mirror of
https://github.com/openclaw/clawhub.git
synced 2026-08-14 08:52:21 +00:00
Compare commits
90
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
53214abd08 | ||
|
|
4a6f4391c4 | ||
|
|
aa0a97bd35 | ||
|
|
311bf1a88a | ||
|
|
8343f0bb23 | ||
|
|
3b73a09d36 | ||
|
|
412249d2d1 | ||
|
|
480125d859 | ||
|
|
0ab1d1e051 | ||
|
|
85374fa44b | ||
|
|
c319e46c8b | ||
|
|
f17087d1f4 | ||
|
|
da4469e1e0 | ||
|
|
42a4648475 | ||
|
|
edc8ec274b | ||
|
|
15b1a05fee | ||
|
|
1e216c03c6 | ||
|
|
0a0b2e6cb1 | ||
|
|
ac6770acff | ||
|
|
bfc87e5932 | ||
|
|
3e1bd19a45 | ||
|
|
d71b747d1c | ||
|
|
4d211fcf73 | ||
|
|
bc5ab8f3e1 | ||
|
|
bb528ea4b9 | ||
|
|
464a04c1e5 | ||
|
|
f4f8e7276f | ||
|
|
3ccf2e05f5 | ||
|
|
0ee2872f5b | ||
|
|
5112d1b215 | ||
|
|
275a170f15 | ||
|
|
17aa24baf9 | ||
|
|
f01476757a | ||
|
|
fe011d00fd | ||
|
|
30ae099825 | ||
|
|
812641342d | ||
|
|
88848c224c | ||
|
|
c107adabac | ||
|
|
38c4a673da | ||
|
|
89933951f5 | ||
|
|
e3523093b1 | ||
|
|
1faf3ee5ed | ||
|
|
286c76a05f | ||
|
|
5745b5a096 | ||
|
|
a060ae3b15 | ||
|
|
77982c5d8e | ||
|
|
4cb84df36a | ||
|
|
d82c8f66c2 | ||
|
|
43fd834d23 | ||
|
|
9b2fc48a55 | ||
|
|
a4dad5dc9d | ||
|
|
84830a268a | ||
|
|
37ef3eb7c5 | ||
|
|
7f987fcc26 | ||
|
|
a0ea45c9a6 | ||
|
|
1f5a782ecd | ||
|
|
c300d4b447 | ||
|
|
c3a6cd7356 | ||
|
|
b75e25c4d6 | ||
|
|
54383665d8 | ||
|
+22 |
697cc1a08f | ||
|
|
652beef9c1 | ||
|
|
146df7b166 | ||
|
|
8f23eb5ee8 | ||
|
|
30b263c27c | ||
|
|
10b704278a | ||
|
|
a289f9cbd9 | ||
|
|
97d68a1be5 | ||
|
|
57e0d39cdc | ||
|
|
1c033868e7 | ||
|
|
4532366009 | ||
|
|
8e9fa44fc2 | ||
|
|
11a66ea148 | ||
|
|
f94e20d4c3 | ||
|
|
71c74f61e2 | ||
|
|
e2592684ed | ||
|
|
79c9381201 | ||
|
|
6a5712fdb6 | ||
|
|
a85faf76ac | ||
|
|
c0a04210e9 | ||
|
|
5adb334cb2 | ||
|
|
182ec8741f | ||
|
|
bafd17b00a | ||
|
|
802ee58054 | ||
|
|
0e83ba00b9 | ||
|
|
3326a5c838 | ||
|
|
eb9a67f2af | ||
|
|
1ae0498595 | ||
|
|
f1a5254755 | ||
|
|
de2542e391 |
@@ -0,0 +1,33 @@
|
||||
name: "Security Gate: Secret Scanning"
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [main, master]
|
||||
|
||||
jobs:
|
||||
trufflehog:
|
||||
name: Scan for Verified Secrets
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read # Required to scan the code in the PR
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0 # necessary to support the scoping requirements below
|
||||
|
||||
- name: TruffleHog OSS
|
||||
id: trufflehog
|
||||
uses: trufflesecurity/trufflehog@e64309e4514a601c7d23f336688782a229a4a754 # Pin to current stable
|
||||
with:
|
||||
path: ./
|
||||
base: ${{ github.event.pull_request.base.sha }} # scope it to the committed files
|
||||
head: ${{ github.event.pull_request.head.sha }}
|
||||
extra_args: --only-verified --debug
|
||||
|
||||
- name: Notify on Failure
|
||||
if: steps.trufflehog.outcome == 'failure'
|
||||
run: |
|
||||
echo "::error::Verified secrets found! This PR contains live credentials that must be rotated immediately."
|
||||
echo "::notice::If these secrets are already in the commit history, they cannot be removed via a simple removal commit/push. A repository owner can contact GitHub Support to purge the cached data: https://support.github.com/contact/private-information"
|
||||
exit 1
|
||||
@@ -4,18 +4,43 @@
|
||||
|
||||
### Added
|
||||
- Admin: add manual unban for banned users (clears `deletedAt` + `banReason`, audit log entry). Revoked API tokens stay revoked.
|
||||
- Admin: bulk restore skills from GitHub backup; reclaim squatted slugs via v1 endpoints + internal tooling (#298) (thanks @autogame-17).
|
||||
- Users: add `trustedPublisher` flag and admin mutations to bypass pending-scan auto-hide for trusted publishers (#298) (thanks @autogame-17).
|
||||
- Skills/Web: show skill owner avatar + handle on skill cards, lists, and detail pages (#312) (thanks @ianalloway).
|
||||
- Skills/Web: add file viewer for skill version files on detail page (#44) (thanks @regenrek).
|
||||
- CLI: add `uninstall` command for skills (#241) (thanks @superlowburn).
|
||||
- CI/Security: add TruffleHog pull-request scanning for verified leaked credentials (#505) (thanks @akses0).
|
||||
|
||||
### Changed
|
||||
- Quality gate: language-aware word counting (`Intl.Segmenter`) and new `cjkChars` signal to reduce false rejects for non-Latin docs.
|
||||
- Jobs: run skill stat event processing every 5 minutes (was 15).
|
||||
- API performance: batch resolve skill/soul tags in v1 list/get endpoints (fewer action->query round-trips) (#112) (thanks @mkrokosz).
|
||||
- Skills: reserve deleted slugs for prior owners (90-day cooldown) to prevent squatting; add admin reclaim flow (#298) (thanks @autogame-17).
|
||||
- Moderation: ban flow soft-deletes owned skills (reversible) and removes them from vector search (#298) (thanks @autogame-17).
|
||||
- LLM helpers: centralize OpenAI Responses text extraction for changelog/summary/eval flows (#502) (thanks @ianalloway).
|
||||
- Rate limiting: apply authenticated quotas by user bucket (vs shared IP), emit delay-based reset headers, and improve CLI 429 guidance/retries (#412) (thanks @lc0rp).
|
||||
- Search/listing performance: cut embedding hydration and badge read bandwidth via `embeddingSkillMap` + denormalized skill badges; shift stat-doc sync to low-frequency cron (#441) (thanks @sethconvex).
|
||||
|
||||
### Fixed
|
||||
- Admin API: `POST /api/v1/users/reclaim` now performs non-destructive root-slug owner transfer
|
||||
(preserves existing skill versions/stats/metadata) and clears active slug reservations.
|
||||
- Users: sync handle on ensure when GitHub login changes (#293) (thanks @christianhpoe).
|
||||
- Users/Auth: throttle GitHub profile sync on login; also sync avatar when it changes (#312) (thanks @ianalloway).
|
||||
- Upload gate: fetch GitHub account age by immutable account ID (prevents username swaps) (#116) (thanks @mkrokosz).
|
||||
- VT fallback: activate only VT-pending hidden skills when scans are unavailable/stale; keep quality/scanner-blocked skills hidden (#300) (thanks @superlowburn).
|
||||
- API: return proper status codes for delete/undelete errors (#35) (thanks @sergical).
|
||||
- API: for owners, return clearer status/messages for hidden/soft-deleted skills instead of a generic 404.
|
||||
- Web: allow copying OpenClaw scan summary text (thanks @borisolver, #322).
|
||||
- HTTP/CORS: add preflight handler + include CORS headers on API/download errors; CLI: include auth token for owner-visible installs/updates (#146) (thanks @Grenghis-Khan).
|
||||
- CLI: clarify `logout` only removes the local token; token remains valid until revoked in the web UI (#166) (thanks @aronchick).
|
||||
- CLI: validate skill slugs used for filesystem operations (prevents path traversal) (#241) (thanks @superlowburn).
|
||||
- Skills: keep global sorting across pagination on `/skills` (thanks @CodeBBakGoSu, #98).
|
||||
- Skills: allow updating skill description/summary from frontmatter on subsequent publishes (#312) (thanks @ianalloway).
|
||||
- Skills/Web: prevent filtered pagination dead-ends and loading-state flicker on `/skills`; move highlighted browse filtering into server list query (#339) (thanks @Marvae).
|
||||
- Web: align `/skills` total count with public visibility and format header count (thanks @rknoche6, #76).
|
||||
- Skills/Web: centralize public visibility checks and keep `globalStats` skill counts in sync incrementally; remove duplicate `/skills` default-sort fallback and share browse test mocks (thanks @rknoche6, #76).
|
||||
- Moderation: clear stale `flagged.suspicious` flags when VirusTotal rescans improve to clean verdicts (#418) (thanks @Phineas1500).
|
||||
- CLI: respect `HTTPS_PROXY`/`HTTP_PROXY`/`NO_PROXY` env vars for outbound registry requests, with troubleshooting docs (#363) (thanks @kerrypotter).
|
||||
|
||||
## 0.6.1 - 2026-02-13
|
||||
|
||||
|
||||
@@ -13,6 +13,7 @@ onlycrabs.ai is the **SOUL.md registry**: publish and share system lore the same
|
||||
|
||||
Live: `https://clawhub.ai`
|
||||
onlycrabs.ai: `https://onlycrabs.ai`
|
||||
Vision: [`VISION.md`](VISION.md)
|
||||
|
||||
## What you can do with it
|
||||
|
||||
@@ -37,6 +38,25 @@ onlycrabs.ai: `https://onlycrabs.ai`
|
||||
- Search: OpenAI embeddings (`text-embedding-3-small`) + Convex vector search.
|
||||
- API schema + routes: `packages/schema` (`clawhub-schema`).
|
||||
|
||||
## CLI
|
||||
|
||||
Common CLI flows:
|
||||
|
||||
- Auth: `clawhub login`, `clawhub whoami`
|
||||
- Discover: `clawhub search ...`, `clawhub explore`
|
||||
- Manage local installs: `clawhub install <slug>`, `clawhub uninstall <slug>`, `clawhub list`, `clawhub update --all`
|
||||
- Inspect without installing: `clawhub inspect <slug>`
|
||||
- Publish/sync: `clawhub publish <path>`, `clawhub sync`
|
||||
|
||||
Docs: `docs/quickstart.md`, `docs/cli.md`.
|
||||
|
||||
### Removal permissions
|
||||
|
||||
- `clawhub uninstall <slug>` only removes a local install on your machine.
|
||||
- Uploaded registry skills use soft-delete/restore (`clawhub delete <slug>` / `clawhub undelete <slug>` or API equivalents).
|
||||
- Soft-delete/restore is allowed for the skill owner, moderators, and admins.
|
||||
- Hard delete is admin-only (management tools / ban flows).
|
||||
|
||||
|
||||
## Telemetry
|
||||
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
## OpenClaw Vision
|
||||
|
||||
OpenClaw is the AI that actually does things.
|
||||
It runs on your devices, in your channels, with your rules.
|
||||
|
||||
This document explains the current state and direction of the project.
|
||||
We are still early, so iteration is fast.
|
||||
Project overview and developer docs: [`README.md`](README.md)
|
||||
|
||||
OpenClaw started as my personal playground to learn AI and build something genuinely useful:
|
||||
an assistant that can run real tasks on my computer.
|
||||
It evolved through several names and shells: Warelay -> Clawdbot -> Moltbot -> OpenClaw.
|
||||
|
||||
The goal? A personal assistant that's easy to use, supports a wide range of platforms, and respects your privacy and security.
|
||||
|
||||
The current focus is:
|
||||
|
||||
Priority:
|
||||
- Security and safe defaults
|
||||
- Bug fixes and stability
|
||||
- Setup reliability and first-run UX
|
||||
|
||||
Next priorities:
|
||||
- Supporting all major model providers
|
||||
- Improving support for major messaging channels (and adding a few high-demand ones)
|
||||
- Performance and test infrastructure
|
||||
- Better computer-use and agent harness capabilities
|
||||
- Ergonomics across CLI and web frontend
|
||||
- Companion apps on macOS, iOS, Android, Windows, and Linux
|
||||
|
||||
## Security
|
||||
|
||||
Security in OpenClaw is a deliberate tradeoff: strong defaults without killing capability.
|
||||
The goal is to stay powerful for real work while making risky paths explicit and operator-controlled.
|
||||
|
||||
Canonical security policy and reporting:
|
||||
- https://github.com/openclaw/openclaw/blob/main/SECURITY.md
|
||||
|
||||
We prioritize secure defaults, but we also expose clear knobs for trusted high-power workflows.
|
||||
|
||||
## Plugins & Memory
|
||||
|
||||
OpenClaw has an extensive plugin API.
|
||||
Core stays lean; optional capability should usually ship as plugins.
|
||||
|
||||
Preferred plugin path is npm package distribution plus local extension loading for development.
|
||||
If you build a plugin, please host and maintain it in your own repository.
|
||||
The bar for adding optional plugins to core is intentionally high.
|
||||
|
||||
Memory is a special plugin slot where only one memory plugin can be active at a time.
|
||||
Today we ship multiple memory options; over time we plan to converge on one recommended default path.
|
||||
|
||||
### Skills
|
||||
|
||||
We still ship some bundled skills for baseline UX.
|
||||
New skills should be published to ClawHub first (`clawhub.ai`), not added to core by default.
|
||||
Core skill additions should be rare and require a strong product or security reason.
|
||||
|
||||
### MCP Support
|
||||
|
||||
OpenClaw supports MCP through `mcporter`: https://github.com/steipete/mcporter
|
||||
|
||||
This keeps MCP integration flexible and decoupled from core runtime:
|
||||
- add or change MCP servers without restarting the gateway
|
||||
- keep core tool/context surface lean
|
||||
- reduce MCP churn impact on core stability and security
|
||||
|
||||
For now, we prefer this bridge model over building first-class MCP runtime into core.
|
||||
If there is an MCP server or feature `mcporter` does not support yet, please open an issue there.
|
||||
|
||||
### Setup
|
||||
|
||||
OpenClaw is currently terminal-first by design.
|
||||
This keeps setup explicit: users see docs, auth, permissions, and security posture up front.
|
||||
|
||||
Long term, we want easier onboarding flows as hardening matures.
|
||||
We do not want convenience wrappers that hide critical security decisions from users.
|
||||
|
||||
### Why TypeScript?
|
||||
|
||||
OpenClaw is primarily an orchestration system: prompts, tools, protocols, and integrations.
|
||||
TypeScript was chosen to keep OpenClaw hackable by default.
|
||||
It is widely known, fast to iterate in, and easy to read, modify, and extend.
|
||||
|
||||
## What We Will Not Merge (For Now)
|
||||
|
||||
- New core skills when they can live on ClawHub
|
||||
- Commercial service integrations that do not clearly fit the model-provider category
|
||||
- Wrapper channels around already supported channels without a clear capability or security gap
|
||||
- First-class MCP runtime in core when `mcporter` already provides the integration path
|
||||
- Heavy orchestration layers that duplicate existing agent and tool infrastructure
|
||||
|
||||
This list is a roadmap guardrail, not a law of physics.
|
||||
Strong user demand and strong technical rationale can change it.
|
||||
@@ -63,7 +63,7 @@
|
||||
},
|
||||
"packages/clawdhub": {
|
||||
"name": "clawhub",
|
||||
"version": "0.6.1",
|
||||
"version": "0.7.0",
|
||||
"bin": {
|
||||
"clawhub": "bin/clawdhub.js",
|
||||
"clawdhub": "bin/clawdhub.js",
|
||||
|
||||
Vendored
+32
@@ -16,29 +16,45 @@ import type * as devSeedExtra from "../devSeedExtra.js";
|
||||
import type * as downloads from "../downloads.js";
|
||||
import type * as githubBackups from "../githubBackups.js";
|
||||
import type * as githubBackupsNode from "../githubBackupsNode.js";
|
||||
import type * as githubIdentity from "../githubIdentity.js";
|
||||
import type * as githubImport from "../githubImport.js";
|
||||
import type * as githubRestore from "../githubRestore.js";
|
||||
import type * as githubRestoreMutations from "../githubRestoreMutations.js";
|
||||
import type * as githubSoulBackups from "../githubSoulBackups.js";
|
||||
import type * as githubSoulBackupsNode from "../githubSoulBackupsNode.js";
|
||||
import type * as http from "../http.js";
|
||||
import type * as httpApi from "../httpApi.js";
|
||||
import type * as httpApiV1 from "../httpApiV1.js";
|
||||
import type * as httpApiV1_shared from "../httpApiV1/shared.js";
|
||||
import type * as httpApiV1_skillsV1 from "../httpApiV1/skillsV1.js";
|
||||
import type * as httpApiV1_soulsV1 from "../httpApiV1/soulsV1.js";
|
||||
import type * as httpApiV1_starsV1 from "../httpApiV1/starsV1.js";
|
||||
import type * as httpApiV1_usersV1 from "../httpApiV1/usersV1.js";
|
||||
import type * as httpApiV1_whoamiV1 from "../httpApiV1/whoamiV1.js";
|
||||
import type * as httpPreflight from "../httpPreflight.js";
|
||||
import type * as leaderboards from "../leaderboards.js";
|
||||
import type * as lib_access from "../lib/access.js";
|
||||
import type * as lib_apiTokenAuth from "../lib/apiTokenAuth.js";
|
||||
import type * as lib_badges from "../lib/badges.js";
|
||||
import type * as lib_batching from "../lib/batching.js";
|
||||
import type * as lib_changelog from "../lib/changelog.js";
|
||||
import type * as lib_contentTypes from "../lib/contentTypes.js";
|
||||
import type * as lib_embeddingVisibility from "../lib/embeddingVisibility.js";
|
||||
import type * as lib_embeddings from "../lib/embeddings.js";
|
||||
import type * as lib_githubAccount from "../lib/githubAccount.js";
|
||||
import type * as lib_githubBackup from "../lib/githubBackup.js";
|
||||
import type * as lib_githubIdentity from "../lib/githubIdentity.js";
|
||||
import type * as lib_githubImport from "../lib/githubImport.js";
|
||||
import type * as lib_githubProfileSync from "../lib/githubProfileSync.js";
|
||||
import type * as lib_githubRestoreHelpers from "../lib/githubRestoreHelpers.js";
|
||||
import type * as lib_githubSoulBackup from "../lib/githubSoulBackup.js";
|
||||
import type * as lib_globalStats from "../lib/globalStats.js";
|
||||
import type * as lib_httpHeaders from "../lib/httpHeaders.js";
|
||||
import type * as lib_httpRateLimit from "../lib/httpRateLimit.js";
|
||||
import type * as lib_leaderboards from "../lib/leaderboards.js";
|
||||
import type * as lib_moderation from "../lib/moderation.js";
|
||||
import type * as lib_public from "../lib/public.js";
|
||||
import type * as lib_reservedSlugs from "../lib/reservedSlugs.js";
|
||||
import type * as lib_searchText from "../lib/searchText.js";
|
||||
import type * as lib_securityPrompt from "../lib/securityPrompt.js";
|
||||
import type * as lib_skillBackfill from "../lib/skillBackfill.js";
|
||||
@@ -90,29 +106,45 @@ declare const fullApi: ApiFromModules<{
|
||||
downloads: typeof downloads;
|
||||
githubBackups: typeof githubBackups;
|
||||
githubBackupsNode: typeof githubBackupsNode;
|
||||
githubIdentity: typeof githubIdentity;
|
||||
githubImport: typeof githubImport;
|
||||
githubRestore: typeof githubRestore;
|
||||
githubRestoreMutations: typeof githubRestoreMutations;
|
||||
githubSoulBackups: typeof githubSoulBackups;
|
||||
githubSoulBackupsNode: typeof githubSoulBackupsNode;
|
||||
http: typeof http;
|
||||
httpApi: typeof httpApi;
|
||||
httpApiV1: typeof httpApiV1;
|
||||
"httpApiV1/shared": typeof httpApiV1_shared;
|
||||
"httpApiV1/skillsV1": typeof httpApiV1_skillsV1;
|
||||
"httpApiV1/soulsV1": typeof httpApiV1_soulsV1;
|
||||
"httpApiV1/starsV1": typeof httpApiV1_starsV1;
|
||||
"httpApiV1/usersV1": typeof httpApiV1_usersV1;
|
||||
"httpApiV1/whoamiV1": typeof httpApiV1_whoamiV1;
|
||||
httpPreflight: typeof httpPreflight;
|
||||
leaderboards: typeof leaderboards;
|
||||
"lib/access": typeof lib_access;
|
||||
"lib/apiTokenAuth": typeof lib_apiTokenAuth;
|
||||
"lib/badges": typeof lib_badges;
|
||||
"lib/batching": typeof lib_batching;
|
||||
"lib/changelog": typeof lib_changelog;
|
||||
"lib/contentTypes": typeof lib_contentTypes;
|
||||
"lib/embeddingVisibility": typeof lib_embeddingVisibility;
|
||||
"lib/embeddings": typeof lib_embeddings;
|
||||
"lib/githubAccount": typeof lib_githubAccount;
|
||||
"lib/githubBackup": typeof lib_githubBackup;
|
||||
"lib/githubIdentity": typeof lib_githubIdentity;
|
||||
"lib/githubImport": typeof lib_githubImport;
|
||||
"lib/githubProfileSync": typeof lib_githubProfileSync;
|
||||
"lib/githubRestoreHelpers": typeof lib_githubRestoreHelpers;
|
||||
"lib/githubSoulBackup": typeof lib_githubSoulBackup;
|
||||
"lib/globalStats": typeof lib_globalStats;
|
||||
"lib/httpHeaders": typeof lib_httpHeaders;
|
||||
"lib/httpRateLimit": typeof lib_httpRateLimit;
|
||||
"lib/leaderboards": typeof lib_leaderboards;
|
||||
"lib/moderation": typeof lib_moderation;
|
||||
"lib/public": typeof lib_public;
|
||||
"lib/reservedSlugs": typeof lib_reservedSlugs;
|
||||
"lib/searchText": typeof lib_searchText;
|
||||
"lib/securityPrompt": typeof lib_securityPrompt;
|
||||
"lib/skillBackfill": typeof lib_skillBackfill;
|
||||
|
||||
+19
-3
@@ -2,7 +2,9 @@ import GitHub from '@auth/core/providers/github'
|
||||
import { convexAuth } from '@convex-dev/auth/server'
|
||||
import type { GenericMutationCtx } from 'convex/server'
|
||||
import { ConvexError } from 'convex/values'
|
||||
import { internal } from './_generated/api'
|
||||
import type { DataModel, Id } from './_generated/dataModel'
|
||||
import { shouldScheduleGitHubProfileSync } from './lib/githubProfileSync'
|
||||
|
||||
export const BANNED_REAUTH_MESSAGE =
|
||||
'Your account has been banned for uploading malicious skills. If you believe this is a mistake, please contact security@openclaw.ai and we will work with you to restore access.'
|
||||
@@ -14,8 +16,9 @@ const REAUTH_BLOCKING_BAN_ACTIONS = new Set(['user.ban', 'user.autoban.malware']
|
||||
export async function handleDeletedUserSignIn(
|
||||
ctx: GenericMutationCtx<DataModel>,
|
||||
args: { userId: Id<'users'>; existingUserId: Id<'users'> | null },
|
||||
userOverride?: { deletedAt?: number; deactivatedAt?: number; purgedAt?: number } | null,
|
||||
) {
|
||||
const user = await ctx.db.get(args.userId)
|
||||
const user = userOverride !== undefined ? userOverride : await ctx.db.get(args.userId)
|
||||
if (!user?.deletedAt && !user?.deactivatedAt) return
|
||||
|
||||
// Verify that the incoming identity matches the existing account to prevent bypass.
|
||||
@@ -69,14 +72,27 @@ export const { auth, signIn, signOut, store, isAuthenticated } = convexAuth({
|
||||
],
|
||||
callbacks: {
|
||||
/**
|
||||
* Block sign-in for deleted/deactivated users.
|
||||
* Block sign-in for deleted/deactivated users and sync GitHub profile.
|
||||
*
|
||||
* Performance note: This callback runs on every OAuth sign-in, but the
|
||||
* audit log query ONLY executes when a legacy deleted user attempts to sign
|
||||
* in (user.deletedAt is set). For active users, this is a single field check.
|
||||
*
|
||||
* The GitHub profile sync is scheduled as a background action to handle
|
||||
* the case where a user renames their GitHub account (fixes #303).
|
||||
*/
|
||||
async afterUserCreatedOrUpdated(ctx, args) {
|
||||
await handleDeletedUserSignIn(ctx, args)
|
||||
const user = await ctx.db.get(args.userId)
|
||||
await handleDeletedUserSignIn(ctx, args, user)
|
||||
|
||||
// Schedule GitHub profile sync to handle username renames (fixes #303)
|
||||
// This runs as a background action so it doesn't block sign-in
|
||||
const now = Date.now()
|
||||
if (shouldScheduleGitHubProfileSync(user, now)) {
|
||||
await ctx.scheduler.runAfter(0, internal.users.syncGitHubProfileAction, {
|
||||
userId: args.userId,
|
||||
})
|
||||
}
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
+21
-2
@@ -19,18 +19,37 @@ crons.interval(
|
||||
|
||||
crons.interval(
|
||||
'skill-stats-backfill',
|
||||
{ minutes: 10 },
|
||||
{ hours: 6 },
|
||||
internal.statsMaintenance.runSkillStatBackfillInternal,
|
||||
{ batchSize: 200, maxBatches: 5 },
|
||||
)
|
||||
|
||||
// Runs frequently to keep dailyStats/trending accurate,
|
||||
// but does NOT patch skill documents (only writes to skillDailyStats).
|
||||
crons.interval(
|
||||
'skill-stat-events',
|
||||
{ minutes: 5 },
|
||||
{ minutes: 15 },
|
||||
internal.skillStatEvents.processSkillStatEventsAction,
|
||||
{},
|
||||
)
|
||||
|
||||
// Syncs accumulated stat deltas to skill documents every 6 hours.
|
||||
// Runs infrequently to avoid thundering-herd reactive query invalidation.
|
||||
// Uses processedAt field to track progress (independent of the action cursor).
|
||||
crons.interval(
|
||||
'skill-doc-stat-sync',
|
||||
{ hours: 6 },
|
||||
internal.skillStatEvents.processSkillStatEventsInternal,
|
||||
{ batchSize: 500 },
|
||||
)
|
||||
|
||||
crons.interval(
|
||||
'global-stats-update',
|
||||
{ minutes: 60 },
|
||||
internal.statsMaintenance.updateGlobalStatsInternal,
|
||||
{},
|
||||
)
|
||||
|
||||
crons.interval('vt-pending-scans', { minutes: 5 }, internal.vt.pollPendingScans, { batchSize: 100 })
|
||||
|
||||
crons.interval('vt-cache-backfill', { minutes: 30 }, internal.vt.backfillActiveSkillsVTCache, {
|
||||
|
||||
@@ -435,6 +435,7 @@ export const seedSkillMutation = internalMutation({
|
||||
visibility: 'latest-approved',
|
||||
updatedAt: now,
|
||||
})
|
||||
await ctx.db.insert('embeddingSkillMap', { embeddingId, skillId })
|
||||
|
||||
await ctx.db.patch(skillId, {
|
||||
latestVersionId: versionId,
|
||||
|
||||
@@ -39,6 +39,7 @@ export type SyncGitHubBackupsResult = {
|
||||
skillsScanned: number
|
||||
skillsSkipped: number
|
||||
skillsBackedUp: number
|
||||
skillsDeleted: number
|
||||
skillsMissingVersion: number
|
||||
skillsMissingOwner: number
|
||||
errors: number
|
||||
|
||||
@@ -7,9 +7,12 @@ import type { ActionCtx } from './_generated/server'
|
||||
import { internalAction } from './_generated/server'
|
||||
import {
|
||||
backupSkillToGitHub,
|
||||
deleteGitHubSkillBackup,
|
||||
fetchGitHubSkillMeta,
|
||||
getGitHubBackupContext,
|
||||
isGitHubBackupConfigured,
|
||||
listGitHubSkillBackupEntries,
|
||||
normalizeOwner,
|
||||
} from './lib/githubBackup'
|
||||
|
||||
const DEFAULT_BATCH_SIZE = 50
|
||||
@@ -35,6 +38,7 @@ export type GitHubBackupSyncStats = {
|
||||
skillsScanned: number
|
||||
skillsSkipped: number
|
||||
skillsBackedUp: number
|
||||
skillsDeleted: number
|
||||
skillsMissingVersion: number
|
||||
skillsMissingOwner: number
|
||||
errors: number
|
||||
@@ -87,6 +91,7 @@ export async function syncGitHubBackupsInternalHandler(
|
||||
skillsScanned: 0,
|
||||
skillsSkipped: 0,
|
||||
skillsBackedUp: 0,
|
||||
skillsDeleted: 0,
|
||||
skillsMissingVersion: 0,
|
||||
skillsMissingOwner: 0,
|
||||
errors: 0,
|
||||
@@ -166,9 +171,69 @@ export async function syncGitHubBackupsInternalHandler(
|
||||
if (isDone) break
|
||||
}
|
||||
|
||||
await pruneDeletedSkillBackups(ctx, context, dryRun, stats)
|
||||
|
||||
return { stats, cursor, isDone }
|
||||
}
|
||||
|
||||
async function pruneDeletedSkillBackups(
|
||||
ctx: ActionCtx,
|
||||
context: Awaited<ReturnType<typeof getGitHubBackupContext>>,
|
||||
dryRun: boolean,
|
||||
stats: GitHubBackupSyncStats,
|
||||
) {
|
||||
let entries: Awaited<ReturnType<typeof listGitHubSkillBackupEntries>>
|
||||
try {
|
||||
entries = await listGitHubSkillBackupEntries(context)
|
||||
} catch (error) {
|
||||
console.error('GitHub backup cleanup list failed', error)
|
||||
stats.errors += 1
|
||||
return
|
||||
}
|
||||
|
||||
for (const entry of entries) {
|
||||
try {
|
||||
const skill = (await ctx.runQuery(internal.skills.getSkillBySlugInternal, {
|
||||
slug: entry.slug,
|
||||
})) as Doc<'skills'> | null
|
||||
if (!skill || skill.softDeletedAt) {
|
||||
await deleteBackupIfNeeded(context, entry, dryRun, stats)
|
||||
continue
|
||||
}
|
||||
|
||||
const owner = (await ctx.runQuery(internal.users.getByIdInternal, {
|
||||
userId: skill.ownerUserId,
|
||||
})) as Doc<'users'> | null
|
||||
if (!owner || owner.deletedAt || owner.deactivatedAt) {
|
||||
await deleteBackupIfNeeded(context, entry, dryRun, stats)
|
||||
continue
|
||||
}
|
||||
|
||||
const ownerHandle = normalizeOwner(owner.handle ?? owner._id)
|
||||
if (ownerHandle !== entry.owner) {
|
||||
await deleteBackupIfNeeded(context, entry, dryRun, stats)
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('GitHub backup cleanup failed', error)
|
||||
stats.errors += 1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function deleteBackupIfNeeded(
|
||||
context: Awaited<ReturnType<typeof getGitHubBackupContext>>,
|
||||
entry: Awaited<ReturnType<typeof listGitHubSkillBackupEntries>>[number],
|
||||
dryRun: boolean,
|
||||
stats: GitHubBackupSyncStats,
|
||||
) {
|
||||
const result = dryRun
|
||||
? { deleted: true as const }
|
||||
: await deleteGitHubSkillBackup(context, entry.owner, entry.slug)
|
||||
if (result.deleted) {
|
||||
stats.skillsDeleted += 1
|
||||
}
|
||||
}
|
||||
|
||||
export const syncGitHubBackupsInternal = internalAction({
|
||||
args: {
|
||||
dryRun: v.optional(v.boolean()),
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
import { v } from 'convex/values'
|
||||
import { internalQuery } from './_generated/server'
|
||||
import { getGitHubProviderAccountId } from './lib/githubIdentity'
|
||||
|
||||
export const getGitHubProviderAccountIdInternal = internalQuery({
|
||||
args: { userId: v.id('users') },
|
||||
handler: async (ctx, args) => getGitHubProviderAccountId(ctx, args.userId),
|
||||
})
|
||||
|
||||
@@ -0,0 +1,216 @@
|
||||
'use node'
|
||||
|
||||
import { v } from 'convex/values'
|
||||
import { internal } from './_generated/api'
|
||||
import type { Doc, Id } from './_generated/dataModel'
|
||||
import { internalAction } from './_generated/server'
|
||||
import {
|
||||
fetchGitHubSkillMeta,
|
||||
getGitHubBackupContext,
|
||||
isGitHubBackupConfigured,
|
||||
} from './lib/githubBackup'
|
||||
import { assertAdmin } from './lib/access'
|
||||
import {
|
||||
listGitHubBackupFiles,
|
||||
readGitHubBackupFile,
|
||||
} from './lib/githubRestoreHelpers'
|
||||
import { publishVersionForUser } from './lib/skillPublish'
|
||||
import { guessContentTypeForPath } from './lib/contentTypes'
|
||||
|
||||
type RestoreResult = {
|
||||
slug: string
|
||||
status: 'restored' | 'slug_conflict' | 'already_exists' | 'no_backup' | 'error'
|
||||
detail?: string
|
||||
}
|
||||
|
||||
type BulkRestoreResult = {
|
||||
results: RestoreResult[]
|
||||
totalRestored: number
|
||||
totalConflicts: number
|
||||
totalSkipped: number
|
||||
totalErrors: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Admin-only: restore a single skill from GitHub backup.
|
||||
* Reads the backup files from the GitHub repo and re-creates the skill in the database.
|
||||
*/
|
||||
export const restoreSkillFromBackup = internalAction({
|
||||
args: {
|
||||
actorUserId: v.id('users'),
|
||||
ownerHandle: v.string(),
|
||||
ownerUserId: v.id('users'),
|
||||
slug: v.string(),
|
||||
forceOverwriteSquatter: v.optional(v.boolean()),
|
||||
},
|
||||
handler: async (ctx, args): Promise<RestoreResult> => {
|
||||
try {
|
||||
const actor = await ctx.runQuery(internal.users.getByIdInternal, {
|
||||
userId: args.actorUserId,
|
||||
})
|
||||
if (!actor || actor.deletedAt || actor.deactivatedAt) {
|
||||
return { slug: args.slug, status: 'error', detail: 'Actor not found' }
|
||||
}
|
||||
assertAdmin(actor as Doc<'users'>)
|
||||
|
||||
if (!isGitHubBackupConfigured()) {
|
||||
return { slug: args.slug, status: 'error', detail: 'GitHub backup not configured' }
|
||||
}
|
||||
|
||||
const ghContext = await getGitHubBackupContext()
|
||||
|
||||
// Check if skill already exists in the DB
|
||||
const existingSkill = (await ctx.runQuery(internal.skills.getSkillBySlugInternal, {
|
||||
slug: args.slug,
|
||||
})) as Doc<'skills'> | null
|
||||
|
||||
if (existingSkill) {
|
||||
if (existingSkill.ownerUserId === args.ownerUserId) {
|
||||
return { slug: args.slug, status: 'already_exists', detail: 'Skill already owned by user' }
|
||||
}
|
||||
|
||||
if (!args.forceOverwriteSquatter) {
|
||||
return {
|
||||
slug: args.slug,
|
||||
status: 'slug_conflict',
|
||||
detail: `Slug occupied by another user. Set forceOverwriteSquatter=true to reclaim.`,
|
||||
}
|
||||
}
|
||||
|
||||
// Free the slug in-transaction by renaming the squatter, then enqueue cleanup.
|
||||
await ctx.runMutation(internal.githubRestoreMutations.evictSquatterSkillForRestoreInternal, {
|
||||
actorUserId: args.actorUserId,
|
||||
slug: args.slug,
|
||||
rightfulOwnerUserId: args.ownerUserId,
|
||||
})
|
||||
}
|
||||
|
||||
// Fetch metadata from GitHub backup
|
||||
const meta = await fetchGitHubSkillMeta(ghContext, args.ownerHandle, args.slug)
|
||||
if (!meta) {
|
||||
return { slug: args.slug, status: 'no_backup', detail: 'No backup found in GitHub repo' }
|
||||
}
|
||||
|
||||
// Read the actual files from the backup
|
||||
const backupFiles = await listGitHubBackupFiles(ghContext, args.ownerHandle, args.slug)
|
||||
if (backupFiles.length === 0) {
|
||||
return { slug: args.slug, status: 'no_backup', detail: 'Backup has no files' }
|
||||
}
|
||||
|
||||
// Download and store each file in Convex storage
|
||||
const storedFiles: Array<{
|
||||
path: string
|
||||
size: number
|
||||
storageId: Id<'_storage'>
|
||||
sha256: string
|
||||
contentType: string
|
||||
}> = []
|
||||
|
||||
for (const filePath of backupFiles) {
|
||||
const fileContent = await readGitHubBackupFile(ghContext, args.ownerHandle, args.slug, filePath)
|
||||
if (!fileContent) continue
|
||||
|
||||
const sha256 = await sha256Hex(fileContent)
|
||||
const contentType = guessContentTypeForPath(filePath)
|
||||
const blob = new Blob([Buffer.from(fileContent)], { type: contentType })
|
||||
const storageId = await ctx.storage.store(blob)
|
||||
|
||||
storedFiles.push({
|
||||
path: filePath,
|
||||
size: fileContent.byteLength,
|
||||
storageId,
|
||||
sha256,
|
||||
contentType,
|
||||
})
|
||||
}
|
||||
|
||||
if (storedFiles.length === 0) {
|
||||
return { slug: args.slug, status: 'error', detail: 'Could not download any backup files' }
|
||||
}
|
||||
|
||||
await publishVersionForUser(
|
||||
ctx,
|
||||
args.ownerUserId,
|
||||
{
|
||||
slug: args.slug,
|
||||
displayName: meta.displayName,
|
||||
version: meta.latest.version,
|
||||
changelog: 'Restored from GitHub backup',
|
||||
files: storedFiles,
|
||||
},
|
||||
{
|
||||
bypassGitHubAccountAge: true,
|
||||
bypassNewSkillRateLimit: true,
|
||||
bypassQualityGate: true,
|
||||
skipBackup: true,
|
||||
skipWebhook: true,
|
||||
},
|
||||
)
|
||||
|
||||
return { slug: args.slug, status: 'restored' }
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : 'Unknown error'
|
||||
console.error(`[restore] Failed to restore ${args.slug}:`, message)
|
||||
return { slug: args.slug, status: 'error', detail: message }
|
||||
}
|
||||
},
|
||||
})
|
||||
|
||||
/**
|
||||
* Admin-only: bulk restore all skills for a user from GitHub backup.
|
||||
*/
|
||||
export const restoreUserSkillsFromBackup = internalAction({
|
||||
args: {
|
||||
actorUserId: v.id('users'),
|
||||
ownerHandle: v.string(),
|
||||
ownerUserId: v.id('users'),
|
||||
slugs: v.array(v.string()),
|
||||
forceOverwriteSquatter: v.optional(v.boolean()),
|
||||
},
|
||||
handler: async (ctx, args): Promise<BulkRestoreResult> => {
|
||||
const results: RestoreResult[] = []
|
||||
let totalRestored = 0
|
||||
let totalConflicts = 0
|
||||
let totalSkipped = 0
|
||||
let totalErrors = 0
|
||||
|
||||
for (const slug of args.slugs) {
|
||||
const result = (await ctx.runAction(internal.githubRestore.restoreSkillFromBackup, {
|
||||
actorUserId: args.actorUserId,
|
||||
ownerHandle: args.ownerHandle,
|
||||
ownerUserId: args.ownerUserId,
|
||||
slug,
|
||||
forceOverwriteSquatter: args.forceOverwriteSquatter,
|
||||
})) as RestoreResult
|
||||
|
||||
results.push(result)
|
||||
|
||||
switch (result.status) {
|
||||
case 'restored':
|
||||
totalRestored += 1
|
||||
break
|
||||
case 'slug_conflict':
|
||||
totalConflicts += 1
|
||||
break
|
||||
case 'already_exists':
|
||||
case 'no_backup':
|
||||
totalSkipped += 1
|
||||
break
|
||||
case 'error':
|
||||
totalErrors += 1
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
return { results, totalRestored, totalConflicts, totalSkipped, totalErrors }
|
||||
},
|
||||
})
|
||||
|
||||
async function sha256Hex(bytes: Uint8Array) {
|
||||
const { createHash } = await import('node:crypto')
|
||||
const hash = createHash('sha256')
|
||||
hash.update(bytes)
|
||||
return hash.digest('hex')
|
||||
}
|
||||
|
||||
// guessContentTypeForPath in lib/contentTypes.ts
|
||||
@@ -0,0 +1,84 @@
|
||||
import { v } from 'convex/values'
|
||||
import { internal } from './_generated/api'
|
||||
import { internalMutation } from './_generated/server'
|
||||
import { assertAdmin } from './lib/access'
|
||||
|
||||
export const evictSquatterSkillForRestoreInternal = internalMutation({
|
||||
args: {
|
||||
actorUserId: v.id('users'),
|
||||
slug: v.string(),
|
||||
rightfulOwnerUserId: v.id('users'),
|
||||
},
|
||||
handler: async (ctx, args) => {
|
||||
const actor = await ctx.db.get(args.actorUserId)
|
||||
if (!actor || actor.deletedAt || actor.deactivatedAt) throw new Error('Actor not found')
|
||||
assertAdmin(actor)
|
||||
|
||||
const slug = args.slug.trim().toLowerCase()
|
||||
if (!slug) throw new Error('Slug required')
|
||||
|
||||
const now = Date.now()
|
||||
|
||||
const existingSkill = await ctx.db
|
||||
.query('skills')
|
||||
.withIndex('by_slug', (q) => q.eq('slug', slug))
|
||||
.unique()
|
||||
if (!existingSkill) return { ok: true as const, action: 'noop' as const }
|
||||
if (existingSkill.ownerUserId === args.rightfulOwnerUserId) {
|
||||
return { ok: true as const, action: 'already_owned' as const }
|
||||
}
|
||||
|
||||
const evictedSlug = buildEvictedSlug(slug, now)
|
||||
|
||||
// Free the slug immediately (same transaction) by renaming the squatter's skill.
|
||||
await ctx.db.patch(existingSkill._id, {
|
||||
slug: evictedSlug,
|
||||
softDeletedAt: now,
|
||||
hiddenAt: existingSkill.hiddenAt ?? now,
|
||||
hiddenBy: existingSkill.hiddenBy ?? actor._id,
|
||||
updatedAt: now,
|
||||
})
|
||||
|
||||
// Remove from vector search ASAP.
|
||||
const embeddings = await ctx.db
|
||||
.query('skillEmbeddings')
|
||||
.withIndex('by_skill', (q) => q.eq('skillId', existingSkill._id))
|
||||
.collect()
|
||||
for (const embedding of embeddings) {
|
||||
await ctx.db.patch(embedding._id, {
|
||||
visibility: 'deleted',
|
||||
updatedAt: now,
|
||||
})
|
||||
}
|
||||
|
||||
// Cleanup the rest asynchronously (versions, fingerprints, installs, etc.)
|
||||
await ctx.scheduler.runAfter(0, internal.skills.hardDeleteInternal, {
|
||||
skillId: existingSkill._id,
|
||||
actorUserId: actor._id,
|
||||
phase: 'versions',
|
||||
})
|
||||
|
||||
await ctx.db.insert('auditLogs', {
|
||||
actorUserId: actor._id,
|
||||
action: 'slug.reclaim.sync',
|
||||
targetType: 'skill',
|
||||
targetId: existingSkill._id,
|
||||
metadata: {
|
||||
slug,
|
||||
evictedSlug,
|
||||
squatterUserId: existingSkill.ownerUserId,
|
||||
rightfulOwnerUserId: args.rightfulOwnerUserId,
|
||||
reason: 'Synchronous eviction during GitHub restore',
|
||||
},
|
||||
createdAt: now,
|
||||
})
|
||||
|
||||
return { ok: true as const, action: 'evicted' as const, evictedSlug }
|
||||
},
|
||||
})
|
||||
|
||||
function buildEvictedSlug(slug: string, now: number) {
|
||||
const suffix = now.toString(36)
|
||||
return `${slug}-evicted-${suffix}`
|
||||
}
|
||||
|
||||
@@ -28,6 +28,12 @@ function isRateLimitArgs(args: unknown): args is RateLimitArgs {
|
||||
)
|
||||
}
|
||||
|
||||
function hasSlugArgs(args: unknown): args is { slug: string } {
|
||||
if (!args || typeof args !== 'object') return false
|
||||
const value = args as Record<string, unknown>
|
||||
return typeof value.slug === 'string'
|
||||
}
|
||||
|
||||
function makeCtx(partial: Record<string, unknown>) {
|
||||
const partialRunQuery =
|
||||
typeof partial.runQuery === 'function'
|
||||
@@ -81,6 +87,124 @@ describe('httpApiV1 handlers', () => {
|
||||
expect(runAction).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('users/restore forbids non-admin api tokens', async () => {
|
||||
const runQuery = vi.fn()
|
||||
const runAction = vi.fn()
|
||||
const runMutation = vi.fn().mockResolvedValue(okRate())
|
||||
vi.mocked(requireApiTokenUser).mockResolvedValue({
|
||||
userId: 'users:actor',
|
||||
user: { _id: 'users:actor', role: 'user' },
|
||||
} as never)
|
||||
|
||||
const response = await __handlers.usersPostRouterV1Handler(
|
||||
makeCtx({ runQuery, runAction, runMutation }),
|
||||
new Request('https://example.com/api/v1/users/restore', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ handle: 'target', slugs: ['a'] }),
|
||||
}),
|
||||
)
|
||||
expect(response.status).toBe(403)
|
||||
expect(runQuery).not.toHaveBeenCalled()
|
||||
expect(runAction).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('users/restore calls restore action for admin', async () => {
|
||||
const runAction = vi.fn().mockResolvedValue({ ok: true, totalRestored: 1, results: [] })
|
||||
const runMutation = vi.fn(async (_mutation: unknown, args: Record<string, unknown>) => {
|
||||
if (isRateLimitArgs(args)) return okRate()
|
||||
return { ok: true }
|
||||
})
|
||||
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
|
||||
if ('handle' in args) return { _id: 'users:target' }
|
||||
return null
|
||||
})
|
||||
vi.mocked(requireApiTokenUser).mockResolvedValue({
|
||||
userId: 'users:admin',
|
||||
user: { _id: 'users:admin', role: 'admin' },
|
||||
} as never)
|
||||
|
||||
const response = await __handlers.usersPostRouterV1Handler(
|
||||
makeCtx({ runQuery, runAction, runMutation }),
|
||||
new Request('https://example.com/api/v1/users/restore', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({
|
||||
handle: 'Target',
|
||||
slugs: ['a', 'b'],
|
||||
forceOverwriteSquatter: true,
|
||||
}),
|
||||
}),
|
||||
)
|
||||
if (response.status !== 200) throw new Error(await response.text())
|
||||
expect(runAction).toHaveBeenCalledWith(expect.anything(), {
|
||||
actorUserId: 'users:admin',
|
||||
ownerHandle: 'target',
|
||||
ownerUserId: 'users:target',
|
||||
slugs: ['a', 'b'],
|
||||
forceOverwriteSquatter: true,
|
||||
})
|
||||
})
|
||||
|
||||
it('users/reclaim forbids non-admin api tokens', async () => {
|
||||
const runQuery = vi.fn()
|
||||
const runAction = vi.fn()
|
||||
const runMutation = vi.fn().mockResolvedValue(okRate())
|
||||
vi.mocked(requireApiTokenUser).mockResolvedValue({
|
||||
userId: 'users:actor',
|
||||
user: { _id: 'users:actor', role: 'user' },
|
||||
} as never)
|
||||
|
||||
const response = await __handlers.usersPostRouterV1Handler(
|
||||
makeCtx({ runQuery, runAction, runMutation }),
|
||||
new Request('https://example.com/api/v1/users/reclaim', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ handle: 'target', slugs: ['a'] }),
|
||||
}),
|
||||
)
|
||||
expect(response.status).toBe(403)
|
||||
expect(runQuery).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('users/reclaim calls reclaim mutation for admin', async () => {
|
||||
const runMutation = vi.fn(async (_mutation: unknown, args: Record<string, unknown>) => {
|
||||
if (isRateLimitArgs(args)) return okRate()
|
||||
return { ok: true, action: 'ownership_transferred' }
|
||||
})
|
||||
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
|
||||
if ('handle' in args) return { _id: 'users:target' }
|
||||
return null
|
||||
})
|
||||
vi.mocked(requireApiTokenUser).mockResolvedValue({
|
||||
userId: 'users:admin',
|
||||
user: { _id: 'users:admin', role: 'admin' },
|
||||
} as never)
|
||||
|
||||
const response = await __handlers.usersPostRouterV1Handler(
|
||||
makeCtx({ runQuery, runAction: vi.fn(), runMutation }),
|
||||
new Request('https://example.com/api/v1/users/reclaim', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ handle: 'Target', slugs: [' A ', 'b'], reason: 'r' }),
|
||||
}),
|
||||
)
|
||||
if (response.status !== 200) throw new Error(await response.text())
|
||||
|
||||
const reclaimCalls = runMutation.mock.calls.filter(([, args]) => hasSlugArgs(args))
|
||||
expect(reclaimCalls).toHaveLength(2)
|
||||
expect(reclaimCalls[0]?.[1]).toMatchObject({
|
||||
actorUserId: 'users:admin',
|
||||
slug: 'a',
|
||||
rightfulOwnerUserId: 'users:target',
|
||||
reason: 'r',
|
||||
transferRootSlugOnly: true,
|
||||
})
|
||||
expect(reclaimCalls[1]?.[1]).toMatchObject({
|
||||
actorUserId: 'users:admin',
|
||||
slug: 'b',
|
||||
rightfulOwnerUserId: 'users:target',
|
||||
reason: 'r',
|
||||
transferRootSlugOnly: true,
|
||||
})
|
||||
})
|
||||
|
||||
it('search forwards limit and highlightedOnly', async () => {
|
||||
const runAction = vi.fn().mockResolvedValue([
|
||||
{
|
||||
@@ -920,6 +1044,7 @@ describe('httpApiV1 handlers', () => {
|
||||
})
|
||||
|
||||
it('stars add succeeds', async () => {
|
||||
vi.mocked(getOptionalApiTokenUserId).mockResolvedValue('users:1' as never)
|
||||
vi.mocked(requireApiTokenUser).mockResolvedValue({
|
||||
userId: 'users:1',
|
||||
user: { handle: 'p' },
|
||||
@@ -928,7 +1053,6 @@ describe('httpApiV1 handlers', () => {
|
||||
const runMutation = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce(okRate())
|
||||
.mockResolvedValueOnce(okRate())
|
||||
.mockResolvedValueOnce({ ok: true, starred: true, alreadyStarred: false })
|
||||
const response = await __handlers.starsPostRouterV1Handler(
|
||||
makeCtx({ runQuery, runMutation }),
|
||||
@@ -944,6 +1068,7 @@ describe('httpApiV1 handlers', () => {
|
||||
})
|
||||
|
||||
it('stars delete succeeds', async () => {
|
||||
vi.mocked(getOptionalApiTokenUserId).mockResolvedValue('users:1' as never)
|
||||
vi.mocked(requireApiTokenUser).mockResolvedValue({
|
||||
userId: 'users:1',
|
||||
user: { handle: 'p' },
|
||||
@@ -952,7 +1077,6 @@ describe('httpApiV1 handlers', () => {
|
||||
const runMutation = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce(okRate())
|
||||
.mockResolvedValueOnce(okRate())
|
||||
.mockResolvedValueOnce({ ok: true, unstarred: true, alreadyUnstarred: false })
|
||||
const response = await __handlers.starsDeleteRouterV1Handler(
|
||||
makeCtx({ runQuery, runMutation }),
|
||||
@@ -966,4 +1090,61 @@ describe('httpApiV1 handlers', () => {
|
||||
expect(json.ok).toBe(true)
|
||||
expect(json.unstarred).toBe(true)
|
||||
})
|
||||
|
||||
it('delete/undelete map forbidden/not-found/unknown to 403/404/500', async () => {
|
||||
vi.mocked(requireApiTokenUser).mockResolvedValue({
|
||||
userId: 'users:1',
|
||||
user: { handle: 'p' },
|
||||
} as never)
|
||||
|
||||
const runMutationForbidden = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
|
||||
if ('key' in args) return okRate()
|
||||
throw new Error('Forbidden')
|
||||
})
|
||||
const forbidden = await __handlers.skillsDeleteRouterV1Handler(
|
||||
makeCtx({ runMutation: runMutationForbidden }),
|
||||
new Request('https://example.com/api/v1/skills/demo', {
|
||||
method: 'DELETE',
|
||||
headers: { Authorization: 'Bearer clh_test' },
|
||||
}),
|
||||
)
|
||||
expect(forbidden.status).toBe(403)
|
||||
expect(await forbidden.text()).toBe('Forbidden')
|
||||
|
||||
vi.mocked(requireApiTokenUser).mockResolvedValue({
|
||||
userId: 'users:1',
|
||||
user: { handle: 'p' },
|
||||
} as never)
|
||||
const runMutationNotFound = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
|
||||
if ('key' in args) return okRate()
|
||||
throw new Error('Skill not found')
|
||||
})
|
||||
const notFound = await __handlers.skillsPostRouterV1Handler(
|
||||
makeCtx({ runMutation: runMutationNotFound }),
|
||||
new Request('https://example.com/api/v1/skills/demo/undelete', {
|
||||
method: 'POST',
|
||||
headers: { Authorization: 'Bearer clh_test' },
|
||||
}),
|
||||
)
|
||||
expect(notFound.status).toBe(404)
|
||||
expect(await notFound.text()).toBe('Skill not found')
|
||||
|
||||
vi.mocked(requireApiTokenUser).mockResolvedValue({
|
||||
userId: 'users:1',
|
||||
user: { handle: 'p' },
|
||||
} as never)
|
||||
const runMutationUnknown = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
|
||||
if ('key' in args) return okRate()
|
||||
throw new Error('boom')
|
||||
})
|
||||
const unknown = await __handlers.soulsDeleteRouterV1Handler(
|
||||
makeCtx({ runMutation: runMutationUnknown }),
|
||||
new Request('https://example.com/api/v1/souls/demo-soul', {
|
||||
method: 'DELETE',
|
||||
headers: { Authorization: 'Bearer clh_test' },
|
||||
}),
|
||||
)
|
||||
expect(unknown.status).toBe(500)
|
||||
expect(await unknown.text()).toBe('Internal Server Error')
|
||||
})
|
||||
})
|
||||
|
||||
+24
-1296
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,324 @@
|
||||
import { CliPublishRequestSchema, parseArk } from 'clawhub-schema'
|
||||
import { internal } from '../_generated/api'
|
||||
import type { Doc, Id } from '../_generated/dataModel'
|
||||
import type { ActionCtx } from '../_generated/server'
|
||||
import { assertAdmin } from '../lib/access'
|
||||
import { requireApiTokenUser } from '../lib/apiTokenAuth'
|
||||
import { corsHeaders, mergeHeaders } from '../lib/httpHeaders'
|
||||
|
||||
export const MAX_RAW_FILE_BYTES = 200 * 1024
|
||||
|
||||
const SAFE_TEXT_FILE_CSP =
|
||||
"default-src 'none'; base-uri 'none'; form-action 'none'; frame-ancestors 'none'"
|
||||
|
||||
function isSvgLike(contentType: string | undefined, path: string) {
|
||||
return contentType?.toLowerCase().includes('svg') || path.toLowerCase().endsWith('.svg')
|
||||
}
|
||||
|
||||
export function safeTextFileResponse(params: {
|
||||
textContent: string
|
||||
path: string
|
||||
contentType?: string
|
||||
sha256: string
|
||||
size: number
|
||||
headers?: HeadersInit
|
||||
}) {
|
||||
const isSvg = isSvgLike(params.contentType, params.path)
|
||||
|
||||
// For any text response that a browser might try to render, lock it down.
|
||||
// In particular, this prevents SVG <foreignObject> script execution from reading
|
||||
// localStorage tokens on this origin.
|
||||
const headers = mergeHeaders(
|
||||
params.headers,
|
||||
{
|
||||
'Content-Type': params.contentType
|
||||
? `${params.contentType}; charset=utf-8`
|
||||
: 'text/plain; charset=utf-8',
|
||||
'Cache-Control': 'private, max-age=60',
|
||||
ETag: params.sha256,
|
||||
'X-Content-SHA256': params.sha256,
|
||||
'X-Content-Size': String(params.size),
|
||||
'X-Content-Type-Options': 'nosniff',
|
||||
'X-Frame-Options': 'DENY',
|
||||
'Content-Security-Policy': SAFE_TEXT_FILE_CSP,
|
||||
...(isSvg ? { 'Content-Disposition': 'attachment' } : {}),
|
||||
},
|
||||
corsHeaders(),
|
||||
)
|
||||
|
||||
return new Response(params.textContent, { status: 200, headers })
|
||||
}
|
||||
|
||||
export function json(value: unknown, status = 200, headers?: HeadersInit) {
|
||||
return new Response(JSON.stringify(value), {
|
||||
status,
|
||||
headers: mergeHeaders(
|
||||
{
|
||||
'Content-Type': 'application/json',
|
||||
'Cache-Control': 'no-store',
|
||||
},
|
||||
headers,
|
||||
corsHeaders(),
|
||||
),
|
||||
})
|
||||
}
|
||||
|
||||
export function text(value: string, status: number, headers?: HeadersInit) {
|
||||
return new Response(value, {
|
||||
status,
|
||||
headers: mergeHeaders(
|
||||
{
|
||||
'Content-Type': 'text/plain; charset=utf-8',
|
||||
'Cache-Control': 'no-store',
|
||||
},
|
||||
headers,
|
||||
corsHeaders(),
|
||||
),
|
||||
})
|
||||
}
|
||||
|
||||
export async function parseJsonPayload(request: Request, headers: HeadersInit) {
|
||||
try {
|
||||
const payload = (await request.json()) as Record<string, unknown>
|
||||
return { ok: true as const, payload }
|
||||
} catch {
|
||||
return { ok: false as const, response: text('Invalid JSON', 400, headers) }
|
||||
}
|
||||
}
|
||||
|
||||
export async function requireApiTokenUserOrResponse(
|
||||
ctx: ActionCtx,
|
||||
request: Request,
|
||||
headers: HeadersInit,
|
||||
) {
|
||||
try {
|
||||
const auth = await requireApiTokenUser(ctx, request)
|
||||
return { ok: true as const, userId: auth.userId, user: auth.user as Doc<'users'> }
|
||||
} catch {
|
||||
return { ok: false as const, response: text('Unauthorized', 401, headers) }
|
||||
}
|
||||
}
|
||||
|
||||
export function requireAdminOrResponse(user: Doc<'users'>, headers: HeadersInit) {
|
||||
try {
|
||||
assertAdmin(user)
|
||||
return { ok: true as const }
|
||||
} catch {
|
||||
return { ok: false as const, response: text('Forbidden', 403, headers) }
|
||||
}
|
||||
}
|
||||
|
||||
export function getPathSegments(request: Request, prefix: string) {
|
||||
const pathname = new URL(request.url).pathname
|
||||
if (!pathname.startsWith(prefix)) return []
|
||||
const rest = pathname.slice(prefix.length)
|
||||
return rest
|
||||
.split('/')
|
||||
.map((segment) => segment.trim())
|
||||
.filter(Boolean)
|
||||
.map((segment) => decodeURIComponent(segment))
|
||||
}
|
||||
|
||||
export function toOptionalNumber(value: string | null) {
|
||||
if (!value) return undefined
|
||||
const parsed = Number.parseInt(value, 10)
|
||||
return Number.isFinite(parsed) ? parsed : undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Batch resolve soul version tags to version strings.
|
||||
* Collects all version IDs, fetches them in a single query, then maps back.
|
||||
* Reduces N sequential queries to 1 batch query.
|
||||
*/
|
||||
export async function resolveSoulTagsBatch(
|
||||
ctx: ActionCtx,
|
||||
tagsList: Array<Record<string, Id<'soulVersions'>>>,
|
||||
): Promise<Array<Record<string, string>>> {
|
||||
return resolveVersionTagsBatch(ctx, tagsList, internal.souls.getVersionsByIdsInternal)
|
||||
}
|
||||
|
||||
export async function resolveTagsBatch(
|
||||
ctx: ActionCtx,
|
||||
tagsList: Array<Record<string, Id<'skillVersions'>>>,
|
||||
): Promise<Array<Record<string, string>>> {
|
||||
return resolveVersionTagsBatch(ctx, tagsList, internal.skills.getVersionsByIdsInternal)
|
||||
}
|
||||
|
||||
/**
|
||||
* Batch resolve version tags to version strings.
|
||||
* Collects all version IDs, fetches them in a single query, then maps back.
|
||||
*
|
||||
* Notes:
|
||||
* - Uses `internal.*` queries to avoid expanding the public Convex API surface.
|
||||
* - Sorts ids for stable query args (helps caching/log diffs).
|
||||
*/
|
||||
export async function resolveVersionTagsBatch<TTable extends 'skillVersions' | 'soulVersions'>(
|
||||
ctx: ActionCtx,
|
||||
tagsList: Array<Record<string, Id<TTable>>>,
|
||||
getVersionsByIdsQuery: unknown,
|
||||
): Promise<Array<Record<string, string>>> {
|
||||
const allVersionIds = new Set<Id<TTable>>()
|
||||
for (const tags of tagsList) {
|
||||
for (const versionId of Object.values(tags)) allVersionIds.add(versionId)
|
||||
}
|
||||
|
||||
if (allVersionIds.size === 0) return tagsList.map(() => ({}))
|
||||
|
||||
const versionIds = [...allVersionIds].sort() as Array<Id<TTable>>
|
||||
const versions =
|
||||
((await ctx.runQuery(getVersionsByIdsQuery as never, { versionIds } as never)) as Array<{
|
||||
_id: Id<TTable>
|
||||
version: string
|
||||
softDeletedAt?: unknown
|
||||
}> | null) ?? []
|
||||
|
||||
const versionMap = new Map<Id<TTable>, string>()
|
||||
for (const v of versions) {
|
||||
if (!v?.softDeletedAt) versionMap.set(v._id, v.version)
|
||||
}
|
||||
|
||||
return tagsList.map((tags) => {
|
||||
const resolved: Record<string, string> = {}
|
||||
for (const [tag, versionId] of Object.entries(tags)) {
|
||||
const version = versionMap.get(versionId)
|
||||
if (version) resolved[tag] = version
|
||||
}
|
||||
return resolved
|
||||
})
|
||||
}
|
||||
|
||||
async function sha256Hex(bytes: Uint8Array) {
|
||||
const data = new Uint8Array(bytes)
|
||||
const digest = await crypto.subtle.digest('SHA-256', data)
|
||||
return toHex(new Uint8Array(digest))
|
||||
}
|
||||
|
||||
function toHex(bytes: Uint8Array) {
|
||||
let out = ''
|
||||
for (const byte of bytes) out += byte.toString(16).padStart(2, '0')
|
||||
return out
|
||||
}
|
||||
|
||||
type FileLike = {
|
||||
name: string
|
||||
size: number
|
||||
type: string
|
||||
arrayBuffer: () => Promise<ArrayBuffer>
|
||||
}
|
||||
|
||||
type FileLikeEntry = FormDataEntryValue & FileLike
|
||||
|
||||
function toFileLike(entry: FormDataEntryValue): FileLikeEntry | null {
|
||||
if (typeof entry === 'string') return null
|
||||
const candidate = entry as Partial<FileLike>
|
||||
if (typeof candidate.name !== 'string') return null
|
||||
if (typeof candidate.size !== 'number') return null
|
||||
if (typeof candidate.arrayBuffer !== 'function') return null
|
||||
return entry as FileLikeEntry
|
||||
}
|
||||
|
||||
export async function parseMultipartPublish(
|
||||
ctx: ActionCtx,
|
||||
request: Request,
|
||||
): Promise<{
|
||||
slug: string
|
||||
displayName: string
|
||||
version: string
|
||||
changelog: string
|
||||
tags?: string[]
|
||||
forkOf?: { slug: string; version?: string }
|
||||
files: Array<{
|
||||
path: string
|
||||
size: number
|
||||
storageId: Id<'_storage'>
|
||||
sha256: string
|
||||
contentType?: string
|
||||
}>
|
||||
}> {
|
||||
const form = await request.formData()
|
||||
const payloadRaw = form.get('payload')
|
||||
if (!payloadRaw || typeof payloadRaw !== 'string') {
|
||||
throw new Error('Missing payload')
|
||||
}
|
||||
let payload: Record<string, unknown>
|
||||
try {
|
||||
payload = JSON.parse(payloadRaw) as Record<string, unknown>
|
||||
} catch {
|
||||
throw new Error('Invalid JSON payload')
|
||||
}
|
||||
|
||||
const files: Array<{
|
||||
path: string
|
||||
size: number
|
||||
storageId: Id<'_storage'>
|
||||
sha256: string
|
||||
contentType?: string
|
||||
}> = []
|
||||
|
||||
for (const entry of form.getAll('files')) {
|
||||
const file = toFileLike(entry)
|
||||
if (!file) continue
|
||||
const path = file.name
|
||||
const size = file.size
|
||||
const contentType = file.type || undefined
|
||||
const buffer = new Uint8Array(await file.arrayBuffer())
|
||||
const sha256 = await sha256Hex(buffer)
|
||||
const storageId = await ctx.storage.store(file as Blob)
|
||||
files.push({ path, size, storageId, sha256, contentType })
|
||||
}
|
||||
|
||||
const forkOf = payload.forkOf && typeof payload.forkOf === 'object' ? payload.forkOf : undefined
|
||||
const body = {
|
||||
slug: payload.slug,
|
||||
displayName: payload.displayName,
|
||||
version: payload.version,
|
||||
changelog: typeof payload.changelog === 'string' ? payload.changelog : '',
|
||||
tags: Array.isArray(payload.tags) ? payload.tags : undefined,
|
||||
...(payload.source ? { source: payload.source } : {}),
|
||||
files,
|
||||
...(forkOf ? { forkOf } : {}),
|
||||
}
|
||||
|
||||
return parsePublishBody(body)
|
||||
}
|
||||
|
||||
export function parsePublishBody(body: unknown) {
|
||||
const parsed = parseArk(CliPublishRequestSchema, body, 'Publish payload')
|
||||
if (parsed.files.length === 0) throw new Error('files required')
|
||||
const tags = parsed.tags && parsed.tags.length > 0 ? parsed.tags : undefined
|
||||
return {
|
||||
slug: parsed.slug,
|
||||
displayName: parsed.displayName,
|
||||
version: parsed.version,
|
||||
changelog: parsed.changelog,
|
||||
tags,
|
||||
source: parsed.source ?? undefined,
|
||||
forkOf: parsed.forkOf
|
||||
? {
|
||||
slug: parsed.forkOf.slug,
|
||||
version: parsed.forkOf.version ?? undefined,
|
||||
}
|
||||
: undefined,
|
||||
files: parsed.files.map((file) => ({
|
||||
...file,
|
||||
storageId: file.storageId as Id<'_storage'>,
|
||||
})),
|
||||
}
|
||||
}
|
||||
|
||||
export function softDeleteErrorToResponse(
|
||||
entity: 'skill' | 'soul',
|
||||
error: unknown,
|
||||
headers: HeadersInit,
|
||||
) {
|
||||
const message = error instanceof Error ? error.message : `${entity} delete failed`
|
||||
const lower = message.toLowerCase()
|
||||
|
||||
if (lower.includes('unauthorized')) return text('Unauthorized', 401, headers)
|
||||
if (lower.includes('forbidden')) return text('Forbidden', 403, headers)
|
||||
if (lower.includes('not found')) return text(message, 404, headers)
|
||||
if (lower.includes('slug required')) return text('Slug required', 400, headers)
|
||||
|
||||
// Unknown: server-side failure. Keep body generic.
|
||||
return text('Internal Server Error', 500, headers)
|
||||
}
|
||||
@@ -0,0 +1,495 @@
|
||||
import { api, internal } from '../_generated/api'
|
||||
import type { Doc, Id } from '../_generated/dataModel'
|
||||
import type { ActionCtx } from '../_generated/server'
|
||||
import { getOptionalApiTokenUserId, requireApiTokenUser } from '../lib/apiTokenAuth'
|
||||
import { applyRateLimit, parseBearerToken } from '../lib/httpRateLimit'
|
||||
import { publishVersionForUser } from '../skills'
|
||||
import {
|
||||
MAX_RAW_FILE_BYTES,
|
||||
getPathSegments,
|
||||
json,
|
||||
parseMultipartPublish,
|
||||
parsePublishBody,
|
||||
resolveTagsBatch,
|
||||
safeTextFileResponse,
|
||||
softDeleteErrorToResponse,
|
||||
text,
|
||||
toOptionalNumber,
|
||||
} from './shared'
|
||||
|
||||
type SearchSkillEntry = {
|
||||
score: number
|
||||
skill: {
|
||||
slug?: string
|
||||
displayName?: string
|
||||
summary?: string | null
|
||||
updatedAt?: number
|
||||
} | null
|
||||
version: { version?: string; createdAt?: number } | null
|
||||
}
|
||||
|
||||
type ListSkillsResult = {
|
||||
items: Array<{
|
||||
skill: {
|
||||
_id: Id<'skills'>
|
||||
slug: string
|
||||
displayName: string
|
||||
summary?: string
|
||||
tags: Record<string, Id<'skillVersions'>>
|
||||
stats: unknown
|
||||
createdAt: number
|
||||
updatedAt: number
|
||||
latestVersionId?: Id<'skillVersions'>
|
||||
}
|
||||
latestVersion: { version: string; createdAt: number; changelog: string } | null
|
||||
}>
|
||||
nextCursor: string | null
|
||||
}
|
||||
|
||||
type SkillFile = Doc<'skillVersions'>['files'][number]
|
||||
|
||||
type GetBySlugResult = {
|
||||
skill: {
|
||||
_id: Id<'skills'>
|
||||
slug: string
|
||||
displayName: string
|
||||
summary?: string
|
||||
tags: Record<string, Id<'skillVersions'>>
|
||||
stats: unknown
|
||||
createdAt: number
|
||||
updatedAt: number
|
||||
} | null
|
||||
latestVersion: Doc<'skillVersions'> | null
|
||||
owner: { _id: Id<'users'>; handle?: string; displayName?: string; image?: string } | null
|
||||
moderationInfo?: {
|
||||
isPendingScan: boolean
|
||||
isMalwareBlocked: boolean
|
||||
isSuspicious: boolean
|
||||
isHiddenByMod: boolean
|
||||
isRemoved: boolean
|
||||
reason?: string
|
||||
} | null
|
||||
} | null
|
||||
|
||||
type ListVersionsResult = {
|
||||
items: Array<{
|
||||
version: string
|
||||
createdAt: number
|
||||
changelog: string
|
||||
changelogSource?: 'auto' | 'user'
|
||||
files: Array<{
|
||||
path: string
|
||||
size: number
|
||||
storageId: Id<'_storage'>
|
||||
sha256: string
|
||||
contentType?: string
|
||||
}>
|
||||
softDeletedAt?: number
|
||||
}>
|
||||
nextCursor: string | null
|
||||
}
|
||||
|
||||
export async function searchSkillsV1Handler(ctx: ActionCtx, request: Request) {
|
||||
const rate = await applyRateLimit(ctx, request, 'read')
|
||||
if (!rate.ok) return rate.response
|
||||
|
||||
const url = new URL(request.url)
|
||||
const query = url.searchParams.get('q')?.trim() ?? ''
|
||||
const limit = toOptionalNumber(url.searchParams.get('limit'))
|
||||
const highlightedOnly = url.searchParams.get('highlightedOnly') === 'true'
|
||||
|
||||
if (!query) return json({ results: [] }, 200, rate.headers)
|
||||
|
||||
const results = (await ctx.runAction(api.search.searchSkills, {
|
||||
query,
|
||||
limit,
|
||||
highlightedOnly: highlightedOnly || undefined,
|
||||
})) as SearchSkillEntry[]
|
||||
|
||||
return json(
|
||||
{
|
||||
results: results.map((result) => ({
|
||||
score: result.score,
|
||||
slug: result.skill?.slug,
|
||||
displayName: result.skill?.displayName,
|
||||
summary: result.skill?.summary ?? null,
|
||||
version: result.version?.version ?? null,
|
||||
updatedAt: result.skill?.updatedAt,
|
||||
})),
|
||||
},
|
||||
200,
|
||||
rate.headers,
|
||||
)
|
||||
}
|
||||
|
||||
export async function resolveSkillVersionV1Handler(ctx: ActionCtx, request: Request) {
|
||||
const rate = await applyRateLimit(ctx, request, 'read')
|
||||
if (!rate.ok) return rate.response
|
||||
|
||||
const url = new URL(request.url)
|
||||
const slug = url.searchParams.get('slug')?.trim().toLowerCase()
|
||||
const hash = url.searchParams.get('hash')?.trim().toLowerCase()
|
||||
if (!slug || !hash) return text('Missing slug or hash', 400, rate.headers)
|
||||
if (!/^[a-f0-9]{64}$/.test(hash)) return text('Invalid hash', 400, rate.headers)
|
||||
|
||||
const resolved = await ctx.runQuery(api.skills.resolveVersionByHash, { slug, hash })
|
||||
if (!resolved) return text('Skill not found', 404, rate.headers)
|
||||
|
||||
return json({ slug, match: resolved.match, latestVersion: resolved.latestVersion }, 200, rate.headers)
|
||||
}
|
||||
|
||||
type SkillListSort =
|
||||
| 'updated'
|
||||
| 'downloads'
|
||||
| 'stars'
|
||||
| 'installsCurrent'
|
||||
| 'installsAllTime'
|
||||
| 'trending'
|
||||
|
||||
function parseListSort(value: string | null): SkillListSort {
|
||||
const normalized = value?.trim().toLowerCase()
|
||||
if (normalized === 'downloads') return 'downloads'
|
||||
if (normalized === 'stars' || normalized === 'rating') return 'stars'
|
||||
if (
|
||||
normalized === 'installs' ||
|
||||
normalized === 'install' ||
|
||||
normalized === 'installscurrent' ||
|
||||
normalized === 'installs-current'
|
||||
) {
|
||||
return 'installsCurrent'
|
||||
}
|
||||
if (normalized === 'installsalltime' || normalized === 'installs-all-time') {
|
||||
return 'installsAllTime'
|
||||
}
|
||||
if (normalized === 'trending') return 'trending'
|
||||
return 'updated'
|
||||
}
|
||||
|
||||
export async function listSkillsV1Handler(ctx: ActionCtx, request: Request) {
|
||||
const rate = await applyRateLimit(ctx, request, 'read')
|
||||
if (!rate.ok) return rate.response
|
||||
|
||||
const url = new URL(request.url)
|
||||
const limit = toOptionalNumber(url.searchParams.get('limit'))
|
||||
const rawCursor = url.searchParams.get('cursor')?.trim() || undefined
|
||||
const sort = parseListSort(url.searchParams.get('sort'))
|
||||
const cursor = sort === 'trending' ? undefined : rawCursor
|
||||
|
||||
const result = (await ctx.runQuery(api.skills.listPublicPage, {
|
||||
limit,
|
||||
cursor,
|
||||
sort,
|
||||
})) as ListSkillsResult
|
||||
|
||||
// Batch resolve all tags in a single query instead of N queries
|
||||
const resolvedTagsList = await resolveTagsBatch(
|
||||
ctx,
|
||||
result.items.map((item) => item.skill.tags),
|
||||
)
|
||||
|
||||
const items = result.items.map((item, idx) => ({
|
||||
slug: item.skill.slug,
|
||||
displayName: item.skill.displayName,
|
||||
summary: item.skill.summary ?? null,
|
||||
tags: resolvedTagsList[idx],
|
||||
stats: item.skill.stats,
|
||||
createdAt: item.skill.createdAt,
|
||||
updatedAt: item.skill.updatedAt,
|
||||
latestVersion: item.latestVersion
|
||||
? {
|
||||
version: item.latestVersion.version,
|
||||
createdAt: item.latestVersion.createdAt,
|
||||
changelog: item.latestVersion.changelog,
|
||||
}
|
||||
: null,
|
||||
}))
|
||||
|
||||
return json({ items, nextCursor: result.nextCursor ?? null }, 200, rate.headers)
|
||||
}
|
||||
|
||||
async function describeOwnerVisibleSkillState(
|
||||
ctx: ActionCtx,
|
||||
request: Request,
|
||||
slug: string,
|
||||
): Promise<{ status: number; message: string } | null> {
|
||||
const skill = await ctx.runQuery(internal.skills.getSkillBySlugInternal, { slug })
|
||||
if (!skill) return null
|
||||
|
||||
const apiTokenUserId = await getOptionalApiTokenUserId(ctx, request)
|
||||
const isOwner = Boolean(apiTokenUserId && apiTokenUserId === skill.ownerUserId)
|
||||
if (!isOwner) return null
|
||||
|
||||
if (skill.softDeletedAt) {
|
||||
return {
|
||||
status: 410,
|
||||
message: `Skill is hidden/deleted. Run "clawhub undelete ${slug}" to restore it.`,
|
||||
}
|
||||
}
|
||||
|
||||
if (skill.moderationStatus === 'hidden') {
|
||||
if (skill.moderationReason === 'pending.scan' || skill.moderationReason === 'scanner.vt.pending') {
|
||||
return {
|
||||
status: 423,
|
||||
message: 'Skill is hidden while security scan is pending. Try again in a few minutes.',
|
||||
}
|
||||
}
|
||||
if (skill.moderationReason === 'quality.low') {
|
||||
return {
|
||||
status: 403,
|
||||
message:
|
||||
'Skill is hidden by quality checks. Update SKILL.md content or run "clawhub undelete <slug>" after review.',
|
||||
}
|
||||
}
|
||||
return {
|
||||
status: 403,
|
||||
message: `Skill is hidden by moderation${
|
||||
skill.moderationReason ? ` (${skill.moderationReason})` : ''
|
||||
}.`,
|
||||
}
|
||||
}
|
||||
|
||||
if (skill.moderationStatus === 'removed') {
|
||||
return { status: 410, message: 'Skill has been removed by moderation.' }
|
||||
}
|
||||
|
||||
return null
|
||||
}
|
||||
|
||||
export async function skillsGetRouterV1Handler(ctx: ActionCtx, request: Request) {
|
||||
const rate = await applyRateLimit(ctx, request, 'read')
|
||||
if (!rate.ok) return rate.response
|
||||
|
||||
const segments = getPathSegments(request, '/api/v1/skills/')
|
||||
if (segments.length === 0) return text('Missing slug', 400, rate.headers)
|
||||
const slug = segments[0]?.trim().toLowerCase() ?? ''
|
||||
const second = segments[1]
|
||||
const third = segments[2]
|
||||
|
||||
if (segments.length === 1) {
|
||||
const result = (await ctx.runQuery(api.skills.getBySlug, { slug })) as GetBySlugResult
|
||||
if (!result?.skill) {
|
||||
const hidden = await describeOwnerVisibleSkillState(ctx, request, slug)
|
||||
if (hidden) return text(hidden.message, hidden.status, rate.headers)
|
||||
return text('Skill not found', 404, rate.headers)
|
||||
}
|
||||
|
||||
const [tags] = await resolveTagsBatch(ctx, [result.skill.tags])
|
||||
return json(
|
||||
{
|
||||
skill: {
|
||||
slug: result.skill.slug,
|
||||
displayName: result.skill.displayName,
|
||||
summary: result.skill.summary ?? null,
|
||||
tags,
|
||||
stats: result.skill.stats,
|
||||
createdAt: result.skill.createdAt,
|
||||
updatedAt: result.skill.updatedAt,
|
||||
},
|
||||
latestVersion: result.latestVersion
|
||||
? {
|
||||
version: result.latestVersion.version,
|
||||
createdAt: result.latestVersion.createdAt,
|
||||
changelog: result.latestVersion.changelog,
|
||||
}
|
||||
: null,
|
||||
owner: result.owner
|
||||
? {
|
||||
handle: result.owner.handle ?? null,
|
||||
userId: result.owner._id,
|
||||
displayName: result.owner.displayName ?? null,
|
||||
image: result.owner.image ?? null,
|
||||
}
|
||||
: null,
|
||||
moderation: result.moderationInfo
|
||||
? {
|
||||
isSuspicious: result.moderationInfo.isSuspicious ?? false,
|
||||
isMalwareBlocked: result.moderationInfo.isMalwareBlocked ?? false,
|
||||
}
|
||||
: null,
|
||||
},
|
||||
200,
|
||||
rate.headers,
|
||||
)
|
||||
}
|
||||
|
||||
if (second === 'versions' && segments.length === 2) {
|
||||
const skill = await ctx.runQuery(internal.skills.getSkillBySlugInternal, { slug })
|
||||
if (!skill || skill.softDeletedAt) return text('Skill not found', 404, rate.headers)
|
||||
|
||||
const url = new URL(request.url)
|
||||
const limit = toOptionalNumber(url.searchParams.get('limit'))
|
||||
const cursor = url.searchParams.get('cursor')?.trim() || undefined
|
||||
const result = (await ctx.runQuery(api.skills.listVersionsPage, {
|
||||
skillId: skill._id,
|
||||
limit,
|
||||
cursor,
|
||||
})) as ListVersionsResult
|
||||
|
||||
const items = result.items
|
||||
.filter((version) => !version.softDeletedAt)
|
||||
.map((version) => ({
|
||||
version: version.version,
|
||||
createdAt: version.createdAt,
|
||||
changelog: version.changelog,
|
||||
changelogSource: version.changelogSource ?? null,
|
||||
}))
|
||||
|
||||
return json({ items, nextCursor: result.nextCursor ?? null }, 200, rate.headers)
|
||||
}
|
||||
|
||||
if (second === 'versions' && third && segments.length === 3) {
|
||||
const skill = await ctx.runQuery(internal.skills.getSkillBySlugInternal, { slug })
|
||||
if (!skill || skill.softDeletedAt) return text('Skill not found', 404, rate.headers)
|
||||
|
||||
const version = await ctx.runQuery(api.skills.getVersionBySkillAndVersion, {
|
||||
skillId: skill._id,
|
||||
version: third,
|
||||
})
|
||||
if (!version) return text('Version not found', 404, rate.headers)
|
||||
if (version.softDeletedAt) return text('Version not available', 410, rate.headers)
|
||||
|
||||
return json(
|
||||
{
|
||||
skill: { slug: skill.slug, displayName: skill.displayName },
|
||||
version: {
|
||||
version: version.version,
|
||||
createdAt: version.createdAt,
|
||||
changelog: version.changelog,
|
||||
changelogSource: version.changelogSource ?? null,
|
||||
files: version.files.map((file: SkillFile) => ({
|
||||
path: file.path,
|
||||
size: file.size,
|
||||
sha256: file.sha256,
|
||||
contentType: file.contentType ?? null,
|
||||
})),
|
||||
},
|
||||
},
|
||||
200,
|
||||
rate.headers,
|
||||
)
|
||||
}
|
||||
|
||||
if (second === 'file' && segments.length === 2) {
|
||||
const url = new URL(request.url)
|
||||
const path = url.searchParams.get('path')?.trim()
|
||||
if (!path) return text('Missing path', 400, rate.headers)
|
||||
const versionParam = url.searchParams.get('version')?.trim()
|
||||
const tagParam = url.searchParams.get('tag')?.trim()
|
||||
|
||||
const skillResult = (await ctx.runQuery(api.skills.getBySlug, { slug })) as GetBySlugResult
|
||||
if (!skillResult?.skill) return text('Skill not found', 404, rate.headers)
|
||||
|
||||
let version = skillResult.latestVersion
|
||||
if (versionParam) {
|
||||
version = await ctx.runQuery(api.skills.getVersionBySkillAndVersion, {
|
||||
skillId: skillResult.skill._id,
|
||||
version: versionParam,
|
||||
})
|
||||
} else if (tagParam) {
|
||||
const versionId = skillResult.skill.tags[tagParam]
|
||||
if (versionId) {
|
||||
version = await ctx.runQuery(api.skills.getVersionById, { versionId })
|
||||
}
|
||||
}
|
||||
|
||||
if (!version) return text('Version not found', 404, rate.headers)
|
||||
if (version.softDeletedAt) return text('Version not available', 410, rate.headers)
|
||||
|
||||
const normalized = path.trim()
|
||||
const normalizedLower = normalized.toLowerCase()
|
||||
const file =
|
||||
version.files.find((entry) => entry.path === normalized) ??
|
||||
version.files.find((entry) => entry.path.toLowerCase() === normalizedLower)
|
||||
if (!file) return text('File not found', 404, rate.headers)
|
||||
if (file.size > MAX_RAW_FILE_BYTES) return text('File exceeds 200KB limit', 413, rate.headers)
|
||||
|
||||
const blob = await ctx.storage.get(file.storageId)
|
||||
if (!blob) return text('File missing in storage', 410, rate.headers)
|
||||
const textContent = await blob.text()
|
||||
return safeTextFileResponse({
|
||||
textContent,
|
||||
path: file.path,
|
||||
contentType: file.contentType ?? undefined,
|
||||
sha256: file.sha256,
|
||||
size: file.size,
|
||||
headers: rate.headers,
|
||||
})
|
||||
}
|
||||
|
||||
return text('Not found', 404, rate.headers)
|
||||
}
|
||||
|
||||
export async function publishSkillV1Handler(ctx: ActionCtx, request: Request) {
|
||||
const rate = await applyRateLimit(ctx, request, 'write')
|
||||
if (!rate.ok) return rate.response
|
||||
|
||||
try {
|
||||
if (!parseBearerToken(request)) return text('Unauthorized', 401, rate.headers)
|
||||
} catch {
|
||||
return text('Unauthorized', 401, rate.headers)
|
||||
}
|
||||
const { userId } = await requireApiTokenUser(ctx, request)
|
||||
|
||||
const contentType = request.headers.get('content-type') ?? ''
|
||||
try {
|
||||
if (contentType.includes('application/json')) {
|
||||
const body = await request.json()
|
||||
const payload = parsePublishBody(body)
|
||||
const result = await publishVersionForUser(ctx, userId, payload)
|
||||
return json({ ok: true, ...result }, 200, rate.headers)
|
||||
}
|
||||
|
||||
if (contentType.includes('multipart/form-data')) {
|
||||
const payload = await parseMultipartPublish(ctx, request)
|
||||
const result = await publishVersionForUser(ctx, userId, payload)
|
||||
return json({ ok: true, ...result }, 200, rate.headers)
|
||||
}
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : 'Publish failed'
|
||||
return text(message, 400, rate.headers)
|
||||
}
|
||||
|
||||
return text('Unsupported content type', 415, rate.headers)
|
||||
}
|
||||
|
||||
export async function skillsPostRouterV1Handler(ctx: ActionCtx, request: Request) {
|
||||
const rate = await applyRateLimit(ctx, request, 'write')
|
||||
if (!rate.ok) return rate.response
|
||||
|
||||
const segments = getPathSegments(request, '/api/v1/skills/')
|
||||
if (segments.length !== 2 || segments[1] !== 'undelete') {
|
||||
return text('Not found', 404, rate.headers)
|
||||
}
|
||||
const slug = segments[0]?.trim().toLowerCase() ?? ''
|
||||
try {
|
||||
const { userId } = await requireApiTokenUser(ctx, request)
|
||||
await ctx.runMutation(internal.skills.setSkillSoftDeletedInternal, {
|
||||
userId,
|
||||
slug,
|
||||
deleted: false,
|
||||
})
|
||||
return json({ ok: true }, 200, rate.headers)
|
||||
} catch (error) {
|
||||
return softDeleteErrorToResponse('skill', error, rate.headers)
|
||||
}
|
||||
}
|
||||
|
||||
export async function skillsDeleteRouterV1Handler(ctx: ActionCtx, request: Request) {
|
||||
const rate = await applyRateLimit(ctx, request, 'write')
|
||||
if (!rate.ok) return rate.response
|
||||
|
||||
const segments = getPathSegments(request, '/api/v1/skills/')
|
||||
if (segments.length !== 1) return text('Not found', 404, rate.headers)
|
||||
const slug = segments[0]?.trim().toLowerCase() ?? ''
|
||||
try {
|
||||
const { userId } = await requireApiTokenUser(ctx, request)
|
||||
await ctx.runMutation(internal.skills.setSkillSoftDeletedInternal, {
|
||||
userId,
|
||||
slug,
|
||||
deleted: true,
|
||||
})
|
||||
return json({ ok: true }, 200, rate.headers)
|
||||
} catch (error) {
|
||||
return softDeleteErrorToResponse('skill', error, rate.headers)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,340 @@
|
||||
import { api, internal } from '../_generated/api'
|
||||
import type { Doc, Id } from '../_generated/dataModel'
|
||||
import type { ActionCtx } from '../_generated/server'
|
||||
import { requireApiTokenUser } from '../lib/apiTokenAuth'
|
||||
import { applyRateLimit, parseBearerToken } from '../lib/httpRateLimit'
|
||||
import { publishSoulVersionForUser } from '../souls'
|
||||
import {
|
||||
MAX_RAW_FILE_BYTES,
|
||||
getPathSegments,
|
||||
json,
|
||||
parseMultipartPublish,
|
||||
parsePublishBody,
|
||||
resolveSoulTagsBatch,
|
||||
safeTextFileResponse,
|
||||
softDeleteErrorToResponse,
|
||||
text,
|
||||
toOptionalNumber,
|
||||
} from './shared'
|
||||
|
||||
type ListSoulsResult = {
|
||||
items: Array<{
|
||||
soul: {
|
||||
_id: Id<'souls'>
|
||||
slug: string
|
||||
displayName: string
|
||||
summary?: string
|
||||
tags: Record<string, Id<'soulVersions'>>
|
||||
stats: unknown
|
||||
createdAt: number
|
||||
updatedAt: number
|
||||
latestVersionId?: Id<'soulVersions'>
|
||||
}
|
||||
latestVersion: { version: string; createdAt: number; changelog: string } | null
|
||||
}>
|
||||
nextCursor: string | null
|
||||
}
|
||||
|
||||
type GetSoulBySlugResult = {
|
||||
soul: {
|
||||
_id: Id<'souls'>
|
||||
slug: string
|
||||
displayName: string
|
||||
summary?: string
|
||||
tags: Record<string, Id<'soulVersions'>>
|
||||
stats: unknown
|
||||
createdAt: number
|
||||
updatedAt: number
|
||||
} | null
|
||||
latestVersion: Doc<'soulVersions'> | null
|
||||
owner: { handle?: string; displayName?: string; image?: string } | null
|
||||
} | null
|
||||
|
||||
type ListSoulVersionsResult = {
|
||||
items: Array<{
|
||||
version: string
|
||||
createdAt: number
|
||||
changelog: string
|
||||
changelogSource?: 'auto' | 'user'
|
||||
files: Array<{
|
||||
path: string
|
||||
size: number
|
||||
storageId: Id<'_storage'>
|
||||
sha256: string
|
||||
contentType?: string
|
||||
}>
|
||||
softDeletedAt?: number
|
||||
}>
|
||||
nextCursor: string | null
|
||||
}
|
||||
|
||||
type SoulFile = Doc<'soulVersions'>['files'][number]
|
||||
|
||||
export async function listSoulsV1Handler(ctx: ActionCtx, request: Request) {
|
||||
const rate = await applyRateLimit(ctx, request, 'read')
|
||||
if (!rate.ok) return rate.response
|
||||
|
||||
const url = new URL(request.url)
|
||||
const limit = toOptionalNumber(url.searchParams.get('limit'))
|
||||
const cursor = url.searchParams.get('cursor')?.trim() || undefined
|
||||
|
||||
const result = (await ctx.runQuery(api.souls.listPublicPage, {
|
||||
limit,
|
||||
cursor,
|
||||
})) as ListSoulsResult
|
||||
|
||||
// Batch resolve all tags in a single query instead of N queries
|
||||
const resolvedTagsList = await resolveSoulTagsBatch(
|
||||
ctx,
|
||||
result.items.map((item) => item.soul.tags),
|
||||
)
|
||||
|
||||
const items = result.items.map((item, idx) => ({
|
||||
slug: item.soul.slug,
|
||||
displayName: item.soul.displayName,
|
||||
summary: item.soul.summary ?? null,
|
||||
tags: resolvedTagsList[idx],
|
||||
stats: item.soul.stats,
|
||||
createdAt: item.soul.createdAt,
|
||||
updatedAt: item.soul.updatedAt,
|
||||
latestVersion: item.latestVersion
|
||||
? {
|
||||
version: item.latestVersion.version,
|
||||
createdAt: item.latestVersion.createdAt,
|
||||
changelog: item.latestVersion.changelog,
|
||||
}
|
||||
: null,
|
||||
}))
|
||||
|
||||
return json({ items, nextCursor: result.nextCursor ?? null }, 200, rate.headers)
|
||||
}
|
||||
|
||||
export async function soulsGetRouterV1Handler(ctx: ActionCtx, request: Request) {
|
||||
const rate = await applyRateLimit(ctx, request, 'read')
|
||||
if (!rate.ok) return rate.response
|
||||
|
||||
const segments = getPathSegments(request, '/api/v1/souls/')
|
||||
if (segments.length === 0) return text('Missing slug', 400, rate.headers)
|
||||
const slug = segments[0]?.trim().toLowerCase() ?? ''
|
||||
const second = segments[1]
|
||||
const third = segments[2]
|
||||
|
||||
if (segments.length === 1) {
|
||||
const result = (await ctx.runQuery(api.souls.getBySlug, { slug })) as GetSoulBySlugResult
|
||||
if (!result?.soul) return text('Soul not found', 404, rate.headers)
|
||||
|
||||
const [tags] = await resolveSoulTagsBatch(ctx, [result.soul.tags])
|
||||
return json(
|
||||
{
|
||||
soul: {
|
||||
slug: result.soul.slug,
|
||||
displayName: result.soul.displayName,
|
||||
summary: result.soul.summary ?? null,
|
||||
tags,
|
||||
stats: result.soul.stats,
|
||||
createdAt: result.soul.createdAt,
|
||||
updatedAt: result.soul.updatedAt,
|
||||
},
|
||||
latestVersion: result.latestVersion
|
||||
? {
|
||||
version: result.latestVersion.version,
|
||||
createdAt: result.latestVersion.createdAt,
|
||||
changelog: result.latestVersion.changelog,
|
||||
}
|
||||
: null,
|
||||
owner: result.owner
|
||||
? {
|
||||
handle: result.owner.handle ?? null,
|
||||
displayName: result.owner.displayName ?? null,
|
||||
image: result.owner.image ?? null,
|
||||
}
|
||||
: null,
|
||||
},
|
||||
200,
|
||||
rate.headers,
|
||||
)
|
||||
}
|
||||
|
||||
if (second === 'versions' && segments.length === 2) {
|
||||
const soul = await ctx.runQuery(internal.souls.getSoulBySlugInternal, { slug })
|
||||
if (!soul || soul.softDeletedAt) return text('Soul not found', 404, rate.headers)
|
||||
|
||||
const url = new URL(request.url)
|
||||
const limit = toOptionalNumber(url.searchParams.get('limit'))
|
||||
const cursor = url.searchParams.get('cursor')?.trim() || undefined
|
||||
const result = (await ctx.runQuery(api.souls.listVersionsPage, {
|
||||
soulId: soul._id,
|
||||
limit,
|
||||
cursor,
|
||||
})) as ListSoulVersionsResult
|
||||
|
||||
const items = result.items
|
||||
.filter((version) => !version.softDeletedAt)
|
||||
.map((version) => ({
|
||||
version: version.version,
|
||||
createdAt: version.createdAt,
|
||||
changelog: version.changelog,
|
||||
changelogSource: version.changelogSource ?? null,
|
||||
}))
|
||||
|
||||
return json({ items, nextCursor: result.nextCursor ?? null }, 200, rate.headers)
|
||||
}
|
||||
|
||||
if (second === 'versions' && third && segments.length === 3) {
|
||||
const soul = await ctx.runQuery(internal.souls.getSoulBySlugInternal, { slug })
|
||||
if (!soul || soul.softDeletedAt) return text('Soul not found', 404, rate.headers)
|
||||
|
||||
const version = await ctx.runQuery(api.souls.getVersionBySoulAndVersion, {
|
||||
soulId: soul._id,
|
||||
version: third,
|
||||
})
|
||||
if (!version) return text('Version not found', 404, rate.headers)
|
||||
if (version.softDeletedAt) return text('Version not available', 410, rate.headers)
|
||||
|
||||
return json(
|
||||
{
|
||||
soul: { slug: soul.slug, displayName: soul.displayName },
|
||||
version: {
|
||||
version: version.version,
|
||||
createdAt: version.createdAt,
|
||||
changelog: version.changelog,
|
||||
changelogSource: version.changelogSource ?? null,
|
||||
files: version.files.map((file: SoulFile) => ({
|
||||
path: file.path,
|
||||
size: file.size,
|
||||
sha256: file.sha256,
|
||||
contentType: file.contentType ?? null,
|
||||
})),
|
||||
},
|
||||
},
|
||||
200,
|
||||
rate.headers,
|
||||
)
|
||||
}
|
||||
|
||||
if (second === 'file' && segments.length === 2) {
|
||||
const url = new URL(request.url)
|
||||
const path = url.searchParams.get('path')?.trim()
|
||||
if (!path) return text('Missing path', 400, rate.headers)
|
||||
const versionParam = url.searchParams.get('version')?.trim()
|
||||
const tagParam = url.searchParams.get('tag')?.trim()
|
||||
|
||||
const soulResult = (await ctx.runQuery(api.souls.getBySlug, { slug })) as GetSoulBySlugResult
|
||||
if (!soulResult?.soul) return text('Soul not found', 404, rate.headers)
|
||||
|
||||
let version = soulResult.latestVersion
|
||||
if (versionParam) {
|
||||
version = await ctx.runQuery(api.souls.getVersionBySoulAndVersion, {
|
||||
soulId: soulResult.soul._id,
|
||||
version: versionParam,
|
||||
})
|
||||
} else if (tagParam) {
|
||||
const versionId = soulResult.soul.tags[tagParam]
|
||||
if (versionId) {
|
||||
version = await ctx.runQuery(api.souls.getVersionById, { versionId })
|
||||
}
|
||||
}
|
||||
|
||||
if (!version) return text('Version not found', 404, rate.headers)
|
||||
if (version.softDeletedAt) return text('Version not available', 410, rate.headers)
|
||||
|
||||
const normalized = path.trim()
|
||||
const normalizedLower = normalized.toLowerCase()
|
||||
const file =
|
||||
version.files.find((entry) => entry.path === normalized) ??
|
||||
version.files.find((entry) => entry.path.toLowerCase() === normalizedLower)
|
||||
if (!file) return text('File not found', 404, rate.headers)
|
||||
if (file.size > MAX_RAW_FILE_BYTES) return text('File exceeds 200KB limit', 413, rate.headers)
|
||||
|
||||
const blob = await ctx.storage.get(file.storageId)
|
||||
if (!blob) return text('File missing in storage', 410, rate.headers)
|
||||
const textContent = await blob.text()
|
||||
|
||||
void ctx.runMutation(api.soulDownloads.increment, { soulId: soulResult.soul._id })
|
||||
return safeTextFileResponse({
|
||||
textContent,
|
||||
path: file.path,
|
||||
contentType: file.contentType ?? undefined,
|
||||
sha256: file.sha256,
|
||||
size: file.size,
|
||||
headers: rate.headers,
|
||||
})
|
||||
}
|
||||
|
||||
return text('Not found', 404, rate.headers)
|
||||
}
|
||||
|
||||
export async function publishSoulV1Handler(ctx: ActionCtx, request: Request) {
|
||||
const rate = await applyRateLimit(ctx, request, 'write')
|
||||
if (!rate.ok) return rate.response
|
||||
|
||||
try {
|
||||
if (!parseBearerToken(request)) return text('Unauthorized', 401, rate.headers)
|
||||
} catch {
|
||||
return text('Unauthorized', 401, rate.headers)
|
||||
}
|
||||
const { userId } = await requireApiTokenUser(ctx, request)
|
||||
|
||||
const contentType = request.headers.get('content-type') ?? ''
|
||||
try {
|
||||
if (contentType.includes('application/json')) {
|
||||
const body = await request.json()
|
||||
const payload = parsePublishBody(body)
|
||||
const result = await publishSoulVersionForUser(ctx, userId, payload)
|
||||
return json({ ok: true, ...result }, 200, rate.headers)
|
||||
}
|
||||
|
||||
if (contentType.includes('multipart/form-data')) {
|
||||
const payload = await parseMultipartPublish(ctx, request)
|
||||
const result = await publishSoulVersionForUser(ctx, userId, payload)
|
||||
return json({ ok: true, ...result }, 200, rate.headers)
|
||||
}
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : 'Publish failed'
|
||||
return text(message, 400, rate.headers)
|
||||
}
|
||||
|
||||
return text('Unsupported content type', 415, rate.headers)
|
||||
}
|
||||
|
||||
export async function soulsPostRouterV1Handler(ctx: ActionCtx, request: Request) {
|
||||
const rate = await applyRateLimit(ctx, request, 'write')
|
||||
if (!rate.ok) return rate.response
|
||||
|
||||
const segments = getPathSegments(request, '/api/v1/souls/')
|
||||
if (segments.length !== 2 || segments[1] !== 'undelete') {
|
||||
return text('Not found', 404, rate.headers)
|
||||
}
|
||||
const slug = segments[0]?.trim().toLowerCase() ?? ''
|
||||
try {
|
||||
const { userId } = await requireApiTokenUser(ctx, request)
|
||||
await ctx.runMutation(internal.souls.setSoulSoftDeletedInternal, {
|
||||
userId,
|
||||
slug,
|
||||
deleted: false,
|
||||
})
|
||||
return json({ ok: true }, 200, rate.headers)
|
||||
} catch (error) {
|
||||
return softDeleteErrorToResponse('soul', error, rate.headers)
|
||||
}
|
||||
}
|
||||
|
||||
export async function soulsDeleteRouterV1Handler(ctx: ActionCtx, request: Request) {
|
||||
const rate = await applyRateLimit(ctx, request, 'write')
|
||||
if (!rate.ok) return rate.response
|
||||
|
||||
const segments = getPathSegments(request, '/api/v1/souls/')
|
||||
if (segments.length !== 1) return text('Not found', 404, rate.headers)
|
||||
const slug = segments[0]?.trim().toLowerCase() ?? ''
|
||||
try {
|
||||
const { userId } = await requireApiTokenUser(ctx, request)
|
||||
await ctx.runMutation(internal.souls.setSoulSoftDeletedInternal, {
|
||||
userId,
|
||||
slug,
|
||||
deleted: true,
|
||||
})
|
||||
return json({ ok: true }, 200, rate.headers)
|
||||
} catch (error) {
|
||||
return softDeleteErrorToResponse('soul', error, rate.headers)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
import { internal } from '../_generated/api'
|
||||
import type { ActionCtx } from '../_generated/server'
|
||||
import { requireApiTokenUser } from '../lib/apiTokenAuth'
|
||||
import { applyRateLimit } from '../lib/httpRateLimit'
|
||||
import { getPathSegments, json, text } from './shared'
|
||||
|
||||
export async function starsPostRouterV1Handler(ctx: ActionCtx, request: Request) {
|
||||
const rate = await applyRateLimit(ctx, request, 'write')
|
||||
if (!rate.ok) return rate.response
|
||||
|
||||
const segments = getPathSegments(request, '/api/v1/stars/')
|
||||
if (segments.length !== 1) return text('Not found', 404, rate.headers)
|
||||
const slug = segments[0]?.trim().toLowerCase() ?? ''
|
||||
|
||||
try {
|
||||
const { userId } = await requireApiTokenUser(ctx, request)
|
||||
const skill = await ctx.runQuery(internal.skills.getSkillBySlugInternal, { slug })
|
||||
if (!skill) return text('Skill not found', 404, rate.headers)
|
||||
|
||||
const result = await ctx.runMutation(internal.stars.addStarInternal, {
|
||||
userId,
|
||||
skillId: skill._id,
|
||||
})
|
||||
return json(result, 200, rate.headers)
|
||||
} catch {
|
||||
return text('Unauthorized', 401, rate.headers)
|
||||
}
|
||||
}
|
||||
|
||||
export async function starsDeleteRouterV1Handler(ctx: ActionCtx, request: Request) {
|
||||
const rate = await applyRateLimit(ctx, request, 'write')
|
||||
if (!rate.ok) return rate.response
|
||||
|
||||
const segments = getPathSegments(request, '/api/v1/stars/')
|
||||
if (segments.length !== 1) return text('Not found', 404, rate.headers)
|
||||
const slug = segments[0]?.trim().toLowerCase() ?? ''
|
||||
|
||||
try {
|
||||
const { userId } = await requireApiTokenUser(ctx, request)
|
||||
const skill = await ctx.runQuery(internal.skills.getSkillBySlugInternal, { slug })
|
||||
if (!skill) return text('Skill not found', 404, rate.headers)
|
||||
|
||||
const result = await ctx.runMutation(internal.stars.removeStarInternal, {
|
||||
userId,
|
||||
skillId: skill._id,
|
||||
})
|
||||
return json(result, 200, rate.headers)
|
||||
} catch {
|
||||
return text('Unauthorized', 401, rate.headers)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,247 @@
|
||||
import { api, internal } from '../_generated/api'
|
||||
import type { Id } from '../_generated/dataModel'
|
||||
import type { ActionCtx } from '../_generated/server'
|
||||
import { requireApiTokenUser } from '../lib/apiTokenAuth'
|
||||
import { applyRateLimit } from '../lib/httpRateLimit'
|
||||
import {
|
||||
getPathSegments,
|
||||
json,
|
||||
parseJsonPayload,
|
||||
requireAdminOrResponse,
|
||||
requireApiTokenUserOrResponse,
|
||||
text,
|
||||
toOptionalNumber,
|
||||
} from './shared'
|
||||
|
||||
export async function usersPostRouterV1Handler(ctx: ActionCtx, request: Request) {
|
||||
const rate = await applyRateLimit(ctx, request, 'write')
|
||||
if (!rate.ok) return rate.response
|
||||
|
||||
const segments = getPathSegments(request, '/api/v1/users/')
|
||||
if (segments.length !== 1) {
|
||||
return text('Not found', 404, rate.headers)
|
||||
}
|
||||
const action = segments[0]
|
||||
if (action !== 'ban' && action !== 'role' && action !== 'restore' && action !== 'reclaim') {
|
||||
return text('Not found', 404, rate.headers)
|
||||
}
|
||||
|
||||
const payloadResult = await parseJsonPayload(request, rate.headers)
|
||||
if (!payloadResult.ok) return payloadResult.response
|
||||
const payload = payloadResult.payload
|
||||
|
||||
const authResult = await requireApiTokenUserOrResponse(ctx, request, rate.headers)
|
||||
if (!authResult.ok) return authResult.response
|
||||
const actorUserId = authResult.userId
|
||||
const actorUser = authResult.user
|
||||
|
||||
// Restore and reclaim have different parameter shapes, handle them separately
|
||||
if (action === 'restore') {
|
||||
const admin = requireAdminOrResponse(actorUser, rate.headers)
|
||||
if (!admin.ok) return admin.response
|
||||
return handleAdminRestore(ctx, request, payload, actorUserId, rate.headers)
|
||||
}
|
||||
|
||||
if (action === 'reclaim') {
|
||||
const admin = requireAdminOrResponse(actorUser, rate.headers)
|
||||
if (!admin.ok) return admin.response
|
||||
return handleAdminReclaim(ctx, request, payload, actorUserId, rate.headers)
|
||||
}
|
||||
|
||||
const handleRaw = typeof payload.handle === 'string' ? payload.handle.trim() : ''
|
||||
const userIdRaw = typeof payload.userId === 'string' ? payload.userId.trim() : ''
|
||||
const reasonRaw = typeof payload.reason === 'string' ? payload.reason.trim() : ''
|
||||
if (!handleRaw && !userIdRaw) {
|
||||
return text('Missing userId or handle', 400, rate.headers)
|
||||
}
|
||||
|
||||
const roleRaw = typeof payload.role === 'string' ? payload.role.trim().toLowerCase() : ''
|
||||
if (action === 'role' && !roleRaw) {
|
||||
return text('Missing role', 400, rate.headers)
|
||||
}
|
||||
const role = roleRaw === 'user' || roleRaw === 'moderator' || roleRaw === 'admin' ? roleRaw : null
|
||||
if (action === 'role' && !role) {
|
||||
return text('Invalid role', 400, rate.headers)
|
||||
}
|
||||
|
||||
let targetUserId: Id<'users'> | null = userIdRaw ? (userIdRaw as Id<'users'>) : null
|
||||
if (!targetUserId) {
|
||||
const handle = handleRaw.toLowerCase()
|
||||
const user = await ctx.runQuery(api.users.getByHandle, { handle })
|
||||
if (!user?._id) return text('User not found', 404, rate.headers)
|
||||
targetUserId = user._id
|
||||
}
|
||||
|
||||
if (action === 'ban') {
|
||||
const reason = reasonRaw.length > 0 ? reasonRaw : undefined
|
||||
if (reason && reason.length > 500) {
|
||||
return text('Reason too long (max 500 chars)', 400, rate.headers)
|
||||
}
|
||||
try {
|
||||
const result = await ctx.runMutation(internal.users.banUserInternal, {
|
||||
actorUserId,
|
||||
targetUserId,
|
||||
reason,
|
||||
})
|
||||
return json(result, 200, rate.headers)
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : 'Ban failed'
|
||||
if (message.toLowerCase().includes('forbidden')) {
|
||||
return text('Forbidden', 403, rate.headers)
|
||||
}
|
||||
if (message.toLowerCase().includes('not found')) {
|
||||
return text(message, 404, rate.headers)
|
||||
}
|
||||
return text(message, 400, rate.headers)
|
||||
}
|
||||
}
|
||||
|
||||
if (!role) {
|
||||
return text('Invalid role', 400, rate.headers)
|
||||
}
|
||||
|
||||
try {
|
||||
const result = await ctx.runMutation(internal.users.setRoleInternal, {
|
||||
actorUserId,
|
||||
targetUserId,
|
||||
role,
|
||||
})
|
||||
return json({ ok: true, role: result.role ?? role }, 200, rate.headers)
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : 'Role change failed'
|
||||
if (message.toLowerCase().includes('forbidden')) {
|
||||
return text('Forbidden', 403, rate.headers)
|
||||
}
|
||||
if (message.toLowerCase().includes('not found')) {
|
||||
return text(message, 404, rate.headers)
|
||||
}
|
||||
return text(message, 400, rate.headers)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* POST /api/v1/users/restore
|
||||
* Admin-only: restore skills from GitHub backup for a user.
|
||||
* Body: { handle: string, slugs: string[], forceOverwriteSquatter?: boolean }
|
||||
*/
|
||||
async function handleAdminRestore(
|
||||
ctx: ActionCtx,
|
||||
_request: Request,
|
||||
payload: Record<string, unknown>,
|
||||
actorUserId: Id<'users'>,
|
||||
headers: HeadersInit,
|
||||
) {
|
||||
const handle = typeof payload.handle === 'string' ? payload.handle.trim().toLowerCase() : ''
|
||||
if (!handle) return text('Missing handle', 400, headers)
|
||||
|
||||
const slugs = Array.isArray(payload.slugs) ? payload.slugs.filter((s): s is string => typeof s === 'string') : []
|
||||
if (slugs.length === 0) return text('Missing slugs array', 400, headers)
|
||||
if (slugs.length > 100) return text('Too many slugs (max 100)', 400, headers)
|
||||
|
||||
const forceOverwriteSquatter = Boolean(payload.forceOverwriteSquatter)
|
||||
|
||||
const targetUser = await ctx.runQuery(api.users.getByHandle, { handle })
|
||||
if (!targetUser?._id) return text('User not found', 404, headers)
|
||||
|
||||
try {
|
||||
const result = await ctx.runAction(internal.githubRestore.restoreUserSkillsFromBackup, {
|
||||
actorUserId,
|
||||
ownerHandle: handle,
|
||||
ownerUserId: targetUser._id,
|
||||
slugs,
|
||||
forceOverwriteSquatter,
|
||||
})
|
||||
return json(result, 200, headers)
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : 'Restore failed'
|
||||
if (message.toLowerCase().includes('forbidden')) {
|
||||
return text('Forbidden', 403, headers)
|
||||
}
|
||||
return text(message, 400, headers)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* POST /api/v1/users/reclaim
|
||||
* Admin-only: reclaim root slugs for the rightful owner.
|
||||
* Default behavior is non-destructive owner transfer for existing skills
|
||||
* (preserves versions/stats/metadata) and leaves missing slugs untouched.
|
||||
* Body: { handle: string, slugs: string[], reason?: string }
|
||||
*/
|
||||
async function handleAdminReclaim(
|
||||
ctx: ActionCtx,
|
||||
_request: Request,
|
||||
payload: Record<string, unknown>,
|
||||
actorUserId: Id<'users'>,
|
||||
headers: HeadersInit,
|
||||
) {
|
||||
const handle = typeof payload.handle === 'string' ? payload.handle.trim().toLowerCase() : ''
|
||||
if (!handle) return text('Missing handle', 400, headers)
|
||||
|
||||
const slugs = Array.isArray(payload.slugs) ? payload.slugs.filter((s): s is string => typeof s === 'string') : []
|
||||
if (slugs.length === 0) return text('Missing slugs array', 400, headers)
|
||||
if (slugs.length > 200) return text('Too many slugs (max 200)', 400, headers)
|
||||
|
||||
const reason = typeof payload.reason === 'string' ? payload.reason.trim() : undefined
|
||||
|
||||
const targetUser = await ctx.runQuery(api.users.getByHandle, { handle })
|
||||
if (!targetUser?._id) return text('User not found', 404, headers)
|
||||
|
||||
const results: Array<{ slug: string; ok: boolean; action?: string; error?: string }> = []
|
||||
for (const slug of slugs) {
|
||||
try {
|
||||
const result = (await ctx.runMutation(internal.skills.reclaimSlugInternal, {
|
||||
actorUserId,
|
||||
slug: slug.trim().toLowerCase(),
|
||||
rightfulOwnerUserId: targetUser._id,
|
||||
reason,
|
||||
transferRootSlugOnly: true,
|
||||
})) as { action?: string }
|
||||
results.push({ slug, ok: true, action: result.action })
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : 'Reclaim failed'
|
||||
results.push({ slug, ok: false, error: message })
|
||||
}
|
||||
}
|
||||
|
||||
const succeeded = results.filter((r) => r.ok).length
|
||||
const failed = results.filter((r) => !r.ok).length
|
||||
|
||||
return json({ ok: true, results, succeeded, failed }, 200, headers)
|
||||
}
|
||||
|
||||
export async function usersListV1Handler(ctx: ActionCtx, request: Request) {
|
||||
const rate = await applyRateLimit(ctx, request, 'read')
|
||||
if (!rate.ok) return rate.response
|
||||
|
||||
const url = new URL(request.url)
|
||||
const limitRaw = toOptionalNumber(url.searchParams.get('limit'))
|
||||
const query = url.searchParams.get('q') ?? url.searchParams.get('query') ?? ''
|
||||
|
||||
let actorUserId: Id<'users'>
|
||||
try {
|
||||
const auth = await requireApiTokenUser(ctx, request)
|
||||
actorUserId = auth.userId
|
||||
} catch {
|
||||
return text('Unauthorized', 401, rate.headers)
|
||||
}
|
||||
|
||||
const limit = Math.min(Math.max(limitRaw ?? 20, 1), 200)
|
||||
try {
|
||||
const result = await ctx.runQuery(internal.users.searchInternal, {
|
||||
actorUserId,
|
||||
query,
|
||||
limit,
|
||||
})
|
||||
return json(result, 200, rate.headers)
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : 'User search failed'
|
||||
if (message.toLowerCase().includes('forbidden')) {
|
||||
return text('Forbidden', 403, rate.headers)
|
||||
}
|
||||
if (message.toLowerCase().includes('unauthorized')) {
|
||||
return text('Unauthorized', 401, rate.headers)
|
||||
}
|
||||
return text(message, 400, rate.headers)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
import type { ActionCtx } from '../_generated/server'
|
||||
import { requireApiTokenUser } from '../lib/apiTokenAuth'
|
||||
import { applyRateLimit } from '../lib/httpRateLimit'
|
||||
import { json, text } from './shared'
|
||||
|
||||
export async function whoamiV1Handler(ctx: ActionCtx, request: Request) {
|
||||
const rate = await applyRateLimit(ctx, request, 'read')
|
||||
if (!rate.ok) return rate.response
|
||||
|
||||
try {
|
||||
const { user } = await requireApiTokenUser(ctx, request)
|
||||
return json(
|
||||
{
|
||||
user: {
|
||||
handle: user.handle ?? null,
|
||||
displayName: user.displayName ?? null,
|
||||
image: user.image ?? null,
|
||||
},
|
||||
},
|
||||
200,
|
||||
rate.headers,
|
||||
)
|
||||
} catch {
|
||||
return text('Unauthorized', 401, rate.headers)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
|
||||
vi.mock('@convex-dev/auth/server', () => ({
|
||||
getAuthUserId: vi.fn(),
|
||||
}))
|
||||
|
||||
const { getAuthUserId } = await import('@convex-dev/auth/server')
|
||||
const {
|
||||
assertAdmin,
|
||||
assertModerator,
|
||||
assertRole,
|
||||
requireUser,
|
||||
requireUserFromAction,
|
||||
} = await import('./access')
|
||||
|
||||
describe('access.requireUser', () => {
|
||||
it('throws when auth is missing', async () => {
|
||||
vi.mocked(getAuthUserId).mockResolvedValue(null)
|
||||
await expect(
|
||||
requireUser({
|
||||
db: { get: vi.fn() },
|
||||
} as never),
|
||||
).rejects.toThrow('Unauthorized')
|
||||
})
|
||||
|
||||
it('throws when user is deleted/deactivated/missing', async () => {
|
||||
vi.mocked(getAuthUserId).mockResolvedValue('users:1' as never)
|
||||
|
||||
for (const value of [null, { _id: 'users:1', deletedAt: Date.now() }, { _id: 'users:1', deactivatedAt: Date.now() }]) {
|
||||
const dbGet = vi.fn().mockResolvedValue(value as never)
|
||||
await expect(
|
||||
requireUser({
|
||||
db: { get: dbGet },
|
||||
} as never),
|
||||
).rejects.toThrow('User not found')
|
||||
}
|
||||
})
|
||||
|
||||
it('returns auth user when active', async () => {
|
||||
vi.mocked(getAuthUserId).mockResolvedValue('users:2' as never)
|
||||
const user = { _id: 'users:2', role: 'user' }
|
||||
const dbGet = vi.fn().mockResolvedValue(user as never)
|
||||
|
||||
const result = await requireUser({
|
||||
db: { get: dbGet },
|
||||
} as never)
|
||||
|
||||
expect(dbGet).toHaveBeenCalledWith('users:2')
|
||||
expect(result).toEqual({ userId: 'users:2', user })
|
||||
})
|
||||
})
|
||||
|
||||
describe('access.requireUserFromAction', () => {
|
||||
it('throws when auth is missing', async () => {
|
||||
vi.mocked(getAuthUserId).mockResolvedValue(null)
|
||||
await expect(
|
||||
requireUserFromAction({
|
||||
runQuery: vi.fn(),
|
||||
} as never),
|
||||
).rejects.toThrow('Unauthorized')
|
||||
})
|
||||
|
||||
it('throws when action lookup returns deleted/deactivated/missing user', async () => {
|
||||
vi.mocked(getAuthUserId).mockResolvedValue('users:1' as never)
|
||||
|
||||
for (const value of [null, { _id: 'users:1', deletedAt: Date.now() }, { _id: 'users:1', deactivatedAt: Date.now() }]) {
|
||||
const runQuery = vi.fn().mockResolvedValue(value as never)
|
||||
await expect(
|
||||
requireUserFromAction({
|
||||
runQuery,
|
||||
} as never),
|
||||
).rejects.toThrow('User not found')
|
||||
}
|
||||
})
|
||||
|
||||
it('returns active user from action query', async () => {
|
||||
vi.mocked(getAuthUserId).mockResolvedValue('users:9' as never)
|
||||
const user = { _id: 'users:9', role: 'admin' }
|
||||
const runQuery = vi.fn().mockResolvedValue(user as never)
|
||||
|
||||
const result = await requireUserFromAction({
|
||||
runQuery,
|
||||
} as never)
|
||||
|
||||
expect(runQuery).toHaveBeenCalledTimes(1)
|
||||
expect(result).toEqual({ userId: 'users:9', user })
|
||||
})
|
||||
})
|
||||
|
||||
describe('access role assertions', () => {
|
||||
it('assertRole allows matching roles and rejects missing role', () => {
|
||||
expect(() => assertRole({ role: 'admin' } as never, ['admin'])).not.toThrow()
|
||||
expect(() => assertRole({ role: undefined } as never, ['admin'])).toThrow('Forbidden')
|
||||
expect(() => assertRole({ role: 'user' } as never, ['admin'])).toThrow('Forbidden')
|
||||
})
|
||||
|
||||
it('assertAdmin/assertModerator enforce expected policy', () => {
|
||||
expect(() => assertAdmin({ role: 'admin' } as never)).not.toThrow()
|
||||
expect(() => assertAdmin({ role: 'moderator' } as never)).toThrow('Forbidden')
|
||||
|
||||
expect(() => assertModerator({ role: 'admin' } as never)).not.toThrow()
|
||||
expect(() => assertModerator({ role: 'moderator' } as never)).not.toThrow()
|
||||
expect(() => assertModerator({ role: 'user' } as never)).toThrow('Forbidden')
|
||||
})
|
||||
})
|
||||
@@ -35,7 +35,7 @@ export async function getSkillBadgeMap(
|
||||
const records = await ctx.db
|
||||
.query('skillBadges')
|
||||
.withIndex('by_skill', (q) => q.eq('skillId', skillId))
|
||||
.collect()
|
||||
.take(10)
|
||||
return buildBadgeMap(records)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
import type { Scheduler } from 'convex/server'
|
||||
|
||||
export function scheduleNextBatchIfNeeded<TArgs extends { cursor?: string }>(
|
||||
scheduler: Scheduler,
|
||||
fn: unknown,
|
||||
args: TArgs,
|
||||
isDone: boolean,
|
||||
continueCursor: string | null,
|
||||
) {
|
||||
if (isDone) return
|
||||
void scheduler.runAfter(0, fn as never, {
|
||||
...args,
|
||||
cursor: continueCursor ?? undefined,
|
||||
} as never)
|
||||
}
|
||||
+1
-21
@@ -1,6 +1,7 @@
|
||||
import { internal } from '../_generated/api'
|
||||
import type { Doc, Id } from '../_generated/dataModel'
|
||||
import type { ActionCtx } from '../_generated/server'
|
||||
import { extractResponseText } from './openaiResponse'
|
||||
|
||||
const CHANGELOG_MODEL = process.env.OPENAI_CHANGELOG_MODEL ?? 'gpt-4.1'
|
||||
const MAX_README_CHARS = 8_000
|
||||
@@ -59,27 +60,6 @@ function pickPaths(values: string[]) {
|
||||
return values.slice(0, MAX_PATHS_IN_PROMPT)
|
||||
}
|
||||
|
||||
function extractResponseText(payload: unknown) {
|
||||
if (!payload || typeof payload !== 'object') return null
|
||||
const output = (payload as { output?: unknown }).output
|
||||
if (!Array.isArray(output)) return null
|
||||
const chunks: string[] = []
|
||||
for (const item of output) {
|
||||
if (!item || typeof item !== 'object') continue
|
||||
if ((item as { type?: unknown }).type !== 'message') continue
|
||||
const content = (item as { content?: unknown }).content
|
||||
if (!Array.isArray(content)) continue
|
||||
for (const part of content) {
|
||||
if (!part || typeof part !== 'object') continue
|
||||
if ((part as { type?: unknown }).type !== 'output_text') continue
|
||||
const text = (part as { text?: unknown }).text
|
||||
if (typeof text === 'string' && text.trim()) chunks.push(text)
|
||||
}
|
||||
}
|
||||
const joined = chunks.join('\n').trim()
|
||||
return joined || null
|
||||
}
|
||||
|
||||
async function generateWithOpenAI(args: {
|
||||
slug: string
|
||||
version: string
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
const EXT_TO_TYPE: Record<string, string> = {
|
||||
md: 'text/markdown',
|
||||
mdx: 'text/markdown',
|
||||
json: 'application/json',
|
||||
json5: 'application/json',
|
||||
yaml: 'application/yaml',
|
||||
yml: 'application/yaml',
|
||||
toml: 'application/toml',
|
||||
svg: 'image/svg+xml',
|
||||
}
|
||||
|
||||
export function guessContentTypeForPath(path: string) {
|
||||
const trimmed = path.trim().toLowerCase()
|
||||
if (!trimmed) return 'application/octet-stream'
|
||||
const ext = trimmed.split('.').at(-1) ?? ''
|
||||
return EXT_TO_TYPE[ext] ?? 'application/octet-stream'
|
||||
}
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
export type EmbeddingVisibility =
|
||||
| 'latest'
|
||||
| 'latest-approved'
|
||||
| 'archived'
|
||||
| 'archived-approved'
|
||||
| 'deleted'
|
||||
|
||||
export function embeddingVisibilityFor(isLatest: boolean, isApproved: boolean): Exclude<
|
||||
EmbeddingVisibility,
|
||||
'deleted'
|
||||
> {
|
||||
if (isLatest && isApproved) return 'latest-approved'
|
||||
if (isLatest) return 'latest'
|
||||
if (isApproved) return 'archived-approved'
|
||||
return 'archived'
|
||||
}
|
||||
|
||||
@@ -2,14 +2,17 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
import { internal } from '../_generated/api'
|
||||
import { requireGitHubAccountAge } from './githubAccount'
|
||||
import { requireGitHubAccountAge, syncGitHubProfile } from './githubAccount'
|
||||
|
||||
vi.mock('../_generated/api', () => ({
|
||||
internal: {
|
||||
githubIdentity: {
|
||||
getGitHubProviderAccountIdInternal: Symbol('getGitHubProviderAccountIdInternal'),
|
||||
},
|
||||
users: {
|
||||
getByIdInternal: Symbol('getByIdInternal'),
|
||||
getGitHubProviderAccountIdInternal: Symbol('getGitHubProviderAccountIdInternal'),
|
||||
updateGithubMetaInternal: Symbol('updateGithubMetaInternal'),
|
||||
setGitHubCreatedAtInternal: Symbol('setGitHubCreatedAtInternal'),
|
||||
syncGitHubProfileInternal: Symbol('syncGitHubProfileInternal'),
|
||||
},
|
||||
},
|
||||
}))
|
||||
@@ -37,7 +40,6 @@ describe('requireGitHubAccountAge', () => {
|
||||
const runQuery = vi.fn().mockResolvedValue({
|
||||
_id: 'users:1',
|
||||
githubCreatedAt: now.getTime() - 10 * ONE_DAY_MS,
|
||||
githubFetchedAt: now.getTime() - ONE_DAY_MS,
|
||||
})
|
||||
const runMutation = vi.fn()
|
||||
const fetchMock = vi.fn()
|
||||
@@ -48,9 +50,10 @@ describe('requireGitHubAccountAge', () => {
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
expect(runMutation).not.toHaveBeenCalled()
|
||||
expect(runQuery).toHaveBeenCalledWith(internal.users.getByIdInternal, { userId: 'users:1' })
|
||||
expect(runQuery).not.toHaveBeenCalledWith(internal.users.getGitHubProviderAccountIdInternal, {
|
||||
userId: 'users:1',
|
||||
})
|
||||
expect(runQuery).not.toHaveBeenCalledWith(
|
||||
internal.githubIdentity.getGitHubProviderAccountIdInternal,
|
||||
{ userId: 'users:1' },
|
||||
)
|
||||
})
|
||||
|
||||
it('rejects deactivated users', async () => {
|
||||
@@ -77,7 +80,6 @@ describe('requireGitHubAccountAge', () => {
|
||||
const runQuery = vi.fn().mockResolvedValue({
|
||||
_id: 'users:1',
|
||||
githubCreatedAt: now.getTime() - 2 * ONE_DAY_MS,
|
||||
githubFetchedAt: now.getTime() - ONE_DAY_MS / 2,
|
||||
})
|
||||
const runMutation = vi.fn()
|
||||
|
||||
@@ -95,7 +97,6 @@ describe('requireGitHubAccountAge', () => {
|
||||
.mockResolvedValueOnce({
|
||||
_id: 'users:1',
|
||||
githubCreatedAt: undefined,
|
||||
githubFetchedAt: 0,
|
||||
})
|
||||
.mockResolvedValueOnce('12345')
|
||||
const runMutation = vi.fn()
|
||||
@@ -115,10 +116,9 @@ describe('requireGitHubAccountAge', () => {
|
||||
headers: expect.objectContaining({ 'User-Agent': 'clawhub' }),
|
||||
}),
|
||||
)
|
||||
expect(runMutation).toHaveBeenCalledWith(internal.users.updateGithubMetaInternal, {
|
||||
expect(runMutation).toHaveBeenCalledWith(internal.users.setGitHubCreatedAtInternal, {
|
||||
userId: 'users:1',
|
||||
githubCreatedAt: Date.parse('2020-01-01T00:00:00Z'),
|
||||
githubFetchedAt: now.getTime(),
|
||||
})
|
||||
})
|
||||
|
||||
@@ -127,7 +127,6 @@ describe('requireGitHubAccountAge', () => {
|
||||
.mockResolvedValueOnce({
|
||||
_id: 'users:1',
|
||||
githubCreatedAt: undefined,
|
||||
githubFetchedAt: 0,
|
||||
})
|
||||
.mockResolvedValueOnce(null)
|
||||
const runMutation = vi.fn()
|
||||
@@ -146,7 +145,6 @@ describe('requireGitHubAccountAge', () => {
|
||||
.mockResolvedValueOnce({
|
||||
_id: 'users:1',
|
||||
githubCreatedAt: undefined,
|
||||
githubFetchedAt: 0,
|
||||
})
|
||||
.mockResolvedValueOnce('abc123')
|
||||
const runMutation = vi.fn()
|
||||
@@ -165,7 +163,6 @@ describe('requireGitHubAccountAge', () => {
|
||||
.mockResolvedValueOnce({
|
||||
_id: 'users:1',
|
||||
githubCreatedAt: undefined,
|
||||
githubFetchedAt: 0,
|
||||
})
|
||||
.mockResolvedValueOnce('12345')
|
||||
const runMutation = vi.fn()
|
||||
@@ -182,7 +179,6 @@ describe('requireGitHubAccountAge', () => {
|
||||
.mockResolvedValueOnce({
|
||||
_id: 'users:1',
|
||||
githubCreatedAt: undefined,
|
||||
githubFetchedAt: 0,
|
||||
})
|
||||
.mockResolvedValueOnce('12345')
|
||||
const runMutation = vi.fn()
|
||||
@@ -199,7 +195,6 @@ describe('requireGitHubAccountAge', () => {
|
||||
.mockResolvedValueOnce({
|
||||
_id: 'users:1',
|
||||
githubCreatedAt: undefined,
|
||||
githubFetchedAt: 0,
|
||||
})
|
||||
.mockResolvedValueOnce('12345')
|
||||
const runMutation = vi.fn()
|
||||
@@ -216,7 +211,6 @@ describe('requireGitHubAccountAge', () => {
|
||||
.mockResolvedValueOnce({
|
||||
_id: 'users:1',
|
||||
githubCreatedAt: undefined,
|
||||
githubFetchedAt: 0,
|
||||
})
|
||||
.mockResolvedValueOnce('12345')
|
||||
const runMutation = vi.fn()
|
||||
@@ -242,7 +236,6 @@ describe('requireGitHubAccountAge', () => {
|
||||
.mockResolvedValueOnce({
|
||||
_id: 'users:1',
|
||||
githubCreatedAt: undefined,
|
||||
githubFetchedAt: 0,
|
||||
})
|
||||
.mockResolvedValueOnce('12345')
|
||||
const runMutation = vi.fn()
|
||||
@@ -268,3 +261,136 @@ describe('requireGitHubAccountAge', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('syncGitHubProfile', () => {
|
||||
beforeEach(() => {
|
||||
vi.restoreAllMocks()
|
||||
vi.unstubAllEnvs()
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
vi.useRealTimers()
|
||||
vi.unstubAllEnvs()
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
|
||||
it('skips recent syncs (throttle)', async () => {
|
||||
vi.useFakeTimers()
|
||||
const now = new Date('2026-02-02T12:00:00Z')
|
||||
vi.setSystemTime(now)
|
||||
|
||||
const runQuery = vi.fn()
|
||||
.mockResolvedValueOnce({
|
||||
_id: 'users:1',
|
||||
name: 'oldname',
|
||||
githubProfileSyncedAt: now.getTime(),
|
||||
})
|
||||
const runMutation = vi.fn()
|
||||
const fetchMock = vi.fn()
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
|
||||
await syncGitHubProfile({ runQuery, runMutation } as never, 'users:1' as never)
|
||||
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
expect(runMutation).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('updates profile even when only avatar changes', async () => {
|
||||
vi.useFakeTimers()
|
||||
const now = new Date('2026-02-02T12:00:00Z')
|
||||
vi.setSystemTime(now)
|
||||
|
||||
const runQuery = vi.fn()
|
||||
.mockResolvedValueOnce({
|
||||
_id: 'users:1',
|
||||
name: 'same',
|
||||
image: 'https://avatars.githubusercontent.com/u/1?v=3',
|
||||
githubProfileSyncedAt: now.getTime() - 10 * ONE_DAY_MS,
|
||||
})
|
||||
.mockResolvedValueOnce('12345')
|
||||
const runMutation = vi.fn()
|
||||
const fetchMock = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
login: 'same',
|
||||
avatar_url: 'https://avatars.githubusercontent.com/u/1?v=4',
|
||||
}),
|
||||
})
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
|
||||
await syncGitHubProfile({ runQuery, runMutation } as never, 'users:1' as never)
|
||||
|
||||
expect(runMutation).toHaveBeenCalledWith(internal.users.syncGitHubProfileInternal, {
|
||||
userId: 'users:1',
|
||||
name: 'same',
|
||||
image: 'https://avatars.githubusercontent.com/u/1?v=4',
|
||||
syncedAt: now.getTime(),
|
||||
})
|
||||
})
|
||||
|
||||
it('updates name and records sync timestamp', async () => {
|
||||
vi.useFakeTimers()
|
||||
const now = new Date('2026-02-02T12:00:00Z')
|
||||
vi.setSystemTime(now)
|
||||
|
||||
const runQuery = vi.fn()
|
||||
.mockResolvedValueOnce({
|
||||
_id: 'users:1',
|
||||
name: 'old',
|
||||
githubProfileSyncedAt: now.getTime() - 10 * ONE_DAY_MS,
|
||||
})
|
||||
.mockResolvedValueOnce('12345')
|
||||
const runMutation = vi.fn()
|
||||
const fetchMock = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
login: 'new',
|
||||
avatar_url: 'https://avatars.githubusercontent.com/u/1?v=1',
|
||||
}),
|
||||
})
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
|
||||
await syncGitHubProfile({ runQuery, runMutation } as never, 'users:1' as never)
|
||||
|
||||
expect(runMutation).toHaveBeenCalledWith(internal.users.syncGitHubProfileInternal, {
|
||||
userId: 'users:1',
|
||||
name: 'new',
|
||||
image: 'https://avatars.githubusercontent.com/u/1?v=1',
|
||||
syncedAt: now.getTime(),
|
||||
})
|
||||
})
|
||||
|
||||
it('forwards GitHub profile name (full name) when present', async () => {
|
||||
vi.useFakeTimers()
|
||||
const now = new Date('2026-02-02T12:00:00Z')
|
||||
vi.setSystemTime(now)
|
||||
|
||||
const runQuery = vi.fn()
|
||||
.mockResolvedValueOnce({
|
||||
_id: 'users:1',
|
||||
name: 'same',
|
||||
githubProfileSyncedAt: now.getTime() - 10 * ONE_DAY_MS,
|
||||
})
|
||||
.mockResolvedValueOnce('12345')
|
||||
const runMutation = vi.fn()
|
||||
const fetchMock = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
login: 'same',
|
||||
name: 'Real Name',
|
||||
avatar_url: 'https://avatars.githubusercontent.com/u/1?v=1',
|
||||
}),
|
||||
})
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
|
||||
await syncGitHubProfile({ runQuery, runMutation } as never, 'users:1' as never)
|
||||
|
||||
expect(runMutation).toHaveBeenCalledWith(internal.users.syncGitHubProfileInternal, {
|
||||
userId: 'users:1',
|
||||
name: 'same',
|
||||
image: 'https://avatars.githubusercontent.com/u/1?v=1',
|
||||
profileName: 'Real Name',
|
||||
syncedAt: now.getTime(),
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
+83
-15
@@ -2,14 +2,33 @@ import { ConvexError } from 'convex/values'
|
||||
import { internal } from '../_generated/api'
|
||||
import type { Id } from '../_generated/dataModel'
|
||||
import type { ActionCtx } from '../_generated/server'
|
||||
import { GITHUB_PROFILE_SYNC_WINDOW_MS } from './githubProfileSync'
|
||||
|
||||
const GITHUB_API = 'https://api.github.com'
|
||||
const MIN_ACCOUNT_AGE_MS = 7 * 24 * 60 * 60 * 1000
|
||||
|
||||
type GitHubUser = {
|
||||
login?: string
|
||||
name?: string
|
||||
avatar_url?: string
|
||||
created_at?: string
|
||||
}
|
||||
|
||||
function assertGitHubNumericId(providerAccountId: string) {
|
||||
if (!/^[0-9]+$/.test(providerAccountId)) {
|
||||
throw new ConvexError('GitHub account lookup failed')
|
||||
}
|
||||
}
|
||||
|
||||
function buildGitHubHeaders() {
|
||||
const headers: Record<string, string> = { 'User-Agent': 'clawhub' }
|
||||
const token = process.env.GITHUB_TOKEN
|
||||
if (token) {
|
||||
headers.Authorization = `Bearer ${token}`
|
||||
}
|
||||
return headers
|
||||
}
|
||||
|
||||
export async function requireGitHubAccountAge(ctx: ActionCtx, userId: Id<'users'>) {
|
||||
const user = await ctx.runQuery(internal.users.getByIdInternal, { userId })
|
||||
if (!user || user.deletedAt || user.deactivatedAt) throw new ConvexError('User not found')
|
||||
@@ -18,26 +37,19 @@ export async function requireGitHubAccountAge(ctx: ActionCtx, userId: Id<'users'
|
||||
let createdAt = user.githubCreatedAt ?? null
|
||||
|
||||
if (!createdAt) {
|
||||
const providerAccountId = await ctx.runQuery(internal.users.getGitHubProviderAccountIdInternal, {
|
||||
userId,
|
||||
})
|
||||
const providerAccountId = await ctx.runQuery(
|
||||
internal.githubIdentity.getGitHubProviderAccountIdInternal,
|
||||
{ userId },
|
||||
)
|
||||
if (!providerAccountId) {
|
||||
// Invariant: GitHub is our only auth provider, so this should never happen.
|
||||
throw new ConvexError('GitHub account required')
|
||||
}
|
||||
if (!/^[0-9]+$/.test(providerAccountId)) {
|
||||
throw new ConvexError('GitHub account lookup failed')
|
||||
}
|
||||
|
||||
const headers: Record<string, string> = { 'User-Agent': 'clawhub' }
|
||||
const token = process.env.GITHUB_TOKEN
|
||||
if (token) {
|
||||
headers.Authorization = `Bearer ${token}`
|
||||
}
|
||||
assertGitHubNumericId(providerAccountId)
|
||||
|
||||
// Fetch by immutable GitHub numeric ID to avoid username swap attacks entirely.
|
||||
const response = await fetch(`${GITHUB_API}/user/${providerAccountId}`, {
|
||||
headers,
|
||||
headers: buildGitHubHeaders(),
|
||||
})
|
||||
if (!response.ok) {
|
||||
if (response.status === 403 || response.status === 429) {
|
||||
@@ -51,10 +63,9 @@ export async function requireGitHubAccountAge(ctx: ActionCtx, userId: Id<'users'
|
||||
if (!Number.isFinite(parsed)) throw new ConvexError('GitHub account lookup failed')
|
||||
|
||||
createdAt = parsed
|
||||
await ctx.runMutation(internal.users.updateGithubMetaInternal, {
|
||||
await ctx.runMutation(internal.users.setGitHubCreatedAtInternal, {
|
||||
userId,
|
||||
githubCreatedAt: createdAt,
|
||||
githubFetchedAt: now,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -71,3 +82,60 @@ export async function requireGitHubAccountAge(ctx: ActionCtx, userId: Id<'users'
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Sync the user's GitHub profile (username, avatar) from the GitHub API.
|
||||
* This handles the case where a user renames their GitHub account.
|
||||
* Uses the immutable GitHub numeric ID to fetch the current profile.
|
||||
*/
|
||||
export async function syncGitHubProfile(ctx: ActionCtx, userId: Id<'users'>) {
|
||||
const user = await ctx.runQuery(internal.users.getByIdInternal, { userId })
|
||||
if (!user || user.deletedAt || user.deactivatedAt) return
|
||||
|
||||
const now = Date.now()
|
||||
const lastSyncedAt = user.githubProfileSyncedAt ?? null
|
||||
if (lastSyncedAt && now - lastSyncedAt < GITHUB_PROFILE_SYNC_WINDOW_MS) return
|
||||
|
||||
const providerAccountId = await ctx.runQuery(
|
||||
internal.githubIdentity.getGitHubProviderAccountIdInternal,
|
||||
{ userId },
|
||||
)
|
||||
if (!providerAccountId) return
|
||||
|
||||
assertGitHubNumericId(providerAccountId)
|
||||
|
||||
const response = await fetch(`${GITHUB_API}/user/${providerAccountId}`, {
|
||||
headers: buildGitHubHeaders(),
|
||||
})
|
||||
if (!response.ok) {
|
||||
// Silently fail - this is a best-effort sync, not critical path
|
||||
console.warn(`[syncGitHubProfile] GitHub API error for user ${userId}: ${response.status}`)
|
||||
return
|
||||
}
|
||||
|
||||
const payload = (await response.json()) as GitHubUser
|
||||
const newLogin = payload.login?.trim()
|
||||
const newImage = payload.avatar_url?.trim()
|
||||
|
||||
const profileName = payload.name?.trim()
|
||||
|
||||
if (!newLogin) return
|
||||
|
||||
const args: {
|
||||
userId: Id<'users'>
|
||||
name: string
|
||||
image?: string
|
||||
syncedAt: number
|
||||
profileName?: string
|
||||
} = {
|
||||
userId,
|
||||
name: newLogin,
|
||||
image: newImage,
|
||||
syncedAt: now,
|
||||
}
|
||||
if (profileName && profileName !== newLogin) {
|
||||
args.profileName = profileName
|
||||
}
|
||||
|
||||
await ctx.runMutation(internal.users.syncGitHubProfileInternal, args)
|
||||
}
|
||||
|
||||
+105
-1
@@ -74,6 +74,13 @@ export type GitHubBackupContext = {
|
||||
root: string
|
||||
}
|
||||
|
||||
export type GitHubSkillBackupEntry = {
|
||||
owner: string
|
||||
slug: string
|
||||
rootPath: string
|
||||
metaPath: string
|
||||
}
|
||||
|
||||
export function isGitHubBackupConfigured() {
|
||||
return Boolean(
|
||||
process.env.GITHUB_APP_ID &&
|
||||
@@ -108,6 +115,103 @@ export async function fetchGitHubSkillMeta(
|
||||
)
|
||||
}
|
||||
|
||||
export async function listGitHubSkillBackupEntries(
|
||||
context: GitHubBackupContext,
|
||||
): Promise<GitHubSkillBackupEntry[]> {
|
||||
const ref = await githubGet<GitRef>(
|
||||
context.token,
|
||||
`/repos/${context.repoOwner}/${context.repoName}/git/ref/heads/${context.branch}`,
|
||||
)
|
||||
const baseCommit = await githubGet<GitCommit>(
|
||||
context.token,
|
||||
`/repos/${context.repoOwner}/${context.repoName}/git/commits/${ref.object.sha}`,
|
||||
)
|
||||
const tree = await githubGet<GitTree>(
|
||||
context.token,
|
||||
`/repos/${context.repoOwner}/${context.repoName}/git/trees/${baseCommit.tree.sha}?recursive=1`,
|
||||
)
|
||||
|
||||
const prefix = context.root ? `${context.root}/` : ''
|
||||
const entries: GitHubSkillBackupEntry[] = []
|
||||
for (const entry of tree.tree ?? []) {
|
||||
if (entry.type !== 'blob' || !entry.path) continue
|
||||
if (!entry.path.startsWith(prefix) || !entry.path.endsWith(`/${META_FILENAME}`)) continue
|
||||
const relative = entry.path.slice(prefix.length)
|
||||
const segments = relative.split('/')
|
||||
if (segments.length !== 3) continue
|
||||
const [owner, slug, file] = segments
|
||||
if (file !== META_FILENAME) continue
|
||||
const rootPath = prefix ? `${prefix}${owner}/${slug}` : `${owner}/${slug}`
|
||||
entries.push({ owner, slug, rootPath, metaPath: entry.path })
|
||||
}
|
||||
|
||||
return entries
|
||||
}
|
||||
|
||||
export async function deleteGitHubSkillBackup(
|
||||
context: GitHubBackupContext,
|
||||
ownerHandle: string,
|
||||
slug: string,
|
||||
) {
|
||||
const skillRoot = buildSkillRoot(context.root, ownerHandle, slug)
|
||||
const ref = await githubGet<GitRef>(
|
||||
context.token,
|
||||
`/repos/${context.repoOwner}/${context.repoName}/git/ref/heads/${context.branch}`,
|
||||
)
|
||||
const baseCommitSha = ref.object.sha
|
||||
const baseCommit = await githubGet<GitCommit>(
|
||||
context.token,
|
||||
`/repos/${context.repoOwner}/${context.repoName}/git/commits/${baseCommitSha}`,
|
||||
)
|
||||
const baseTreeSha = baseCommit.tree.sha
|
||||
const existingTree = await githubGet<GitTree>(
|
||||
context.token,
|
||||
`/repos/${context.repoOwner}/${context.repoName}/git/trees/${baseTreeSha}?recursive=1`,
|
||||
)
|
||||
|
||||
const prefix = `${skillRoot}/`
|
||||
const pathsToDelete = (existingTree.tree ?? [])
|
||||
.filter((entry) => entry.type === 'blob' && entry.path?.startsWith(prefix))
|
||||
.map((entry) => entry.path ?? '')
|
||||
.filter(Boolean)
|
||||
|
||||
if (!pathsToDelete.length) return { deleted: false as const }
|
||||
|
||||
const treeEntries = pathsToDelete.map((path) => ({
|
||||
path,
|
||||
mode: '100644' as const,
|
||||
type: 'blob' as const,
|
||||
sha: null,
|
||||
}))
|
||||
|
||||
const newTree = await githubPost<{ sha: string }>(
|
||||
context.token,
|
||||
`/repos/${context.repoOwner}/${context.repoName}/git/trees`,
|
||||
{
|
||||
base_tree: baseTreeSha,
|
||||
tree: treeEntries,
|
||||
},
|
||||
)
|
||||
|
||||
const commit = await githubPost<GitCommit>(
|
||||
context.token,
|
||||
`/repos/${context.repoOwner}/${context.repoName}/git/commits`,
|
||||
{
|
||||
message: `delete: ${skillRoot}`,
|
||||
tree: newTree.sha,
|
||||
parents: [baseCommitSha],
|
||||
},
|
||||
)
|
||||
|
||||
await githubPatch(
|
||||
context.token,
|
||||
`/repos/${context.repoOwner}/${context.repoName}/git/refs/heads/${context.branch}`,
|
||||
{ sha: commit.sha },
|
||||
)
|
||||
|
||||
return { deleted: true as const }
|
||||
}
|
||||
|
||||
export async function backupSkillToGitHub(
|
||||
ctx: ActionCtx,
|
||||
params: BackupParams,
|
||||
@@ -397,7 +501,7 @@ function parseRepo(repo: string) {
|
||||
return [owner, name] as const
|
||||
}
|
||||
|
||||
function normalizeOwner(value: string) {
|
||||
export function normalizeOwner(value: string) {
|
||||
const normalized = value
|
||||
.trim()
|
||||
.toLowerCase()
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
export const GITHUB_PROFILE_SYNC_WINDOW_MS = 6 * 60 * 60 * 1000
|
||||
|
||||
export function shouldScheduleGitHubProfileSync(
|
||||
user:
|
||||
| {
|
||||
deletedAt?: number
|
||||
deactivatedAt?: number
|
||||
githubProfileSyncedAt?: number
|
||||
}
|
||||
| null
|
||||
| undefined,
|
||||
now: number,
|
||||
) {
|
||||
if (!user || user.deletedAt || user.deactivatedAt) return false
|
||||
const lastSyncedAt = user.githubProfileSyncedAt ?? null
|
||||
if (lastSyncedAt && now - lastSyncedAt < GITHUB_PROFILE_SYNC_WINDOW_MS) return false
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
/* @vitest-environment node */
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
import type { GitHubBackupContext } from './githubBackup'
|
||||
import { readGitHubBackupFile } from './githubRestoreHelpers'
|
||||
|
||||
function makeContext(): GitHubBackupContext {
|
||||
return {
|
||||
token: 'token',
|
||||
repo: 'owner/repo',
|
||||
repoOwner: 'owner',
|
||||
repoName: 'repo',
|
||||
branch: 'main',
|
||||
root: 'skills',
|
||||
}
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals()
|
||||
vi.restoreAllMocks()
|
||||
})
|
||||
|
||||
describe('githubRestoreHelpers', () => {
|
||||
it('decodes base64 payloads (including newlines) into bytes', async () => {
|
||||
const content = 'SGVs\n bG8h' // "Hello!" with whitespace/newline
|
||||
vi.stubGlobal(
|
||||
'fetch',
|
||||
vi.fn(async () => ({
|
||||
ok: true,
|
||||
json: async () => ({ content, encoding: 'base64' }),
|
||||
text: async () => '',
|
||||
})),
|
||||
)
|
||||
|
||||
const bytes = await readGitHubBackupFile(makeContext(), 'Owner', 'slug', 'SKILL.md')
|
||||
expect(bytes).not.toBeNull()
|
||||
expect(Buffer.from(bytes!).toString('utf8')).toBe('Hello!')
|
||||
})
|
||||
|
||||
it('throws on unsupported GitHub content encoding', async () => {
|
||||
vi.stubGlobal(
|
||||
'fetch',
|
||||
vi.fn(async () => ({
|
||||
ok: true,
|
||||
json: async () => ({ content: 'eA==', encoding: 'utf-16' }),
|
||||
text: async () => '',
|
||||
})),
|
||||
)
|
||||
|
||||
await expect(readGitHubBackupFile(makeContext(), 'Owner', 'slug', 'SKILL.md')).rejects.toThrow(
|
||||
/Unsupported GitHub content encoding/i,
|
||||
)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,159 @@
|
||||
'use node'
|
||||
|
||||
import type { GitHubBackupContext } from './githubBackup'
|
||||
|
||||
const GITHUB_API = 'https://api.github.com'
|
||||
const META_FILENAME = '_meta.json'
|
||||
const USER_AGENT = 'clawhub/skills-restore'
|
||||
|
||||
type GitHubContentsEntry = {
|
||||
name?: string
|
||||
path?: string
|
||||
type?: string // 'file' | 'dir'
|
||||
size?: number
|
||||
}
|
||||
|
||||
type GitHubBlobResponse = {
|
||||
content?: string
|
||||
encoding?: string
|
||||
size?: number
|
||||
}
|
||||
|
||||
/**
|
||||
* List all files in a skill's backup directory (excluding _meta.json).
|
||||
* Uses the Contents API scoped to the target directory instead of fetching
|
||||
* the entire repository tree, which is critical for bulk restore performance.
|
||||
* Returns relative file paths (e.g. "SKILL.md", "lib/helper.ts").
|
||||
*/
|
||||
export async function listGitHubBackupFiles(
|
||||
context: GitHubBackupContext,
|
||||
ownerHandle: string,
|
||||
slug: string,
|
||||
): Promise<string[]> {
|
||||
const skillRoot = buildSkillRoot(context.root, ownerHandle, slug)
|
||||
return listFilesRecursive(context, skillRoot, '')
|
||||
}
|
||||
|
||||
/**
|
||||
* Recursively list files under a directory using the GitHub Contents API.
|
||||
* Each call is scoped to one directory, avoiding full-repo tree downloads.
|
||||
*/
|
||||
async function listFilesRecursive(
|
||||
context: GitHubBackupContext,
|
||||
basePath: string,
|
||||
relativePath: string,
|
||||
): Promise<string[]> {
|
||||
const dirPath = relativePath ? `${basePath}/${relativePath}` : basePath
|
||||
|
||||
try {
|
||||
const entries = await githubGet<GitHubContentsEntry[]>(
|
||||
context.token,
|
||||
`/repos/${context.repoOwner}/${context.repoName}/contents/${encodePath(dirPath)}?ref=${context.branch}`,
|
||||
)
|
||||
|
||||
if (!Array.isArray(entries)) return []
|
||||
|
||||
const files: string[] = []
|
||||
for (const entry of entries) {
|
||||
if (!entry.name || !entry.type) continue
|
||||
|
||||
const entryRelative = relativePath ? `${relativePath}/${entry.name}` : entry.name
|
||||
|
||||
if (entry.type === 'file') {
|
||||
// Skip the meta file
|
||||
if (entry.name === META_FILENAME) continue
|
||||
files.push(entryRelative)
|
||||
} else if (entry.type === 'dir') {
|
||||
// Recurse into subdirectories
|
||||
const subFiles = await listFilesRecursive(context, basePath, entryRelative)
|
||||
files.push(...subFiles)
|
||||
}
|
||||
}
|
||||
|
||||
return files
|
||||
} catch (error) {
|
||||
if (isNotFoundError(error)) return []
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a single file from the GitHub backup repository.
|
||||
* Returns the file content as a Uint8Array, or null if not found.
|
||||
*/
|
||||
export async function readGitHubBackupFile(
|
||||
context: GitHubBackupContext,
|
||||
ownerHandle: string,
|
||||
slug: string,
|
||||
filePath: string,
|
||||
): Promise<Uint8Array | null> {
|
||||
const skillRoot = buildSkillRoot(context.root, ownerHandle, slug)
|
||||
const fullPath = `${skillRoot}/${filePath}`
|
||||
|
||||
try {
|
||||
const response = await githubGet<GitHubBlobResponse>(
|
||||
context.token,
|
||||
`/repos/${context.repoOwner}/${context.repoName}/contents/${encodePath(fullPath)}?ref=${context.branch}`,
|
||||
)
|
||||
|
||||
if (!response.content) return null
|
||||
|
||||
if (response.encoding && response.encoding !== 'base64') {
|
||||
throw new Error(`Unsupported GitHub content encoding: ${response.encoding}`)
|
||||
}
|
||||
|
||||
return fromBase64Bytes(response.content)
|
||||
} catch (error) {
|
||||
if (isNotFoundError(error)) return null
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
function buildSkillRoot(root: string, ownerHandle: string, slug: string) {
|
||||
const ownerSegment = normalizeOwner(ownerHandle)
|
||||
return `${root}/${ownerSegment}/${slug}`
|
||||
}
|
||||
|
||||
function normalizeOwner(value: string) {
|
||||
const normalized = value
|
||||
.trim()
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9-]/g, '-')
|
||||
.replace(/-+/g, '-')
|
||||
.replace(/^-+|-+$/g, '')
|
||||
return normalized || 'unknown'
|
||||
}
|
||||
|
||||
function encodePath(path: string) {
|
||||
return path
|
||||
.split('/')
|
||||
.map((segment) => encodeURIComponent(segment))
|
||||
.join('/')
|
||||
}
|
||||
|
||||
function fromBase64Bytes(value: string) {
|
||||
// GitHub may include newlines in the base64 payload.
|
||||
const normalized = value.replace(/\s/g, '')
|
||||
return new Uint8Array(Buffer.from(normalized, 'base64'))
|
||||
}
|
||||
|
||||
async function githubGet<T>(token: string, path: string): Promise<T> {
|
||||
const response = await fetch(`${GITHUB_API}${path}`, {
|
||||
headers: {
|
||||
Authorization: `token ${token}`,
|
||||
Accept: 'application/vnd.github+json',
|
||||
'User-Agent': USER_AGENT,
|
||||
},
|
||||
})
|
||||
if (!response.ok) {
|
||||
const message = await response.text()
|
||||
throw new Error(`GitHub GET ${path} failed: ${message}`)
|
||||
}
|
||||
return (await response.json()) as T
|
||||
}
|
||||
|
||||
function isNotFoundError(error: unknown) {
|
||||
return (
|
||||
error instanceof Error && (error.message.includes('404') || error.message.includes('Not Found'))
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
import type { Doc } from '../_generated/dataModel'
|
||||
import type { MutationCtx, QueryCtx } from '../_generated/server'
|
||||
|
||||
export const GLOBAL_STATS_KEY = 'default'
|
||||
|
||||
type SkillVisibilityFields = Pick<
|
||||
Doc<'skills'>,
|
||||
'softDeletedAt' | 'moderationStatus' | 'moderationFlags'
|
||||
>
|
||||
|
||||
type GlobalStatsReadCtx = Pick<MutationCtx | QueryCtx, 'db'>
|
||||
type GlobalStatsWriteCtx = Pick<MutationCtx, 'db'>
|
||||
|
||||
export function isPublicSkillDoc(skill: SkillVisibilityFields | null | undefined) {
|
||||
if (!skill || skill.softDeletedAt) return false
|
||||
if (skill.moderationStatus && skill.moderationStatus !== 'active') return false
|
||||
if (skill.moderationFlags?.includes('blocked.malware')) return false
|
||||
return true
|
||||
}
|
||||
|
||||
export function getPublicSkillVisibilityDelta(
|
||||
before: SkillVisibilityFields | null | undefined,
|
||||
after: SkillVisibilityFields | null | undefined,
|
||||
) {
|
||||
const beforePublic = isPublicSkillDoc(before)
|
||||
const afterPublic = isPublicSkillDoc(after)
|
||||
if (beforePublic === afterPublic) return 0
|
||||
return afterPublic ? 1 : -1
|
||||
}
|
||||
|
||||
function getErrorMessage(error: unknown) {
|
||||
if (typeof error === 'string') return error
|
||||
if (error && typeof error === 'object' && 'message' in error) {
|
||||
const message = (error as { message?: unknown }).message
|
||||
if (typeof message === 'string') return message
|
||||
}
|
||||
return ''
|
||||
}
|
||||
|
||||
export function isGlobalStatsStorageNotReadyError(error: unknown) {
|
||||
const message = getErrorMessage(error).toLowerCase()
|
||||
if (!message) return false
|
||||
const referencesGlobalStats = message.includes('globalstats') || message.includes('by_key')
|
||||
if (!referencesGlobalStats) return false
|
||||
return (
|
||||
message.includes('table') ||
|
||||
message.includes('index') ||
|
||||
message.includes('schema') ||
|
||||
message.includes('not found') ||
|
||||
message.includes('does not exist') ||
|
||||
message.includes('unknown')
|
||||
)
|
||||
}
|
||||
|
||||
export async function countPublicSkillsForGlobalStats(ctx: GlobalStatsReadCtx) {
|
||||
const skills = await ctx.db
|
||||
.query('skills')
|
||||
.withIndex('by_active_updated', (q) => q.eq('softDeletedAt', undefined))
|
||||
.collect()
|
||||
let count = 0
|
||||
for (const skill of skills) {
|
||||
if (isPublicSkillDoc(skill)) count += 1
|
||||
}
|
||||
return count
|
||||
}
|
||||
|
||||
export async function setGlobalPublicSkillsCount(
|
||||
ctx: GlobalStatsWriteCtx,
|
||||
count: number,
|
||||
now = Date.now(),
|
||||
) {
|
||||
const normalizedCount = Math.max(0, Math.trunc(Number.isFinite(count) ? count : 0))
|
||||
try {
|
||||
const existing = await ctx.db
|
||||
.query('globalStats')
|
||||
.withIndex('by_key', (q) => q.eq('key', GLOBAL_STATS_KEY))
|
||||
.unique()
|
||||
|
||||
if (existing) {
|
||||
await ctx.db.patch(existing._id, { activeSkillsCount: normalizedCount, updatedAt: now })
|
||||
} else {
|
||||
await ctx.db.insert('globalStats', {
|
||||
key: GLOBAL_STATS_KEY,
|
||||
activeSkillsCount: normalizedCount,
|
||||
updatedAt: now,
|
||||
})
|
||||
}
|
||||
} catch (error) {
|
||||
if (isGlobalStatsStorageNotReadyError(error)) return
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
export async function adjustGlobalPublicSkillsCount(
|
||||
ctx: GlobalStatsWriteCtx,
|
||||
delta: number,
|
||||
now = Date.now(),
|
||||
) {
|
||||
const normalizedDelta = Math.trunc(Number.isFinite(delta) ? delta : 0)
|
||||
if (normalizedDelta === 0) return
|
||||
|
||||
let existing:
|
||||
| {
|
||||
_id: Doc<'globalStats'>['_id']
|
||||
activeSkillsCount: number
|
||||
}
|
||||
| null
|
||||
| undefined
|
||||
try {
|
||||
existing = await ctx.db
|
||||
.query('globalStats')
|
||||
.withIndex('by_key', (q) => q.eq('key', GLOBAL_STATS_KEY))
|
||||
.unique()
|
||||
} catch (error) {
|
||||
if (isGlobalStatsStorageNotReadyError(error)) return
|
||||
throw error
|
||||
}
|
||||
|
||||
if (!existing) {
|
||||
// No baseline yet (e.g. fresh deploy). Initialize via full recount once.
|
||||
const count = await countPublicSkillsForGlobalStats(ctx)
|
||||
await setGlobalPublicSkillsCount(ctx, count, now)
|
||||
return
|
||||
}
|
||||
|
||||
const nextCount = Math.max(0, existing.activeSkillsCount + normalizedDelta)
|
||||
await ctx.db.patch(existing._id, { activeSkillsCount: nextCount, updatedAt: now })
|
||||
}
|
||||
|
||||
export async function readGlobalPublicSkillsCount(ctx: GlobalStatsReadCtx) {
|
||||
try {
|
||||
const stats = await ctx.db
|
||||
.query('globalStats')
|
||||
.withIndex('by_key', (q) => q.eq('key', GLOBAL_STATS_KEY))
|
||||
.unique()
|
||||
return stats?.activeSkillsCount ?? null
|
||||
} catch (error) {
|
||||
if (isGlobalStatsStorageNotReadyError(error)) return null
|
||||
throw error
|
||||
}
|
||||
}
|
||||
@@ -1,19 +1,76 @@
|
||||
/* @vitest-environment node */
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getClientIp } from './httpRateLimit'
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { applyRateLimit, getClientIp } from './httpRateLimit'
|
||||
|
||||
type MockRateLimitStatus = {
|
||||
allowed: boolean
|
||||
remaining: number
|
||||
limit: number
|
||||
resetAt: number
|
||||
}
|
||||
|
||||
type MockRateLimitPlan = {
|
||||
ip: MockRateLimitStatus
|
||||
user?: MockRateLimitStatus
|
||||
tokenValid?: boolean
|
||||
userActive?: boolean
|
||||
}
|
||||
|
||||
function makeRateLimitCtx(plan: MockRateLimitPlan) {
|
||||
const runQuery = vi.fn(async (_fn: unknown, args: Record<string, unknown>) => {
|
||||
if ('tokenHash' in args) {
|
||||
if (plan.tokenValid === false) return null
|
||||
return { _id: 'token_1', revokedAt: undefined }
|
||||
}
|
||||
if ('tokenId' in args) {
|
||||
if (plan.userActive === false) return null
|
||||
return { _id: 'users_123', deletedAt: undefined, deactivatedAt: undefined }
|
||||
}
|
||||
if ('key' in args && 'limit' in args && 'windowMs' in args) {
|
||||
const key = String(args.key)
|
||||
if (key.startsWith('ip:')) return plan.ip
|
||||
if (key.startsWith('user:')) return plan.user
|
||||
}
|
||||
throw new Error(`Unexpected runQuery args: ${JSON.stringify(args)}`)
|
||||
})
|
||||
|
||||
const runMutation = vi.fn(async (_fn: unknown, args: Record<string, unknown>) => {
|
||||
const key = String(args.key)
|
||||
const source = key.startsWith('user:') ? plan.user : plan.ip
|
||||
if (!source) throw new Error(`Missing rate limit source for ${key}`)
|
||||
return { allowed: source.allowed, remaining: source.remaining }
|
||||
})
|
||||
|
||||
return {
|
||||
runQuery,
|
||||
runMutation,
|
||||
} as unknown as Parameters<typeof applyRateLimit>[0]
|
||||
}
|
||||
|
||||
describe('getClientIp', () => {
|
||||
it('uses forwarded headers by default when cf-connecting-ip is missing', () => {
|
||||
let prev: string | undefined
|
||||
beforeEach(() => {
|
||||
prev = process.env.TRUST_FORWARDED_IPS
|
||||
})
|
||||
afterEach(() => {
|
||||
if (prev === undefined) {
|
||||
delete process.env.TRUST_FORWARDED_IPS
|
||||
} else {
|
||||
process.env.TRUST_FORWARDED_IPS = prev
|
||||
}
|
||||
})
|
||||
|
||||
it('returns null when cf-connecting-ip is missing (CF-only default)', () => {
|
||||
const request = new Request('https://example.com', {
|
||||
headers: {
|
||||
'x-forwarded-for': '203.0.113.9',
|
||||
},
|
||||
})
|
||||
delete process.env.TRUST_FORWARDED_IPS
|
||||
expect(getClientIp(request)).toBe('203.0.113.9')
|
||||
expect(getClientIp(request)).toBeNull()
|
||||
})
|
||||
|
||||
it('can disable forwarded headers explicitly', () => {
|
||||
it('keeps forwarded headers disabled when TRUST_FORWARDED_IPS=false', () => {
|
||||
const request = new Request('https://example.com', {
|
||||
headers: {
|
||||
'x-forwarded-for': '203.0.113.9',
|
||||
@@ -21,7 +78,6 @@ describe('getClientIp', () => {
|
||||
})
|
||||
process.env.TRUST_FORWARDED_IPS = 'false'
|
||||
expect(getClientIp(request)).toBeNull()
|
||||
delete process.env.TRUST_FORWARDED_IPS
|
||||
})
|
||||
|
||||
it('returns first ip from cf-connecting-ip', () => {
|
||||
@@ -41,6 +97,200 @@ describe('getClientIp', () => {
|
||||
})
|
||||
process.env.TRUST_FORWARDED_IPS = 'true'
|
||||
expect(getClientIp(request)).toBe('203.0.113.9')
|
||||
delete process.env.TRUST_FORWARDED_IPS
|
||||
})
|
||||
|
||||
it('prefers x-forwarded-for over x-real-ip when trusted mode is enabled', () => {
|
||||
const request = new Request('https://example.com', {
|
||||
headers: {
|
||||
'x-forwarded-for': '203.0.113.9, 198.51.100.2',
|
||||
'x-real-ip': '198.51.100.77',
|
||||
},
|
||||
})
|
||||
process.env.TRUST_FORWARDED_IPS = 'true'
|
||||
expect(getClientIp(request)).toBe('203.0.113.9')
|
||||
})
|
||||
})
|
||||
|
||||
describe('applyRateLimit headers', () => {
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks()
|
||||
})
|
||||
|
||||
it('returns delay-seconds Retry-After on 429 (not epoch)', async () => {
|
||||
vi.spyOn(Date, 'now').mockReturnValue(1_000_000)
|
||||
const runMutation = vi.fn()
|
||||
const ctx = {
|
||||
runQuery: vi.fn().mockResolvedValue({
|
||||
allowed: false,
|
||||
remaining: 0,
|
||||
limit: 20,
|
||||
resetAt: 1_030_500,
|
||||
}),
|
||||
runMutation,
|
||||
} as unknown as Parameters<typeof applyRateLimit>[0]
|
||||
const request = new Request('https://example.com', {
|
||||
headers: { 'cf-connecting-ip': '203.0.113.1' },
|
||||
})
|
||||
|
||||
const result = await applyRateLimit(ctx, request, 'download')
|
||||
expect(result.ok).toBe(false)
|
||||
if (result.ok) return
|
||||
expect(result.response.status).toBe(429)
|
||||
expect(result.response.headers.get('Retry-After')).toBe('31')
|
||||
expect(result.response.headers.get('X-RateLimit-Reset')).toBe('1031')
|
||||
expect(result.response.headers.get('RateLimit-Reset')).toBe('31')
|
||||
expect(runMutation).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('includes rate-limit headers without Retry-After when allowed', async () => {
|
||||
vi.spyOn(Date, 'now').mockReturnValue(2_000_000)
|
||||
const ctx = {
|
||||
runQuery: vi.fn().mockResolvedValue({
|
||||
allowed: true,
|
||||
remaining: 19,
|
||||
limit: 20,
|
||||
resetAt: 2_015_000,
|
||||
}),
|
||||
runMutation: vi.fn().mockResolvedValue({
|
||||
allowed: true,
|
||||
remaining: 18,
|
||||
}),
|
||||
} as unknown as Parameters<typeof applyRateLimit>[0]
|
||||
const request = new Request('https://example.com', {
|
||||
headers: { 'cf-connecting-ip': '203.0.113.1' },
|
||||
})
|
||||
|
||||
const result = await applyRateLimit(ctx, request, 'download')
|
||||
expect(result.ok).toBe(true)
|
||||
if (!result.ok) return
|
||||
const headers = new Headers(result.headers)
|
||||
expect(headers.get('X-RateLimit-Limit')).toBe('20')
|
||||
expect(headers.get('X-RateLimit-Remaining')).toBe('18')
|
||||
expect(headers.get('X-RateLimit-Reset')).toBe('2015')
|
||||
expect(headers.get('RateLimit-Limit')).toBe('20')
|
||||
expect(headers.get('RateLimit-Remaining')).toBe('18')
|
||||
expect(headers.get('RateLimit-Reset')).toBe('15')
|
||||
expect(headers.get('Retry-After')).toBeNull()
|
||||
})
|
||||
|
||||
it('allows authenticated users when user bucket is healthy and shared ip bucket is exhausted', async () => {
|
||||
vi.spyOn(Date, 'now').mockReturnValue(3_000_000)
|
||||
const ctx = makeRateLimitCtx({
|
||||
ip: {
|
||||
allowed: false,
|
||||
remaining: 0,
|
||||
limit: 20,
|
||||
resetAt: 3_040_000,
|
||||
},
|
||||
user: {
|
||||
allowed: true,
|
||||
remaining: 42,
|
||||
limit: 120,
|
||||
resetAt: 3_010_000,
|
||||
},
|
||||
})
|
||||
const request = new Request('https://example.com', {
|
||||
headers: {
|
||||
authorization: 'Bearer clh_token',
|
||||
'cf-connecting-ip': '203.0.113.1',
|
||||
},
|
||||
})
|
||||
|
||||
const result = await applyRateLimit(ctx, request, 'download')
|
||||
expect(result.ok).toBe(true)
|
||||
if (!result.ok) return
|
||||
const headers = new Headers(result.headers)
|
||||
expect(headers.get('X-RateLimit-Limit')).toBe('120')
|
||||
expect(headers.get('X-RateLimit-Remaining')).toBe('42')
|
||||
expect(headers.get('Retry-After')).toBeNull()
|
||||
})
|
||||
|
||||
it('does not consume ip bucket for authenticated requests', async () => {
|
||||
vi.spyOn(Date, 'now').mockReturnValue(3_100_000)
|
||||
const ctx = makeRateLimitCtx({
|
||||
ip: {
|
||||
allowed: true,
|
||||
remaining: 19,
|
||||
limit: 20,
|
||||
resetAt: 3_140_000,
|
||||
},
|
||||
user: {
|
||||
allowed: true,
|
||||
remaining: 41,
|
||||
limit: 120,
|
||||
resetAt: 3_110_000,
|
||||
},
|
||||
})
|
||||
const request = new Request('https://example.com', {
|
||||
headers: {
|
||||
authorization: 'Bearer clh_token',
|
||||
'cf-connecting-ip': '203.0.113.1',
|
||||
},
|
||||
})
|
||||
|
||||
const result = await applyRateLimit(ctx, request, 'download')
|
||||
expect(result.ok).toBe(true)
|
||||
const runMutation = (ctx as unknown as { runMutation: ReturnType<typeof vi.fn> }).runMutation
|
||||
const consumedKeys = runMutation.mock.calls.map(([, args]) => String(args.key))
|
||||
expect(consumedKeys.some((key) => key.startsWith('user:'))).toBe(true)
|
||||
expect(consumedKeys.some((key) => key.startsWith('ip:'))).toBe(false)
|
||||
})
|
||||
|
||||
it('denies authenticated users when user bucket is exhausted even if ip bucket is healthy', async () => {
|
||||
vi.spyOn(Date, 'now').mockReturnValue(4_000_000)
|
||||
const ctx = makeRateLimitCtx({
|
||||
ip: {
|
||||
allowed: true,
|
||||
remaining: 19,
|
||||
limit: 20,
|
||||
resetAt: 4_020_000,
|
||||
},
|
||||
user: {
|
||||
allowed: false,
|
||||
remaining: 0,
|
||||
limit: 120,
|
||||
resetAt: 4_030_000,
|
||||
},
|
||||
})
|
||||
const request = new Request('https://example.com', {
|
||||
headers: {
|
||||
authorization: 'Bearer clh_token',
|
||||
'cf-connecting-ip': '203.0.113.1',
|
||||
},
|
||||
})
|
||||
|
||||
const result = await applyRateLimit(ctx, request, 'download')
|
||||
expect(result.ok).toBe(false)
|
||||
if (result.ok) return
|
||||
expect(result.response.status).toBe(429)
|
||||
expect(result.response.headers.get('X-RateLimit-Limit')).toBe('120')
|
||||
expect(result.response.headers.get('X-RateLimit-Remaining')).toBe('0')
|
||||
expect(result.response.headers.get('Retry-After')).toBe('30')
|
||||
})
|
||||
|
||||
it('falls back to ip enforcement when bearer token is invalid', async () => {
|
||||
vi.spyOn(Date, 'now').mockReturnValue(5_000_000)
|
||||
const ctx = makeRateLimitCtx({
|
||||
tokenValid: false,
|
||||
ip: {
|
||||
allowed: false,
|
||||
remaining: 0,
|
||||
limit: 20,
|
||||
resetAt: 5_030_000,
|
||||
},
|
||||
})
|
||||
const request = new Request('https://example.com', {
|
||||
headers: {
|
||||
authorization: 'Bearer invalid',
|
||||
'cf-connecting-ip': '203.0.113.1',
|
||||
},
|
||||
})
|
||||
|
||||
const result = await applyRateLimit(ctx, request, 'download')
|
||||
expect(result.ok).toBe(false)
|
||||
if (result.ok) return
|
||||
expect(result.response.status).toBe(429)
|
||||
expect(result.response.headers.get('X-RateLimit-Limit')).toBe('20')
|
||||
expect(result.response.headers.get('Retry-After')).toBe('30')
|
||||
})
|
||||
})
|
||||
|
||||
+67
-20
@@ -1,7 +1,7 @@
|
||||
import { internal } from '../_generated/api'
|
||||
import type { ActionCtx } from '../_generated/server'
|
||||
import { getOptionalApiTokenUserId } from './apiTokenAuth'
|
||||
import { corsHeaders, mergeHeaders } from './httpHeaders'
|
||||
import { hashToken } from './tokens'
|
||||
|
||||
const RATE_LIMIT_WINDOW_MS = 60_000
|
||||
export const RATE_LIMITS = {
|
||||
@@ -22,17 +22,56 @@ export async function applyRateLimit(
|
||||
request: Request,
|
||||
kind: keyof typeof RATE_LIMITS,
|
||||
): Promise<{ ok: true; headers: HeadersInit } | { ok: false; response: Response }> {
|
||||
const userId = await getOptionalApiTokenUserId(ctx, request)
|
||||
const ip = getClientIp(request) ?? 'unknown'
|
||||
const ipSource = getClientIpSource(request)
|
||||
const hasClientIp = ip !== 'unknown'
|
||||
|
||||
// Authenticated requests are enforced and consumed by user bucket only to
|
||||
// avoid draining shared IP quota.
|
||||
if (userId) {
|
||||
const userResult = await checkRateLimit(ctx, `user:${userId}`, RATE_LIMITS[kind].key)
|
||||
const headers = rateHeaders(userResult)
|
||||
if (!userResult.allowed) {
|
||||
console.info('rate_limit_denied', {
|
||||
kind,
|
||||
auth: true,
|
||||
userAllowed: false,
|
||||
ipAllowed: null,
|
||||
ipSource,
|
||||
hasClientIp,
|
||||
})
|
||||
return {
|
||||
ok: false,
|
||||
response: new Response('Rate limit exceeded', {
|
||||
status: 429,
|
||||
headers: mergeHeaders(
|
||||
{
|
||||
'Content-Type': 'text/plain; charset=utf-8',
|
||||
'Cache-Control': 'no-store',
|
||||
},
|
||||
headers,
|
||||
corsHeaders(),
|
||||
),
|
||||
}),
|
||||
}
|
||||
}
|
||||
return { ok: true, headers }
|
||||
}
|
||||
|
||||
// Anonymous requests remain IP-enforced.
|
||||
const ipResult = await checkRateLimit(ctx, `ip:${ip}`, RATE_LIMITS[kind].ip)
|
||||
const token = parseBearerToken(request)
|
||||
const keyResult = token
|
||||
? await checkRateLimit(ctx, `key:${await hashToken(token)}`, RATE_LIMITS[kind].key)
|
||||
: null
|
||||
const headers = rateHeaders(ipResult)
|
||||
|
||||
const chosen = pickMostRestrictive(ipResult, keyResult)
|
||||
const headers = rateHeaders(chosen)
|
||||
|
||||
if (!ipResult.allowed || (keyResult && !keyResult.allowed)) {
|
||||
if (!ipResult.allowed) {
|
||||
console.info('rate_limit_denied', {
|
||||
kind,
|
||||
auth: false,
|
||||
userAllowed: null,
|
||||
ipAllowed: ipResult.allowed,
|
||||
ipSource,
|
||||
hasClientIp,
|
||||
})
|
||||
return {
|
||||
ok: false,
|
||||
response: new Response('Rate limit exceeded', {
|
||||
@@ -59,13 +98,22 @@ export function getClientIp(request: Request) {
|
||||
if (!shouldTrustForwardedIps()) return null
|
||||
|
||||
const forwarded =
|
||||
request.headers.get('x-real-ip') ??
|
||||
request.headers.get('x-forwarded-for') ??
|
||||
request.headers.get('x-real-ip') ??
|
||||
request.headers.get('fly-client-ip')
|
||||
|
||||
return splitFirstIp(forwarded)
|
||||
}
|
||||
|
||||
function getClientIpSource(request: Request) {
|
||||
if (request.headers.get('cf-connecting-ip')) return 'cf-connecting-ip'
|
||||
if (!shouldTrustForwardedIps()) return 'none'
|
||||
if (request.headers.get('x-forwarded-for')) return 'x-forwarded-for'
|
||||
if (request.headers.get('x-real-ip')) return 'x-real-ip'
|
||||
if (request.headers.get('fly-client-ip')) return 'fly-client-ip'
|
||||
return 'none'
|
||||
}
|
||||
|
||||
async function checkRateLimit(
|
||||
ctx: ActionCtx,
|
||||
key: string,
|
||||
@@ -110,20 +158,18 @@ async function checkRateLimit(
|
||||
}
|
||||
}
|
||||
|
||||
function pickMostRestrictive(primary: RateLimitResult, secondary: RateLimitResult | null) {
|
||||
if (!secondary) return primary
|
||||
if (!primary.allowed) return primary
|
||||
if (!secondary.allowed) return secondary
|
||||
return secondary.remaining < primary.remaining ? secondary : primary
|
||||
}
|
||||
|
||||
function rateHeaders(result: RateLimitResult): HeadersInit {
|
||||
const nowMs = Date.now()
|
||||
const resetSeconds = Math.ceil(result.resetAt / 1000)
|
||||
const resetDelaySeconds = Math.max(1, Math.ceil((result.resetAt - nowMs) / 1000))
|
||||
return {
|
||||
'X-RateLimit-Limit': String(result.limit),
|
||||
'X-RateLimit-Remaining': String(result.remaining),
|
||||
'X-RateLimit-Reset': String(resetSeconds),
|
||||
...(result.allowed ? {} : { 'Retry-After': String(resetSeconds) }),
|
||||
'RateLimit-Limit': String(result.limit),
|
||||
'RateLimit-Remaining': String(result.remaining),
|
||||
'RateLimit-Reset': String(resetDelaySeconds),
|
||||
...(result.allowed ? {} : { 'Retry-After': String(resetDelaySeconds) }),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -147,9 +193,10 @@ function shouldTrustForwardedIps() {
|
||||
const value = String(process.env.TRUST_FORWARDED_IPS ?? '')
|
||||
.trim()
|
||||
.toLowerCase()
|
||||
if (!value) return true
|
||||
// Hardening default: CF-only. Forwarded headers are trivial to spoof unless you
|
||||
// control the trusted proxy layer.
|
||||
if (!value) return false
|
||||
if (value === '1' || value === 'true' || value === 'yes') return true
|
||||
if (value === '0' || value === 'false' || value === 'no') return false
|
||||
return false
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { extractResponseText } from './openaiResponse'
|
||||
|
||||
describe('extractResponseText', () => {
|
||||
it('returns null for invalid payload shapes', () => {
|
||||
expect(extractResponseText(null)).toBeNull()
|
||||
expect(extractResponseText({})).toBeNull()
|
||||
expect(extractResponseText({ output: {} })).toBeNull()
|
||||
})
|
||||
|
||||
it('extracts output_text chunks from message content', () => {
|
||||
const payload = {
|
||||
output: [
|
||||
{ type: 'reasoning', content: [] },
|
||||
{
|
||||
type: 'message',
|
||||
content: [
|
||||
{ type: 'output_text', text: 'First line' },
|
||||
{ type: 'output_text', text: 'Second line' },
|
||||
],
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
expect(extractResponseText(payload)).toBe('First line\nSecond line')
|
||||
})
|
||||
|
||||
it('ignores blank and non-output_text parts', () => {
|
||||
const payload = {
|
||||
output: [
|
||||
{
|
||||
type: 'message',
|
||||
content: [
|
||||
{ type: 'input_text', text: 'ignored' },
|
||||
{ type: 'output_text', text: ' ' },
|
||||
{ type: 'output_text', text: 'kept' },
|
||||
],
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
expect(extractResponseText(payload)).toBe('kept')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,20 @@
|
||||
export function extractResponseText(payload: unknown): string | null {
|
||||
if (!payload || typeof payload !== 'object') return null
|
||||
const output = (payload as { output?: unknown }).output
|
||||
if (!Array.isArray(output)) return null
|
||||
const chunks: string[] = []
|
||||
for (const item of output) {
|
||||
if (!item || typeof item !== 'object') continue
|
||||
if ((item as { type?: unknown }).type !== 'message') continue
|
||||
const content = (item as { content?: unknown }).content
|
||||
if (!Array.isArray(content)) continue
|
||||
for (const part of content) {
|
||||
if (!part || typeof part !== 'object') continue
|
||||
if ((part as { type?: unknown }).type !== 'output_text') continue
|
||||
const text = (part as { text?: unknown }).text
|
||||
if (typeof text === 'string' && text.trim()) chunks.push(text)
|
||||
}
|
||||
}
|
||||
const joined = chunks.join('\n').trim()
|
||||
return joined || null
|
||||
}
|
||||
@@ -65,4 +65,32 @@ describe('public skill mapping', () => {
|
||||
comments: 0,
|
||||
})
|
||||
})
|
||||
|
||||
it('returns skill when moderationStatus is active', () => {
|
||||
const skill = makeSkill({ moderationStatus: 'active' })
|
||||
expect(toPublicSkill(skill)).not.toBeNull()
|
||||
})
|
||||
|
||||
it('filters out skill when moderationStatus is hidden', () => {
|
||||
const skill = makeSkill({ moderationStatus: 'hidden' })
|
||||
expect(toPublicSkill(skill)).toBeNull()
|
||||
})
|
||||
|
||||
it('returns skill when moderationStatus is undefined (legacy)', () => {
|
||||
const skill = makeSkill({ moderationStatus: undefined as unknown as string })
|
||||
expect(toPublicSkill(skill)).not.toBeNull()
|
||||
})
|
||||
|
||||
it('filters out soft-deleted skills', () => {
|
||||
const skill = makeSkill({ softDeletedAt: Date.now() })
|
||||
expect(toPublicSkill(skill)).toBeNull()
|
||||
})
|
||||
|
||||
it('filters out skills with blocked.malware flag', () => {
|
||||
const skill = makeSkill({
|
||||
moderationStatus: 'active',
|
||||
moderationFlags: ['blocked.malware'],
|
||||
})
|
||||
expect(toPublicSkill(skill)).toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import type { Doc } from '../_generated/dataModel'
|
||||
import { isPublicSkillDoc } from './globalStats'
|
||||
|
||||
export type PublicUser = Pick<
|
||||
Doc<'users'>,
|
||||
@@ -52,9 +53,8 @@ export function toPublicUser(user: Doc<'users'> | null | undefined): PublicUser
|
||||
}
|
||||
|
||||
export function toPublicSkill(skill: Doc<'skills'> | null | undefined): PublicSkill | null {
|
||||
if (!skill || skill.softDeletedAt) return null
|
||||
if (skill.moderationStatus && skill.moderationStatus !== 'active') return null
|
||||
if (skill.moderationFlags?.includes('blocked.malware')) return null
|
||||
if (!skill) return null
|
||||
if (!isPublicSkillDoc(skill)) return null
|
||||
const stats = {
|
||||
downloads:
|
||||
typeof skill.statsDownloads === 'number'
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
import type { Doc, Id } from '../_generated/dataModel'
|
||||
import type { MutationCtx, QueryCtx } from '../_generated/server'
|
||||
|
||||
type ReservedSlug = Doc<'reservedSlugs'>
|
||||
|
||||
const DEFAULT_ACTIVE_LIMIT = 25
|
||||
|
||||
function reservedSlugQuery(ctx: QueryCtx | MutationCtx, slug: string) {
|
||||
return ctx.db
|
||||
.query('reservedSlugs')
|
||||
.withIndex('by_slug_active_deletedAt', (q) => q.eq('slug', slug).eq('releasedAt', undefined))
|
||||
.order('desc')
|
||||
}
|
||||
|
||||
export async function listActiveReservedSlugsForSlug(
|
||||
ctx: QueryCtx | MutationCtx,
|
||||
slug: string,
|
||||
limit = DEFAULT_ACTIVE_LIMIT,
|
||||
) {
|
||||
return reservedSlugQuery(ctx, slug).take(limit)
|
||||
}
|
||||
|
||||
export async function getLatestActiveReservedSlug(ctx: QueryCtx | MutationCtx, slug: string) {
|
||||
return (await reservedSlugQuery(ctx, slug).take(1))[0] ?? null
|
||||
}
|
||||
|
||||
export async function releaseDuplicateActiveReservations(
|
||||
ctx: MutationCtx,
|
||||
active: ReservedSlug[],
|
||||
keepId: Id<'reservedSlugs'> | null | undefined,
|
||||
releasedAt: number,
|
||||
) {
|
||||
for (const stale of active) {
|
||||
if (keepId && stale._id === keepId) continue
|
||||
await ctx.db.patch(stale._id, { releasedAt })
|
||||
}
|
||||
}
|
||||
|
||||
export async function reserveSlugForHardDeleteFinalize(
|
||||
ctx: MutationCtx,
|
||||
params: {
|
||||
slug: string
|
||||
originalOwnerUserId: Id<'users'>
|
||||
deletedAt: number
|
||||
expiresAt: number
|
||||
},
|
||||
) {
|
||||
const active = await listActiveReservedSlugsForSlug(ctx, params.slug)
|
||||
const latest = active[0] ?? null
|
||||
|
||||
if (latest) {
|
||||
// Only extend reservation if it matches the owner being deleted.
|
||||
// If it points elsewhere, it likely came from a reclaim flow; do not overwrite.
|
||||
if (latest.originalOwnerUserId === params.originalOwnerUserId) {
|
||||
await ctx.db.patch(latest._id, {
|
||||
deletedAt: params.deletedAt,
|
||||
expiresAt: params.expiresAt,
|
||||
releasedAt: undefined,
|
||||
})
|
||||
}
|
||||
await releaseDuplicateActiveReservations(ctx, active, latest._id, params.deletedAt)
|
||||
return
|
||||
}
|
||||
|
||||
const inserted = await ctx.db.insert('reservedSlugs', {
|
||||
slug: params.slug,
|
||||
originalOwnerUserId: params.originalOwnerUserId,
|
||||
deletedAt: params.deletedAt,
|
||||
expiresAt: params.expiresAt,
|
||||
})
|
||||
await releaseDuplicateActiveReservations(ctx, active, inserted, params.deletedAt)
|
||||
}
|
||||
|
||||
export async function upsertReservedSlugForRightfulOwner(
|
||||
ctx: MutationCtx,
|
||||
params: {
|
||||
slug: string
|
||||
rightfulOwnerUserId: Id<'users'>
|
||||
deletedAt: number
|
||||
expiresAt: number
|
||||
reason?: string
|
||||
},
|
||||
) {
|
||||
const active = await listActiveReservedSlugsForSlug(ctx, params.slug)
|
||||
const latest = active[0] ?? null
|
||||
|
||||
let keepId: Id<'reservedSlugs'>
|
||||
if (latest) {
|
||||
keepId = latest._id
|
||||
await ctx.db.patch(latest._id, {
|
||||
originalOwnerUserId: params.rightfulOwnerUserId,
|
||||
deletedAt: params.deletedAt,
|
||||
expiresAt: params.expiresAt,
|
||||
reason: params.reason ?? latest.reason,
|
||||
releasedAt: undefined,
|
||||
})
|
||||
} else {
|
||||
keepId = await ctx.db.insert('reservedSlugs', {
|
||||
slug: params.slug,
|
||||
originalOwnerUserId: params.rightfulOwnerUserId,
|
||||
deletedAt: params.deletedAt,
|
||||
expiresAt: params.expiresAt,
|
||||
reason: params.reason,
|
||||
})
|
||||
}
|
||||
|
||||
await releaseDuplicateActiveReservations(ctx, active, keepId, params.deletedAt)
|
||||
}
|
||||
|
||||
export async function enforceReservedSlugCooldownForNewSkill(
|
||||
ctx: MutationCtx,
|
||||
params: { slug: string; userId: Id<'users'>; now: number },
|
||||
) {
|
||||
const active = await listActiveReservedSlugsForSlug(ctx, params.slug)
|
||||
const latest = active[0] ?? null
|
||||
if (!latest) return
|
||||
|
||||
if (latest.expiresAt > params.now && latest.originalOwnerUserId !== params.userId) {
|
||||
throw new Error(
|
||||
`Slug "${params.slug}" is reserved for its previous owner until ${new Date(latest.expiresAt).toISOString()}. ` +
|
||||
'Please choose a different slug.',
|
||||
)
|
||||
}
|
||||
|
||||
await ctx.db.patch(latest._id, { releasedAt: params.now })
|
||||
await releaseDuplicateActiveReservations(ctx, active, latest._id, params.now)
|
||||
}
|
||||
|
||||
+39
-17
@@ -19,6 +19,7 @@ import { generateSkillSummary } from './skillSummary'
|
||||
import {
|
||||
buildEmbeddingText,
|
||||
getFrontmatterMetadata,
|
||||
getFrontmatterValue,
|
||||
hashSkillFiles,
|
||||
isTextFile,
|
||||
parseClawdisMetadata,
|
||||
@@ -63,10 +64,19 @@ export type PublishVersionArgs = {
|
||||
}>
|
||||
}
|
||||
|
||||
export type PublishOptions = {
|
||||
bypassGitHubAccountAge?: boolean
|
||||
bypassNewSkillRateLimit?: boolean
|
||||
bypassQualityGate?: boolean
|
||||
skipBackup?: boolean
|
||||
skipWebhook?: boolean
|
||||
}
|
||||
|
||||
export async function publishVersionForUser(
|
||||
ctx: ActionCtx,
|
||||
userId: Id<'users'>,
|
||||
args: PublishVersionArgs,
|
||||
options: PublishOptions = {},
|
||||
): Promise<PublishResult> {
|
||||
const version = args.version.trim()
|
||||
const slug = args.slug.trim().toLowerCase()
|
||||
@@ -79,7 +89,9 @@ export async function publishVersionForUser(
|
||||
throw new ConvexError('Version must be valid semver')
|
||||
}
|
||||
|
||||
await requireGitHubAccountAge(ctx, userId)
|
||||
if (!options.bypassGitHubAccountAge) {
|
||||
await requireGitHubAccountAge(ctx, userId)
|
||||
}
|
||||
const existingSkill = (await ctx.runQuery(internal.skills.getSkillBySlugInternal, {
|
||||
slug,
|
||||
})) as Doc<'skills'> | null
|
||||
@@ -122,13 +134,18 @@ export async function publishVersionForUser(
|
||||
const ownerCreatedAt = owner?.createdAt ?? owner?._creationTime ?? Date.now()
|
||||
const now = Date.now()
|
||||
const frontmatterMetadata = getFrontmatterMetadata(frontmatter)
|
||||
const summaryFromFrontmatter =
|
||||
// Check for description in metadata.description (nested) or description (direct frontmatter field)
|
||||
const metadataDescription =
|
||||
frontmatterMetadata &&
|
||||
typeof frontmatterMetadata === 'object' &&
|
||||
!Array.isArray(frontmatterMetadata) &&
|
||||
typeof (frontmatterMetadata as Record<string, unknown>).description === 'string'
|
||||
? ((frontmatterMetadata as Record<string, unknown>).description as string)
|
||||
: undefined
|
||||
const directDescription = getFrontmatterValue(frontmatter, 'description')
|
||||
// Prioritize the new description from frontmatter over the existing skill summary
|
||||
// This ensures updates to the description are reflected on subsequent publishes (#301)
|
||||
const summaryFromFrontmatter = metadataDescription ?? directDescription
|
||||
const summary = await generateSkillSummary({
|
||||
slug,
|
||||
displayName,
|
||||
@@ -137,7 +154,7 @@ export async function publishVersionForUser(
|
||||
})
|
||||
|
||||
let qualityAssessment: QualityAssessment | null = null
|
||||
if (isNewSkill) {
|
||||
if (isNewSkill && !options.bypassQualityGate) {
|
||||
const ownerActivity = (await ctx.runQuery(internal.skills.getOwnerSkillActivityInternal, {
|
||||
ownerUserId: userId,
|
||||
limit: QUALITY_ACTIVITY_LIMIT,
|
||||
@@ -240,6 +257,7 @@ export async function publishVersionForUser(
|
||||
version: args.forkOf.version?.trim() || undefined,
|
||||
}
|
||||
: undefined,
|
||||
bypassNewSkillRateLimit: options.bypassNewSkillRateLimit || undefined,
|
||||
files: safeFiles.map((file) => ({
|
||||
...file,
|
||||
path: file.path,
|
||||
@@ -273,24 +291,28 @@ export async function publishVersionForUser(
|
||||
|
||||
const ownerHandle = owner?.handle ?? owner?.displayName ?? owner?.name ?? 'unknown'
|
||||
|
||||
void ctx.scheduler
|
||||
.runAfter(0, internal.githubBackupsNode.backupSkillForPublishInternal, {
|
||||
if (!options.skipBackup) {
|
||||
void ctx.scheduler
|
||||
.runAfter(0, internal.githubBackupsNode.backupSkillForPublishInternal, {
|
||||
slug,
|
||||
version,
|
||||
displayName,
|
||||
ownerHandle,
|
||||
files: safeFiles,
|
||||
publishedAt: Date.now(),
|
||||
})
|
||||
.catch((error) => {
|
||||
console.error('GitHub backup scheduling failed', error)
|
||||
})
|
||||
}
|
||||
|
||||
if (!options.skipWebhook) {
|
||||
void schedulePublishWebhook(ctx, {
|
||||
slug,
|
||||
version,
|
||||
displayName,
|
||||
ownerHandle,
|
||||
files: safeFiles,
|
||||
publishedAt: Date.now(),
|
||||
})
|
||||
.catch((error) => {
|
||||
console.error('GitHub backup scheduling failed', error)
|
||||
})
|
||||
|
||||
void schedulePublishWebhook(ctx, {
|
||||
slug,
|
||||
version,
|
||||
displayName,
|
||||
})
|
||||
}
|
||||
|
||||
return publishResult
|
||||
}
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { getFrontmatterValue, parseFrontmatter } from './skills'
|
||||
import { extractResponseText } from './openaiResponse'
|
||||
|
||||
const SKILL_SUMMARY_MODEL = process.env.OPENAI_SKILL_SUMMARY_MODEL ?? 'gpt-4.1-mini'
|
||||
const MAX_README_CHARS = 8_000
|
||||
@@ -61,27 +62,6 @@ function deriveIdentityFallback(args: { slug: string; displayName: string }) {
|
||||
return normalizeSummary(`Automation skill for ${base}.`)
|
||||
}
|
||||
|
||||
function extractResponseText(payload: unknown) {
|
||||
if (!payload || typeof payload !== 'object') return null
|
||||
const output = (payload as { output?: unknown }).output
|
||||
if (!Array.isArray(output)) return null
|
||||
const chunks: string[] = []
|
||||
for (const item of output) {
|
||||
if (!item || typeof item !== 'object') continue
|
||||
if ((item as { type?: unknown }).type !== 'message') continue
|
||||
const content = (item as { content?: unknown }).content
|
||||
if (!Array.isArray(content)) continue
|
||||
for (const part of content) {
|
||||
if (!part || typeof part !== 'object') continue
|
||||
if ((part as { type?: unknown }).type !== 'output_text') continue
|
||||
const text = (part as { text?: unknown }).text
|
||||
if (typeof text === 'string' && text.trim()) chunks.push(text)
|
||||
}
|
||||
}
|
||||
const joined = chunks.join('\n').trim()
|
||||
return joined || null
|
||||
}
|
||||
|
||||
export async function generateSkillSummary(args: {
|
||||
slug: string
|
||||
displayName: string
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { internal } from '../_generated/api'
|
||||
import type { Doc } from '../_generated/dataModel'
|
||||
import type { ActionCtx } from '../_generated/server'
|
||||
import { extractResponseText } from './openaiResponse'
|
||||
|
||||
const CHANGELOG_MODEL = process.env.OPENAI_CHANGELOG_MODEL ?? 'gpt-4.1'
|
||||
const MAX_README_CHARS = 8_000
|
||||
@@ -59,27 +60,6 @@ function pickPaths(values: string[]) {
|
||||
return values.slice(0, MAX_PATHS_IN_PROMPT)
|
||||
}
|
||||
|
||||
function extractResponseText(payload: unknown) {
|
||||
if (!payload || typeof payload !== 'object') return null
|
||||
const output = (payload as { output?: unknown }).output
|
||||
if (!Array.isArray(output)) return null
|
||||
const chunks: string[] = []
|
||||
for (const item of output) {
|
||||
if (!item || typeof item !== 'object') continue
|
||||
if ((item as { type?: unknown }).type !== 'message') continue
|
||||
const content = (item as { content?: unknown }).content
|
||||
if (!Array.isArray(content)) continue
|
||||
for (const part of content) {
|
||||
if (!part || typeof part !== 'object') continue
|
||||
if ((part as { type?: unknown }).type !== 'output_text') continue
|
||||
const text = (part as { text?: unknown }).text
|
||||
if (typeof text === 'string' && text.trim()) chunks.push(text)
|
||||
}
|
||||
}
|
||||
const joined = chunks.join('\n').trim()
|
||||
return joined || null
|
||||
}
|
||||
|
||||
async function generateWithOpenAI(args: {
|
||||
slug: string
|
||||
version: string
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { buildUserSearchResults } from './userSearch'
|
||||
|
||||
function makeUser(overrides: Record<string, unknown> = {}) {
|
||||
return {
|
||||
_id: 'users:1',
|
||||
_creationTime: 1,
|
||||
handle: 'alice',
|
||||
name: 'alice-gh',
|
||||
displayName: 'Alice',
|
||||
email: 'alice@example.com',
|
||||
...overrides,
|
||||
} as never
|
||||
}
|
||||
|
||||
describe('buildUserSearchResults', () => {
|
||||
it('returns all users when query is empty', () => {
|
||||
const users = [makeUser({ _id: 'users:1' }), makeUser({ _id: 'users:2', handle: 'bob' })]
|
||||
const result = buildUserSearchResults(users)
|
||||
expect(result.total).toBe(2)
|
||||
expect(result.items).toHaveLength(2)
|
||||
})
|
||||
|
||||
it('matches compact handle/search variants', () => {
|
||||
const users = [makeUser({ handle: 'alice-dev' }), makeUser({ _id: 'users:2', handle: 'bob' })]
|
||||
const result = buildUserSearchResults(users, 'alicedev')
|
||||
expect(result.total).toBe(1)
|
||||
expect(result.items[0]?.handle).toBe('alice-dev')
|
||||
})
|
||||
|
||||
it('does not throw on malformed legacy field types', () => {
|
||||
const users = [
|
||||
makeUser({
|
||||
_id: 'users:legacy',
|
||||
handle: 42,
|
||||
name: { bad: true },
|
||||
displayName: null,
|
||||
email: ['legacy@example.com'],
|
||||
}),
|
||||
makeUser({ _id: 'users:2', handle: 'carol' }),
|
||||
]
|
||||
|
||||
expect(() => buildUserSearchResults(users, 'car')).not.toThrow()
|
||||
const result = buildUserSearchResults(users, 'car')
|
||||
expect(result.total).toBe(1)
|
||||
expect(result.items[0]?._id).toBe('users:2')
|
||||
})
|
||||
|
||||
it('ranks exact id match above fuzzy matches', () => {
|
||||
const users = [
|
||||
makeUser({ _id: 'users:target', handle: 'target-user', _creationTime: 1 }),
|
||||
makeUser({ _id: 'users:2', handle: 'users:target', _creationTime: 10 }),
|
||||
]
|
||||
|
||||
const result = buildUserSearchResults(users, 'users:target')
|
||||
expect(result.total).toBe(2)
|
||||
expect(result.items[0]?._id).toBe('users:target')
|
||||
})
|
||||
|
||||
it('uses creation time as tie-break when scores are equal', () => {
|
||||
const users = [
|
||||
makeUser({ _id: 'users:older', handle: 'alpha', _creationTime: 1 }),
|
||||
makeUser({ _id: 'users:newer', handle: 'alpha-two', _creationTime: 50 }),
|
||||
]
|
||||
|
||||
const result = buildUserSearchResults(users, 'pha')
|
||||
expect(result.total).toBe(2)
|
||||
expect(result.items[0]?._id).toBe('users:newer')
|
||||
expect(result.items[1]?._id).toBe('users:older')
|
||||
})
|
||||
})
|
||||
@@ -14,11 +14,15 @@ function normalizeCompact(value: string) {
|
||||
return value.toLowerCase().replace(/[^a-z0-9]/g, '')
|
||||
}
|
||||
|
||||
function toSearchText(value: unknown) {
|
||||
return typeof value === 'string' ? value.toLowerCase() : ''
|
||||
}
|
||||
|
||||
function scoreUser(user: Doc<'users'>, query: string, compactQuery: string) {
|
||||
const handle = user.handle?.toLowerCase() ?? ''
|
||||
const name = user.name?.toLowerCase() ?? ''
|
||||
const displayName = user.displayName?.toLowerCase() ?? ''
|
||||
const email = user.email?.toLowerCase() ?? ''
|
||||
const handle = toSearchText(user.handle)
|
||||
const name = toSearchText(user.name)
|
||||
const displayName = toSearchText(user.displayName)
|
||||
const email = toSearchText(user.email)
|
||||
const id = String(user._id).toLowerCase()
|
||||
|
||||
let score = 0
|
||||
|
||||
+1
-21
@@ -11,32 +11,12 @@ import {
|
||||
parseLlmEvalResponse,
|
||||
SECURITY_EVALUATOR_SYSTEM_PROMPT,
|
||||
} from './lib/securityPrompt'
|
||||
import { extractResponseText } from './lib/openaiResponse'
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function extractResponseText(payload: unknown): string | null {
|
||||
if (!payload || typeof payload !== 'object') return null
|
||||
const output = (payload as { output?: unknown }).output
|
||||
if (!Array.isArray(output)) return null
|
||||
const chunks: string[] = []
|
||||
for (const item of output) {
|
||||
if (!item || typeof item !== 'object') continue
|
||||
if ((item as { type?: unknown }).type !== 'message') continue
|
||||
const content = (item as { content?: unknown }).content
|
||||
if (!Array.isArray(content)) continue
|
||||
for (const part of content) {
|
||||
if (!part || typeof part !== 'object') continue
|
||||
if ((part as { type?: unknown }).type !== 'output_text') continue
|
||||
const text = (part as { text?: unknown }).text
|
||||
if (typeof text === 'string' && text.trim()) chunks.push(text)
|
||||
}
|
||||
}
|
||||
const joined = chunks.join('\n').trim()
|
||||
return joined || null
|
||||
}
|
||||
|
||||
function verdictToStatus(verdict: string): string {
|
||||
switch (verdict) {
|
||||
case 'benign':
|
||||
|
||||
@@ -37,6 +37,7 @@ const {
|
||||
backfillSkillSummariesInternalHandler,
|
||||
cleanupEmptySkillsInternalHandler,
|
||||
nominateEmptySkillSpammersInternalHandler,
|
||||
upsertSkillBadgeRecordInternal,
|
||||
} = await import('./maintenance')
|
||||
const { internal } = await import('./_generated/api')
|
||||
const { generateSkillSummary } = await import('./lib/skillSummary')
|
||||
@@ -196,6 +197,81 @@ describe('maintenance backfill', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('maintenance badge denormalization', () => {
|
||||
it('upserts table badge and keeps skill.badges in sync', async () => {
|
||||
const unique = vi.fn().mockResolvedValue(null)
|
||||
const query = vi.fn().mockReturnValue({
|
||||
withIndex: () => ({ unique }),
|
||||
})
|
||||
const insert = vi.fn().mockResolvedValue('skillBadges:1')
|
||||
const get = vi.fn().mockResolvedValue({ _id: 'skills:1', badges: undefined })
|
||||
const patch = vi.fn().mockResolvedValue(undefined)
|
||||
|
||||
const ctx = {
|
||||
db: {
|
||||
query,
|
||||
insert,
|
||||
get,
|
||||
patch,
|
||||
},
|
||||
} as never
|
||||
|
||||
const result = await (upsertSkillBadgeRecordInternal as { _handler: Function })._handler(ctx, {
|
||||
skillId: 'skills:1',
|
||||
kind: 'highlighted',
|
||||
byUserId: 'users:1',
|
||||
at: 123,
|
||||
})
|
||||
|
||||
expect(result).toEqual({ inserted: true })
|
||||
expect(insert).toHaveBeenCalledWith('skillBadges', {
|
||||
skillId: 'skills:1',
|
||||
kind: 'highlighted',
|
||||
byUserId: 'users:1',
|
||||
at: 123,
|
||||
})
|
||||
expect(patch).toHaveBeenCalledWith('skills:1', {
|
||||
badges: {
|
||||
highlighted: { byUserId: 'users:1', at: 123 },
|
||||
},
|
||||
})
|
||||
})
|
||||
|
||||
it('resyncs denormalized badge even when table record already exists', async () => {
|
||||
const unique = vi.fn().mockResolvedValue({ _id: 'skillBadges:existing' })
|
||||
const query = vi.fn().mockReturnValue({
|
||||
withIndex: () => ({ unique }),
|
||||
})
|
||||
const insert = vi.fn()
|
||||
const get = vi.fn().mockResolvedValue({ _id: 'skills:1', badges: {} })
|
||||
const patch = vi.fn().mockResolvedValue(undefined)
|
||||
|
||||
const ctx = {
|
||||
db: {
|
||||
query,
|
||||
insert,
|
||||
get,
|
||||
patch,
|
||||
},
|
||||
} as never
|
||||
|
||||
const result = await (upsertSkillBadgeRecordInternal as { _handler: Function })._handler(ctx, {
|
||||
skillId: 'skills:1',
|
||||
kind: 'official',
|
||||
byUserId: 'users:2',
|
||||
at: 456,
|
||||
})
|
||||
|
||||
expect(result).toEqual({ inserted: false })
|
||||
expect(insert).not.toHaveBeenCalled()
|
||||
expect(patch).toHaveBeenCalledWith('skills:1', {
|
||||
badges: {
|
||||
official: { byUserId: 'users:2', at: 456 },
|
||||
},
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
describe('maintenance fingerprint backfill', () => {
|
||||
it('backfills fingerprint field and inserts index entry', async () => {
|
||||
const { hashSkillFiles } = await import('./lib/skills')
|
||||
|
||||
+119
-1
@@ -642,17 +642,32 @@ export const upsertSkillBadgeRecordInternal = internalMutation({
|
||||
at: v.number(),
|
||||
},
|
||||
handler: async (ctx, args) => {
|
||||
const syncDenormalizedBadge = async () => {
|
||||
const skill = await ctx.db.get(args.skillId)
|
||||
if (!skill) return
|
||||
await ctx.db.patch(args.skillId, {
|
||||
badges: {
|
||||
...(skill.badges as Record<string, unknown> | undefined),
|
||||
[args.kind]: { byUserId: args.byUserId, at: args.at },
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
const existing = await ctx.db
|
||||
.query('skillBadges')
|
||||
.withIndex('by_skill_kind', (q) => q.eq('skillId', args.skillId).eq('kind', args.kind))
|
||||
.unique()
|
||||
if (existing) return { inserted: false as const }
|
||||
if (existing) {
|
||||
await syncDenormalizedBadge()
|
||||
return { inserted: false as const }
|
||||
}
|
||||
await ctx.db.insert('skillBadges', {
|
||||
skillId: args.skillId,
|
||||
kind: args.kind,
|
||||
byUserId: args.byUserId,
|
||||
at: args.at,
|
||||
})
|
||||
await syncDenormalizedBadge()
|
||||
return { inserted: true as const }
|
||||
},
|
||||
})
|
||||
@@ -1411,6 +1426,109 @@ export const nominateEmptySkillSpammers: ReturnType<typeof action> = action({
|
||||
},
|
||||
})
|
||||
|
||||
// Backfill embeddingSkillMap from existing skillEmbeddings.
|
||||
// Run once after deploying the schema change:
|
||||
// npx convex run maintenance:backfillEmbeddingSkillMapInternal --prod
|
||||
export const backfillEmbeddingSkillMapInternal = internalMutation({
|
||||
args: {
|
||||
cursor: v.optional(v.string()),
|
||||
batchSize: v.optional(v.number()),
|
||||
},
|
||||
handler: async (ctx, args) => {
|
||||
const batchSize = clampInt(args.batchSize ?? 200, 10, 500)
|
||||
const { page, continueCursor, isDone } = await ctx.db
|
||||
.query('skillEmbeddings')
|
||||
.paginate({ cursor: args.cursor ?? null, numItems: batchSize })
|
||||
|
||||
let inserted = 0
|
||||
for (const embedding of page) {
|
||||
const existing = await ctx.db
|
||||
.query('embeddingSkillMap')
|
||||
.withIndex('by_embedding', (q) => q.eq('embeddingId', embedding._id))
|
||||
.unique()
|
||||
if (!existing) {
|
||||
await ctx.db.insert('embeddingSkillMap', {
|
||||
embeddingId: embedding._id,
|
||||
skillId: embedding.skillId,
|
||||
})
|
||||
inserted++
|
||||
}
|
||||
}
|
||||
|
||||
if (!isDone) {
|
||||
await ctx.scheduler.runAfter(0, internal.maintenance.backfillEmbeddingSkillMapInternal, {
|
||||
cursor: continueCursor,
|
||||
batchSize: args.batchSize,
|
||||
})
|
||||
}
|
||||
|
||||
return { inserted, isDone, scanned: page.length }
|
||||
},
|
||||
})
|
||||
|
||||
// Sync skillBadges table → denormalized skill.badges field.
|
||||
// Run after deploying the badge-read removal to ensure all skills
|
||||
// have up-to-date badges on the skill doc itself.
|
||||
export const backfillDenormalizedBadgesInternal = internalMutation({
|
||||
args: {
|
||||
cursor: v.optional(v.string()),
|
||||
batchSize: v.optional(v.number()),
|
||||
},
|
||||
handler: async (ctx, args) => {
|
||||
const batchSize = clampInt(args.batchSize ?? 100, 10, 200)
|
||||
const { page, continueCursor, isDone } = await ctx.db
|
||||
.query('skills')
|
||||
.paginate({ cursor: args.cursor ?? null, numItems: batchSize })
|
||||
|
||||
let patched = 0
|
||||
for (const skill of page) {
|
||||
const records = await ctx.db
|
||||
.query('skillBadges')
|
||||
.withIndex('by_skill', (q) => q.eq('skillId', skill._id))
|
||||
.take(10)
|
||||
|
||||
// Build canonical badge map from the table
|
||||
const canonical: Record<string, { byUserId: Id<'users'>; at: number }> = {}
|
||||
for (const r of records) {
|
||||
canonical[r.kind] = { byUserId: r.byUserId, at: r.at }
|
||||
}
|
||||
|
||||
// Compare with existing denormalized badges (keys + values)
|
||||
const existing = (skill.badges ?? {}) as Record<
|
||||
string,
|
||||
{ byUserId?: Id<'users'>; at?: number } | undefined
|
||||
>
|
||||
const canonicalKeys = Object.keys(canonical)
|
||||
const existingKeys = Object.keys(existing).filter((k) => existing[k] !== undefined)
|
||||
const needsPatch =
|
||||
canonicalKeys.length !== existingKeys.length ||
|
||||
canonicalKeys.some((k) => {
|
||||
const current = existing[k]
|
||||
const next = canonical[k]
|
||||
return (
|
||||
!current ||
|
||||
current.byUserId !== next.byUserId ||
|
||||
current.at !== next.at
|
||||
)
|
||||
})
|
||||
|
||||
if (needsPatch) {
|
||||
await ctx.db.patch(skill._id, { badges: canonical })
|
||||
patched++
|
||||
}
|
||||
}
|
||||
|
||||
if (!isDone) {
|
||||
await ctx.scheduler.runAfter(0, internal.maintenance.backfillDenormalizedBadgesInternal, {
|
||||
cursor: continueCursor,
|
||||
batchSize: args.batchSize,
|
||||
})
|
||||
}
|
||||
|
||||
return { patched, isDone, scanned: page.length }
|
||||
},
|
||||
})
|
||||
|
||||
function clampInt(value: number, min: number, max: number) {
|
||||
const rounded = Math.trunc(value)
|
||||
if (!Number.isFinite(rounded)) return min
|
||||
|
||||
@@ -17,6 +17,8 @@ const users = defineTable({
|
||||
role: v.optional(v.union(v.literal('admin'), v.literal('moderator'), v.literal('user'))),
|
||||
githubCreatedAt: v.optional(v.number()),
|
||||
githubFetchedAt: v.optional(v.number()),
|
||||
githubProfileSyncedAt: v.optional(v.number()),
|
||||
trustedPublisher: v.optional(v.boolean()),
|
||||
deactivatedAt: v.optional(v.number()),
|
||||
purgedAt: v.optional(v.number()),
|
||||
deletedAt: v.optional(v.number()),
|
||||
@@ -308,6 +310,14 @@ const skillEmbeddings = defineTable({
|
||||
filterFields: ['visibility'],
|
||||
})
|
||||
|
||||
// Lightweight lookup: embeddingId → skillId (~100 bytes per doc).
|
||||
// Avoids reading full skillEmbeddings docs (~12KB each with vector)
|
||||
// during search hydration.
|
||||
const embeddingSkillMap = defineTable({
|
||||
embeddingId: v.id('skillEmbeddings'),
|
||||
skillId: v.id('skills'),
|
||||
}).index('by_embedding', ['embeddingId'])
|
||||
|
||||
const skillDailyStats = defineTable({
|
||||
skillId: v.id('skills'),
|
||||
day: v.number(),
|
||||
@@ -340,6 +350,12 @@ const skillStatBackfillState = defineTable({
|
||||
updatedAt: v.number(),
|
||||
}).index('by_key', ['key'])
|
||||
|
||||
const globalStats = defineTable({
|
||||
key: v.string(),
|
||||
activeSkillsCount: v.number(),
|
||||
updatedAt: v.number(),
|
||||
}).index('by_key', ['key'])
|
||||
|
||||
const skillStatEvents = defineTable({
|
||||
skillId: v.id('skills'),
|
||||
kind: v.union(
|
||||
@@ -500,6 +516,19 @@ const downloadDedupes = defineTable({
|
||||
.index('by_skill_identity_hour', ['skillId', 'identityHash', 'hourStart'])
|
||||
.index('by_hour', ['hourStart'])
|
||||
|
||||
const reservedSlugs = defineTable({
|
||||
slug: v.string(),
|
||||
originalOwnerUserId: v.id('users'),
|
||||
deletedAt: v.number(),
|
||||
expiresAt: v.number(),
|
||||
reason: v.optional(v.string()),
|
||||
releasedAt: v.optional(v.number()),
|
||||
})
|
||||
.index('by_slug', ['slug'])
|
||||
.index('by_slug_active_deletedAt', ['slug', 'releasedAt', 'deletedAt'])
|
||||
.index('by_owner', ['originalOwnerUserId'])
|
||||
.index('by_expiry', ['expiresAt'])
|
||||
|
||||
const githubBackupSyncState = defineTable({
|
||||
key: v.string(),
|
||||
cursor: v.optional(v.string()),
|
||||
@@ -555,10 +584,12 @@ export default defineSchema({
|
||||
skillBadges,
|
||||
soulVersionFingerprints,
|
||||
skillEmbeddings,
|
||||
embeddingSkillMap,
|
||||
soulEmbeddings,
|
||||
skillDailyStats,
|
||||
skillLeaderboards,
|
||||
skillStatBackfillState,
|
||||
globalStats,
|
||||
skillStatEvents,
|
||||
skillStatUpdateCursors,
|
||||
comments,
|
||||
@@ -571,6 +602,7 @@ export default defineSchema({
|
||||
apiTokens,
|
||||
rateLimits,
|
||||
downloadDedupes,
|
||||
reservedSlugs,
|
||||
githubBackupSyncState,
|
||||
userSyncRoots,
|
||||
userSkillInstalls,
|
||||
|
||||
+21
-27
@@ -4,9 +4,8 @@ import { describe, expect, it, vi } from 'vitest'
|
||||
import { tokenize } from './lib/searchText'
|
||||
import { __test, hydrateResults, lexicalFallbackSkills, searchSkills } from './search'
|
||||
|
||||
const { generateEmbeddingMock, getSkillBadgeMapsMock } = vi.hoisted(() => ({
|
||||
const { generateEmbeddingMock } = vi.hoisted(() => ({
|
||||
generateEmbeddingMock: vi.fn(),
|
||||
getSkillBadgeMapsMock: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('./lib/embeddings', () => ({
|
||||
@@ -14,7 +13,6 @@ vi.mock('./lib/embeddings', () => ({
|
||||
}))
|
||||
|
||||
vi.mock('./lib/badges', () => ({
|
||||
getSkillBadgeMaps: getSkillBadgeMapsMock,
|
||||
isSkillHighlighted: (skill: { badges?: Record<string, unknown> }) =>
|
||||
Boolean(skill.badges?.highlighted),
|
||||
}))
|
||||
@@ -45,13 +43,13 @@ describe('search helpers', () => {
|
||||
skill: makePublicSkill({ id: 'skills:orf', slug: 'orf', displayName: 'ORF' }),
|
||||
version: null,
|
||||
ownerHandle: 'steipete',
|
||||
owner: null,
|
||||
},
|
||||
]
|
||||
const runQuery = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce([])
|
||||
.mockResolvedValueOnce([])
|
||||
.mockResolvedValueOnce(fallback)
|
||||
.mockResolvedValueOnce([]) // hydrateResults
|
||||
.mockResolvedValueOnce(fallback) // lexicalFallbackSkills
|
||||
|
||||
const result = await searchSkillsHandler(
|
||||
{
|
||||
@@ -70,18 +68,15 @@ describe('search helpers', () => {
|
||||
})
|
||||
|
||||
it('applies highlightedOnly filtering in lexical fallback', async () => {
|
||||
const highlighted = makeSkillDoc({
|
||||
id: 'skills:hl',
|
||||
slug: 'orf-highlighted',
|
||||
displayName: 'ORF Highlighted',
|
||||
})
|
||||
const highlighted = {
|
||||
...makeSkillDoc({
|
||||
id: 'skills:hl',
|
||||
slug: 'orf-highlighted',
|
||||
displayName: 'ORF Highlighted',
|
||||
}),
|
||||
badges: { highlighted: { byUserId: 'users:mod', at: 1 } },
|
||||
}
|
||||
const plain = makeSkillDoc({ id: 'skills:plain', slug: 'orf-plain', displayName: 'ORF Plain' })
|
||||
getSkillBadgeMapsMock.mockResolvedValueOnce(
|
||||
new Map([
|
||||
['skills:hl', { highlighted: { byUserId: 'users:mod', at: 1 } }],
|
||||
['skills:plain', {}],
|
||||
]),
|
||||
)
|
||||
|
||||
const result = await lexicalFallbackSkillsHandler(
|
||||
makeLexicalCtx({
|
||||
@@ -103,12 +98,6 @@ describe('search helpers', () => {
|
||||
moderationFlags: ['flagged.suspicious'],
|
||||
})
|
||||
const clean = makeSkillDoc({ id: 'skills:clean', slug: 'orf-clean', displayName: 'ORF Clean' })
|
||||
getSkillBadgeMapsMock.mockResolvedValueOnce(
|
||||
new Map([
|
||||
['skills:suspicious', {}],
|
||||
['skills:clean', {}],
|
||||
]),
|
||||
)
|
||||
|
||||
const result = await lexicalFallbackSkillsHandler(
|
||||
makeLexicalCtx({
|
||||
@@ -124,7 +113,6 @@ describe('search helpers', () => {
|
||||
|
||||
it('includes exact slug match from by_slug even when recent scan is empty', async () => {
|
||||
const exactSlugSkill = makeSkillDoc({ id: 'skills:orf', slug: 'orf', displayName: 'ORF' })
|
||||
getSkillBadgeMapsMock.mockResolvedValueOnce(new Map([['skills:orf', {}]]))
|
||||
const ctx = makeLexicalCtx({
|
||||
exactSlugSkill,
|
||||
recentSkills: [],
|
||||
@@ -154,6 +142,7 @@ describe('search helpers', () => {
|
||||
}),
|
||||
version: null,
|
||||
ownerHandle: 'one',
|
||||
owner: null,
|
||||
},
|
||||
{
|
||||
embeddingId: 'skillEmbeddings:b',
|
||||
@@ -165,6 +154,7 @@ describe('search helpers', () => {
|
||||
}),
|
||||
version: null,
|
||||
ownerHandle: 'two',
|
||||
owner: null,
|
||||
},
|
||||
]
|
||||
const fallbackEntries = [
|
||||
@@ -177,6 +167,7 @@ describe('search helpers', () => {
|
||||
}),
|
||||
version: null,
|
||||
ownerHandle: 'one',
|
||||
owner: null,
|
||||
},
|
||||
{
|
||||
skill: makePublicSkill({
|
||||
@@ -187,14 +178,14 @@ describe('search helpers', () => {
|
||||
}),
|
||||
version: null,
|
||||
ownerHandle: 'three',
|
||||
owner: null,
|
||||
},
|
||||
]
|
||||
|
||||
const runQuery = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce(vectorEntries)
|
||||
.mockResolvedValueOnce([])
|
||||
.mockResolvedValueOnce(fallbackEntries)
|
||||
.mockResolvedValueOnce(vectorEntries) // hydrateResults
|
||||
.mockResolvedValueOnce(fallbackEntries) // lexicalFallbackSkills
|
||||
|
||||
const result = await searchSkillsHandler(
|
||||
{
|
||||
@@ -232,6 +223,9 @@ describe('search helpers', () => {
|
||||
if (id === 'skillVersions:1') return { _id: 'skillVersions:1', version: '1.0.0' }
|
||||
return null
|
||||
}),
|
||||
query: vi.fn(() => ({
|
||||
withIndex: () => ({ unique: vi.fn().mockResolvedValue(null) }),
|
||||
})),
|
||||
},
|
||||
},
|
||||
{ embeddingIds: ['skillEmbeddings:1'], nonSuspiciousOnly: true },
|
||||
|
||||
+57
-81
@@ -1,18 +1,36 @@
|
||||
import { v } from 'convex/values'
|
||||
import { internal } from './_generated/api'
|
||||
import type { Doc, Id } from './_generated/dataModel'
|
||||
import type { QueryCtx } from './_generated/server'
|
||||
import { action, internalQuery } from './_generated/server'
|
||||
import { getSkillBadgeMaps, isSkillHighlighted, type SkillBadgeMap } from './lib/badges'
|
||||
import { isSkillHighlighted } from './lib/badges'
|
||||
import { generateEmbedding } from './lib/embeddings'
|
||||
import { toPublicSkill, toPublicSoul } from './lib/public'
|
||||
import { toPublicSkill, toPublicSoul, toPublicUser } from './lib/public'
|
||||
import { matchesExactTokens, tokenize } from './lib/searchText'
|
||||
import { isSkillSuspicious } from './lib/skillSafety'
|
||||
|
||||
type OwnerInfo = { handle: string | null; owner: ReturnType<typeof toPublicUser> | null }
|
||||
|
||||
function makeOwnerInfoGetter(ctx: Pick<QueryCtx, 'db'>) {
|
||||
const ownerCache = new Map<Id<'users'>, Promise<OwnerInfo>>()
|
||||
return (ownerUserId: Id<'users'>) => {
|
||||
const cached = ownerCache.get(ownerUserId)
|
||||
if (cached) return cached
|
||||
const ownerPromise = ctx.db.get(ownerUserId).then((ownerDoc) => ({
|
||||
handle: ownerDoc?.handle ?? (ownerDoc?._id ? String(ownerDoc._id) : null),
|
||||
owner: toPublicUser(ownerDoc),
|
||||
}))
|
||||
ownerCache.set(ownerUserId, ownerPromise)
|
||||
return ownerPromise
|
||||
}
|
||||
}
|
||||
|
||||
type SkillSearchEntry = {
|
||||
embeddingId?: Id<'skillEmbeddings'>
|
||||
skill: NonNullable<ReturnType<typeof toPublicSkill>>
|
||||
version: Doc<'skillVersions'> | null
|
||||
ownerHandle: string | null
|
||||
owner: ReturnType<typeof toPublicUser> | null
|
||||
}
|
||||
|
||||
type SearchResult = SkillSearchEntry & { score: number }
|
||||
@@ -22,7 +40,7 @@ const SLUG_PREFIX_BOOST = 0.8
|
||||
const NAME_EXACT_BOOST = 1.1
|
||||
const NAME_PREFIX_BOOST = 0.6
|
||||
const POPULARITY_WEIGHT = 0.08
|
||||
const FALLBACK_SCAN_LIMIT = 1200
|
||||
const FALLBACK_SCAN_LIMIT = 500
|
||||
|
||||
function getNextCandidateLimit(current: number, max: number) {
|
||||
const next = Math.min(current * 2, max)
|
||||
@@ -131,21 +149,11 @@ export const searchSkills: ReturnType<typeof action> = action({
|
||||
results.map((result) => [result._id, result._score]),
|
||||
)
|
||||
|
||||
const badgeMapEntries = (await ctx.runQuery(internal.search.getSkillBadgeMapsInternal, {
|
||||
skillIds: hydrated.map((entry) => entry.skill._id),
|
||||
})) as Array<[Id<'skills'>, SkillBadgeMap]>
|
||||
const badgeMapBySkillId = new Map(badgeMapEntries)
|
||||
const hydratedWithBadges = hydrated.map((entry) => ({
|
||||
...entry,
|
||||
skill: {
|
||||
...entry.skill,
|
||||
badges: badgeMapBySkillId.get(entry.skill._id) ?? {},
|
||||
},
|
||||
}))
|
||||
|
||||
// Skills already have badges from their docs (via toPublicSkill).
|
||||
// No need for a separate badge table lookup.
|
||||
const filtered = args.highlightedOnly
|
||||
? hydratedWithBadges.filter((entry) => isSkillHighlighted(entry.skill))
|
||||
: hydratedWithBadges
|
||||
? hydrated.filter((entry) => isSkillHighlighted(entry.skill))
|
||||
: hydrated
|
||||
|
||||
exactMatches = filtered.filter((entry) =>
|
||||
matchesExactTokens(queryTokens, [
|
||||
@@ -197,46 +205,39 @@ export const searchSkills: ReturnType<typeof action> = action({
|
||||
},
|
||||
})
|
||||
|
||||
export const getBadgeMapsForSkills = internalQuery({
|
||||
args: { skillIds: v.array(v.id('skills')) },
|
||||
handler: async (ctx, args): Promise<Array<[Id<'skills'>, SkillBadgeMap]>> => {
|
||||
const badgeMap = await getSkillBadgeMaps(ctx, args.skillIds)
|
||||
return Array.from(badgeMap.entries())
|
||||
},
|
||||
})
|
||||
|
||||
export const hydrateResults = internalQuery({
|
||||
args: {
|
||||
embeddingIds: v.array(v.id('skillEmbeddings')),
|
||||
nonSuspiciousOnly: v.optional(v.boolean()),
|
||||
},
|
||||
handler: async (ctx, args): Promise<SkillSearchEntry[]> => {
|
||||
const ownerHandleCache = new Map<Id<'users'>, Promise<string | null>>()
|
||||
|
||||
const getOwnerHandle = (ownerUserId: Id<'users'>) => {
|
||||
const cached = ownerHandleCache.get(ownerUserId)
|
||||
if (cached) return cached
|
||||
const handlePromise = ctx.db
|
||||
.get(ownerUserId)
|
||||
.then((owner) => owner?.handle ?? owner?._id ?? null)
|
||||
ownerHandleCache.set(ownerUserId, handlePromise)
|
||||
return handlePromise
|
||||
}
|
||||
const getOwnerInfo = makeOwnerInfoGetter(ctx)
|
||||
|
||||
const entries: Array<SkillSearchEntry | null> = await Promise.all(
|
||||
args.embeddingIds.map(async (embeddingId) => {
|
||||
const embedding = await ctx.db.get(embeddingId)
|
||||
if (!embedding) return null
|
||||
const skill = await ctx.db.get(embedding.skillId)
|
||||
// Use lightweight lookup table (~100 bytes) instead of full embedding doc (~12KB).
|
||||
const lookup = await ctx.db
|
||||
.query('embeddingSkillMap')
|
||||
.withIndex('by_embedding', (q) => q.eq('embeddingId', embeddingId))
|
||||
.unique()
|
||||
// Fallback to full embedding doc for rows not yet backfilled.
|
||||
const skillId = lookup
|
||||
? lookup.skillId
|
||||
: await ctx.db.get(embeddingId).then((e) => e?.skillId)
|
||||
if (!skillId) return null
|
||||
const skill = await ctx.db.get(skillId)
|
||||
if (!skill || skill.softDeletedAt) return null
|
||||
if (args.nonSuspiciousOnly && isSkillSuspicious(skill)) return null
|
||||
const [version, ownerHandle] = await Promise.all([
|
||||
ctx.db.get(embedding.versionId),
|
||||
getOwnerHandle(skill.ownerUserId),
|
||||
])
|
||||
const ownerInfo = await getOwnerInfo(skill.ownerUserId)
|
||||
const publicSkill = toPublicSkill(skill)
|
||||
if (!publicSkill) return null
|
||||
return { embeddingId, skill: publicSkill, version, ownerHandle }
|
||||
return {
|
||||
embeddingId,
|
||||
skill: publicSkill,
|
||||
version: null as Doc<'skillVersions'> | null,
|
||||
ownerHandle: ownerInfo.handle,
|
||||
owner: ownerInfo.owner,
|
||||
}
|
||||
}),
|
||||
)
|
||||
|
||||
@@ -291,46 +292,29 @@ export const lexicalFallbackSkills = internalQuery({
|
||||
)
|
||||
if (matched.length === 0) return []
|
||||
|
||||
const ownerHandleCache = new Map<Id<'users'>, Promise<string | null>>()
|
||||
const getOwnerHandle = (ownerUserId: Id<'users'>) => {
|
||||
const cached = ownerHandleCache.get(ownerUserId)
|
||||
if (cached) return cached
|
||||
const handlePromise = ctx.db
|
||||
.get(ownerUserId)
|
||||
.then((owner) => owner?.handle ?? owner?._id ?? null)
|
||||
ownerHandleCache.set(ownerUserId, handlePromise)
|
||||
return handlePromise
|
||||
}
|
||||
const getOwnerInfo = makeOwnerInfoGetter(ctx)
|
||||
|
||||
const entries = await Promise.all(
|
||||
matched.map(async (skill) => {
|
||||
const [version, ownerHandle] = await Promise.all([
|
||||
skill.latestVersionId ? ctx.db.get(skill.latestVersionId) : Promise.resolve(null),
|
||||
getOwnerHandle(skill.ownerUserId),
|
||||
])
|
||||
const ownerInfo = await getOwnerInfo(skill.ownerUserId)
|
||||
const publicSkill = toPublicSkill(skill)
|
||||
if (!publicSkill) return null
|
||||
return { skill: publicSkill, version, ownerHandle }
|
||||
return {
|
||||
skill: publicSkill,
|
||||
version: null as Doc<'skillVersions'> | null,
|
||||
ownerHandle: ownerInfo.handle,
|
||||
owner: ownerInfo.owner,
|
||||
}
|
||||
}),
|
||||
)
|
||||
const validEntries = entries.filter((entry): entry is SkillSearchEntry => entry !== null)
|
||||
if (validEntries.length === 0) return []
|
||||
|
||||
const badgeMap = await getSkillBadgeMaps(
|
||||
ctx,
|
||||
validEntries.map((entry) => entry.skill._id),
|
||||
)
|
||||
const withBadges = validEntries.map((entry) => ({
|
||||
...entry,
|
||||
skill: {
|
||||
...entry.skill,
|
||||
badges: badgeMap.get(entry.skill._id) ?? {},
|
||||
},
|
||||
}))
|
||||
|
||||
// Skills already have badges from their docs (via toPublicSkill).
|
||||
// No need for a separate badge table lookup.
|
||||
const filtered = args.highlightedOnly
|
||||
? withBadges.filter((entry) => isSkillHighlighted(entry.skill))
|
||||
: withBadges
|
||||
? validEntries.filter((entry) => isSkillHighlighted(entry.skill))
|
||||
: validEntries
|
||||
return filtered.slice(0, limit)
|
||||
},
|
||||
})
|
||||
@@ -430,14 +414,6 @@ export const hydrateSoulResults = internalQuery({
|
||||
},
|
||||
})
|
||||
|
||||
export const getSkillBadgeMapsInternal = internalQuery({
|
||||
args: { skillIds: v.array(v.id('skills')) },
|
||||
handler: async (ctx, args) => {
|
||||
const badgeMap = await getSkillBadgeMaps(ctx, args.skillIds)
|
||||
return Array.from(badgeMap.entries())
|
||||
},
|
||||
})
|
||||
|
||||
export const __test = {
|
||||
getNextCandidateLimit,
|
||||
matchesAllTokens,
|
||||
|
||||
+23
-60
@@ -3,14 +3,18 @@
|
||||
*
|
||||
* Instead of updating skill stats synchronously in the hot path (which can cause
|
||||
* contention when multiple users download/star/install the same skill), we insert
|
||||
* lightweight event records and process them in batches via a cron job.
|
||||
* lightweight event records and process them in batches via cron jobs.
|
||||
*
|
||||
* Flow:
|
||||
* 1. User action (download, star, install) → insertStatEvent() writes to skillStatEvents table
|
||||
* 2. Cron job runs every 5 minutes → processSkillStatEventsInternal() processes batches
|
||||
* 3. Events are aggregated per-skill to minimize database operations
|
||||
* 4. Stats are applied to skill documents and daily stats tables
|
||||
* 5. Events are marked as processed (kept forever for auditing)
|
||||
* Two processing paths run at different frequencies to balance freshness vs bandwidth:
|
||||
*
|
||||
* 1. **Daily stats (15-minute cron)** — `processSkillStatEventsAction`
|
||||
* Writes to skillDailyStats for trending/leaderboards. Uses a cursor in
|
||||
* skillStatUpdateCursors. Does NOT touch skill documents.
|
||||
*
|
||||
* 2. **Skill doc sync (6-hour cron)** — `processSkillStatEventsInternal`
|
||||
* Patches skill documents with accumulated stat deltas. Uses processedAt
|
||||
* field to track progress. Runs infrequently because patching skill docs
|
||||
* invalidates reactive queries for all subscribers (thundering herd).
|
||||
*/
|
||||
|
||||
import { v } from 'convex/values'
|
||||
@@ -175,7 +179,7 @@ function aggregateEvents(events: Doc<'skillStatEvents'>[]): AggregatedDeltas {
|
||||
/**
|
||||
* Process a batch of unprocessed stat events.
|
||||
*
|
||||
* Called by cron every 5 minutes. Processes up to batchSize events (default 100).
|
||||
* Called by the 6-hour cron to sync stats to skill docs. Processes up to batchSize events (default 500).
|
||||
* If the batch is full, schedules an immediate follow-up run to drain the queue.
|
||||
*
|
||||
* Processing steps:
|
||||
@@ -198,7 +202,7 @@ function aggregateEvents(events: Doc<'skillStatEvents'>[]): AggregatedDeltas {
|
||||
export const processSkillStatEventsInternal = internalMutation({
|
||||
args: { batchSize: v.optional(v.number()) },
|
||||
handler: async (ctx, args) => {
|
||||
const batchSize = args.batchSize ?? 100
|
||||
const batchSize = args.batchSize ?? 500
|
||||
const now = Date.now()
|
||||
|
||||
// Level 1: Fetch a batch of unprocessed events
|
||||
@@ -252,25 +256,13 @@ export const processSkillStatEventsInternal = internalMutation({
|
||||
installsAllTime: deltas.installsAllTime,
|
||||
installsCurrent: deltas.installsCurrent,
|
||||
})
|
||||
await ctx.db.patch(skill._id, {
|
||||
...patch,
|
||||
updatedAt: now,
|
||||
})
|
||||
// Don't update `updatedAt` — stat changes shouldn't move the
|
||||
// skill's position in the by_active_updated index.
|
||||
await ctx.db.patch(skill._id, patch)
|
||||
}
|
||||
|
||||
// Update daily stats for trending/leaderboards
|
||||
// We use the ORIGINAL event timestamp (occurredAt) so that:
|
||||
// - A download at Mon 11:55 PM counts toward Monday's stats
|
||||
// - Even if the cron processes it on Tuesday
|
||||
//
|
||||
// Level 4: bumpDailySkillStats does its own coalescing - multiple
|
||||
// events on the same day will update the same daily record
|
||||
for (const occurredAt of deltas.downloadEvents) {
|
||||
await bumpDailySkillStats(ctx, { skillId, now: occurredAt, downloads: 1 })
|
||||
}
|
||||
for (const occurredAt of deltas.installNewEvents) {
|
||||
await bumpDailySkillStats(ctx, { skillId, now: occurredAt, installs: 1 })
|
||||
}
|
||||
// NOTE: Daily stats (skillDailyStats) are written by the 15-minute
|
||||
// action cron (processSkillStatEventsAction), not here.
|
||||
|
||||
// Mark all events for this skill as processed
|
||||
for (const event of skillEvents) {
|
||||
@@ -352,11 +344,11 @@ const skillDeltaValidator = v.object({
|
||||
})
|
||||
|
||||
/**
|
||||
* Apply aggregated stats to skills and update the cursor.
|
||||
* Write aggregated daily stats and advance the cursor.
|
||||
* This is a single atomic mutation that:
|
||||
* 1. Updates all affected skills with their aggregated deltas
|
||||
* 2. Updates daily stats for trending
|
||||
* 3. Advances the cursor to the new position
|
||||
* 1. Updates daily stats for trending/leaderboards (skillDailyStats)
|
||||
* 2. Advances the cursor to the new position
|
||||
* NOTE: Does NOT patch skill documents — that's handled by processSkillStatEventsInternal.
|
||||
*/
|
||||
export const applyAggregatedStatsAndUpdateCursor = internalMutation({
|
||||
args: {
|
||||
@@ -366,37 +358,8 @@ export const applyAggregatedStatsAndUpdateCursor = internalMutation({
|
||||
handler: async (ctx, args) => {
|
||||
const now = Date.now()
|
||||
|
||||
// Process each skill's aggregated deltas
|
||||
// Update daily stats for trending/leaderboards
|
||||
for (const delta of args.skillDeltas) {
|
||||
const skill = await ctx.db.get(delta.skillId)
|
||||
|
||||
// Skill was deleted - skip
|
||||
if (!skill) {
|
||||
continue
|
||||
}
|
||||
|
||||
// Apply aggregated deltas to skill stats
|
||||
if (
|
||||
delta.downloads !== 0 ||
|
||||
delta.stars !== 0 ||
|
||||
delta.comments !== 0 ||
|
||||
delta.installsAllTime !== 0 ||
|
||||
delta.installsCurrent !== 0
|
||||
) {
|
||||
const patch = applySkillStatDeltas(skill, {
|
||||
downloads: delta.downloads,
|
||||
stars: delta.stars,
|
||||
comments: delta.comments,
|
||||
installsAllTime: delta.installsAllTime,
|
||||
installsCurrent: delta.installsCurrent,
|
||||
})
|
||||
await ctx.db.patch(skill._id, {
|
||||
...patch,
|
||||
updatedAt: now,
|
||||
})
|
||||
}
|
||||
|
||||
// Update daily stats for trending/leaderboards
|
||||
for (const occurredAt of delta.downloadEvents) {
|
||||
await bumpDailySkillStats(ctx, { skillId: delta.skillId, now: occurredAt, downloads: 1 })
|
||||
}
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { countPublicSkills } from './skills'
|
||||
|
||||
type WrappedHandler<TArgs, TResult> = {
|
||||
_handler: (ctx: unknown, args: TArgs) => Promise<TResult>
|
||||
}
|
||||
|
||||
const countPublicSkillsHandler = (
|
||||
countPublicSkills as unknown as WrappedHandler<Record<string, never>, number>
|
||||
)._handler
|
||||
|
||||
function makeSkillsQuery(skills: Array<{ softDeletedAt?: number; moderationStatus?: string | null }>) {
|
||||
return {
|
||||
withIndex: (name: string) => {
|
||||
if (name !== 'by_active_updated') throw new Error(`unexpected skills index ${name}`)
|
||||
return {
|
||||
collect: async () => skills,
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
describe('skills.countPublicSkills', () => {
|
||||
it('returns precomputed global stats count when available', async () => {
|
||||
const ctx = {
|
||||
db: {
|
||||
query: vi.fn((table: string) => {
|
||||
if (table === 'globalStats') {
|
||||
return {
|
||||
withIndex: () => ({
|
||||
unique: async () => ({ _id: 'globalStats:1', activeSkillsCount: 123 }),
|
||||
}),
|
||||
}
|
||||
}
|
||||
if (table === 'skills') {
|
||||
return makeSkillsQuery([])
|
||||
}
|
||||
throw new Error(`unexpected table ${table}`)
|
||||
}),
|
||||
},
|
||||
}
|
||||
|
||||
const result = await countPublicSkillsHandler(ctx, {})
|
||||
expect(result).toBe(123)
|
||||
})
|
||||
|
||||
it('falls back to live count when global stats row is missing', async () => {
|
||||
const ctx = {
|
||||
db: {
|
||||
query: vi.fn((table: string) => {
|
||||
if (table === 'globalStats') {
|
||||
return {
|
||||
withIndex: () => ({
|
||||
unique: async () => null,
|
||||
}),
|
||||
}
|
||||
}
|
||||
if (table === 'skills') {
|
||||
return makeSkillsQuery([
|
||||
{ softDeletedAt: undefined, moderationStatus: 'active' },
|
||||
{ softDeletedAt: undefined, moderationStatus: 'hidden' },
|
||||
{ softDeletedAt: undefined, moderationStatus: 'active' },
|
||||
])
|
||||
}
|
||||
throw new Error(`unexpected table ${table}`)
|
||||
}),
|
||||
},
|
||||
}
|
||||
|
||||
const result = await countPublicSkillsHandler(ctx, {})
|
||||
expect(result).toBe(2)
|
||||
})
|
||||
|
||||
it('falls back to live count when globalStats table is unavailable', async () => {
|
||||
const ctx = {
|
||||
db: {
|
||||
query: vi.fn((table: string) => {
|
||||
if (table === 'globalStats') {
|
||||
throw new Error('unexpected table globalStats')
|
||||
}
|
||||
if (table === 'skills') {
|
||||
return makeSkillsQuery([
|
||||
{ softDeletedAt: undefined, moderationStatus: 'active' },
|
||||
{ softDeletedAt: undefined, moderationStatus: 'active' },
|
||||
])
|
||||
}
|
||||
throw new Error(`unexpected table ${table}`)
|
||||
}),
|
||||
},
|
||||
}
|
||||
|
||||
const result = await countPublicSkillsHandler(ctx, {})
|
||||
expect(result).toBe(2)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,317 @@
|
||||
/* @vitest-environment node */
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const { getSkillBadgeMapMock, getSkillBadgeMapsMock, isSkillHighlightedMock } = vi.hoisted(() => ({
|
||||
getSkillBadgeMapMock: vi.fn(),
|
||||
getSkillBadgeMapsMock: vi.fn(),
|
||||
isSkillHighlightedMock: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('./lib/badges', () => ({
|
||||
getSkillBadgeMap: getSkillBadgeMapMock,
|
||||
getSkillBadgeMaps: getSkillBadgeMapsMock,
|
||||
isSkillHighlighted: isSkillHighlightedMock,
|
||||
}))
|
||||
|
||||
import { listPublicPageV2 } from './skills'
|
||||
|
||||
type ListArgs = {
|
||||
paginationOpts: { cursor: string | null; numItems: number; id?: number }
|
||||
sort?: 'newest' | 'updated' | 'downloads' | 'installs' | 'stars' | 'name'
|
||||
dir?: 'asc' | 'desc'
|
||||
highlightedOnly?: boolean
|
||||
nonSuspiciousOnly?: boolean
|
||||
}
|
||||
|
||||
type ListResult = {
|
||||
page: Array<{ skill: { slug: string } }>
|
||||
continueCursor: string | null
|
||||
isDone: boolean
|
||||
}
|
||||
|
||||
type WrappedHandler<TArgs, TResult> = {
|
||||
_handler: (ctx: unknown, args: TArgs) => Promise<TResult>
|
||||
}
|
||||
|
||||
const listPublicPageV2Handler = (listPublicPageV2 as unknown as WrappedHandler<ListArgs, ListResult>)
|
||||
._handler
|
||||
|
||||
describe('skills.listPublicPageV2', () => {
|
||||
beforeEach(() => {
|
||||
getSkillBadgeMapMock.mockReset()
|
||||
getSkillBadgeMapsMock.mockReset()
|
||||
getSkillBadgeMapsMock.mockResolvedValue(new Map())
|
||||
isSkillHighlightedMock.mockReset()
|
||||
isSkillHighlightedMock.mockImplementation((skill: { slug?: string }) =>
|
||||
Boolean(skill.slug?.startsWith('hl-')),
|
||||
)
|
||||
})
|
||||
|
||||
it('applies highlightedOnly and nonSuspiciousOnly together', async () => {
|
||||
const highlightedClean = makeSkill('skills:hl-clean', 'hl-clean', 'users:1', 'skillVersions:1')
|
||||
const plainClean = makeSkill('skills:plain', 'plain', 'users:2', 'skillVersions:2')
|
||||
const highlightedSuspicious = makeSkill(
|
||||
'skills:hl-suspicious',
|
||||
'hl-suspicious',
|
||||
'users:3',
|
||||
'skillVersions:3',
|
||||
['flagged.suspicious'],
|
||||
)
|
||||
|
||||
const paginateMock = vi.fn().mockResolvedValue({
|
||||
page: [highlightedClean, plainClean, highlightedSuspicious],
|
||||
continueCursor: 'next-cursor',
|
||||
isDone: false,
|
||||
pageStatus: null,
|
||||
splitCursor: null,
|
||||
})
|
||||
const orderMock = vi.fn(() => ({ paginate: paginateMock }))
|
||||
const eqMock = vi.fn(() => ({}))
|
||||
const withIndexMock = vi.fn((_index: string, builder: (q: { eq: typeof eqMock }) => unknown) => {
|
||||
builder({ eq: eqMock })
|
||||
return { order: orderMock }
|
||||
})
|
||||
const getMock = vi.fn(async (id: string) => {
|
||||
if (id.startsWith('users:')) return makeUser(id)
|
||||
if (id.startsWith('skillVersions:')) return makeVersion(id)
|
||||
return null
|
||||
})
|
||||
const ctx = {
|
||||
db: {
|
||||
query: vi.fn((table: string) => {
|
||||
if (table !== 'skills') throw new Error(`unexpected table ${table}`)
|
||||
return { withIndex: withIndexMock }
|
||||
}),
|
||||
get: getMock,
|
||||
},
|
||||
}
|
||||
|
||||
const result = await listPublicPageV2Handler(ctx, {
|
||||
paginationOpts: { cursor: null, numItems: 25 },
|
||||
sort: 'downloads',
|
||||
dir: 'desc',
|
||||
highlightedOnly: true,
|
||||
nonSuspiciousOnly: true,
|
||||
})
|
||||
|
||||
expect(result.page).toHaveLength(1)
|
||||
expect(result.page[0]?.skill.slug).toBe('hl-clean')
|
||||
expect(result.continueCursor).toBe('next-cursor')
|
||||
expect(result.isDone).toBe(false)
|
||||
expect(withIndexMock).toHaveBeenCalledWith('by_active_stats_downloads', expect.any(Function))
|
||||
expect(orderMock).toHaveBeenCalledWith('desc')
|
||||
expect(paginateMock).toHaveBeenCalledWith({ cursor: null, numItems: 25 })
|
||||
expect(eqMock).toHaveBeenCalledWith('softDeletedAt', undefined)
|
||||
})
|
||||
|
||||
it('preserves pagination cursor when filtering removes the whole page', async () => {
|
||||
const plain = makeSkill('skills:plain', 'plain', 'users:1', 'skillVersions:1')
|
||||
const paginateMock = vi.fn().mockResolvedValue({
|
||||
page: [plain],
|
||||
continueCursor: 'next-cursor',
|
||||
isDone: false,
|
||||
pageStatus: null,
|
||||
splitCursor: null,
|
||||
})
|
||||
const ctx = {
|
||||
db: {
|
||||
query: vi.fn(() => ({
|
||||
withIndex: vi.fn(() => ({
|
||||
order: vi.fn(() => ({ paginate: paginateMock })),
|
||||
})),
|
||||
})),
|
||||
get: vi.fn(),
|
||||
},
|
||||
}
|
||||
|
||||
const result = await listPublicPageV2Handler(ctx, {
|
||||
paginationOpts: { cursor: null, numItems: 25 },
|
||||
sort: 'downloads',
|
||||
dir: 'desc',
|
||||
highlightedOnly: true,
|
||||
nonSuspiciousOnly: false,
|
||||
})
|
||||
|
||||
expect(result.page).toEqual([])
|
||||
expect(result.continueCursor).toBe('next-cursor')
|
||||
expect(result.isDone).toBe(false)
|
||||
})
|
||||
|
||||
it('restarts pagination from first page when cursor is stale', async () => {
|
||||
const plain = makeSkill('skills:plain', 'plain', 'users:1', 'skillVersions:1')
|
||||
const paginateMock = vi
|
||||
.fn()
|
||||
.mockRejectedValueOnce(new Error('Failed to parse cursor'))
|
||||
.mockResolvedValueOnce({
|
||||
page: [plain],
|
||||
continueCursor: 'next-cursor',
|
||||
isDone: false,
|
||||
pageStatus: null,
|
||||
splitCursor: null,
|
||||
})
|
||||
const ctx = {
|
||||
db: {
|
||||
query: vi.fn(() => ({
|
||||
withIndex: vi.fn(() => ({
|
||||
order: vi.fn(() => ({ paginate: paginateMock })),
|
||||
})),
|
||||
})),
|
||||
get: vi.fn(async (id: string) => {
|
||||
if (id.startsWith('users:')) return makeUser(id)
|
||||
if (id.startsWith('skillVersions:')) return makeVersion(id)
|
||||
return null
|
||||
}),
|
||||
},
|
||||
}
|
||||
|
||||
const result = await listPublicPageV2Handler(ctx, {
|
||||
paginationOpts: { cursor: 'stale-cursor', numItems: 25, id: 123456 },
|
||||
sort: 'downloads',
|
||||
dir: 'desc',
|
||||
highlightedOnly: false,
|
||||
nonSuspiciousOnly: false,
|
||||
})
|
||||
|
||||
expect(result.page).toHaveLength(1)
|
||||
expect(result.page[0]?.skill.slug).toBe('plain')
|
||||
expect(result.continueCursor).toBe('next-cursor')
|
||||
expect(result.isDone).toBe(false)
|
||||
expect(paginateMock).toHaveBeenNthCalledWith(1, { cursor: 'stale-cursor', numItems: 25 })
|
||||
expect(paginateMock).toHaveBeenNthCalledWith(2, { cursor: null, numItems: 25 })
|
||||
expect(paginateMock).not.toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
id: expect.any(Number),
|
||||
}),
|
||||
)
|
||||
})
|
||||
|
||||
it('drops pagination id from client options on first-page queries', async () => {
|
||||
const plain = makeSkill('skills:plain', 'plain', 'users:1', 'skillVersions:1')
|
||||
const paginateMock = vi.fn().mockResolvedValue({
|
||||
page: [plain],
|
||||
continueCursor: 'next-cursor',
|
||||
isDone: false,
|
||||
pageStatus: null,
|
||||
splitCursor: null,
|
||||
})
|
||||
const ctx = {
|
||||
db: {
|
||||
query: vi.fn(() => ({
|
||||
withIndex: vi.fn(() => ({
|
||||
order: vi.fn(() => ({ paginate: paginateMock })),
|
||||
})),
|
||||
})),
|
||||
get: vi.fn(async (id: string) => {
|
||||
if (id.startsWith('users:')) return makeUser(id)
|
||||
if (id.startsWith('skillVersions:')) return makeVersion(id)
|
||||
return null
|
||||
}),
|
||||
},
|
||||
}
|
||||
|
||||
const result = await listPublicPageV2Handler(ctx, {
|
||||
paginationOpts: { cursor: null, numItems: 25, id: 999_999_999 },
|
||||
sort: 'downloads',
|
||||
dir: 'desc',
|
||||
highlightedOnly: false,
|
||||
nonSuspiciousOnly: false,
|
||||
})
|
||||
|
||||
expect(result.page).toHaveLength(1)
|
||||
expect(paginateMock).toHaveBeenCalledTimes(1)
|
||||
expect(paginateMock).toHaveBeenCalledWith({ cursor: null, numItems: 25 })
|
||||
expect(paginateMock).not.toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
id: expect.any(Number),
|
||||
}),
|
||||
)
|
||||
})
|
||||
|
||||
it('does not swallow non-cursor paginate errors', async () => {
|
||||
const paginateMock = vi.fn().mockRejectedValue(new Error('database unavailable'))
|
||||
const ctx = {
|
||||
db: {
|
||||
query: vi.fn(() => ({
|
||||
withIndex: vi.fn(() => ({
|
||||
order: vi.fn(() => ({ paginate: paginateMock })),
|
||||
})),
|
||||
})),
|
||||
get: vi.fn(),
|
||||
},
|
||||
}
|
||||
|
||||
await expect(
|
||||
listPublicPageV2Handler(ctx, {
|
||||
paginationOpts: { cursor: 'stale-cursor', numItems: 25, id: 999_999_999 },
|
||||
sort: 'downloads',
|
||||
dir: 'desc',
|
||||
highlightedOnly: false,
|
||||
nonSuspiciousOnly: false,
|
||||
}),
|
||||
).rejects.toThrow('database unavailable')
|
||||
|
||||
expect(paginateMock).toHaveBeenCalledTimes(1)
|
||||
expect(paginateMock).toHaveBeenCalledWith({ cursor: 'stale-cursor', numItems: 25 })
|
||||
})
|
||||
})
|
||||
|
||||
function makeSkill(
|
||||
id: string,
|
||||
slug: string,
|
||||
ownerUserId: string,
|
||||
latestVersionId: string,
|
||||
moderationFlags?: string[],
|
||||
) {
|
||||
return {
|
||||
_id: id,
|
||||
_creationTime: 1,
|
||||
slug,
|
||||
displayName: slug,
|
||||
summary: `${slug} summary`,
|
||||
ownerUserId,
|
||||
canonicalSkillId: undefined,
|
||||
forkOf: undefined,
|
||||
latestVersionId,
|
||||
tags: {},
|
||||
badges: {},
|
||||
stats: {
|
||||
downloads: 0,
|
||||
stars: 0,
|
||||
installsCurrent: 0,
|
||||
installsAllTime: 0,
|
||||
versions: 1,
|
||||
comments: 0,
|
||||
},
|
||||
createdAt: 1,
|
||||
updatedAt: 1,
|
||||
softDeletedAt: undefined,
|
||||
moderationStatus: 'active',
|
||||
moderationFlags,
|
||||
}
|
||||
}
|
||||
|
||||
function makeUser(id: string) {
|
||||
return {
|
||||
_id: id,
|
||||
_creationTime: 1,
|
||||
handle: 'owner',
|
||||
name: 'Owner',
|
||||
displayName: 'Owner',
|
||||
image: null,
|
||||
bio: null,
|
||||
deletedAt: undefined,
|
||||
deactivatedAt: undefined,
|
||||
}
|
||||
}
|
||||
|
||||
function makeVersion(id: string) {
|
||||
return {
|
||||
_id: id,
|
||||
_creationTime: 1,
|
||||
version: '1.0.0',
|
||||
createdAt: 1,
|
||||
changelog: '',
|
||||
changelogSource: 'user',
|
||||
parsed: {},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,202 @@
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { getPendingScanSkillsInternal } from './skills'
|
||||
|
||||
type PendingScanResult = Array<{
|
||||
skillId: string
|
||||
versionId: string | null
|
||||
sha256hash: string | null
|
||||
checkCount: number
|
||||
}>
|
||||
|
||||
type WrappedHandler<TArgs, TResult> = {
|
||||
_handler: (ctx: unknown, args: TArgs) => Promise<TResult>
|
||||
}
|
||||
|
||||
const getPendingScanSkillsHandler = (
|
||||
getPendingScanSkillsInternal as unknown as WrappedHandler<Record<string, unknown>, PendingScanResult>
|
||||
)._handler
|
||||
|
||||
describe('skills.getPendingScanSkillsInternal', () => {
|
||||
it('includes unresolved VT records from the oldest slice and skips finalized ones', async () => {
|
||||
const recentSkills = [
|
||||
makeSkill('skills:recent-clean', 'skillVersions:recent-clean', 'scanner.llm.clean'),
|
||||
makeSkill('skills:recent-malicious', 'skillVersions:recent-malicious', 'scanner.vt.pending'),
|
||||
]
|
||||
const oldestSkills = [
|
||||
makeSkill('skills:old-pending', 'skillVersions:old-pending', 'scanner.vt.pending'),
|
||||
makeSkill('skills:old-stale', 'skillVersions:old-stale', 'scanner.llm.clean'),
|
||||
makeSkill('skills:old-no-hash', 'skillVersions:old-no-hash', 'scanner.vt.pending'),
|
||||
]
|
||||
|
||||
const versions = new Map<string, unknown>([
|
||||
[
|
||||
'skillVersions:recent-clean',
|
||||
{ _id: 'skillVersions:recent-clean', sha256hash: 'a'.repeat(64), vtAnalysis: { status: 'clean' } },
|
||||
],
|
||||
[
|
||||
'skillVersions:recent-malicious',
|
||||
{
|
||||
_id: 'skillVersions:recent-malicious',
|
||||
sha256hash: 'b'.repeat(64),
|
||||
vtAnalysis: { status: 'malicious' },
|
||||
},
|
||||
],
|
||||
[
|
||||
'skillVersions:old-pending',
|
||||
{ _id: 'skillVersions:old-pending', sha256hash: 'c'.repeat(64), vtAnalysis: { status: 'pending' } },
|
||||
],
|
||||
[
|
||||
'skillVersions:old-stale',
|
||||
{ _id: 'skillVersions:old-stale', sha256hash: 'd'.repeat(64), vtAnalysis: { status: 'stale' } },
|
||||
],
|
||||
['skillVersions:old-no-hash', { _id: 'skillVersions:old-no-hash' }],
|
||||
])
|
||||
|
||||
const ctx = {
|
||||
db: {
|
||||
query: vi.fn((table: string) => {
|
||||
if (table !== 'skills') throw new Error(`unexpected table ${table}`)
|
||||
return {
|
||||
withIndex: (
|
||||
indexName: string,
|
||||
builder: (q: { eq: (field: string, value: unknown) => unknown }) => unknown,
|
||||
) => {
|
||||
builder({ eq: () => ({}) })
|
||||
if (indexName === 'by_active_updated') {
|
||||
return {
|
||||
order: () => ({
|
||||
take: async () => recentSkills,
|
||||
}),
|
||||
}
|
||||
}
|
||||
if (indexName === 'by_active_created') {
|
||||
return {
|
||||
order: () => ({
|
||||
take: async () => oldestSkills,
|
||||
}),
|
||||
}
|
||||
}
|
||||
throw new Error(`unexpected index ${indexName}`)
|
||||
},
|
||||
}
|
||||
}),
|
||||
get: vi.fn(async (id: string) => versions.get(id) ?? null),
|
||||
},
|
||||
}
|
||||
|
||||
const result = await getPendingScanSkillsHandler(ctx, {
|
||||
limit: 25,
|
||||
skipRecentMinutes: 0,
|
||||
})
|
||||
|
||||
const ids = new Set(result.map((entry) => entry.skillId))
|
||||
expect(ids.has('skills:old-pending')).toBe(true)
|
||||
expect(ids.has('skills:old-stale')).toBe(true)
|
||||
expect(ids.has('skills:recent-clean')).toBe(false)
|
||||
expect(ids.has('skills:recent-malicious')).toBe(false)
|
||||
expect(ids.has('skills:old-no-hash')).toBe(false)
|
||||
})
|
||||
|
||||
it('exhaustive mode ignores recent-check suppression for manual backfills', async () => {
|
||||
const now = Date.now()
|
||||
const allSkills = [
|
||||
makeSkill('skills:recently-checked', 'skillVersions:recently-checked', 'scanner.vt.pending', now),
|
||||
]
|
||||
const versions = new Map<string, unknown>([
|
||||
[
|
||||
'skillVersions:recently-checked',
|
||||
{ _id: 'skillVersions:recently-checked', sha256hash: 'e'.repeat(64) },
|
||||
],
|
||||
])
|
||||
|
||||
const withIndex = vi.fn(
|
||||
(
|
||||
indexName: string,
|
||||
builder: (q: { eq: (field: string, value: unknown) => unknown }) => unknown,
|
||||
) => {
|
||||
builder({ eq: () => ({}) })
|
||||
if (indexName !== 'by_active_updated') throw new Error(`unexpected index ${indexName}`)
|
||||
return {
|
||||
collect: async () => allSkills,
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
const ctx = {
|
||||
db: {
|
||||
query: vi.fn((table: string) => {
|
||||
if (table !== 'skills') throw new Error(`unexpected table ${table}`)
|
||||
return { withIndex }
|
||||
}),
|
||||
get: vi.fn(async (id: string) => versions.get(id) ?? null),
|
||||
},
|
||||
}
|
||||
|
||||
const result = await getPendingScanSkillsHandler(ctx, {
|
||||
limit: 25,
|
||||
skipRecentMinutes: 60,
|
||||
exhaustive: true,
|
||||
})
|
||||
|
||||
expect(result).toHaveLength(1)
|
||||
expect(result[0]?.skillId).toBe('skills:recently-checked')
|
||||
})
|
||||
|
||||
it('does not clamp exhaustive mode to 100 records', async () => {
|
||||
const allSkills = Array.from({ length: 150 }, (_, i) =>
|
||||
makeSkill(`skills:bulk-${i}`, `skillVersions:bulk-${i}`, 'scanner.vt.pending'),
|
||||
)
|
||||
const versions = new Map<string, unknown>(
|
||||
allSkills.map((skill) => {
|
||||
const versionId = skill.latestVersionId as string
|
||||
return [versionId, { _id: versionId, sha256hash: `${String(versionId).slice(-8)}${'f'.repeat(56)}` }]
|
||||
}),
|
||||
)
|
||||
|
||||
const withIndex = vi.fn(
|
||||
(
|
||||
indexName: string,
|
||||
builder: (q: { eq: (field: string, value: unknown) => unknown }) => unknown,
|
||||
) => {
|
||||
builder({ eq: () => ({}) })
|
||||
if (indexName !== 'by_active_updated') throw new Error(`unexpected index ${indexName}`)
|
||||
return {
|
||||
collect: async () => allSkills,
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
const ctx = {
|
||||
db: {
|
||||
query: vi.fn((table: string) => {
|
||||
if (table !== 'skills') throw new Error(`unexpected table ${table}`)
|
||||
return { withIndex }
|
||||
}),
|
||||
get: vi.fn(async (id: string) => versions.get(id) ?? null),
|
||||
},
|
||||
}
|
||||
|
||||
const result = await getPendingScanSkillsHandler(ctx, {
|
||||
limit: 10000,
|
||||
exhaustive: true,
|
||||
skipRecentMinutes: 0,
|
||||
})
|
||||
|
||||
expect(result).toHaveLength(150)
|
||||
})
|
||||
})
|
||||
|
||||
function makeSkill(
|
||||
id: string,
|
||||
versionId: string,
|
||||
moderationReason: string,
|
||||
scanLastCheckedAt?: number,
|
||||
) {
|
||||
return {
|
||||
_id: id,
|
||||
moderationStatus: 'active',
|
||||
moderationReason,
|
||||
latestVersionId: versionId,
|
||||
scanLastCheckedAt,
|
||||
}
|
||||
}
|
||||
@@ -22,6 +22,21 @@ const clearOwnerSuspiciousFlagsHandler = (
|
||||
clearOwnerSuspiciousFlagsInternal as unknown as WrappedHandler<Record<string, unknown>>
|
||||
)._handler
|
||||
|
||||
function buildGlobalStatsQuery(table: string) {
|
||||
if (table !== 'globalStats') return null
|
||||
return {
|
||||
withIndex: (name: string) => {
|
||||
if (name !== 'by_key') throw new Error(`unexpected globalStats index ${name}`)
|
||||
return {
|
||||
unique: async () => ({
|
||||
_id: 'globalStats:1',
|
||||
activeSkillsCount: 100,
|
||||
}),
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
function createPublishArgs(overrides?: Partial<Record<string, unknown>>) {
|
||||
return {
|
||||
userId: 'users:owner',
|
||||
@@ -67,6 +82,8 @@ describe('skills anti-spam guards', () => {
|
||||
deletedAt: undefined,
|
||||
})),
|
||||
query: vi.fn((table: string) => {
|
||||
const globalStatsQuery = buildGlobalStatsQuery(table)
|
||||
if (globalStatsQuery) return globalStatsQuery
|
||||
if (table === 'skills') {
|
||||
return {
|
||||
withIndex: (name: string) => {
|
||||
@@ -84,6 +101,16 @@ describe('skills anti-spam guards', () => {
|
||||
},
|
||||
}
|
||||
}
|
||||
if (table === 'reservedSlugs') {
|
||||
return {
|
||||
withIndex: (name: string) => {
|
||||
if (name === 'by_slug_active_deletedAt') {
|
||||
return { order: () => ({ take: async () => [] }) }
|
||||
}
|
||||
throw new Error(`unexpected index ${name}`)
|
||||
},
|
||||
}
|
||||
}
|
||||
throw new Error(`unexpected table ${table}`)
|
||||
}),
|
||||
}
|
||||
@@ -117,6 +144,8 @@ describe('skills anti-spam guards', () => {
|
||||
return null
|
||||
}),
|
||||
query: vi.fn((table: string) => {
|
||||
const globalStatsQuery = buildGlobalStatsQuery(table)
|
||||
if (globalStatsQuery) return globalStatsQuery
|
||||
if (table === 'skillVersions') {
|
||||
return {
|
||||
withIndex: () => ({
|
||||
@@ -187,6 +216,8 @@ describe('skills anti-spam guards', () => {
|
||||
return null
|
||||
}),
|
||||
query: vi.fn((table: string) => {
|
||||
const globalStatsQuery = buildGlobalStatsQuery(table)
|
||||
if (globalStatsQuery) return globalStatsQuery
|
||||
if (table === 'skillVersions') {
|
||||
return {
|
||||
withIndex: () => ({
|
||||
@@ -255,6 +286,8 @@ describe('skills anti-spam guards', () => {
|
||||
return null
|
||||
}),
|
||||
query: vi.fn((table: string) => {
|
||||
const globalStatsQuery = buildGlobalStatsQuery(table)
|
||||
if (globalStatsQuery) return globalStatsQuery
|
||||
if (table === 'skillVersions') {
|
||||
return {
|
||||
withIndex: () => ({
|
||||
@@ -314,6 +347,8 @@ describe('skills anti-spam guards', () => {
|
||||
return null
|
||||
}),
|
||||
query: vi.fn((table: string) => {
|
||||
const globalStatsQuery = buildGlobalStatsQuery(table)
|
||||
if (globalStatsQuery) return globalStatsQuery
|
||||
if (table === 'skills') {
|
||||
return {
|
||||
withIndex: (name: string) => {
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { reclaimSlugInternal } from './skills'
|
||||
|
||||
type WrappedHandler<TArgs> = {
|
||||
_handler: (ctx: unknown, args: TArgs) => Promise<unknown>
|
||||
}
|
||||
|
||||
const reclaimSlugInternalHandler = (
|
||||
reclaimSlugInternal as unknown as WrappedHandler<Record<string, unknown>>
|
||||
)._handler
|
||||
|
||||
describe('skills reclaim ownership transfer', () => {
|
||||
it('transfers ownership in-place when transferRootSlugOnly is true', async () => {
|
||||
const now = Date.now()
|
||||
const patch = vi.fn(async () => {})
|
||||
const insert = vi.fn(async () => {})
|
||||
const runAfter = vi.fn(async () => {})
|
||||
|
||||
const existingSkill = {
|
||||
_id: 'skills:1',
|
||||
slug: 'capability-evolver',
|
||||
ownerUserId: 'users:old',
|
||||
}
|
||||
const activeReservation = {
|
||||
_id: 'reservedSlugs:1',
|
||||
slug: 'capability-evolver',
|
||||
originalOwnerUserId: 'users:old',
|
||||
deletedAt: now - 1_000,
|
||||
expiresAt: now + 10_000,
|
||||
}
|
||||
|
||||
const db = {
|
||||
get: vi.fn(async (id: string) => {
|
||||
if (id === 'users:admin') return { _id: 'users:admin', role: 'admin' }
|
||||
if (id === 'users:new') return { _id: 'users:new', role: 'user' }
|
||||
return null
|
||||
}),
|
||||
query: vi.fn((table: string) => {
|
||||
if (table === 'skills') {
|
||||
return {
|
||||
withIndex: (name: string) => {
|
||||
if (name !== 'by_slug') throw new Error(`unexpected skills index ${name}`)
|
||||
return { unique: async () => existingSkill }
|
||||
},
|
||||
}
|
||||
}
|
||||
if (table === 'skillEmbeddings') {
|
||||
return {
|
||||
withIndex: (name: string) => {
|
||||
if (name !== 'by_skill') throw new Error(`unexpected embeddings index ${name}`)
|
||||
return {
|
||||
collect: async () => [{ _id: 'skillEmbeddings:1', skillId: 'skills:1', ownerId: 'users:old' }],
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
if (table === 'reservedSlugs') {
|
||||
return {
|
||||
withIndex: (name: string) => {
|
||||
if (name !== 'by_slug_active_deletedAt') {
|
||||
throw new Error(`unexpected reservedSlugs index ${name}`)
|
||||
}
|
||||
return {
|
||||
order: () => ({
|
||||
take: async () => [activeReservation],
|
||||
}),
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
throw new Error(`unexpected table ${table}`)
|
||||
}),
|
||||
patch,
|
||||
insert,
|
||||
}
|
||||
|
||||
const result = (await reclaimSlugInternalHandler(
|
||||
{ db, scheduler: { runAfter } } as never,
|
||||
{
|
||||
actorUserId: 'users:admin',
|
||||
slug: 'Capability-Evolver',
|
||||
rightfulOwnerUserId: 'users:new',
|
||||
transferRootSlugOnly: true,
|
||||
} as never,
|
||||
)) as { ok: boolean; action: string }
|
||||
|
||||
expect(result).toEqual({ ok: true, action: 'ownership_transferred' })
|
||||
expect(runAfter).not.toHaveBeenCalled()
|
||||
expect(patch).toHaveBeenCalledWith(
|
||||
'skills:1',
|
||||
expect.objectContaining({
|
||||
ownerUserId: 'users:new',
|
||||
}),
|
||||
)
|
||||
expect(patch).toHaveBeenCalledWith(
|
||||
'skillEmbeddings:1',
|
||||
expect.objectContaining({
|
||||
ownerId: 'users:new',
|
||||
}),
|
||||
)
|
||||
expect(patch).toHaveBeenCalledWith(
|
||||
'reservedSlugs:1',
|
||||
expect.objectContaining({
|
||||
releasedAt: expect.any(Number),
|
||||
}),
|
||||
)
|
||||
})
|
||||
|
||||
it('returns missing without reserving when transferRootSlugOnly is true and slug does not exist', async () => {
|
||||
const insert = vi.fn(async () => {})
|
||||
const patch = vi.fn(async () => {})
|
||||
const runAfter = vi.fn(async () => {})
|
||||
|
||||
const db = {
|
||||
get: vi.fn(async (id: string) => {
|
||||
if (id === 'users:admin') return { _id: 'users:admin', role: 'admin' }
|
||||
if (id === 'users:new') return { _id: 'users:new', role: 'user' }
|
||||
return null
|
||||
}),
|
||||
query: vi.fn((table: string) => {
|
||||
if (table === 'skills') {
|
||||
return {
|
||||
withIndex: (name: string) => {
|
||||
if (name !== 'by_slug') throw new Error(`unexpected skills index ${name}`)
|
||||
return { unique: async () => null }
|
||||
},
|
||||
}
|
||||
}
|
||||
throw new Error(`unexpected table ${table}`)
|
||||
}),
|
||||
patch,
|
||||
insert,
|
||||
}
|
||||
|
||||
const result = (await reclaimSlugInternalHandler(
|
||||
{ db, scheduler: { runAfter } } as never,
|
||||
{
|
||||
actorUserId: 'users:admin',
|
||||
slug: 'missing-slug',
|
||||
rightfulOwnerUserId: 'users:new',
|
||||
transferRootSlugOnly: true,
|
||||
} as never,
|
||||
)) as { ok: boolean; action: string }
|
||||
|
||||
expect(result).toEqual({ ok: true, action: 'missing' })
|
||||
expect(runAfter).not.toHaveBeenCalled()
|
||||
expect(patch).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
|
||||
+720
-172
File diff suppressed because it is too large
Load Diff
+5
-11
@@ -3,6 +3,7 @@ import { internal } from './_generated/api'
|
||||
import type { Doc, Id } from './_generated/dataModel'
|
||||
import { action, internalMutation, internalQuery, mutation, query } from './_generated/server'
|
||||
import { assertModerator, requireUser, requireUserFromAction } from './lib/access'
|
||||
import { embeddingVisibilityFor } from './lib/embeddingVisibility'
|
||||
import { toPublicSoul, toPublicUser } from './lib/public'
|
||||
import { getFrontmatterValue, hashSkillFiles } from './lib/skills'
|
||||
import { generateSoulChangelogPreview } from './lib/soulChangelog'
|
||||
@@ -357,7 +358,7 @@ export const updateTags = mutation({
|
||||
const isLatest = embedding.versionId === latestEntry.versionId
|
||||
await ctx.db.patch(embedding._id, {
|
||||
isLatest,
|
||||
visibility: visibilityFor(isLatest, embedding.isApproved),
|
||||
visibility: embeddingVisibilityFor(isLatest, embedding.isApproved),
|
||||
updatedAt: Date.now(),
|
||||
})
|
||||
}
|
||||
@@ -479,7 +480,7 @@ export const insertVersion = internalMutation({
|
||||
embedding: args.embedding,
|
||||
isLatest: true,
|
||||
isApproved: true,
|
||||
visibility: visibilityFor(true, true),
|
||||
visibility: embeddingVisibilityFor(true, true),
|
||||
updatedAt: now,
|
||||
})
|
||||
|
||||
@@ -491,7 +492,7 @@ export const insertVersion = internalMutation({
|
||||
if (previousEmbedding) {
|
||||
await ctx.db.patch(previousEmbedding._id, {
|
||||
isLatest: false,
|
||||
visibility: visibilityFor(false, previousEmbedding.isApproved),
|
||||
visibility: embeddingVisibilityFor(false, previousEmbedding.isApproved),
|
||||
updatedAt: now,
|
||||
})
|
||||
}
|
||||
@@ -547,7 +548,7 @@ export const setSoulSoftDeletedInternal = internalMutation({
|
||||
await ctx.db.patch(embedding._id, {
|
||||
visibility: args.deleted
|
||||
? 'deleted'
|
||||
: visibilityFor(embedding.isLatest, embedding.isApproved),
|
||||
: embeddingVisibilityFor(embedding.isLatest, embedding.isApproved),
|
||||
updatedAt: now,
|
||||
})
|
||||
}
|
||||
@@ -565,13 +566,6 @@ export const setSoulSoftDeletedInternal = internalMutation({
|
||||
},
|
||||
})
|
||||
|
||||
function visibilityFor(isLatest: boolean, isApproved: boolean) {
|
||||
if (isLatest && isApproved) return 'latest-approved'
|
||||
if (isLatest) return 'latest'
|
||||
if (isApproved) return 'archived-approved'
|
||||
return 'archived'
|
||||
}
|
||||
|
||||
function clampInt(value: number, min: number, max: number) {
|
||||
const rounded = Number.isFinite(value) ? Math.round(value) : min
|
||||
return Math.min(max, Math.max(min, rounded))
|
||||
|
||||
@@ -3,6 +3,10 @@ import { internal } from './_generated/api'
|
||||
import type { Doc } from './_generated/dataModel'
|
||||
import type { ActionCtx } from './_generated/server'
|
||||
import { internalAction, internalMutation, internalQuery } from './_generated/server'
|
||||
import {
|
||||
countPublicSkillsForGlobalStats,
|
||||
setGlobalPublicSkillsCount,
|
||||
} from './lib/globalStats'
|
||||
|
||||
const DEFAULT_BATCH_SIZE = 200
|
||||
const MAX_BATCH_SIZE = 1000
|
||||
@@ -299,3 +303,11 @@ export const runReconcileSkillStarCountsInternal = internalAction({
|
||||
function clampInt(value: number, min: number, max: number) {
|
||||
return Math.min(Math.max(value, min), max)
|
||||
}
|
||||
|
||||
export const updateGlobalStatsInternal = internalMutation({
|
||||
args: {},
|
||||
handler: async (ctx) => {
|
||||
const count = await countPublicSkillsForGlobalStats(ctx)
|
||||
await setGlobalPublicSkillsCount(ctx, count)
|
||||
},
|
||||
})
|
||||
|
||||
+356
-1
@@ -6,7 +6,7 @@ vi.mock('./lib/access', async () => {
|
||||
})
|
||||
|
||||
const { requireUser } = await import('./lib/access')
|
||||
const { ensureHandler } = await import('./users')
|
||||
const { ensureHandler, list, searchInternal } = await import('./users')
|
||||
|
||||
function makeCtx() {
|
||||
const patch = vi.fn()
|
||||
@@ -14,6 +14,22 @@ function makeCtx() {
|
||||
return { ctx: { db: { patch, get } } as never, patch, get }
|
||||
}
|
||||
|
||||
function makeListCtx(users: Array<Record<string, unknown>>) {
|
||||
const take = vi.fn(async (n: number) => users.slice(0, n))
|
||||
const collect = vi.fn(async () => users)
|
||||
const order = vi.fn(() => ({ take, collect }))
|
||||
const query = vi.fn(() => ({ order }))
|
||||
const get = vi.fn()
|
||||
return {
|
||||
ctx: { db: { query, get } } as never,
|
||||
take,
|
||||
collect,
|
||||
order,
|
||||
query,
|
||||
get,
|
||||
}
|
||||
}
|
||||
|
||||
describe('ensureHandler', () => {
|
||||
afterEach(() => {
|
||||
vi.mocked(requireUser).mockReset()
|
||||
@@ -87,4 +103,343 @@ describe('ensureHandler', () => {
|
||||
updatedAt: expect.any(Number),
|
||||
})
|
||||
})
|
||||
|
||||
it('does not patch when user metadata is already normalized', async () => {
|
||||
const { ctx, patch, get } = makeCtx()
|
||||
get.mockResolvedValue({
|
||||
_id: 'users:4',
|
||||
handle: 'steady',
|
||||
displayName: 'Steady Name',
|
||||
name: 'steady',
|
||||
role: 'user',
|
||||
_creationTime: 1,
|
||||
createdAt: 1,
|
||||
})
|
||||
vi.mocked(requireUser).mockResolvedValue({
|
||||
userId: 'users:4',
|
||||
user: {
|
||||
_creationTime: 1,
|
||||
handle: 'steady',
|
||||
displayName: 'Steady Name',
|
||||
name: 'steady',
|
||||
role: 'user',
|
||||
createdAt: 1,
|
||||
},
|
||||
} as never)
|
||||
|
||||
const result = await ensureHandler(ctx)
|
||||
|
||||
expect(patch).not.toHaveBeenCalled()
|
||||
expect(get).toHaveBeenCalledWith('users:4')
|
||||
expect(result).toMatchObject({ _id: 'users:4' })
|
||||
})
|
||||
|
||||
it('sets admin role when normalized handle is steipete and role is missing', async () => {
|
||||
const { ctx, patch } = makeCtx()
|
||||
vi.mocked(requireUser).mockResolvedValue({
|
||||
userId: 'users:admin',
|
||||
user: {
|
||||
_creationTime: 1,
|
||||
handle: 'steipete',
|
||||
displayName: 'steipete',
|
||||
name: 'steipete',
|
||||
role: undefined,
|
||||
createdAt: 1,
|
||||
},
|
||||
} as never)
|
||||
|
||||
await ensureHandler(ctx)
|
||||
|
||||
expect(patch).toHaveBeenCalledWith('users:admin', {
|
||||
displayName: 'steipete',
|
||||
role: 'admin',
|
||||
updatedAt: expect.any(Number),
|
||||
})
|
||||
})
|
||||
|
||||
it('derives handle/display name from email when missing', async () => {
|
||||
const { ctx, patch } = makeCtx()
|
||||
vi.mocked(requireUser).mockResolvedValue({
|
||||
userId: 'users:email',
|
||||
user: {
|
||||
_creationTime: 1,
|
||||
handle: undefined,
|
||||
displayName: undefined,
|
||||
name: undefined,
|
||||
email: 'owner@example.com',
|
||||
role: undefined,
|
||||
createdAt: undefined,
|
||||
},
|
||||
} as never)
|
||||
|
||||
await ensureHandler(ctx)
|
||||
|
||||
expect(patch).toHaveBeenCalledWith('users:email', {
|
||||
handle: 'owner',
|
||||
displayName: 'owner',
|
||||
role: 'user',
|
||||
createdAt: 1,
|
||||
updatedAt: expect.any(Number),
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
describe('users.list', () => {
|
||||
afterEach(() => {
|
||||
vi.mocked(requireUser).mockReset()
|
||||
})
|
||||
|
||||
it('uses take(limit) without full collect when search is empty', async () => {
|
||||
vi.mocked(requireUser).mockResolvedValue({
|
||||
userId: 'users:admin',
|
||||
user: { _id: 'users:admin', role: 'admin' },
|
||||
} as never)
|
||||
const users = [
|
||||
{ _id: 'users:1', _creationTime: 3, handle: 'alice', role: 'user' },
|
||||
{ _id: 'users:2', _creationTime: 2, handle: 'bob', role: 'user' },
|
||||
{ _id: 'users:3', _creationTime: 1, handle: 'carol', role: 'user' },
|
||||
]
|
||||
const { ctx, take, collect } = makeListCtx(users)
|
||||
const listHandler = (list as unknown as { _handler: (ctx: unknown, args: unknown) => Promise<unknown> })
|
||||
._handler
|
||||
|
||||
const result = (await listHandler(ctx, { limit: 2 })) as {
|
||||
items: Array<Record<string, unknown>>
|
||||
total: number
|
||||
}
|
||||
|
||||
expect(take).toHaveBeenCalledWith(2)
|
||||
expect(collect).not.toHaveBeenCalled()
|
||||
expect(result.total).toBe(2)
|
||||
expect(result.items).toHaveLength(2)
|
||||
})
|
||||
|
||||
it('uses bounded scan for search instead of full collect', async () => {
|
||||
vi.mocked(requireUser).mockResolvedValue({
|
||||
userId: 'users:admin',
|
||||
user: { _id: 'users:admin', role: 'admin' },
|
||||
} as never)
|
||||
const users = [
|
||||
{ _id: 'users:1', _creationTime: 3, handle: 'alice', role: 'user' },
|
||||
{ _id: 'users:2', _creationTime: 2, handle: 'bob', role: 'user' },
|
||||
{ _id: 'users:3', _creationTime: 1, handle: 'carol', role: 'user' },
|
||||
]
|
||||
const { ctx, take, collect } = makeListCtx(users)
|
||||
const listHandler = (list as unknown as { _handler: (ctx: unknown, args: unknown) => Promise<unknown> })
|
||||
._handler
|
||||
|
||||
const result = (await listHandler(ctx, { limit: 50, search: 'ali' })) as {
|
||||
items: Array<Record<string, unknown>>
|
||||
total: number
|
||||
}
|
||||
|
||||
expect(take).toHaveBeenCalledWith(500)
|
||||
expect(collect).not.toHaveBeenCalled()
|
||||
expect(result.total).toBe(1)
|
||||
expect(result.items).toHaveLength(1)
|
||||
expect(result.items[0]?.handle).toBe('alice')
|
||||
})
|
||||
|
||||
it('clamps large limit and search scan size', async () => {
|
||||
vi.mocked(requireUser).mockResolvedValue({
|
||||
userId: 'users:admin',
|
||||
user: { _id: 'users:admin', role: 'admin' },
|
||||
} as never)
|
||||
const users = Array.from({ length: 8_000 }, (_value, index) => ({
|
||||
_id: `users:${index}`,
|
||||
_creationTime: 10_000 - index,
|
||||
handle: `user-${index}`,
|
||||
role: 'user',
|
||||
}))
|
||||
const { ctx, take } = makeListCtx(users)
|
||||
const listHandler = (list as unknown as { _handler: (ctx: unknown, args: unknown) => Promise<unknown> })
|
||||
._handler
|
||||
|
||||
await listHandler(ctx, { limit: 999, search: 'user' })
|
||||
|
||||
expect(take).toHaveBeenCalledWith(2_000)
|
||||
})
|
||||
|
||||
it('handles malformed legacy user fields without throwing', async () => {
|
||||
vi.mocked(requireUser).mockResolvedValue({
|
||||
userId: 'users:admin',
|
||||
user: { _id: 'users:admin', role: 'admin' },
|
||||
} as never)
|
||||
const users = [
|
||||
{
|
||||
_id: 'users:legacy',
|
||||
_creationTime: 99,
|
||||
handle: 123,
|
||||
name: { broken: true },
|
||||
displayName: null,
|
||||
email: ['legacy@example.com'],
|
||||
role: 'user',
|
||||
},
|
||||
{
|
||||
_id: 'users:2',
|
||||
_creationTime: 98,
|
||||
handle: 'carol',
|
||||
role: 'user',
|
||||
},
|
||||
]
|
||||
const { ctx } = makeListCtx(users)
|
||||
const listHandler = (list as unknown as { _handler: (ctx: unknown, args: unknown) => Promise<unknown> })
|
||||
._handler
|
||||
|
||||
await expect(listHandler(ctx, { limit: 50, search: 'car' })).resolves.toMatchObject({
|
||||
total: 1,
|
||||
items: [{ _id: 'users:2' }],
|
||||
})
|
||||
})
|
||||
|
||||
it('treats whitespace search as empty search', async () => {
|
||||
vi.mocked(requireUser).mockResolvedValue({
|
||||
userId: 'users:admin',
|
||||
user: { _id: 'users:admin', role: 'admin' },
|
||||
} as never)
|
||||
const users = [
|
||||
{ _id: 'users:1', _creationTime: 2, handle: 'alice', role: 'user' },
|
||||
{ _id: 'users:2', _creationTime: 1, handle: 'bob', role: 'user' },
|
||||
]
|
||||
const { ctx, take, collect } = makeListCtx(users)
|
||||
const listHandler = (list as unknown as { _handler: (ctx: unknown, args: unknown) => Promise<unknown> })
|
||||
._handler
|
||||
|
||||
const result = (await listHandler(ctx, { limit: 50, search: ' ' })) as {
|
||||
items: Array<Record<string, unknown>>
|
||||
total: number
|
||||
}
|
||||
|
||||
expect(take).toHaveBeenCalledWith(50)
|
||||
expect(collect).not.toHaveBeenCalled()
|
||||
expect(result.total).toBe(2)
|
||||
})
|
||||
|
||||
it('clamps non-positive limit to one', async () => {
|
||||
vi.mocked(requireUser).mockResolvedValue({
|
||||
userId: 'users:admin',
|
||||
user: { _id: 'users:admin', role: 'admin' },
|
||||
} as never)
|
||||
const users = [
|
||||
{ _id: 'users:1', _creationTime: 2, handle: 'alice', role: 'user' },
|
||||
{ _id: 'users:2', _creationTime: 1, handle: 'bob', role: 'user' },
|
||||
]
|
||||
const { ctx, take } = makeListCtx(users)
|
||||
const listHandler = (list as unknown as { _handler: (ctx: unknown, args: unknown) => Promise<unknown> })
|
||||
._handler
|
||||
|
||||
const result = (await listHandler(ctx, { limit: 0 })) as {
|
||||
items: Array<Record<string, unknown>>
|
||||
total: number
|
||||
}
|
||||
|
||||
expect(take).toHaveBeenCalledWith(1)
|
||||
expect(result.total).toBe(1)
|
||||
expect(result.items).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('rejects non-admin actors', async () => {
|
||||
vi.mocked(requireUser).mockResolvedValue({
|
||||
userId: 'users:basic',
|
||||
user: { _id: 'users:basic', role: 'user' },
|
||||
} as never)
|
||||
const { ctx } = makeListCtx([])
|
||||
const listHandler = (list as unknown as { _handler: (ctx: unknown, args: unknown) => Promise<unknown> })
|
||||
._handler
|
||||
|
||||
await expect(listHandler(ctx, { limit: 10 })).rejects.toThrow('Forbidden')
|
||||
})
|
||||
})
|
||||
|
||||
describe('users.searchInternal', () => {
|
||||
it('rejects missing actor', async () => {
|
||||
const { ctx, get } = makeListCtx([])
|
||||
const handler = (
|
||||
searchInternal as unknown as { _handler: (ctx: unknown, args: unknown) => Promise<unknown> }
|
||||
)._handler
|
||||
get.mockResolvedValue(null)
|
||||
|
||||
await expect(handler(ctx, { actorUserId: 'users:missing' })).rejects.toThrow('Unauthorized')
|
||||
})
|
||||
|
||||
it('uses bounded scan and returns mapped fields', async () => {
|
||||
const users = [
|
||||
{ _id: 'users:1', _creationTime: 2, handle: 'alice', name: 'alice', role: 'user' },
|
||||
{ _id: 'users:2', _creationTime: 1, handle: 'bob', name: 'bob', role: 'moderator' },
|
||||
]
|
||||
const { ctx, take, collect, get } = makeListCtx(users)
|
||||
const handler = (
|
||||
searchInternal as unknown as { _handler: (ctx: unknown, args: unknown) => Promise<unknown> }
|
||||
)._handler
|
||||
get.mockResolvedValue({ _id: 'users:admin', role: 'admin' })
|
||||
|
||||
const result = (await handler(ctx, {
|
||||
actorUserId: 'users:admin',
|
||||
query: 'ali',
|
||||
limit: 25,
|
||||
})) as {
|
||||
items: Array<Record<string, unknown>>
|
||||
total: number
|
||||
}
|
||||
|
||||
expect(take).toHaveBeenCalledWith(500)
|
||||
expect(collect).not.toHaveBeenCalled()
|
||||
expect(result.total).toBe(1)
|
||||
expect(result.items).toEqual([
|
||||
{
|
||||
userId: 'users:1',
|
||||
handle: 'alice',
|
||||
displayName: null,
|
||||
name: 'alice',
|
||||
role: 'user',
|
||||
},
|
||||
])
|
||||
})
|
||||
|
||||
it('rejects deactivated actors', async () => {
|
||||
const { ctx, get } = makeListCtx([])
|
||||
const handler = (
|
||||
searchInternal as unknown as { _handler: (ctx: unknown, args: unknown) => Promise<unknown> }
|
||||
)._handler
|
||||
get.mockResolvedValue({ _id: 'users:ghost', role: 'admin', deactivatedAt: Date.now() })
|
||||
|
||||
await expect(handler(ctx, { actorUserId: 'users:ghost' })).rejects.toThrow('Unauthorized')
|
||||
})
|
||||
|
||||
it('rejects non-admin actors', async () => {
|
||||
const { ctx, get } = makeListCtx([])
|
||||
const handler = (
|
||||
searchInternal as unknown as { _handler: (ctx: unknown, args: unknown) => Promise<unknown> }
|
||||
)._handler
|
||||
get.mockResolvedValue({ _id: 'users:mod', role: 'moderator' })
|
||||
|
||||
await expect(handler(ctx, { actorUserId: 'users:mod', query: 'a' })).rejects.toThrow(
|
||||
'Forbidden',
|
||||
)
|
||||
})
|
||||
|
||||
it('clamps limit for empty query and uses non-search path', async () => {
|
||||
const users = Array.from({ length: 400 }, (_value, index) => ({
|
||||
_id: `users:${index}`,
|
||||
_creationTime: 1_000 - index,
|
||||
handle: `user-${index}`,
|
||||
role: 'user',
|
||||
}))
|
||||
const { ctx, take, collect, get } = makeListCtx(users)
|
||||
const handler = (
|
||||
searchInternal as unknown as { _handler: (ctx: unknown, args: unknown) => Promise<unknown> }
|
||||
)._handler
|
||||
get.mockResolvedValue({ _id: 'users:admin', role: 'admin' })
|
||||
|
||||
const result = (await handler(ctx, {
|
||||
actorUserId: 'users:admin',
|
||||
limit: 999,
|
||||
query: ' ',
|
||||
})) as { items: Array<Record<string, unknown>>; total: number }
|
||||
|
||||
expect(take).toHaveBeenCalledWith(200)
|
||||
expect(collect).not.toHaveBeenCalled()
|
||||
expect(result.total).toBe(200)
|
||||
expect(result.items).toHaveLength(200)
|
||||
})
|
||||
})
|
||||
|
||||
+232
-57
@@ -2,14 +2,18 @@ import { getAuthUserId } from '@convex-dev/auth/server'
|
||||
import { v } from 'convex/values'
|
||||
import { internal } from './_generated/api'
|
||||
import type { Doc, Id } from './_generated/dataModel'
|
||||
import type { MutationCtx } from './_generated/server'
|
||||
import { internalMutation, internalQuery, mutation, query } from './_generated/server'
|
||||
import type { ActionCtx, MutationCtx } from './_generated/server'
|
||||
import { internalAction, internalMutation, internalQuery, mutation, query } from './_generated/server'
|
||||
import { assertAdmin, assertModerator, requireUser } from './lib/access'
|
||||
import { syncGitHubProfile } from './lib/githubAccount'
|
||||
import { toPublicUser } from './lib/public'
|
||||
import { buildUserSearchResults } from './lib/userSearch'
|
||||
|
||||
const DEFAULT_ROLE = 'user'
|
||||
const ADMIN_HANDLE = 'steipete'
|
||||
const MAX_USER_LIST_LIMIT = 200
|
||||
const MAX_USER_SEARCH_SCAN = 5_000
|
||||
const MIN_USER_SEARCH_SCAN = 500
|
||||
|
||||
export const getById = query({
|
||||
args: { userId: v.id('users') },
|
||||
@@ -21,17 +25,6 @@ export const getByIdInternal = internalQuery({
|
||||
handler: async (ctx, args) => ctx.db.get(args.userId),
|
||||
})
|
||||
|
||||
export const getGitHubProviderAccountIdInternal = internalQuery({
|
||||
args: { userId: v.id('users') },
|
||||
handler: async (ctx, args) => {
|
||||
const account = await ctx.db
|
||||
.query('authAccounts')
|
||||
.withIndex('userIdAndProvider', (q) => q.eq('userId', args.userId).eq('provider', 'github'))
|
||||
.unique()
|
||||
return account?.providerAccountId ?? null
|
||||
},
|
||||
})
|
||||
|
||||
export const searchInternal = internalQuery({
|
||||
args: {
|
||||
actorUserId: v.id('users'),
|
||||
@@ -43,10 +36,9 @@ export const searchInternal = internalQuery({
|
||||
if (!actor || actor.deletedAt || actor.deactivatedAt) throw new Error('Unauthorized')
|
||||
assertAdmin(actor)
|
||||
|
||||
const limit = Math.min(Math.max(args.limit ?? 20, 1), 200)
|
||||
const users = await ctx.db.query('users').order('desc').collect()
|
||||
const result = buildUserSearchResults(users, args.query)
|
||||
const items = result.items.slice(0, limit).map((user) => ({
|
||||
const limit = clampInt(args.limit ?? 20, 1, MAX_USER_LIST_LIMIT)
|
||||
const result = await queryUsersForAdminList(ctx, { limit, search: args.query })
|
||||
const items = result.items.map((user) => ({
|
||||
userId: user._id,
|
||||
handle: user.handle ?? null,
|
||||
displayName: user.displayName ?? null,
|
||||
@@ -56,21 +48,97 @@ export const searchInternal = internalQuery({
|
||||
return { items, total: result.total }
|
||||
},
|
||||
})
|
||||
export const updateGithubMetaInternal = internalMutation({
|
||||
|
||||
export const setGitHubCreatedAtInternal = internalMutation({
|
||||
args: {
|
||||
userId: v.id('users'),
|
||||
githubCreatedAt: v.number(),
|
||||
githubFetchedAt: v.number(),
|
||||
},
|
||||
handler: async (ctx, args) => {
|
||||
await ctx.db.patch(args.userId, {
|
||||
githubCreatedAt: args.githubCreatedAt,
|
||||
githubFetchedAt: args.githubFetchedAt,
|
||||
updatedAt: args.githubFetchedAt,
|
||||
updatedAt: Date.now(),
|
||||
})
|
||||
},
|
||||
})
|
||||
|
||||
/**
|
||||
* Sync the user's GitHub profile (username, avatar) when it changes.
|
||||
* This handles the case where a user renames their GitHub account.
|
||||
*/
|
||||
export const syncGitHubProfileInternal = internalMutation({
|
||||
args: {
|
||||
userId: v.id('users'),
|
||||
name: v.string(),
|
||||
image: v.optional(v.string()),
|
||||
profileName: v.optional(v.string()),
|
||||
syncedAt: v.number(),
|
||||
},
|
||||
handler: async (ctx, args) => {
|
||||
const user = await ctx.db.get(args.userId)
|
||||
if (!user || user.deletedAt || user.deactivatedAt) return
|
||||
|
||||
const updates: Partial<Doc<'users'>> = { githubProfileSyncedAt: args.syncedAt }
|
||||
let didChangeProfile = false
|
||||
|
||||
if (user.name !== args.name) {
|
||||
updates.name = args.name
|
||||
didChangeProfile = true
|
||||
}
|
||||
|
||||
// Update handle if it was derived from the old username
|
||||
if (user.handle === user.name && user.name !== args.name) {
|
||||
updates.handle = args.name
|
||||
didChangeProfile = true
|
||||
}
|
||||
|
||||
// Update displayName if it was derived from the old username
|
||||
if (
|
||||
(user.displayName === user.name || user.displayName === user.handle) &&
|
||||
user.name !== args.name
|
||||
) {
|
||||
updates.displayName = args.name
|
||||
didChangeProfile = true
|
||||
}
|
||||
|
||||
// If displayName is derived/missing, prefer the GitHub profile "name" (full name).
|
||||
const profileName = args.profileName?.trim()
|
||||
if (profileName && profileName !== args.name) {
|
||||
const currentDisplay = user.displayName?.trim()
|
||||
const currentHandle = user.handle?.trim()
|
||||
const currentLogin = user.name?.trim()
|
||||
const isDerivedOrMissing =
|
||||
!currentDisplay || currentDisplay === currentHandle || currentDisplay === currentLogin
|
||||
if (isDerivedOrMissing && currentDisplay !== profileName) {
|
||||
updates.displayName = profileName
|
||||
didChangeProfile = true
|
||||
}
|
||||
}
|
||||
|
||||
// Update avatar if provided
|
||||
if (args.image && args.image !== user.image) {
|
||||
updates.image = args.image
|
||||
didChangeProfile = true
|
||||
}
|
||||
|
||||
if (didChangeProfile) {
|
||||
updates.updatedAt = Date.now()
|
||||
}
|
||||
await ctx.db.patch(args.userId, updates)
|
||||
},
|
||||
})
|
||||
|
||||
/**
|
||||
* Internal action to sync GitHub profile from the GitHub API.
|
||||
* This is called after login to ensure the username is up-to-date.
|
||||
*/
|
||||
export const syncGitHubProfileAction = internalAction({
|
||||
args: { userId: v.id('users') },
|
||||
handler: async (ctx: ActionCtx, args) => {
|
||||
await syncGitHubProfile(ctx, args.userId)
|
||||
},
|
||||
})
|
||||
|
||||
export const me = query({
|
||||
args: {},
|
||||
handler: async (ctx) => {
|
||||
@@ -191,7 +259,6 @@ export const deleteAccount = mutation({
|
||||
isAnonymous: undefined,
|
||||
bio: undefined,
|
||||
githubCreatedAt: undefined,
|
||||
githubFetchedAt: undefined,
|
||||
updatedAt: now,
|
||||
})
|
||||
await ctx.runMutation(internal.telemetry.clearUserTelemetryInternal, { userId })
|
||||
@@ -203,14 +270,41 @@ export const list = query({
|
||||
handler: async (ctx, args) => {
|
||||
const { user } = await requireUser(ctx)
|
||||
assertAdmin(user)
|
||||
const limit = Math.min(Math.max(args.limit ?? 50, 1), 200)
|
||||
const query = args.search?.trim().toLowerCase()
|
||||
const users = await ctx.db.query('users').order('desc').collect()
|
||||
const result = buildUserSearchResults(users, query)
|
||||
return { items: result.items.slice(0, limit), total: result.total }
|
||||
const limit = clampInt(args.limit ?? 50, 1, MAX_USER_LIST_LIMIT)
|
||||
return queryUsersForAdminList(ctx, { limit, search: args.search })
|
||||
},
|
||||
})
|
||||
|
||||
function normalizeSearchQuery(search?: string) {
|
||||
const trimmed = search?.trim().toLowerCase()
|
||||
return trimmed ? trimmed : undefined
|
||||
}
|
||||
|
||||
function computeUserSearchScanLimit(limit: number) {
|
||||
return clampInt(limit * 10, MIN_USER_SEARCH_SCAN, MAX_USER_SEARCH_SCAN)
|
||||
}
|
||||
|
||||
async function queryUsersForAdminList(
|
||||
ctx: { db: { query: (table: 'users') => { order: (order: 'desc') => { take: (n: number) => Promise<Doc<'users'>[]> } } } },
|
||||
args: { limit: number; search?: string },
|
||||
) {
|
||||
const normalizedSearch = normalizeSearchQuery(args.search)
|
||||
const orderedUsers = ctx.db.query('users').order('desc')
|
||||
|
||||
if (!normalizedSearch) {
|
||||
const items = await orderedUsers.take(args.limit)
|
||||
return { items, total: items.length }
|
||||
}
|
||||
|
||||
const scannedUsers = await orderedUsers.take(computeUserSearchScanLimit(args.limit))
|
||||
const result = buildUserSearchResults(scannedUsers, normalizedSearch)
|
||||
return { items: result.items.slice(0, args.limit), total: result.total }
|
||||
}
|
||||
|
||||
function clampInt(value: number, min: number, max: number) {
|
||||
return Math.min(Math.max(Math.trunc(value), min), max)
|
||||
}
|
||||
|
||||
export const getByHandle = query({
|
||||
args: { handle: v.string() },
|
||||
handler: async (ctx, args) => {
|
||||
@@ -335,24 +429,26 @@ async function banUserWithActor(
|
||||
return { ok: true as const, alreadyBanned: true, deletedSkills: 0 }
|
||||
}
|
||||
|
||||
const skills = await ctx.db
|
||||
.query('skills')
|
||||
.withIndex('by_owner', (q) => q.eq('ownerUserId', targetUserId))
|
||||
.collect()
|
||||
|
||||
for (const skill of skills) {
|
||||
await ctx.scheduler.runAfter(0, internal.skills.hardDeleteInternal, {
|
||||
skillId: skill._id,
|
||||
actorUserId: actor._id,
|
||||
})
|
||||
}
|
||||
const banSkillsResult = (await ctx.runMutation(
|
||||
internal.skills.applyBanToOwnedSkillsBatchInternal,
|
||||
{
|
||||
ownerUserId: targetUserId,
|
||||
bannedAt: now,
|
||||
hiddenBy: actor._id,
|
||||
cursor: undefined,
|
||||
},
|
||||
)) as { hiddenCount?: number; scheduled?: boolean }
|
||||
const hiddenCount = banSkillsResult.hiddenCount ?? 0
|
||||
const scheduledSkills = banSkillsResult.scheduled ?? false
|
||||
|
||||
const tokens = await ctx.db
|
||||
.query('apiTokens')
|
||||
.withIndex('by_user', (q) => q.eq('userId', targetUserId))
|
||||
.collect()
|
||||
for (const token of tokens) {
|
||||
await ctx.db.patch(token._id, { revokedAt: now })
|
||||
if (!token.revokedAt) {
|
||||
await ctx.db.patch(token._id, { revokedAt: now })
|
||||
}
|
||||
}
|
||||
|
||||
await ctx.db.patch(targetUserId, {
|
||||
@@ -369,11 +465,11 @@ async function banUserWithActor(
|
||||
action: 'user.ban',
|
||||
targetType: 'user',
|
||||
targetId: targetUserId,
|
||||
metadata: { deletedSkills: skills.length, reason: reason || undefined },
|
||||
metadata: { hiddenSkills: hiddenCount, reason: reason || undefined },
|
||||
createdAt: now,
|
||||
})
|
||||
|
||||
return { ok: true as const, alreadyBanned: false, deletedSkills: skills.length }
|
||||
return { ok: true as const, alreadyBanned: false, deletedSkills: hiddenCount, scheduledSkills }
|
||||
}
|
||||
|
||||
async function unbanUserWithActor(
|
||||
@@ -400,6 +496,7 @@ async function unbanUserWithActor(
|
||||
}
|
||||
|
||||
const now = Date.now()
|
||||
const bannedAt = target.deletedAt
|
||||
await ctx.db.patch(targetUserId, {
|
||||
deletedAt: undefined,
|
||||
banReason: undefined,
|
||||
@@ -407,18 +504,97 @@ async function unbanUserWithActor(
|
||||
updatedAt: now,
|
||||
})
|
||||
|
||||
const restoreSkillsResult = (await ctx.runMutation(
|
||||
internal.skills.restoreOwnedSkillsForUnbanBatchInternal,
|
||||
{
|
||||
ownerUserId: targetUserId,
|
||||
bannedAt,
|
||||
cursor: undefined,
|
||||
},
|
||||
)) as { restoredCount?: number; scheduled?: boolean }
|
||||
const restoredCount = restoreSkillsResult.restoredCount ?? 0
|
||||
const scheduledSkills = restoreSkillsResult.scheduled ?? false
|
||||
|
||||
await ctx.db.insert('auditLogs', {
|
||||
actorUserId: actor._id,
|
||||
action: 'user.unban',
|
||||
targetType: 'user',
|
||||
targetId: targetUserId,
|
||||
metadata: { reason: reason || undefined },
|
||||
metadata: { reason: reason || undefined, restoredSkills: restoredCount },
|
||||
createdAt: now,
|
||||
})
|
||||
|
||||
return { ok: true as const, alreadyUnbanned: false }
|
||||
return { ok: true as const, alreadyUnbanned: false, restoredSkills: restoredCount, scheduledSkills }
|
||||
}
|
||||
|
||||
/**
|
||||
* Admin-only: set or unset the trustedPublisher flag for a user.
|
||||
* Trusted publishers bypass the pending.scan auto-hide for new skill publishes.
|
||||
*/
|
||||
export const setTrustedPublisher = mutation({
|
||||
args: {
|
||||
userId: v.id('users'),
|
||||
trusted: v.boolean(),
|
||||
},
|
||||
handler: async (ctx, args) => {
|
||||
const { user } = await requireUser(ctx)
|
||||
assertAdmin(user)
|
||||
|
||||
const target = await ctx.db.get(args.userId)
|
||||
if (!target) throw new Error('User not found')
|
||||
|
||||
const now = Date.now()
|
||||
await ctx.db.patch(args.userId, {
|
||||
trustedPublisher: args.trusted || undefined,
|
||||
updatedAt: now,
|
||||
})
|
||||
|
||||
await ctx.db.insert('auditLogs', {
|
||||
actorUserId: user._id,
|
||||
action: args.trusted ? 'user.trusted.set' : 'user.trusted.unset',
|
||||
targetType: 'user',
|
||||
targetId: args.userId,
|
||||
metadata: { trusted: args.trusted },
|
||||
createdAt: now,
|
||||
})
|
||||
|
||||
return { ok: true as const, trusted: args.trusted }
|
||||
},
|
||||
})
|
||||
|
||||
export const setTrustedPublisherInternal = internalMutation({
|
||||
args: {
|
||||
actorUserId: v.id('users'),
|
||||
targetUserId: v.id('users'),
|
||||
trusted: v.boolean(),
|
||||
},
|
||||
handler: async (ctx, args) => {
|
||||
const actor = await ctx.db.get(args.actorUserId)
|
||||
if (!actor || actor.deletedAt || actor.deactivatedAt) throw new Error('User not found')
|
||||
assertAdmin(actor)
|
||||
|
||||
const target = await ctx.db.get(args.targetUserId)
|
||||
if (!target) throw new Error('User not found')
|
||||
|
||||
const now = Date.now()
|
||||
await ctx.db.patch(args.targetUserId, {
|
||||
trustedPublisher: args.trusted || undefined,
|
||||
updatedAt: now,
|
||||
})
|
||||
|
||||
await ctx.db.insert('auditLogs', {
|
||||
actorUserId: args.actorUserId,
|
||||
action: args.trusted ? 'user.trusted.set' : 'user.trusted.unset',
|
||||
targetType: 'user',
|
||||
targetId: args.targetUserId,
|
||||
metadata: { trusted: args.trusted },
|
||||
createdAt: now,
|
||||
})
|
||||
|
||||
return { ok: true as const, trusted: args.trusted }
|
||||
},
|
||||
})
|
||||
|
||||
/**
|
||||
* Auto-ban a user whose skill was flagged malicious by VT.
|
||||
* Skips moderators/admins. No actor required — this is a system-level action.
|
||||
@@ -442,17 +618,16 @@ export const autobanMalwareAuthorInternal = internalMutation({
|
||||
|
||||
const now = Date.now()
|
||||
|
||||
// Soft-delete all their skills
|
||||
const skills = await ctx.db
|
||||
.query('skills')
|
||||
.withIndex('by_owner', (q) => q.eq('ownerUserId', args.ownerUserId))
|
||||
.collect()
|
||||
|
||||
for (const skill of skills) {
|
||||
if (!skill.softDeletedAt) {
|
||||
await ctx.db.patch(skill._id, { softDeletedAt: now, updatedAt: now })
|
||||
}
|
||||
}
|
||||
const banSkillsResult = (await ctx.runMutation(
|
||||
internal.skills.applyBanToOwnedSkillsBatchInternal,
|
||||
{
|
||||
ownerUserId: args.ownerUserId,
|
||||
bannedAt: now,
|
||||
cursor: undefined,
|
||||
},
|
||||
)) as { hiddenCount?: number; scheduled?: boolean }
|
||||
const hiddenCount = banSkillsResult.hiddenCount ?? 0
|
||||
const scheduledSkills = banSkillsResult.scheduled ?? false
|
||||
|
||||
// Revoke all API tokens
|
||||
const tokens = await ctx.db
|
||||
@@ -477,7 +652,7 @@ export const autobanMalwareAuthorInternal = internalMutation({
|
||||
userId: args.ownerUserId,
|
||||
})
|
||||
|
||||
// Audit log — use the target as actor since there's no human actor
|
||||
// Audit log -- use the target as actor since there's no human actor
|
||||
await ctx.db.insert('auditLogs', {
|
||||
actorUserId: args.ownerUserId,
|
||||
action: 'user.autoban.malware',
|
||||
@@ -487,7 +662,7 @@ export const autobanMalwareAuthorInternal = internalMutation({
|
||||
trigger: 'vt.malicious',
|
||||
sha256hash: args.sha256hash,
|
||||
slug: args.slug,
|
||||
deletedSkills: skills.length,
|
||||
hiddenSkills: hiddenCount,
|
||||
},
|
||||
createdAt: now,
|
||||
})
|
||||
@@ -496,6 +671,6 @@ export const autobanMalwareAuthorInternal = internalMutation({
|
||||
`[autoban] Banned ${target.handle ?? args.ownerUserId} — malicious skill: ${args.slug}`,
|
||||
)
|
||||
|
||||
return { ok: true, alreadyBanned: false, deletedSkills: skills.length }
|
||||
return { ok: true, alreadyBanned: false, deletedSkills: hiddenCount, scheduledSkills }
|
||||
},
|
||||
})
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { __test } from './vt'
|
||||
|
||||
describe('vt activation fallback', () => {
|
||||
it('activates only VT-pending hidden skills', () => {
|
||||
expect(
|
||||
__test.shouldActivateWhenVtUnavailable({
|
||||
moderationStatus: 'hidden',
|
||||
moderationReason: 'pending.scan',
|
||||
}),
|
||||
).toBe(true)
|
||||
|
||||
expect(
|
||||
__test.shouldActivateWhenVtUnavailable({
|
||||
moderationStatus: 'hidden',
|
||||
moderationReason: 'scanner.vt.pending',
|
||||
}),
|
||||
).toBe(true)
|
||||
|
||||
expect(
|
||||
__test.shouldActivateWhenVtUnavailable({
|
||||
moderationStatus: 'hidden',
|
||||
moderationReason: 'pending.scan.stale',
|
||||
}),
|
||||
).toBe(true)
|
||||
})
|
||||
|
||||
it('does not activate quality or scanner-hidden skills', () => {
|
||||
expect(
|
||||
__test.shouldActivateWhenVtUnavailable({
|
||||
moderationStatus: 'hidden',
|
||||
moderationReason: 'quality.low',
|
||||
}),
|
||||
).toBe(false)
|
||||
|
||||
expect(
|
||||
__test.shouldActivateWhenVtUnavailable({
|
||||
moderationStatus: 'hidden',
|
||||
moderationReason: 'scanner.llm.malicious',
|
||||
}),
|
||||
).toBe(false)
|
||||
})
|
||||
|
||||
it('does not activate blocked or already-active skills', () => {
|
||||
expect(
|
||||
__test.shouldActivateWhenVtUnavailable({
|
||||
moderationStatus: 'hidden',
|
||||
moderationReason: 'pending.scan',
|
||||
moderationFlags: ['blocked.malware'],
|
||||
}),
|
||||
).toBe(false)
|
||||
|
||||
expect(
|
||||
__test.shouldActivateWhenVtUnavailable({
|
||||
moderationStatus: 'active',
|
||||
moderationReason: 'pending.scan',
|
||||
}),
|
||||
).toBe(false)
|
||||
})
|
||||
})
|
||||
+50
-2
@@ -1,6 +1,7 @@
|
||||
import { v } from 'convex/values'
|
||||
import { internal } from './_generated/api'
|
||||
import type { Id } from './_generated/dataModel'
|
||||
import type { ActionCtx } from './_generated/server'
|
||||
import { action, internalAction, internalMutation } from './_generated/server'
|
||||
import { buildDeterministicZip } from './lib/skillZip'
|
||||
|
||||
@@ -136,6 +137,13 @@ type PendingScanSkill = {
|
||||
checkCount: number
|
||||
}
|
||||
|
||||
type SkillActivationCandidate = {
|
||||
moderationStatus?: string
|
||||
moderationReason?: string
|
||||
moderationFlags?: string[]
|
||||
softDeletedAt?: number
|
||||
}
|
||||
|
||||
type PollPendingScansResult = {
|
||||
processed: number
|
||||
updated: number
|
||||
@@ -228,6 +236,23 @@ type SyncModerationReasonsResult = {
|
||||
done: boolean
|
||||
}
|
||||
|
||||
const VT_PENDING_REASONS = new Set(['pending.scan', 'scanner.vt.pending', 'pending.scan.stale'])
|
||||
|
||||
function shouldActivateWhenVtUnavailable(skill: SkillActivationCandidate | null | undefined) {
|
||||
if (!skill || skill.softDeletedAt) return false
|
||||
if (skill.moderationFlags?.includes('blocked.malware')) return false
|
||||
if (skill.moderationStatus === 'active') return false
|
||||
const reason = skill.moderationReason
|
||||
return typeof reason === 'string' && VT_PENDING_REASONS.has(reason)
|
||||
}
|
||||
|
||||
async function activateSkillWhenVtUnavailable(ctx: ActionCtx, skillId: Id<'skills'>) {
|
||||
const skill = await ctx.runQuery(internal.skills.getSkillByIdInternal, { skillId })
|
||||
if (!shouldActivateWhenVtUnavailable(skill)) return
|
||||
|
||||
await ctx.runMutation(internal.skills.setSkillModerationStatusActiveInternal, { skillId })
|
||||
}
|
||||
|
||||
export const fetchResults = action({
|
||||
args: {
|
||||
sha256hash: v.optional(v.string()),
|
||||
@@ -305,7 +330,13 @@ export const scanWithVirusTotal = internalAction({
|
||||
handler: async (ctx, args) => {
|
||||
const apiKey = process.env.VT_API_KEY
|
||||
if (!apiKey) {
|
||||
console.log('VT_API_KEY not configured, skipping scan')
|
||||
console.log('VT_API_KEY not configured, skipping scan — activating skill')
|
||||
const version = await ctx.runQuery(internal.skills.getVersionByIdInternal, {
|
||||
versionId: args.versionId,
|
||||
})
|
||||
if (version) {
|
||||
await activateSkillWhenVtUnavailable(ctx, version.skillId)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
@@ -524,6 +555,7 @@ export const pollPendingScans = internalAction({
|
||||
versionId,
|
||||
vtAnalysis: { status: 'stale', checkedAt: Date.now() },
|
||||
})
|
||||
await activateSkillWhenVtUnavailable(ctx, skillId)
|
||||
staled++
|
||||
}
|
||||
continue
|
||||
@@ -549,6 +581,7 @@ export const pollPendingScans = internalAction({
|
||||
versionId,
|
||||
vtAnalysis: { status: 'stale', checkedAt: Date.now() },
|
||||
})
|
||||
await activateSkillWhenVtUnavailable(ctx, skillId)
|
||||
staled++
|
||||
}
|
||||
continue
|
||||
@@ -650,6 +683,10 @@ async function requestRescan(apiKey: string, sha256hash: string): Promise<boolea
|
||||
}
|
||||
}
|
||||
|
||||
export const __test = {
|
||||
shouldActivateWhenVtUnavailable,
|
||||
}
|
||||
|
||||
/**
|
||||
* Backfill function to process ALL pending skills at once
|
||||
* Run manually to clear backlog
|
||||
@@ -672,6 +709,8 @@ export const backfillPendingScans = internalAction({
|
||||
internal.skills.getPendingScanSkillsInternal,
|
||||
{
|
||||
limit: 10000,
|
||||
exhaustive: true,
|
||||
skipRecentMinutes: 0,
|
||||
},
|
||||
)
|
||||
|
||||
@@ -787,7 +826,7 @@ export const rescanActiveSkills = internalAction({
|
||||
`[vt:rescan] Processing batch of ${batch.skills.length} skills (cursor=${cursor}, accumulated=${accTotal})`,
|
||||
)
|
||||
|
||||
for (const { versionId, sha256hash, slug } of batch.skills) {
|
||||
for (const { versionId, sha256hash, slug, wasFlagged } of batch.skills) {
|
||||
try {
|
||||
const vtResult = await checkExistingFile(apiKey, sha256hash)
|
||||
|
||||
@@ -834,6 +873,15 @@ export const rescanActiveSkills = internalAction({
|
||||
status,
|
||||
})
|
||||
accUpdated++
|
||||
} else if (wasFlagged && status === 'clean') {
|
||||
// Verdict improved from suspicious → clean: clear the stale moderation flag
|
||||
console.log(`[vt:rescan] ${slug}: verdict improved to clean, clearing suspicious flag`)
|
||||
await ctx.runMutation(internal.skills.approveSkillByHashInternal, {
|
||||
sha256hash,
|
||||
scanner: 'vt',
|
||||
status,
|
||||
})
|
||||
accUpdated++
|
||||
} else {
|
||||
accUnchanged++
|
||||
}
|
||||
|
||||
+31
-2
@@ -18,12 +18,41 @@ OpenAPI: `/api/v1/openapi.json`
|
||||
|
||||
## Rate limits
|
||||
|
||||
Per IP + per API key:
|
||||
Auth-aware enforcement:
|
||||
|
||||
- Anonymous requests: per IP.
|
||||
- Authenticated requests (valid Bearer token): per user bucket.
|
||||
- Missing/invalid token falls back to IP enforcement.
|
||||
|
||||
- Read: 120/min per IP, 600/min per key
|
||||
- Write: 30/min per IP, 120/min per key
|
||||
|
||||
Headers: `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`, `Retry-After` (on 429).
|
||||
Headers: `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`, `RateLimit-Limit`, `RateLimit-Remaining`, `RateLimit-Reset`, `Retry-After` (on 429).
|
||||
|
||||
Semantics:
|
||||
|
||||
- `X-RateLimit-Reset`: Unix epoch seconds (absolute reset time)
|
||||
- `RateLimit-Reset`: delay seconds until reset
|
||||
- `Retry-After`: delay seconds to wait on `429`
|
||||
|
||||
Example `429`:
|
||||
|
||||
```http
|
||||
HTTP/2 429
|
||||
x-ratelimit-limit: 20
|
||||
x-ratelimit-remaining: 0
|
||||
x-ratelimit-reset: 1771404540
|
||||
ratelimit-limit: 20
|
||||
ratelimit-remaining: 0
|
||||
ratelimit-reset: 34
|
||||
retry-after: 34
|
||||
```
|
||||
|
||||
Client handling:
|
||||
|
||||
- Prefer `Retry-After` when present.
|
||||
- Otherwise use `RateLimit-Reset` or derive delay from `X-RateLimit-Reset`.
|
||||
- Add jitter to retries.
|
||||
|
||||
## Endpoints
|
||||
|
||||
|
||||
@@ -10,6 +10,7 @@ read_when:
|
||||
## Web auth (GitHub OAuth)
|
||||
|
||||
- Convex Auth + GitHub OAuth App.
|
||||
- GitHub is the only supported login provider.
|
||||
- Env vars:
|
||||
- `AUTH_GITHUB_ID`
|
||||
- `AUTH_GITHUB_SECRET`
|
||||
|
||||
+38
-4
@@ -29,6 +29,34 @@ Env equivalents:
|
||||
- `CLAWHUB_REGISTRY` (legacy `CLAWDHUB_REGISTRY`)
|
||||
- `CLAWHUB_WORKDIR` (legacy `CLAWDHUB_WORKDIR`)
|
||||
|
||||
### HTTP proxy
|
||||
|
||||
The CLI respects standard HTTP proxy environment variables for systems behind
|
||||
corporate proxies or restricted networks:
|
||||
|
||||
- `HTTPS_PROXY` / `https_proxy`
|
||||
- `HTTP_PROXY` / `http_proxy`
|
||||
- `NO_PROXY` / `no_proxy`
|
||||
|
||||
When any of these variables is set, the CLI routes outbound requests through
|
||||
the specified proxy. `HTTPS_PROXY` is used for HTTPS requests, `HTTP_PROXY`
|
||||
for plain HTTP. `NO_PROXY` / `no_proxy` is respected to bypass the proxy for
|
||||
specific hosts or domains.
|
||||
|
||||
This is required on systems where direct outbound connections are blocked
|
||||
(e.g. Docker containers, Hetzner VPS with proxy-only internet, corporate
|
||||
firewalls).
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
export HTTPS_PROXY=http://proxy.example.com:3128
|
||||
export NO_PROXY=localhost,127.0.0.1
|
||||
clawhub search "my query"
|
||||
```
|
||||
|
||||
When no proxy variable is set, behavior is unchanged (direct connections).
|
||||
|
||||
## Config file
|
||||
|
||||
Stores your API token + cached registry URL.
|
||||
@@ -86,6 +114,12 @@ Stores your API token + cached registry URL.
|
||||
- `<workdir>/.clawhub/lock.json` (legacy `.clawdhub`)
|
||||
- `<skill>/.clawhub/origin.json` (legacy `.clawdhub`)
|
||||
|
||||
### `uninstall <slug>`
|
||||
|
||||
- Removes `<workdir>/<dir>/<slug>` and deletes the lockfile entry.
|
||||
- Interactive: asks for confirmation.
|
||||
- Non-interactive (`--no-input`): requires `--yes`.
|
||||
|
||||
### `list`
|
||||
|
||||
- Reads `<workdir>/.clawhub/lock.json` (legacy `.clawdhub`).
|
||||
@@ -105,24 +139,24 @@ Stores your API token + cached registry URL.
|
||||
|
||||
### `delete <slug>`
|
||||
|
||||
- Soft-delete a skill (moderator/admin only).
|
||||
- Soft-delete a skill (owner, moderator, or admin).
|
||||
- Calls `DELETE /api/v1/skills/{slug}`.
|
||||
- `--yes` skips confirmation.
|
||||
|
||||
### `undelete <slug>`
|
||||
|
||||
- Restore a hidden skill (moderator/admin only).
|
||||
- Restore a hidden skill (owner, moderator, or admin).
|
||||
- Calls `POST /api/v1/skills/{slug}/undelete`.
|
||||
- `--yes` skips confirmation.
|
||||
|
||||
### `hide <slug>`
|
||||
|
||||
- Hide a skill (moderator/admin only).
|
||||
- Hide a skill (owner, moderator, or admin).
|
||||
- Alias for `delete`.
|
||||
|
||||
### `unhide <slug>`
|
||||
|
||||
- Unhide a skill (moderator/admin only).
|
||||
- Unhide a skill (owner, moderator, or admin).
|
||||
- Alias for `undelete`.
|
||||
|
||||
### `ban-user <handleOrId>`
|
||||
|
||||
@@ -78,3 +78,21 @@ Then:
|
||||
clawhub login --site https://<site>
|
||||
clawhub whoami
|
||||
```
|
||||
|
||||
Rate-limit sanity checks:
|
||||
|
||||
```bash
|
||||
curl -i "https://<site>/api/v1/download?slug=gifgrep"
|
||||
```
|
||||
|
||||
Confirm headers are present:
|
||||
|
||||
- `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`
|
||||
- `RateLimit-Limit`, `RateLimit-Remaining`, `RateLimit-Reset`
|
||||
- `Retry-After` on `429`
|
||||
|
||||
Proxy/IP caveat:
|
||||
|
||||
- Default IP source is `cf-connecting-ip`.
|
||||
- For non-Cloudflare trusted proxy setups, set `TRUST_FORWARDED_IPS=true`.
|
||||
- If proxy headers are not forwarded/trusted correctly, multiple users may collapse into one IP and hit false-positive rate limits.
|
||||
|
||||
+48
-5
@@ -15,7 +15,11 @@ OpenAPI: `/api/v1/openapi.json`.
|
||||
|
||||
## Rate limits
|
||||
|
||||
Enforced per IP + per API key:
|
||||
Enforcement model:
|
||||
|
||||
- Anonymous requests: enforced per IP.
|
||||
- Authenticated requests (valid Bearer token): enforced per user bucket.
|
||||
- If token is missing/invalid, behavior falls back to IP enforcement.
|
||||
|
||||
- Read: 120/min per IP, 600/min per key
|
||||
- Write: 30/min per IP, 120/min per key
|
||||
@@ -23,12 +27,43 @@ Enforced per IP + per API key:
|
||||
|
||||
Headers:
|
||||
|
||||
- `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`, `Retry-After` (when limited)
|
||||
- Legacy compatibility: `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`
|
||||
- Standardized: `RateLimit-Limit`, `RateLimit-Remaining`, `RateLimit-Reset`
|
||||
- On `429`: `Retry-After`
|
||||
|
||||
Header semantics:
|
||||
|
||||
- `X-RateLimit-Reset`: absolute Unix epoch seconds
|
||||
- `RateLimit-Reset`: seconds until reset (delay)
|
||||
- `Retry-After`: seconds to wait before retry (delay) on `429`
|
||||
|
||||
Example `429` response:
|
||||
|
||||
```http
|
||||
HTTP/2 429
|
||||
content-type: text/plain; charset=utf-8
|
||||
x-ratelimit-limit: 20
|
||||
x-ratelimit-remaining: 0
|
||||
x-ratelimit-reset: 1771404540
|
||||
ratelimit-limit: 20
|
||||
ratelimit-remaining: 0
|
||||
ratelimit-reset: 34
|
||||
retry-after: 34
|
||||
|
||||
Rate limit exceeded
|
||||
```
|
||||
|
||||
Client guidance:
|
||||
|
||||
- If `Retry-After` exists, wait that many seconds before retry.
|
||||
- Use jittered backoff to avoid synchronized retries.
|
||||
- If `Retry-After` is missing, fallback to `RateLimit-Reset` (or compute from `X-RateLimit-Reset`).
|
||||
|
||||
IP source:
|
||||
|
||||
- Uses `cf-connecting-ip` first, then falls back to `x-real-ip`, `x-forwarded-for`, or `fly-client-ip`.
|
||||
- Set `TRUST_FORWARDED_IPS=false` to disable forwarded-header fallback.
|
||||
- Uses `cf-connecting-ip` (Cloudflare) for client IP by default.
|
||||
- Set `TRUST_FORWARDED_IPS=true` to opt in to `x-forwarded-for`, `x-real-ip`, or `fly-client-ip` (non-Cloudflare deployments).
|
||||
- If you run behind a reverse proxy/load balancer, ensure real client IP headers are preserved and trusted correctly, or rate limits may be too strict due to shared proxy IPs.
|
||||
|
||||
## Public endpoints (no auth)
|
||||
|
||||
@@ -154,7 +189,15 @@ Publishes a new version.
|
||||
|
||||
### `DELETE /api/v1/skills/{slug}` / `POST /api/v1/skills/{slug}/undelete`
|
||||
|
||||
Soft-delete / restore a skill (moderator/admin only).
|
||||
Soft-delete / restore a skill (owner, moderator, or admin).
|
||||
|
||||
Status codes:
|
||||
|
||||
- `200`: ok
|
||||
- `401`: unauthorized
|
||||
- `403`: forbidden
|
||||
- `404`: skill/user not found
|
||||
- `500`: internal server error
|
||||
|
||||
### `POST /api/v1/users/ban`
|
||||
|
||||
|
||||
@@ -62,6 +62,7 @@ Install a skill into `./skills/<slug>` (if Clawdbot is configured, installs into
|
||||
```bash
|
||||
bun clawhub install <slug>
|
||||
bun clawhub list
|
||||
bun clawhub uninstall <slug> --yes
|
||||
```
|
||||
|
||||
You can also install into any folder:
|
||||
|
||||
@@ -62,7 +62,6 @@ read_when:
|
||||
- Lookup uses GitHub `created_at` fetched by the immutable GitHub numeric ID (`providerAccountId`)
|
||||
and caches on the user:
|
||||
- `githubCreatedAt` (source of truth)
|
||||
- `githubFetchedAt` (fetch timestamp; set when `githubCreatedAt` is populated)
|
||||
- Gate applies to web uploads, CLI publish, and GitHub import.
|
||||
- If GitHub responds `403` or `429`, publish fails with:
|
||||
- `GitHub API rate limit exceeded — please try again in a few minutes`
|
||||
|
||||
@@ -18,6 +18,35 @@ read_when:
|
||||
- Token missing or revoked: check your config file (`CLAWHUB_CONFIG_PATH` override?).
|
||||
- Ensure requests include `Authorization: Bearer ...` (CLI does this automatically).
|
||||
|
||||
## CLI/API returns `Rate limit exceeded` (429)
|
||||
|
||||
- Read headers in the response:
|
||||
- `Retry-After` = wait seconds before retry
|
||||
- `RateLimit-Remaining` + `RateLimit-Limit` = current budget
|
||||
- `RateLimit-Reset` (or `X-RateLimit-Reset`) = reset timing
|
||||
- The CLI now includes retry hints in 429 errors (retry delay + remaining budget).
|
||||
- If many users share one egress IP (NAT/proxy), IP limit can be hit even with valid tokens.
|
||||
- For non-Cloudflare deploys behind trusted proxies, set `TRUST_FORWARDED_IPS=true` so forwarded client IPs can be used.
|
||||
|
||||
## `search` / `install` fails with `fetch failed` behind a proxy
|
||||
|
||||
If your system requires an HTTP proxy for outbound connections (e.g. corporate
|
||||
firewalls, Docker containers with proxy-only internet, Hetzner VPS), the CLI
|
||||
will fail with:
|
||||
|
||||
```
|
||||
✖ fetch failed
|
||||
Error: fetch failed
|
||||
```
|
||||
|
||||
**Fix:** Set the standard proxy environment variables:
|
||||
|
||||
```bash
|
||||
export HTTPS_PROXY=http://proxy.example.com:3128
|
||||
clawhub search "my query"
|
||||
```
|
||||
|
||||
The CLI respects `HTTPS_PROXY`, `HTTP_PROXY`, `https_proxy`, and `http_proxy`.
|
||||
## `publish` fails with `OPENAI_API_KEY is not configured`
|
||||
|
||||
- Set `OPENAI_API_KEY` in the Convex environment (not only locally).
|
||||
|
||||
@@ -504,4 +504,49 @@ describe('clawhub e2e', () => {
|
||||
await rm(cfg.dir, { recursive: true, force: true })
|
||||
}
|
||||
}, 180_000)
|
||||
|
||||
it('delete returns proper error for non-existent skill', async () => {
|
||||
const registry = process.env.CLAWDHUB_REGISTRY?.trim() || 'https://clawdhub.com'
|
||||
const site = process.env.CLAWDHUB_SITE?.trim() || 'https://clawdhub.com'
|
||||
const token = mustGetToken() ?? (await readGlobalConfig())?.token ?? null
|
||||
if (!token) {
|
||||
throw new Error('Missing token. Set CLAWDHUB_E2E_TOKEN or run: bun clawdhub auth login')
|
||||
}
|
||||
|
||||
const cfg = await makeTempConfig(registry, token)
|
||||
const workdir = await mkdtemp(join(tmpdir(), 'clawdhub-e2e-delete-'))
|
||||
const nonExistentSlug = `non-existent-skill-${Date.now()}`
|
||||
|
||||
try {
|
||||
const del = spawnSync(
|
||||
'bun',
|
||||
[
|
||||
'clawdhub',
|
||||
'delete',
|
||||
nonExistentSlug,
|
||||
'--yes',
|
||||
'--site',
|
||||
site,
|
||||
'--registry',
|
||||
registry,
|
||||
'--workdir',
|
||||
workdir,
|
||||
],
|
||||
{
|
||||
cwd: process.cwd(),
|
||||
env: { ...process.env, CLAWDHUB_CONFIG_PATH: cfg.path, CLAWDHUB_DISABLE_TELEMETRY: '1' },
|
||||
encoding: 'utf8',
|
||||
},
|
||||
)
|
||||
// Should fail with non-zero exit code
|
||||
expect(del.status).not.toBe(0)
|
||||
// Error should mention "not found" - not generic "Unauthorized"
|
||||
const output = (del.stdout + del.stderr).toLowerCase()
|
||||
expect(output).toMatch(/not found|404|does not exist/i)
|
||||
expect(output).not.toMatch(/unauthorized/i)
|
||||
} finally {
|
||||
await rm(workdir, { recursive: true, force: true })
|
||||
await rm(cfg.dir, { recursive: true, force: true })
|
||||
}
|
||||
}, 30_000)
|
||||
})
|
||||
|
||||
@@ -42,8 +42,8 @@ test('header menu routes render', async ({ page }) => {
|
||||
}
|
||||
|
||||
if (label === 'Search') {
|
||||
await expect(page).toHaveURL(/\/?(\?|$)/)
|
||||
await expect(page.locator('h1', { hasText: 'ClawHub' })).toBeVisible()
|
||||
await expect(page).toHaveURL(/\/skills(\?|$)/)
|
||||
await expect(page.locator('h1', { hasText: 'Skills' })).toBeVisible()
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "clawhub",
|
||||
"version": "0.6.1",
|
||||
"version": "0.7.0",
|
||||
"description": "ClawHub CLI \\u2014 install, update, search, and publish agent skills.",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
|
||||
@@ -14,7 +14,14 @@ import {
|
||||
import { cmdInspect } from './cli/commands/inspect.js'
|
||||
import { cmdBanUser, cmdSetRole } from './cli/commands/moderation.js'
|
||||
import { cmdPublish } from './cli/commands/publish.js'
|
||||
import { cmdExplore, cmdInstall, cmdList, cmdSearch, cmdUpdate } from './cli/commands/skills.js'
|
||||
import {
|
||||
cmdExplore,
|
||||
cmdInstall,
|
||||
cmdList,
|
||||
cmdSearch,
|
||||
cmdUninstall,
|
||||
cmdUpdate,
|
||||
} from './cli/commands/skills.js'
|
||||
import { cmdStarSkill } from './cli/commands/star.js'
|
||||
import { cmdSync } from './cli/commands/sync.js'
|
||||
import { cmdUnstarSkill } from './cli/commands/unstar.js'
|
||||
@@ -198,6 +205,16 @@ program
|
||||
await cmdUpdate(opts, slug, options, isInputAllowed())
|
||||
})
|
||||
|
||||
program
|
||||
.command('uninstall')
|
||||
.description('Uninstall a skill')
|
||||
.argument('<slug>', 'Skill slug')
|
||||
.option('--yes', 'Skip confirmation')
|
||||
.action(async (slug, options) => {
|
||||
const opts = await resolveGlobalOpts()
|
||||
await cmdUninstall(opts, slug, options, isInputAllowed())
|
||||
})
|
||||
|
||||
program
|
||||
.command('list')
|
||||
.description('List installed skills (from lockfile)')
|
||||
|
||||
@@ -3,6 +3,7 @@ import { mkdir, mkdtemp, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join, resolve } from 'node:path'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import { resolveHome } from '../homedir.js'
|
||||
import { resolveClawdbotDefaultWorkspace, resolveClawdbotSkillRoots } from './clawdbotConfig.js'
|
||||
|
||||
const originalEnv = { ...process.env }
|
||||
@@ -177,6 +178,40 @@ describe('resolveClawdbotSkillRoots', () => {
|
||||
expect(labels[resolve(openclawStateDir, 'skills')]).toBe('OpenClaw: Shared skills')
|
||||
})
|
||||
|
||||
it('uses $HOME over os.homedir() for tilde expansion', async () => {
|
||||
const base = await mkdtemp(join(tmpdir(), 'clawhub-home-override-'))
|
||||
const customHome = join(base, 'custom-home')
|
||||
const stateDir = join(base, 'state')
|
||||
const configPath = join(base, 'clawdbot.json')
|
||||
const openclawStateDir = join(base, 'openclaw-state')
|
||||
|
||||
process.env.HOME = customHome
|
||||
process.env.CLAWDBOT_STATE_DIR = stateDir
|
||||
process.env.CLAWDBOT_CONFIG_PATH = configPath
|
||||
process.env.OPENCLAW_STATE_DIR = openclawStateDir
|
||||
process.env.OPENCLAW_CONFIG_PATH = join(openclawStateDir, 'openclaw.json')
|
||||
|
||||
const config = `{
|
||||
agents: {
|
||||
defaults: { workspace: "~/my-workspace" },
|
||||
},
|
||||
}`
|
||||
await writeFile(configPath, config, 'utf8')
|
||||
|
||||
const workspace = await resolveClawdbotDefaultWorkspace()
|
||||
expect(workspace).toBe(resolve(customHome, 'my-workspace'))
|
||||
expect(resolveHome()).toBe(customHome)
|
||||
})
|
||||
|
||||
it('normalizes trailing separators in $HOME', async () => {
|
||||
const base = await mkdtemp(join(tmpdir(), 'clawhub-home-trailing-'))
|
||||
const customHome = join(base, 'custom-home')
|
||||
|
||||
process.env.HOME = `${customHome}/`
|
||||
|
||||
expect(resolveHome()).toBe(customHome)
|
||||
})
|
||||
|
||||
it('supports OpenClaw configuration files', async () => {
|
||||
const base = await mkdtemp(join(tmpdir(), 'clawhub-openclaw-'))
|
||||
const stateDir = join(base, 'openclaw-state')
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { readFile } from 'node:fs/promises'
|
||||
import { homedir } from 'node:os'
|
||||
import { basename, join, resolve } from 'node:path'
|
||||
import JSON5 from 'json5'
|
||||
import { resolveHome } from '../homedir.js'
|
||||
|
||||
type ClawdbotConfig = {
|
||||
agent?: { workspace?: string }
|
||||
@@ -95,7 +95,7 @@ export async function resolveClawdbotDefaultWorkspace(): Promise<string | null>
|
||||
function resolveClawdbotStateDir() {
|
||||
const override = process.env.CLAWDBOT_STATE_DIR?.trim()
|
||||
if (override) return resolveUserPath(override)
|
||||
return join(homedir(), '.clawdbot')
|
||||
return join(resolveHome(), '.clawdbot')
|
||||
}
|
||||
|
||||
function resolveClawdbotConfigPath() {
|
||||
@@ -107,7 +107,7 @@ function resolveClawdbotConfigPath() {
|
||||
function resolveOpenclawStateDir() {
|
||||
const override = process.env.OPENCLAW_STATE_DIR?.trim()
|
||||
if (override) return resolveUserPath(override)
|
||||
return join(homedir(), '.openclaw')
|
||||
return join(resolveHome(), '.openclaw')
|
||||
}
|
||||
|
||||
function resolveOpenclawConfigPath() {
|
||||
@@ -120,7 +120,7 @@ function resolveUserPath(input: string) {
|
||||
const trimmed = input.trim()
|
||||
if (!trimmed) return ''
|
||||
if (trimmed.startsWith('~')) {
|
||||
return resolve(trimmed.replace(/^~(?=$|[\\/])/, homedir()))
|
||||
return resolve(trimmed.replace(/^~(?=$|[\\/])/, resolveHome()))
|
||||
}
|
||||
return resolve(trimmed)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
/* @vitest-environment node */
|
||||
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import type { GlobalOpts } from '../types'
|
||||
|
||||
const mockReadGlobalConfig = vi.fn(async () => null as { registry?: string; token?: string } | null)
|
||||
const mockWriteGlobalConfig = vi.fn(async (_cfg: unknown) => {})
|
||||
vi.mock('../../config.js', () => ({
|
||||
readGlobalConfig: () => mockReadGlobalConfig(),
|
||||
writeGlobalConfig: (cfg: unknown) => mockWriteGlobalConfig(cfg),
|
||||
}))
|
||||
|
||||
const mockGetRegistry = vi.fn(async () => 'https://clawhub.ai')
|
||||
vi.mock('../registry.js', () => ({
|
||||
getRegistry: () => mockGetRegistry(),
|
||||
}))
|
||||
|
||||
const { cmdLogout } = await import('./auth')
|
||||
|
||||
const mockLog = vi.spyOn(console, 'log').mockImplementation(() => {})
|
||||
|
||||
function makeOpts(): GlobalOpts {
|
||||
return {
|
||||
workdir: '/work',
|
||||
dir: '/work/skills',
|
||||
site: 'https://clawhub.ai',
|
||||
registry: 'https://clawhub.ai',
|
||||
registrySource: 'default',
|
||||
}
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
vi.clearAllMocks()
|
||||
mockLog.mockClear()
|
||||
})
|
||||
|
||||
describe('cmdLogout', () => {
|
||||
it('removes token and logs a clear message', async () => {
|
||||
mockReadGlobalConfig.mockResolvedValueOnce({ registry: 'https://clawhub.ai', token: 'tkn' })
|
||||
|
||||
await cmdLogout(makeOpts())
|
||||
|
||||
expect(mockWriteGlobalConfig).toHaveBeenCalledWith({
|
||||
registry: 'https://clawhub.ai',
|
||||
token: undefined,
|
||||
})
|
||||
expect(mockGetRegistry).not.toHaveBeenCalled()
|
||||
expect(mockLog).toHaveBeenCalledWith(
|
||||
'OK. Logged out locally. Token still valid until revoked (Settings -> API tokens).',
|
||||
)
|
||||
})
|
||||
|
||||
it('falls back to resolved registry when config has no registry', async () => {
|
||||
mockReadGlobalConfig.mockResolvedValueOnce({ token: 'tkn' })
|
||||
mockGetRegistry.mockResolvedValueOnce('https://registry.example')
|
||||
|
||||
await cmdLogout(makeOpts())
|
||||
|
||||
expect(mockGetRegistry).toHaveBeenCalled()
|
||||
expect(mockWriteGlobalConfig).toHaveBeenCalledWith({
|
||||
registry: 'https://registry.example',
|
||||
token: undefined,
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -75,7 +75,7 @@ export async function cmdLogout(opts: GlobalOpts) {
|
||||
const cfg = await readGlobalConfig()
|
||||
const registry = cfg?.registry || (await getRegistry(opts, { cache: true }))
|
||||
await writeGlobalConfig({ registry, token: undefined })
|
||||
console.log('OK. Logged out.')
|
||||
console.log('OK. Logged out locally. Token still valid until revoked (Settings -> API tokens).')
|
||||
}
|
||||
|
||||
export async function cmdWhoami(opts: GlobalOpts) {
|
||||
|
||||
@@ -29,14 +29,16 @@ const mockSpinner = {
|
||||
isSpinning: false,
|
||||
text: '',
|
||||
}
|
||||
const mockIsInteractive = vi.fn(() => false)
|
||||
const mockPromptConfirm = vi.fn(async () => false)
|
||||
vi.mock('../ui.js', () => ({
|
||||
createSpinner: vi.fn(() => mockSpinner),
|
||||
fail: (message: string) => {
|
||||
throw new Error(message)
|
||||
},
|
||||
formatError: (error: unknown) => (error instanceof Error ? error.message : String(error)),
|
||||
isInteractive: () => false,
|
||||
promptConfirm: vi.fn(async () => false),
|
||||
isInteractive: mockIsInteractive,
|
||||
promptConfirm: mockPromptConfirm,
|
||||
}))
|
||||
|
||||
vi.mock('../../skills.js', () => ({
|
||||
@@ -55,7 +57,7 @@ vi.mock('node:fs/promises', () => ({
|
||||
stat: vi.fn(),
|
||||
}))
|
||||
|
||||
const { clampLimit, cmdExplore, cmdInstall, cmdUpdate, formatExploreLine } = await import('./skills')
|
||||
const { clampLimit, cmdExplore, cmdInstall, cmdUninstall, cmdUpdate, formatExploreLine } = await import('./skills')
|
||||
const {
|
||||
extractZipToDir,
|
||||
hashSkillFiles,
|
||||
@@ -220,3 +222,143 @@ describe('cmdInstall', () => {
|
||||
expect(zipArgs?.token).toBe('tkn')
|
||||
})
|
||||
})
|
||||
|
||||
describe('cmdUninstall', () => {
|
||||
it('requires --yes when input is disabled', async () => {
|
||||
vi.mocked(readLockfile).mockResolvedValue({
|
||||
version: 1,
|
||||
skills: { demo: { version: '1.0.0', installedAt: 123 } },
|
||||
})
|
||||
|
||||
await expect(cmdUninstall(makeOpts(), 'demo', {}, false)).rejects.toThrow(/--yes/i)
|
||||
})
|
||||
|
||||
it('prompts when interactive and proceeds on confirm', async () => {
|
||||
vi.mocked(readLockfile).mockResolvedValue({
|
||||
version: 1,
|
||||
skills: { demo: { version: '1.0.0', installedAt: 123 } },
|
||||
})
|
||||
vi.mocked(writeLockfile).mockResolvedValue()
|
||||
vi.mocked(rm).mockResolvedValue()
|
||||
mockIsInteractive.mockReturnValue(true)
|
||||
mockPromptConfirm.mockResolvedValue(true)
|
||||
|
||||
await cmdUninstall(makeOpts(), 'demo', {}, true)
|
||||
|
||||
expect(mockPromptConfirm).toHaveBeenCalledWith('Uninstall demo?')
|
||||
expect(rm).toHaveBeenCalledWith('/work/skills/demo', { recursive: true, force: true })
|
||||
expect(writeLockfile).toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('prints Cancelled and does not remove when prompt declines', async () => {
|
||||
vi.mocked(readLockfile).mockResolvedValue({
|
||||
version: 1,
|
||||
skills: { demo: { version: '1.0.0', installedAt: 123 } },
|
||||
})
|
||||
mockIsInteractive.mockReturnValue(true)
|
||||
mockPromptConfirm.mockResolvedValue(false)
|
||||
|
||||
await cmdUninstall(makeOpts(), 'demo', {}, true)
|
||||
|
||||
expect(mockLog).toHaveBeenCalledWith('Cancelled.')
|
||||
expect(rm).not.toHaveBeenCalled()
|
||||
expect(writeLockfile).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects unsafe slugs', async () => {
|
||||
await expect(cmdUninstall(makeOpts(), '../evil', { yes: true }, false)).rejects.toThrow(
|
||||
/invalid slug/i,
|
||||
)
|
||||
await expect(cmdUninstall(makeOpts(), 'demo/evil', { yes: true }, false)).rejects.toThrow(
|
||||
/invalid slug/i,
|
||||
)
|
||||
})
|
||||
|
||||
it('fails when skill is not installed', async () => {
|
||||
vi.mocked(readLockfile).mockResolvedValue({ version: 1, skills: {} })
|
||||
|
||||
await expect(cmdUninstall(makeOpts(), 'missing', {}, false)).rejects.toThrow(
|
||||
'Not installed: missing',
|
||||
)
|
||||
})
|
||||
|
||||
it('removes skill directory and lockfile entry with --yes flag', async () => {
|
||||
vi.mocked(readLockfile).mockResolvedValue({
|
||||
version: 1,
|
||||
skills: { demo: { version: '1.0.0', installedAt: 123 } },
|
||||
})
|
||||
vi.mocked(writeLockfile).mockResolvedValue()
|
||||
vi.mocked(rm).mockResolvedValue()
|
||||
|
||||
await cmdUninstall(makeOpts(), 'demo', { yes: true }, false)
|
||||
|
||||
expect(rm).toHaveBeenCalledWith('/work/skills/demo', { recursive: true, force: true })
|
||||
expect(writeLockfile).toHaveBeenCalledWith('/work', {
|
||||
version: 1,
|
||||
skills: {},
|
||||
})
|
||||
expect(mockSpinner.succeed).toHaveBeenCalledWith('Uninstalled demo')
|
||||
})
|
||||
|
||||
it('does not update lockfile if remove fails', async () => {
|
||||
vi.mocked(readLockfile).mockResolvedValue({
|
||||
version: 1,
|
||||
skills: { demo: { version: '1.0.0', installedAt: 123 } },
|
||||
})
|
||||
vi.mocked(rm).mockRejectedValue(new Error('nope'))
|
||||
|
||||
await expect(cmdUninstall(makeOpts(), 'demo', { yes: true }, false)).rejects.toThrow('nope')
|
||||
|
||||
expect(writeLockfile).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('updates lockfile after removing directory', async () => {
|
||||
vi.mocked(readLockfile).mockResolvedValue({
|
||||
version: 1,
|
||||
skills: { demo: { version: '1.0.0', installedAt: 123 } },
|
||||
})
|
||||
vi.mocked(writeLockfile).mockResolvedValue()
|
||||
vi.mocked(rm).mockResolvedValue()
|
||||
|
||||
await cmdUninstall(makeOpts(), 'demo', { yes: true }, false)
|
||||
|
||||
const rmMock = vi.mocked(rm)
|
||||
const writeLockfileMock = vi.mocked(writeLockfile)
|
||||
expect(rmMock.mock.invocationCallOrder[0]).toBeLessThan(
|
||||
writeLockfileMock.mock.invocationCallOrder[0],
|
||||
)
|
||||
})
|
||||
|
||||
it('removes skill and updates lockfile keeping other skills', async () => {
|
||||
vi.mocked(readLockfile).mockResolvedValue({
|
||||
version: 1,
|
||||
skills: {
|
||||
demo: { version: '1.0.0', installedAt: 123 },
|
||||
other: { version: '2.0.0', installedAt: 456 },
|
||||
},
|
||||
})
|
||||
vi.mocked(writeLockfile).mockResolvedValue()
|
||||
vi.mocked(rm).mockResolvedValue()
|
||||
|
||||
await cmdUninstall(makeOpts(), 'demo', { yes: true }, false)
|
||||
|
||||
expect(rm).toHaveBeenCalledWith('/work/skills/demo', { recursive: true, force: true })
|
||||
expect(writeLockfile).toHaveBeenCalledWith('/work', {
|
||||
version: 1,
|
||||
skills: { other: { version: '2.0.0', installedAt: 456 } },
|
||||
})
|
||||
})
|
||||
|
||||
it('trims slug whitespace', async () => {
|
||||
vi.mocked(readLockfile).mockResolvedValue({
|
||||
version: 1,
|
||||
skills: { demo: { version: '1.0.0', installedAt: 123 } },
|
||||
})
|
||||
vi.mocked(writeLockfile).mockResolvedValue()
|
||||
vi.mocked(rm).mockResolvedValue()
|
||||
|
||||
await cmdUninstall(makeOpts(), ' demo ', { yes: true }, false)
|
||||
|
||||
expect(rm).toHaveBeenCalledWith('/work/skills/demo', { recursive: true, force: true })
|
||||
})
|
||||
})
|
||||
|
||||
@@ -23,6 +23,20 @@ import type { GlobalOpts, ResolveResult } from '../types.js'
|
||||
import { createSpinner, fail, formatError, isInteractive, promptConfirm } from '../ui.js'
|
||||
import { getOptionalAuthToken } from '../authToken.js'
|
||||
|
||||
function normalizeSkillSlugOrFail(raw: string) {
|
||||
const slug = raw.trim()
|
||||
if (!slug) fail('Slug required')
|
||||
// Safety: never allow path traversal or nested paths to become filesystem operations.
|
||||
if (slug.includes('/') || slug.includes('\\') || slug.includes('..')) {
|
||||
fail(`Invalid slug: ${slug}`)
|
||||
}
|
||||
return slug
|
||||
}
|
||||
|
||||
function isSafeSkillSlug(slug: string) {
|
||||
return Boolean(slug) && !slug.includes('/') && !slug.includes('\\') && !slug.includes('..')
|
||||
}
|
||||
|
||||
export async function cmdSearch(opts: GlobalOpts, query: string, limit?: number) {
|
||||
if (!query) fail('Query required')
|
||||
|
||||
@@ -59,8 +73,7 @@ export async function cmdInstall(
|
||||
versionFlag?: string,
|
||||
force = false,
|
||||
) {
|
||||
const trimmed = slug.trim()
|
||||
if (!trimmed) fail('Slug required')
|
||||
const trimmed = normalizeSkillSlugOrFail(slug)
|
||||
|
||||
const token = await getOptionalAuthToken()
|
||||
|
||||
@@ -139,19 +152,19 @@ export async function cmdUpdate(
|
||||
options: { all?: boolean; version?: string; force?: boolean },
|
||||
inputAllowed: boolean,
|
||||
) {
|
||||
const slug = slugArg?.trim()
|
||||
const slug = slugArg ? normalizeSkillSlugOrFail(slugArg) : undefined
|
||||
const all = Boolean(options.all)
|
||||
if (!slug && !all) fail('Provide <slug> or --all')
|
||||
if (slug && all) fail('Use either <slug> or --all')
|
||||
if (options.version && !slug) fail('--version requires a single <slug>')
|
||||
if (options.version && !semver.valid(options.version)) fail('--version must be valid semver')
|
||||
const allowPrompt = isInteractive() && inputAllowed !== false
|
||||
const allowPrompt = isInteractive() && inputAllowed
|
||||
|
||||
const token = await getOptionalAuthToken()
|
||||
|
||||
const registry = await getRegistry(opts, { cache: true })
|
||||
const lock = await readLockfile(opts.workdir)
|
||||
const slugs = slug ? [slug] : Object.keys(lock.skills)
|
||||
const slugs = slug ? [slug] : Object.keys(lock.skills).filter(isSafeSkillSlug)
|
||||
if (slugs.length === 0) {
|
||||
console.log('No installed skills.')
|
||||
return
|
||||
@@ -295,6 +308,45 @@ export async function cmdList(opts: GlobalOpts) {
|
||||
}
|
||||
}
|
||||
|
||||
export async function cmdUninstall(
|
||||
opts: GlobalOpts,
|
||||
slug: string,
|
||||
options: { yes?: boolean } = {},
|
||||
inputAllowed: boolean,
|
||||
) {
|
||||
const trimmed = normalizeSkillSlugOrFail(slug)
|
||||
|
||||
const lock = await readLockfile(opts.workdir)
|
||||
if (!lock.skills[trimmed]) {
|
||||
fail(`Not installed: ${trimmed}`)
|
||||
}
|
||||
|
||||
const allowPrompt = isInteractive() && inputAllowed
|
||||
if (!options.yes) {
|
||||
if (!allowPrompt) fail('Pass --yes (no input)')
|
||||
const confirm = await promptConfirm(`Uninstall ${trimmed}?`)
|
||||
if (!confirm) {
|
||||
console.log('Cancelled.')
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
const spinner = createSpinner(`Uninstalling ${trimmed}`)
|
||||
try {
|
||||
const target = join(opts.dir, trimmed)
|
||||
|
||||
await rm(target, { recursive: true, force: true })
|
||||
|
||||
delete lock.skills[trimmed]
|
||||
await writeLockfile(opts.workdir, lock)
|
||||
|
||||
spinner.succeed(`Uninstalled ${trimmed}`)
|
||||
} catch (error) {
|
||||
spinner.fail(formatError(error))
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
type ExploreSort = 'newest' | 'downloads' | 'rating' | 'installs' | 'installsAllTime' | 'trending'
|
||||
type ApiExploreSort =
|
||||
| 'updated'
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { createHash } from 'node:crypto'
|
||||
import { realpath } from 'node:fs/promises'
|
||||
import { homedir } from 'node:os'
|
||||
import { resolve } from 'node:path'
|
||||
import { resolveHome } from '../../homedir.js'
|
||||
import { isCancel, multiselect } from '@clack/prompts'
|
||||
import semver from 'semver'
|
||||
import { apiRequest, downloadZip } from '../../http.js'
|
||||
@@ -338,7 +338,7 @@ export function printSection(title: string, body?: string) {
|
||||
}
|
||||
|
||||
function abbreviatePath(value: string) {
|
||||
const home = homedir()
|
||||
const home = resolveHome()
|
||||
if (value.startsWith(home)) return `~${value.slice(home.length)}`
|
||||
return value
|
||||
}
|
||||
@@ -348,7 +348,7 @@ function rootTelemetryId(value: string) {
|
||||
}
|
||||
|
||||
function formatRootLabel(value: string) {
|
||||
const home = homedir()
|
||||
const home = resolveHome()
|
||||
if (value === home) return '~'
|
||||
|
||||
const normalized = value.replaceAll('\\', '/')
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { readdir, stat } from 'node:fs/promises'
|
||||
import { homedir } from 'node:os'
|
||||
import { basename, join, resolve } from 'node:path'
|
||||
import { resolveHome } from '../homedir.js'
|
||||
import { sanitizeSlug, titleCase } from './slug.js'
|
||||
|
||||
export type SkillFolder = {
|
||||
@@ -30,7 +30,7 @@ export async function findSkillFolders(root: string): Promise<SkillFolder[]> {
|
||||
}
|
||||
|
||||
export function getFallbackSkillRoots(workdir: string) {
|
||||
const home = homedir()
|
||||
const home = resolveHome()
|
||||
const roots = [
|
||||
// adjacent repo installs
|
||||
resolve(workdir, '..', 'clawdis', 'skills'),
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { existsSync } from 'node:fs'
|
||||
import { chmod, mkdir, readFile, writeFile } from 'node:fs/promises'
|
||||
import { homedir } from 'node:os'
|
||||
import { dirname, join, resolve } from 'node:path'
|
||||
import { resolveHome } from './homedir.js'
|
||||
import { type GlobalConfig, GlobalConfigSchema, parseArk } from './schema/index.js'
|
||||
|
||||
/**
|
||||
@@ -27,7 +27,7 @@ export function getGlobalConfigPath() {
|
||||
process.env.CLAWHUB_CONFIG_PATH?.trim() ?? process.env.CLAWDHUB_CONFIG_PATH?.trim()
|
||||
if (override) return resolve(override)
|
||||
|
||||
const home = homedir()
|
||||
const home = resolveHome()
|
||||
|
||||
if (process.platform === 'darwin') {
|
||||
return resolveConfigPath(join(home, 'Library', 'Application Support'))
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
import { homedir } from 'node:os'
|
||||
import { win32 } from 'node:path'
|
||||
|
||||
/**
|
||||
* Resolve the user's home directory, preferring environment variables over
|
||||
* os.homedir(). On Linux, os.homedir() reads from /etc/passwd which can
|
||||
* return a stale path after a user rename (usermod -l). The $HOME env var
|
||||
* is set by the login process and reflects the current session.
|
||||
*/
|
||||
export function resolveHome(): string {
|
||||
if (process.platform === 'win32') {
|
||||
return normalizeHome(process.env.USERPROFILE) || normalizeHome(process.env.HOME) || homedir()
|
||||
}
|
||||
return normalizeHome(process.env.HOME) || homedir()
|
||||
}
|
||||
|
||||
function normalizeHome(value: string | undefined): string {
|
||||
const trimmed = value?.trim()
|
||||
if (!trimmed) return ''
|
||||
|
||||
if (process.platform === 'win32') {
|
||||
const root = win32.parse(trimmed).root
|
||||
if (trimmed === root) return trimmed
|
||||
return trimmed.replace(/[\\/]+$/, '')
|
||||
}
|
||||
|
||||
if (trimmed === '/') return '/'
|
||||
return trimmed.replace(/\/+$/, '')
|
||||
}
|
||||
@@ -0,0 +1,256 @@
|
||||
/* @vitest-environment node */
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const originalBunVersion = (process.versions as Record<string, string | undefined>).bun
|
||||
|
||||
function enableBunRuntime() {
|
||||
Object.defineProperty(process.versions, 'bun', {
|
||||
value: '1.2.3',
|
||||
configurable: true,
|
||||
})
|
||||
}
|
||||
|
||||
function restoreBunRuntime() {
|
||||
if (originalBunVersion === undefined) {
|
||||
Reflect.deleteProperty(process.versions, 'bun')
|
||||
return
|
||||
}
|
||||
Object.defineProperty(process.versions, 'bun', {
|
||||
value: originalBunVersion,
|
||||
configurable: true,
|
||||
})
|
||||
}
|
||||
|
||||
function mockImmediateTimeouts() {
|
||||
const setTimeoutMock = vi.fn((callback: () => void) => {
|
||||
callback()
|
||||
return 1 as unknown as ReturnType<typeof setTimeout>
|
||||
})
|
||||
const clearTimeoutMock = vi.fn()
|
||||
vi.stubGlobal('setTimeout', setTimeoutMock as unknown as typeof setTimeout)
|
||||
vi.stubGlobal('clearTimeout', clearTimeoutMock as typeof clearTimeout)
|
||||
return { setTimeoutMock, clearTimeoutMock }
|
||||
}
|
||||
|
||||
async function loadHttpModuleWithBunMocks(opts?: {
|
||||
spawnImpl?: ReturnType<typeof vi.fn>
|
||||
mkdtempValue?: string
|
||||
readFileValue?: Buffer | null
|
||||
}) {
|
||||
const spawnSync = opts?.spawnImpl ?? vi.fn()
|
||||
const mkdtemp = vi.fn(async () => opts?.mkdtempValue ?? '/tmp/clawhub-test')
|
||||
const rm = vi.fn(async () => undefined)
|
||||
const writeFile = vi.fn(async () => undefined)
|
||||
const readFile = vi.fn(async () => opts?.readFileValue ?? Buffer.from([1, 2, 3]))
|
||||
|
||||
vi.doMock('node:child_process', () => ({ spawnSync }))
|
||||
vi.doMock('node:fs/promises', () => ({ mkdtemp, rm, writeFile, readFile }))
|
||||
|
||||
const http = await import('./http')
|
||||
return { http, spawnSync, mkdtemp, rm, writeFile, readFile }
|
||||
}
|
||||
|
||||
describe('http bun runtime', () => {
|
||||
beforeEach(() => {
|
||||
vi.resetModules()
|
||||
vi.clearAllMocks()
|
||||
vi.unstubAllGlobals()
|
||||
enableBunRuntime()
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
restoreBunRuntime()
|
||||
vi.doUnmock('node:child_process')
|
||||
vi.doUnmock('node:fs/promises')
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
|
||||
it('uses curl for apiRequest GET and parses JSON', async () => {
|
||||
const spawnSync = vi.fn().mockReturnValue({
|
||||
status: 0,
|
||||
stdout: '{"ok":true}\n200',
|
||||
stderr: '',
|
||||
})
|
||||
const { http } = await loadHttpModuleWithBunMocks({ spawnImpl: spawnSync })
|
||||
const fetchMock = vi.fn()
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
|
||||
const result = await http.apiRequest<{ ok: boolean }>('https://registry.example', {
|
||||
method: 'GET',
|
||||
path: '/v1/ping',
|
||||
token: 'clh_token',
|
||||
})
|
||||
|
||||
expect(result).toEqual({ ok: true })
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
expect(spawnSync).toHaveBeenCalledTimes(1)
|
||||
const [, args] = spawnSync.mock.calls[0] as [string, string[]]
|
||||
expect(args).toContain('GET')
|
||||
expect(args).toContain('https://registry.example/v1/ping')
|
||||
expect(args).toContain('Accept: application/json')
|
||||
expect(args).toContain('Authorization: Bearer clh_token')
|
||||
})
|
||||
|
||||
it('uses curl for apiRequest POST with json body', async () => {
|
||||
const spawnSync = vi.fn().mockReturnValue({
|
||||
status: 0,
|
||||
stdout: '{"ok":true}\n200',
|
||||
stderr: '',
|
||||
})
|
||||
const { http } = await loadHttpModuleWithBunMocks({ spawnImpl: spawnSync })
|
||||
|
||||
await http.apiRequest('https://registry.example', {
|
||||
method: 'POST',
|
||||
path: '/v1/ping',
|
||||
body: { a: 1 },
|
||||
})
|
||||
|
||||
const [, args] = spawnSync.mock.calls[0] as [string, string[]]
|
||||
expect(args).toContain('Content-Type: application/json')
|
||||
expect(args).toContain('--data-binary')
|
||||
expect(args).toContain('{"a":1}')
|
||||
})
|
||||
|
||||
it('retries bun apiRequest on 429 errors', async () => {
|
||||
const spawnSync = vi.fn().mockReturnValue({
|
||||
status: 0,
|
||||
stdout: 'rate limited\n429',
|
||||
stderr: '',
|
||||
})
|
||||
const { http } = await loadHttpModuleWithBunMocks({ spawnImpl: spawnSync })
|
||||
|
||||
await expect(
|
||||
http.apiRequest('https://registry.example', {
|
||||
method: 'GET',
|
||||
path: '/v1/ping',
|
||||
}),
|
||||
).rejects.toThrow('rate limited')
|
||||
|
||||
expect(spawnSync).toHaveBeenCalledTimes(3)
|
||||
})
|
||||
|
||||
it('includes rate-limit guidance from curl metadata on 429', async () => {
|
||||
mockImmediateTimeouts()
|
||||
const spawnSync = vi.fn().mockReturnValue({
|
||||
status: 0,
|
||||
stdout:
|
||||
'rate limited\n__CLAWHUB_CURL_META__\n429\n20\n0\n1771404540\n20\n0\n34\n34\n',
|
||||
stderr: '',
|
||||
})
|
||||
const { http } = await loadHttpModuleWithBunMocks({ spawnImpl: spawnSync })
|
||||
|
||||
await expect(
|
||||
http.apiRequest('https://registry.example', {
|
||||
method: 'GET',
|
||||
path: '/v1/ping',
|
||||
}),
|
||||
).rejects.toThrow(/retry in 34s.*remaining: 0\/20.*reset in 34s/i)
|
||||
|
||||
expect(spawnSync).toHaveBeenCalledTimes(3)
|
||||
})
|
||||
|
||||
it('does not retry bun apiRequest on 404 errors', async () => {
|
||||
const spawnSync = vi.fn().mockReturnValue({
|
||||
status: 0,
|
||||
stdout: 'missing\n404',
|
||||
stderr: '',
|
||||
})
|
||||
const { http } = await loadHttpModuleWithBunMocks({ spawnImpl: spawnSync })
|
||||
|
||||
await expect(
|
||||
http.apiRequest('https://registry.example', {
|
||||
method: 'GET',
|
||||
path: '/v1/ping',
|
||||
}),
|
||||
).rejects.toThrow('missing')
|
||||
|
||||
expect(spawnSync).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('supports fetchText bun path and propagates status fallback', async () => {
|
||||
const spawnSync = vi
|
||||
.fn()
|
||||
.mockReturnValueOnce({
|
||||
status: 0,
|
||||
stdout: 'hello world\n200',
|
||||
stderr: '',
|
||||
})
|
||||
.mockReturnValueOnce({
|
||||
status: 0,
|
||||
stdout: '\n400',
|
||||
stderr: '',
|
||||
})
|
||||
const { http } = await loadHttpModuleWithBunMocks({ spawnImpl: spawnSync })
|
||||
|
||||
const text = await http.fetchText('https://registry.example', { path: '/v1/readme' })
|
||||
expect(text).toBe('hello world')
|
||||
|
||||
await expect(
|
||||
http.fetchText('https://registry.example', { path: '/v1/readme' }),
|
||||
).rejects.toThrow('HTTP 400')
|
||||
})
|
||||
|
||||
it('handles downloadZip bun path and cleans up temp dir', async () => {
|
||||
const spawnSync = vi
|
||||
.fn()
|
||||
.mockReturnValueOnce({
|
||||
status: 0,
|
||||
stdout: '200',
|
||||
stderr: '',
|
||||
})
|
||||
.mockReturnValueOnce({
|
||||
status: 0,
|
||||
stdout: '404',
|
||||
stderr: '',
|
||||
})
|
||||
const { http, rm, readFile } = await loadHttpModuleWithBunMocks({
|
||||
spawnImpl: spawnSync,
|
||||
mkdtempValue: '/tmp/clawhub-download-abc',
|
||||
readFileValue: Buffer.from('not found'),
|
||||
})
|
||||
|
||||
const bytes = await http.downloadZip('https://registry.example', { slug: 'demo', token: 't' })
|
||||
expect(Array.from(bytes)).toEqual(Array.from(Buffer.from('not found')))
|
||||
|
||||
await expect(
|
||||
http.downloadZip('https://registry.example', { slug: 'demo', token: 't' }),
|
||||
).rejects.toThrow('not found')
|
||||
|
||||
expect(readFile).toHaveBeenCalled()
|
||||
expect(rm).toHaveBeenCalledWith('/tmp/clawhub-download-abc', {
|
||||
recursive: true,
|
||||
force: true,
|
||||
})
|
||||
})
|
||||
|
||||
it('posts multipart form via curl in bun path', async () => {
|
||||
const spawnSync = vi.fn().mockReturnValue({
|
||||
status: 0,
|
||||
stdout: '{"ok":true}\n200',
|
||||
stderr: '',
|
||||
})
|
||||
const { http, writeFile, rm } = await loadHttpModuleWithBunMocks({
|
||||
spawnImpl: spawnSync,
|
||||
mkdtempValue: '/tmp/clawhub-upload-abc',
|
||||
})
|
||||
|
||||
const form = new FormData()
|
||||
form.append('name', 'demo')
|
||||
form.append('file', new Blob(['abc'], { type: 'text/plain' }), 'demo.txt')
|
||||
|
||||
const result = await http.apiRequestForm<{ ok: boolean }>('https://registry.example', {
|
||||
method: 'POST',
|
||||
path: '/upload',
|
||||
form,
|
||||
})
|
||||
|
||||
expect(result).toEqual({ ok: true })
|
||||
expect(writeFile).toHaveBeenCalled()
|
||||
expect(rm).toHaveBeenCalledWith('/tmp/clawhub-upload-abc', { recursive: true, force: true })
|
||||
const [, args] = spawnSync.mock.calls[0] as [string, string[]]
|
||||
expect(args).toContain('-F')
|
||||
expect(args.some((arg) => arg.includes('name=demo'))).toBe(true)
|
||||
expect(args.some((arg) => arg.includes('file=@/tmp/clawhub-upload-abc/demo.txt'))).toBe(true)
|
||||
})
|
||||
})
|
||||
@@ -1,7 +1,7 @@
|
||||
/* @vitest-environment node */
|
||||
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { apiRequest, apiRequestForm, downloadZip, fetchText } from './http'
|
||||
import { apiRequest, apiRequestForm, downloadZip, fetchText, shouldUseProxyFromEnv } from './http'
|
||||
import { ApiV1WhoamiResponseSchema } from './schema/index.js'
|
||||
|
||||
function mockImmediateTimeouts() {
|
||||
@@ -36,6 +36,35 @@ function createAbortingFetchMock() {
|
||||
})
|
||||
}
|
||||
|
||||
describe('shouldUseProxyFromEnv', () => {
|
||||
it('detects standard proxy variables', () => {
|
||||
expect(
|
||||
shouldUseProxyFromEnv({
|
||||
HTTPS_PROXY: 'http://proxy.example:3128',
|
||||
} as NodeJS.ProcessEnv),
|
||||
).toBe(true)
|
||||
expect(
|
||||
shouldUseProxyFromEnv({
|
||||
HTTP_PROXY: 'http://proxy.example:3128',
|
||||
} as NodeJS.ProcessEnv),
|
||||
).toBe(true)
|
||||
expect(
|
||||
shouldUseProxyFromEnv({
|
||||
https_proxy: 'http://proxy.example:3128',
|
||||
} as NodeJS.ProcessEnv),
|
||||
).toBe(true)
|
||||
})
|
||||
|
||||
it('ignores NO_PROXY-only configs', () => {
|
||||
expect(
|
||||
shouldUseProxyFromEnv({
|
||||
NO_PROXY: 'localhost,127.0.0.1',
|
||||
} as NodeJS.ProcessEnv),
|
||||
).toBe(false)
|
||||
expect(shouldUseProxyFromEnv({} as NodeJS.ProcessEnv)).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('apiRequest', () => {
|
||||
it('adds bearer token and parses json', async () => {
|
||||
const fetchMock = vi.fn().mockResolvedValue({
|
||||
@@ -86,6 +115,50 @@ describe('apiRequest', () => {
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
|
||||
it('includes rate-limit guidance from headers on 429', async () => {
|
||||
mockImmediateTimeouts()
|
||||
const fetchMock = vi.fn().mockResolvedValue({
|
||||
ok: false,
|
||||
status: 429,
|
||||
headers: new Headers({
|
||||
'Retry-After': '34',
|
||||
'X-RateLimit-Limit': '20',
|
||||
'X-RateLimit-Remaining': '0',
|
||||
'X-RateLimit-Reset': '1771404540',
|
||||
}),
|
||||
text: async () => 'Rate limit exceeded',
|
||||
})
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
|
||||
await expect(apiRequest('https://example.com', { method: 'GET', path: '/x' })).rejects.toThrow(
|
||||
/retry in 34s.*remaining: 0\/20.*reset in 34s/i,
|
||||
)
|
||||
expect(fetchMock).toHaveBeenCalledTimes(3)
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
|
||||
it('interprets legacy epoch Retry-After values as reset delays', async () => {
|
||||
mockImmediateTimeouts()
|
||||
vi.spyOn(Date, 'now').mockReturnValue(1_771_404_500_000)
|
||||
const fetchMock = vi.fn().mockResolvedValue({
|
||||
ok: false,
|
||||
status: 429,
|
||||
headers: new Headers({
|
||||
'Retry-After': '1771404540',
|
||||
'X-RateLimit-Limit': '20',
|
||||
'X-RateLimit-Remaining': '0',
|
||||
}),
|
||||
text: async () => 'Rate limit exceeded',
|
||||
})
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
|
||||
await expect(apiRequest('https://example.com', { method: 'GET', path: '/x' })).rejects.toThrow(
|
||||
/retry in 40s.*remaining: 0\/20/i,
|
||||
)
|
||||
vi.restoreAllMocks()
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
|
||||
it('falls back to HTTP status when body is empty', async () => {
|
||||
const fetchMock = vi.fn().mockResolvedValue({
|
||||
ok: false,
|
||||
|
||||
+267
-50
@@ -3,20 +3,45 @@ import { mkdtemp, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import pRetry, { AbortError } from 'p-retry'
|
||||
import { Agent, setGlobalDispatcher } from 'undici'
|
||||
import { Agent, EnvHttpProxyAgent, setGlobalDispatcher } from 'undici'
|
||||
import type { ArkValidator } from './schema/index.js'
|
||||
import { ApiRoutes, parseArk } from './schema/index.js'
|
||||
|
||||
const REQUEST_TIMEOUT_MS = 15_000
|
||||
const REQUEST_TIMEOUT_SECONDS = Math.ceil(REQUEST_TIMEOUT_MS / 1000)
|
||||
const RETRY_COUNT = 2
|
||||
const RETRY_BACKOFF_BASE_MS = 300
|
||||
const RETRY_BACKOFF_MAX_MS = 5_000
|
||||
const RETRY_AFTER_JITTER_MS = 250
|
||||
const CURL_META_MARKER = '__CLAWHUB_CURL_META__'
|
||||
const CURL_WRITE_OUT_FORMAT = [
|
||||
'',
|
||||
CURL_META_MARKER,
|
||||
'%{http_code}',
|
||||
'%{header:x-ratelimit-limit}',
|
||||
'%{header:x-ratelimit-remaining}',
|
||||
'%{header:x-ratelimit-reset}',
|
||||
'%{header:ratelimit-limit}',
|
||||
'%{header:ratelimit-remaining}',
|
||||
'%{header:ratelimit-reset}',
|
||||
'%{header:retry-after}',
|
||||
].join('\n')
|
||||
const isBun = typeof process !== 'undefined' && Boolean(process.versions?.bun)
|
||||
|
||||
export function shouldUseProxyFromEnv(env: NodeJS.ProcessEnv = process.env): boolean {
|
||||
return Boolean(env.HTTPS_PROXY || env.HTTP_PROXY || env.https_proxy || env.http_proxy)
|
||||
}
|
||||
|
||||
if (typeof process !== 'undefined' && process.versions?.node) {
|
||||
try {
|
||||
setGlobalDispatcher(
|
||||
new Agent({
|
||||
connect: { timeout: REQUEST_TIMEOUT_MS },
|
||||
}),
|
||||
shouldUseProxyFromEnv(process.env)
|
||||
? new EnvHttpProxyAgent({
|
||||
connect: { timeout: REQUEST_TIMEOUT_MS },
|
||||
})
|
||||
: new Agent({
|
||||
connect: { timeout: REQUEST_TIMEOUT_MS },
|
||||
}),
|
||||
)
|
||||
} catch {
|
||||
// ignore dispatcher setup failures in non-node runtimes
|
||||
@@ -27,6 +52,27 @@ type RequestArgs =
|
||||
| { method: 'GET' | 'POST' | 'DELETE'; path: string; token?: string; body?: unknown }
|
||||
| { method: 'GET' | 'POST' | 'DELETE'; url: string; token?: string; body?: unknown }
|
||||
|
||||
type HeaderSource = Headers | Record<string, string> | null | undefined
|
||||
|
||||
type RateLimitInfo = {
|
||||
limit?: number
|
||||
remaining?: number
|
||||
resetDelaySeconds?: number
|
||||
retryAfterSeconds?: number
|
||||
}
|
||||
|
||||
class HttpStatusError extends Error {
|
||||
readonly status: number
|
||||
readonly rateLimit: RateLimitInfo
|
||||
|
||||
constructor(status: number, message: string, rateLimit: RateLimitInfo) {
|
||||
super(message)
|
||||
this.name = 'HttpStatusError'
|
||||
this.status = status
|
||||
this.rateLimit = rateLimit
|
||||
}
|
||||
}
|
||||
|
||||
export async function apiRequest<T>(registry: string, args: RequestArgs): Promise<T>
|
||||
export async function apiRequest<T>(
|
||||
registry: string,
|
||||
@@ -39,7 +85,7 @@ export async function apiRequest<T>(
|
||||
schema?: ArkValidator<T>,
|
||||
): Promise<T> {
|
||||
const url = 'url' in args ? args.url : new URL(args.path, registry).toString()
|
||||
const json = await pRetry(
|
||||
const json = await runWithRetries(
|
||||
async () => {
|
||||
if (isBun) {
|
||||
return await fetchJsonViaCurl(url, args)
|
||||
@@ -58,11 +104,10 @@ export async function apiRequest<T>(
|
||||
body,
|
||||
})
|
||||
if (!response.ok) {
|
||||
throwHttpStatusError(response.status, await readResponseTextSafe(response))
|
||||
throwHttpStatusError(response.status, await readResponseTextSafe(response), response.headers)
|
||||
}
|
||||
return (await response.json()) as unknown
|
||||
},
|
||||
{ retries: 2 },
|
||||
)
|
||||
if (schema) return parseArk(schema, json, 'API response')
|
||||
return json as T
|
||||
@@ -84,7 +129,7 @@ export async function apiRequestForm<T>(
|
||||
schema?: ArkValidator<T>,
|
||||
): Promise<T> {
|
||||
const url = 'url' in args ? args.url : new URL(args.path, registry).toString()
|
||||
const json = await pRetry(
|
||||
const json = await runWithRetries(
|
||||
async () => {
|
||||
if (isBun) {
|
||||
return await fetchJsonFormViaCurl(url, args)
|
||||
@@ -98,11 +143,10 @@ export async function apiRequestForm<T>(
|
||||
body: args.form,
|
||||
})
|
||||
if (!response.ok) {
|
||||
throwHttpStatusError(response.status, await readResponseTextSafe(response))
|
||||
throwHttpStatusError(response.status, await readResponseTextSafe(response), response.headers)
|
||||
}
|
||||
return (await response.json()) as unknown
|
||||
},
|
||||
{ retries: 2 },
|
||||
)
|
||||
if (schema) return parseArk(schema, json, 'API response')
|
||||
return json as T
|
||||
@@ -112,7 +156,7 @@ type TextRequestArgs = { path: string; token?: string } | { url: string; token?:
|
||||
|
||||
export async function fetchText(registry: string, args: TextRequestArgs): Promise<string> {
|
||||
const url = 'url' in args ? args.url : new URL(args.path, registry).toString()
|
||||
return pRetry(
|
||||
return runWithRetries(
|
||||
async () => {
|
||||
if (isBun) {
|
||||
return await fetchTextViaCurl(url, args)
|
||||
@@ -123,11 +167,10 @@ export async function fetchText(registry: string, args: TextRequestArgs): Promis
|
||||
const response = await fetchWithTimeout(url, { method: 'GET', headers })
|
||||
const text = await response.text()
|
||||
if (!response.ok) {
|
||||
throwHttpStatusError(response.status, text)
|
||||
throwHttpStatusError(response.status, text, response.headers)
|
||||
}
|
||||
return text
|
||||
},
|
||||
{ retries: 2 },
|
||||
)
|
||||
}
|
||||
|
||||
@@ -138,7 +181,7 @@ export async function downloadZip(
|
||||
const url = new URL(ApiRoutes.download, registry)
|
||||
url.searchParams.set('slug', args.slug)
|
||||
if (args.version) url.searchParams.set('version', args.version)
|
||||
return pRetry(
|
||||
return runWithRetries(
|
||||
async () => {
|
||||
if (isBun) {
|
||||
return await fetchBinaryViaCurl(url.toString(), args.token)
|
||||
@@ -149,11 +192,10 @@ export async function downloadZip(
|
||||
|
||||
const response = await fetchWithTimeout(url.toString(), { method: 'GET', headers })
|
||||
if (!response.ok) {
|
||||
throwHttpStatusError(response.status, await readResponseTextSafe(response))
|
||||
throwHttpStatusError(response.status, await readResponseTextSafe(response), response.headers)
|
||||
}
|
||||
return new Uint8Array(await response.arrayBuffer())
|
||||
},
|
||||
{ retries: 2 },
|
||||
)
|
||||
}
|
||||
|
||||
@@ -171,14 +213,152 @@ async function readResponseTextSafe(response: Response): Promise<string> {
|
||||
return await response.text().catch(() => '')
|
||||
}
|
||||
|
||||
function throwHttpStatusError(status: number, text: string): never {
|
||||
const message = text || `HTTP ${status}`
|
||||
async function runWithRetries<T>(fn: () => Promise<T>): Promise<T> {
|
||||
return await pRetry(fn, {
|
||||
retries: RETRY_COUNT,
|
||||
minTimeout: 0,
|
||||
maxTimeout: 0,
|
||||
factor: 1,
|
||||
randomize: false,
|
||||
onFailedAttempt: async (attemptError) => {
|
||||
const delayMs = getRetryDelayMs(attemptError)
|
||||
if (delayMs <= 0) return
|
||||
await sleep(delayMs)
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
function getRetryDelayMs(attemptError: unknown): number {
|
||||
const failed = attemptError as {
|
||||
attemptNumber?: number
|
||||
cause?: unknown
|
||||
error?: unknown
|
||||
}
|
||||
const attemptNumber = Math.max(1, Number(failed.attemptNumber ?? 1))
|
||||
const rootError = failed.cause ?? failed.error ?? attemptError
|
||||
if (rootError instanceof HttpStatusError && rootError.rateLimit.retryAfterSeconds !== undefined) {
|
||||
return rootError.rateLimit.retryAfterSeconds * 1000 + jitterMs(RETRY_AFTER_JITTER_MS)
|
||||
}
|
||||
const baseMs = Math.min(RETRY_BACKOFF_MAX_MS, RETRY_BACKOFF_BASE_MS * 2 ** (attemptNumber - 1))
|
||||
return baseMs + jitterMs(RETRY_BACKOFF_BASE_MS)
|
||||
}
|
||||
|
||||
function sleep(ms: number): Promise<void> {
|
||||
return new Promise((resolve) => {
|
||||
setTimeout(resolve, ms)
|
||||
})
|
||||
}
|
||||
|
||||
function jitterMs(maxMs: number): number {
|
||||
if (maxMs <= 0) return 0
|
||||
return Math.floor(Math.random() * maxMs)
|
||||
}
|
||||
|
||||
function throwHttpStatusError(status: number, text: string, headers?: HeaderSource): never {
|
||||
const rateLimit = parseRateLimitInfo(headers)
|
||||
const message = buildHttpErrorMessage(status, text, rateLimit)
|
||||
if (status === 429 || status >= 500) {
|
||||
throw new Error(message)
|
||||
throw new HttpStatusError(status, message, rateLimit)
|
||||
}
|
||||
throw new AbortError(message)
|
||||
}
|
||||
|
||||
function buildHttpErrorMessage(status: number, text: string, rateLimit: RateLimitInfo): string {
|
||||
const base = text || `HTTP ${status}`
|
||||
const details: string[] = []
|
||||
if (rateLimit.retryAfterSeconds !== undefined) {
|
||||
details.push(`retry in ${rateLimit.retryAfterSeconds}s`)
|
||||
}
|
||||
if (rateLimit.remaining !== undefined && rateLimit.limit !== undefined) {
|
||||
details.push(`remaining: ${rateLimit.remaining}/${rateLimit.limit}`)
|
||||
}
|
||||
if (rateLimit.resetDelaySeconds !== undefined) {
|
||||
details.push(`reset in ${rateLimit.resetDelaySeconds}s`)
|
||||
}
|
||||
if (details.length === 0) {
|
||||
return base
|
||||
}
|
||||
return `${base} (${details.join(', ')})`
|
||||
}
|
||||
|
||||
function parseRateLimitInfo(headers?: HeaderSource): RateLimitInfo {
|
||||
if (!headers) return {}
|
||||
const limit = parseIntHeader(getHeader(headers, 'x-ratelimit-limit') ?? getHeader(headers, 'ratelimit-limit'))
|
||||
const remaining = parseIntHeader(
|
||||
getHeader(headers, 'x-ratelimit-remaining') ?? getHeader(headers, 'ratelimit-remaining'),
|
||||
)
|
||||
const nowMs = Date.now()
|
||||
const retryAfterSeconds = parseRetryAfterSeconds(getHeader(headers, 'retry-after'), nowMs)
|
||||
const resetDelaySeconds = parseResetDelaySeconds(headers, nowMs, retryAfterSeconds)
|
||||
|
||||
return {
|
||||
limit,
|
||||
remaining,
|
||||
resetDelaySeconds,
|
||||
retryAfterSeconds,
|
||||
}
|
||||
}
|
||||
|
||||
function parseResetDelaySeconds(
|
||||
headers: HeaderSource,
|
||||
nowMs: number,
|
||||
retryAfterSeconds: number | undefined,
|
||||
): number | undefined {
|
||||
if (retryAfterSeconds !== undefined) return retryAfterSeconds
|
||||
|
||||
const standardized = parseIntHeader(getHeader(headers, 'ratelimit-reset'))
|
||||
if (standardized !== undefined) {
|
||||
return Math.max(1, standardized)
|
||||
}
|
||||
const legacyEpochSeconds = parseIntHeader(getHeader(headers, 'x-ratelimit-reset'))
|
||||
if (legacyEpochSeconds === undefined) return undefined
|
||||
const nowSeconds = Math.floor(nowMs / 1000)
|
||||
return Math.max(1, legacyEpochSeconds - nowSeconds)
|
||||
}
|
||||
|
||||
function parseRetryAfterSeconds(value: string | undefined, nowMs: number): number | undefined {
|
||||
if (!value) return undefined
|
||||
const trimmed = value.trim()
|
||||
if (!trimmed) return undefined
|
||||
|
||||
const asNumber = Number(trimmed)
|
||||
if (Number.isFinite(asNumber) && asNumber >= 0) {
|
||||
// Compatibility guard for older servers that accidentally sent Unix epoch seconds.
|
||||
if (asNumber > 31_536_000) {
|
||||
const nowSeconds = Math.floor(nowMs / 1000)
|
||||
return Math.max(1, Math.ceil(asNumber - nowSeconds))
|
||||
}
|
||||
return Math.max(1, Math.ceil(asNumber))
|
||||
}
|
||||
|
||||
const asDateMs = Date.parse(trimmed)
|
||||
if (!Number.isFinite(asDateMs)) return undefined
|
||||
return Math.max(1, Math.ceil((asDateMs - nowMs) / 1000))
|
||||
}
|
||||
|
||||
function parseIntHeader(value: string | undefined): number | undefined {
|
||||
if (!value) return undefined
|
||||
const parsed = Number.parseInt(value, 10)
|
||||
if (!Number.isFinite(parsed)) return undefined
|
||||
return parsed
|
||||
}
|
||||
|
||||
function getHeader(headers: HeaderSource, key: string): string | undefined {
|
||||
if (!headers) return undefined
|
||||
if (headers instanceof Headers) {
|
||||
const value = headers.get(key)
|
||||
return value === null ? undefined : value
|
||||
}
|
||||
const normalizedKey = key.toLowerCase()
|
||||
const direct = headers[normalizedKey] ?? headers[key]
|
||||
if (typeof direct === 'string' && direct.trim()) return direct.trim()
|
||||
const match = Object.entries(headers).find(
|
||||
([entryKey, entryValue]) =>
|
||||
entryKey.toLowerCase() === normalizedKey && typeof entryValue === 'string' && entryValue.trim(),
|
||||
)
|
||||
return typeof match?.[1] === 'string' ? match[1].trim() : undefined
|
||||
}
|
||||
|
||||
async function fetchJsonViaCurl(url: string, args: RequestArgs) {
|
||||
const headers = ['-H', 'Accept: application/json']
|
||||
if (args.token) {
|
||||
@@ -191,7 +371,7 @@ async function fetchJsonViaCurl(url: string, args: RequestArgs) {
|
||||
'--max-time',
|
||||
String(REQUEST_TIMEOUT_SECONDS),
|
||||
'--write-out',
|
||||
'\n%{http_code}',
|
||||
CURL_WRITE_OUT_FORMAT,
|
||||
'-X',
|
||||
args.method,
|
||||
...headers,
|
||||
@@ -206,14 +386,9 @@ async function fetchJsonViaCurl(url: string, args: RequestArgs) {
|
||||
if (result.status !== 0) {
|
||||
throw new Error(result.stderr || 'curl failed')
|
||||
}
|
||||
const output = result.stdout ?? ''
|
||||
const splitAt = output.lastIndexOf('\n')
|
||||
if (splitAt === -1) throw new Error('curl response missing status')
|
||||
const body = output.slice(0, splitAt)
|
||||
const status = Number(output.slice(splitAt + 1).trim())
|
||||
if (!Number.isFinite(status)) throw new Error('curl response missing status')
|
||||
const { body, status, headers: responseHeaders } = parseCurlBodyAndMeta(result.stdout ?? '')
|
||||
if (status < 200 || status >= 300) {
|
||||
throwHttpStatusError(status, body)
|
||||
throwHttpStatusError(status, body, responseHeaders)
|
||||
}
|
||||
return JSON.parse(body || 'null') as unknown
|
||||
}
|
||||
@@ -246,7 +421,7 @@ async function fetchJsonFormViaCurl(url: string, args: FormRequestArgs) {
|
||||
'--max-time',
|
||||
String(REQUEST_TIMEOUT_SECONDS),
|
||||
'--write-out',
|
||||
'\n%{http_code}',
|
||||
CURL_WRITE_OUT_FORMAT,
|
||||
'-X',
|
||||
args.method,
|
||||
...headers,
|
||||
@@ -258,14 +433,9 @@ async function fetchJsonFormViaCurl(url: string, args: FormRequestArgs) {
|
||||
if (result.status !== 0) {
|
||||
throw new Error(result.stderr || 'curl failed')
|
||||
}
|
||||
const output = result.stdout ?? ''
|
||||
const splitAt = output.lastIndexOf('\n')
|
||||
if (splitAt === -1) throw new Error('curl response missing status')
|
||||
const body = output.slice(0, splitAt)
|
||||
const status = Number(output.slice(splitAt + 1).trim())
|
||||
if (!Number.isFinite(status)) throw new Error('curl response missing status')
|
||||
const { body, status, headers: responseHeaders } = parseCurlBodyAndMeta(result.stdout ?? '')
|
||||
if (status < 200 || status >= 300) {
|
||||
throwHttpStatusError(status, body)
|
||||
throwHttpStatusError(status, body, responseHeaders)
|
||||
}
|
||||
return JSON.parse(body || 'null') as unknown
|
||||
} finally {
|
||||
@@ -285,7 +455,7 @@ async function fetchTextViaCurl(url: string, args: { token?: string }) {
|
||||
'--max-time',
|
||||
String(REQUEST_TIMEOUT_SECONDS),
|
||||
'--write-out',
|
||||
'\n%{http_code}',
|
||||
CURL_WRITE_OUT_FORMAT,
|
||||
'-X',
|
||||
'GET',
|
||||
...headers,
|
||||
@@ -295,17 +465,9 @@ async function fetchTextViaCurl(url: string, args: { token?: string }) {
|
||||
if (result.status !== 0) {
|
||||
throw new Error(result.stderr || 'curl failed')
|
||||
}
|
||||
const output = result.stdout ?? ''
|
||||
const splitAt = output.lastIndexOf('\n')
|
||||
if (splitAt === -1) throw new Error('curl response missing status')
|
||||
const body = output.slice(0, splitAt)
|
||||
const status = Number(output.slice(splitAt + 1).trim())
|
||||
if (!Number.isFinite(status)) throw new Error('curl response missing status')
|
||||
const { body, status, headers: responseHeaders } = parseCurlBodyAndMeta(result.stdout ?? '')
|
||||
if (status < 200 || status >= 300) {
|
||||
if (status === 429 || status >= 500) {
|
||||
throw new Error(body || `HTTP ${status}`)
|
||||
}
|
||||
throw new AbortError(body || `HTTP ${status}`)
|
||||
throwHttpStatusError(status, body, responseHeaders)
|
||||
}
|
||||
return body
|
||||
}
|
||||
@@ -329,18 +491,17 @@ async function fetchBinaryViaCurl(url: string, token?: string) {
|
||||
'-o',
|
||||
filePath,
|
||||
'--write-out',
|
||||
'%{http_code}',
|
||||
CURL_WRITE_OUT_FORMAT,
|
||||
url,
|
||||
]
|
||||
const result = spawnSync('curl', curlArgs, { encoding: 'utf8' })
|
||||
if (result.status !== 0) {
|
||||
throw new Error(result.stderr || 'curl failed')
|
||||
}
|
||||
const status = Number((result.stdout ?? '').trim())
|
||||
if (!Number.isFinite(status)) throw new Error('curl response missing status')
|
||||
const { status, headers: responseHeaders } = parseCurlBodyAndMeta(result.stdout ?? '')
|
||||
if (status < 200 || status >= 300) {
|
||||
const body = await readFileSafe(filePath)
|
||||
throwHttpStatusError(status, body ? new TextDecoder().decode(body) : '')
|
||||
throwHttpStatusError(status, body ? new TextDecoder().decode(body) : '', responseHeaders)
|
||||
}
|
||||
const bytes = await readFileSafe(filePath)
|
||||
return bytes ? new Uint8Array(bytes) : new Uint8Array()
|
||||
@@ -349,6 +510,62 @@ async function fetchBinaryViaCurl(url: string, token?: string) {
|
||||
}
|
||||
}
|
||||
|
||||
function parseCurlBodyAndMeta(output: string): {
|
||||
body: string
|
||||
status: number
|
||||
headers: Record<string, string>
|
||||
} {
|
||||
const marker = `\n${CURL_META_MARKER}\n`
|
||||
const markerIndex = output.lastIndexOf(marker)
|
||||
if (markerIndex === -1) {
|
||||
// Backward compatibility for older tests that only provide "<body>\n<status>".
|
||||
const splitAt = output.lastIndexOf('\n')
|
||||
if (splitAt === -1) {
|
||||
const statusOnly = Number(output.trim())
|
||||
if (!Number.isFinite(statusOnly)) throw new Error('curl response missing status')
|
||||
return { body: '', status: statusOnly, headers: {} }
|
||||
}
|
||||
const body = output.slice(0, splitAt)
|
||||
const status = Number(output.slice(splitAt + 1).trim())
|
||||
if (!Number.isFinite(status)) throw new Error('curl response missing status')
|
||||
return { body, status, headers: {} }
|
||||
}
|
||||
|
||||
const body = output.slice(0, markerIndex)
|
||||
const meta = output.slice(markerIndex + marker.length).replace(/\r/g, '')
|
||||
const lines = meta.split('\n')
|
||||
const status = Number((lines[0] ?? '').trim())
|
||||
if (!Number.isFinite(status)) throw new Error('curl response missing status')
|
||||
|
||||
const [
|
||||
xRateLimitLimit,
|
||||
xRateLimitRemaining,
|
||||
xRateLimitReset,
|
||||
rateLimitLimit,
|
||||
rateLimitRemaining,
|
||||
rateLimitReset,
|
||||
retryAfter,
|
||||
] = lines.slice(1)
|
||||
|
||||
const headers: Record<string, string> = {}
|
||||
setHeaderIfPresent(headers, 'x-ratelimit-limit', xRateLimitLimit)
|
||||
setHeaderIfPresent(headers, 'x-ratelimit-remaining', xRateLimitRemaining)
|
||||
setHeaderIfPresent(headers, 'x-ratelimit-reset', xRateLimitReset)
|
||||
setHeaderIfPresent(headers, 'ratelimit-limit', rateLimitLimit)
|
||||
setHeaderIfPresent(headers, 'ratelimit-remaining', rateLimitRemaining)
|
||||
setHeaderIfPresent(headers, 'ratelimit-reset', rateLimitReset)
|
||||
setHeaderIfPresent(headers, 'retry-after', retryAfter)
|
||||
|
||||
return { body, status, headers }
|
||||
}
|
||||
|
||||
function setHeaderIfPresent(headers: Record<string, string>, key: string, value: string | undefined) {
|
||||
if (typeof value !== 'string') return
|
||||
const trimmed = value.trim()
|
||||
if (!trimmed) return
|
||||
headers[key] = trimmed
|
||||
}
|
||||
|
||||
async function readFileSafe(path: string) {
|
||||
try {
|
||||
const { readFile } = await import('node:fs/promises')
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
import { vi } from 'vitest'
|
||||
|
||||
export const convexReactMocks = {
|
||||
useAction: vi.fn(),
|
||||
useQuery: vi.fn(),
|
||||
usePaginatedQuery: vi.fn(),
|
||||
}
|
||||
|
||||
export function resetConvexReactMocks() {
|
||||
convexReactMocks.useAction.mockReset()
|
||||
convexReactMocks.useQuery.mockReset()
|
||||
convexReactMocks.usePaginatedQuery.mockReset()
|
||||
}
|
||||
|
||||
export function setupDefaultConvexReactMocks() {
|
||||
convexReactMocks.useAction.mockReturnValue(() => Promise.resolve([]))
|
||||
convexReactMocks.useQuery.mockReturnValue(null)
|
||||
}
|
||||
@@ -1,6 +1,5 @@
|
||||
import { fireEvent, render, screen, waitFor } from '@testing-library/react'
|
||||
import { vi } from 'vitest'
|
||||
|
||||
import { SkillDetailPage } from '../components/SkillDetailPage'
|
||||
|
||||
const navigateMock = vi.fn()
|
||||
@@ -130,4 +129,57 @@ describe('SkillDetailPage', () => {
|
||||
expect(await screen.findByRole('dialog')).toBeTruthy()
|
||||
expect(screen.getByText(/Report skill/i)).toBeTruthy()
|
||||
})
|
||||
|
||||
it('defers compare version query until compare tab is requested', async () => {
|
||||
useQueryMock.mockImplementation((_fn: unknown, args: unknown) => {
|
||||
if (args === 'skip') return undefined
|
||||
if (args && typeof args === 'object' && 'limit' in args) {
|
||||
return []
|
||||
}
|
||||
if (args && typeof args === 'object' && 'skillId' in args) return []
|
||||
if (args && typeof args === 'object' && 'slug' in args) {
|
||||
return {
|
||||
skill: {
|
||||
_id: 'skills:1',
|
||||
slug: 'weather',
|
||||
displayName: 'Weather',
|
||||
summary: 'Get current weather.',
|
||||
ownerUserId: 'users:1',
|
||||
tags: {},
|
||||
stats: { stars: 0, downloads: 0 },
|
||||
},
|
||||
owner: { handle: 'steipete', name: 'Peter' },
|
||||
latestVersion: { _id: 'skillVersions:1', version: '1.0.0', parsed: {}, files: [] },
|
||||
}
|
||||
}
|
||||
return undefined
|
||||
})
|
||||
|
||||
render(<SkillDetailPage slug="weather" />)
|
||||
expect(await screen.findByText('Weather')).toBeTruthy()
|
||||
|
||||
expect(
|
||||
useQueryMock.mock.calls.some(
|
||||
([, args]: [unknown, unknown]) =>
|
||||
typeof args === 'object' &&
|
||||
args !== null &&
|
||||
'limit' in args &&
|
||||
(args as { limit: number }).limit === 200,
|
||||
),
|
||||
).toBe(false)
|
||||
|
||||
fireEvent.click(screen.getByRole('button', { name: /compare/i }))
|
||||
|
||||
await waitFor(() => {
|
||||
expect(
|
||||
useQueryMock.mock.calls.some(
|
||||
([, args]: [unknown, unknown]) =>
|
||||
typeof args === 'object' &&
|
||||
args !== null &&
|
||||
'limit' in args &&
|
||||
(args as { limit: number }).limit === 200,
|
||||
),
|
||||
).toBe(true)
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
@@ -2,12 +2,15 @@
|
||||
import { act, render } from '@testing-library/react'
|
||||
import type { ReactNode } from 'react'
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import {
|
||||
convexReactMocks,
|
||||
resetConvexReactMocks,
|
||||
setupDefaultConvexReactMocks,
|
||||
} from './helpers/convexReactMocks'
|
||||
|
||||
import { SkillsIndex } from '../routes/skills/index'
|
||||
|
||||
const navigateMock = vi.fn()
|
||||
const useActionMock = vi.fn()
|
||||
const usePaginatedQueryMock = vi.fn()
|
||||
let searchMock: Record<string, unknown> = {}
|
||||
|
||||
vi.mock('@tanstack/react-router', () => ({
|
||||
@@ -20,17 +23,17 @@ vi.mock('@tanstack/react-router', () => ({
|
||||
}))
|
||||
|
||||
vi.mock('convex/react', () => ({
|
||||
useAction: (...args: unknown[]) => useActionMock(...args),
|
||||
usePaginatedQuery: (...args: unknown[]) => usePaginatedQueryMock(...args),
|
||||
useAction: (...args: unknown[]) => convexReactMocks.useAction(...args),
|
||||
useQuery: (...args: unknown[]) => convexReactMocks.useQuery(...args),
|
||||
usePaginatedQuery: (...args: unknown[]) => convexReactMocks.usePaginatedQuery(...args),
|
||||
}))
|
||||
|
||||
describe('SkillsIndex load-more observer', () => {
|
||||
beforeEach(() => {
|
||||
usePaginatedQueryMock.mockReset()
|
||||
useActionMock.mockReset()
|
||||
resetConvexReactMocks()
|
||||
navigateMock.mockReset()
|
||||
searchMock = {}
|
||||
useActionMock.mockReturnValue(() => Promise.resolve([]))
|
||||
setupDefaultConvexReactMocks()
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
@@ -39,7 +42,7 @@ describe('SkillsIndex load-more observer', () => {
|
||||
|
||||
it('triggers one request for repeated intersection callbacks', async () => {
|
||||
const loadMorePaginated = vi.fn()
|
||||
usePaginatedQueryMock.mockReturnValue({
|
||||
convexReactMocks.usePaginatedQuery.mockReturnValue({
|
||||
results: [makeListResult('skill-0', 'Skill 0')],
|
||||
status: 'CanLoadMore',
|
||||
loadMore: loadMorePaginated,
|
||||
|
||||
@@ -2,12 +2,15 @@
|
||||
import { act, fireEvent, render, screen } from '@testing-library/react'
|
||||
import type { ReactNode } from 'react'
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import {
|
||||
convexReactMocks,
|
||||
resetConvexReactMocks,
|
||||
setupDefaultConvexReactMocks,
|
||||
} from './helpers/convexReactMocks'
|
||||
|
||||
import { SkillsIndex } from '../routes/skills/index'
|
||||
|
||||
const navigateMock = vi.fn()
|
||||
const useActionMock = vi.fn()
|
||||
const usePaginatedQueryMock = vi.fn()
|
||||
let searchMock: Record<string, unknown> = {}
|
||||
|
||||
vi.mock('@tanstack/react-router', () => ({
|
||||
@@ -20,19 +23,19 @@ vi.mock('@tanstack/react-router', () => ({
|
||||
}))
|
||||
|
||||
vi.mock('convex/react', () => ({
|
||||
useAction: (...args: unknown[]) => useActionMock(...args),
|
||||
usePaginatedQuery: (...args: unknown[]) => usePaginatedQueryMock(...args),
|
||||
useAction: (...args: unknown[]) => convexReactMocks.useAction(...args),
|
||||
useQuery: (...args: unknown[]) => convexReactMocks.useQuery(...args),
|
||||
usePaginatedQuery: (...args: unknown[]) => convexReactMocks.usePaginatedQuery(...args),
|
||||
}))
|
||||
|
||||
describe('SkillsIndex', () => {
|
||||
beforeEach(() => {
|
||||
usePaginatedQueryMock.mockReset()
|
||||
useActionMock.mockReset()
|
||||
resetConvexReactMocks()
|
||||
navigateMock.mockReset()
|
||||
searchMock = {}
|
||||
useActionMock.mockReturnValue(() => Promise.resolve([]))
|
||||
setupDefaultConvexReactMocks()
|
||||
// Default: return empty results with Exhausted status
|
||||
usePaginatedQueryMock.mockReturnValue({
|
||||
convexReactMocks.usePaginatedQuery.mockReturnValue({
|
||||
results: [],
|
||||
status: 'Exhausted',
|
||||
loadMore: vi.fn(),
|
||||
@@ -47,9 +50,9 @@ describe('SkillsIndex', () => {
|
||||
it('requests the first skills page', () => {
|
||||
render(<SkillsIndex />)
|
||||
// usePaginatedQuery should be called with the API endpoint and sort/dir args
|
||||
expect(usePaginatedQueryMock).toHaveBeenCalledWith(
|
||||
expect(convexReactMocks.usePaginatedQuery).toHaveBeenCalledWith(
|
||||
expect.anything(),
|
||||
{ sort: 'downloads', dir: 'desc', nonSuspiciousOnly: false },
|
||||
{ sort: 'downloads', dir: 'desc', highlightedOnly: false, nonSuspiciousOnly: false },
|
||||
{ initialNumItems: 25 },
|
||||
)
|
||||
})
|
||||
@@ -59,16 +62,100 @@ describe('SkillsIndex', () => {
|
||||
expect(screen.getByText('No skills match that filter.')).toBeTruthy()
|
||||
})
|
||||
|
||||
it('shows loading state instead of empty state when pagination is not exhausted', () => {
|
||||
// When status is not 'Exhausted', we should show loading, not "No skills match"
|
||||
convexReactMocks.usePaginatedQuery.mockReturnValue({
|
||||
results: [],
|
||||
status: 'CanLoadMore',
|
||||
loadMore: vi.fn(),
|
||||
})
|
||||
render(<SkillsIndex />)
|
||||
expect(screen.getByText('Loading skills…')).toBeTruthy()
|
||||
expect(screen.queryByText('No skills match that filter.')).toBeNull()
|
||||
})
|
||||
|
||||
it('keeps load-more reachable when results are empty but pagination can continue', () => {
|
||||
convexReactMocks.usePaginatedQuery.mockReturnValue({
|
||||
results: [],
|
||||
status: 'CanLoadMore',
|
||||
loadMore: vi.fn(),
|
||||
})
|
||||
render(<SkillsIndex />)
|
||||
expect(screen.getByRole('button', { name: 'Load more' })).toBeTruthy()
|
||||
})
|
||||
|
||||
it('shows loading indicator during pagination instead of hiding load more', () => {
|
||||
// When status is 'LoadingMore', keep showing the load more area with loading text
|
||||
const mockEntry = {
|
||||
skill: {
|
||||
_id: 'test-id',
|
||||
slug: 'test-skill',
|
||||
displayName: 'Test Skill',
|
||||
stats: { downloads: 0, installsAllTime: 0, stars: 0 },
|
||||
},
|
||||
latestVersion: null,
|
||||
owner: null,
|
||||
ownerHandle: null,
|
||||
}
|
||||
convexReactMocks.usePaginatedQuery.mockReturnValue({
|
||||
results: [mockEntry],
|
||||
status: 'LoadingMore',
|
||||
loadMore: vi.fn(),
|
||||
})
|
||||
render(<SkillsIndex />)
|
||||
// The load more button should still be visible with loading state
|
||||
expect(screen.getByText('Loading…')).toBeTruthy()
|
||||
})
|
||||
|
||||
it('handles LoadingMore with empty results gracefully', () => {
|
||||
// Edge case: user changes filter while loading more, results become empty
|
||||
convexReactMocks.usePaginatedQuery.mockReturnValue({
|
||||
results: [],
|
||||
status: 'LoadingMore',
|
||||
loadMore: vi.fn(),
|
||||
})
|
||||
render(<SkillsIndex />)
|
||||
// Should show loading message, not "No skills match"
|
||||
expect(screen.getByText('Loading skills…')).toBeTruthy()
|
||||
expect(screen.queryByText('No skills match that filter.')).toBeNull()
|
||||
// Keep the pagination control mounted so loading can continue.
|
||||
expect(screen.getByText('Loading…')).toBeTruthy()
|
||||
})
|
||||
|
||||
it('shows empty state immediately when search returns no results', async () => {
|
||||
// When searching and results are empty, show "No skills match" not "Loading"
|
||||
// This tests the hasQuery condition in the empty state logic
|
||||
searchMock = { q: 'nonexistent-skill-xyz' }
|
||||
const actionFn = vi.fn().mockResolvedValue([])
|
||||
convexReactMocks.useAction.mockReturnValue(actionFn)
|
||||
// Pagination is skipped in search mode, so status stays 'LoadingFirstPage'
|
||||
convexReactMocks.usePaginatedQuery.mockReturnValue({
|
||||
results: [],
|
||||
status: 'LoadingFirstPage',
|
||||
loadMore: vi.fn(),
|
||||
})
|
||||
vi.useFakeTimers()
|
||||
|
||||
render(<SkillsIndex />)
|
||||
await act(async () => {
|
||||
await vi.runAllTimersAsync()
|
||||
})
|
||||
|
||||
// Should show empty state, not loading
|
||||
expect(screen.getByText('No skills match that filter.')).toBeTruthy()
|
||||
expect(screen.queryByText('Loading skills…')).toBeNull()
|
||||
})
|
||||
|
||||
it('skips list query and calls search when query is set', async () => {
|
||||
searchMock = { q: 'remind' }
|
||||
const actionFn = vi.fn().mockResolvedValue([])
|
||||
useActionMock.mockReturnValue(actionFn)
|
||||
convexReactMocks.useAction.mockReturnValue(actionFn)
|
||||
vi.useFakeTimers()
|
||||
|
||||
render(<SkillsIndex />)
|
||||
|
||||
// usePaginatedQuery should be called with 'skip' when there's a search query
|
||||
expect(usePaginatedQueryMock).toHaveBeenCalledWith(expect.anything(), 'skip', {
|
||||
expect(convexReactMocks.usePaginatedQuery).toHaveBeenCalledWith(expect.anything(), 'skip', {
|
||||
initialNumItems: 25,
|
||||
})
|
||||
await act(async () => {
|
||||
@@ -98,7 +185,7 @@ describe('SkillsIndex', () => {
|
||||
.fn()
|
||||
.mockResolvedValueOnce(makeSearchResults(25))
|
||||
.mockResolvedValueOnce(makeSearchResults(50))
|
||||
useActionMock.mockReturnValue(actionFn)
|
||||
convexReactMocks.useAction.mockReturnValue(actionFn)
|
||||
vi.useFakeTimers()
|
||||
|
||||
render(<SkillsIndex />)
|
||||
@@ -129,7 +216,7 @@ describe('SkillsIndex', () => {
|
||||
makeSearchEntry({ slug: 'skill-b', displayName: 'Skill B', stars: 5, updatedAt: 200 }),
|
||||
makeSearchEntry({ slug: 'skill-c', displayName: 'Skill C', stars: 4, updatedAt: 999 }),
|
||||
])
|
||||
useActionMock.mockReturnValue(actionFn)
|
||||
convexReactMocks.useAction.mockReturnValue(actionFn)
|
||||
vi.useFakeTimers()
|
||||
|
||||
render(<SkillsIndex />)
|
||||
@@ -154,7 +241,7 @@ describe('SkillsIndex', () => {
|
||||
makeSearchResult('newer-low-score', 'Newer Low Score', 0.1, 2000),
|
||||
makeSearchResult('older-high-score', 'Older High Score', 0.9, 1000),
|
||||
])
|
||||
useActionMock.mockReturnValue(actionFn)
|
||||
convexReactMocks.useAction.mockReturnValue(actionFn)
|
||||
vi.useFakeTimers()
|
||||
|
||||
render(<SkillsIndex />)
|
||||
@@ -174,9 +261,20 @@ describe('SkillsIndex', () => {
|
||||
searchMock = { nonSuspicious: true }
|
||||
render(<SkillsIndex />)
|
||||
|
||||
expect(usePaginatedQueryMock).toHaveBeenCalledWith(
|
||||
expect(convexReactMocks.usePaginatedQuery).toHaveBeenCalledWith(
|
||||
expect.anything(),
|
||||
{ sort: 'downloads', dir: 'desc', nonSuspiciousOnly: true },
|
||||
{ sort: 'downloads', dir: 'desc', highlightedOnly: false, nonSuspiciousOnly: true },
|
||||
{ initialNumItems: 25 },
|
||||
)
|
||||
})
|
||||
|
||||
it('passes highlightedOnly to list query when filter is active', () => {
|
||||
searchMock = { highlighted: true }
|
||||
render(<SkillsIndex />)
|
||||
|
||||
expect(convexReactMocks.usePaginatedQuery).toHaveBeenCalledWith(
|
||||
expect.anything(),
|
||||
{ sort: 'downloads', dir: 'desc', highlightedOnly: true, nonSuspiciousOnly: false },
|
||||
{ initialNumItems: 25 },
|
||||
)
|
||||
})
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
import { useMutation, useQuery } from 'convex/react'
|
||||
import { useState } from 'react'
|
||||
import { api } from '../../convex/_generated/api'
|
||||
import type { Doc, Id } from '../../convex/_generated/dataModel'
|
||||
import { isModerator } from '../lib/roles'
|
||||
|
||||
type SkillCommentsPanelProps = {
|
||||
skillId: Id<'skills'>
|
||||
isAuthenticated: boolean
|
||||
me: Doc<'users'> | null
|
||||
}
|
||||
|
||||
export function SkillCommentsPanel({ skillId, isAuthenticated, me }: SkillCommentsPanelProps) {
|
||||
const addComment = useMutation(api.comments.add)
|
||||
const removeComment = useMutation(api.comments.remove)
|
||||
const [comment, setComment] = useState('')
|
||||
const [isSubmitting, setIsSubmitting] = useState(false)
|
||||
const [submitError, setSubmitError] = useState<string | null>(null)
|
||||
const [deleteError, setDeleteError] = useState<string | null>(null)
|
||||
const [deletingCommentId, setDeletingCommentId] = useState<Id<'comments'> | null>(null)
|
||||
const comments = useQuery(api.comments.listBySkill, { skillId, limit: 50 })
|
||||
|
||||
const submitComment = async () => {
|
||||
const body = comment.trim()
|
||||
if (!body || isSubmitting) return
|
||||
setIsSubmitting(true)
|
||||
setSubmitError(null)
|
||||
try {
|
||||
await addComment({ skillId, body })
|
||||
setComment('')
|
||||
} catch (error) {
|
||||
setSubmitError(error instanceof Error ? error.message : 'Failed to post comment')
|
||||
} finally {
|
||||
setIsSubmitting(false)
|
||||
}
|
||||
}
|
||||
|
||||
const deleteComment = async (commentId: Id<'comments'>) => {
|
||||
if (deletingCommentId) return
|
||||
setDeleteError(null)
|
||||
setDeletingCommentId(commentId)
|
||||
try {
|
||||
await removeComment({ commentId })
|
||||
} catch (error) {
|
||||
setDeleteError(error instanceof Error ? error.message : 'Failed to delete comment')
|
||||
} finally {
|
||||
setDeletingCommentId(null)
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="card">
|
||||
<h2 className="section-title" style={{ fontSize: '1.2rem', margin: 0 }}>
|
||||
Comments
|
||||
</h2>
|
||||
{isAuthenticated ? (
|
||||
<form
|
||||
onSubmit={(event) => {
|
||||
event.preventDefault()
|
||||
void submitComment()
|
||||
}}
|
||||
className="comment-form"
|
||||
>
|
||||
<textarea
|
||||
className="comment-input"
|
||||
rows={4}
|
||||
value={comment}
|
||||
onChange={(event) => setComment(event.target.value)}
|
||||
placeholder="Leave a note…"
|
||||
disabled={isSubmitting}
|
||||
/>
|
||||
{submitError ? <div className="report-dialog-error">{submitError}</div> : null}
|
||||
<button className="btn comment-submit" type="submit" disabled={isSubmitting}>
|
||||
{isSubmitting ? 'Posting…' : 'Post comment'}
|
||||
</button>
|
||||
</form>
|
||||
) : (
|
||||
<p className="section-subtitle">Sign in to comment.</p>
|
||||
)}
|
||||
{deleteError ? <div className="report-dialog-error">{deleteError}</div> : null}
|
||||
<div style={{ display: 'grid', gap: 12, marginTop: 16 }}>
|
||||
{(comments ?? []).length === 0 ? (
|
||||
<div className="stat">No comments yet.</div>
|
||||
) : (
|
||||
(comments ?? []).map((entry) => (
|
||||
<div key={entry.comment._id} className="comment-item">
|
||||
<div className="comment-body">
|
||||
<strong>@{entry.user?.handle ?? entry.user?.name ?? 'user'}</strong>
|
||||
<div className="comment-body-text">{entry.comment.body}</div>
|
||||
</div>
|
||||
{isAuthenticated && me && (me._id === entry.comment.userId || isModerator(me)) ? (
|
||||
<button
|
||||
className="btn comment-delete"
|
||||
type="button"
|
||||
onClick={() => void deleteComment(entry.comment._id)}
|
||||
disabled={Boolean(deletingCommentId) || isSubmitting}
|
||||
>
|
||||
{deletingCommentId === entry.comment._id ? 'Deleting…' : 'Delete'}
|
||||
</button>
|
||||
) : null}
|
||||
</div>
|
||||
))
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
+164
-1099
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,100 @@
|
||||
import { lazy, Suspense } from 'react'
|
||||
import type { Doc, Id } from '../../convex/_generated/dataModel'
|
||||
import { SkillVersionsPanel } from './SkillVersionsPanel'
|
||||
|
||||
const SkillDiffCard = lazy(() =>
|
||||
import('./SkillDiffCard').then((module) => ({ default: module.SkillDiffCard })),
|
||||
)
|
||||
|
||||
const SkillFilesPanel = lazy(() =>
|
||||
import('./SkillFilesPanel').then((module) => ({ default: module.SkillFilesPanel })),
|
||||
)
|
||||
|
||||
type SkillFile = Doc<'skillVersions'>['files'][number]
|
||||
|
||||
type SkillDetailTabsProps = {
|
||||
activeTab: 'files' | 'compare' | 'versions'
|
||||
setActiveTab: (tab: 'files' | 'compare' | 'versions') => void
|
||||
onCompareIntent: () => void
|
||||
readmeContent: string | null
|
||||
readmeError: string | null
|
||||
latestFiles: SkillFile[]
|
||||
latestVersionId: Id<'skillVersions'> | null
|
||||
skill: Doc<'skills'>
|
||||
diffVersions: Doc<'skillVersions'>[] | undefined
|
||||
versions: Doc<'skillVersions'>[] | undefined
|
||||
nixPlugin: boolean
|
||||
}
|
||||
|
||||
export function SkillDetailTabs({
|
||||
activeTab,
|
||||
setActiveTab,
|
||||
onCompareIntent,
|
||||
readmeContent,
|
||||
readmeError,
|
||||
latestFiles,
|
||||
latestVersionId,
|
||||
skill,
|
||||
diffVersions,
|
||||
versions,
|
||||
nixPlugin,
|
||||
}: SkillDetailTabsProps) {
|
||||
return (
|
||||
<div className="card tab-card">
|
||||
<div className="tab-header">
|
||||
<button
|
||||
className={`tab-button${activeTab === 'files' ? ' is-active' : ''}`}
|
||||
type="button"
|
||||
onClick={() => setActiveTab('files')}
|
||||
>
|
||||
Files
|
||||
</button>
|
||||
<button
|
||||
className={`tab-button${activeTab === 'compare' ? ' is-active' : ''}`}
|
||||
type="button"
|
||||
onClick={() => setActiveTab('compare')}
|
||||
onMouseEnter={() => {
|
||||
onCompareIntent()
|
||||
void import('./SkillDiffCard')
|
||||
}}
|
||||
onFocus={() => {
|
||||
onCompareIntent()
|
||||
void import('./SkillDiffCard')
|
||||
}}
|
||||
>
|
||||
Compare
|
||||
</button>
|
||||
<button
|
||||
className={`tab-button${activeTab === 'versions' ? ' is-active' : ''}`}
|
||||
type="button"
|
||||
onClick={() => setActiveTab('versions')}
|
||||
>
|
||||
Versions
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{activeTab === 'files' ? (
|
||||
<Suspense fallback={<div className="tab-body stat">Loading file viewer…</div>}>
|
||||
<SkillFilesPanel
|
||||
versionId={latestVersionId}
|
||||
readmeContent={readmeContent}
|
||||
readmeError={readmeError}
|
||||
latestFiles={latestFiles}
|
||||
/>
|
||||
</Suspense>
|
||||
) : null}
|
||||
|
||||
{activeTab === 'compare' ? (
|
||||
<div className="tab-body">
|
||||
<Suspense fallback={<div className="stat">Loading diff viewer…</div>}>
|
||||
<SkillDiffCard skill={skill} versions={diffVersions ?? []} variant="embedded" />
|
||||
</Suspense>
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
{activeTab === 'versions' ? (
|
||||
<SkillVersionsPanel versions={versions} nixPlugin={nixPlugin} skillSlug={skill.slug} />
|
||||
) : null}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
import { fireEvent, render, screen, waitFor } from '@testing-library/react'
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import type { Doc, Id } from '../../convex/_generated/dataModel'
|
||||
import { SkillFilesPanel } from './SkillFilesPanel'
|
||||
|
||||
const getFileTextMock = vi.fn()
|
||||
|
||||
vi.mock('convex/react', () => ({
|
||||
useAction: () => getFileTextMock,
|
||||
}))
|
||||
|
||||
vi.mock('react-markdown', () => ({
|
||||
default: ({ children }: { children: string }) => <div>{children}</div>,
|
||||
}))
|
||||
|
||||
vi.mock('remark-gfm', () => ({
|
||||
default: {},
|
||||
}))
|
||||
|
||||
type SkillFile = Doc<'skillVersions'>['files'][number]
|
||||
|
||||
function makeFile(path: string, size: number): SkillFile {
|
||||
return { path, size } as unknown as SkillFile
|
||||
}
|
||||
|
||||
describe('SkillFilesPanel', () => {
|
||||
beforeEach(() => {
|
||||
getFileTextMock.mockReset()
|
||||
})
|
||||
|
||||
it('caches loaded files and avoids duplicate fetches', async () => {
|
||||
getFileTextMock.mockResolvedValue({
|
||||
text: 'echo hello',
|
||||
size: 10,
|
||||
sha256: 'a'.repeat(64),
|
||||
})
|
||||
|
||||
render(
|
||||
<SkillFilesPanel
|
||||
versionId={'skillVersions:1' as Id<'skillVersions'>}
|
||||
readmeContent={'# skill'}
|
||||
readmeError={null}
|
||||
latestFiles={[makeFile('scripts/run.sh', 10)]}
|
||||
/>,
|
||||
)
|
||||
|
||||
const fileButton = screen.getByRole('button', { name: /scripts\/run\.sh/i })
|
||||
fireEvent.click(fileButton)
|
||||
|
||||
await screen.findByText('echo hello')
|
||||
|
||||
fireEvent.click(fileButton)
|
||||
|
||||
await waitFor(() => {
|
||||
expect(getFileTextMock).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
})
|
||||
|
||||
it('ignores stale responses when newer file selection is active', async () => {
|
||||
const resolvers: Record<string, (value: { text: string; size: number; sha256: string }) => void> = {}
|
||||
|
||||
getFileTextMock.mockImplementation(
|
||||
({ path }: { path: string }) =>
|
||||
new Promise<{ text: string; size: number; sha256: string }>((resolve) => {
|
||||
resolvers[path] = resolve
|
||||
}),
|
||||
)
|
||||
|
||||
render(
|
||||
<SkillFilesPanel
|
||||
versionId={'skillVersions:1' as Id<'skillVersions'>}
|
||||
readmeContent={'# skill'}
|
||||
readmeError={null}
|
||||
latestFiles={[makeFile('a.txt', 5), makeFile('b.txt', 6)]}
|
||||
/>,
|
||||
)
|
||||
|
||||
fireEvent.click(screen.getByRole('button', { name: /a\.txt/i }))
|
||||
fireEvent.click(screen.getByRole('button', { name: /b\.txt/i }))
|
||||
|
||||
resolvers['a.txt']({ text: 'alpha', size: 5, sha256: 'b'.repeat(64) })
|
||||
resolvers['b.txt']({ text: 'beta', size: 6, sha256: 'c'.repeat(64) })
|
||||
|
||||
await screen.findByText('beta')
|
||||
expect(screen.queryByText('alpha')).toBeNull()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,164 @@
|
||||
import { useAction } from 'convex/react'
|
||||
import { useCallback, useEffect, useRef, useState } from 'react'
|
||||
import ReactMarkdown from 'react-markdown'
|
||||
import remarkGfm from 'remark-gfm'
|
||||
import { api } from '../../convex/_generated/api'
|
||||
import type { Doc, Id } from '../../convex/_generated/dataModel'
|
||||
import { formatBytes } from './skillDetailUtils'
|
||||
|
||||
type SkillFile = Doc<'skillVersions'>['files'][number]
|
||||
|
||||
type SkillFilesPanelProps = {
|
||||
versionId: Id<'skillVersions'> | null
|
||||
readmeContent: string | null
|
||||
readmeError: string | null
|
||||
latestFiles: SkillFile[]
|
||||
}
|
||||
|
||||
export function SkillFilesPanel({
|
||||
versionId,
|
||||
readmeContent,
|
||||
readmeError,
|
||||
latestFiles,
|
||||
}: SkillFilesPanelProps) {
|
||||
const getFileText = useAction(api.skills.getFileText)
|
||||
const [selectedPath, setSelectedPath] = useState<string | null>(null)
|
||||
const [fileContent, setFileContent] = useState<string | null>(null)
|
||||
const [fileMeta, setFileMeta] = useState<{ size: number; sha256: string } | null>(null)
|
||||
const [fileError, setFileError] = useState<string | null>(null)
|
||||
const [isLoading, setIsLoading] = useState(false)
|
||||
const isMounted = useRef(true)
|
||||
const requestId = useRef(0)
|
||||
const fileCache = useRef(new Map<string, { text: string; size: number; sha256: string }>())
|
||||
|
||||
useEffect(() => {
|
||||
isMounted.current = true
|
||||
return () => {
|
||||
isMounted.current = false
|
||||
requestId.current += 1
|
||||
}
|
||||
}, [])
|
||||
|
||||
useEffect(() => {
|
||||
requestId.current += 1
|
||||
|
||||
setSelectedPath(null)
|
||||
setFileContent(null)
|
||||
setFileMeta(null)
|
||||
setFileError(null)
|
||||
setIsLoading(false)
|
||||
|
||||
if (versionId === null) return
|
||||
}, [versionId])
|
||||
|
||||
const handleSelect = useCallback(
|
||||
(path: string) => {
|
||||
if (!versionId) return
|
||||
const cacheKey = `${versionId}:${path}`
|
||||
const cached = fileCache.current.get(cacheKey)
|
||||
|
||||
requestId.current += 1
|
||||
const current = requestId.current
|
||||
setSelectedPath(path)
|
||||
setFileError(null)
|
||||
if (cached) {
|
||||
setFileContent(cached.text)
|
||||
setFileMeta({ size: cached.size, sha256: cached.sha256 })
|
||||
setIsLoading(false)
|
||||
return
|
||||
}
|
||||
|
||||
setFileContent(null)
|
||||
setFileMeta(null)
|
||||
setIsLoading(true)
|
||||
void getFileText({ versionId, path })
|
||||
.then((data) => {
|
||||
if (!isMounted.current) return
|
||||
if (requestId.current !== current) return
|
||||
fileCache.current.set(cacheKey, data)
|
||||
setFileContent(data.text)
|
||||
setFileMeta({ size: data.size, sha256: data.sha256 })
|
||||
setIsLoading(false)
|
||||
})
|
||||
.catch((error) => {
|
||||
if (!isMounted.current) return
|
||||
if (requestId.current !== current) return
|
||||
setFileError(error instanceof Error ? error.message : 'Failed to load file')
|
||||
setIsLoading(false)
|
||||
})
|
||||
},
|
||||
[getFileText, versionId],
|
||||
)
|
||||
|
||||
return (
|
||||
<div className="tab-body">
|
||||
<div>
|
||||
<h2 className="section-title" style={{ fontSize: '1.2rem', margin: 0 }}>
|
||||
SKILL.md
|
||||
</h2>
|
||||
<div className="markdown">
|
||||
{readmeContent ? (
|
||||
<ReactMarkdown remarkPlugins={[remarkGfm]}>{readmeContent}</ReactMarkdown>
|
||||
) : readmeError ? (
|
||||
<div className="stat">Failed to load SKILL.md: {readmeError}</div>
|
||||
) : (
|
||||
<div>Loading…</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
<div className="file-browser">
|
||||
<div className="file-list">
|
||||
<div className="file-list-header">
|
||||
<h3 className="section-title" style={{ fontSize: '1.05rem', margin: 0 }}>
|
||||
Files
|
||||
</h3>
|
||||
<span className="section-subtitle" style={{ margin: 0 }}>
|
||||
{latestFiles.length} total
|
||||
</span>
|
||||
</div>
|
||||
<div className="file-list-body">
|
||||
{latestFiles.length === 0 ? (
|
||||
<div className="stat">No files available.</div>
|
||||
) : (
|
||||
latestFiles.map((file) => (
|
||||
<button
|
||||
key={file.path}
|
||||
className={`file-row file-row-button${
|
||||
selectedPath === file.path ? ' is-active' : ''
|
||||
}`}
|
||||
type="button"
|
||||
onClick={() => handleSelect(file.path)}
|
||||
aria-current={selectedPath === file.path ? 'true' : undefined}
|
||||
>
|
||||
<span className="file-path">{file.path}</span>
|
||||
<span className="file-meta">{formatBytes(file.size)}</span>
|
||||
</button>
|
||||
))
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
<div className="file-viewer">
|
||||
<div className="file-viewer-header">
|
||||
<div className="file-path">{selectedPath ?? 'Select a file'}</div>
|
||||
{fileMeta ? (
|
||||
<span className="file-meta">
|
||||
{formatBytes(fileMeta.size)} · {fileMeta.sha256.slice(0, 12)}…
|
||||
</span>
|
||||
) : null}
|
||||
</div>
|
||||
<div className="file-viewer-body">
|
||||
{isLoading ? (
|
||||
<div className="stat">Loading…</div>
|
||||
) : fileError ? (
|
||||
<div className="stat">Failed to load file: {fileError}</div>
|
||||
) : fileContent ? (
|
||||
<pre className="file-viewer-code">{fileContent}</pre>
|
||||
) : (
|
||||
<div className="stat">Select a file to preview.</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user