Compare commits

...
Author SHA1 Message Date
Peter Steinberger 53214abd08 fix: finalize proxy env support + changelog credits (#363) (thanks @kerrypotter) 2026-02-25 12:13:25 +00:00
Jarvis 4a6f4391c4 fix: use EnvHttpProxyAgent for proper proxy support
Address review feedback:
- Use undici's EnvHttpProxyAgent instead of ProxyAgent. This properly
  handles HTTPS_PROXY vs HTTP_PROXY per-scheme, respects NO_PROXY,
  and uses connect.timeout instead of requestTls.
- Update docs to mention NO_PROXY support.
2026-02-25 12:11:43 +00:00
Jarvis aa0a97bd35 fix: respect HTTP_PROXY/HTTPS_PROXY environment variables
The CLI creates a custom undici Agent via setGlobalDispatcher() which
overrides any proxy configuration. Since Node.js native fetch (backed
by undici) does not automatically respect HTTP_PROXY/HTTPS_PROXY env
vars, the CLI fails with 'fetch failed' on systems that require a
proxy for outbound connections.

Import ProxyAgent from undici and use it when any of the standard proxy
environment variables (HTTPS_PROXY, HTTP_PROXY, https_proxy, http_proxy)
is set. When no proxy variable is present, behavior is unchanged.

Also adds proxy documentation to cli.md and a troubleshooting entry.
2026-02-25 12:11:18 +00:00
Phineas1500 311bf1a88a fix: guard de-escalation on clean status and fix alreadyFlagged blocking clean verdicts
Address review feedback:
- Guard rescan de-escalation with `status === 'clean'` so pending/unknown
  verdicts don't accidentally clear the suspicious flag
- Fix approveSkillByHashInternal where `alreadyFlagged` in the condition
  `(isSuspicious || alreadyFlagged) && !bypassSuspicious` prevented clean
  verdicts from reaching the isClean branch that properly checks whether
  a different scanner set the flag
2026-02-25 12:09:41 +00:00
Phineas1500 8343f0bb23 fix: clear stale suspicious flag when VT verdict improves to clean
The daily VT rescan updated vtAnalysis on the version but only called
escalateByVtInternal for suspicious/malicious verdicts. When a verdict
improved from suspicious to clean, the version's vtAnalysis was updated
(website shows "Benign") but the skill's moderationFlags kept the stale
"flagged.suspicious" entry (CLI warns "suspicious"). Now the rescan
calls approveSkillByHashInternal to clear the flag on de-escalation.
2026-02-25 12:09:41 +00:00
Peter Steinberger 3b73a09d36 fix: keep denormalized badge reads consistent (#441) (thanks @sethconvex) 2026-02-25 03:00:34 +00:00
Seth RaphaelandClaude Opus 4.6 412249d2d1 fix: use destructuring instead of undefined assignment in removeSkillBadge
Avoids leaving an explicit undefined key in the badges object which
could fail Convex validation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:00:34 +00:00
Seth RaphaelandClaude Opus 4.6 480125d859 feat: add backfill for denormalized skill badges, clean up fallback
- Add backfillDenormalizedBadgesInternal: syncs skillBadges table →
  skill.badges field so listing/search reads are correct
- Simplify hydrateResults fallback

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:00:34 +00:00
Seth RaphaelandClaude Opus 4.6 0ab1d1e051 fix: restore fallback in hydrateResults for un-backfilled embeddings
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:00:34 +00:00
Seth RaphaelandClaude Opus 4.6 85374fa44b perf: eliminate redundant badge reads and add embedding lookup table
- Remove badge table queries from listing and search paths (~200 queries
  per page load eliminated). Use denormalized skill.badges field instead.
- Sync skill.badges when badges are mutated (upsertSkillBadge/removeSkillBadge).
- Add embeddingSkillMap lookup table (~100 bytes/doc) so search hydration
  can skip reading full skillEmbeddings docs (~12KB each with vector).
- Remove dead badge query exports from search module.
- Reduce lexical fallback scan limit from 1200 to 500.
- Add backfill mutation for embeddingSkillMap with graceful fallback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:00:34 +00:00
Seth RaphaelandClaude Opus 4.6 c319e46c8b perf: cap badge query to 10 records per skill
Skills should never have more than a handful of badge records.
Using .take(10) instead of .collect() avoids unbounded reads.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:00:34 +00:00
Seth RaphaelandClaude Opus 4.6 f17087d1f4 perf: reduce db bandwidth by splitting stat processing into two frequencies
The 5-minute stat event processor was patching skill documents on every run,
which invalidated listPublicPageV2 reactive queries for ALL subscribers —
causing a thundering herd responsible for ~17 TB (59%) of the 28.65 TB
monthly db bandwidth.

Split into two paths:
- Daily stats (15-min cron): writes to skillDailyStats only, no skill doc patches
- Skill doc sync (6-hour cron): patches skill documents with accumulated deltas

Also skip reading version docs in listPublicPageV2 and search hydration
(version data is only needed on detail pages, not listings).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:00:34 +00:00
LK da4469e1e0 fix(rate-limit): skip ip consumption for authenticated requests 2026-02-25 02:56:20 +00:00
LK 42a4648475 refactor(http): parse rate-limit headers once per error 2026-02-25 02:56:20 +00:00
Lukeandgreptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> edc8ec274b Update docs/http-api.md
fix header precedence in docs to match code

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-02-25 02:56:20 +00:00
LK 15b1a05fee fix(rate-limit): make limiter auth-aware for shared proxy ips 2026-02-25 02:56:20 +00:00
LK 1e216c03c6 fix(rate-limit): standardize retry-after and improve 429 ux 2026-02-25 02:56:20 +00:00
ɐʞsǝs 0a0b2e6cb1 pin the th version for better stability / security 2026-02-25 02:50:17 +00:00
ɐʞsǝsandgreptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> ac6770acff Update .github/workflows/secret-scan.yml
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-02-25 02:50:17 +00:00
ɐʞsǝs bfc87e5932 refined message to give better guidance 2026-02-25 02:50:17 +00:00
ɐʞsǝs 3e1bd19a45 Add secret scanning workflow using TruffleHog
This ensures that a given PR will immediately error on vulnerable, live credentials found by trufflehog
2026-02-25 02:50:17 +00:00
Peter Steinberger d71b747d1c fix: harden VT fallback activation rules (#300) (thanks @superlowburn) 2026-02-25 02:48:57 +00:00
Steve 4d211fcf73 fix: check moderation reason before activating skills
- Prevent activating skills with quality.low moderation reason
- Add skill lookup and moderationReason check in 3 locations where skills are activated
- This ensures quality gate quarantine is not bypassed when VT scan is unavailable or stale

Resolves review comments on #300
2026-02-25 02:48:57 +00:00
SteveandClaude Opus 4.6 bc5ab8f3e1 fix: activate skills when VT scan is unavailable or stales out
Published skills stay permanently hidden in search when VirusTotal
cannot produce a verdict. Three code paths leave moderationStatus as
'hidden' with no recovery:

1. VT_API_KEY not configured — scan skipped, skill stays hidden
2. VT hash not found after 10 poll attempts — marked stale, stays hidden
3. VT hash found but no Code Insight after 10 attempts — same

Fix: call setSkillModerationStatusActiveInternal in all three paths so
the skill becomes searchable. If VT later returns a malicious verdict,
approveSkillByHashInternal will correctly re-hide and flag it.

Closes #139

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 02:48:57 +00:00
Peter Steinberger bb528ea4b9 fix: dedupe OpenAI response parsing (#502) (thanks @ianalloway) 2026-02-25 02:46:35 +00:00
Nimrod Gutman 464a04c1e5 fix(vt): prevent pending scan starvation and retry unresolved results (#468)
* fix(vt): prevent pending scan starvation and retry unresolved results

* fix(vt): remove exhaustive pending-scan clamp
2026-02-23 21:26:47 -06:00
Peter Steinberger f4f8e7276f docs: clarify skill delete/undelete permissions 2026-02-18 17:24:09 +01:00
Peter Steinberger 3ccf2e05f5 fix: make users reclaim transfer root slug ownership in place 2026-02-18 17:10:08 +01:00
Shadow 0ee2872f5b fix: prune deleted skill backups 2026-02-17 11:02:57 -06:00
Peter Steinberger 5112d1b215 docs: cross-link README and vision 2026-02-17 17:35:28 +01:00
Peter Steinberger 275a170f15 docs: clarify MCP support policy in vision 2026-02-17 17:33:54 +01:00
Peter Steinberger 17aa24baf9 docs: refine vision priorities and security stance 2026-02-17 17:31:24 +01:00
Peter Steinberger f01476757a fix(ui): remove upvote-style metric and clarify installs icon 2026-02-17 15:08:47 +01:00
Peter Steinberger fe011d00fd test(e2e): update search menu route expectation 2026-02-17 15:08:42 +01:00
Peter Steinberger 30ae099825 fix: avoid paginated fallback in skills count query (#76) 2026-02-17 00:32:22 +01:00
Peter Steinberger 812641342d fix: harden /skills count when globalStats is missing (#76) 2026-02-17 00:25:46 +01:00
Peter Steinberger 88848c224c fix: keep public skill counts consistent after moderation/visibility changes (#76) (thanks @rknoche6) 2026-02-17 00:25:46 +01:00
c107adabac display total skills count on /skills page (#76)
Co-authored-by: rknoche <richard.knoche@holidaycheck.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-17 00:25:46 +01:00
58 changed files with 2637 additions and 443 deletions
+33
View File
@@ -0,0 +1,33 @@
name: "Security Gate: Secret Scanning"
on:
pull_request:
branches: [main, master]
jobs:
trufflehog:
name: Scan for Verified Secrets
runs-on: ubuntu-latest
permissions:
contents: read # Required to scan the code in the PR
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0 # necessary to support the scoping requirements below
- name: TruffleHog OSS
id: trufflehog
uses: trufflesecurity/trufflehog@e64309e4514a601c7d23f336688782a229a4a754 # Pin to current stable
with:
path: ./
base: ${{ github.event.pull_request.base.sha }} # scope it to the committed files
head: ${{ github.event.pull_request.head.sha }}
extra_args: --only-verified --debug
- name: Notify on Failure
if: steps.trufflehog.outcome == 'failure'
run: |
echo "::error::Verified secrets found! This PR contains live credentials that must be rotated immediately."
echo "::notice::If these secrets are already in the commit history, they cannot be removed via a simple removal commit/push. A repository owner can contact GitHub Support to purge the cached data: https://support.github.com/contact/private-information"
exit 1
+11
View File
@@ -9,6 +9,7 @@
- Skills/Web: show skill owner avatar + handle on skill cards, lists, and detail pages (#312) (thanks @ianalloway).
- Skills/Web: add file viewer for skill version files on detail page (#44) (thanks @regenrek).
- CLI: add `uninstall` command for skills (#241) (thanks @superlowburn).
- CI/Security: add TruffleHog pull-request scanning for verified leaked credentials (#505) (thanks @akses0).
### Changed
- Quality gate: language-aware word counting (`Intl.Segmenter`) and new `cjkChars` signal to reduce false rejects for non-Latin docs.
@@ -16,11 +17,17 @@
- API performance: batch resolve skill/soul tags in v1 list/get endpoints (fewer action->query round-trips) (#112) (thanks @mkrokosz).
- Skills: reserve deleted slugs for prior owners (90-day cooldown) to prevent squatting; add admin reclaim flow (#298) (thanks @autogame-17).
- Moderation: ban flow soft-deletes owned skills (reversible) and removes them from vector search (#298) (thanks @autogame-17).
- LLM helpers: centralize OpenAI Responses text extraction for changelog/summary/eval flows (#502) (thanks @ianalloway).
- Rate limiting: apply authenticated quotas by user bucket (vs shared IP), emit delay-based reset headers, and improve CLI 429 guidance/retries (#412) (thanks @lc0rp).
- Search/listing performance: cut embedding hydration and badge read bandwidth via `embeddingSkillMap` + denormalized skill badges; shift stat-doc sync to low-frequency cron (#441) (thanks @sethconvex).
### Fixed
- Admin API: `POST /api/v1/users/reclaim` now performs non-destructive root-slug owner transfer
(preserves existing skill versions/stats/metadata) and clears active slug reservations.
- Users: sync handle on ensure when GitHub login changes (#293) (thanks @christianhpoe).
- Users/Auth: throttle GitHub profile sync on login; also sync avatar when it changes (#312) (thanks @ianalloway).
- Upload gate: fetch GitHub account age by immutable account ID (prevents username swaps) (#116) (thanks @mkrokosz).
- VT fallback: activate only VT-pending hidden skills when scans are unavailable/stale; keep quality/scanner-blocked skills hidden (#300) (thanks @superlowburn).
- API: return proper status codes for delete/undelete errors (#35) (thanks @sergical).
- API: for owners, return clearer status/messages for hidden/soft-deleted skills instead of a generic 404.
- Web: allow copying OpenClaw scan summary text (thanks @borisolver, #322).
@@ -30,6 +37,10 @@
- Skills: keep global sorting across pagination on `/skills` (thanks @CodeBBakGoSu, #98).
- Skills: allow updating skill description/summary from frontmatter on subsequent publishes (#312) (thanks @ianalloway).
- Skills/Web: prevent filtered pagination dead-ends and loading-state flicker on `/skills`; move highlighted browse filtering into server list query (#339) (thanks @Marvae).
- Web: align `/skills` total count with public visibility and format header count (thanks @rknoche6, #76).
- Skills/Web: centralize public visibility checks and keep `globalStats` skill counts in sync incrementally; remove duplicate `/skills` default-sort fallback and share browse test mocks (thanks @rknoche6, #76).
- Moderation: clear stale `flagged.suspicious` flags when VirusTotal rescans improve to clean verdicts (#418) (thanks @Phineas1500).
- CLI: respect `HTTPS_PROXY`/`HTTP_PROXY`/`NO_PROXY` env vars for outbound registry requests, with troubleshooting docs (#363) (thanks @kerrypotter).
## 0.6.1 - 2026-02-13
+8
View File
@@ -13,6 +13,7 @@ onlycrabs.ai is the **SOUL.md registry**: publish and share system lore the same
Live: `https://clawhub.ai`
onlycrabs.ai: `https://onlycrabs.ai`
Vision: [`VISION.md`](VISION.md)
## What you can do with it
@@ -49,6 +50,13 @@ Common CLI flows:
Docs: `docs/quickstart.md`, `docs/cli.md`.
### Removal permissions
- `clawhub uninstall <slug>` only removes a local install on your machine.
- Uploaded registry skills use soft-delete/restore (`clawhub delete <slug>` / `clawhub undelete <slug>` or API equivalents).
- Soft-delete/restore is allowed for the skill owner, moderators, and admins.
- Hard delete is admin-only (management tools / ban flows).
## Telemetry
+94
View File
@@ -0,0 +1,94 @@
## OpenClaw Vision
OpenClaw is the AI that actually does things.
It runs on your devices, in your channels, with your rules.
This document explains the current state and direction of the project.
We are still early, so iteration is fast.
Project overview and developer docs: [`README.md`](README.md)
OpenClaw started as my personal playground to learn AI and build something genuinely useful:
an assistant that can run real tasks on my computer.
It evolved through several names and shells: Warelay -> Clawdbot -> Moltbot -> OpenClaw.
The goal? A personal assistant that's easy to use, supports a wide range of platforms, and respects your privacy and security.
The current focus is:
Priority:
- Security and safe defaults
- Bug fixes and stability
- Setup reliability and first-run UX
Next priorities:
- Supporting all major model providers
- Improving support for major messaging channels (and adding a few high-demand ones)
- Performance and test infrastructure
- Better computer-use and agent harness capabilities
- Ergonomics across CLI and web frontend
- Companion apps on macOS, iOS, Android, Windows, and Linux
## Security
Security in OpenClaw is a deliberate tradeoff: strong defaults without killing capability.
The goal is to stay powerful for real work while making risky paths explicit and operator-controlled.
Canonical security policy and reporting:
- https://github.com/openclaw/openclaw/blob/main/SECURITY.md
We prioritize secure defaults, but we also expose clear knobs for trusted high-power workflows.
## Plugins & Memory
OpenClaw has an extensive plugin API.
Core stays lean; optional capability should usually ship as plugins.
Preferred plugin path is npm package distribution plus local extension loading for development.
If you build a plugin, please host and maintain it in your own repository.
The bar for adding optional plugins to core is intentionally high.
Memory is a special plugin slot where only one memory plugin can be active at a time.
Today we ship multiple memory options; over time we plan to converge on one recommended default path.
### Skills
We still ship some bundled skills for baseline UX.
New skills should be published to ClawHub first (`clawhub.ai`), not added to core by default.
Core skill additions should be rare and require a strong product or security reason.
### MCP Support
OpenClaw supports MCP through `mcporter`: https://github.com/steipete/mcporter
This keeps MCP integration flexible and decoupled from core runtime:
- add or change MCP servers without restarting the gateway
- keep core tool/context surface lean
- reduce MCP churn impact on core stability and security
For now, we prefer this bridge model over building first-class MCP runtime into core.
If there is an MCP server or feature `mcporter` does not support yet, please open an issue there.
### Setup
OpenClaw is currently terminal-first by design.
This keeps setup explicit: users see docs, auth, permissions, and security posture up front.
Long term, we want easier onboarding flows as hardening matures.
We do not want convenience wrappers that hide critical security decisions from users.
### Why TypeScript?
OpenClaw is primarily an orchestration system: prompts, tools, protocols, and integrations.
TypeScript was chosen to keep OpenClaw hackable by default.
It is widely known, fast to iterate in, and easy to read, modify, and extend.
## What We Will Not Merge (For Now)
- New core skills when they can live on ClawHub
- Commercial service integrations that do not clearly fit the model-provider category
- Wrapper channels around already supported channels without a clear capability or security gap
- First-class MCP runtime in core when `mcporter` already provides the integration path
- Heavy orchestration layers that duplicate existing agent and tool infrastructure
This list is a roadmap guardrail, not a law of physics.
Strong user demand and strong technical rationale can change it.
+2
View File
@@ -48,6 +48,7 @@ import type * as lib_githubImport from "../lib/githubImport.js";
import type * as lib_githubProfileSync from "../lib/githubProfileSync.js";
import type * as lib_githubRestoreHelpers from "../lib/githubRestoreHelpers.js";
import type * as lib_githubSoulBackup from "../lib/githubSoulBackup.js";
import type * as lib_globalStats from "../lib/globalStats.js";
import type * as lib_httpHeaders from "../lib/httpHeaders.js";
import type * as lib_httpRateLimit from "../lib/httpRateLimit.js";
import type * as lib_leaderboards from "../lib/leaderboards.js";
@@ -137,6 +138,7 @@ declare const fullApi: ApiFromModules<{
"lib/githubProfileSync": typeof lib_githubProfileSync;
"lib/githubRestoreHelpers": typeof lib_githubRestoreHelpers;
"lib/githubSoulBackup": typeof lib_githubSoulBackup;
"lib/globalStats": typeof lib_globalStats;
"lib/httpHeaders": typeof lib_httpHeaders;
"lib/httpRateLimit": typeof lib_httpRateLimit;
"lib/leaderboards": typeof lib_leaderboards;
+21 -2
View File
@@ -19,18 +19,37 @@ crons.interval(
crons.interval(
'skill-stats-backfill',
{ minutes: 10 },
{ hours: 6 },
internal.statsMaintenance.runSkillStatBackfillInternal,
{ batchSize: 200, maxBatches: 5 },
)
// Runs frequently to keep dailyStats/trending accurate,
// but does NOT patch skill documents (only writes to skillDailyStats).
crons.interval(
'skill-stat-events',
{ minutes: 5 },
{ minutes: 15 },
internal.skillStatEvents.processSkillStatEventsAction,
{},
)
// Syncs accumulated stat deltas to skill documents every 6 hours.
// Runs infrequently to avoid thundering-herd reactive query invalidation.
// Uses processedAt field to track progress (independent of the action cursor).
crons.interval(
'skill-doc-stat-sync',
{ hours: 6 },
internal.skillStatEvents.processSkillStatEventsInternal,
{ batchSize: 500 },
)
crons.interval(
'global-stats-update',
{ minutes: 60 },
internal.statsMaintenance.updateGlobalStatsInternal,
{},
)
crons.interval('vt-pending-scans', { minutes: 5 }, internal.vt.pollPendingScans, { batchSize: 100 })
crons.interval('vt-cache-backfill', { minutes: 30 }, internal.vt.backfillActiveSkillsVTCache, {
+1
View File
@@ -435,6 +435,7 @@ export const seedSkillMutation = internalMutation({
visibility: 'latest-approved',
updatedAt: now,
})
await ctx.db.insert('embeddingSkillMap', { embeddingId, skillId })
await ctx.db.patch(skillId, {
latestVersionId: versionId,
+1
View File
@@ -39,6 +39,7 @@ export type SyncGitHubBackupsResult = {
skillsScanned: number
skillsSkipped: number
skillsBackedUp: number
skillsDeleted: number
skillsMissingVersion: number
skillsMissingOwner: number
errors: number
+65
View File
@@ -7,9 +7,12 @@ import type { ActionCtx } from './_generated/server'
import { internalAction } from './_generated/server'
import {
backupSkillToGitHub,
deleteGitHubSkillBackup,
fetchGitHubSkillMeta,
getGitHubBackupContext,
isGitHubBackupConfigured,
listGitHubSkillBackupEntries,
normalizeOwner,
} from './lib/githubBackup'
const DEFAULT_BATCH_SIZE = 50
@@ -35,6 +38,7 @@ export type GitHubBackupSyncStats = {
skillsScanned: number
skillsSkipped: number
skillsBackedUp: number
skillsDeleted: number
skillsMissingVersion: number
skillsMissingOwner: number
errors: number
@@ -87,6 +91,7 @@ export async function syncGitHubBackupsInternalHandler(
skillsScanned: 0,
skillsSkipped: 0,
skillsBackedUp: 0,
skillsDeleted: 0,
skillsMissingVersion: 0,
skillsMissingOwner: 0,
errors: 0,
@@ -166,9 +171,69 @@ export async function syncGitHubBackupsInternalHandler(
if (isDone) break
}
await pruneDeletedSkillBackups(ctx, context, dryRun, stats)
return { stats, cursor, isDone }
}
async function pruneDeletedSkillBackups(
ctx: ActionCtx,
context: Awaited<ReturnType<typeof getGitHubBackupContext>>,
dryRun: boolean,
stats: GitHubBackupSyncStats,
) {
let entries: Awaited<ReturnType<typeof listGitHubSkillBackupEntries>>
try {
entries = await listGitHubSkillBackupEntries(context)
} catch (error) {
console.error('GitHub backup cleanup list failed', error)
stats.errors += 1
return
}
for (const entry of entries) {
try {
const skill = (await ctx.runQuery(internal.skills.getSkillBySlugInternal, {
slug: entry.slug,
})) as Doc<'skills'> | null
if (!skill || skill.softDeletedAt) {
await deleteBackupIfNeeded(context, entry, dryRun, stats)
continue
}
const owner = (await ctx.runQuery(internal.users.getByIdInternal, {
userId: skill.ownerUserId,
})) as Doc<'users'> | null
if (!owner || owner.deletedAt || owner.deactivatedAt) {
await deleteBackupIfNeeded(context, entry, dryRun, stats)
continue
}
const ownerHandle = normalizeOwner(owner.handle ?? owner._id)
if (ownerHandle !== entry.owner) {
await deleteBackupIfNeeded(context, entry, dryRun, stats)
}
} catch (error) {
console.error('GitHub backup cleanup failed', error)
stats.errors += 1
}
}
}
async function deleteBackupIfNeeded(
context: Awaited<ReturnType<typeof getGitHubBackupContext>>,
entry: Awaited<ReturnType<typeof listGitHubSkillBackupEntries>>[number],
dryRun: boolean,
stats: GitHubBackupSyncStats,
) {
const result = dryRun
? { deleted: true as const }
: await deleteGitHubSkillBackup(context, entry.owner, entry.slug)
if (result.deleted) {
stats.skillsDeleted += 1
}
}
export const syncGitHubBackupsInternal = internalAction({
args: {
dryRun: v.optional(v.boolean()),
+5 -3
View File
@@ -167,7 +167,7 @@ describe('httpApiV1 handlers', () => {
it('users/reclaim calls reclaim mutation for admin', async () => {
const runMutation = vi.fn(async (_mutation: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate()
return { ok: true }
return { ok: true, action: 'ownership_transferred' }
})
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('handle' in args) return { _id: 'users:target' }
@@ -194,12 +194,14 @@ describe('httpApiV1 handlers', () => {
slug: 'a',
rightfulOwnerUserId: 'users:target',
reason: 'r',
transferRootSlugOnly: true,
})
expect(reclaimCalls[1]?.[1]).toMatchObject({
actorUserId: 'users:admin',
slug: 'b',
rightfulOwnerUserId: 'users:target',
reason: 'r',
transferRootSlugOnly: true,
})
})
@@ -1042,6 +1044,7 @@ describe('httpApiV1 handlers', () => {
})
it('stars add succeeds', async () => {
vi.mocked(getOptionalApiTokenUserId).mockResolvedValue('users:1' as never)
vi.mocked(requireApiTokenUser).mockResolvedValue({
userId: 'users:1',
user: { handle: 'p' },
@@ -1050,7 +1053,6 @@ describe('httpApiV1 handlers', () => {
const runMutation = vi
.fn()
.mockResolvedValueOnce(okRate())
.mockResolvedValueOnce(okRate())
.mockResolvedValueOnce({ ok: true, starred: true, alreadyStarred: false })
const response = await __handlers.starsPostRouterV1Handler(
makeCtx({ runQuery, runMutation }),
@@ -1066,6 +1068,7 @@ describe('httpApiV1 handlers', () => {
})
it('stars delete succeeds', async () => {
vi.mocked(getOptionalApiTokenUserId).mockResolvedValue('users:1' as never)
vi.mocked(requireApiTokenUser).mockResolvedValue({
userId: 'users:1',
user: { handle: 'p' },
@@ -1074,7 +1077,6 @@ describe('httpApiV1 handlers', () => {
const runMutation = vi
.fn()
.mockResolvedValueOnce(okRate())
.mockResolvedValueOnce(okRate())
.mockResolvedValueOnce({ ok: true, unstarred: true, alreadyUnstarred: false })
const response = await __handlers.starsDeleteRouterV1Handler(
makeCtx({ runQuery, runMutation }),
+8 -5
View File
@@ -163,7 +163,9 @@ async function handleAdminRestore(
/**
* POST /api/v1/users/reclaim
* Admin-only: reclaim squatted slugs and reserve them for the rightful owner.
* Admin-only: reclaim root slugs for the rightful owner.
* Default behavior is non-destructive owner transfer for existing skills
* (preserves versions/stats/metadata) and leaves missing slugs untouched.
* Body: { handle: string, slugs: string[], reason?: string }
*/
async function handleAdminReclaim(
@@ -185,16 +187,17 @@ async function handleAdminReclaim(
const targetUser = await ctx.runQuery(api.users.getByHandle, { handle })
if (!targetUser?._id) return text('User not found', 404, headers)
const results: Array<{ slug: string; ok: boolean; error?: string }> = []
const results: Array<{ slug: string; ok: boolean; action?: string; error?: string }> = []
for (const slug of slugs) {
try {
await ctx.runMutation(internal.skills.reclaimSlugInternal, {
const result = (await ctx.runMutation(internal.skills.reclaimSlugInternal, {
actorUserId,
slug: slug.trim().toLowerCase(),
rightfulOwnerUserId: targetUser._id,
reason,
})
results.push({ slug, ok: true })
transferRootSlugOnly: true,
})) as { action?: string }
results.push({ slug, ok: true, action: result.action })
} catch (error) {
const message = error instanceof Error ? error.message : 'Reclaim failed'
results.push({ slug, ok: false, error: message })
+1 -1
View File
@@ -35,7 +35,7 @@ export async function getSkillBadgeMap(
const records = await ctx.db
.query('skillBadges')
.withIndex('by_skill', (q) => q.eq('skillId', skillId))
.collect()
.take(10)
return buildBadgeMap(records)
}
+1 -21
View File
@@ -1,6 +1,7 @@
import { internal } from '../_generated/api'
import type { Doc, Id } from '../_generated/dataModel'
import type { ActionCtx } from '../_generated/server'
import { extractResponseText } from './openaiResponse'
const CHANGELOG_MODEL = process.env.OPENAI_CHANGELOG_MODEL ?? 'gpt-4.1'
const MAX_README_CHARS = 8_000
@@ -59,27 +60,6 @@ function pickPaths(values: string[]) {
return values.slice(0, MAX_PATHS_IN_PROMPT)
}
function extractResponseText(payload: unknown) {
if (!payload || typeof payload !== 'object') return null
const output = (payload as { output?: unknown }).output
if (!Array.isArray(output)) return null
const chunks: string[] = []
for (const item of output) {
if (!item || typeof item !== 'object') continue
if ((item as { type?: unknown }).type !== 'message') continue
const content = (item as { content?: unknown }).content
if (!Array.isArray(content)) continue
for (const part of content) {
if (!part || typeof part !== 'object') continue
if ((part as { type?: unknown }).type !== 'output_text') continue
const text = (part as { text?: unknown }).text
if (typeof text === 'string' && text.trim()) chunks.push(text)
}
}
const joined = chunks.join('\n').trim()
return joined || null
}
async function generateWithOpenAI(args: {
slug: string
version: string
+1
View File
@@ -116,6 +116,7 @@ export async function syncGitHubProfile(ctx: ActionCtx, userId: Id<'users'>) {
const payload = (await response.json()) as GitHubUser
const newLogin = payload.login?.trim()
const newImage = payload.avatar_url?.trim()
const profileName = payload.name?.trim()
if (!newLogin) return
+105 -1
View File
@@ -74,6 +74,13 @@ export type GitHubBackupContext = {
root: string
}
export type GitHubSkillBackupEntry = {
owner: string
slug: string
rootPath: string
metaPath: string
}
export function isGitHubBackupConfigured() {
return Boolean(
process.env.GITHUB_APP_ID &&
@@ -108,6 +115,103 @@ export async function fetchGitHubSkillMeta(
)
}
export async function listGitHubSkillBackupEntries(
context: GitHubBackupContext,
): Promise<GitHubSkillBackupEntry[]> {
const ref = await githubGet<GitRef>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/ref/heads/${context.branch}`,
)
const baseCommit = await githubGet<GitCommit>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/commits/${ref.object.sha}`,
)
const tree = await githubGet<GitTree>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/trees/${baseCommit.tree.sha}?recursive=1`,
)
const prefix = context.root ? `${context.root}/` : ''
const entries: GitHubSkillBackupEntry[] = []
for (const entry of tree.tree ?? []) {
if (entry.type !== 'blob' || !entry.path) continue
if (!entry.path.startsWith(prefix) || !entry.path.endsWith(`/${META_FILENAME}`)) continue
const relative = entry.path.slice(prefix.length)
const segments = relative.split('/')
if (segments.length !== 3) continue
const [owner, slug, file] = segments
if (file !== META_FILENAME) continue
const rootPath = prefix ? `${prefix}${owner}/${slug}` : `${owner}/${slug}`
entries.push({ owner, slug, rootPath, metaPath: entry.path })
}
return entries
}
export async function deleteGitHubSkillBackup(
context: GitHubBackupContext,
ownerHandle: string,
slug: string,
) {
const skillRoot = buildSkillRoot(context.root, ownerHandle, slug)
const ref = await githubGet<GitRef>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/ref/heads/${context.branch}`,
)
const baseCommitSha = ref.object.sha
const baseCommit = await githubGet<GitCommit>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/commits/${baseCommitSha}`,
)
const baseTreeSha = baseCommit.tree.sha
const existingTree = await githubGet<GitTree>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/trees/${baseTreeSha}?recursive=1`,
)
const prefix = `${skillRoot}/`
const pathsToDelete = (existingTree.tree ?? [])
.filter((entry) => entry.type === 'blob' && entry.path?.startsWith(prefix))
.map((entry) => entry.path ?? '')
.filter(Boolean)
if (!pathsToDelete.length) return { deleted: false as const }
const treeEntries = pathsToDelete.map((path) => ({
path,
mode: '100644' as const,
type: 'blob' as const,
sha: null,
}))
const newTree = await githubPost<{ sha: string }>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/trees`,
{
base_tree: baseTreeSha,
tree: treeEntries,
},
)
const commit = await githubPost<GitCommit>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/commits`,
{
message: `delete: ${skillRoot}`,
tree: newTree.sha,
parents: [baseCommitSha],
},
)
await githubPatch(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/refs/heads/${context.branch}`,
{ sha: commit.sha },
)
return { deleted: true as const }
}
export async function backupSkillToGitHub(
ctx: ActionCtx,
params: BackupParams,
@@ -397,7 +501,7 @@ function parseRepo(repo: string) {
return [owner, name] as const
}
function normalizeOwner(value: string) {
export function normalizeOwner(value: string) {
const normalized = value
.trim()
.toLowerCase()
+141
View File
@@ -0,0 +1,141 @@
import type { Doc } from '../_generated/dataModel'
import type { MutationCtx, QueryCtx } from '../_generated/server'
export const GLOBAL_STATS_KEY = 'default'
type SkillVisibilityFields = Pick<
Doc<'skills'>,
'softDeletedAt' | 'moderationStatus' | 'moderationFlags'
>
type GlobalStatsReadCtx = Pick<MutationCtx | QueryCtx, 'db'>
type GlobalStatsWriteCtx = Pick<MutationCtx, 'db'>
export function isPublicSkillDoc(skill: SkillVisibilityFields | null | undefined) {
if (!skill || skill.softDeletedAt) return false
if (skill.moderationStatus && skill.moderationStatus !== 'active') return false
if (skill.moderationFlags?.includes('blocked.malware')) return false
return true
}
export function getPublicSkillVisibilityDelta(
before: SkillVisibilityFields | null | undefined,
after: SkillVisibilityFields | null | undefined,
) {
const beforePublic = isPublicSkillDoc(before)
const afterPublic = isPublicSkillDoc(after)
if (beforePublic === afterPublic) return 0
return afterPublic ? 1 : -1
}
function getErrorMessage(error: unknown) {
if (typeof error === 'string') return error
if (error && typeof error === 'object' && 'message' in error) {
const message = (error as { message?: unknown }).message
if (typeof message === 'string') return message
}
return ''
}
export function isGlobalStatsStorageNotReadyError(error: unknown) {
const message = getErrorMessage(error).toLowerCase()
if (!message) return false
const referencesGlobalStats = message.includes('globalstats') || message.includes('by_key')
if (!referencesGlobalStats) return false
return (
message.includes('table') ||
message.includes('index') ||
message.includes('schema') ||
message.includes('not found') ||
message.includes('does not exist') ||
message.includes('unknown')
)
}
export async function countPublicSkillsForGlobalStats(ctx: GlobalStatsReadCtx) {
const skills = await ctx.db
.query('skills')
.withIndex('by_active_updated', (q) => q.eq('softDeletedAt', undefined))
.collect()
let count = 0
for (const skill of skills) {
if (isPublicSkillDoc(skill)) count += 1
}
return count
}
export async function setGlobalPublicSkillsCount(
ctx: GlobalStatsWriteCtx,
count: number,
now = Date.now(),
) {
const normalizedCount = Math.max(0, Math.trunc(Number.isFinite(count) ? count : 0))
try {
const existing = await ctx.db
.query('globalStats')
.withIndex('by_key', (q) => q.eq('key', GLOBAL_STATS_KEY))
.unique()
if (existing) {
await ctx.db.patch(existing._id, { activeSkillsCount: normalizedCount, updatedAt: now })
} else {
await ctx.db.insert('globalStats', {
key: GLOBAL_STATS_KEY,
activeSkillsCount: normalizedCount,
updatedAt: now,
})
}
} catch (error) {
if (isGlobalStatsStorageNotReadyError(error)) return
throw error
}
}
export async function adjustGlobalPublicSkillsCount(
ctx: GlobalStatsWriteCtx,
delta: number,
now = Date.now(),
) {
const normalizedDelta = Math.trunc(Number.isFinite(delta) ? delta : 0)
if (normalizedDelta === 0) return
let existing:
| {
_id: Doc<'globalStats'>['_id']
activeSkillsCount: number
}
| null
| undefined
try {
existing = await ctx.db
.query('globalStats')
.withIndex('by_key', (q) => q.eq('key', GLOBAL_STATS_KEY))
.unique()
} catch (error) {
if (isGlobalStatsStorageNotReadyError(error)) return
throw error
}
if (!existing) {
// No baseline yet (e.g. fresh deploy). Initialize via full recount once.
const count = await countPublicSkillsForGlobalStats(ctx)
await setGlobalPublicSkillsCount(ctx, count, now)
return
}
const nextCount = Math.max(0, existing.activeSkillsCount + normalizedDelta)
await ctx.db.patch(existing._id, { activeSkillsCount: nextCount, updatedAt: now })
}
export async function readGlobalPublicSkillsCount(ctx: GlobalStatsReadCtx) {
try {
const stats = await ctx.db
.query('globalStats')
.withIndex('by_key', (q) => q.eq('key', GLOBAL_STATS_KEY))
.unique()
return stats?.activeSkillsCount ?? null
} catch (error) {
if (isGlobalStatsStorageNotReadyError(error)) return null
throw error
}
}
+242 -2
View File
@@ -1,6 +1,51 @@
/* @vitest-environment node */
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import { getClientIp } from './httpRateLimit'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { applyRateLimit, getClientIp } from './httpRateLimit'
type MockRateLimitStatus = {
allowed: boolean
remaining: number
limit: number
resetAt: number
}
type MockRateLimitPlan = {
ip: MockRateLimitStatus
user?: MockRateLimitStatus
tokenValid?: boolean
userActive?: boolean
}
function makeRateLimitCtx(plan: MockRateLimitPlan) {
const runQuery = vi.fn(async (_fn: unknown, args: Record<string, unknown>) => {
if ('tokenHash' in args) {
if (plan.tokenValid === false) return null
return { _id: 'token_1', revokedAt: undefined }
}
if ('tokenId' in args) {
if (plan.userActive === false) return null
return { _id: 'users_123', deletedAt: undefined, deactivatedAt: undefined }
}
if ('key' in args && 'limit' in args && 'windowMs' in args) {
const key = String(args.key)
if (key.startsWith('ip:')) return plan.ip
if (key.startsWith('user:')) return plan.user
}
throw new Error(`Unexpected runQuery args: ${JSON.stringify(args)}`)
})
const runMutation = vi.fn(async (_fn: unknown, args: Record<string, unknown>) => {
const key = String(args.key)
const source = key.startsWith('user:') ? plan.user : plan.ip
if (!source) throw new Error(`Missing rate limit source for ${key}`)
return { allowed: source.allowed, remaining: source.remaining }
})
return {
runQuery,
runMutation,
} as unknown as Parameters<typeof applyRateLimit>[0]
}
describe('getClientIp', () => {
let prev: string | undefined
@@ -53,4 +98,199 @@ describe('getClientIp', () => {
process.env.TRUST_FORWARDED_IPS = 'true'
expect(getClientIp(request)).toBe('203.0.113.9')
})
it('prefers x-forwarded-for over x-real-ip when trusted mode is enabled', () => {
const request = new Request('https://example.com', {
headers: {
'x-forwarded-for': '203.0.113.9, 198.51.100.2',
'x-real-ip': '198.51.100.77',
},
})
process.env.TRUST_FORWARDED_IPS = 'true'
expect(getClientIp(request)).toBe('203.0.113.9')
})
})
describe('applyRateLimit headers', () => {
afterEach(() => {
vi.restoreAllMocks()
})
it('returns delay-seconds Retry-After on 429 (not epoch)', async () => {
vi.spyOn(Date, 'now').mockReturnValue(1_000_000)
const runMutation = vi.fn()
const ctx = {
runQuery: vi.fn().mockResolvedValue({
allowed: false,
remaining: 0,
limit: 20,
resetAt: 1_030_500,
}),
runMutation,
} as unknown as Parameters<typeof applyRateLimit>[0]
const request = new Request('https://example.com', {
headers: { 'cf-connecting-ip': '203.0.113.1' },
})
const result = await applyRateLimit(ctx, request, 'download')
expect(result.ok).toBe(false)
if (result.ok) return
expect(result.response.status).toBe(429)
expect(result.response.headers.get('Retry-After')).toBe('31')
expect(result.response.headers.get('X-RateLimit-Reset')).toBe('1031')
expect(result.response.headers.get('RateLimit-Reset')).toBe('31')
expect(runMutation).not.toHaveBeenCalled()
})
it('includes rate-limit headers without Retry-After when allowed', async () => {
vi.spyOn(Date, 'now').mockReturnValue(2_000_000)
const ctx = {
runQuery: vi.fn().mockResolvedValue({
allowed: true,
remaining: 19,
limit: 20,
resetAt: 2_015_000,
}),
runMutation: vi.fn().mockResolvedValue({
allowed: true,
remaining: 18,
}),
} as unknown as Parameters<typeof applyRateLimit>[0]
const request = new Request('https://example.com', {
headers: { 'cf-connecting-ip': '203.0.113.1' },
})
const result = await applyRateLimit(ctx, request, 'download')
expect(result.ok).toBe(true)
if (!result.ok) return
const headers = new Headers(result.headers)
expect(headers.get('X-RateLimit-Limit')).toBe('20')
expect(headers.get('X-RateLimit-Remaining')).toBe('18')
expect(headers.get('X-RateLimit-Reset')).toBe('2015')
expect(headers.get('RateLimit-Limit')).toBe('20')
expect(headers.get('RateLimit-Remaining')).toBe('18')
expect(headers.get('RateLimit-Reset')).toBe('15')
expect(headers.get('Retry-After')).toBeNull()
})
it('allows authenticated users when user bucket is healthy and shared ip bucket is exhausted', async () => {
vi.spyOn(Date, 'now').mockReturnValue(3_000_000)
const ctx = makeRateLimitCtx({
ip: {
allowed: false,
remaining: 0,
limit: 20,
resetAt: 3_040_000,
},
user: {
allowed: true,
remaining: 42,
limit: 120,
resetAt: 3_010_000,
},
})
const request = new Request('https://example.com', {
headers: {
authorization: 'Bearer clh_token',
'cf-connecting-ip': '203.0.113.1',
},
})
const result = await applyRateLimit(ctx, request, 'download')
expect(result.ok).toBe(true)
if (!result.ok) return
const headers = new Headers(result.headers)
expect(headers.get('X-RateLimit-Limit')).toBe('120')
expect(headers.get('X-RateLimit-Remaining')).toBe('42')
expect(headers.get('Retry-After')).toBeNull()
})
it('does not consume ip bucket for authenticated requests', async () => {
vi.spyOn(Date, 'now').mockReturnValue(3_100_000)
const ctx = makeRateLimitCtx({
ip: {
allowed: true,
remaining: 19,
limit: 20,
resetAt: 3_140_000,
},
user: {
allowed: true,
remaining: 41,
limit: 120,
resetAt: 3_110_000,
},
})
const request = new Request('https://example.com', {
headers: {
authorization: 'Bearer clh_token',
'cf-connecting-ip': '203.0.113.1',
},
})
const result = await applyRateLimit(ctx, request, 'download')
expect(result.ok).toBe(true)
const runMutation = (ctx as unknown as { runMutation: ReturnType<typeof vi.fn> }).runMutation
const consumedKeys = runMutation.mock.calls.map(([, args]) => String(args.key))
expect(consumedKeys.some((key) => key.startsWith('user:'))).toBe(true)
expect(consumedKeys.some((key) => key.startsWith('ip:'))).toBe(false)
})
it('denies authenticated users when user bucket is exhausted even if ip bucket is healthy', async () => {
vi.spyOn(Date, 'now').mockReturnValue(4_000_000)
const ctx = makeRateLimitCtx({
ip: {
allowed: true,
remaining: 19,
limit: 20,
resetAt: 4_020_000,
},
user: {
allowed: false,
remaining: 0,
limit: 120,
resetAt: 4_030_000,
},
})
const request = new Request('https://example.com', {
headers: {
authorization: 'Bearer clh_token',
'cf-connecting-ip': '203.0.113.1',
},
})
const result = await applyRateLimit(ctx, request, 'download')
expect(result.ok).toBe(false)
if (result.ok) return
expect(result.response.status).toBe(429)
expect(result.response.headers.get('X-RateLimit-Limit')).toBe('120')
expect(result.response.headers.get('X-RateLimit-Remaining')).toBe('0')
expect(result.response.headers.get('Retry-After')).toBe('30')
})
it('falls back to ip enforcement when bearer token is invalid', async () => {
vi.spyOn(Date, 'now').mockReturnValue(5_000_000)
const ctx = makeRateLimitCtx({
tokenValid: false,
ip: {
allowed: false,
remaining: 0,
limit: 20,
resetAt: 5_030_000,
},
})
const request = new Request('https://example.com', {
headers: {
authorization: 'Bearer invalid',
'cf-connecting-ip': '203.0.113.1',
},
})
const result = await applyRateLimit(ctx, request, 'download')
expect(result.ok).toBe(false)
if (result.ok) return
expect(result.response.status).toBe(429)
expect(result.response.headers.get('X-RateLimit-Limit')).toBe('20')
expect(result.response.headers.get('Retry-After')).toBe('30')
})
})
+64 -18
View File
@@ -1,7 +1,7 @@
import { internal } from '../_generated/api'
import type { ActionCtx } from '../_generated/server'
import { getOptionalApiTokenUserId } from './apiTokenAuth'
import { corsHeaders, mergeHeaders } from './httpHeaders'
import { hashToken } from './tokens'
const RATE_LIMIT_WINDOW_MS = 60_000
export const RATE_LIMITS = {
@@ -22,17 +22,56 @@ export async function applyRateLimit(
request: Request,
kind: keyof typeof RATE_LIMITS,
): Promise<{ ok: true; headers: HeadersInit } | { ok: false; response: Response }> {
const userId = await getOptionalApiTokenUserId(ctx, request)
const ip = getClientIp(request) ?? 'unknown'
const ipSource = getClientIpSource(request)
const hasClientIp = ip !== 'unknown'
// Authenticated requests are enforced and consumed by user bucket only to
// avoid draining shared IP quota.
if (userId) {
const userResult = await checkRateLimit(ctx, `user:${userId}`, RATE_LIMITS[kind].key)
const headers = rateHeaders(userResult)
if (!userResult.allowed) {
console.info('rate_limit_denied', {
kind,
auth: true,
userAllowed: false,
ipAllowed: null,
ipSource,
hasClientIp,
})
return {
ok: false,
response: new Response('Rate limit exceeded', {
status: 429,
headers: mergeHeaders(
{
'Content-Type': 'text/plain; charset=utf-8',
'Cache-Control': 'no-store',
},
headers,
corsHeaders(),
),
}),
}
}
return { ok: true, headers }
}
// Anonymous requests remain IP-enforced.
const ipResult = await checkRateLimit(ctx, `ip:${ip}`, RATE_LIMITS[kind].ip)
const token = parseBearerToken(request)
const keyResult = token
? await checkRateLimit(ctx, `key:${await hashToken(token)}`, RATE_LIMITS[kind].key)
: null
const headers = rateHeaders(ipResult)
const chosen = pickMostRestrictive(ipResult, keyResult)
const headers = rateHeaders(chosen)
if (!ipResult.allowed || (keyResult && !keyResult.allowed)) {
if (!ipResult.allowed) {
console.info('rate_limit_denied', {
kind,
auth: false,
userAllowed: null,
ipAllowed: ipResult.allowed,
ipSource,
hasClientIp,
})
return {
ok: false,
response: new Response('Rate limit exceeded', {
@@ -59,13 +98,22 @@ export function getClientIp(request: Request) {
if (!shouldTrustForwardedIps()) return null
const forwarded =
request.headers.get('x-real-ip') ??
request.headers.get('x-forwarded-for') ??
request.headers.get('x-real-ip') ??
request.headers.get('fly-client-ip')
return splitFirstIp(forwarded)
}
function getClientIpSource(request: Request) {
if (request.headers.get('cf-connecting-ip')) return 'cf-connecting-ip'
if (!shouldTrustForwardedIps()) return 'none'
if (request.headers.get('x-forwarded-for')) return 'x-forwarded-for'
if (request.headers.get('x-real-ip')) return 'x-real-ip'
if (request.headers.get('fly-client-ip')) return 'fly-client-ip'
return 'none'
}
async function checkRateLimit(
ctx: ActionCtx,
key: string,
@@ -110,20 +158,18 @@ async function checkRateLimit(
}
}
function pickMostRestrictive(primary: RateLimitResult, secondary: RateLimitResult | null) {
if (!secondary) return primary
if (!primary.allowed) return primary
if (!secondary.allowed) return secondary
return secondary.remaining < primary.remaining ? secondary : primary
}
function rateHeaders(result: RateLimitResult): HeadersInit {
const nowMs = Date.now()
const resetSeconds = Math.ceil(result.resetAt / 1000)
const resetDelaySeconds = Math.max(1, Math.ceil((result.resetAt - nowMs) / 1000))
return {
'X-RateLimit-Limit': String(result.limit),
'X-RateLimit-Remaining': String(result.remaining),
'X-RateLimit-Reset': String(resetSeconds),
...(result.allowed ? {} : { 'Retry-After': String(resetSeconds) }),
'RateLimit-Limit': String(result.limit),
'RateLimit-Remaining': String(result.remaining),
'RateLimit-Reset': String(resetDelaySeconds),
...(result.allowed ? {} : { 'Retry-After': String(resetDelaySeconds) }),
}
}
+44
View File
@@ -0,0 +1,44 @@
import { describe, expect, it } from 'vitest'
import { extractResponseText } from './openaiResponse'
describe('extractResponseText', () => {
it('returns null for invalid payload shapes', () => {
expect(extractResponseText(null)).toBeNull()
expect(extractResponseText({})).toBeNull()
expect(extractResponseText({ output: {} })).toBeNull()
})
it('extracts output_text chunks from message content', () => {
const payload = {
output: [
{ type: 'reasoning', content: [] },
{
type: 'message',
content: [
{ type: 'output_text', text: 'First line' },
{ type: 'output_text', text: 'Second line' },
],
},
],
}
expect(extractResponseText(payload)).toBe('First line\nSecond line')
})
it('ignores blank and non-output_text parts', () => {
const payload = {
output: [
{
type: 'message',
content: [
{ type: 'input_text', text: 'ignored' },
{ type: 'output_text', text: ' ' },
{ type: 'output_text', text: 'kept' },
],
},
],
}
expect(extractResponseText(payload)).toBe('kept')
})
})
+20
View File
@@ -0,0 +1,20 @@
export function extractResponseText(payload: unknown): string | null {
if (!payload || typeof payload !== 'object') return null
const output = (payload as { output?: unknown }).output
if (!Array.isArray(output)) return null
const chunks: string[] = []
for (const item of output) {
if (!item || typeof item !== 'object') continue
if ((item as { type?: unknown }).type !== 'message') continue
const content = (item as { content?: unknown }).content
if (!Array.isArray(content)) continue
for (const part of content) {
if (!part || typeof part !== 'object') continue
if ((part as { type?: unknown }).type !== 'output_text') continue
const text = (part as { text?: unknown }).text
if (typeof text === 'string' && text.trim()) chunks.push(text)
}
}
const joined = chunks.join('\n').trim()
return joined || null
}
+28
View File
@@ -65,4 +65,32 @@ describe('public skill mapping', () => {
comments: 0,
})
})
it('returns skill when moderationStatus is active', () => {
const skill = makeSkill({ moderationStatus: 'active' })
expect(toPublicSkill(skill)).not.toBeNull()
})
it('filters out skill when moderationStatus is hidden', () => {
const skill = makeSkill({ moderationStatus: 'hidden' })
expect(toPublicSkill(skill)).toBeNull()
})
it('returns skill when moderationStatus is undefined (legacy)', () => {
const skill = makeSkill({ moderationStatus: undefined as unknown as string })
expect(toPublicSkill(skill)).not.toBeNull()
})
it('filters out soft-deleted skills', () => {
const skill = makeSkill({ softDeletedAt: Date.now() })
expect(toPublicSkill(skill)).toBeNull()
})
it('filters out skills with blocked.malware flag', () => {
const skill = makeSkill({
moderationStatus: 'active',
moderationFlags: ['blocked.malware'],
})
expect(toPublicSkill(skill)).toBeNull()
})
})
+3 -3
View File
@@ -1,4 +1,5 @@
import type { Doc } from '../_generated/dataModel'
import { isPublicSkillDoc } from './globalStats'
export type PublicUser = Pick<
Doc<'users'>,
@@ -52,9 +53,8 @@ export function toPublicUser(user: Doc<'users'> | null | undefined): PublicUser
}
export function toPublicSkill(skill: Doc<'skills'> | null | undefined): PublicSkill | null {
if (!skill || skill.softDeletedAt) return null
if (skill.moderationStatus && skill.moderationStatus !== 'active') return null
if (skill.moderationFlags?.includes('blocked.malware')) return null
if (!skill) return null
if (!isPublicSkillDoc(skill)) return null
const stats = {
downloads:
typeof skill.statsDownloads === 'number'
+1 -21
View File
@@ -1,4 +1,5 @@
import { getFrontmatterValue, parseFrontmatter } from './skills'
import { extractResponseText } from './openaiResponse'
const SKILL_SUMMARY_MODEL = process.env.OPENAI_SKILL_SUMMARY_MODEL ?? 'gpt-4.1-mini'
const MAX_README_CHARS = 8_000
@@ -61,27 +62,6 @@ function deriveIdentityFallback(args: { slug: string; displayName: string }) {
return normalizeSummary(`Automation skill for ${base}.`)
}
function extractResponseText(payload: unknown) {
if (!payload || typeof payload !== 'object') return null
const output = (payload as { output?: unknown }).output
if (!Array.isArray(output)) return null
const chunks: string[] = []
for (const item of output) {
if (!item || typeof item !== 'object') continue
if ((item as { type?: unknown }).type !== 'message') continue
const content = (item as { content?: unknown }).content
if (!Array.isArray(content)) continue
for (const part of content) {
if (!part || typeof part !== 'object') continue
if ((part as { type?: unknown }).type !== 'output_text') continue
const text = (part as { text?: unknown }).text
if (typeof text === 'string' && text.trim()) chunks.push(text)
}
}
const joined = chunks.join('\n').trim()
return joined || null
}
export async function generateSkillSummary(args: {
slug: string
displayName: string
+1 -21
View File
@@ -1,6 +1,7 @@
import { internal } from '../_generated/api'
import type { Doc } from '../_generated/dataModel'
import type { ActionCtx } from '../_generated/server'
import { extractResponseText } from './openaiResponse'
const CHANGELOG_MODEL = process.env.OPENAI_CHANGELOG_MODEL ?? 'gpt-4.1'
const MAX_README_CHARS = 8_000
@@ -59,27 +60,6 @@ function pickPaths(values: string[]) {
return values.slice(0, MAX_PATHS_IN_PROMPT)
}
function extractResponseText(payload: unknown) {
if (!payload || typeof payload !== 'object') return null
const output = (payload as { output?: unknown }).output
if (!Array.isArray(output)) return null
const chunks: string[] = []
for (const item of output) {
if (!item || typeof item !== 'object') continue
if ((item as { type?: unknown }).type !== 'message') continue
const content = (item as { content?: unknown }).content
if (!Array.isArray(content)) continue
for (const part of content) {
if (!part || typeof part !== 'object') continue
if ((part as { type?: unknown }).type !== 'output_text') continue
const text = (part as { text?: unknown }).text
if (typeof text === 'string' && text.trim()) chunks.push(text)
}
}
const joined = chunks.join('\n').trim()
return joined || null
}
async function generateWithOpenAI(args: {
slug: string
version: string
+1 -21
View File
@@ -11,32 +11,12 @@ import {
parseLlmEvalResponse,
SECURITY_EVALUATOR_SYSTEM_PROMPT,
} from './lib/securityPrompt'
import { extractResponseText } from './lib/openaiResponse'
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
function extractResponseText(payload: unknown): string | null {
if (!payload || typeof payload !== 'object') return null
const output = (payload as { output?: unknown }).output
if (!Array.isArray(output)) return null
const chunks: string[] = []
for (const item of output) {
if (!item || typeof item !== 'object') continue
if ((item as { type?: unknown }).type !== 'message') continue
const content = (item as { content?: unknown }).content
if (!Array.isArray(content)) continue
for (const part of content) {
if (!part || typeof part !== 'object') continue
if ((part as { type?: unknown }).type !== 'output_text') continue
const text = (part as { text?: unknown }).text
if (typeof text === 'string' && text.trim()) chunks.push(text)
}
}
const joined = chunks.join('\n').trim()
return joined || null
}
function verdictToStatus(verdict: string): string {
switch (verdict) {
case 'benign':
+76
View File
@@ -37,6 +37,7 @@ const {
backfillSkillSummariesInternalHandler,
cleanupEmptySkillsInternalHandler,
nominateEmptySkillSpammersInternalHandler,
upsertSkillBadgeRecordInternal,
} = await import('./maintenance')
const { internal } = await import('./_generated/api')
const { generateSkillSummary } = await import('./lib/skillSummary')
@@ -196,6 +197,81 @@ describe('maintenance backfill', () => {
})
})
describe('maintenance badge denormalization', () => {
it('upserts table badge and keeps skill.badges in sync', async () => {
const unique = vi.fn().mockResolvedValue(null)
const query = vi.fn().mockReturnValue({
withIndex: () => ({ unique }),
})
const insert = vi.fn().mockResolvedValue('skillBadges:1')
const get = vi.fn().mockResolvedValue({ _id: 'skills:1', badges: undefined })
const patch = vi.fn().mockResolvedValue(undefined)
const ctx = {
db: {
query,
insert,
get,
patch,
},
} as never
const result = await (upsertSkillBadgeRecordInternal as { _handler: Function })._handler(ctx, {
skillId: 'skills:1',
kind: 'highlighted',
byUserId: 'users:1',
at: 123,
})
expect(result).toEqual({ inserted: true })
expect(insert).toHaveBeenCalledWith('skillBadges', {
skillId: 'skills:1',
kind: 'highlighted',
byUserId: 'users:1',
at: 123,
})
expect(patch).toHaveBeenCalledWith('skills:1', {
badges: {
highlighted: { byUserId: 'users:1', at: 123 },
},
})
})
it('resyncs denormalized badge even when table record already exists', async () => {
const unique = vi.fn().mockResolvedValue({ _id: 'skillBadges:existing' })
const query = vi.fn().mockReturnValue({
withIndex: () => ({ unique }),
})
const insert = vi.fn()
const get = vi.fn().mockResolvedValue({ _id: 'skills:1', badges: {} })
const patch = vi.fn().mockResolvedValue(undefined)
const ctx = {
db: {
query,
insert,
get,
patch,
},
} as never
const result = await (upsertSkillBadgeRecordInternal as { _handler: Function })._handler(ctx, {
skillId: 'skills:1',
kind: 'official',
byUserId: 'users:2',
at: 456,
})
expect(result).toEqual({ inserted: false })
expect(insert).not.toHaveBeenCalled()
expect(patch).toHaveBeenCalledWith('skills:1', {
badges: {
official: { byUserId: 'users:2', at: 456 },
},
})
})
})
describe('maintenance fingerprint backfill', () => {
it('backfills fingerprint field and inserts index entry', async () => {
const { hashSkillFiles } = await import('./lib/skills')
+119 -1
View File
@@ -642,17 +642,32 @@ export const upsertSkillBadgeRecordInternal = internalMutation({
at: v.number(),
},
handler: async (ctx, args) => {
const syncDenormalizedBadge = async () => {
const skill = await ctx.db.get(args.skillId)
if (!skill) return
await ctx.db.patch(args.skillId, {
badges: {
...(skill.badges as Record<string, unknown> | undefined),
[args.kind]: { byUserId: args.byUserId, at: args.at },
},
})
}
const existing = await ctx.db
.query('skillBadges')
.withIndex('by_skill_kind', (q) => q.eq('skillId', args.skillId).eq('kind', args.kind))
.unique()
if (existing) return { inserted: false as const }
if (existing) {
await syncDenormalizedBadge()
return { inserted: false as const }
}
await ctx.db.insert('skillBadges', {
skillId: args.skillId,
kind: args.kind,
byUserId: args.byUserId,
at: args.at,
})
await syncDenormalizedBadge()
return { inserted: true as const }
},
})
@@ -1411,6 +1426,109 @@ export const nominateEmptySkillSpammers: ReturnType<typeof action> = action({
},
})
// Backfill embeddingSkillMap from existing skillEmbeddings.
// Run once after deploying the schema change:
// npx convex run maintenance:backfillEmbeddingSkillMapInternal --prod
export const backfillEmbeddingSkillMapInternal = internalMutation({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
},
handler: async (ctx, args) => {
const batchSize = clampInt(args.batchSize ?? 200, 10, 500)
const { page, continueCursor, isDone } = await ctx.db
.query('skillEmbeddings')
.paginate({ cursor: args.cursor ?? null, numItems: batchSize })
let inserted = 0
for (const embedding of page) {
const existing = await ctx.db
.query('embeddingSkillMap')
.withIndex('by_embedding', (q) => q.eq('embeddingId', embedding._id))
.unique()
if (!existing) {
await ctx.db.insert('embeddingSkillMap', {
embeddingId: embedding._id,
skillId: embedding.skillId,
})
inserted++
}
}
if (!isDone) {
await ctx.scheduler.runAfter(0, internal.maintenance.backfillEmbeddingSkillMapInternal, {
cursor: continueCursor,
batchSize: args.batchSize,
})
}
return { inserted, isDone, scanned: page.length }
},
})
// Sync skillBadges table → denormalized skill.badges field.
// Run after deploying the badge-read removal to ensure all skills
// have up-to-date badges on the skill doc itself.
export const backfillDenormalizedBadgesInternal = internalMutation({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
},
handler: async (ctx, args) => {
const batchSize = clampInt(args.batchSize ?? 100, 10, 200)
const { page, continueCursor, isDone } = await ctx.db
.query('skills')
.paginate({ cursor: args.cursor ?? null, numItems: batchSize })
let patched = 0
for (const skill of page) {
const records = await ctx.db
.query('skillBadges')
.withIndex('by_skill', (q) => q.eq('skillId', skill._id))
.take(10)
// Build canonical badge map from the table
const canonical: Record<string, { byUserId: Id<'users'>; at: number }> = {}
for (const r of records) {
canonical[r.kind] = { byUserId: r.byUserId, at: r.at }
}
// Compare with existing denormalized badges (keys + values)
const existing = (skill.badges ?? {}) as Record<
string,
{ byUserId?: Id<'users'>; at?: number } | undefined
>
const canonicalKeys = Object.keys(canonical)
const existingKeys = Object.keys(existing).filter((k) => existing[k] !== undefined)
const needsPatch =
canonicalKeys.length !== existingKeys.length ||
canonicalKeys.some((k) => {
const current = existing[k]
const next = canonical[k]
return (
!current ||
current.byUserId !== next.byUserId ||
current.at !== next.at
)
})
if (needsPatch) {
await ctx.db.patch(skill._id, { badges: canonical })
patched++
}
}
if (!isDone) {
await ctx.scheduler.runAfter(0, internal.maintenance.backfillDenormalizedBadgesInternal, {
cursor: continueCursor,
batchSize: args.batchSize,
})
}
return { patched, isDone, scanned: page.length }
},
})
function clampInt(value: number, min: number, max: number) {
const rounded = Math.trunc(value)
if (!Number.isFinite(rounded)) return min
+16
View File
@@ -310,6 +310,14 @@ const skillEmbeddings = defineTable({
filterFields: ['visibility'],
})
// Lightweight lookup: embeddingId → skillId (~100 bytes per doc).
// Avoids reading full skillEmbeddings docs (~12KB each with vector)
// during search hydration.
const embeddingSkillMap = defineTable({
embeddingId: v.id('skillEmbeddings'),
skillId: v.id('skills'),
}).index('by_embedding', ['embeddingId'])
const skillDailyStats = defineTable({
skillId: v.id('skills'),
day: v.number(),
@@ -342,6 +350,12 @@ const skillStatBackfillState = defineTable({
updatedAt: v.number(),
}).index('by_key', ['key'])
const globalStats = defineTable({
key: v.string(),
activeSkillsCount: v.number(),
updatedAt: v.number(),
}).index('by_key', ['key'])
const skillStatEvents = defineTable({
skillId: v.id('skills'),
kind: v.union(
@@ -570,10 +584,12 @@ export default defineSchema({
skillBadges,
soulVersionFingerprints,
skillEmbeddings,
embeddingSkillMap,
soulEmbeddings,
skillDailyStats,
skillLeaderboards,
skillStatBackfillState,
globalStats,
skillStatEvents,
skillStatUpdateCursors,
comments,
+16 -27
View File
@@ -4,9 +4,8 @@ import { describe, expect, it, vi } from 'vitest'
import { tokenize } from './lib/searchText'
import { __test, hydrateResults, lexicalFallbackSkills, searchSkills } from './search'
const { generateEmbeddingMock, getSkillBadgeMapsMock } = vi.hoisted(() => ({
const { generateEmbeddingMock } = vi.hoisted(() => ({
generateEmbeddingMock: vi.fn(),
getSkillBadgeMapsMock: vi.fn(),
}))
vi.mock('./lib/embeddings', () => ({
@@ -14,7 +13,6 @@ vi.mock('./lib/embeddings', () => ({
}))
vi.mock('./lib/badges', () => ({
getSkillBadgeMaps: getSkillBadgeMapsMock,
isSkillHighlighted: (skill: { badges?: Record<string, unknown> }) =>
Boolean(skill.badges?.highlighted),
}))
@@ -50,9 +48,8 @@ describe('search helpers', () => {
]
const runQuery = vi
.fn()
.mockResolvedValueOnce([])
.mockResolvedValueOnce([])
.mockResolvedValueOnce(fallback)
.mockResolvedValueOnce([]) // hydrateResults
.mockResolvedValueOnce(fallback) // lexicalFallbackSkills
const result = await searchSkillsHandler(
{
@@ -71,18 +68,15 @@ describe('search helpers', () => {
})
it('applies highlightedOnly filtering in lexical fallback', async () => {
const highlighted = makeSkillDoc({
id: 'skills:hl',
slug: 'orf-highlighted',
displayName: 'ORF Highlighted',
})
const highlighted = {
...makeSkillDoc({
id: 'skills:hl',
slug: 'orf-highlighted',
displayName: 'ORF Highlighted',
}),
badges: { highlighted: { byUserId: 'users:mod', at: 1 } },
}
const plain = makeSkillDoc({ id: 'skills:plain', slug: 'orf-plain', displayName: 'ORF Plain' })
getSkillBadgeMapsMock.mockResolvedValueOnce(
new Map([
['skills:hl', { highlighted: { byUserId: 'users:mod', at: 1 } }],
['skills:plain', {}],
]),
)
const result = await lexicalFallbackSkillsHandler(
makeLexicalCtx({
@@ -104,12 +98,6 @@ describe('search helpers', () => {
moderationFlags: ['flagged.suspicious'],
})
const clean = makeSkillDoc({ id: 'skills:clean', slug: 'orf-clean', displayName: 'ORF Clean' })
getSkillBadgeMapsMock.mockResolvedValueOnce(
new Map([
['skills:suspicious', {}],
['skills:clean', {}],
]),
)
const result = await lexicalFallbackSkillsHandler(
makeLexicalCtx({
@@ -125,7 +113,6 @@ describe('search helpers', () => {
it('includes exact slug match from by_slug even when recent scan is empty', async () => {
const exactSlugSkill = makeSkillDoc({ id: 'skills:orf', slug: 'orf', displayName: 'ORF' })
getSkillBadgeMapsMock.mockResolvedValueOnce(new Map([['skills:orf', {}]]))
const ctx = makeLexicalCtx({
exactSlugSkill,
recentSkills: [],
@@ -197,9 +184,8 @@ describe('search helpers', () => {
const runQuery = vi
.fn()
.mockResolvedValueOnce(vectorEntries)
.mockResolvedValueOnce([])
.mockResolvedValueOnce(fallbackEntries)
.mockResolvedValueOnce(vectorEntries) // hydrateResults
.mockResolvedValueOnce(fallbackEntries) // lexicalFallbackSkills
const result = await searchSkillsHandler(
{
@@ -237,6 +223,9 @@ describe('search helpers', () => {
if (id === 'skillVersions:1') return { _id: 'skillVersions:1', version: '1.0.0' }
return null
}),
query: vi.fn(() => ({
withIndex: () => ({ unique: vi.fn().mockResolvedValue(null) }),
})),
},
},
{ embeddingIds: ['skillEmbeddings:1'], nonSuspiciousOnly: true },
+25 -59
View File
@@ -3,7 +3,7 @@ import { internal } from './_generated/api'
import type { Doc, Id } from './_generated/dataModel'
import type { QueryCtx } from './_generated/server'
import { action, internalQuery } from './_generated/server'
import { getSkillBadgeMaps, isSkillHighlighted, type SkillBadgeMap } from './lib/badges'
import { isSkillHighlighted } from './lib/badges'
import { generateEmbedding } from './lib/embeddings'
import { toPublicSkill, toPublicSoul, toPublicUser } from './lib/public'
import { matchesExactTokens, tokenize } from './lib/searchText'
@@ -40,7 +40,7 @@ const SLUG_PREFIX_BOOST = 0.8
const NAME_EXACT_BOOST = 1.1
const NAME_PREFIX_BOOST = 0.6
const POPULARITY_WEIGHT = 0.08
const FALLBACK_SCAN_LIMIT = 1200
const FALLBACK_SCAN_LIMIT = 500
function getNextCandidateLimit(current: number, max: number) {
const next = Math.min(current * 2, max)
@@ -149,21 +149,11 @@ export const searchSkills: ReturnType<typeof action> = action({
results.map((result) => [result._id, result._score]),
)
const badgeMapEntries = (await ctx.runQuery(internal.search.getSkillBadgeMapsInternal, {
skillIds: hydrated.map((entry) => entry.skill._id),
})) as Array<[Id<'skills'>, SkillBadgeMap]>
const badgeMapBySkillId = new Map(badgeMapEntries)
const hydratedWithBadges = hydrated.map((entry) => ({
...entry,
skill: {
...entry.skill,
badges: badgeMapBySkillId.get(entry.skill._id) ?? {},
},
}))
// Skills already have badges from their docs (via toPublicSkill).
// No need for a separate badge table lookup.
const filtered = args.highlightedOnly
? hydratedWithBadges.filter((entry) => isSkillHighlighted(entry.skill))
: hydratedWithBadges
? hydrated.filter((entry) => isSkillHighlighted(entry.skill))
: hydrated
exactMatches = filtered.filter((entry) =>
matchesExactTokens(queryTokens, [
@@ -215,14 +205,6 @@ export const searchSkills: ReturnType<typeof action> = action({
},
})
export const getBadgeMapsForSkills = internalQuery({
args: { skillIds: v.array(v.id('skills')) },
handler: async (ctx, args): Promise<Array<[Id<'skills'>, SkillBadgeMap]>> => {
const badgeMap = await getSkillBadgeMaps(ctx, args.skillIds)
return Array.from(badgeMap.entries())
},
})
export const hydrateResults = internalQuery({
args: {
embeddingIds: v.array(v.id('skillEmbeddings')),
@@ -233,21 +215,26 @@ export const hydrateResults = internalQuery({
const entries: Array<SkillSearchEntry | null> = await Promise.all(
args.embeddingIds.map(async (embeddingId) => {
const embedding = await ctx.db.get(embeddingId)
if (!embedding) return null
const skill = await ctx.db.get(embedding.skillId)
// Use lightweight lookup table (~100 bytes) instead of full embedding doc (~12KB).
const lookup = await ctx.db
.query('embeddingSkillMap')
.withIndex('by_embedding', (q) => q.eq('embeddingId', embeddingId))
.unique()
// Fallback to full embedding doc for rows not yet backfilled.
const skillId = lookup
? lookup.skillId
: await ctx.db.get(embeddingId).then((e) => e?.skillId)
if (!skillId) return null
const skill = await ctx.db.get(skillId)
if (!skill || skill.softDeletedAt) return null
if (args.nonSuspiciousOnly && isSkillSuspicious(skill)) return null
const [version, ownerInfo] = await Promise.all([
ctx.db.get(embedding.versionId),
getOwnerInfo(skill.ownerUserId),
])
const ownerInfo = await getOwnerInfo(skill.ownerUserId)
const publicSkill = toPublicSkill(skill)
if (!publicSkill) return null
return {
embeddingId,
skill: publicSkill,
version,
version: null as Doc<'skillVersions'> | null,
ownerHandle: ownerInfo.handle,
owner: ownerInfo.owner,
}
@@ -309,15 +296,12 @@ export const lexicalFallbackSkills = internalQuery({
const entries = await Promise.all(
matched.map(async (skill) => {
const [version, ownerInfo] = await Promise.all([
skill.latestVersionId ? ctx.db.get(skill.latestVersionId) : Promise.resolve(null),
getOwnerInfo(skill.ownerUserId),
])
const ownerInfo = await getOwnerInfo(skill.ownerUserId)
const publicSkill = toPublicSkill(skill)
if (!publicSkill) return null
return {
skill: publicSkill,
version,
version: null as Doc<'skillVersions'> | null,
ownerHandle: ownerInfo.handle,
owner: ownerInfo.owner,
}
@@ -326,21 +310,11 @@ export const lexicalFallbackSkills = internalQuery({
const validEntries = entries.filter((entry): entry is SkillSearchEntry => entry !== null)
if (validEntries.length === 0) return []
const badgeMap = await getSkillBadgeMaps(
ctx,
validEntries.map((entry) => entry.skill._id),
)
const withBadges = validEntries.map((entry) => ({
...entry,
skill: {
...entry.skill,
badges: badgeMap.get(entry.skill._id) ?? {},
},
}))
// Skills already have badges from their docs (via toPublicSkill).
// No need for a separate badge table lookup.
const filtered = args.highlightedOnly
? withBadges.filter((entry) => isSkillHighlighted(entry.skill))
: withBadges
? validEntries.filter((entry) => isSkillHighlighted(entry.skill))
: validEntries
return filtered.slice(0, limit)
},
})
@@ -440,14 +414,6 @@ export const hydrateSoulResults = internalQuery({
},
})
export const getSkillBadgeMapsInternal = internalQuery({
args: { skillIds: v.array(v.id('skills')) },
handler: async (ctx, args) => {
const badgeMap = await getSkillBadgeMaps(ctx, args.skillIds)
return Array.from(badgeMap.entries())
},
})
export const __test = {
getNextCandidateLimit,
matchesAllTokens,
+23 -60
View File
@@ -3,14 +3,18 @@
*
* Instead of updating skill stats synchronously in the hot path (which can cause
* contention when multiple users download/star/install the same skill), we insert
* lightweight event records and process them in batches via a cron job.
* lightweight event records and process them in batches via cron jobs.
*
* Flow:
* 1. User action (download, star, install) → insertStatEvent() writes to skillStatEvents table
* 2. Cron job runs every 5 minutes → processSkillStatEventsInternal() processes batches
* 3. Events are aggregated per-skill to minimize database operations
* 4. Stats are applied to skill documents and daily stats tables
* 5. Events are marked as processed (kept forever for auditing)
* Two processing paths run at different frequencies to balance freshness vs bandwidth:
*
* 1. **Daily stats (15-minute cron)** — `processSkillStatEventsAction`
* Writes to skillDailyStats for trending/leaderboards. Uses a cursor in
* skillStatUpdateCursors. Does NOT touch skill documents.
*
* 2. **Skill doc sync (6-hour cron)** — `processSkillStatEventsInternal`
* Patches skill documents with accumulated stat deltas. Uses processedAt
* field to track progress. Runs infrequently because patching skill docs
* invalidates reactive queries for all subscribers (thundering herd).
*/
import { v } from 'convex/values'
@@ -175,7 +179,7 @@ function aggregateEvents(events: Doc<'skillStatEvents'>[]): AggregatedDeltas {
/**
* Process a batch of unprocessed stat events.
*
* Called by cron every 5 minutes. Processes up to batchSize events (default 100).
* Called by the 6-hour cron to sync stats to skill docs. Processes up to batchSize events (default 500).
* If the batch is full, schedules an immediate follow-up run to drain the queue.
*
* Processing steps:
@@ -198,7 +202,7 @@ function aggregateEvents(events: Doc<'skillStatEvents'>[]): AggregatedDeltas {
export const processSkillStatEventsInternal = internalMutation({
args: { batchSize: v.optional(v.number()) },
handler: async (ctx, args) => {
const batchSize = args.batchSize ?? 100
const batchSize = args.batchSize ?? 500
const now = Date.now()
// Level 1: Fetch a batch of unprocessed events
@@ -252,25 +256,13 @@ export const processSkillStatEventsInternal = internalMutation({
installsAllTime: deltas.installsAllTime,
installsCurrent: deltas.installsCurrent,
})
await ctx.db.patch(skill._id, {
...patch,
updatedAt: now,
})
// Don't update `updatedAt` — stat changes shouldn't move the
// skill's position in the by_active_updated index.
await ctx.db.patch(skill._id, patch)
}
// Update daily stats for trending/leaderboards
// We use the ORIGINAL event timestamp (occurredAt) so that:
// - A download at Mon 11:55 PM counts toward Monday's stats
// - Even if the cron processes it on Tuesday
//
// Level 4: bumpDailySkillStats does its own coalescing - multiple
// events on the same day will update the same daily record
for (const occurredAt of deltas.downloadEvents) {
await bumpDailySkillStats(ctx, { skillId, now: occurredAt, downloads: 1 })
}
for (const occurredAt of deltas.installNewEvents) {
await bumpDailySkillStats(ctx, { skillId, now: occurredAt, installs: 1 })
}
// NOTE: Daily stats (skillDailyStats) are written by the 15-minute
// action cron (processSkillStatEventsAction), not here.
// Mark all events for this skill as processed
for (const event of skillEvents) {
@@ -352,11 +344,11 @@ const skillDeltaValidator = v.object({
})
/**
* Apply aggregated stats to skills and update the cursor.
* Write aggregated daily stats and advance the cursor.
* This is a single atomic mutation that:
* 1. Updates all affected skills with their aggregated deltas
* 2. Updates daily stats for trending
* 3. Advances the cursor to the new position
* 1. Updates daily stats for trending/leaderboards (skillDailyStats)
* 2. Advances the cursor to the new position
* NOTE: Does NOT patch skill documents — that's handled by processSkillStatEventsInternal.
*/
export const applyAggregatedStatsAndUpdateCursor = internalMutation({
args: {
@@ -366,37 +358,8 @@ export const applyAggregatedStatsAndUpdateCursor = internalMutation({
handler: async (ctx, args) => {
const now = Date.now()
// Process each skill's aggregated deltas
// Update daily stats for trending/leaderboards
for (const delta of args.skillDeltas) {
const skill = await ctx.db.get(delta.skillId)
// Skill was deleted - skip
if (!skill) {
continue
}
// Apply aggregated deltas to skill stats
if (
delta.downloads !== 0 ||
delta.stars !== 0 ||
delta.comments !== 0 ||
delta.installsAllTime !== 0 ||
delta.installsCurrent !== 0
) {
const patch = applySkillStatDeltas(skill, {
downloads: delta.downloads,
stars: delta.stars,
comments: delta.comments,
installsAllTime: delta.installsAllTime,
installsCurrent: delta.installsCurrent,
})
await ctx.db.patch(skill._id, {
...patch,
updatedAt: now,
})
}
// Update daily stats for trending/leaderboards
for (const occurredAt of delta.downloadEvents) {
await bumpDailySkillStats(ctx, { skillId: delta.skillId, now: occurredAt, downloads: 1 })
}
+95
View File
@@ -0,0 +1,95 @@
import { describe, expect, it, vi } from 'vitest'
import { countPublicSkills } from './skills'
type WrappedHandler<TArgs, TResult> = {
_handler: (ctx: unknown, args: TArgs) => Promise<TResult>
}
const countPublicSkillsHandler = (
countPublicSkills as unknown as WrappedHandler<Record<string, never>, number>
)._handler
function makeSkillsQuery(skills: Array<{ softDeletedAt?: number; moderationStatus?: string | null }>) {
return {
withIndex: (name: string) => {
if (name !== 'by_active_updated') throw new Error(`unexpected skills index ${name}`)
return {
collect: async () => skills,
}
},
}
}
describe('skills.countPublicSkills', () => {
it('returns precomputed global stats count when available', async () => {
const ctx = {
db: {
query: vi.fn((table: string) => {
if (table === 'globalStats') {
return {
withIndex: () => ({
unique: async () => ({ _id: 'globalStats:1', activeSkillsCount: 123 }),
}),
}
}
if (table === 'skills') {
return makeSkillsQuery([])
}
throw new Error(`unexpected table ${table}`)
}),
},
}
const result = await countPublicSkillsHandler(ctx, {})
expect(result).toBe(123)
})
it('falls back to live count when global stats row is missing', async () => {
const ctx = {
db: {
query: vi.fn((table: string) => {
if (table === 'globalStats') {
return {
withIndex: () => ({
unique: async () => null,
}),
}
}
if (table === 'skills') {
return makeSkillsQuery([
{ softDeletedAt: undefined, moderationStatus: 'active' },
{ softDeletedAt: undefined, moderationStatus: 'hidden' },
{ softDeletedAt: undefined, moderationStatus: 'active' },
])
}
throw new Error(`unexpected table ${table}`)
}),
},
}
const result = await countPublicSkillsHandler(ctx, {})
expect(result).toBe(2)
})
it('falls back to live count when globalStats table is unavailable', async () => {
const ctx = {
db: {
query: vi.fn((table: string) => {
if (table === 'globalStats') {
throw new Error('unexpected table globalStats')
}
if (table === 'skills') {
return makeSkillsQuery([
{ softDeletedAt: undefined, moderationStatus: 'active' },
{ softDeletedAt: undefined, moderationStatus: 'active' },
])
}
throw new Error(`unexpected table ${table}`)
}),
},
}
const result = await countPublicSkillsHandler(ctx, {})
expect(result).toBe(2)
})
})
+202
View File
@@ -0,0 +1,202 @@
import { describe, expect, it, vi } from 'vitest'
import { getPendingScanSkillsInternal } from './skills'
type PendingScanResult = Array<{
skillId: string
versionId: string | null
sha256hash: string | null
checkCount: number
}>
type WrappedHandler<TArgs, TResult> = {
_handler: (ctx: unknown, args: TArgs) => Promise<TResult>
}
const getPendingScanSkillsHandler = (
getPendingScanSkillsInternal as unknown as WrappedHandler<Record<string, unknown>, PendingScanResult>
)._handler
describe('skills.getPendingScanSkillsInternal', () => {
it('includes unresolved VT records from the oldest slice and skips finalized ones', async () => {
const recentSkills = [
makeSkill('skills:recent-clean', 'skillVersions:recent-clean', 'scanner.llm.clean'),
makeSkill('skills:recent-malicious', 'skillVersions:recent-malicious', 'scanner.vt.pending'),
]
const oldestSkills = [
makeSkill('skills:old-pending', 'skillVersions:old-pending', 'scanner.vt.pending'),
makeSkill('skills:old-stale', 'skillVersions:old-stale', 'scanner.llm.clean'),
makeSkill('skills:old-no-hash', 'skillVersions:old-no-hash', 'scanner.vt.pending'),
]
const versions = new Map<string, unknown>([
[
'skillVersions:recent-clean',
{ _id: 'skillVersions:recent-clean', sha256hash: 'a'.repeat(64), vtAnalysis: { status: 'clean' } },
],
[
'skillVersions:recent-malicious',
{
_id: 'skillVersions:recent-malicious',
sha256hash: 'b'.repeat(64),
vtAnalysis: { status: 'malicious' },
},
],
[
'skillVersions:old-pending',
{ _id: 'skillVersions:old-pending', sha256hash: 'c'.repeat(64), vtAnalysis: { status: 'pending' } },
],
[
'skillVersions:old-stale',
{ _id: 'skillVersions:old-stale', sha256hash: 'd'.repeat(64), vtAnalysis: { status: 'stale' } },
],
['skillVersions:old-no-hash', { _id: 'skillVersions:old-no-hash' }],
])
const ctx = {
db: {
query: vi.fn((table: string) => {
if (table !== 'skills') throw new Error(`unexpected table ${table}`)
return {
withIndex: (
indexName: string,
builder: (q: { eq: (field: string, value: unknown) => unknown }) => unknown,
) => {
builder({ eq: () => ({}) })
if (indexName === 'by_active_updated') {
return {
order: () => ({
take: async () => recentSkills,
}),
}
}
if (indexName === 'by_active_created') {
return {
order: () => ({
take: async () => oldestSkills,
}),
}
}
throw new Error(`unexpected index ${indexName}`)
},
}
}),
get: vi.fn(async (id: string) => versions.get(id) ?? null),
},
}
const result = await getPendingScanSkillsHandler(ctx, {
limit: 25,
skipRecentMinutes: 0,
})
const ids = new Set(result.map((entry) => entry.skillId))
expect(ids.has('skills:old-pending')).toBe(true)
expect(ids.has('skills:old-stale')).toBe(true)
expect(ids.has('skills:recent-clean')).toBe(false)
expect(ids.has('skills:recent-malicious')).toBe(false)
expect(ids.has('skills:old-no-hash')).toBe(false)
})
it('exhaustive mode ignores recent-check suppression for manual backfills', async () => {
const now = Date.now()
const allSkills = [
makeSkill('skills:recently-checked', 'skillVersions:recently-checked', 'scanner.vt.pending', now),
]
const versions = new Map<string, unknown>([
[
'skillVersions:recently-checked',
{ _id: 'skillVersions:recently-checked', sha256hash: 'e'.repeat(64) },
],
])
const withIndex = vi.fn(
(
indexName: string,
builder: (q: { eq: (field: string, value: unknown) => unknown }) => unknown,
) => {
builder({ eq: () => ({}) })
if (indexName !== 'by_active_updated') throw new Error(`unexpected index ${indexName}`)
return {
collect: async () => allSkills,
}
},
)
const ctx = {
db: {
query: vi.fn((table: string) => {
if (table !== 'skills') throw new Error(`unexpected table ${table}`)
return { withIndex }
}),
get: vi.fn(async (id: string) => versions.get(id) ?? null),
},
}
const result = await getPendingScanSkillsHandler(ctx, {
limit: 25,
skipRecentMinutes: 60,
exhaustive: true,
})
expect(result).toHaveLength(1)
expect(result[0]?.skillId).toBe('skills:recently-checked')
})
it('does not clamp exhaustive mode to 100 records', async () => {
const allSkills = Array.from({ length: 150 }, (_, i) =>
makeSkill(`skills:bulk-${i}`, `skillVersions:bulk-${i}`, 'scanner.vt.pending'),
)
const versions = new Map<string, unknown>(
allSkills.map((skill) => {
const versionId = skill.latestVersionId as string
return [versionId, { _id: versionId, sha256hash: `${String(versionId).slice(-8)}${'f'.repeat(56)}` }]
}),
)
const withIndex = vi.fn(
(
indexName: string,
builder: (q: { eq: (field: string, value: unknown) => unknown }) => unknown,
) => {
builder({ eq: () => ({}) })
if (indexName !== 'by_active_updated') throw new Error(`unexpected index ${indexName}`)
return {
collect: async () => allSkills,
}
},
)
const ctx = {
db: {
query: vi.fn((table: string) => {
if (table !== 'skills') throw new Error(`unexpected table ${table}`)
return { withIndex }
}),
get: vi.fn(async (id: string) => versions.get(id) ?? null),
},
}
const result = await getPendingScanSkillsHandler(ctx, {
limit: 10000,
exhaustive: true,
skipRecentMinutes: 0,
})
expect(result).toHaveLength(150)
})
})
function makeSkill(
id: string,
versionId: string,
moderationReason: string,
scanLastCheckedAt?: number,
) {
return {
_id: id,
moderationStatus: 'active',
moderationReason,
latestVersionId: versionId,
scanLastCheckedAt,
}
}
+25
View File
@@ -22,6 +22,21 @@ const clearOwnerSuspiciousFlagsHandler = (
clearOwnerSuspiciousFlagsInternal as unknown as WrappedHandler<Record<string, unknown>>
)._handler
function buildGlobalStatsQuery(table: string) {
if (table !== 'globalStats') return null
return {
withIndex: (name: string) => {
if (name !== 'by_key') throw new Error(`unexpected globalStats index ${name}`)
return {
unique: async () => ({
_id: 'globalStats:1',
activeSkillsCount: 100,
}),
}
},
}
}
function createPublishArgs(overrides?: Partial<Record<string, unknown>>) {
return {
userId: 'users:owner',
@@ -67,6 +82,8 @@ describe('skills anti-spam guards', () => {
deletedAt: undefined,
})),
query: vi.fn((table: string) => {
const globalStatsQuery = buildGlobalStatsQuery(table)
if (globalStatsQuery) return globalStatsQuery
if (table === 'skills') {
return {
withIndex: (name: string) => {
@@ -127,6 +144,8 @@ describe('skills anti-spam guards', () => {
return null
}),
query: vi.fn((table: string) => {
const globalStatsQuery = buildGlobalStatsQuery(table)
if (globalStatsQuery) return globalStatsQuery
if (table === 'skillVersions') {
return {
withIndex: () => ({
@@ -197,6 +216,8 @@ describe('skills anti-spam guards', () => {
return null
}),
query: vi.fn((table: string) => {
const globalStatsQuery = buildGlobalStatsQuery(table)
if (globalStatsQuery) return globalStatsQuery
if (table === 'skillVersions') {
return {
withIndex: () => ({
@@ -265,6 +286,8 @@ describe('skills anti-spam guards', () => {
return null
}),
query: vi.fn((table: string) => {
const globalStatsQuery = buildGlobalStatsQuery(table)
if (globalStatsQuery) return globalStatsQuery
if (table === 'skillVersions') {
return {
withIndex: () => ({
@@ -324,6 +347,8 @@ describe('skills anti-spam guards', () => {
return null
}),
query: vi.fn((table: string) => {
const globalStatsQuery = buildGlobalStatsQuery(table)
if (globalStatsQuery) return globalStatsQuery
if (table === 'skills') {
return {
withIndex: (name: string) => {
+150
View File
@@ -0,0 +1,150 @@
import { describe, expect, it, vi } from 'vitest'
import { reclaimSlugInternal } from './skills'
type WrappedHandler<TArgs> = {
_handler: (ctx: unknown, args: TArgs) => Promise<unknown>
}
const reclaimSlugInternalHandler = (
reclaimSlugInternal as unknown as WrappedHandler<Record<string, unknown>>
)._handler
describe('skills reclaim ownership transfer', () => {
it('transfers ownership in-place when transferRootSlugOnly is true', async () => {
const now = Date.now()
const patch = vi.fn(async () => {})
const insert = vi.fn(async () => {})
const runAfter = vi.fn(async () => {})
const existingSkill = {
_id: 'skills:1',
slug: 'capability-evolver',
ownerUserId: 'users:old',
}
const activeReservation = {
_id: 'reservedSlugs:1',
slug: 'capability-evolver',
originalOwnerUserId: 'users:old',
deletedAt: now - 1_000,
expiresAt: now + 10_000,
}
const db = {
get: vi.fn(async (id: string) => {
if (id === 'users:admin') return { _id: 'users:admin', role: 'admin' }
if (id === 'users:new') return { _id: 'users:new', role: 'user' }
return null
}),
query: vi.fn((table: string) => {
if (table === 'skills') {
return {
withIndex: (name: string) => {
if (name !== 'by_slug') throw new Error(`unexpected skills index ${name}`)
return { unique: async () => existingSkill }
},
}
}
if (table === 'skillEmbeddings') {
return {
withIndex: (name: string) => {
if (name !== 'by_skill') throw new Error(`unexpected embeddings index ${name}`)
return {
collect: async () => [{ _id: 'skillEmbeddings:1', skillId: 'skills:1', ownerId: 'users:old' }],
}
},
}
}
if (table === 'reservedSlugs') {
return {
withIndex: (name: string) => {
if (name !== 'by_slug_active_deletedAt') {
throw new Error(`unexpected reservedSlugs index ${name}`)
}
return {
order: () => ({
take: async () => [activeReservation],
}),
}
},
}
}
throw new Error(`unexpected table ${table}`)
}),
patch,
insert,
}
const result = (await reclaimSlugInternalHandler(
{ db, scheduler: { runAfter } } as never,
{
actorUserId: 'users:admin',
slug: 'Capability-Evolver',
rightfulOwnerUserId: 'users:new',
transferRootSlugOnly: true,
} as never,
)) as { ok: boolean; action: string }
expect(result).toEqual({ ok: true, action: 'ownership_transferred' })
expect(runAfter).not.toHaveBeenCalled()
expect(patch).toHaveBeenCalledWith(
'skills:1',
expect.objectContaining({
ownerUserId: 'users:new',
}),
)
expect(patch).toHaveBeenCalledWith(
'skillEmbeddings:1',
expect.objectContaining({
ownerId: 'users:new',
}),
)
expect(patch).toHaveBeenCalledWith(
'reservedSlugs:1',
expect.objectContaining({
releasedAt: expect.any(Number),
}),
)
})
it('returns missing without reserving when transferRootSlugOnly is true and slug does not exist', async () => {
const insert = vi.fn(async () => {})
const patch = vi.fn(async () => {})
const runAfter = vi.fn(async () => {})
const db = {
get: vi.fn(async (id: string) => {
if (id === 'users:admin') return { _id: 'users:admin', role: 'admin' }
if (id === 'users:new') return { _id: 'users:new', role: 'user' }
return null
}),
query: vi.fn((table: string) => {
if (table === 'skills') {
return {
withIndex: (name: string) => {
if (name !== 'by_slug') throw new Error(`unexpected skills index ${name}`)
return { unique: async () => null }
},
}
}
throw new Error(`unexpected table ${table}`)
}),
patch,
insert,
}
const result = (await reclaimSlugInternalHandler(
{ db, scheduler: { runAfter } } as never,
{
actorUserId: 'users:admin',
slug: 'missing-slug',
rightfulOwnerUserId: 'users:new',
transferRootSlugOnly: true,
} as never,
)) as { ok: boolean; action: string }
expect(result).toEqual({ ok: true, action: 'missing' })
expect(runAfter).not.toHaveBeenCalled()
expect(patch).not.toHaveBeenCalled()
})
})
+265 -55
View File
@@ -17,13 +17,18 @@ import {
getSkillBadgeMap,
getSkillBadgeMaps,
isSkillHighlighted,
type SkillBadgeMap,
} from './lib/badges'
import { generateChangelogPreview as buildChangelogPreview } from './lib/changelog'
import {
canHealSkillOwnershipByGitHubProviderAccountId,
getGitHubProviderAccountId,
} from './lib/githubIdentity'
import {
adjustGlobalPublicSkillsCount,
countPublicSkillsForGlobalStats,
getPublicSkillVisibilityDelta,
readGlobalPublicSkillsCount,
} from './lib/globalStats'
import { buildTrendingLeaderboard } from './lib/leaderboards'
import { deriveModerationFlags } from './lib/moderation'
import { toPublicSkill, toPublicUser } from './lib/public'
@@ -32,6 +37,7 @@ import { scheduleNextBatchIfNeeded } from './lib/batching'
import {
enforceReservedSlugCooldownForNewSkill,
getLatestActiveReservedSlug,
listActiveReservedSlugsForSlug,
reserveSlugForHardDeleteFinalize,
upsertReservedSlugForRightfulOwner,
} from './lib/reservedSlugs'
@@ -54,7 +60,6 @@ const MAX_LIST_LIMIT = 50
const MAX_PUBLIC_LIST_LIMIT = 200
const MAX_LIST_BULK_LIMIT = 200
const MAX_LIST_TAKE = 1000
const MAX_BADGE_LOOKUP_SKILLS = 200
const HARD_DELETE_BATCH_SIZE = 100
const HARD_DELETE_VERSION_BATCH_SIZE = 10
const HARD_DELETE_LEADERBOARD_BATCH_SIZE = 25
@@ -117,6 +122,16 @@ function normalizeScannerSuspiciousReason(reason: string | undefined) {
return `${reason.slice(0, -'.suspicious'.length)}.clean`
}
async function adjustGlobalPublicCountForSkillChange(
ctx: MutationCtx,
previousSkill: Doc<'skills'> | null | undefined,
nextSkill: Doc<'skills'> | null | undefined,
) {
const delta = getPublicSkillVisibilityDelta(previousSkill, nextSkill)
if (delta === 0) return
await adjustGlobalPublicSkillsCount(ctx, delta)
}
async function getOwnerTrustSignals(
ctx: QueryCtx | MutationCtx,
owner: Doc<'users'>,
@@ -219,7 +234,9 @@ async function hardDeleteSkillStep(
if (Object.keys(patch).length) {
patch.lastReviewedAt = now
patch.updatedAt = now
const nextSkill = { ...skill, ...patch }
await ctx.db.patch(skill._id, patch)
await adjustGlobalPublicCountForSkillChange(ctx, skill, nextSkill)
}
switch (phase) {
@@ -491,18 +508,16 @@ type ManagementSkillEntry = {
type BadgeKind = Doc<'skillBadges'>['kind']
async function buildPublicSkillEntries(ctx: QueryCtx, skills: Doc<'skills'>[]) {
async function buildPublicSkillEntries(
ctx: QueryCtx,
skills: Doc<'skills'>[],
opts?: { includeVersion?: boolean },
) {
const includeVersion = opts?.includeVersion ?? true
const ownerInfoCache = new Map<
Id<'users'>,
Promise<{ ownerHandle: string | null; owner: ReturnType<typeof toPublicUser> | null }>
>()
const badgeMapBySkillId: Map<Id<'skills'>, SkillBadgeMap> = skills.length <=
MAX_BADGE_LOOKUP_SKILLS
? await getSkillBadgeMaps(
ctx,
skills.map((skill) => skill._id),
)
: new Map()
const getOwnerInfo = (ownerUserId: Id<'users'>) => {
const cached = ownerInfoCache.get(ownerUserId)
@@ -523,11 +538,10 @@ async function buildPublicSkillEntries(ctx: QueryCtx, skills: Doc<'skills'>[]) {
const entries = await Promise.all(
skills.map(async (skill) => {
const [latestVersionDoc, ownerInfo] = await Promise.all([
skill.latestVersionId ? ctx.db.get(skill.latestVersionId) : null,
includeVersion && skill.latestVersionId ? ctx.db.get(skill.latestVersionId) : null,
getOwnerInfo(skill.ownerUserId),
])
const badges = badgeMapBySkillId.get(skill._id) ?? {}
const publicSkill = toPublicSkill({ ...skill, badges })
const publicSkill = toPublicSkill(skill)
if (!publicSkill) return null
const latestVersion = toPublicSkillListVersion(latestVersionDoc)
return {
@@ -626,14 +640,24 @@ async function upsertSkillBadge(
.unique()
if (existing) {
await ctx.db.patch(existing._id, { byUserId: userId, at })
return existing._id
} else {
await ctx.db.insert('skillBadges', {
skillId,
kind,
byUserId: userId,
at,
})
}
// Keep denormalized badges field on skill doc in sync
const skill = await ctx.db.get(skillId)
if (skill) {
await ctx.db.patch(skillId, {
badges: {
...(skill.badges as Record<string, unknown> | undefined),
[kind]: { byUserId: userId, at },
},
})
}
return ctx.db.insert('skillBadges', {
skillId,
kind,
byUserId: userId,
at,
})
}
async function removeSkillBadge(ctx: MutationCtx, skillId: Id<'skills'>, kind: BadgeKind) {
@@ -644,6 +668,12 @@ async function removeSkillBadge(ctx: MutationCtx, skillId: Id<'skills'>, kind: B
if (existing) {
await ctx.db.delete(existing._id)
}
// Keep denormalized badges field on skill doc in sync
const skill = await ctx.db.get(skillId)
if (skill) {
const { [kind]: _, ...remainingBadges } = (skill.badges ?? {}) as Record<string, unknown>
await ctx.db.patch(skillId, { badges: remainingBadges })
}
}
export const getBySlug = query({
@@ -887,7 +917,9 @@ export const clearOwnerSuspiciousFlagsInternal = internalMutation({
patch.moderationStatus = 'active'
}
const nextSkill = { ...skill, ...patch }
await ctx.db.patch(skill._id, patch)
await adjustGlobalPublicCountForSkillChange(ctx, skill, nextSkill)
updated += 1
}
@@ -1462,7 +1494,9 @@ export const report = mutation({
})
}
const nextSkill = { ...skill, ...updates }
await ctx.db.patch(skill._id, updates)
await adjustGlobalPublicCountForSkillChange(ctx, skill, nextSkill)
if (shouldAutoHide) {
await setSkillEmbeddingsSoftDeleted(ctx, skill._id, true, now)
@@ -1591,8 +1625,9 @@ export const listPublicPageV2 = query({
})
: result.page
// Build the public skill entries (fetch latestVersion + ownerHandle)
const items = await buildPublicSkillEntries(ctx, filteredPage)
// Build the public skill entries — skip version doc reads to reduce bandwidth.
// Version data is only needed for detail pages, not the listing.
const items = await buildPublicSkillEntries(ctx, filteredPage, { includeVersion: false })
return { ...result, page: items }
},
})
@@ -1632,6 +1667,16 @@ function isCursorParseError(error: unknown) {
return false
}
export const countPublicSkills = query({
args: {},
handler: async (ctx) => {
const statsCount = await readGlobalPublicSkillsCount(ctx)
if (typeof statsCount === 'number') return statsCount
// Fallback for uninitialized/missing globalStats storage.
return countPublicSkillsForGlobalStats(ctx)
},
})
function sortToIndex(
sort: 'downloads' | 'stars' | 'installsCurrent' | 'installsAllTime',
):
@@ -1723,19 +1768,47 @@ export const getSkillByIdInternal = internalQuery({
})
export const getPendingScanSkillsInternal = internalQuery({
args: { limit: v.optional(v.number()), skipRecentMinutes: v.optional(v.number()) },
args: {
limit: v.optional(v.number()),
skipRecentMinutes: v.optional(v.number()),
exhaustive: v.optional(v.boolean()),
},
handler: async (ctx, args) => {
const limit = clampInt(args.limit ?? 10, 1, 100)
const skipRecentMinutes = args.skipRecentMinutes ?? 60
const exhaustive = args.exhaustive ?? false
const limit = exhaustive ? Math.max(1, Math.floor(args.limit ?? 10000)) : clampInt(args.limit ?? 10, 1, 100)
const skipRecentMinutes = exhaustive ? 0 : (args.skipRecentMinutes ?? 60)
const skipThreshold = Date.now() - skipRecentMinutes * 60 * 1000
// Use an indexed query and bounded scan to avoid full-table reads under spam/high volume.
const poolSize = Math.min(Math.max(limit * 20, 200), 1000)
const allSkills = await ctx.db
.query('skills')
.withIndex('by_active_updated', (q) => q.eq('softDeletedAt', undefined))
.order('desc')
.take(poolSize)
let allSkills: Doc<'skills'>[] = []
if (exhaustive) {
// Used by manual/backfill tooling where fairness matters more than query cost.
allSkills = await ctx.db
.query('skills')
.withIndex('by_active_updated', (q) => q.eq('softDeletedAt', undefined))
.collect()
} else {
// Mix "most recently updated" with "oldest created" slices so older pending
// items don't starve behind high-churn records.
const poolSize = Math.min(Math.max(limit * 20, 200), 1000)
const [recentSkills, oldestSkills] = await Promise.all([
ctx.db
.query('skills')
.withIndex('by_active_updated', (q) => q.eq('softDeletedAt', undefined))
.order('desc')
.take(poolSize),
ctx.db
.query('skills')
.withIndex('by_active_created', (q) => q.eq('softDeletedAt', undefined))
.order('asc')
.take(poolSize),
])
const deduped = new Map<Id<'skills'>, Doc<'skills'>>()
for (const skill of [...recentSkills, ...oldestSkills]) {
deduped.set(skill._id, skill)
}
allSkills = [...deduped.values()]
}
const candidates = allSkills.filter((skill) => {
const reason = skill.moderationReason
@@ -1750,10 +1823,11 @@ export const getPendingScanSkillsInternal = internalQuery({
)
})
// Filter out recently checked skills
const skills = candidates.filter(
(s) => !s.scanLastCheckedAt || s.scanLastCheckedAt < skipThreshold,
)
// Filter out recently checked skills unless caller explicitly disables recency filtering.
const skills =
skipRecentMinutes <= 0
? candidates
: candidates.filter((s) => !s.scanLastCheckedAt || s.scanLastCheckedAt < skipThreshold)
// Shuffle and take the requested limit (Fisher-Yates)
for (let i = skills.length - 1; i > 0; i--) {
@@ -1769,10 +1843,13 @@ export const getPendingScanSkillsInternal = internalQuery({
checkCount: number
}> = []
const FINAL_VT_STATUSES = new Set(['clean', 'malicious', 'suspicious'])
for (const skill of selected) {
const version = skill.latestVersionId ? await ctx.db.get(skill.latestVersionId) : null
// Skip skills where version already has vtAnalysis or lacks sha256hash
if (version?.vtAnalysis || !version?.sha256hash) continue
if (!version?.sha256hash) continue
const vtStatus = version.vtAnalysis?.status?.trim().toLowerCase()
// Keep retrying unresolved VT results (pending/stale/error), but skip finalized outcomes.
if (vtStatus && FINAL_VT_STATUSES.has(vtStatus)) continue
results.push({
skillId: skill._id,
versionId: version?._id ?? null,
@@ -1954,6 +2031,7 @@ export const getActiveSkillBatchForRescanInternal = internalQuery({
versionId: Id<'skillVersions'>
sha256hash: string
slug: string
wasFlagged: boolean
}> = []
let nextCursor = cursor
@@ -1974,6 +2052,8 @@ export const getActiveSkillBatchForRescanInternal = internalQuery({
versionId: version._id,
sha256hash: version.sha256hash,
slug: skill.slug,
wasFlagged:
(skill.moderationFlags as string[] | undefined)?.includes('flagged.suspicious') ?? false,
})
}
@@ -2171,9 +2251,13 @@ export const getSkillsWithNullModerationStatusInternal = internalQuery({
export const setSkillModerationStatusActiveInternal = internalMutation({
args: { skillId: v.id('skills') },
handler: async (ctx, args) => {
await ctx.db.patch(args.skillId, {
moderationStatus: 'active',
})
const skill = await ctx.db.get(args.skillId)
if (!skill) return
const patch: Partial<Doc<'skills'>> = { moderationStatus: 'active' }
const nextSkill = { ...skill, ...patch }
await ctx.db.patch(args.skillId, patch)
await adjustGlobalPublicCountForSkillChange(ctx, skill, nextSkill)
},
})
@@ -2279,7 +2363,9 @@ export const applyBanToOwnedSkillsBatchInternal = internalMutation({
hiddenCount += 1
}
const nextSkill = { ...skill, ...patch }
await ctx.db.patch(skill._id, patch)
await adjustGlobalPublicCountForSkillChange(ctx, skill, nextSkill)
await setSkillEmbeddingsSoftDeleted(ctx, skill._id, true, args.bannedAt)
}
@@ -2319,7 +2405,7 @@ export const restoreOwnedSkillsForUnbanBatchInternal = internalMutation({
continue
}
await ctx.db.patch(skill._id, {
const patch: Partial<Doc<'skills'>> = {
softDeletedAt: undefined,
moderationStatus: 'active',
moderationReason: 'restored.unban',
@@ -2327,7 +2413,10 @@ export const restoreOwnedSkillsForUnbanBatchInternal = internalMutation({
hiddenBy: undefined,
lastReviewedAt: now,
updatedAt: now,
})
}
const nextSkill = { ...skill, ...patch }
await ctx.db.patch(skill._id, patch)
await adjustGlobalPublicCountForSkillChange(ctx, skill, nextSkill)
await setSkillEmbeddingsSoftDeleted(ctx, skill._id, false, now)
restoredCount += 1
@@ -2557,7 +2646,6 @@ export const approveSkillByHashInternal = internalMutation({
const existingFlags: string[] = (skill.moderationFlags as string[] | undefined) ?? []
const existingReason: string | undefined = skill.moderationReason as string | undefined
const alreadyBlocked = existingFlags.includes('blocked.malware')
const alreadyFlagged = existingFlags.includes('flagged.suspicious')
const bypassSuspicious =
isSuspicious && !alreadyBlocked && isPrivilegedOwnerForSuspiciousBypass(owner)
@@ -2566,8 +2654,8 @@ export const approveSkillByHashInternal = internalMutation({
if (isMalicious || alreadyBlocked) {
// Malicious from ANY scanner → blocked.malware (upgrade from suspicious)
newFlags = ['blocked.malware']
} else if ((isSuspicious || alreadyFlagged) && !bypassSuspicious) {
// Suspicious from ANY scanner → flagged.suspicious
} else if (isSuspicious && !bypassSuspicious) {
// Suspicious from this scanner → flagged.suspicious
newFlags = ['flagged.suspicious']
} else if (isClean) {
// Clean from this scanner — only clear if no other scanner has flagged
@@ -2595,7 +2683,7 @@ export const approveSkillByHashInternal = internalMutation({
'Quality gate quarantine is still active. Manual moderation review required.')
: undefined
await ctx.db.patch(skill._id, {
const patch: Partial<Doc<'skills'>> = {
moderationStatus: nextModerationStatus,
moderationReason: nextModerationReason,
moderationFlags: newFlags,
@@ -2604,7 +2692,10 @@ export const approveSkillByHashInternal = internalMutation({
hiddenBy: undefined,
lastReviewedAt: nextModerationStatus === 'hidden' ? now : undefined,
updatedAt: now,
})
}
const nextSkill = { ...skill, ...patch }
await ctx.db.patch(skill._id, patch)
await adjustGlobalPublicCountForSkillChange(ctx, skill, nextSkill)
// Auto-ban authors of malicious skills (skips moderators/admins)
if (isMalicious && skill.ownerUserId) {
@@ -2657,7 +2748,7 @@ export const escalateByVtInternal = internalMutation({
newFlags = ['flagged.suspicious']
}
const patch: Record<string, unknown> = {
const patch: Partial<Doc<'skills'>> = {
moderationFlags: newFlags.length ? newFlags : undefined,
updatedAt: Date.now(),
}
@@ -2672,7 +2763,9 @@ export const escalateByVtInternal = internalMutation({
patch.moderationStatus = 'hidden'
}
const nextSkill = { ...skill, ...patch }
await ctx.db.patch(skill._id, patch)
await adjustGlobalPublicCountForSkillChange(ctx, skill, nextSkill)
// Auto-ban authors of malicious skills
if (isMalicious && skill.ownerUserId) {
@@ -2962,14 +3055,17 @@ export const setSoftDeleted = mutation({
if (!skill) throw new Error('Skill not found')
const now = Date.now()
await ctx.db.patch(skill._id, {
const patch: Partial<Doc<'skills'>> = {
softDeletedAt: args.deleted ? now : undefined,
moderationStatus: args.deleted ? 'hidden' : 'active',
hiddenAt: args.deleted ? now : undefined,
hiddenBy: args.deleted ? user._id : undefined,
lastReviewedAt: now,
updatedAt: now,
})
}
const nextSkill = { ...skill, ...patch }
await ctx.db.patch(skill._id, patch)
await adjustGlobalPublicCountForSkillChange(ctx, skill, nextSkill)
await setSkillEmbeddingsSoftDeleted(ctx, skill._id, args.deleted, now)
@@ -3024,6 +3120,38 @@ export const changeOwner = mutation({
},
})
async function transferSkillOwnershipAndEmbeddings(
ctx: MutationCtx,
params: {
skill: Doc<'skills'>
ownerUserId: Id<'users'>
now: number
},
) {
if (params.skill.ownerUserId === params.ownerUserId) return
await ctx.db.patch(params.skill._id, {
ownerUserId: params.ownerUserId,
lastReviewedAt: params.now,
updatedAt: params.now,
})
const embeddings = await listSkillEmbeddingsForSkill(ctx, params.skill._id)
for (const embedding of embeddings) {
await ctx.db.patch(embedding._id, {
ownerId: params.ownerUserId,
updatedAt: params.now,
})
}
}
async function releaseActiveReservationsForSlug(ctx: MutationCtx, slug: string, releasedAt: number) {
const active = await listActiveReservedSlugsForSlug(ctx, slug)
for (const reservation of active) {
await ctx.db.patch(reservation._id, { releasedAt })
}
}
/**
* Admin-only: reclaim a squatted slug by hard-deleting the squatter's skill
* and reserving the slug for the rightful owner.
@@ -3102,6 +3230,7 @@ export const reclaimSlugInternal = internalMutation({
slug: v.string(),
rightfulOwnerUserId: v.id('users'),
reason: v.optional(v.string()),
transferRootSlugOnly: v.optional(v.boolean()),
},
handler: async (ctx, args) => {
const actor = await ctx.db.get(args.actorUserId)
@@ -3112,12 +3241,82 @@ export const reclaimSlugInternal = internalMutation({
if (!slug) throw new Error('Slug required')
const now = Date.now()
const transferRootSlugOnly = args.transferRootSlugOnly === true
const rightfulOwner = await ctx.db.get(args.rightfulOwnerUserId)
if (!rightfulOwner || rightfulOwner.deletedAt || rightfulOwner.deactivatedAt) {
throw new Error('Rightful owner not found')
}
const existingSkill = await ctx.db
.query('skills')
.withIndex('by_slug', (q) => q.eq('slug', slug))
.unique()
if (transferRootSlugOnly) {
if (!existingSkill) {
await ctx.db.insert('auditLogs', {
actorUserId: args.actorUserId,
action: 'slug.reclaim',
targetType: 'slug',
targetId: slug,
metadata: {
slug,
rightfulOwnerUserId: args.rightfulOwnerUserId,
transferRootSlugOnly: true,
action: 'missing',
reason: args.reason || undefined,
},
createdAt: now,
})
return { ok: true as const, action: 'missing' as const }
}
if (existingSkill.ownerUserId === args.rightfulOwnerUserId) {
await releaseActiveReservationsForSlug(ctx, slug, now)
await ctx.db.insert('auditLogs', {
actorUserId: args.actorUserId,
action: 'slug.reclaim',
targetType: 'slug',
targetId: slug,
metadata: {
slug,
rightfulOwnerUserId: args.rightfulOwnerUserId,
transferRootSlugOnly: true,
action: 'already_owned',
reason: args.reason || undefined,
},
createdAt: now,
})
return { ok: true as const, action: 'already_owned' as const }
}
await transferSkillOwnershipAndEmbeddings(ctx, {
skill: existingSkill,
ownerUserId: args.rightfulOwnerUserId,
now,
})
await releaseActiveReservationsForSlug(ctx, slug, now)
await ctx.db.insert('auditLogs', {
actorUserId: args.actorUserId,
action: 'slug.reclaim',
targetType: 'slug',
targetId: slug,
metadata: {
slug,
rightfulOwnerUserId: args.rightfulOwnerUserId,
previousOwnerUserId: existingSkill.ownerUserId,
hadSquatter: true,
transferRootSlugOnly: true,
action: 'ownership_transferred',
reason: args.reason || undefined,
},
createdAt: now,
})
return { ok: true as const, action: 'ownership_transferred' as const }
}
if (existingSkill && existingSkill.ownerUserId !== args.rightfulOwnerUserId) {
await ctx.scheduler.runAfter(0, internal.skills.hardDeleteInternal, {
skillId: existingSkill._id,
@@ -3513,6 +3712,9 @@ export const insertVersion = internalMutation({
updatedAt: now,
})
skill = await ctx.db.get(skillId)
if (skill) {
await adjustGlobalPublicCountForSkillChange(ctx, null, skill)
}
}
if (!skill) throw new Error('Skill creation failed')
@@ -3554,7 +3756,7 @@ export const insertVersion = internalMutation({
files: args.files,
})
await ctx.db.patch(skill._id, {
const patch: Partial<Doc<'skills'>> = {
displayName: args.displayName,
summary: nextSummary ?? undefined,
latestVersionId: versionId,
@@ -3567,7 +3769,10 @@ export const insertVersion = internalMutation({
quality: qualityRecord ?? skill.quality,
moderationFlags: moderationFlags.length ? moderationFlags : undefined,
updatedAt: now,
})
}
const nextSkill = { ...skill, ...patch }
await ctx.db.patch(skill._id, patch)
await adjustGlobalPublicCountForSkillChange(ctx, skill, nextSkill)
const badgeMap = await getSkillBadgeMap(ctx, skill._id)
const isApproved = Boolean(badgeMap.redactionApproved)
@@ -3582,6 +3787,8 @@ export const insertVersion = internalMutation({
visibility: embeddingVisibilityFor(true, isApproved),
updatedAt: now,
})
// Lightweight lookup so search hydration can skip reading the 12KB embedding doc
await ctx.db.insert('embeddingSkillMap', { embeddingId, skillId: skill._id })
if (latestBefore) {
const previousEmbedding = await ctx.db
@@ -3632,14 +3839,17 @@ export const setSkillSoftDeletedInternal = internalMutation({
}
const now = Date.now()
await ctx.db.patch(skill._id, {
const patch: Partial<Doc<'skills'>> = {
softDeletedAt: args.deleted ? now : undefined,
moderationStatus: args.deleted ? 'hidden' : 'active',
hiddenAt: args.deleted ? now : undefined,
hiddenBy: args.deleted ? args.userId : undefined,
lastReviewedAt: now,
updatedAt: now,
})
}
const nextSkill = { ...skill, ...patch }
await ctx.db.patch(skill._id, patch)
await adjustGlobalPublicCountForSkillChange(ctx, skill, nextSkill)
await setSkillEmbeddingsSoftDeleted(ctx, skill._id, args.deleted, now)
+12
View File
@@ -3,6 +3,10 @@ import { internal } from './_generated/api'
import type { Doc } from './_generated/dataModel'
import type { ActionCtx } from './_generated/server'
import { internalAction, internalMutation, internalQuery } from './_generated/server'
import {
countPublicSkillsForGlobalStats,
setGlobalPublicSkillsCount,
} from './lib/globalStats'
const DEFAULT_BATCH_SIZE = 200
const MAX_BATCH_SIZE = 1000
@@ -299,3 +303,11 @@ export const runReconcileSkillStarCountsInternal = internalAction({
function clampInt(value: number, min: number, max: number) {
return Math.min(Math.max(value, min), max)
}
export const updateGlobalStatsInternal = internalMutation({
args: {},
handler: async (ctx) => {
const count = await countPublicSkillsForGlobalStats(ctx)
await setGlobalPublicSkillsCount(ctx, count)
},
})
+60
View File
@@ -0,0 +1,60 @@
import { describe, expect, it } from 'vitest'
import { __test } from './vt'
describe('vt activation fallback', () => {
it('activates only VT-pending hidden skills', () => {
expect(
__test.shouldActivateWhenVtUnavailable({
moderationStatus: 'hidden',
moderationReason: 'pending.scan',
}),
).toBe(true)
expect(
__test.shouldActivateWhenVtUnavailable({
moderationStatus: 'hidden',
moderationReason: 'scanner.vt.pending',
}),
).toBe(true)
expect(
__test.shouldActivateWhenVtUnavailable({
moderationStatus: 'hidden',
moderationReason: 'pending.scan.stale',
}),
).toBe(true)
})
it('does not activate quality or scanner-hidden skills', () => {
expect(
__test.shouldActivateWhenVtUnavailable({
moderationStatus: 'hidden',
moderationReason: 'quality.low',
}),
).toBe(false)
expect(
__test.shouldActivateWhenVtUnavailable({
moderationStatus: 'hidden',
moderationReason: 'scanner.llm.malicious',
}),
).toBe(false)
})
it('does not activate blocked or already-active skills', () => {
expect(
__test.shouldActivateWhenVtUnavailable({
moderationStatus: 'hidden',
moderationReason: 'pending.scan',
moderationFlags: ['blocked.malware'],
}),
).toBe(false)
expect(
__test.shouldActivateWhenVtUnavailable({
moderationStatus: 'active',
moderationReason: 'pending.scan',
}),
).toBe(false)
})
})
+50 -2
View File
@@ -1,6 +1,7 @@
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Id } from './_generated/dataModel'
import type { ActionCtx } from './_generated/server'
import { action, internalAction, internalMutation } from './_generated/server'
import { buildDeterministicZip } from './lib/skillZip'
@@ -136,6 +137,13 @@ type PendingScanSkill = {
checkCount: number
}
type SkillActivationCandidate = {
moderationStatus?: string
moderationReason?: string
moderationFlags?: string[]
softDeletedAt?: number
}
type PollPendingScansResult = {
processed: number
updated: number
@@ -228,6 +236,23 @@ type SyncModerationReasonsResult = {
done: boolean
}
const VT_PENDING_REASONS = new Set(['pending.scan', 'scanner.vt.pending', 'pending.scan.stale'])
function shouldActivateWhenVtUnavailable(skill: SkillActivationCandidate | null | undefined) {
if (!skill || skill.softDeletedAt) return false
if (skill.moderationFlags?.includes('blocked.malware')) return false
if (skill.moderationStatus === 'active') return false
const reason = skill.moderationReason
return typeof reason === 'string' && VT_PENDING_REASONS.has(reason)
}
async function activateSkillWhenVtUnavailable(ctx: ActionCtx, skillId: Id<'skills'>) {
const skill = await ctx.runQuery(internal.skills.getSkillByIdInternal, { skillId })
if (!shouldActivateWhenVtUnavailable(skill)) return
await ctx.runMutation(internal.skills.setSkillModerationStatusActiveInternal, { skillId })
}
export const fetchResults = action({
args: {
sha256hash: v.optional(v.string()),
@@ -305,7 +330,13 @@ export const scanWithVirusTotal = internalAction({
handler: async (ctx, args) => {
const apiKey = process.env.VT_API_KEY
if (!apiKey) {
console.log('VT_API_KEY not configured, skipping scan')
console.log('VT_API_KEY not configured, skipping scan — activating skill')
const version = await ctx.runQuery(internal.skills.getVersionByIdInternal, {
versionId: args.versionId,
})
if (version) {
await activateSkillWhenVtUnavailable(ctx, version.skillId)
}
return
}
@@ -524,6 +555,7 @@ export const pollPendingScans = internalAction({
versionId,
vtAnalysis: { status: 'stale', checkedAt: Date.now() },
})
await activateSkillWhenVtUnavailable(ctx, skillId)
staled++
}
continue
@@ -549,6 +581,7 @@ export const pollPendingScans = internalAction({
versionId,
vtAnalysis: { status: 'stale', checkedAt: Date.now() },
})
await activateSkillWhenVtUnavailable(ctx, skillId)
staled++
}
continue
@@ -650,6 +683,10 @@ async function requestRescan(apiKey: string, sha256hash: string): Promise<boolea
}
}
export const __test = {
shouldActivateWhenVtUnavailable,
}
/**
* Backfill function to process ALL pending skills at once
* Run manually to clear backlog
@@ -672,6 +709,8 @@ export const backfillPendingScans = internalAction({
internal.skills.getPendingScanSkillsInternal,
{
limit: 10000,
exhaustive: true,
skipRecentMinutes: 0,
},
)
@@ -787,7 +826,7 @@ export const rescanActiveSkills = internalAction({
`[vt:rescan] Processing batch of ${batch.skills.length} skills (cursor=${cursor}, accumulated=${accTotal})`,
)
for (const { versionId, sha256hash, slug } of batch.skills) {
for (const { versionId, sha256hash, slug, wasFlagged } of batch.skills) {
try {
const vtResult = await checkExistingFile(apiKey, sha256hash)
@@ -834,6 +873,15 @@ export const rescanActiveSkills = internalAction({
status,
})
accUpdated++
} else if (wasFlagged && status === 'clean') {
// Verdict improved from suspicious → clean: clear the stale moderation flag
console.log(`[vt:rescan] ${slug}: verdict improved to clean, clearing suspicious flag`)
await ctx.runMutation(internal.skills.approveSkillByHashInternal, {
sha256hash,
scanner: 'vt',
status,
})
accUpdated++
} else {
accUnchanged++
}
+31 -2
View File
@@ -18,12 +18,41 @@ OpenAPI: `/api/v1/openapi.json`
## Rate limits
Per IP + per API key:
Auth-aware enforcement:
- Anonymous requests: per IP.
- Authenticated requests (valid Bearer token): per user bucket.
- Missing/invalid token falls back to IP enforcement.
- Read: 120/min per IP, 600/min per key
- Write: 30/min per IP, 120/min per key
Headers: `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`, `Retry-After` (on 429).
Headers: `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`, `RateLimit-Limit`, `RateLimit-Remaining`, `RateLimit-Reset`, `Retry-After` (on 429).
Semantics:
- `X-RateLimit-Reset`: Unix epoch seconds (absolute reset time)
- `RateLimit-Reset`: delay seconds until reset
- `Retry-After`: delay seconds to wait on `429`
Example `429`:
```http
HTTP/2 429
x-ratelimit-limit: 20
x-ratelimit-remaining: 0
x-ratelimit-reset: 1771404540
ratelimit-limit: 20
ratelimit-remaining: 0
ratelimit-reset: 34
retry-after: 34
```
Client handling:
- Prefer `Retry-After` when present.
- Otherwise use `RateLimit-Reset` or derive delay from `X-RateLimit-Reset`.
- Add jitter to retries.
## Endpoints
+32 -4
View File
@@ -29,6 +29,34 @@ Env equivalents:
- `CLAWHUB_REGISTRY` (legacy `CLAWDHUB_REGISTRY`)
- `CLAWHUB_WORKDIR` (legacy `CLAWDHUB_WORKDIR`)
### HTTP proxy
The CLI respects standard HTTP proxy environment variables for systems behind
corporate proxies or restricted networks:
- `HTTPS_PROXY` / `https_proxy`
- `HTTP_PROXY` / `http_proxy`
- `NO_PROXY` / `no_proxy`
When any of these variables is set, the CLI routes outbound requests through
the specified proxy. `HTTPS_PROXY` is used for HTTPS requests, `HTTP_PROXY`
for plain HTTP. `NO_PROXY` / `no_proxy` is respected to bypass the proxy for
specific hosts or domains.
This is required on systems where direct outbound connections are blocked
(e.g. Docker containers, Hetzner VPS with proxy-only internet, corporate
firewalls).
Example:
```bash
export HTTPS_PROXY=http://proxy.example.com:3128
export NO_PROXY=localhost,127.0.0.1
clawhub search "my query"
```
When no proxy variable is set, behavior is unchanged (direct connections).
## Config file
Stores your API token + cached registry URL.
@@ -111,24 +139,24 @@ Stores your API token + cached registry URL.
### `delete <slug>`
- Soft-delete a skill (moderator/admin only).
- Soft-delete a skill (owner, moderator, or admin).
- Calls `DELETE /api/v1/skills/{slug}`.
- `--yes` skips confirmation.
### `undelete <slug>`
- Restore a hidden skill (moderator/admin only).
- Restore a hidden skill (owner, moderator, or admin).
- Calls `POST /api/v1/skills/{slug}/undelete`.
- `--yes` skips confirmation.
### `hide <slug>`
- Hide a skill (moderator/admin only).
- Hide a skill (owner, moderator, or admin).
- Alias for `delete`.
### `unhide <slug>`
- Unhide a skill (moderator/admin only).
- Unhide a skill (owner, moderator, or admin).
- Alias for `undelete`.
### `ban-user <handleOrId>`
+18
View File
@@ -78,3 +78,21 @@ Then:
clawhub login --site https://<site>
clawhub whoami
```
Rate-limit sanity checks:
```bash
curl -i "https://<site>/api/v1/download?slug=gifgrep"
```
Confirm headers are present:
- `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`
- `RateLimit-Limit`, `RateLimit-Remaining`, `RateLimit-Reset`
- `Retry-After` on `429`
Proxy/IP caveat:
- Default IP source is `cf-connecting-ip`.
- For non-Cloudflare trusted proxy setups, set `TRUST_FORWARDED_IPS=true`.
- If proxy headers are not forwarded/trusted correctly, multiple users may collapse into one IP and hit false-positive rate limits.
+39 -4
View File
@@ -15,7 +15,11 @@ OpenAPI: `/api/v1/openapi.json`.
## Rate limits
Enforced per IP + per API key:
Enforcement model:
- Anonymous requests: enforced per IP.
- Authenticated requests (valid Bearer token): enforced per user bucket.
- If token is missing/invalid, behavior falls back to IP enforcement.
- Read: 120/min per IP, 600/min per key
- Write: 30/min per IP, 120/min per key
@@ -23,12 +27,43 @@ Enforced per IP + per API key:
Headers:
- `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`, `Retry-After` (when limited)
- Legacy compatibility: `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`
- Standardized: `RateLimit-Limit`, `RateLimit-Remaining`, `RateLimit-Reset`
- On `429`: `Retry-After`
Header semantics:
- `X-RateLimit-Reset`: absolute Unix epoch seconds
- `RateLimit-Reset`: seconds until reset (delay)
- `Retry-After`: seconds to wait before retry (delay) on `429`
Example `429` response:
```http
HTTP/2 429
content-type: text/plain; charset=utf-8
x-ratelimit-limit: 20
x-ratelimit-remaining: 0
x-ratelimit-reset: 1771404540
ratelimit-limit: 20
ratelimit-remaining: 0
ratelimit-reset: 34
retry-after: 34
Rate limit exceeded
```
Client guidance:
- If `Retry-After` exists, wait that many seconds before retry.
- Use jittered backoff to avoid synchronized retries.
- If `Retry-After` is missing, fallback to `RateLimit-Reset` (or compute from `X-RateLimit-Reset`).
IP source:
- Uses `cf-connecting-ip` (Cloudflare) for client IP by default.
- Set `TRUST_FORWARDED_IPS=true` to opt in to `x-real-ip`, `x-forwarded-for`, or `fly-client-ip` (non-Cloudflare deployments).
- Set `TRUST_FORWARDED_IPS=true` to opt in to `x-forwarded-for`, `x-real-ip`, or `fly-client-ip` (non-Cloudflare deployments).
- If you run behind a reverse proxy/load balancer, ensure real client IP headers are preserved and trusted correctly, or rate limits may be too strict due to shared proxy IPs.
## Public endpoints (no auth)
@@ -154,7 +189,7 @@ Publishes a new version.
### `DELETE /api/v1/skills/{slug}` / `POST /api/v1/skills/{slug}/undelete`
Soft-delete / restore a skill (moderator/admin only).
Soft-delete / restore a skill (owner, moderator, or admin).
Status codes:
+29
View File
@@ -18,6 +18,35 @@ read_when:
- Token missing or revoked: check your config file (`CLAWHUB_CONFIG_PATH` override?).
- Ensure requests include `Authorization: Bearer ...` (CLI does this automatically).
## CLI/API returns `Rate limit exceeded` (429)
- Read headers in the response:
- `Retry-After` = wait seconds before retry
- `RateLimit-Remaining` + `RateLimit-Limit` = current budget
- `RateLimit-Reset` (or `X-RateLimit-Reset`) = reset timing
- The CLI now includes retry hints in 429 errors (retry delay + remaining budget).
- If many users share one egress IP (NAT/proxy), IP limit can be hit even with valid tokens.
- For non-Cloudflare deploys behind trusted proxies, set `TRUST_FORWARDED_IPS=true` so forwarded client IPs can be used.
## `search` / `install` fails with `fetch failed` behind a proxy
If your system requires an HTTP proxy for outbound connections (e.g. corporate
firewalls, Docker containers with proxy-only internet, Hetzner VPS), the CLI
will fail with:
```
✖ fetch failed
Error: fetch failed
```
**Fix:** Set the standard proxy environment variables:
```bash
export HTTPS_PROXY=http://proxy.example.com:3128
clawhub search "my query"
```
The CLI respects `HTTPS_PROXY`, `HTTP_PROXY`, `https_proxy`, and `http_proxy`.
## `publish` fails with `OPENAI_API_KEY is not configured`
- Set `OPENAI_API_KEY` in the Convex environment (not only locally).
+2 -2
View File
@@ -42,8 +42,8 @@ test('header menu routes render', async ({ page }) => {
}
if (label === 'Search') {
await expect(page).toHaveURL(/\/?(\?|$)/)
await expect(page.locator('h1', { hasText: 'ClawHub' })).toBeVisible()
await expect(page).toHaveURL(/\/skills(\?|$)/)
await expect(page.locator('h1', { hasText: 'Skills' })).toBeVisible()
}
}
})
+31
View File
@@ -22,6 +22,17 @@ function restoreBunRuntime() {
})
}
function mockImmediateTimeouts() {
const setTimeoutMock = vi.fn((callback: () => void) => {
callback()
return 1 as unknown as ReturnType<typeof setTimeout>
})
const clearTimeoutMock = vi.fn()
vi.stubGlobal('setTimeout', setTimeoutMock as unknown as typeof setTimeout)
vi.stubGlobal('clearTimeout', clearTimeoutMock as typeof clearTimeout)
return { setTimeoutMock, clearTimeoutMock }
}
async function loadHttpModuleWithBunMocks(opts?: {
spawnImpl?: ReturnType<typeof vi.fn>
mkdtempValue?: string
@@ -119,6 +130,26 @@ describe('http bun runtime', () => {
expect(spawnSync).toHaveBeenCalledTimes(3)
})
it('includes rate-limit guidance from curl metadata on 429', async () => {
mockImmediateTimeouts()
const spawnSync = vi.fn().mockReturnValue({
status: 0,
stdout:
'rate limited\n__CLAWHUB_CURL_META__\n429\n20\n0\n1771404540\n20\n0\n34\n34\n',
stderr: '',
})
const { http } = await loadHttpModuleWithBunMocks({ spawnImpl: spawnSync })
await expect(
http.apiRequest('https://registry.example', {
method: 'GET',
path: '/v1/ping',
}),
).rejects.toThrow(/retry in 34s.*remaining: 0\/20.*reset in 34s/i)
expect(spawnSync).toHaveBeenCalledTimes(3)
})
it('does not retry bun apiRequest on 404 errors', async () => {
const spawnSync = vi.fn().mockReturnValue({
status: 0,
+74 -1
View File
@@ -1,7 +1,7 @@
/* @vitest-environment node */
import { describe, expect, it, vi } from 'vitest'
import { apiRequest, apiRequestForm, downloadZip, fetchText } from './http'
import { apiRequest, apiRequestForm, downloadZip, fetchText, shouldUseProxyFromEnv } from './http'
import { ApiV1WhoamiResponseSchema } from './schema/index.js'
function mockImmediateTimeouts() {
@@ -36,6 +36,35 @@ function createAbortingFetchMock() {
})
}
describe('shouldUseProxyFromEnv', () => {
it('detects standard proxy variables', () => {
expect(
shouldUseProxyFromEnv({
HTTPS_PROXY: 'http://proxy.example:3128',
} as NodeJS.ProcessEnv),
).toBe(true)
expect(
shouldUseProxyFromEnv({
HTTP_PROXY: 'http://proxy.example:3128',
} as NodeJS.ProcessEnv),
).toBe(true)
expect(
shouldUseProxyFromEnv({
https_proxy: 'http://proxy.example:3128',
} as NodeJS.ProcessEnv),
).toBe(true)
})
it('ignores NO_PROXY-only configs', () => {
expect(
shouldUseProxyFromEnv({
NO_PROXY: 'localhost,127.0.0.1',
} as NodeJS.ProcessEnv),
).toBe(false)
expect(shouldUseProxyFromEnv({} as NodeJS.ProcessEnv)).toBe(false)
})
})
describe('apiRequest', () => {
it('adds bearer token and parses json', async () => {
const fetchMock = vi.fn().mockResolvedValue({
@@ -86,6 +115,50 @@ describe('apiRequest', () => {
vi.unstubAllGlobals()
})
it('includes rate-limit guidance from headers on 429', async () => {
mockImmediateTimeouts()
const fetchMock = vi.fn().mockResolvedValue({
ok: false,
status: 429,
headers: new Headers({
'Retry-After': '34',
'X-RateLimit-Limit': '20',
'X-RateLimit-Remaining': '0',
'X-RateLimit-Reset': '1771404540',
}),
text: async () => 'Rate limit exceeded',
})
vi.stubGlobal('fetch', fetchMock)
await expect(apiRequest('https://example.com', { method: 'GET', path: '/x' })).rejects.toThrow(
/retry in 34s.*remaining: 0\/20.*reset in 34s/i,
)
expect(fetchMock).toHaveBeenCalledTimes(3)
vi.unstubAllGlobals()
})
it('interprets legacy epoch Retry-After values as reset delays', async () => {
mockImmediateTimeouts()
vi.spyOn(Date, 'now').mockReturnValue(1_771_404_500_000)
const fetchMock = vi.fn().mockResolvedValue({
ok: false,
status: 429,
headers: new Headers({
'Retry-After': '1771404540',
'X-RateLimit-Limit': '20',
'X-RateLimit-Remaining': '0',
}),
text: async () => 'Rate limit exceeded',
})
vi.stubGlobal('fetch', fetchMock)
await expect(apiRequest('https://example.com', { method: 'GET', path: '/x' })).rejects.toThrow(
/retry in 40s.*remaining: 0\/20/i,
)
vi.restoreAllMocks()
vi.unstubAllGlobals()
})
it('falls back to HTTP status when body is empty', async () => {
const fetchMock = vi.fn().mockResolvedValue({
ok: false,
+267 -50
View File
@@ -3,20 +3,45 @@ import { mkdtemp, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import pRetry, { AbortError } from 'p-retry'
import { Agent, setGlobalDispatcher } from 'undici'
import { Agent, EnvHttpProxyAgent, setGlobalDispatcher } from 'undici'
import type { ArkValidator } from './schema/index.js'
import { ApiRoutes, parseArk } from './schema/index.js'
const REQUEST_TIMEOUT_MS = 15_000
const REQUEST_TIMEOUT_SECONDS = Math.ceil(REQUEST_TIMEOUT_MS / 1000)
const RETRY_COUNT = 2
const RETRY_BACKOFF_BASE_MS = 300
const RETRY_BACKOFF_MAX_MS = 5_000
const RETRY_AFTER_JITTER_MS = 250
const CURL_META_MARKER = '__CLAWHUB_CURL_META__'
const CURL_WRITE_OUT_FORMAT = [
'',
CURL_META_MARKER,
'%{http_code}',
'%{header:x-ratelimit-limit}',
'%{header:x-ratelimit-remaining}',
'%{header:x-ratelimit-reset}',
'%{header:ratelimit-limit}',
'%{header:ratelimit-remaining}',
'%{header:ratelimit-reset}',
'%{header:retry-after}',
].join('\n')
const isBun = typeof process !== 'undefined' && Boolean(process.versions?.bun)
export function shouldUseProxyFromEnv(env: NodeJS.ProcessEnv = process.env): boolean {
return Boolean(env.HTTPS_PROXY || env.HTTP_PROXY || env.https_proxy || env.http_proxy)
}
if (typeof process !== 'undefined' && process.versions?.node) {
try {
setGlobalDispatcher(
new Agent({
connect: { timeout: REQUEST_TIMEOUT_MS },
}),
shouldUseProxyFromEnv(process.env)
? new EnvHttpProxyAgent({
connect: { timeout: REQUEST_TIMEOUT_MS },
})
: new Agent({
connect: { timeout: REQUEST_TIMEOUT_MS },
}),
)
} catch {
// ignore dispatcher setup failures in non-node runtimes
@@ -27,6 +52,27 @@ type RequestArgs =
| { method: 'GET' | 'POST' | 'DELETE'; path: string; token?: string; body?: unknown }
| { method: 'GET' | 'POST' | 'DELETE'; url: string; token?: string; body?: unknown }
type HeaderSource = Headers | Record<string, string> | null | undefined
type RateLimitInfo = {
limit?: number
remaining?: number
resetDelaySeconds?: number
retryAfterSeconds?: number
}
class HttpStatusError extends Error {
readonly status: number
readonly rateLimit: RateLimitInfo
constructor(status: number, message: string, rateLimit: RateLimitInfo) {
super(message)
this.name = 'HttpStatusError'
this.status = status
this.rateLimit = rateLimit
}
}
export async function apiRequest<T>(registry: string, args: RequestArgs): Promise<T>
export async function apiRequest<T>(
registry: string,
@@ -39,7 +85,7 @@ export async function apiRequest<T>(
schema?: ArkValidator<T>,
): Promise<T> {
const url = 'url' in args ? args.url : new URL(args.path, registry).toString()
const json = await pRetry(
const json = await runWithRetries(
async () => {
if (isBun) {
return await fetchJsonViaCurl(url, args)
@@ -58,11 +104,10 @@ export async function apiRequest<T>(
body,
})
if (!response.ok) {
throwHttpStatusError(response.status, await readResponseTextSafe(response))
throwHttpStatusError(response.status, await readResponseTextSafe(response), response.headers)
}
return (await response.json()) as unknown
},
{ retries: 2 },
)
if (schema) return parseArk(schema, json, 'API response')
return json as T
@@ -84,7 +129,7 @@ export async function apiRequestForm<T>(
schema?: ArkValidator<T>,
): Promise<T> {
const url = 'url' in args ? args.url : new URL(args.path, registry).toString()
const json = await pRetry(
const json = await runWithRetries(
async () => {
if (isBun) {
return await fetchJsonFormViaCurl(url, args)
@@ -98,11 +143,10 @@ export async function apiRequestForm<T>(
body: args.form,
})
if (!response.ok) {
throwHttpStatusError(response.status, await readResponseTextSafe(response))
throwHttpStatusError(response.status, await readResponseTextSafe(response), response.headers)
}
return (await response.json()) as unknown
},
{ retries: 2 },
)
if (schema) return parseArk(schema, json, 'API response')
return json as T
@@ -112,7 +156,7 @@ type TextRequestArgs = { path: string; token?: string } | { url: string; token?:
export async function fetchText(registry: string, args: TextRequestArgs): Promise<string> {
const url = 'url' in args ? args.url : new URL(args.path, registry).toString()
return pRetry(
return runWithRetries(
async () => {
if (isBun) {
return await fetchTextViaCurl(url, args)
@@ -123,11 +167,10 @@ export async function fetchText(registry: string, args: TextRequestArgs): Promis
const response = await fetchWithTimeout(url, { method: 'GET', headers })
const text = await response.text()
if (!response.ok) {
throwHttpStatusError(response.status, text)
throwHttpStatusError(response.status, text, response.headers)
}
return text
},
{ retries: 2 },
)
}
@@ -138,7 +181,7 @@ export async function downloadZip(
const url = new URL(ApiRoutes.download, registry)
url.searchParams.set('slug', args.slug)
if (args.version) url.searchParams.set('version', args.version)
return pRetry(
return runWithRetries(
async () => {
if (isBun) {
return await fetchBinaryViaCurl(url.toString(), args.token)
@@ -149,11 +192,10 @@ export async function downloadZip(
const response = await fetchWithTimeout(url.toString(), { method: 'GET', headers })
if (!response.ok) {
throwHttpStatusError(response.status, await readResponseTextSafe(response))
throwHttpStatusError(response.status, await readResponseTextSafe(response), response.headers)
}
return new Uint8Array(await response.arrayBuffer())
},
{ retries: 2 },
)
}
@@ -171,14 +213,152 @@ async function readResponseTextSafe(response: Response): Promise<string> {
return await response.text().catch(() => '')
}
function throwHttpStatusError(status: number, text: string): never {
const message = text || `HTTP ${status}`
async function runWithRetries<T>(fn: () => Promise<T>): Promise<T> {
return await pRetry(fn, {
retries: RETRY_COUNT,
minTimeout: 0,
maxTimeout: 0,
factor: 1,
randomize: false,
onFailedAttempt: async (attemptError) => {
const delayMs = getRetryDelayMs(attemptError)
if (delayMs <= 0) return
await sleep(delayMs)
},
})
}
function getRetryDelayMs(attemptError: unknown): number {
const failed = attemptError as {
attemptNumber?: number
cause?: unknown
error?: unknown
}
const attemptNumber = Math.max(1, Number(failed.attemptNumber ?? 1))
const rootError = failed.cause ?? failed.error ?? attemptError
if (rootError instanceof HttpStatusError && rootError.rateLimit.retryAfterSeconds !== undefined) {
return rootError.rateLimit.retryAfterSeconds * 1000 + jitterMs(RETRY_AFTER_JITTER_MS)
}
const baseMs = Math.min(RETRY_BACKOFF_MAX_MS, RETRY_BACKOFF_BASE_MS * 2 ** (attemptNumber - 1))
return baseMs + jitterMs(RETRY_BACKOFF_BASE_MS)
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => {
setTimeout(resolve, ms)
})
}
function jitterMs(maxMs: number): number {
if (maxMs <= 0) return 0
return Math.floor(Math.random() * maxMs)
}
function throwHttpStatusError(status: number, text: string, headers?: HeaderSource): never {
const rateLimit = parseRateLimitInfo(headers)
const message = buildHttpErrorMessage(status, text, rateLimit)
if (status === 429 || status >= 500) {
throw new Error(message)
throw new HttpStatusError(status, message, rateLimit)
}
throw new AbortError(message)
}
function buildHttpErrorMessage(status: number, text: string, rateLimit: RateLimitInfo): string {
const base = text || `HTTP ${status}`
const details: string[] = []
if (rateLimit.retryAfterSeconds !== undefined) {
details.push(`retry in ${rateLimit.retryAfterSeconds}s`)
}
if (rateLimit.remaining !== undefined && rateLimit.limit !== undefined) {
details.push(`remaining: ${rateLimit.remaining}/${rateLimit.limit}`)
}
if (rateLimit.resetDelaySeconds !== undefined) {
details.push(`reset in ${rateLimit.resetDelaySeconds}s`)
}
if (details.length === 0) {
return base
}
return `${base} (${details.join(', ')})`
}
function parseRateLimitInfo(headers?: HeaderSource): RateLimitInfo {
if (!headers) return {}
const limit = parseIntHeader(getHeader(headers, 'x-ratelimit-limit') ?? getHeader(headers, 'ratelimit-limit'))
const remaining = parseIntHeader(
getHeader(headers, 'x-ratelimit-remaining') ?? getHeader(headers, 'ratelimit-remaining'),
)
const nowMs = Date.now()
const retryAfterSeconds = parseRetryAfterSeconds(getHeader(headers, 'retry-after'), nowMs)
const resetDelaySeconds = parseResetDelaySeconds(headers, nowMs, retryAfterSeconds)
return {
limit,
remaining,
resetDelaySeconds,
retryAfterSeconds,
}
}
function parseResetDelaySeconds(
headers: HeaderSource,
nowMs: number,
retryAfterSeconds: number | undefined,
): number | undefined {
if (retryAfterSeconds !== undefined) return retryAfterSeconds
const standardized = parseIntHeader(getHeader(headers, 'ratelimit-reset'))
if (standardized !== undefined) {
return Math.max(1, standardized)
}
const legacyEpochSeconds = parseIntHeader(getHeader(headers, 'x-ratelimit-reset'))
if (legacyEpochSeconds === undefined) return undefined
const nowSeconds = Math.floor(nowMs / 1000)
return Math.max(1, legacyEpochSeconds - nowSeconds)
}
function parseRetryAfterSeconds(value: string | undefined, nowMs: number): number | undefined {
if (!value) return undefined
const trimmed = value.trim()
if (!trimmed) return undefined
const asNumber = Number(trimmed)
if (Number.isFinite(asNumber) && asNumber >= 0) {
// Compatibility guard for older servers that accidentally sent Unix epoch seconds.
if (asNumber > 31_536_000) {
const nowSeconds = Math.floor(nowMs / 1000)
return Math.max(1, Math.ceil(asNumber - nowSeconds))
}
return Math.max(1, Math.ceil(asNumber))
}
const asDateMs = Date.parse(trimmed)
if (!Number.isFinite(asDateMs)) return undefined
return Math.max(1, Math.ceil((asDateMs - nowMs) / 1000))
}
function parseIntHeader(value: string | undefined): number | undefined {
if (!value) return undefined
const parsed = Number.parseInt(value, 10)
if (!Number.isFinite(parsed)) return undefined
return parsed
}
function getHeader(headers: HeaderSource, key: string): string | undefined {
if (!headers) return undefined
if (headers instanceof Headers) {
const value = headers.get(key)
return value === null ? undefined : value
}
const normalizedKey = key.toLowerCase()
const direct = headers[normalizedKey] ?? headers[key]
if (typeof direct === 'string' && direct.trim()) return direct.trim()
const match = Object.entries(headers).find(
([entryKey, entryValue]) =>
entryKey.toLowerCase() === normalizedKey && typeof entryValue === 'string' && entryValue.trim(),
)
return typeof match?.[1] === 'string' ? match[1].trim() : undefined
}
async function fetchJsonViaCurl(url: string, args: RequestArgs) {
const headers = ['-H', 'Accept: application/json']
if (args.token) {
@@ -191,7 +371,7 @@ async function fetchJsonViaCurl(url: string, args: RequestArgs) {
'--max-time',
String(REQUEST_TIMEOUT_SECONDS),
'--write-out',
'\n%{http_code}',
CURL_WRITE_OUT_FORMAT,
'-X',
args.method,
...headers,
@@ -206,14 +386,9 @@ async function fetchJsonViaCurl(url: string, args: RequestArgs) {
if (result.status !== 0) {
throw new Error(result.stderr || 'curl failed')
}
const output = result.stdout ?? ''
const splitAt = output.lastIndexOf('\n')
if (splitAt === -1) throw new Error('curl response missing status')
const body = output.slice(0, splitAt)
const status = Number(output.slice(splitAt + 1).trim())
if (!Number.isFinite(status)) throw new Error('curl response missing status')
const { body, status, headers: responseHeaders } = parseCurlBodyAndMeta(result.stdout ?? '')
if (status < 200 || status >= 300) {
throwHttpStatusError(status, body)
throwHttpStatusError(status, body, responseHeaders)
}
return JSON.parse(body || 'null') as unknown
}
@@ -246,7 +421,7 @@ async function fetchJsonFormViaCurl(url: string, args: FormRequestArgs) {
'--max-time',
String(REQUEST_TIMEOUT_SECONDS),
'--write-out',
'\n%{http_code}',
CURL_WRITE_OUT_FORMAT,
'-X',
args.method,
...headers,
@@ -258,14 +433,9 @@ async function fetchJsonFormViaCurl(url: string, args: FormRequestArgs) {
if (result.status !== 0) {
throw new Error(result.stderr || 'curl failed')
}
const output = result.stdout ?? ''
const splitAt = output.lastIndexOf('\n')
if (splitAt === -1) throw new Error('curl response missing status')
const body = output.slice(0, splitAt)
const status = Number(output.slice(splitAt + 1).trim())
if (!Number.isFinite(status)) throw new Error('curl response missing status')
const { body, status, headers: responseHeaders } = parseCurlBodyAndMeta(result.stdout ?? '')
if (status < 200 || status >= 300) {
throwHttpStatusError(status, body)
throwHttpStatusError(status, body, responseHeaders)
}
return JSON.parse(body || 'null') as unknown
} finally {
@@ -285,7 +455,7 @@ async function fetchTextViaCurl(url: string, args: { token?: string }) {
'--max-time',
String(REQUEST_TIMEOUT_SECONDS),
'--write-out',
'\n%{http_code}',
CURL_WRITE_OUT_FORMAT,
'-X',
'GET',
...headers,
@@ -295,17 +465,9 @@ async function fetchTextViaCurl(url: string, args: { token?: string }) {
if (result.status !== 0) {
throw new Error(result.stderr || 'curl failed')
}
const output = result.stdout ?? ''
const splitAt = output.lastIndexOf('\n')
if (splitAt === -1) throw new Error('curl response missing status')
const body = output.slice(0, splitAt)
const status = Number(output.slice(splitAt + 1).trim())
if (!Number.isFinite(status)) throw new Error('curl response missing status')
const { body, status, headers: responseHeaders } = parseCurlBodyAndMeta(result.stdout ?? '')
if (status < 200 || status >= 300) {
if (status === 429 || status >= 500) {
throw new Error(body || `HTTP ${status}`)
}
throw new AbortError(body || `HTTP ${status}`)
throwHttpStatusError(status, body, responseHeaders)
}
return body
}
@@ -329,18 +491,17 @@ async function fetchBinaryViaCurl(url: string, token?: string) {
'-o',
filePath,
'--write-out',
'%{http_code}',
CURL_WRITE_OUT_FORMAT,
url,
]
const result = spawnSync('curl', curlArgs, { encoding: 'utf8' })
if (result.status !== 0) {
throw new Error(result.stderr || 'curl failed')
}
const status = Number((result.stdout ?? '').trim())
if (!Number.isFinite(status)) throw new Error('curl response missing status')
const { status, headers: responseHeaders } = parseCurlBodyAndMeta(result.stdout ?? '')
if (status < 200 || status >= 300) {
const body = await readFileSafe(filePath)
throwHttpStatusError(status, body ? new TextDecoder().decode(body) : '')
throwHttpStatusError(status, body ? new TextDecoder().decode(body) : '', responseHeaders)
}
const bytes = await readFileSafe(filePath)
return bytes ? new Uint8Array(bytes) : new Uint8Array()
@@ -349,6 +510,62 @@ async function fetchBinaryViaCurl(url: string, token?: string) {
}
}
function parseCurlBodyAndMeta(output: string): {
body: string
status: number
headers: Record<string, string>
} {
const marker = `\n${CURL_META_MARKER}\n`
const markerIndex = output.lastIndexOf(marker)
if (markerIndex === -1) {
// Backward compatibility for older tests that only provide "<body>\n<status>".
const splitAt = output.lastIndexOf('\n')
if (splitAt === -1) {
const statusOnly = Number(output.trim())
if (!Number.isFinite(statusOnly)) throw new Error('curl response missing status')
return { body: '', status: statusOnly, headers: {} }
}
const body = output.slice(0, splitAt)
const status = Number(output.slice(splitAt + 1).trim())
if (!Number.isFinite(status)) throw new Error('curl response missing status')
return { body, status, headers: {} }
}
const body = output.slice(0, markerIndex)
const meta = output.slice(markerIndex + marker.length).replace(/\r/g, '')
const lines = meta.split('\n')
const status = Number((lines[0] ?? '').trim())
if (!Number.isFinite(status)) throw new Error('curl response missing status')
const [
xRateLimitLimit,
xRateLimitRemaining,
xRateLimitReset,
rateLimitLimit,
rateLimitRemaining,
rateLimitReset,
retryAfter,
] = lines.slice(1)
const headers: Record<string, string> = {}
setHeaderIfPresent(headers, 'x-ratelimit-limit', xRateLimitLimit)
setHeaderIfPresent(headers, 'x-ratelimit-remaining', xRateLimitRemaining)
setHeaderIfPresent(headers, 'x-ratelimit-reset', xRateLimitReset)
setHeaderIfPresent(headers, 'ratelimit-limit', rateLimitLimit)
setHeaderIfPresent(headers, 'ratelimit-remaining', rateLimitRemaining)
setHeaderIfPresent(headers, 'ratelimit-reset', rateLimitReset)
setHeaderIfPresent(headers, 'retry-after', retryAfter)
return { body, status, headers }
}
function setHeaderIfPresent(headers: Record<string, string>, key: string, value: string | undefined) {
if (typeof value !== 'string') return
const trimmed = value.trim()
if (!trimmed) return
headers[key] = trimmed
}
async function readFileSafe(path: string) {
try {
const { readFile } = await import('node:fs/promises')
+18
View File
@@ -0,0 +1,18 @@
import { vi } from 'vitest'
export const convexReactMocks = {
useAction: vi.fn(),
useQuery: vi.fn(),
usePaginatedQuery: vi.fn(),
}
export function resetConvexReactMocks() {
convexReactMocks.useAction.mockReset()
convexReactMocks.useQuery.mockReset()
convexReactMocks.usePaginatedQuery.mockReset()
}
export function setupDefaultConvexReactMocks() {
convexReactMocks.useAction.mockReturnValue(() => Promise.resolve([]))
convexReactMocks.useQuery.mockReturnValue(null)
}
+11 -8
View File
@@ -2,12 +2,15 @@
import { act, render } from '@testing-library/react'
import type { ReactNode } from 'react'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import {
convexReactMocks,
resetConvexReactMocks,
setupDefaultConvexReactMocks,
} from './helpers/convexReactMocks'
import { SkillsIndex } from '../routes/skills/index'
const navigateMock = vi.fn()
const useActionMock = vi.fn()
const usePaginatedQueryMock = vi.fn()
let searchMock: Record<string, unknown> = {}
vi.mock('@tanstack/react-router', () => ({
@@ -20,17 +23,17 @@ vi.mock('@tanstack/react-router', () => ({
}))
vi.mock('convex/react', () => ({
useAction: (...args: unknown[]) => useActionMock(...args),
usePaginatedQuery: (...args: unknown[]) => usePaginatedQueryMock(...args),
useAction: (...args: unknown[]) => convexReactMocks.useAction(...args),
useQuery: (...args: unknown[]) => convexReactMocks.useQuery(...args),
usePaginatedQuery: (...args: unknown[]) => convexReactMocks.usePaginatedQuery(...args),
}))
describe('SkillsIndex load-more observer', () => {
beforeEach(() => {
usePaginatedQueryMock.mockReset()
useActionMock.mockReset()
resetConvexReactMocks()
navigateMock.mockReset()
searchMock = {}
useActionMock.mockReturnValue(() => Promise.resolve([]))
setupDefaultConvexReactMocks()
})
afterEach(() => {
@@ -39,7 +42,7 @@ describe('SkillsIndex load-more observer', () => {
it('triggers one request for repeated intersection callbacks', async () => {
const loadMorePaginated = vi.fn()
usePaginatedQueryMock.mockReturnValue({
convexReactMocks.usePaginatedQuery.mockReturnValue({
results: [makeListResult('skill-0', 'Skill 0')],
status: 'CanLoadMore',
loadMore: loadMorePaginated,
+25 -22
View File
@@ -2,12 +2,15 @@
import { act, fireEvent, render, screen } from '@testing-library/react'
import type { ReactNode } from 'react'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import {
convexReactMocks,
resetConvexReactMocks,
setupDefaultConvexReactMocks,
} from './helpers/convexReactMocks'
import { SkillsIndex } from '../routes/skills/index'
const navigateMock = vi.fn()
const useActionMock = vi.fn()
const usePaginatedQueryMock = vi.fn()
let searchMock: Record<string, unknown> = {}
vi.mock('@tanstack/react-router', () => ({
@@ -20,19 +23,19 @@ vi.mock('@tanstack/react-router', () => ({
}))
vi.mock('convex/react', () => ({
useAction: (...args: unknown[]) => useActionMock(...args),
usePaginatedQuery: (...args: unknown[]) => usePaginatedQueryMock(...args),
useAction: (...args: unknown[]) => convexReactMocks.useAction(...args),
useQuery: (...args: unknown[]) => convexReactMocks.useQuery(...args),
usePaginatedQuery: (...args: unknown[]) => convexReactMocks.usePaginatedQuery(...args),
}))
describe('SkillsIndex', () => {
beforeEach(() => {
usePaginatedQueryMock.mockReset()
useActionMock.mockReset()
resetConvexReactMocks()
navigateMock.mockReset()
searchMock = {}
useActionMock.mockReturnValue(() => Promise.resolve([]))
setupDefaultConvexReactMocks()
// Default: return empty results with Exhausted status
usePaginatedQueryMock.mockReturnValue({
convexReactMocks.usePaginatedQuery.mockReturnValue({
results: [],
status: 'Exhausted',
loadMore: vi.fn(),
@@ -47,7 +50,7 @@ describe('SkillsIndex', () => {
it('requests the first skills page', () => {
render(<SkillsIndex />)
// usePaginatedQuery should be called with the API endpoint and sort/dir args
expect(usePaginatedQueryMock).toHaveBeenCalledWith(
expect(convexReactMocks.usePaginatedQuery).toHaveBeenCalledWith(
expect.anything(),
{ sort: 'downloads', dir: 'desc', highlightedOnly: false, nonSuspiciousOnly: false },
{ initialNumItems: 25 },
@@ -61,7 +64,7 @@ describe('SkillsIndex', () => {
it('shows loading state instead of empty state when pagination is not exhausted', () => {
// When status is not 'Exhausted', we should show loading, not "No skills match"
usePaginatedQueryMock.mockReturnValue({
convexReactMocks.usePaginatedQuery.mockReturnValue({
results: [],
status: 'CanLoadMore',
loadMore: vi.fn(),
@@ -72,7 +75,7 @@ describe('SkillsIndex', () => {
})
it('keeps load-more reachable when results are empty but pagination can continue', () => {
usePaginatedQueryMock.mockReturnValue({
convexReactMocks.usePaginatedQuery.mockReturnValue({
results: [],
status: 'CanLoadMore',
loadMore: vi.fn(),
@@ -94,7 +97,7 @@ describe('SkillsIndex', () => {
owner: null,
ownerHandle: null,
}
usePaginatedQueryMock.mockReturnValue({
convexReactMocks.usePaginatedQuery.mockReturnValue({
results: [mockEntry],
status: 'LoadingMore',
loadMore: vi.fn(),
@@ -106,7 +109,7 @@ describe('SkillsIndex', () => {
it('handles LoadingMore with empty results gracefully', () => {
// Edge case: user changes filter while loading more, results become empty
usePaginatedQueryMock.mockReturnValue({
convexReactMocks.usePaginatedQuery.mockReturnValue({
results: [],
status: 'LoadingMore',
loadMore: vi.fn(),
@@ -124,9 +127,9 @@ describe('SkillsIndex', () => {
// This tests the hasQuery condition in the empty state logic
searchMock = { q: 'nonexistent-skill-xyz' }
const actionFn = vi.fn().mockResolvedValue([])
useActionMock.mockReturnValue(actionFn)
convexReactMocks.useAction.mockReturnValue(actionFn)
// Pagination is skipped in search mode, so status stays 'LoadingFirstPage'
usePaginatedQueryMock.mockReturnValue({
convexReactMocks.usePaginatedQuery.mockReturnValue({
results: [],
status: 'LoadingFirstPage',
loadMore: vi.fn(),
@@ -146,13 +149,13 @@ describe('SkillsIndex', () => {
it('skips list query and calls search when query is set', async () => {
searchMock = { q: 'remind' }
const actionFn = vi.fn().mockResolvedValue([])
useActionMock.mockReturnValue(actionFn)
convexReactMocks.useAction.mockReturnValue(actionFn)
vi.useFakeTimers()
render(<SkillsIndex />)
// usePaginatedQuery should be called with 'skip' when there's a search query
expect(usePaginatedQueryMock).toHaveBeenCalledWith(expect.anything(), 'skip', {
expect(convexReactMocks.usePaginatedQuery).toHaveBeenCalledWith(expect.anything(), 'skip', {
initialNumItems: 25,
})
await act(async () => {
@@ -182,7 +185,7 @@ describe('SkillsIndex', () => {
.fn()
.mockResolvedValueOnce(makeSearchResults(25))
.mockResolvedValueOnce(makeSearchResults(50))
useActionMock.mockReturnValue(actionFn)
convexReactMocks.useAction.mockReturnValue(actionFn)
vi.useFakeTimers()
render(<SkillsIndex />)
@@ -213,7 +216,7 @@ describe('SkillsIndex', () => {
makeSearchEntry({ slug: 'skill-b', displayName: 'Skill B', stars: 5, updatedAt: 200 }),
makeSearchEntry({ slug: 'skill-c', displayName: 'Skill C', stars: 4, updatedAt: 999 }),
])
useActionMock.mockReturnValue(actionFn)
convexReactMocks.useAction.mockReturnValue(actionFn)
vi.useFakeTimers()
render(<SkillsIndex />)
@@ -238,7 +241,7 @@ describe('SkillsIndex', () => {
makeSearchResult('newer-low-score', 'Newer Low Score', 0.1, 2000),
makeSearchResult('older-high-score', 'Older High Score', 0.9, 1000),
])
useActionMock.mockReturnValue(actionFn)
convexReactMocks.useAction.mockReturnValue(actionFn)
vi.useFakeTimers()
render(<SkillsIndex />)
@@ -258,7 +261,7 @@ describe('SkillsIndex', () => {
searchMock = { nonSuspicious: true }
render(<SkillsIndex />)
expect(usePaginatedQueryMock).toHaveBeenCalledWith(
expect(convexReactMocks.usePaginatedQuery).toHaveBeenCalledWith(
expect.anything(),
{ sort: 'downloads', dir: 'desc', highlightedOnly: false, nonSuspiciousOnly: true },
{ initialNumItems: 25 },
@@ -269,7 +272,7 @@ describe('SkillsIndex', () => {
searchMock = { highlighted: true }
render(<SkillsIndex />)
expect(usePaginatedQueryMock).toHaveBeenCalledWith(
expect(convexReactMocks.usePaginatedQuery).toHaveBeenCalledWith(
expect.anything(),
{ sort: 'downloads', dir: 'desc', highlightedOnly: true, nonSuspiciousOnly: false },
{ initialNumItems: 25 },
+4 -3
View File
@@ -1,5 +1,6 @@
import { Link } from '@tanstack/react-router'
import type { ClawdisSkillMetadata } from 'clawhub-schema'
import { Package } from 'lucide-react'
import type { Doc, Id } from '../../convex/_generated/dataModel'
import { getSkillBadges } from '../lib/badges'
import { formatCompactStat, formatSkillStatsTriplet } from '../lib/numberFormat'
@@ -188,9 +189,9 @@ export function SkillHeader({
</div>
) : null}
<div className="stat">
{formattedStats.stars} · {formattedStats.downloads} · {' '}
{formatCompactStat(skill.stats.installsCurrent ?? 0)} current ·{' '}
{formattedStats.installsAllTime} all-time
{formattedStats.stars} · <Package size={14} aria-hidden="true" />{' '}
{formattedStats.downloads} · {formatCompactStat(skill.stats.installsCurrent ?? 0)} current
installs · {formattedStats.installsAllTime} all-time installs
</div>
<div className="stat">
<UserBadge user={owner} fallbackHandle={ownerHandle} prefix="by" size="md" showName />
+5 -3
View File
@@ -1,3 +1,4 @@
import { Package } from 'lucide-react'
import { formatSkillStatsTriplet, type SkillStatsTriplet } from '../lib/numberFormat'
type SkillMetricsStats = SkillStatsTriplet & {
@@ -8,7 +9,7 @@ export function SkillStatsTripletLine({ stats }: { stats: SkillStatsTriplet }) {
const formatted = formatSkillStatsTriplet(stats)
return (
<>
{formatted.stars} · {formatted.downloads} · {formatted.installsAllTime}
{formatted.stars} · <Package size={13} aria-hidden="true" /> {formatted.downloads}
</>
)
}
@@ -17,8 +18,9 @@ export function SkillMetricsRow({ stats }: { stats: SkillMetricsStats }) {
const formatted = formatSkillStatsTriplet(stats)
return (
<>
<span> {formatted.downloads}</span>
<span> {formatted.installsAllTime}</span>
<span>
<Package size={13} aria-hidden="true" /> {formatted.downloads}
</span>
<span> {formatted.stars}</span>
<span>{stats.versions} v</span>
</>
+6 -2
View File
@@ -1,3 +1,4 @@
import { Package } from 'lucide-react'
import { formatSoulStatsTriplet, type SoulStatsTriplet } from '../lib/numberFormat'
export function SoulStatsTripletLine({
@@ -10,7 +11,8 @@ export function SoulStatsTripletLine({
const formatted = formatSoulStatsTriplet(stats)
return (
<>
{formatted.stars} · {formatted.downloads} · {formatted.versions} {versionSuffix}
{formatted.stars} · <Package size={13} aria-hidden="true" /> {formatted.downloads} ·{' '}
{formatted.versions} {versionSuffix}
</>
)
}
@@ -19,7 +21,9 @@ export function SoulMetricsRow({ stats }: { stats: SoulStatsTriplet }) {
const formatted = formatSoulStatsTriplet(stats)
return (
<>
<span> {formatted.downloads}</span>
<span>
<Package size={13} aria-hidden="true" /> {formatted.downloads}
</span>
<span> {formatted.stars}</span>
<span>{formatted.versions} v</span>
</>
+3 -1
View File
@@ -85,7 +85,9 @@ function SkillCard({ skill, ownerHandle }: { skill: DashboardSkill; ownerHandle:
</div>
{skill.summary && <p className="dashboard-skill-description">{skill.summary}</p>}
<div className="dashboard-skill-stats">
<span> {formatCompactStat(skill.stats.downloads)}</span>
<span>
<Package size={13} aria-hidden="true" /> {formatCompactStat(skill.stats.downloads)}
</span>
<span> {formatCompactStat(skill.stats.stars)}</span>
<span>{skill.stats.versions} v</span>
</div>
-7
View File
@@ -47,7 +47,6 @@ export function SkillsResults({
<div className="grid">
{sorted.map((entry) => {
const skill = entry.skill
const isPlugin = Boolean(entry.latestVersion?.parsed?.clawdis?.nix?.plugin)
const ownerHandle = entry.owner?.handle ?? entry.owner?.name ?? entry.ownerHandle ?? null
const skillHref = buildSkillHref(skill, ownerHandle)
return (
@@ -56,7 +55,6 @@ export function SkillsResults({
skill={skill}
href={skillHref}
badge={getSkillBadges(skill)}
chip={isPlugin ? 'Plugin bundle (nix)' : undefined}
summaryFallback="Agent-ready skill pack."
meta={
<div className="skill-card-footer-rows">
@@ -74,7 +72,6 @@ export function SkillsResults({
<div className="skills-list">
{sorted.map((entry) => {
const skill = entry.skill
const isPlugin = Boolean(entry.latestVersion?.parsed?.clawdis?.nix?.plugin)
const ownerHandle = entry.owner?.handle ?? entry.owner?.name ?? entry.ownerHandle ?? null
const skillHref = buildSkillHref(skill, ownerHandle)
return (
@@ -88,15 +85,11 @@ export function SkillsResults({
{badge}
</span>
))}
{isPlugin ? <span className="tag tag-accent tag-compact">Plugin bundle (nix)</span> : null}
</div>
<div className="skills-row-summary">{skill.summary ?? 'No summary provided.'}</div>
<div className="skills-row-owner">
<UserBadge user={entry.owner} fallbackHandle={ownerHandle} prefix="by" link={false} />
</div>
{isPlugin ? (
<div className="skills-row-meta">Bundle includes SKILL.md, CLI, and config.</div>
) : null}
</div>
<div className="skills-row-metrics">
<SkillMetricsRow stats={skill.stats} />
@@ -77,17 +77,6 @@ export function useSkillsBrowseModel({
setQuery(search.q ?? '')
}, [search.q])
useEffect(() => {
if (hasQuery || search.sort) return
void navigate({
search: (prev) => ({
...prev,
sort: 'downloads',
}),
replace: true,
})
}, [hasQuery, navigate, search.sort])
useEffect(() => {
if (search.focus === 'search' && searchInputRef.current) {
searchInputRef.current.focus()
+6
View File
@@ -1,5 +1,7 @@
import { createFileRoute, redirect } from '@tanstack/react-router'
import { useQuery } from 'convex/react'
import { useRef } from 'react'
import { api } from '../../../convex/_generated/api'
import { parseSort } from './-params'
import { SkillsResults } from './-SkillsResults'
import { SkillsToolbar } from './-SkillsToolbar'
@@ -49,6 +51,9 @@ export function SkillsIndex() {
const navigate = Route.useNavigate()
const search = Route.useSearch()
const searchInputRef = useRef<HTMLInputElement>(null)
const totalSkills = useQuery(api.skills.countPublicSkills)
const totalSkillsText =
typeof totalSkills === 'number' ? totalSkills.toLocaleString('en-US') : null
const model = useSkillsBrowseModel({
navigate,
@@ -61,6 +66,7 @@ export function SkillsIndex() {
<header className="skills-header-top">
<h1 className="section-title" style={{ marginBottom: 8 }}>
Skills
{totalSkillsText && <span style={{ opacity: 0.55 }}>{` (${totalSkillsText})`}</span>}
</h1>
<p className="section-subtitle" style={{ marginBottom: 0 }}>
{model.isLoadingSkills