From 0b6017548c300cffbf1f400a25185f4ae0970598 Mon Sep 17 00:00:00 2001 From: Patrick Erichsen Date: Wed, 5 Aug 2026 16:30:58 -0700 Subject: [PATCH] feat: add local UI proof fallback (#3429) * feat: add local UI proof fallback * style: format generated Convex skills --- .agents/skills/clawhub-pr-maintainer/SKILL.md | 22 +++- .agents/skills/convex-add/SKILL.md | 8 +- .agents/skills/convex-billing/SKILL.md | 44 ++++--- .../skills/convex-launch-readiness/SKILL.md | 2 +- scripts/ui-proof.mjs | 115 ++++++++++++++++-- scripts/ui-proof.test.mjs | 102 ++++++++++++++++ specs/ui-proof.md | 24 ++-- 7 files changed, 281 insertions(+), 36 deletions(-) diff --git a/.agents/skills/clawhub-pr-maintainer/SKILL.md b/.agents/skills/clawhub-pr-maintainer/SKILL.md index 09f1d0b0..f2b7121e 100644 --- a/.agents/skills/clawhub-pr-maintainer/SKILL.md +++ b/.agents/skills/clawhub-pr-maintainer/SKILL.md @@ -120,8 +120,11 @@ Review output must include: ## Decide UI Proof Mode -Use the `clawhub-ui-proof` skill when the maintainer/agent should generate new -visual evidence. +Generate new visual evidence with the best proof runtime available in the +current session. Use Crabbox through `bun run proof:ui` only when a Crabbox +skill or working Crabbox capability is available. Otherwise ignore Crabbox and +run the existing Playwright proof runtime against a real local ClawHub instance; +missing Crabbox access is not a blocker. - `before-after`: bug fixes, regressions, changed copy, changed layout, or any PR where main-vs-candidate comparison clarifies the change. @@ -134,6 +137,21 @@ Write a temporary Playwright scenario under `.artifacts/proof-scenarios/`; do not infer manual clicks. Keep screenshots and videos in `.artifacts/` until publishing. Never commit proof artifacts. +For the local fallback, start ClawHub with the relevant local Convex state and +run the scenario through the local Playwright runner: + +```sh +bun run proof:ui -- --runner local --mode feature \ + --scenario .artifacts/proof-scenarios/.pw.ts \ + --candidate-url +``` + +For before/after proof, run the same scenario against an `origin/main` checkout +and the candidate checkout, then pass both URLs with `--baseline-url` and +`--candidate-url`. The runner accepts only localhost or loopback URLs and writes +publishable `baseline/` and `candidate/` artifacts. Use the Codex app browser to +inspect the running local instances and captured evidence. + ## Final Review Comment With Proof If this review generated `proof:ui` artifacts, publish them before the final PR diff --git a/.agents/skills/convex-add/SKILL.md b/.agents/skills/convex-add/SKILL.md index 035c79bd..dd2a1688 100644 --- a/.agents/skills/convex-add/SKILL.md +++ b/.agents/skills/convex-add/SKILL.md @@ -13,10 +13,10 @@ Add a named capability to an existing Convex app. Step 1: fetch the served capab 1. Identify the capability the user wants (text after /add or $add). 2. Fetch https://basic-anteater-667.convex.site/capabilities.json?src=agent-skills (4s timeout). Match the request against title/summary/trigger. -3a. If a match is found and tier>0: confirm with user before proceeding. Then fetch /capability/.md and follow its Procedure+Rules sections. -3b. If a match is found and tier=0: fetch /capability/.md and follow its Procedure+Rules sections directly. -4. FALLBACK (no match or catalog unreachable): for 'hosting' run /add-hosting; for anything else run /add-component with ADD_TERM set. Read CANDIDATES output, install best match, wire per README. -5. Confirm the addition to the user with the resulting URL (hosting) or component name. + 3a. If a match is found and tier>0: confirm with user before proceeding. Then fetch /capability/.md and follow its Procedure+Rules sections. + 3b. If a match is found and tier=0: fetch /capability/.md and follow its Procedure+Rules sections directly. +3. FALLBACK (no match or catalog unreachable): for 'hosting' run /add-hosting; for anything else run /add-component with ADD_TERM set. Read CANDIDATES output, install best match, wire per README. +4. Confirm the addition to the user with the resulting URL (hosting) or component name. ## Rules diff --git a/.agents/skills/convex-billing/SKILL.md b/.agents/skills/convex-billing/SKILL.md index 7ffac8b6..4f2d48d2 100644 --- a/.agents/skills/convex-billing/SKILL.md +++ b/.agents/skills/convex-billing/SKILL.md @@ -14,8 +14,8 @@ Wire Stripe to Convex using @convex-dev/stripe: a checkout action, an httpAction 1. Install the component: `npm install @convex-dev/stripe`. 2. Create `convex/convex.config.ts`: ```ts - import { defineApp } from 'convex/server'; - import stripe from '@convex-dev/stripe/convex.config.js'; + import { defineApp } from "convex/server"; + import stripe from "@convex-dev/stripe/convex.config.js"; const app = defineApp(); app.use(stripe); export default app; @@ -23,28 +23,39 @@ Wire Stripe to Convex using @convex-dev/stripe: a checkout action, an httpAction 3. Store Stripe keys in Convex env (use the `env` micro power): `STRIPE_SECRET_KEY` (sk_test_… / sk_live_…) and `STRIPE_WEBHOOK_SECRET` (whsec_…). 4. Create `convex/http.ts` to register the webhook route (the component handles signature verification automatically): ```ts - import { httpRouter } from 'convex/server'; - import { components } from './_generated/api'; - import { registerRoutes } from '@convex-dev/stripe'; + import { httpRouter } from "convex/server"; + import { components } from "./_generated/api"; + import { registerRoutes } from "@convex-dev/stripe"; const http = httpRouter(); - registerRoutes(http, components.stripe, { webhookPath: '/stripe/webhook' }); + registerRoutes(http, components.stripe, { webhookPath: "/stripe/webhook" }); export default http; ``` 5. Create `convex/billing.ts` with a checkout action and a subscription-gate query: ```ts - import { action, query } from './_generated/server'; - import { components } from './_generated/api'; - import { StripeSubscriptions } from '@convex-dev/stripe'; - import { v } from 'convex/values'; + import { action, query } from "./_generated/server"; + import { components } from "./_generated/api"; + import { StripeSubscriptions } from "@convex-dev/stripe"; + import { v } from "convex/values"; const stripeClient = new StripeSubscriptions(components.stripe, {}); export const createSubscriptionCheckout = action({ args: { priceId: v.string() }, returns: v.object({ sessionId: v.string(), url: v.union(v.string(), v.null()) }), handler: async (ctx, args) => { const identity = await ctx.auth.getUserIdentity(); - if (!identity) throw new Error('Not authenticated'); - const customer = await stripeClient.getOrCreateCustomer(ctx, { userId: identity.subject, email: identity.email, name: identity.name }); - return await stripeClient.createCheckoutSession(ctx, { priceId: args.priceId, customerId: customer.customerId, mode: 'subscription', successUrl: `${process.env.SITE_URL ?? 'http://localhost:3000'}/?success=true`, cancelUrl: `${process.env.SITE_URL ?? 'http://localhost:3000'}/?canceled=true`, subscriptionMetadata: { userId: identity.subject } }); + if (!identity) throw new Error("Not authenticated"); + const customer = await stripeClient.getOrCreateCustomer(ctx, { + userId: identity.subject, + email: identity.email, + name: identity.name, + }); + return await stripeClient.createCheckoutSession(ctx, { + priceId: args.priceId, + customerId: customer.customerId, + mode: "subscription", + successUrl: `${process.env.SITE_URL ?? "http://localhost:3000"}/?success=true`, + cancelUrl: `${process.env.SITE_URL ?? "http://localhost:3000"}/?canceled=true`, + subscriptionMetadata: { userId: identity.subject }, + }); }, }); export const isSubscribed = query({ @@ -53,8 +64,11 @@ Wire Stripe to Convex using @convex-dev/stripe: a checkout action, an httpAction handler: async (ctx) => { const identity = await ctx.auth.getUserIdentity(); if (!identity) return false; - const subscriptions = await ctx.runQuery(components.stripe.public.listSubscriptionsByUserId, { userId: identity.subject }); - return subscriptions.some((sub) => sub.status === 'active' || sub.status === 'trialing'); + const subscriptions = await ctx.runQuery( + components.stripe.public.listSubscriptionsByUserId, + { userId: identity.subject }, + ); + return subscriptions.some((sub) => sub.status === "active" || sub.status === "trialing"); }, }); ``` diff --git a/.agents/skills/convex-launch-readiness/SKILL.md b/.agents/skills/convex-launch-readiness/SKILL.md index 8a19d874..9debf51e 100644 --- a/.agents/skills/convex-launch-readiness/SKILL.md +++ b/.agents/skills/convex-launch-readiness/SKILL.md @@ -17,7 +17,7 @@ Readiness is not one check — it's the union of the checks, deduped, ranked, an - convex-reviewer — validators, indexes-not-filter, idiom, error handling. Always runnable on code. - convex-advisor — live read-limit / OCC evidence (only if a deployment with traffic exists; else record 'skipped: no traffic'). - convex-insights — recent failures from logs (only if a deployment exists). - Run independent passes concurrently; each returns findings, not fixes. + Run independent passes concurrently; each returns findings, not fixes. 3. NORMALIZE + DEDUPE: collect all findings into one report. Set each finding's `identity` field to a normalized function/table key (e.g. `messages:list`) that is the SAME whether the pass reported a code-locus or a deployment-locus for that function — so the SAME defect seen from two loci (reviewer flags a missing index at code-locus, advisor flags its read-limit symptom at deployment-locus) collapses to ONE via the bus's (class, identity) dedup and isn't double-counted in the score. Keep the higher-confidence source. Drop nothing silently; a pass that errored/was skipped is a stated coverage gap, not a clean result. 4. SCORE, auditable: start at 100; subtract per CONFIRMED finding by severity (high −15, med −5, low −1), floor at 0; print the exact formula and the per-class breakdown so the number is reproducible, not a vibe. plausible-only findings are listed as candidates but do NOT move the score (evidence-not-vibes). A deployment/traffic-less run reports a code-only score and says so. 5. REPORT: the score, then findings ranked by severity, each with its evidence, its locus, and the fixCapability + a one-line fix note. Group by 'blockers' (high) / 'should-fix' (med) / 'nice-to-have' (low). End with the ordered fix plan: which capability to run next, in what order (authz/data-loss first, then perf/scale, then idiom/observability). diff --git a/scripts/ui-proof.mjs b/scripts/ui-proof.mjs index 31f840c3..2131a52a 100644 --- a/scripts/ui-proof.mjs +++ b/scripts/ui-proof.mjs @@ -9,6 +9,7 @@ const DEFAULT_BASELINE = "origin/main"; const DEFAULT_CANDIDATE = "worktree"; const DEFAULT_MODE = "before-after"; const DEFAULT_PROVIDER = "hetzner"; +const DEFAULT_RUNNER = "crabbox"; const DEFAULT_CLASS = "standard"; const DEFAULT_IDLE_TIMEOUT = "60m"; const DEFAULT_TTL = "120m"; @@ -37,6 +38,7 @@ export function parseProofUiArgs(argv = []) { machineClass: DEFAULT_CLASS, mode: DEFAULT_MODE, provider: DEFAULT_PROVIDER, + runner: DEFAULT_RUNNER, scenario: "", skipInstall: false, ttl: DEFAULT_TTL, @@ -51,9 +53,15 @@ export function parseProofUiArgs(argv = []) { if (arg === "--baseline") { opts.baseline = requireValue(arg, next); index += 1; + } else if (arg === "--baseline-url") { + opts.baselineUrl = requireValue(arg, next); + index += 1; } else if (arg === "--candidate") { opts.candidate = requireValue(arg, next); index += 1; + } else if (arg === "--candidate-url") { + opts.candidateUrl = requireValue(arg, next); + index += 1; } else if (arg === "--class" || arg === "--machine-class") { opts.machineClass = requireValue(arg, next); index += 1; @@ -88,6 +96,9 @@ export function parseProofUiArgs(argv = []) { } else if (arg === "--provider") { opts.provider = requireValue(arg, next); index += 1; + } else if (arg === "--runner") { + opts.runner = requireValue(arg, next); + index += 1; } else if (arg === "--scenario") { opts.scenario = requireValue(arg, next); index += 1; @@ -112,6 +123,20 @@ export function parseProofUiArgs(argv = []) { if (!["before-after", "feature"].includes(opts.mode)) { throw new Error(`Unknown proof:ui mode: ${opts.mode}`); } + if (!["crabbox", "local"].includes(opts.runner)) { + throw new Error(`Unknown proof:ui runner: ${opts.runner}`); + } + if (opts.runner === "local") { + if (!opts.candidateUrl || (opts.mode === "before-after" && !opts.baselineUrl)) { + throw new Error( + opts.mode === "before-after" + ? "local before-after proof requires --baseline-url and --candidate-url" + : "local feature proof requires --candidate-url", + ); + } + if (opts.baselineUrl) assertLocalProofUrl(opts.baselineUrl, "--baseline-url"); + assertLocalProofUrl(opts.candidateUrl, "--candidate-url"); + } return opts; } @@ -134,6 +159,21 @@ function parseEnvAssignment(raw, flag) { return [key, raw.slice(separator + 1)]; } +function assertLocalProofUrl(raw, flag) { + let parsed; + try { + parsed = new URL(raw); + } catch { + throw new Error(`${flag} requires a valid URL`); + } + if ( + !["http:", "https:"].includes(parsed.protocol) || + !["127.0.0.1", "[::1]", "localhost"].includes(parsed.hostname) + ) { + throw new Error(`${flag} must use localhost or a loopback address`); + } +} + function timestamp(now) { return now().toISOString().replace(/[:.]/gu, "-"); } @@ -160,15 +200,19 @@ export function buildProofUiPlan({ now = () => new Date(), opts, repoRoot }) { outputDir: path.join(outputDir, "candidate"), ref: opts.candidate, }); + candidateLane.baseURL = opts.candidateUrl; const lanes = opts.mode === "feature" ? [candidateLane] : [ - buildLane({ - name: "baseline", - outputDir: path.join(outputDir, "baseline"), - ref: opts.baseline, - }), + { + ...buildLane({ + name: "baseline", + outputDir: path.join(outputDir, "baseline"), + ref: opts.baseline, + }), + baseURL: opts.baselineUrl, + }, candidateLane, ]; return { @@ -176,7 +220,8 @@ export function buildProofUiPlan({ now = () => new Date(), opts, repoRoot }) { candidate: opts.candidate, mode: opts.mode, outputDir, - provider: opts.provider, + provider: opts.runner === "local" ? "local" : opts.provider, + runner: opts.runner, scenario: path.resolve(repoRoot, opts.scenario), lanes, }; @@ -485,9 +530,10 @@ function renderReport(summary) { ? "Baseline: not run for feature proof." : `Baseline: \`${summary.baseline}\``, `Candidate: \`${summary.candidate}\``, + `Runner: \`${summary.runner}\``, `Provider: \`${summary.provider}\``, "", - summary.status === "dry-run" ? "Dry run: Crabbox was not invoked." : undefined, + summary.status === "dry-run" ? "Dry run: no proof runtime was invoked." : undefined, "## Artifacts", "", ].filter(Boolean); @@ -686,6 +732,45 @@ async function stopLease({ commandRunner, invocation, leaseId, opts, repoRoot }) }); } +async function runLocalLane({ commandRunner, lane, plan, repoRoot }) { + let error; + await fs.mkdir(lane.outputDir, { recursive: true }); + try { + await commandRunner( + "bun", + [ + path.join(repoRoot, "scripts", "ui-proof-runtime.mjs"), + "run-scenario", + "--scenario", + plan.scenario, + "--base-url", + lane.baseURL, + "--lane", + lane.name, + "--output-dir", + lane.outputDir, + ], + { cwd: repoRoot, stdio: "inherit" }, + ); + } catch (caught) { + error = caught instanceof Error ? caught.message : String(caught); + } + const manifest = await readLaneManifest(lane.outputDir); + const status = manifest.status ?? "fail"; + const laneError = + status === "pass" + ? undefined + : (manifest.error ?? error ?? "Local Playwright proof did not write a result manifest."); + return { + error: laneError, + localOutputDir: lane.outputDir, + name: lane.name, + ref: lane.ref, + status, + steps: manifest.steps ?? [], + }; +} + export async function runProofUi({ args = process.argv.slice(2), commandRunner = defaultCommandRunner, @@ -708,6 +793,7 @@ export async function runProofUi({ mode: plan.mode, outputDir: plan.outputDir, provider: plan.provider, + runner: plan.runner, scenario: plan.scenario, status: opts.dryRun ? "dry-run" : "pending", }; @@ -720,6 +806,21 @@ export async function runProofUi({ }; } + if (opts.runner === "local") { + const lanes = []; + for (const lane of plan.lanes) { + lanes.push(await runLocalLane({ commandRunner, lane, plan, repoRoot })); + } + summary.lanes = lanes; + summary.status = lanes.every((lane) => lane.status === "pass") ? "pass" : "fail"; + await writeSummaryAndReport({ outputDir: plan.outputDir, summary }); + return { + outputDir: plan.outputDir, + status: summary.status, + summaryPath: path.join(plan.outputDir, "summary.json"), + }; + } + const invocation = crabboxInvocation({ opts, repoRoot }); const { created, leaseId } = await warmupLease({ commandRunner, invocation, opts, repoRoot }); summary.crabbox = { createdLease: created, leaseId }; diff --git a/scripts/ui-proof.test.mjs b/scripts/ui-proof.test.mjs index 990d8275..d53065af 100644 --- a/scripts/ui-proof.test.mjs +++ b/scripts/ui-proof.test.mjs @@ -21,11 +21,58 @@ describe("ui-proof", () => { devAuth: false, mode: "before-after", provider: "hetzner", + runner: "crabbox", scenario: ".artifacts/proof-scenarios/demo.pw.ts", }, ); }); + it("parses local proof URLs and rejects non-local targets", () => { + expect( + parseProofUiArgs([ + "--runner", + "local", + "--mode", + "before-after", + "--baseline-url", + "http://127.0.0.1:4317", + "--candidate-url", + "http://localhost:4318", + "--scenario", + ".artifacts/proof-scenarios/demo.pw.ts", + ]), + ).toMatchObject({ + baselineUrl: "http://127.0.0.1:4317", + candidateUrl: "http://localhost:4318", + runner: "local", + }); + + expect(() => + parseProofUiArgs([ + "--runner", + "local", + "--mode", + "feature", + "--candidate-url", + "https://clawhub.ai", + "--scenario", + ".artifacts/proof-scenarios/demo.pw.ts", + ]), + ).toThrow("--candidate-url must use localhost"); + expect(() => + parseProofUiArgs([ + "--runner", + "local", + "--mode", + "before-after", + "--candidate-url", + "http://127.0.0.1:4318", + "--scenario", + ".artifacts/proof-scenarios/demo.pw.ts", + ]), + ).toThrow("local before-after proof requires --baseline-url and --candidate-url"); + }); + it("parses explicit proof modes and rejects unknown modes", () => { expect( parseProofUiArgs([ @@ -177,6 +224,61 @@ describe("ui-proof", () => { ).resolves.toContain('"mode": "feature"'); }); + it("runs a publishable local Playwright proof without invoking Crabbox", async () => { + const repoRoot = await fs.mkdtemp(path.join(os.tmpdir(), "clawhub-proof-")); + const scenario = path.join(repoRoot, ".artifacts/proof-scenarios/demo.pw.ts"); + await fs.mkdir(path.dirname(scenario), { recursive: true }); + await fs.writeFile(scenario, "export default async function demo() {}\n"); + const commands = []; + + const result = await runProofUi({ + args: [ + "--runner", + "local", + "--mode", + "feature", + "--candidate-url", + "http://127.0.0.1:4318", + "--scenario", + scenario, + ], + commandRunner: async (command, commandArgs) => { + commands.push([command, commandArgs]); + const outputDir = commandArgs[commandArgs.indexOf("--output-dir") + 1]; + await fs.mkdir(outputDir, { recursive: true }); + await fs.writeFile( + path.join(outputDir, "proof-steps.json"), + `${JSON.stringify({ + lane: "candidate", + status: "pass", + steps: [ + { + name: "candidate /skills", + screenshot: "screenshots/skills.png", + status: "pass", + }, + ], + })}\n`, + ); + return { stdout: "", stderr: "" }; + }, + now: () => new Date("2026-05-12T12:34:56.000Z"), + repoRoot, + }); + + expect(commands).toHaveLength(1); + expect(commands[0][0]).toBe("bun"); + expect(commands[0][1]).toContain("run-scenario"); + expect(commands[0][1]).toContain("http://127.0.0.1:4318"); + expect(result.status).toBe("pass"); + const summary = JSON.parse(await fs.readFile(result.summaryPath, "utf8")); + expect(summary).toMatchObject({ runner: "local", status: "pass" }); + expect(summary.lanes).toHaveLength(1); + await expect(fs.readFile(path.join(result.outputDir, "report.md"), "utf8")).resolves.toContain( + "Runner: `local`", + ); + }); + it("treats a passing proof manifest as authoritative after a Crabbox transport error", async () => { const repoRoot = await fs.mkdtemp(path.join(os.tmpdir(), "clawhub-proof-")); const scenario = path.join(repoRoot, ".artifacts/proof-scenarios/demo.pw.ts"); diff --git a/specs/ui-proof.md b/specs/ui-proof.md index 34ef5d29..c70e3311 100644 --- a/specs/ui-proof.md +++ b/specs/ui-proof.md @@ -1,13 +1,15 @@ # UI Proof Runtime -`proof:ui` is always full-stack. Each lane starts local Convex from that lane's -checkout, on deterministic lane-specific ports, then builds and previews the -frontend against those local Convex URLs. +`proof:ui` always verifies a full-stack ClawHub instance. Crabbox lanes start +local Convex from that lane's checkout, on deterministic lane-specific ports, +then build and preview the frontend against those local Convex URLs. Local +Playwright lanes attach to maintainer-started ClawHub instances on localhost or +a loopback address. -The proof runner must not provide a shared-backend mode. UI proof is meant to -prove the control plane and data plane together: the Git checkout controls both -frontend and Convex source, and the lane-local Convex URL controls the runtime -backend/data used by the browser. +The proof runner must not provide a shared or production-backend mode. UI proof +is meant to prove the control plane and data plane together: the Git checkout +controls both frontend and Convex source, and the lane-local Convex URL controls +the runtime backend/data used by the browser. Use `--mode before-after` for baseline-vs-candidate proof and `--mode feature` for candidate-only proof. Use `--seed-command` when a scenario needs fixtures. @@ -15,3 +17,11 @@ for candidate-only proof. Use `--seed-command` when a scenario needs fixtures. Dev auth must be explicit. The proof runner must not set `VITE_ENABLE_DEV_AUTH=1` by default; scenarios that need it should pass `--dev-auth` or explicit `--env` values. + +Crabbox is an optional execution environment, not a prerequisite for visual +verification. Agents should use `proof:ui` when a Crabbox skill or working +Crabbox capability is available. Otherwise they should ignore Crabbox and run +the same temporary scenario with `proof:ui --runner local` against real local +ClawHub instances. Local before/after evidence requires separate baseline and +candidate URLs and keeps the results in separate `baseline/` and `candidate/` +directories so `proof:publish` can publish either execution path.