feat: add local UI proof fallback (#3429)

* feat: add local UI proof fallback * style: format generated Convex skills
2026-08-14 00:47:57 +00:00 · 2026-08-05 16:30:58 -07:00
parent cd09e33877
commit 0b6017548c
7 changed files with 281 additions and 36 deletions
@@ -120,8 +120,11 @@ Review output must include:

 ## Decide UI Proof Mode

-Use the `clawhub-ui-proof` skill when the maintainer/agent should generate new
-visual evidence.
+Generate new visual evidence with the best proof runtime available in the
+current session. Use Crabbox through `bun run proof:ui` only when a Crabbox
+skill or working Crabbox capability is available. Otherwise ignore Crabbox and
+run the existing Playwright proof runtime against a real local ClawHub instance;
+missing Crabbox access is not a blocker.

 - `before-after`: bug fixes, regressions, changed copy, changed layout, or any
  PR where main-vs-candidate comparison clarifies the change.
@@ -134,6 +137,21 @@ Write a temporary Playwright scenario under `.artifacts/proof-scenarios/`; do
 not infer manual clicks. Keep screenshots and videos in `.artifacts/` until
 publishing. Never commit proof artifacts.

+For the local fallback, start ClawHub with the relevant local Convex state and
+run the scenario through the local Playwright runner:
+
+```sh
+bun run proof:ui -- --runner local --mode feature \
+  --scenario .artifacts/proof-scenarios/<name>.pw.ts \
+  --candidate-url <local-clawhub-url>
+```
+
+For before/after proof, run the same scenario against an `origin/main` checkout
+and the candidate checkout, then pass both URLs with `--baseline-url` and
+`--candidate-url`. The runner accepts only localhost or loopback URLs and writes
+publishable `baseline/` and `candidate/` artifacts. Use the Codex app browser to
+inspect the running local instances and captured evidence.
+
 ## Final Review Comment With Proof

 If this review generated `proof:ui` artifacts, publish them before the final PR
@@ -13,10 +13,10 @@ Add a named capability to an existing Convex app. Step 1: fetch the served capab

 1. Identify the capability the user wants (text after /add or $add).
 2. Fetch https://basic-anteater-667.convex.site/capabilities.json?src=agent-skills (4s timeout). Match the request against title/summary/trigger.
-3a. If a match is found and tier>0: confirm with user before proceeding. Then fetch /capability/<id>.md and follow its Procedure+Rules sections.
-3b. If a match is found and tier=0: fetch /capability/<id>.md and follow its Procedure+Rules sections directly.
-4. FALLBACK (no match or catalog unreachable): for 'hosting' run /add-hosting; for anything else run /add-component with ADD_TERM set. Read CANDIDATES output, install best match, wire per README.
-5. Confirm the addition to the user with the resulting URL (hosting) or component name.
+   3a. If a match is found and tier>0: confirm with user before proceeding. Then fetch /capability/<id>.md and follow its Procedure+Rules sections.
+   3b. If a match is found and tier=0: fetch /capability/<id>.md and follow its Procedure+Rules sections directly.
+3. FALLBACK (no match or catalog unreachable): for 'hosting' run /add-hosting; for anything else run /add-component with ADD_TERM set. Read CANDIDATES output, install best match, wire per README.
+4. Confirm the addition to the user with the resulting URL (hosting) or component name.

 ## Rules

@@ -14,8 +14,8 @@ Wire Stripe to Convex using @convex-dev/stripe: a checkout action, an httpAction
 1. Install the component: `npm install @convex-dev/stripe`.
 2. Create `convex/convex.config.ts`:
   ```ts
-   import { defineApp } from 'convex/server';
-   import stripe from '@convex-dev/stripe/convex.config.js';
+   import { defineApp } from "convex/server";
+   import stripe from "@convex-dev/stripe/convex.config.js";
   const app = defineApp();
   app.use(stripe);
   export default app;
@@ -23,28 +23,39 @@ Wire Stripe to Convex using @convex-dev/stripe: a checkout action, an httpAction
 3. Store Stripe keys in Convex env (use the `env` micro power): `STRIPE_SECRET_KEY` (sk_test_… / sk_live_…) and `STRIPE_WEBHOOK_SECRET` (whsec_…).
 4. Create `convex/http.ts` to register the webhook route (the component handles signature verification automatically):
   ```ts
-   import { httpRouter } from 'convex/server';
-   import { components } from './_generated/api';
-   import { registerRoutes } from '@convex-dev/stripe';
+   import { httpRouter } from "convex/server";
+   import { components } from "./_generated/api";
+   import { registerRoutes } from "@convex-dev/stripe";
   const http = httpRouter();
-   registerRoutes(http, components.stripe, { webhookPath: '/stripe/webhook' });
+   registerRoutes(http, components.stripe, { webhookPath: "/stripe/webhook" });
   export default http;
   ```
 5. Create `convex/billing.ts` with a checkout action and a subscription-gate query:
   ```ts
-   import { action, query } from './_generated/server';
-   import { components } from './_generated/api';
-   import { StripeSubscriptions } from '@convex-dev/stripe';
-   import { v } from 'convex/values';
+   import { action, query } from "./_generated/server";
+   import { components } from "./_generated/api";
+   import { StripeSubscriptions } from "@convex-dev/stripe";
+   import { v } from "convex/values";
   const stripeClient = new StripeSubscriptions(components.stripe, {});
   export const createSubscriptionCheckout = action({
     args: { priceId: v.string() },
     returns: v.object({ sessionId: v.string(), url: v.union(v.string(), v.null()) }),
     handler: async (ctx, args) => {
       const identity = await ctx.auth.getUserIdentity();
-       if (!identity) throw new Error('Not authenticated');
-       const customer = await stripeClient.getOrCreateCustomer(ctx, { userId: identity.subject, email: identity.email, name: identity.name });
-       return await stripeClient.createCheckoutSession(ctx, { priceId: args.priceId, customerId: customer.customerId, mode: 'subscription', successUrl: `${process.env.SITE_URL ?? 'http://localhost:3000'}/?success=true`, cancelUrl: `${process.env.SITE_URL ?? 'http://localhost:3000'}/?canceled=true`, subscriptionMetadata: { userId: identity.subject } });
+       if (!identity) throw new Error("Not authenticated");
+       const customer = await stripeClient.getOrCreateCustomer(ctx, {
+         userId: identity.subject,
+         email: identity.email,
+         name: identity.name,
+       });
+       return await stripeClient.createCheckoutSession(ctx, {
+         priceId: args.priceId,
+         customerId: customer.customerId,
+         mode: "subscription",
+         successUrl: `${process.env.SITE_URL ?? "http://localhost:3000"}/?success=true`,
+         cancelUrl: `${process.env.SITE_URL ?? "http://localhost:3000"}/?canceled=true`,
+         subscriptionMetadata: { userId: identity.subject },
+       });
     },
   });
   export const isSubscribed = query({
@@ -53,8 +64,11 @@ Wire Stripe to Convex using @convex-dev/stripe: a checkout action, an httpAction
     handler: async (ctx) => {
       const identity = await ctx.auth.getUserIdentity();
       if (!identity) return false;
-       const subscriptions = await ctx.runQuery(components.stripe.public.listSubscriptionsByUserId, { userId: identity.subject });
-       return subscriptions.some((sub) => sub.status === 'active' || sub.status === 'trialing');
+       const subscriptions = await ctx.runQuery(
+         components.stripe.public.listSubscriptionsByUserId,
+         { userId: identity.subject },
+       );
+       return subscriptions.some((sub) => sub.status === "active" || sub.status === "trialing");
     },
   });
   ```
@@ -17,7 +17,7 @@ Readiness is not one check — it's the union of the checks, deduped, ranked, an
   - convex-reviewer — validators, indexes-not-filter, idiom, error handling. Always runnable on code.
   - convex-advisor — live read-limit / OCC evidence (only if a deployment with traffic exists; else record 'skipped: no traffic').
   - convex-insights — recent failures from logs (only if a deployment exists).
-   Run independent passes concurrently; each returns findings, not fixes.
+     Run independent passes concurrently; each returns findings, not fixes.
 3. NORMALIZE + DEDUPE: collect all findings into one report. Set each finding's `identity` field to a normalized function/table key (e.g. `messages:list`) that is the SAME whether the pass reported a code-locus or a deployment-locus for that function — so the SAME defect seen from two loci (reviewer flags a missing index at code-locus, advisor flags its read-limit symptom at deployment-locus) collapses to ONE via the bus's (class, identity) dedup and isn't double-counted in the score. Keep the higher-confidence source. Drop nothing silently; a pass that errored/was skipped is a stated coverage gap, not a clean result.
 4. SCORE, auditable: start at 100; subtract per CONFIRMED finding by severity (high −15, med −5, low −1), floor at 0; print the exact formula and the per-class breakdown so the number is reproducible, not a vibe. plausible-only findings are listed as candidates but do NOT move the score (evidence-not-vibes). A deployment/traffic-less run reports a code-only score and says so.
 5. REPORT: the score, then findings ranked by severity, each with its evidence, its locus, and the fixCapability + a one-line fix note. Group by 'blockers' (high) / 'should-fix' (med) / 'nice-to-have' (low). End with the ordered fix plan: which capability to run next, in what order (authz/data-loss first, then perf/scale, then idiom/observability).
@@ -9,6 +9,7 @@ const DEFAULT_BASELINE = "origin/main";
 const DEFAULT_CANDIDATE = "worktree";
 const DEFAULT_MODE = "before-after";
 const DEFAULT_PROVIDER = "hetzner";
+const DEFAULT_RUNNER = "crabbox";
 const DEFAULT_CLASS = "standard";
 const DEFAULT_IDLE_TIMEOUT = "60m";
 const DEFAULT_TTL = "120m";
@@ -37,6 +38,7 @@ export function parseProofUiArgs(argv = []) {
    machineClass: DEFAULT_CLASS,
    mode: DEFAULT_MODE,
    provider: DEFAULT_PROVIDER,
+    runner: DEFAULT_RUNNER,
    scenario: "",
    skipInstall: false,
    ttl: DEFAULT_TTL,
@@ -51,9 +53,15 @@ export function parseProofUiArgs(argv = []) {
    if (arg === "--baseline") {
      opts.baseline = requireValue(arg, next);
      index += 1;
+    } else if (arg === "--baseline-url") {
+      opts.baselineUrl = requireValue(arg, next);
+      index += 1;
    } else if (arg === "--candidate") {
      opts.candidate = requireValue(arg, next);
      index += 1;
+    } else if (arg === "--candidate-url") {
+      opts.candidateUrl = requireValue(arg, next);
+      index += 1;
    } else if (arg === "--class" || arg === "--machine-class") {
      opts.machineClass = requireValue(arg, next);
      index += 1;
@@ -88,6 +96,9 @@ export function parseProofUiArgs(argv = []) {
    } else if (arg === "--provider") {
      opts.provider = requireValue(arg, next);
      index += 1;
+    } else if (arg === "--runner") {
+      opts.runner = requireValue(arg, next);
+      index += 1;
    } else if (arg === "--scenario") {
      opts.scenario = requireValue(arg, next);
      index += 1;
@@ -112,6 +123,20 @@ export function parseProofUiArgs(argv = []) {
  if (!["before-after", "feature"].includes(opts.mode)) {
    throw new Error(`Unknown proof:ui mode: ${opts.mode}`);
  }
+  if (!["crabbox", "local"].includes(opts.runner)) {
+    throw new Error(`Unknown proof:ui runner: ${opts.runner}`);
+  }
+  if (opts.runner === "local") {
+    if (!opts.candidateUrl || (opts.mode === "before-after" && !opts.baselineUrl)) {
+      throw new Error(
+        opts.mode === "before-after"
+          ? "local before-after proof requires --baseline-url and --candidate-url"
+          : "local feature proof requires --candidate-url",
+      );
+    }
+    if (opts.baselineUrl) assertLocalProofUrl(opts.baselineUrl, "--baseline-url");
+    assertLocalProofUrl(opts.candidateUrl, "--candidate-url");
+  }
  return opts;
 }

@@ -134,6 +159,21 @@ function parseEnvAssignment(raw, flag) {
  return [key, raw.slice(separator + 1)];
 }

+function assertLocalProofUrl(raw, flag) {
+  let parsed;
+  try {
+    parsed = new URL(raw);
+  } catch {
+    throw new Error(`${flag} requires a valid URL`);
+  }
+  if (
+    !["http:", "https:"].includes(parsed.protocol) ||
+    !["127.0.0.1", "[::1]", "localhost"].includes(parsed.hostname)
+  ) {
+    throw new Error(`${flag} must use localhost or a loopback address`);
+  }
+}
+
 function timestamp(now) {
  return now().toISOString().replace(/[:.]/gu, "-");
 }
@@ -160,15 +200,19 @@ export function buildProofUiPlan({ now = () => new Date(), opts, repoRoot }) {
    outputDir: path.join(outputDir, "candidate"),
    ref: opts.candidate,
  });
+  candidateLane.baseURL = opts.candidateUrl;
  const lanes =
    opts.mode === "feature"
      ? [candidateLane]
      : [
-          buildLane({
-            name: "baseline",
-            outputDir: path.join(outputDir, "baseline"),
-            ref: opts.baseline,
-          }),
+          {
+            ...buildLane({
+              name: "baseline",
+              outputDir: path.join(outputDir, "baseline"),
+              ref: opts.baseline,
+            }),
+            baseURL: opts.baselineUrl,
+          },
          candidateLane,
        ];
  return {
@@ -176,7 +220,8 @@ export function buildProofUiPlan({ now = () => new Date(), opts, repoRoot }) {
    candidate: opts.candidate,
    mode: opts.mode,
    outputDir,
-    provider: opts.provider,
+    provider: opts.runner === "local" ? "local" : opts.provider,
+    runner: opts.runner,
    scenario: path.resolve(repoRoot, opts.scenario),
    lanes,
  };
@@ -485,9 +530,10 @@ function renderReport(summary) {
      ? "Baseline: not run for feature proof."
      : `Baseline: \`${summary.baseline}\``,
    `Candidate: \`${summary.candidate}\``,
+    `Runner: \`${summary.runner}\``,
    `Provider: \`${summary.provider}\``,
    "",
-    summary.status === "dry-run" ? "Dry run: Crabbox was not invoked." : undefined,
+    summary.status === "dry-run" ? "Dry run: no proof runtime was invoked." : undefined,
    "## Artifacts",
    "",
  ].filter(Boolean);
@@ -686,6 +732,45 @@ async function stopLease({ commandRunner, invocation, leaseId, opts, repoRoot })
  });
 }

+async function runLocalLane({ commandRunner, lane, plan, repoRoot }) {
+  let error;
+  await fs.mkdir(lane.outputDir, { recursive: true });
+  try {
+    await commandRunner(
+      "bun",
+      [
+        path.join(repoRoot, "scripts", "ui-proof-runtime.mjs"),
+        "run-scenario",
+        "--scenario",
+        plan.scenario,
+        "--base-url",
+        lane.baseURL,
+        "--lane",
+        lane.name,
+        "--output-dir",
+        lane.outputDir,
+      ],
+      { cwd: repoRoot, stdio: "inherit" },
+    );
+  } catch (caught) {
+    error = caught instanceof Error ? caught.message : String(caught);
+  }
+  const manifest = await readLaneManifest(lane.outputDir);
+  const status = manifest.status ?? "fail";
+  const laneError =
+    status === "pass"
+      ? undefined
+      : (manifest.error ?? error ?? "Local Playwright proof did not write a result manifest.");
+  return {
+    error: laneError,
+    localOutputDir: lane.outputDir,
+    name: lane.name,
+    ref: lane.ref,
+    status,
+    steps: manifest.steps ?? [],
+  };
+}
+
 export async function runProofUi({
  args = process.argv.slice(2),
  commandRunner = defaultCommandRunner,
@@ -708,6 +793,7 @@ export async function runProofUi({
    mode: plan.mode,
    outputDir: plan.outputDir,
    provider: plan.provider,
+    runner: plan.runner,
    scenario: plan.scenario,
    status: opts.dryRun ? "dry-run" : "pending",
  };
@@ -720,6 +806,21 @@ export async function runProofUi({
    };
  }

+  if (opts.runner === "local") {
+    const lanes = [];
+    for (const lane of plan.lanes) {
+      lanes.push(await runLocalLane({ commandRunner, lane, plan, repoRoot }));
+    }
+    summary.lanes = lanes;
+    summary.status = lanes.every((lane) => lane.status === "pass") ? "pass" : "fail";
+    await writeSummaryAndReport({ outputDir: plan.outputDir, summary });
+    return {
+      outputDir: plan.outputDir,
+      status: summary.status,
+      summaryPath: path.join(plan.outputDir, "summary.json"),
+    };
+  }
+
  const invocation = crabboxInvocation({ opts, repoRoot });
  const { created, leaseId } = await warmupLease({ commandRunner, invocation, opts, repoRoot });
  summary.crabbox = { createdLease: created, leaseId };
@@ -21,11 +21,58 @@ describe("ui-proof", () => {
        devAuth: false,
        mode: "before-after",
        provider: "hetzner",
+        runner: "crabbox",
        scenario: ".artifacts/proof-scenarios/demo.pw.ts",
      },
    );
  });

+  it("parses local proof URLs and rejects non-local targets", () => {
+    expect(
+      parseProofUiArgs([
+        "--runner",
+        "local",
+        "--mode",
+        "before-after",
+        "--baseline-url",
+        "http://127.0.0.1:4317",
+        "--candidate-url",
+        "http://localhost:4318",
+        "--scenario",
+        ".artifacts/proof-scenarios/demo.pw.ts",
+      ]),
+    ).toMatchObject({
+      baselineUrl: "http://127.0.0.1:4317",
+      candidateUrl: "http://localhost:4318",
+      runner: "local",
+    });
+
+    expect(() =>
+      parseProofUiArgs([
+        "--runner",
+        "local",
+        "--mode",
+        "feature",
+        "--candidate-url",
+        "https://clawhub.ai",
+        "--scenario",
+        ".artifacts/proof-scenarios/demo.pw.ts",
+      ]),
+    ).toThrow("--candidate-url must use localhost");
+    expect(() =>
+      parseProofUiArgs([
+        "--runner",
+        "local",
+        "--mode",
+        "before-after",
+        "--candidate-url",
+        "http://127.0.0.1:4318",
+        "--scenario",
+        ".artifacts/proof-scenarios/demo.pw.ts",
+      ]),
+    ).toThrow("local before-after proof requires --baseline-url and --candidate-url");
+  });
+
  it("parses explicit proof modes and rejects unknown modes", () => {
    expect(
      parseProofUiArgs([
@@ -177,6 +224,61 @@ describe("ui-proof", () => {
    ).resolves.toContain('"mode": "feature"');
  });

+  it("runs a publishable local Playwright proof without invoking Crabbox", async () => {
+    const repoRoot = await fs.mkdtemp(path.join(os.tmpdir(), "clawhub-proof-"));
+    const scenario = path.join(repoRoot, ".artifacts/proof-scenarios/demo.pw.ts");
+    await fs.mkdir(path.dirname(scenario), { recursive: true });
+    await fs.writeFile(scenario, "export default async function demo() {}\n");
+    const commands = [];
+
+    const result = await runProofUi({
+      args: [
+        "--runner",
+        "local",
+        "--mode",
+        "feature",
+        "--candidate-url",
+        "http://127.0.0.1:4318",
+        "--scenario",
+        scenario,
+      ],
+      commandRunner: async (command, commandArgs) => {
+        commands.push([command, commandArgs]);
+        const outputDir = commandArgs[commandArgs.indexOf("--output-dir") + 1];
+        await fs.mkdir(outputDir, { recursive: true });
+        await fs.writeFile(
+          path.join(outputDir, "proof-steps.json"),
+          `${JSON.stringify({
+            lane: "candidate",
+            status: "pass",
+            steps: [
+              {
+                name: "candidate /skills",
+                screenshot: "screenshots/skills.png",
+                status: "pass",
+              },
+            ],
+          })}\n`,
+        );
+        return { stdout: "", stderr: "" };
+      },
+      now: () => new Date("2026-05-12T12:34:56.000Z"),
+      repoRoot,
+    });
+
+    expect(commands).toHaveLength(1);
+    expect(commands[0][0]).toBe("bun");
+    expect(commands[0][1]).toContain("run-scenario");
+    expect(commands[0][1]).toContain("http://127.0.0.1:4318");
+    expect(result.status).toBe("pass");
+    const summary = JSON.parse(await fs.readFile(result.summaryPath, "utf8"));
+    expect(summary).toMatchObject({ runner: "local", status: "pass" });
+    expect(summary.lanes).toHaveLength(1);
+    await expect(fs.readFile(path.join(result.outputDir, "report.md"), "utf8")).resolves.toContain(
+      "Runner: `local`",
+    );
+  });
+
  it("treats a passing proof manifest as authoritative after a Crabbox transport error", async () => {
    const repoRoot = await fs.mkdtemp(path.join(os.tmpdir(), "clawhub-proof-"));
    const scenario = path.join(repoRoot, ".artifacts/proof-scenarios/demo.pw.ts");
@@ -1,13 +1,15 @@
 # UI Proof Runtime

-`proof:ui` is always full-stack. Each lane starts local Convex from that lane's
-checkout, on deterministic lane-specific ports, then builds and previews the
-frontend against those local Convex URLs.
+`proof:ui` always verifies a full-stack ClawHub instance. Crabbox lanes start
+local Convex from that lane's checkout, on deterministic lane-specific ports,
+then build and preview the frontend against those local Convex URLs. Local
+Playwright lanes attach to maintainer-started ClawHub instances on localhost or
+a loopback address.

-The proof runner must not provide a shared-backend mode. UI proof is meant to
-prove the control plane and data plane together: the Git checkout controls both
-frontend and Convex source, and the lane-local Convex URL controls the runtime
-backend/data used by the browser.
+The proof runner must not provide a shared or production-backend mode. UI proof
+is meant to prove the control plane and data plane together: the Git checkout
+controls both frontend and Convex source, and the lane-local Convex URL controls
+the runtime backend/data used by the browser.

 Use `--mode before-after` for baseline-vs-candidate proof and `--mode feature`
 for candidate-only proof. Use `--seed-command` when a scenario needs fixtures.
@@ -15,3 +17,11 @@ for candidate-only proof. Use `--seed-command` when a scenario needs fixtures.
 Dev auth must be explicit. The proof runner must not set
 `VITE_ENABLE_DEV_AUTH=1` by default; scenarios that need it should pass
 `--dev-auth` or explicit `--env` values.
+
+Crabbox is an optional execution environment, not a prerequisite for visual
+verification. Agents should use `proof:ui` when a Crabbox skill or working
+Crabbox capability is available. Otherwise they should ignore Crabbox and run
+the same temporary scenario with `proof:ui --runner local` against real local
+ClawHub instances. Local before/after evidence requires separate baseline and
+candidate URLs and keeps the results in separate `baseline/` and `candidate/`
+directories so `proof:publish` can publish either execution path.