diff --git a/.github/workflows/electron-e2e.yml b/.github/workflows/electron-e2e.yml index 0e4df025..ab2c6894 100644 --- a/.github/workflows/electron-e2e.yml +++ b/.github/workflows/electron-e2e.yml @@ -23,6 +23,7 @@ jobs: - windows-latest env: CI: 'true' + CLAWX_E2E_WORKERS: '2' # Linux runners cannot use Electron's setuid chrome-sandbox; harmless on macOS/Windows. ELECTRON_DISABLE_SANDBOX: '1' diff --git a/AGENTS.md b/AGENTS.md index c099c0f3..49128aa0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,6 +30,7 @@ Standard dev commands are in `package.json` scripts and `README.md`. Key ones: - **pnpm version**: The exact pnpm version is pinned via `packageManager` in `package.json`. Use `corepack enable && corepack prepare` to activate the correct version before installing. - **Electron on headless Linux**: The dbus errors (`Failed to connect to the bus`) are expected and harmless in a headless/cloud environment. The app still runs fine with `$DISPLAY` set (e.g., `:1` via Xvfb/VNC). - **Performance profiling**: `pnpm run perf:chat` writes synthetic Renderer/Main CPU profiles and versioned metrics under ignored Playwright `test-results/`. For live Renderer CDP use `CLAWX_REMOTE_DEBUGGING_PORT=9223 pnpm dev`; for live Main inspection use `pnpm run profile:main` and port 9229. +- **E2E parallel isolation**: Functional Electron specs run concurrently with `CLAWX_E2E_WORKERS=2` by default. Keep tests parallel-safe and test-scoped; apply `E2E_EXCLUSIVE_TAG` from `tests/e2e/parallel-policy.ts` to tests that use the real clipboard or other OS-global state, and `E2E_PERFORMANCE_TAG` to host performance profiles. Extend `tests/unit/e2e-parallel-policy.test.ts` for recognizable new global APIs. - **`pnpm run lint` race condition**: If `pnpm run uv:download` was recently run, ESLint may fail with `ENOENT: no such file or directory, scandir '/workspace/temp_uv_extract'` because the temp directory was created and removed during download. Simply re-run lint after the download script finishes. - **Build scripts warning**: `pnpm install` may warn about ignored build scripts for `@discordjs/opus` and `koffi`. These are optional messaging-channel dependencies and the warnings are safe to ignore. - **`pnpm run init`**: This is a convenience script that runs `pnpm install` followed by `pnpm run uv:download`. Either run `pnpm run init` or run the two steps separately. diff --git a/README.ja-JP.md b/README.ja-JP.md index 286dfe7c..b19d8ddc 100644 --- a/README.ja-JP.md +++ b/README.ja-JP.md @@ -402,6 +402,10 @@ pnpm package:linux # Linux向けにパッケージ化 ヘッドレス Linux では Electron テストに表示サーバーが必要です。`xvfb-run -a pnpm run test:e2e` を利用してください。 +Electron E2E の機能テストはローカルと CI の両方で既定で 2 つの Playwright worker を使用します。通常の並列レーンは `CLAWX_E2E_WORKERS=<正の整数>` でマシンに合わせて調整できます。OS 全体の状態を扱うテストは 1 worker の `exclusive` project に入り、ホストのパフォーマンスプロファイルは機能テスト後に単独で実行されます。新しい E2E テストは既定で並列です。実クリップボードなどのマシン全体で共有されるリソースを使う場合は、`tests/e2e/parallel-policy.ts` の `E2E_EXCLUSIVE_TAG` を適用してください。 + +独占前提を必要としない通常の spec だけを実行する場合は、`pnpm exec playwright test --project=parallel --no-deps` を使用します。 + ### Electron パフォーマンス診断 `pnpm run perf:chat` は隔離された合成 ACP 負荷を実行し、ストリーミング応答と、リッチな静的 Markdown 会話でのサイドバーおよびスクロール操作を測定します。Playwright の `test-results/` には、バージョン付きメトリクスと Renderer/Main CPU プロファイルが出力されます。Renderer プロファイルは本番の store/render 経路とフレームペーシングを対象とします。ストリーミング Main プロファイルは Main から Renderer への IPC fanout を測定し、操作時の Main プロファイルは Renderer 操作中に Main がアイドルのままかを確認します。どちらも上流の OpenClaw/ACP サブプロセスや GPU プロセスの経路は含みません。CPU プロファイルは Chrome DevTools で開けます。アーティファクトには生成されたテスト文字列だけが含まれ、製品テレメトリーには送信されません。測定値はハードウェアに依存するため、共通の絶対閾値ではなく同じマシン上の複数回の結果を比較してください。 diff --git a/README.md b/README.md index 79d8557a..9cf71a8a 100644 --- a/README.md +++ b/README.md @@ -402,6 +402,10 @@ pnpm package:linux # Package for Linux On headless Linux, run Electron tests under a display server such as `xvfb-run -a pnpm run test:e2e`. +Electron E2E functional specs use two Playwright workers by default both locally and in CI; set `CLAWX_E2E_WORKERS=` to tune the ordinary parallel lane for the machine. Tests that touch OS-global state use the one-worker `exclusive` project, and host performance profiles run alone afterward. New E2E tests are parallel by default; apply `E2E_EXCLUSIVE_TAG` from `tests/e2e/parallel-policy.ts` when a test uses the real clipboard or another machine-global resource. + +For a focused ordinary spec that does not need the exclusive prerequisite, run `pnpm exec playwright test --project=parallel --no-deps`. + ### Electron Performance Diagnostics `pnpm run perf:chat` runs isolated synthetic ACP workloads for streaming and for rich static Markdown sidebar/scroll interaction. It writes versioned metrics plus Renderer and Main CPU profiles under the Playwright `test-results/` directory. The Renderer profiles cover the production store/render path and frame pacing. The streaming Main profile measures Main-to-Renderer IPC fanout; the interaction Main profile shows whether Main remains idle while Renderer interactions run. Neither includes the upstream OpenClaw/ACP subprocess or GPU-process paths. Open a CPU profile in Chrome DevTools; the artifacts contain generated fixture text only and are not product telemetry. Results are hardware-dependent, so compare repeated runs on the same machine instead of applying one cross-platform absolute threshold. @@ -432,6 +436,7 @@ from `dist/` and `dist-electron/`, so it does not require manually running - builds the renderer and Electron bundles with `pnpm run build:vite` - starts Electron in an isolated E2E mode with a temporary `HOME` - uses a temporary ClawX `userData` directory +- runs ordinary spec files concurrently while fencing OS-global and performance tests - skips heavy startup side effects such as gateway auto-start, bundled skill installation, tray creation, and CLI auto-install @@ -441,7 +446,7 @@ The first two baseline specs cover: - skipping setup and navigating to the Models page inside the Electron app Add future Electron flows under `tests/e2e/` and reuse the shared fixture in -`tests/e2e/fixtures/electron.ts`. +`tests/e2e/fixtures/electron.ts`. Keep tests parallel-safe by avoiding fixed writable paths, ports, native keychains, and other external shared state; use `E2E_EXCLUSIVE_TAG` when isolation is not possible. ### Tech Stack | Layer | Technology | diff --git a/README.ru-RU.md b/README.ru-RU.md index 5ba8c4a9..30f48b75 100644 --- a/README.ru-RU.md +++ b/README.ru-RU.md @@ -365,6 +365,10 @@ pnpm package:linux # Упаковать для Linux На headless Linux запускайте тесты Electron под сервером отображения, например `xvfb-run -a pnpm run test:e2e`. +Функциональные Electron E2E-тесты локально и в CI по умолчанию используют два worker-процесса Playwright. Число worker-процессов обычной параллельной группы можно настроить через `CLAWX_E2E_WORKERS=<положительное целое>`. Тесты с глобальным состоянием ОС выполняются в однопоточном проекте `exclusive`, а профили производительности хоста запускаются отдельно после функциональных тестов. Новые E2E-тесты параллельны по умолчанию; при работе с реальным буфером обмена или другим общим ресурсом машины используйте `E2E_EXCLUSIVE_TAG` из `tests/e2e/parallel-policy.ts`. + +Чтобы запустить только обычный spec без эксклюзивного предварительного этапа, используйте `pnpm exec playwright test --project=parallel --no-deps`. + ### Проверка регрессии коммуникаций Когда PR изменяет пути коммуникации (события шлюза, поток отправки/получения чата, доставка каналов или откат транспорта), запустите: @@ -385,6 +389,7 @@ pnpm run comms:compare - собирает рендерер и пакеты Electron с `pnpm run build:vite` - запускает Electron в изолированном режиме E2E с временным `HOME` - использует временный каталог `userData` ClawX +- параллельно запускает обычные spec-файлы, изолируя тесты глобальных ресурсов и производительности - пропускает тяжёлые побочные эффекты запуска, такие как автозапуск шлюза, установку упакованных навыков, создание трея и автоустановку CLI Первые два базовых спецификации покрывают: diff --git a/README.zh-CN.md b/README.zh-CN.md index 148c505e..f7155db0 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -403,6 +403,10 @@ pnpm package:linux # 为 Linux 打包 在无头 Linux 环境下，Electron 测试需要显示服务；可使用 `xvfb-run -a pnpm run test:e2e`。 +Electron E2E 功能测试在本地和 CI 中默认使用两个 Playwright worker；可通过 `CLAWX_E2E_WORKERS=<正整数>` 按机器能力调整普通并行通道。访问操作系统全局状态的测试进入单 worker 的 `exclusive` project，主机性能采样则在功能测试结束后独占运行。新增 E2E 测试默认并行；若测试使用真实剪贴板或其他机器级共享资源，请应用 `tests/e2e/parallel-policy.ts` 中的 `E2E_EXCLUSIVE_TAG`。 + +如果只需运行一个不依赖独占前置阶段的普通 spec，可使用 `pnpm exec playwright test --project=parallel --no-deps`。 + ### Electron 性能诊断 `pnpm run perf:chat` 会运行隔离的合成 ACP 负载，分别覆盖流式响应，以及富 Markdown 静态会话中的侧栏和滚动交互，并在 Playwright 的 `test-results/` 目录输出版本化指标与 Renderer/Main CPU Profile。Renderer Profile 覆盖生产 store/render 路径和帧节奏；流式 Main Profile 测量 Main 到 Renderer 的 IPC fanout，交互 Main Profile 用于确认 Renderer 交互期间 Main 是否保持空闲。两者都不包含上游 OpenClaw/ACP 子进程或 GPU 进程路径。CPU Profile 可直接用 Chrome DevTools 打开；其中只包含生成的测试文本，不会上报为产品遥测。性能数据依赖硬件，应在同一机器上多次运行后对比，不应使用统一的跨平台绝对阈值。 diff --git a/harness/reference/e2e-parallelism.md b/harness/reference/e2e-parallelism.md new file mode 100644 index 00000000..dd33288a --- /dev/null +++ b/harness/reference/e2e-parallelism.md @@ -0,0 +1,15 @@ +# Electron E2E Parallelism + +ClawX launches one Electron process per Playwright test with a test-scoped HOME and user-data directory. Ordinary specs can therefore run in separate workers without sharing application stores or OpenClaw files. + +The Playwright project graph has three ordered lanes: + +1. `exclusive` runs tests tagged `@exclusive` with one worker. +2. `parallel` runs all ordinary functional tests with the configured worker count after `exclusive` succeeds. +3. `performance` runs tests tagged `@performance` with one worker after functional tests finish. + +Real clipboard tests are exclusive because Electron renderer instances read and write the same OS clipboard. Renderer performance tests run last because concurrent Electron processes distort CPU, GPU, frame-pacing, and elapsed-time evidence even when their files are otherwise isolated. `test.describe.configure({ mode: 'serial' })` is not sufficient for either case because it does not prevent another spec file or project from running at the same time. + +New tests are parallel by default. A test that uses an OS-global resource must import and apply `E2E_EXCLUSIVE_TAG`; a host performance profile must use `E2E_PERFORMANCE_TAG`. Extend `tests/unit/e2e-parallel-policy.test.ts` when another recognizable global API is introduced. No static check can identify every possible external side effect, so reviewers must classify tests that use native dialogs, keychains, fixed ports, fixed writable paths, external runtimes, or other machine-global state. + +Use `CLAWX_E2E_WORKERS` to override the ordinary worker count on constrained or high-capacity machines. Playwright project dependencies make a directly filtered ordinary spec run the exclusive prerequisite first; add `--project=parallel --no-deps` when a focused command intentionally needs only an audited ordinary spec. `pnpm run perf:chat` selects the performance project without running its dependencies. diff --git a/harness/specs/rules/e2e-parallel-isolation.md b/harness/specs/rules/e2e-parallel-isolation.md new file mode 100644 index 00000000..e62a5dc5 --- /dev/null +++ b/harness/specs/rules/e2e-parallel-isolation.md @@ -0,0 +1,18 @@ +--- +id: e2e-parallel-isolation +title: E2E Parallel Isolation +type: ai-coding-rule +appliesTo: + - gateway-backend-communication +requiredProfiles: + - fast + - e2e +--- + +Electron E2E tests are parallel by default because each test owns its HOME, OpenClaw state directory, Electron user-data directory, and Host API configuration. Keep those fixtures test-scoped. + +Tests that mutate OS-global state must use `E2E_EXCLUSIVE_TAG`. Tests that profile shared host CPU, GPU, display, or frame pacing must use `E2E_PERFORMANCE_TAG`. Do not use Playwright serial mode as a cross-file mutex; serial mode only orders tests within its own group. + +When adding another global resource, extend the automated policy check where the resource has a recognizable API. Unknown external resources still require reviewer classification. + +The project graph, environment isolation, and validation commands are documented in `harness/reference/e2e-parallelism.md`. diff --git a/harness/specs/scenarios/gateway-backend-communication.md b/harness/specs/scenarios/gateway-backend-communication.md index 972c38e5..ef477f8d 100644 --- a/harness/specs/scenarios/gateway-backend-communication.md +++ b/harness/specs/scenarios/gateway-backend-communication.md @@ -60,6 +60,7 @@ requiredRules: - provider-model-selection-authority - sidebar-session-attention-authority - web-browser-security-and-lifecycle + - e2e-parallel-isolation - comms-regression - docs-sync forbiddenPatterns: @@ -96,6 +97,6 @@ Scheduled-task history is Main-owned backend data. Current OpenClaw versions mus The local HTML Preview privileged bridge is also Main-owned: Renderer may load a validated local HTML file or open that current file externally through the typed Host API. The guest is an implementation detail of the existing `preview` tab; there is no `web-browser` artifact tab or general address navigation. The durable guest contract is `harness/reference/web-browser.md`. -Gateway session-catalog subscription, normalization, ordered list/event replay, attention transitions, and reconnect recovery are documented in `harness/reference/sidebar-session-attention.md`. +Gateway session-catalog subscription, normalization, ordered list/event replay, attention transitions, and reconnect recovery are documented in `harness/reference/sidebar-session-attention.md`. Electron test-process isolation and global-resource scheduling are documented in `harness/reference/e2e-parallelism.md`. Gateway WebSocket heartbeat misses are diagnostic availability signals only. They may mark health unresponsive, but must not terminate the socket or replace the Gateway process; authoritative process-exit and socket-close signals retain automatic lifecycle recovery ownership. diff --git a/harness/specs/tasks/parallelize-electron-e2e.md b/harness/specs/tasks/parallelize-electron-e2e.md new file mode 100644 index 00000000..1be51029 --- /dev/null +++ b/harness/specs/tasks/parallelize-electron-e2e.md @@ -0,0 +1,57 @@ +--- +id: parallelize-electron-e2e +title: Parallelize Electron E2E safely +scenario: gateway-backend-communication +taskType: runtime-bridge +intent: Run isolated Electron E2E specs concurrently while fencing OS-global resources and host-sensitive performance profiles. +touchedAreas: + - playwright.config.ts + - package.json + - .github/workflows/electron-e2e.yml + - AGENTS.md + - tests/e2e/fixtures/electron.ts + - tests/e2e/parallel-policy.ts + - tests/e2e/chat-streamdown-rendering.spec.ts + - tests/e2e/chat-acp-attachments.spec.ts + - tests/e2e/markdown-file-preview.spec.ts + - tests/e2e/renderer-performance.spec.ts + - tests/unit/e2e-parallel-policy.test.ts + - harness/reference/e2e-parallelism.md + - harness/specs/rules/e2e-parallel-isolation.md + - harness/specs/scenarios/gateway-backend-communication.md + - harness/specs/tasks/parallelize-electron-e2e.md + - README.md + - README.zh-CN.md + - README.ja-JP.md + - README.ru-RU.md +expectedUserBehavior: + - Local and CI Electron E2E runs execute independent spec files concurrently. + - Tests that use the OS clipboard and host performance profiles never overlap incompatible tests. +requiredProfiles: + - fast + - comms + - e2e +requiredRules: + - backend-communication-boundary + - e2e-parallel-isolation + - docs-sync +requiredTests: + - pnpm exec vitest run tests/unit/e2e-parallel-policy.test.ts + - pnpm harness validate --spec harness/specs/tasks/parallelize-electron-e2e.md + - pnpm run test:e2e + - pnpm run typecheck + - pnpm run lint:check +acceptance: + - The ordinary E2E project uses more than one worker by default and can be overridden for constrained machines. + - OS-global clipboard tests execute in a one-worker prerequisite project. + - Renderer performance profiles execute alone after functional E2E tests. + - Per-test HOME, Electron profile, OpenClaw state, and Host API configuration remain isolated. + - CI opts into the checked-in parallel worker policy on every supported OS. + - A durable policy and automated guard explain how future global-resource tests enter the exclusive lane. +docs: + required: true +--- + +## Scope + +This task changes only Electron E2E scheduling and fixture isolation. It does not change application transport behavior, production Host API routing, or user-visible ClawX features. diff --git a/package.json b/package.json index 1015ade4..d59c5c88 100644 --- a/package.json +++ b/package.json @@ -52,7 +52,7 @@ "test": "vitest run", "test:e2e": "pnpm run build:vite && playwright test", "test:e2e:headed": "pnpm run build:vite && playwright test --headed", - "perf:chat": "pnpm run build:vite && playwright test tests/e2e/renderer-performance.spec.ts --workers=1", + "perf:chat": "pnpm run build:vite && playwright test tests/e2e/renderer-performance.spec.ts --project=performance --no-deps --workers=1", "profile:main": "pnpm run build:vite && electron --inspect=9229 .", "harness": "pnpm --filter @clawx/harness start --", "harness:ci": "pnpm harness list && pnpm harness validate --spec harness/specs/scenarios/gateway-backend-communication.md && pnpm harness validate --spec harness/specs/tasks/maintain-session-catalog-reconciliation.example.md --no-diff && pnpm harness run --spec harness/specs/scenarios/gateway-backend-communication.md --dry-run && pnpm exec vitest run tests/unit/harness-specs.test.ts tests/unit/harness-git.test.ts", diff --git a/playwright.config.ts b/playwright.config.ts index 7c5e8331..e3032ff1 100644 --- a/playwright.config.ts +++ b/playwright.config.ts @@ -1,9 +1,29 @@ import { defineConfig } from '@playwright/test'; +import { + DEFAULT_E2E_WORKERS, + E2E_EXCLUSIVE_TAG, + E2E_PERFORMANCE_TAG, +} from './tests/e2e/parallel-policy'; + +function e2eWorkers(): number { + const configured = process.env.CLAWX_E2E_WORKERS?.trim(); + if (!configured) return DEFAULT_E2E_WORKERS; + + const workers = Number(configured); + if (!Number.isInteger(workers) || workers < 1) { + throw new Error('CLAWX_E2E_WORKERS must be a positive integer'); + } + return workers; +} + +const exclusivePattern = new RegExp(E2E_EXCLUSIVE_TAG); +const performancePattern = new RegExp(E2E_PERFORMANCE_TAG); +const nonParallelPattern = new RegExp(`${E2E_EXCLUSIVE_TAG}|${E2E_PERFORMANCE_TAG}`); export default defineConfig({ testDir: './tests/e2e', fullyParallel: false, - workers: 1, + workers: e2eWorkers(), forbidOnly: Boolean(process.env.CI), retries: process.env.CI ? 2 : 0, timeout: 90_000, @@ -19,4 +39,22 @@ export default defineConfig({ screenshot: 'only-on-failure', video: 'retain-on-failure', }, + projects: [ + { + name: 'exclusive', + grep: exclusivePattern, + workers: 1, + }, + { + name: 'parallel', + grepInvert: nonParallelPattern, + dependencies: ['exclusive'], + }, + { + name: 'performance', + grep: performancePattern, + dependencies: ['parallel'], + workers: 1, + }, + ], }); diff --git a/tests/e2e/chat-acp-attachments.spec.ts b/tests/e2e/chat-acp-attachments.spec.ts index d618eebe..4f50033b 100644 --- a/tests/e2e/chat-acp-attachments.spec.ts +++ b/tests/e2e/chat-acp-attachments.spec.ts @@ -76,6 +76,13 @@ function filesActionCalls( return calls.filter((call) => call.module === 'files' && call.action === action); } +function resolvedAttachmentRefs(calls: RecordedHostInvocation[]): Record[] { + return calls + .filter((call) => call.module === 'files' && call.action === 'resolveAttachment') + .map((call) => call.payload?.ref) + .filter((ref): ref is Record => ref != null); +} + async function openChat(app: ElectronApplication): Promise { const page = await getStableWindow(app); try { @@ -227,14 +234,6 @@ test.describe('ACP media attachments', () => { )); return resolveCall?.payload?.ref ?? null; }).not.toBeNull(); - const resolveCall = (await fixture.getHostInvocations()).find((call) => ( - call.module === 'files' - && call.action === 'resolveAttachment' - && (call.payload?.ref as Record | undefined)?.uri === spreadsheetPath - )); - const resolvedRef = resolveCall?.payload?.ref as Record; - await fixture.clearInvocations(); - await trigger.click(); const menu = page.getByTestId('acp-attachment-open-with-menu'); await expect(menu).toBeVisible(); @@ -249,7 +248,17 @@ test.describe('ACP media attachments', () => { await expect.poll(async () => filesActionCalls( await fixture.getHostInvocations(), 'listAttachmentOpenHandlers', - ).map((call) => call.payload)).toEqual([resolvedRef]); + )).toHaveLength(1); + const openWithRef = filesActionCalls( + await fixture.getHostInvocations(), + 'listAttachmentOpenHandlers', + )[0].payload as Record; + expect(openWithRef).toMatchObject({ + sessionKey: MAIN_SESSION_KEY, + uri: spreadsheetPath, + generation: expect.any(Number), + }); + expect(resolvedAttachmentRefs(await fixture.getHostInvocations())).toContainEqual(openWithRef); const appRows = page.getByTestId('acp-attachment-open-with-app'); await expect(appRows).toHaveCount(2); await expect(appRows.nth(0)).toHaveText('Zulu Sheets'); @@ -265,7 +274,7 @@ test.describe('ACP media attachments', () => { await expect.poll(async () => filesActionCalls( await fixture.getHostInvocations(), 'openAttachmentWith', - ).map((call) => call.payload)).toEqual([{ ref: resolvedRef, handlerId: 'app-alpha' }]); + ).map((call) => call.payload)).toEqual([{ ref: openWithRef, handlerId: 'app-alpha' }]); await expect(page.getByTestId('artifact-panel')).toHaveCount(0); await trigger.click(); } else { @@ -278,7 +287,17 @@ test.describe('ACP media attachments', () => { await expect.poll(async () => filesActionCalls( await fixture.getHostInvocations(), 'revealAttachment', - ).map((call) => call.payload)).toEqual([resolvedRef]); + )).toHaveLength(1); + const revealRef = filesActionCalls( + await fixture.getHostInvocations(), + 'revealAttachment', + )[0].payload as Record; + expect(revealRef).toMatchObject({ + sessionKey: MAIN_SESSION_KEY, + uri: spreadsheetPath, + generation: expect.any(Number), + }); + expect(resolvedAttachmentRefs(await fixture.getHostInvocations())).toContainEqual(revealRef); await expect(page.getByTestId('artifact-panel')).toHaveCount(0); await fixture.clearInvocations(); diff --git a/tests/e2e/chat-streamdown-rendering.spec.ts b/tests/e2e/chat-streamdown-rendering.spec.ts index 8253b528..f47c2f78 100644 --- a/tests/e2e/chat-streamdown-rendering.spec.ts +++ b/tests/e2e/chat-streamdown-rendering.spec.ts @@ -1,5 +1,6 @@ import type { ElectronApplication } from '@playwright/test'; import { closeElectronApp, expect, getStableWindow, installIpcMocks, test } from './fixtures/electron'; +import { E2E_EXCLUSIVE_TAG } from './parallel-policy'; const SESSION_KEY = 'agent:main:main'; const MAIN_WORKSPACE = '/workspace'; @@ -78,7 +79,7 @@ async function resolveAcpPrompt(app: ElectronApplication) { }); } -test.describe('ClawX streaming Markdown rendering', () => { +test.describe('ClawX streaming Markdown rendering', { tag: E2E_EXCLUSIVE_TAG }, () => { test('repairs and animates only the pending assistant response, then clears animation markers', async ({ launchElectronApp }) => { const app = await launchElectronApp({ skipSetup: true }); diff --git a/tests/e2e/fixtures/electron.ts b/tests/e2e/fixtures/electron.ts index 8d566218..b1072d02 100644 --- a/tests/e2e/fixtures/electron.ts +++ b/tests/e2e/fixtures/electron.ts @@ -2,7 +2,6 @@ import electronBinaryPath from 'electron'; import { _electron as electron, expect, test as base, type ElectronApplication, type Page } from '@playwright/test'; import { build as buildWithEsbuild } from 'esbuild'; import { access, mkdir, mkdtemp, realpath, rm, writeFile } from 'node:fs/promises'; -import { createServer } from 'node:net'; import { tmpdir } from 'node:os'; import { dirname, join, relative, resolve } from 'node:path'; import { pathToFileURL } from 'node:url'; @@ -139,28 +138,6 @@ function productionAttachmentBundle(): Promise { return productionAttachmentBundlePromise; } -async function allocatePort(): Promise { - return await new Promise((resolvePort, reject) => { - const server = createServer(); - server.once('error', reject); - server.listen(0, '127.0.0.1', () => { - const address = server.address(); - if (!address || typeof address === 'string') { - server.close(() => reject(new Error('Failed to allocate an ephemeral port'))); - return; - } - const { port } = address; - server.close((error) => { - if (error) { - reject(error); - return; - } - resolvePort(port); - }); - }); - }); -} - async function getStableWindow(app: ElectronApplication): Promise { const deadline = Date.now() + 30_000; let page = await app.firstWindow(); @@ -254,7 +231,10 @@ async function launchClawXElectron( throw new Error('Electron E2E must not bypass application media permission prompts'); } await seedE2eSettings(userDataDir); - const hostApiPort = await allocatePort(); + const inheritedEnv = { ...process.env }; + delete inheritedEnv.CLAWX_E2E_SKIP_SETUP; + delete inheritedEnv.CLAWX_REMOTE_DEBUGGING_PORT; + delete inheritedEnv.VITE_DEV_SERVER_URL; const electronEnv = process.platform === 'linux' ? { ELECTRON_DISABLE_SANDBOX: '1', @@ -265,7 +245,7 @@ async function launchClawXElectron( executablePath: electronBinaryPath, args: ['--lang=en-US', ...(options.additionalArgs ?? []), electronEntry], env: { - ...process.env, + ...inheritedEnv, ...electronEnv, HOME: homeDir, USERPROFILE: homeDir, @@ -277,8 +257,9 @@ async function launchClawXElectron( LANGUAGE: 'en', CLAWX_E2E: '1', CLAWX_USER_DATA_DIR: userDataDir, + OPENCLAW_STATE_DIR: join(homeDir, '.openclaw'), + OPENCLAW_CONFIG_PATH: join(homeDir, '.openclaw', 'openclaw.json'), ...(options.skipSetup ? { CLAWX_E2E_SKIP_SETUP: '1' } : {}), - CLAWX_PORT_CLAWX_HOST_API: String(hostApiPort), }, timeout: 90_000, }); diff --git a/tests/e2e/markdown-file-preview.spec.ts b/tests/e2e/markdown-file-preview.spec.ts index 6fddfc82..7774fbfa 100644 --- a/tests/e2e/markdown-file-preview.spec.ts +++ b/tests/e2e/markdown-file-preview.spec.ts @@ -6,6 +6,7 @@ import { installAttachmentHostFixture, test, } from './fixtures/electron'; +import { E2E_EXCLUSIVE_TAG } from './parallel-policy'; const MAIN_SESSION_KEY = 'agent:main:main'; const FILE_NAME = 'streamdown-preview.md'; @@ -71,7 +72,7 @@ async function openChat(app: ElectronApplication): Promise { return page; } -test.describe('Markdown file preview', () => { +test.describe('Markdown file preview', { tag: E2E_EXCLUSIVE_TAG }, () => { test('renders workspace Markdown through static Streamdown', async ({ launchElectronApp }) => { const app = await launchElectronApp({ skipSetup: true }); diff --git a/tests/e2e/parallel-policy.ts b/tests/e2e/parallel-policy.ts new file mode 100644 index 00000000..fbe6b762 --- /dev/null +++ b/tests/e2e/parallel-policy.ts @@ -0,0 +1,3 @@ +export const DEFAULT_E2E_WORKERS = 2; +export const E2E_EXCLUSIVE_TAG = '@exclusive'; +export const E2E_PERFORMANCE_TAG = '@performance'; diff --git a/tests/e2e/renderer-performance.spec.ts b/tests/e2e/renderer-performance.spec.ts index 39cec73c..166fb7cd 100644 --- a/tests/e2e/renderer-performance.spec.ts +++ b/tests/e2e/renderer-performance.spec.ts @@ -10,6 +10,7 @@ import { stopMainCpuProfile, test, } from './fixtures/electron'; +import { E2E_PERFORMANCE_TAG } from './parallel-policy'; const SESSION_KEY = 'agent:main:performance'; const WORKSPACE = '/synthetic-workspace'; @@ -171,7 +172,9 @@ async function writeArtifact(testInfo: TestInfo, name: string, body: unknown): P test.use({ trace: 'off', video: 'off' }); -test('profiles a populated timeline during a growing Markdown stream', async ({ launchElectronApp }, testInfo) => { +test('profiles a populated timeline during a growing Markdown stream', { + tag: E2E_PERFORMANCE_TAG, +}, async ({ launchElectronApp }, testInfo) => { const app = await launchElectronApp({ skipSetup: true }); try { @@ -314,7 +317,9 @@ test('profiles a populated timeline during a growing Markdown stream', async ({ } }); -test('profiles sidebar animation and scrolling with rich static Markdown', async ({ launchElectronApp }, testInfo) => { +test('profiles sidebar animation and scrolling with rich static Markdown', { + tag: E2E_PERFORMANCE_TAG, +}, async ({ launchElectronApp }, testInfo) => { const app = await launchElectronApp({ skipSetup: true }); try { diff --git a/tests/unit/e2e-parallel-policy.test.ts b/tests/unit/e2e-parallel-policy.test.ts new file mode 100644 index 00000000..318f7720 --- /dev/null +++ b/tests/unit/e2e-parallel-policy.test.ts @@ -0,0 +1,176 @@ +import { readFile, readdir } from 'node:fs/promises'; +import path from 'node:path'; +import ts from 'typescript'; + +import { describe, expect, it } from 'vitest'; + +import playwrightConfig from '../../playwright.config'; +import { + DEFAULT_E2E_WORKERS, + E2E_EXCLUSIVE_TAG, + E2E_PERFORMANCE_TAG, +} from '../e2e/parallel-policy'; + +const e2eDir = path.resolve('tests/e2e'); + +async function readSpecTree(directory: string): Promise> { + const entries = await readdir(directory, { withFileTypes: true }); + const nested = await Promise.all(entries.map(async (entry) => { + const file = path.join(directory, entry.name); + if (entry.isDirectory()) return readSpecTree(file); + if (!entry.isFile() || !/\.spec\.tsx?$/.test(entry.name)) return []; + return [{ file, source: await readFile(file, 'utf8') }]; + })); + return nested.flat(); +} + +function accessPath(expression: ts.Expression): string[] | null { + if (ts.isIdentifier(expression)) return [expression.text]; + if (ts.isPropertyAccessExpression(expression)) { + const parent = accessPath(expression.expression); + return parent ? [...parent, expression.name.text] : null; + } + if (ts.isElementAccessExpression(expression) + && expression.argumentExpression + && ts.isStringLiteralLike(expression.argumentExpression)) { + const parent = accessPath(expression.expression); + return parent ? [...parent, expression.argumentExpression.text] : null; + } + return null; +} + +function isGlobalClipboardCall(node: ts.CallExpression): boolean { + const pathParts = accessPath(node.expression); + if (!pathParts) return false; + if (pathParts.length === 1) return pathParts[0] === 'installWebBrowserPolicyInstrumentation'; + + const method = pathParts.at(-1) ?? ''; + return pathParts.at(-2) === 'clipboard' && /^(?:clear|read\w*|write\w*)$/.test(method); +} + +function appliesTag(call: ts.CallExpression, tagName: string): boolean { + return call.arguments.some((argument) => ( + ts.isObjectLiteralExpression(argument) + && argument.properties.some((property) => ( + ts.isPropertyAssignment(property) + && property.name.getText() === 'tag' + && ts.isIdentifier(property.initializer) + && property.initializer.text === tagName + )) + )); +} + +function hasTaggedTestAncestor(node: ts.Node, tagName: string): boolean { + for (let current: ts.Node | undefined = node; current; current = current.parent) { + if (!ts.isCallExpression(current)) continue; + const expression = current.expression.getText(); + if ((expression === 'test' || expression === 'test.describe') && appliesTag(current, tagName)) { + return true; + } + } + return false; +} + +function untaggedGlobalClipboardCalls(file: string, source: string): string[] { + const sourceFile = ts.createSourceFile(file, source, ts.ScriptTarget.Latest, true); + const untagged: string[] = []; + + const visit = (node: ts.Node) => { + if (ts.isCallExpression(node) + && isGlobalClipboardCall(node) + && !hasTaggedTestAncestor(node, 'E2E_EXCLUSIVE_TAG')) { + const { line, character } = sourceFile.getLineAndCharacterOfPosition(node.getStart(sourceFile)); + untagged.push(`${path.relative(e2eDir, file)}:${line + 1}:${character + 1}`); + } + ts.forEachChild(node, visit); + }; + visit(sourceFile); + return untagged; +} + +function untaggedTestDefinitions(file: string, source: string, tagName: string): string[] { + const sourceFile = ts.createSourceFile(file, source, ts.ScriptTarget.Latest, true); + const untagged: string[] = []; + + const visit = (node: ts.Node) => { + if (ts.isCallExpression(node)) { + const pathParts = accessPath(node.expression); + const isTestDefinition = pathParts?.[0] === 'test' + && (pathParts.length === 1 || ['fixme', 'only', 'skip'].includes(pathParts[1])) + && ts.isStringLiteralLike(node.arguments[0]); + if (isTestDefinition && !hasTaggedTestAncestor(node, tagName)) { + const { line, character } = sourceFile.getLineAndCharacterOfPosition(node.getStart(sourceFile)); + untagged.push(`${path.relative(e2eDir, file)}:${line + 1}:${character + 1}`); + } + } + ts.forEachChild(node, visit); + }; + visit(sourceFile); + return untagged; +} + +describe('Electron E2E parallel policy', () => { + it('runs isolated tests concurrently while fencing global resources and performance profiles', () => { + const projects = playwrightConfig.projects ?? []; + const exclusive = projects.find((project) => project.name === 'exclusive'); + const parallel = projects.find((project) => project.name === 'parallel'); + const performance = projects.find((project) => project.name === 'performance'); + + expect(playwrightConfig.fullyParallel).toBe(false); + expect(DEFAULT_E2E_WORKERS).toBeGreaterThan(1); + expect(playwrightConfig.workers).toBe( + process.env.CLAWX_E2E_WORKERS ? Number(process.env.CLAWX_E2E_WORKERS) : DEFAULT_E2E_WORKERS, + ); + expect(exclusive?.workers).toBe(1); + expect(String(exclusive?.grep)).toContain(E2E_EXCLUSIVE_TAG); + expect(parallel?.dependencies).toEqual(['exclusive']); + expect(String(parallel?.grepInvert)).toContain(E2E_EXCLUSIVE_TAG); + expect(String(parallel?.grepInvert)).toContain(E2E_PERFORMANCE_TAG); + expect(performance?.workers).toBe(1); + expect(performance?.dependencies).toEqual(['parallel']); + expect(String(performance?.grep)).toContain(E2E_PERFORMANCE_TAG); + }); + + it('keeps specs that access the real clipboard in the exclusive project', async () => { + const specs = await readSpecTree(e2eDir); + const untagged = specs.flatMap(({ file, source }) => untaggedGlobalClipboardCalls(file, source)); + expect(untagged, 'OS clipboard calls must be enclosed by an exclusively tagged test or describe') + .toEqual([]); + }); + + it('does not accept an unused exclusive tag import', () => { + const source = [ + "import { E2E_EXCLUSIVE_TAG } from './parallel-policy';", + "test('copy', async () => navigator.clipboard.readText());", + ].join('\n'); + const untagged = untaggedGlobalClipboardCalls(path.join(e2eDir, 'unused-tag.spec.ts'), source); + expect(untagged).toHaveLength(1); + expect(untagged[0]).toMatch(/^unused-tag\.spec\.ts:2:/); + }); + + it('recognizes optional and element access to the OS clipboard', () => { + const source = [ + "test('optional', async () => navigator.clipboard?.readText());", + "test('element', async () => navigator.clipboard['writeText']('value'));", + ].join('\n'); + const untagged = untaggedGlobalClipboardCalls(path.join(e2eDir, 'access-forms.spec.ts'), source); + expect(untagged).toHaveLength(2); + }); + + it('keeps renderer performance profiles in the performance project', async () => { + const file = path.join(e2eDir, 'renderer-performance.spec.ts'); + const source = await readFile(file, 'utf8'); + expect(untaggedTestDefinitions(file, source, 'E2E_PERFORMANCE_TAG')).toEqual([]); + }); + + it('rejects an individually untagged performance test', () => { + const source = [ + "test('tagged', { tag: E2E_PERFORMANCE_TAG }, async () => {});", + "test('untagged', async () => {});", + ].join('\n'); + const file = path.join(e2eDir, 'synthetic-performance.spec.ts'); + expect(untaggedTestDefinitions(file, source, 'E2E_PERFORMANCE_TAG')).toEqual([ + 'synthetic-performance.spec.ts:2:1', + ]); + }); +});