diff --git a/tests/E2E_TESTING_NOTICE.md b/tests/E2E_TESTING_NOTICE.md index 774e18b..2596791 100644 --- a/tests/E2E_TESTING_NOTICE.md +++ b/tests/E2E_TESTING_NOTICE.md @@ -1,6 +1,11 @@ All E2E tests must be performed using the standard procedures defined in `tests/E2E_TESTING_SOP.md`. +Mandatory testing-design rule: + +- E2E tests must be designed to reproduce real user-visible failures and catch bugs early, not merely to pass validation. +- Do not add pass-only E2E checks that cannot fail for the bug class under review. +- For every user-reported or high-risk frontend regression, ask which E2E assertion would have caught it before release, then add or update that assertion. Exception: - strictly documentation-only changes do not require entering the E2E workflow - this exception does not apply once application code, test code, scripts, configs, or generated artifacts change @@ -10,3 +15,21 @@ Scope note: - Backend real-E2E lanes (`tests.test_r122_real_backend_lane`, `tests.test_r123_real_backend_model_list_lane`) are governed by `tests/TEST_SOP.md`. For public/admin/webhook/connector or other user-facing transaction changes, acceptance evidence must include at least one transaction-level probe that verifies the actual submitted outcome; route load or redirect-only evidence is not sufficient on its own. + +## RookieUI-Derived Global E2E Notice + +All E2E tests must follow `tests/E2E_TESTING_SOP.md`. Full acceptance workflow and gate order remain defined by `tests/TEST_SOP.md`. + +Mandatory testing-design rule: + +- E2E tests must be designed to reproduce real user-visible failures and catch bugs early, not merely to pass validation. +- Do not add pass-only E2E checks that cannot fail for the bug class under review. +- For every user-reported or high-risk frontend regression, ask which E2E assertion would have caught it before release, then add or update that assertion. + +Exception: + +- strictly documentation-only changes do not require entering the E2E workflow +- once code/tests/scripts/config/runtime files change, this exception does not apply + +For transaction-sensitive features, acceptance evidence must include at least one action-level assertion of final outcome, not route-load evidence only. + diff --git a/tests/E2E_TESTING_SOP.md b/tests/E2E_TESTING_SOP.md index a76506c..5571cd2 100644 --- a/tests/E2E_TESTING_SOP.md +++ b/tests/E2E_TESTING_SOP.md @@ -1,5 +1,10 @@ # ComfyUI-OpenClaw E2E Testing SOP +## Problem-First Test Design Rule + +E2E scripts and mocked harness flows must be designed to reproduce failures and catch bugs early. The goal is not to make the harness pass; the goal is to make the harness fail when a real user-facing contract breaks. + +When adding or reviewing E2E coverage, prefer assertions that prove final user-visible behavior, request routing, payload shape, state synchronization, and failure feedback. Avoid pass-only checks that only prove the page loaded or a mocked happy path returned. This SOP documents the verified, repeatable steps to run Playwright E2E tests against a local **test harness** (no live ComfyUI backend required). @@ -115,3 +120,63 @@ Non-examples: - loading the entry page only - verifying only that a route exists or returns a redirect - asserting only mocked backend behavior when the production seam is the failure point + +## RookieUI-Derived Global E2E Rules + +This section preserves the repo's existing E2E procedure while adding the shared Playwright/harness baseline used across this workspace. + +### Problem-First Test Design Rule + +E2E scripts and mocked harness flows must be designed to reproduce failures and catch bugs early. The goal is not to make the harness pass; the goal is to make the harness fail when a real user-facing contract breaks. + +When adding or reviewing E2E coverage, prefer assertions that prove final user-visible behavior, request routing, payload shape, state synchronization, and failure feedback. Avoid pass-only checks that only prove the page loaded or a mocked happy path returned. + +### Requirements + +- Node.js 18+ +- npm 9+ when the repo uses npm +- Python command available (`python` or a local shim to `python3`) when the harness serves files through Python +- Playwright Chromium installed with `npx playwright install chromium` when Playwright is used + +### Windows (PowerShell) + +```powershell +node -v +npm -v +python --version + +npm install +npx playwright install chromium +npm test +``` + +### WSL2 (bash) + +```bash +source ~/.nvm/nvm.sh +nvm use 18 +node -v +python3 --version + +mkdir -p .tmp/bin +ln -sf "$(command -v python3)" .tmp/bin/python + +npm install +npx playwright install chromium + +mkdir -p .tmp/playwright +TMPDIR=.tmp/playwright TMP=.tmp/playwright TEMP=.tmp/playwright \ + PATH=".tmp/bin:$PATH" npm test +``` + +### Troubleshooting + +- `python: command not found` on WSL: create `.tmp/bin/python` as a shim to `python3`. +- Port bind failure: use the repo-documented E2E port override or stop the conflicting process. +- Browser missing: run `npx playwright install chromium`. +- Dependency drift: remove `node_modules` and rerun `npm install`. + +### Non-applicable E2E + +If the repo does not have a frontend or Playwright harness, document the non-applicability in `tests/TEST_SOP.md` and identify the replacement smoke, unit, or integration lane. Do not treat a missing E2E harness as an unrecorded pass. + diff --git a/tests/TEST_SOP.md b/tests/TEST_SOP.md index 918d742..9b7ec66 100644 --- a/tests/TEST_SOP.md +++ b/tests/TEST_SOP.md @@ -1,5 +1,12 @@ # Test SOP +### Problem-First Test Design Rule (Mandatory) + +All test scripts, test harnesses, and validation flows must be designed first to reproduce real failures and catch bugs early. + +The purpose of testing is to expose defects, regressions, drift, and broken assumptions before users hit them. Tests must not be designed merely to produce a green validation result, satisfy a checklist, or prove that a happy path still passes. Do not waste validation time on pass-only checks that cannot fail for the bug class under review. + +Every bugfix or high-risk change must start from the question: "Which test would have caught this before release?" If the existing gate missed the bug, update the targeted test or SOP flow so the same class of bug fails deterministically next time. This document defines the **mandatory test workflow** for this repo. Run it **before every push** (unless you explicitly document why you are skipping). ## Acceptance Rule (SOP) @@ -771,3 +778,64 @@ PRE_COMMIT_HOME=/tmp/pre-commit-cache pre-commit run --all-files --show-diff-on- - Check the console error (module import/exports mismatch is the most common cause). - Verify all referenced JS modules exist and export expected names. + +## RookieUI-Derived Global Testing Rules + +These rules preserve this repository's existing test lanes while adding the shared testing baseline used across this workspace. + +### Required Reading Order + +1. `tests/TEST_SOP.md` +2. `tests/E2E_TESTING_NOTICE.md` +3. `tests/E2E_TESTING_SOP.md` + +### Acceptance Rule + +A change is not accepted until required checks pass and evidence is recorded. Existing repo-specific gates remain authoritative; this section adds the shared minimum expectations. + +Required shared gate: + +1. `pre-commit run detect-secrets --all-files` +2. `pre-commit run --all-files --show-diff-on-failure` +3. backend/unit tests through the repo's documented runner, preferring `scripts/run_unittests.py` when present +4. frontend/E2E tests through the repo's documented Playwright or harness lane, usually `npm test` when a Node harness exists +5. targeted type/static validation when the changed surface has a typed frontend or equivalent static contract + +If a repo has no frontend/E2E harness, the SOP must state the non-applicability and identify the replacement smoke, unit, or integration lane that catches the same user-facing risk. + +### Problem-First Test Design Rule + +All test scripts, test harnesses, and validation flows must be designed first to reproduce real failures and catch bugs early. + +The purpose of testing is to expose defects, regressions, drift, and broken assumptions before users hit them. Tests must not be designed merely to produce a green validation result, satisfy a checklist, or prove that a happy path still passes. Do not waste validation time on pass-only checks that cannot fail for the bug class under review. + +Every bugfix or high-risk change must start from the question: "Which test would have caught this before release?" If the existing gate missed the bug, update the targeted test or SOP flow so the same class of bug fails deterministically next time. + +### Bugfix/Hotfix Rule (Reproduce -> Pin -> Sweep) + +For bugfix/hotfix work, acceptance evidence must include: + +1. pre-fix reproduction evidence +2. post-fix targeted regression evidence +3. final full-gate evidence + +A green full gate alone is not sufficient bugfix evidence unless the record also shows how the specific failure was reproduced and pinned. + +### Documentation-only Exception + +If all touched files are documentation/planning text only and no code, tests, scripts, config, generated artifacts, dependency manifests, or runtime behavior changed, full test execution is optional. Once executable or runtime-affecting files change, this exception does not apply. + +### Environment Guardrails + +- Keep the Python interpreter consistent across all commands. +- Prefer a project-local virtual environment: `.venv` on Windows and `.venv-wsl` on WSL/Linux when the repo supports dual-OS validation. +- Do not mix global and venv-installed `pre-commit` accidentally. +- Node.js must be 18+ before running frontend/E2E tests. +- On Windows, prefer repo-local `PRE_COMMIT_HOME` to avoid cache lock issues. +- On WSL, if `python` is missing but `python3` exists, create a local shim before running Playwright or harness commands. +- If pre-commit modifies files, review/stage/commit those changes and rerun hooks until clean. + +### Evidence Recording + +Implementation records must include date/time, OS/environment, command log reference, and pass/fail result for each required stage. If a gate is intentionally skipped as non-applicable, record why and name the replacement validation lane. +