From 035290fb47e8b7a837cefb118d667db29a10074a Mon Sep 17 00:00:00 2001 From: rookiestar28 Date: Fri, 12 Jun 2026 20:19:11 +0800 Subject: [PATCH] docs: refresh host alignment documentation --- README.md | 18 ++++++++++-------- docs/asset_api_adoption_decision.md | 12 +++++++++++- docs/frontend_ux_walkthrough.md | 6 +++--- docs/openapi.yaml | 2 +- docs/release/api_contract.md | 20 ++++++++++++++++++-- docs/release/recent_updates.md | 7 ++++--- docs/troubleshooting.md | 3 ++- 7 files changed, 49 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index b10a041..421fd33 100644 --- a/README.md +++ b/README.md @@ -103,13 +103,14 @@ Deployment profiles and hardening references:
-ComfyUI host compatibility, queue recovery, model folders, and asset-output posture refreshed +ComfyUI host compatibility, media outputs, model folders, and prompt attribution refreshed - Refreshed the published compatibility baseline for ComfyUI `822aca19` (`v0.24.0-60-g822aca19`, pyproject `0.24.0`), standalone frontend `1.46.13`, and Desktop `0.9.4` with core `0.22.3` plus embedded frontend `1.43.18`. - Reconciled active prompt state after backend or SSE reconnects so completed prompts are not left in the active queue lane after a host recovery. - Updated sidebar registration to prefer the current ComfyUI sidebar store API and keep the deprecated frontend facade as a compatibility fallback for older hosts. -- Aligned Model Manager and preflight diagnostics with current ComfyUI model folder names such as `text_encoders`, `diffusion_models`, `geometry_estimation`, and `detection`, while retaining legacy aliases such as `clip` and `unet`. -- Kept output previews on the bounded `/history` + `/view` contract, including `asset_hash` and `hash` aliases for hash-backed previews; upstream asset-only identifiers remain explicit fallback states unless a future feature requires direct `/api/assets` use. +- Aligned Model Manager and preflight diagnostics with current ComfyUI model folder names, including newer managed keys such as `gligen`, `latent_upscale_models`, `hypernetworks`, `photomaker`, `model_patches`, `geometry_estimation`, and `detection`, while retaining legacy aliases such as `clip` and `unet`. +- Made output parsing media-aware for current previewable result groups (`images`, `video`, `audio`, `3d`, and bounded `text`) while keeping image callbacks compatible and keeping asset-only identifiers as explicit fallback states instead of silently upgrading to `/api/assets`. +- OpenClaw prompt submissions now include stable `comfy_usage_source` attribution when missing, without overwriting caller-provided attribution or copying prompt/tenant/trace content into that field. - Updated public release/support/troubleshooting docs to match the refreshed host facts and avoid exposing maintainer-only planning paths or machine-local links.
@@ -419,7 +420,7 @@ The OpenClaw sidebar includes these built-in tabs. Some tabs are capability-gate | Tab | What it does | Related docs | | --- | --- | --- | | `Settings` | Health/config/log visibility, provider/model setup, model connectivity checks, and optional localhost key storage. | [Quick Start](#quick-start-minimal), [API Overview](#api-overview), [Troubleshooting](#troubleshooting) | -| `Jobs` | Tracks prompt IDs, consumes deterministic event/task cursor metadata for polling, and shows output previews for recent jobs across classic history refs plus `asset_hash`/`hash`-backed output refs through the same `/view` contract; refs that only expose asset-service identifiers stay explicit as an operator-visible fallback state instead of silently upgrading to `/api/assets`. | [API Overview](#api-overview), [Remote Control (Connector)](#remote-control-connector) | +| `Jobs` | Tracks prompt IDs, consumes deterministic event/task cursor metadata for polling, and shows recent outputs across classic history refs, `asset_hash`/`hash`-backed refs, and current previewable media groups (`images`, `video`, `audio`, `3d`, bounded `text`). File-like refs stay on the `/view` contract; asset-service-only refs stay explicit instead of silently upgrading to `/api/assets`. | [API Overview](#api-overview), [Remote Control (Connector)](#remote-control-connector) | | `Planner` | Uses assist endpoint to generate structured prompt plans (positive/negative/params). | [Configure an LLM key](#1-configure-an-llm-key-for-plannerrefinervision-helpers), [Nodes](#nodes) | | `Refiner` | Refines existing prompts with optional image context and issue/goal input. | [Configure an LLM key](#1-configure-an-llm-key-for-plannerrefinervision-helpers), [Nodes](#nodes) | | `Variants` | Local helper for generating batch variant parameter JSON (seed/range-style sweeps). | [Nodes](#nodes), [Operator UX Features](#operator-ux-features) | @@ -428,7 +429,7 @@ The OpenClaw sidebar includes these built-in tabs. Some tabs are capability-gate | `Explorer` | Inventory/preflight diagnostics and snapshot/checkpoint troubleshooting workflows, including snapshot-first inventory refresh state (`snapshot_ts`, `scan_state`, `stale`, `last_error`) and suppressed inactive-branch findings. | [Operator UX Features](#operator-ux-features), [Troubleshooting](#troubleshooting) | | `Packs` | Dedicated pack lifecycle tab for import/export/delete under admin boundary. | [API Overview](#api-overview) | | `PNG Info` | Inspects saved generation images through drag-and-drop, file picker, or scoped paste, parses A1111 infotext plus ComfyUI `prompt` / `workflow` metadata, shows extracted prompt and generation fields when recoverable, and keeps raw metadata visible for operator inspection. | [API Overview](#api-overview), [Troubleshooting](#troubleshooting) | -| `Model Manager` | Searches model catalog/install records, queues managed downloads, monitors task lifecycle, and imports completed tasks into the managed install root with current ComfyUI folder-key normalization, including `geometry_estimation` and `detection`, plus legacy type aliases. | [API Overview](#api-overview), [Troubleshooting](#troubleshooting) | +| `Model Manager` | Searches model catalog/install records, queues managed downloads, monitors task lifecycle, and imports completed tasks into the managed install root with current ComfyUI folder-key normalization, including `gligen`, `latent_upscale_models`, `hypernetworks`, `photomaker`, `model_patches`, `geometry_estimation`, and `detection`, plus legacy type aliases. | [API Overview](#api-overview), [Troubleshooting](#troubleshooting) | | `Parameter Lab` | Runs bounded sweep/compare experiments, stores history, and replays parameters back into the graph. | [Operator UX Features](#operator-ux-features) | ## Operator UX Features @@ -522,8 +523,9 @@ Key operational notes: - Observability remains token-gated for remote access and redacts provider reasoning-like content plus marked internal maintenance/helper content by default. - Event/model-download polling and preflight inventory are snapshot/cursor-driven contracts; clients should consume `snapshot_ts`, `scan_state`, `stale`, and cursor metadata instead of assuming full-refresh polling. -- Model Manager and preflight consumers should use current ComfyUI folder keys for model types where possible, including `text_encoders`, `diffusion_models`, `geometry_estimation`, and `detection`; compatibility aliases such as `clip`, `unet`, `ckpt`, and plural legacy names are normalized before lookup/import. -- Output/history-facing consumers should keep using the bounded `/history` + `/view` contract; `asset_hash` and `hash` values that map to `blake3:...` preview normally, while refs that only upstream asset services can resolve remain explicit `asset_api_required` compatibility states. +- Model Manager and preflight consumers should use current ComfyUI folder keys for model types where possible, including `text_encoders`, `diffusion_models`, `gligen`, `latent_upscale_models`, `hypernetworks`, `photomaker`, `model_patches`, `geometry_estimation`, and `detection`; compatibility aliases such as `clip`, `unet`, `ckpt`, and plural legacy names are normalized before lookup/import. +- Output/history-facing consumers should keep using the bounded `/history` + `/view` contract; current previewable output groups include `images`, `video`, `audio`, `3d`, and bounded `text`, while refs that only upstream asset services can resolve remain explicit `asset_api_required` compatibility states. +- Queue submissions add stable ComfyUI usage-source attribution (`comfyui-openclaw`) when callers do not provide one; callers that already supply `extra_data.comfy_usage_source` keep ownership of that value. - Connector diagnostics expose redacted token references only, and `/openclaw/connector/extraction-contract` is structural packaging metadata and static SecretRef policy rather than a live installation-health, environment, or token-status feed. ## Advanced Security and Runtime Setup @@ -693,7 +695,7 @@ Quick jumps: - backend not loaded / route 404 startup failures - Operator Doctor usage -- Jobs preview fallback for asset-api-only output refs +- Jobs preview and media fallback for asset-api-only output refs - audit chain verification after restart or rotation - external-tool allowlist, sandbox runtime, interpreter, timeout, or workspace diagnostics - webhook auth not configured diff --git a/docs/asset_api_adoption_decision.md b/docs/asset_api_adoption_decision.md index c62b61a..a81eb51 100644 --- a/docs/asset_api_adoption_decision.md +++ b/docs/asset_api_adoption_decision.md @@ -1,5 +1,12 @@ # ComfyUI Asset API Adoption Decision (2026-04-16) +## 2026-06-12 reconfirmation + +- Current output parsing is media-aware for ComfyUI result groups `images`, `video`, `audio`, `3d`, and bounded `text`. +- File-like media refs still use `/view` when they provide `filename` or hash-backed preview metadata. +- Text output previews are bounded and rendered as text, not HTML. +- Asset-service-only identifiers remain explicit fallback states and still do not trigger automatic direct `/api/assets` fetches. + ## 2026-05-31 reconfirmation - Current host reference evidence shows upstream asset responses may expose `hash` alongside `asset_hash`. @@ -15,6 +22,7 @@ - Current history/output-facing interop already accepts: - classic ComfyUI output refs (`filename`, `subfolder`, `type`) - asset-hash-backed refs that still resolve through `/view?filename=blake3:...` + - media-aware output groups (`images`, `video`, `audio`, `3d`, and bounded `text`) - Current ComfyUI `822aca19` / `v0.24.0-60-g822aca19` / pyproject `0.24.0` reference facts: - `/api/assets*` routes exist, but operational use is feature-gated behind `--enable-assets` - `/features` exposes the `assets` capability flag so hosts can report whether the asset system is enabled @@ -31,7 +39,7 @@ - **No-go for first-class `/api/assets` runtime adoption in phase 2.** - OpenClaw keeps `/history` + `/view` as the supported runtime contract for normal output handling. -- Asset-api-only identifiers are now treated as explicit unsupported contracts rather than implicit fetch targets. +- Asset-api-only identifiers are treated as explicit unsupported contracts rather than implicit fetch targets. ## Rationale @@ -44,6 +52,8 @@ - Preserve current supported refs exactly: - classic refs -> `/view?filename=...&type=...` - asset-hash-backed refs -> `/view?filename=blake3:...` + - file-like media refs -> `/view` fallback/link surfaces when preview metadata is present + - bounded text refs -> escaped text surfaces, not HTML - For refs that expose only asset-service identifiers and are not representable through `/view`: - keep them in normalized output payloads - mark them as `asset_api_required` diff --git a/docs/frontend_ux_walkthrough.md b/docs/frontend_ux_walkthrough.md index 09c9869..3d7a602 100644 --- a/docs/frontend_ux_walkthrough.md +++ b/docs/frontend_ux_walkthrough.md @@ -14,7 +14,7 @@ This document summarizes the current OpenClaw sidebar UI structure and how to ve - Tabs: `web/openclaw_tabs.js` manages tab registration, rendering, and remount safety. - API: `web/openclaw_api.js` provides a normalized fetch wrapper and OpenClaw endpoints (legacy Moltbot endpoints still work). - Host surface: `web/openclaw_host_surface.js` resolves the active frontend host surface and stamps explicit metadata so standalone frontend vs desktop-embedded behavior stays testable. -- Output refs: `web/openclaw_asset_refs.js` normalizes classic history refs and newer asset-backed output refs onto the same bounded `/view` preview contract, while keeping asset-service-only refs explicit as a fallback state instead of silently auto-fetching `/api/assets`. +- Output refs: `web/openclaw_asset_refs.js` normalizes classic history refs, `asset_hash`/`hash` refs, and current previewable media groups (`images`, `video`, `audio`, `3d`, bounded `text`) onto one media-aware contract. File-like refs stay on the bounded `/view` preview path, text refs stay escaped and bounded, and asset-service-only refs remain explicit fallback states instead of silently auto-fetching `/api/assets`. - Styles: `web/openclaw.css` provides shared design tokens and component classes. - Errors and compatibility helpers: `web/openclaw_utils.js` provides `showError()` / `clearError()` plus runtime legacy-class alias helpers used to keep canonical `openclaw-*` markup compatible with existing `moltbot-*` selectors. @@ -24,7 +24,7 @@ Refactor note: - New tab markup should use canonical `openclaw-*` classes; legacy `moltbot-*` aliases are generated centrally at runtime instead of being duplicated in each template. - New host sidebar registration changes should stay in `web/openclaw_sidebar_registration.js` rather than duplicating ComfyUI frontend API detection inside the extension entrypoint. - Host-sensitive behaviors should consume the shared host-surface helper rather than inferring desktop vs standalone frontend from ad-hoc globals. -- Output preview flows should consume the shared asset-ref normalizer rather than assembling `/view` URLs independently in each tab or silently widening runtime behavior to direct `/api/assets` fetches. +- Output preview flows should consume the shared asset-ref normalizer rather than assembling `/view` URLs independently in each tab, treating non-image media as broken images, or silently widening runtime behavior to direct `/api/assets` fetches. - Explorer/preflight consumers should treat inventory diagnostics as snapshot-first and surface `snapshot_ts`, `scan_state`, `stale`, and `last_error` instead of blocking the UI on full rescans. - Explorer/preflight rendering should keep actionable missing-node/model failures separate from suppressed inactive-branch findings returned by the backend. @@ -83,7 +83,7 @@ If `assist_streaming` is unavailable or the stream transport degrades, Planner/R 3. Confirm the sidebar host-surface metadata resolves correctly for the current environment instead of defaulting silently. 4. Planner: click **Plan Generation** with minimal input and confirm either live preview/stage updates appear (when streaming is supported) or a readable fallback result/error appears. 5. Refiner: click **Refine Prompts** (with or without image) and confirm either live preview/stage updates appear (when streaming is supported) or a readable fallback result/error appears. -6. Jobs: verify output previews still resolve for both classic history refs and any asset-backed refs surfaced by callback/history payloads, that asset-service-only refs stay explicit as a bounded fallback state, and that repeated polls do not duplicate rows after reconnect/resume. +6. Jobs: verify output previews still resolve for classic history refs, hash-backed refs, and supported media-aware refs (`images`, `video`, `audio`, `3d`, bounded `text`); asset-service-only refs should stay explicit as a bounded fallback state, and repeated polls should not duplicate rows after reconnect/resume. 7. Explorer: verify preflight inventory can show `refreshing` / `stale` / `error` state without freezing the tab while deep scan work continues, and verify inactive-branch suppressed findings render separately from actionable failures. 8. Library/Approvals: if backend endpoints are not enabled, confirm the UI shows a clear error state (no crashes). 9. If you simulate/fake a stream failure in dev tools, confirm Planner/Refiner retry through the classic non-stream path without duplicate submits or broken loading state. diff --git a/docs/openapi.yaml b/docs/openapi.yaml index ec05bd6..94f54e6 100644 --- a/docs/openapi.yaml +++ b/docs/openapi.yaml @@ -1,7 +1,7 @@ openapi: "3.0.3" info: title: "ComfyUI-OpenClaw API" - version: "1.0.11" + version: "1.0.12" description: "Generated from docs/release/api_contract.md." servers: - url: "/openclaw" diff --git a/docs/release/api_contract.md b/docs/release/api_contract.md index bbf9b61..634113d 100644 --- a/docs/release/api_contract.md +++ b/docs/release/api_contract.md @@ -1,8 +1,8 @@ # OpenClaw API Contract (v1) > **Status**: normative -> **Version**: 1.0.11 -> **Date**: 2026-06-04 +> **Version**: 1.0.12 +> **Date**: 2026-06-12 This document defines the public API contract for OpenClaw. It serves as the authoritative baseline for client compatibility and breaking change policies. @@ -75,6 +75,15 @@ Preflight workflow diagnostics contract: - suppressed findings represent muted or bypassed root nodes or subgraph branches when the submitted workflow shape provides enough frontend ancestry metadata - clients SHOULD display suppressed findings as informational context rather than blocking workflow readiness +History and output-ref contract: + +- history/output consumers SHOULD treat the normalized output-ref contract as media-aware +- current previewable output groups are `images`, `video`, `audio`, `3d`, and bounded `text` +- file-like refs that can be represented through `/view` remain on the bounded `/history` + `/view` preview path +- refs with `asset_hash` or `hash` values that map to `blake3:...` preview through `/view?filename=blake3:...` +- refs that only expose upstream asset-service identifiers remain explicit `asset_api_required` states; clients MUST NOT silently infer direct `/api/assets` fetching from that marker +- legacy callback/image-only consumers may continue using image-only extraction paths; non-image media refs should be rendered as explicit fallback/link/text surfaces unless the client implements a safe media-specific renderer + ### 1.2 Webhooks & Triggers **Auth**: Requires configured webhook secret or Admin Token. @@ -86,6 +95,13 @@ Preflight workflow diagnostics contract: | `POST` | `/webhook/validate` | `/moltbot/webhook/validate` | Webhook Secret | Dry-run validation of webhook payload. | | `POST` | `/triggers/fire` | `/moltbot/triggers/fire` | Admin | Fire an ad-hoc workflow trigger from external system. | +ComfyUI prompt submission interoperability: + +- OpenClaw-generated ComfyUI `/prompt` payloads include `extra_data.comfy_usage_source = "comfyui-openclaw"` when the caller has not supplied a value +- caller-provided `extra_data.comfy_usage_source` is preserved +- attribution is a stable product identifier and MUST NOT include prompt text, tenant ids, trace ids, URLs, tokens, or secrets +- existing `extra_data.openclaw` and legacy `extra_data.moltbot` metadata remain caller-owned except for OpenClaw tenant metadata insertion under `extra_data.openclaw.tenant_id` + ### 1.3 Assist, LLM & Chat **Assist Base Path**: `/openclaw/assist/` diff --git a/docs/release/recent_updates.md b/docs/release/recent_updates.md index 7cd1416..19be1e3 100644 --- a/docs/release/recent_updates.md +++ b/docs/release/recent_updates.md @@ -19,13 +19,14 @@ Newest entries appear first.
-ComfyUI host compatibility, queue recovery, model folders, and asset-output posture refreshed +ComfyUI host compatibility, media outputs, model folders, and prompt attribution refreshed - Refreshed the published compatibility baseline for ComfyUI `822aca19` (`v0.24.0-60-g822aca19`, pyproject `0.24.0`), standalone frontend `1.46.13`, and Desktop `0.9.4` with core `0.22.3` plus embedded frontend `1.43.18`. - Reconciled active prompt state after backend or SSE reconnects so completed prompts are not left in the active queue lane after a host recovery. - Updated sidebar registration to prefer the current ComfyUI sidebar store API and keep the deprecated frontend facade as a compatibility fallback for older hosts. -- Aligned Model Manager and preflight diagnostics with current ComfyUI model folder names such as `text_encoders`, `diffusion_models`, `geometry_estimation`, and `detection`, while retaining legacy aliases such as `clip` and `unet`. -- Kept output previews on the bounded `/history` + `/view` contract, including `asset_hash` and `hash` aliases for hash-backed previews; upstream asset-only identifiers remain explicit fallback states unless a future feature requires direct `/api/assets` use. +- Aligned Model Manager and preflight diagnostics with current ComfyUI model folder names, including newer managed keys such as `gligen`, `latent_upscale_models`, `hypernetworks`, `photomaker`, `model_patches`, `geometry_estimation`, and `detection`, while retaining legacy aliases such as `clip` and `unet`. +- Made output parsing media-aware for current previewable result groups (`images`, `video`, `audio`, `3d`, and bounded `text`) while keeping image callbacks compatible and keeping asset-only identifiers as explicit fallback states instead of silently upgrading to `/api/assets`. +- OpenClaw prompt submissions now include stable `comfy_usage_source` attribution when missing, without overwriting caller-provided attribution or copying prompt/tenant/trace content into that field.
diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 69b7f04..7f4b334 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -91,7 +91,7 @@ Notes: - Runtime cache and sandbox scratch paths are generated state, not package resources. - OpenClaw does not automatically repair, migrate, or delete runtime dependency caches. -## Jobs preview shows an explicit asset fallback state instead of an image preview +## Jobs preview shows an explicit media or asset fallback state Current OpenClaw builds keep `/history` + `/view` as the supported runtime preview contract for job results. @@ -101,6 +101,7 @@ What this means: - `asset_api_required` is a bounded compatibility state, not a generic parser failure. - Classic history refs and hash-backed refs exposed as `asset_hash` or `hash` that still map onto `/view` should continue to preview normally. +- Current media-aware outputs can include `images`, `video`, `audio`, `3d`, and bounded `text`; images render as thumbnails, text renders as escaped bounded text, and other file-like media may appear as explicit fallback/link tiles instead of image elements. - If an operator workflow starts depending on direct asset-service identifiers, treat that as a contract gap and review [`docs/asset_api_adoption_decision.md`](asset_api_adoption_decision.md) before widening the runtime dependency. ## Verify audit-chain continuity after restart or rotation