mirror of
https://github.com/ValueCell-ai/ClawX.git
synced 2026-08-16 18:02:05 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1f67454c15 | ||
|
|
e811bb8c0a | ||
|
|
767b6f11ed | ||
|
|
2107f8a1fe | ||
|
|
51bc92336d | ||
|
|
32bf17e0a1 | ||
|
|
546f727951 | ||
|
|
0059db5fbd | ||
|
|
f281f7bd22 | ||
|
|
2198f7340a | ||
|
|
0c50ddd7cc | ||
|
|
0d79d608bd | ||
|
|
a1365c41f7 | ||
|
|
2040180ec4 | ||
|
|
4018cbece3 | ||
|
|
05581314e1 | ||
|
|
cfc5bca9c9 | ||
|
|
33a58f621b | ||
|
|
b60e8c6f81 | ||
|
|
6bbb050750 | ||
|
|
88cfc31fa7 | ||
|
|
bd1aac8e94 | ||
|
|
0116f6e968 |
@@ -88,6 +88,16 @@ AIエージェントの構築にコマンドラインの習得は不要である
|
||||
|
||||
> 機能の詳細は [docs/ja-JP/features.md](docs/ja-JP/features.md) を参照してください。
|
||||
|
||||
### リアルタイム Talk
|
||||
|
||||
OpenClaw のカタログが設定済みで利用可能なリアルタイムプロバイダーを返すとき、チャットからリアルタイム Talk を開始できます。Talk は **Gateway Relay** のみを使用します。Electron Main が Gateway プロトコルと RPC ルーティングを所有し、Renderer が Gateway やプロバイダーへ直接接続することはありません。アクティブなリレーはグローバルで一つだけです。
|
||||
|
||||
直接プロバイダーの音声と文字起こしバブルは一時的なもので、Talk の停止、セッション変更、チャット再読み込み、アプリ再起動で消去されます。Agent 相談は選択中の OpenClaw セッションを使用し、その永続的な結果は通常の **ACP** 履歴リプレイでのみ表示されます。ClawX は別個の Talk 文字起こしストアや合成 ACP エントリを作成しません。
|
||||
|
||||
Talk はカタログの利用可否を表示・確認してから開始します。プロバイダー、モデル、話者の音声、利用可否は **設定 → Talk** で設定します。プロバイダー認証情報は Gateway が所有し、Main が OpenClaw 設定トランザクションを通じて許可された `talk.realtime` 設定を適用します。Talk 設定にはシークレット、トランスポート選択、VAD コントロールは表示されません。
|
||||
|
||||
手動の macOS 検証には、設定済みのリアルタイムプロバイダーとマイク権限が必要です。直接プロバイダー応答、Agent 相談応答、バージイン、停止、Gateway 再接続はその設定済み環境で確認する必要があります。自動テストはこれらのプロバイダー依存ケースを検証しません。
|
||||
|
||||
### 主なユースケース
|
||||
|
||||
- **🤖 パーソナルAIアシスタント**:質問への回答、メールの下書き、ドキュメントの要約、日常タスクの支援を行う汎用AIエージェントを、クリーンなデスクトップインターフェースから設定できます。
|
||||
|
||||
@@ -88,6 +88,16 @@ Building AI agents shouldn't require mastering the command line. ClawX was desig
|
||||
|
||||
> For full feature details, see [docs/en-US/features.md](docs/en-US/features.md).
|
||||
|
||||
### Realtime Talk
|
||||
|
||||
Chat can start a realtime Talk session when the OpenClaw catalog reports a configured, ready realtime provider. Talk uses **Gateway Relay only**: Electron Main owns Gateway protocol and RPC routing, while the renderer has no direct Gateway or provider connection. Only one relay is active globally.
|
||||
|
||||
Direct-provider audio and transcript bubbles are transient and clear when Talk stops, the session changes, Chat reloads, or the app restarts. An Agent consult instead uses the selected OpenClaw session; its durable result appears through normal **ACP** history replay. ClawX does not create a separate Talk transcript store or synthetic ACP entries.
|
||||
|
||||
Talk displays and checks catalog readiness before it can start. Configure a provider, model, speaker voice, and readiness in **Settings -> Talk**. Provider credentials remain Gateway-owned; Main applies the allowed `talk.realtime` settings through the OpenClaw configuration transaction. The Talk settings surface does not expose secrets, transport selection, or VAD controls.
|
||||
|
||||
Manual macOS validation still requires a configured realtime provider and microphone permission. Direct provider response, Agent consult response, barge-in, stop, and Gateway reconnect must be checked in that configured environment; automated coverage does not validate those provider-dependent cases.
|
||||
|
||||
### Typical Use Cases
|
||||
|
||||
- **🤖 Personal AI Assistant**: Configure a general-purpose AI agent to answer questions, draft emails, summarize documents, and help with everyday tasks from a clean desktop interface.
|
||||
|
||||
@@ -87,6 +87,16 @@ ClawX 预置了最佳实践的模型供应商配置,原生支持 Windows 平
|
||||
|
||||
> 对于功能细节的完整说明,请参阅 [docs/zh-CN/features.md](docs/zh-CN/features.md)。
|
||||
|
||||
### 实时 Talk
|
||||
|
||||
当 OpenClaw 目录返回已配置且可用的实时提供商时,聊天页面可以启动实时 Talk。Talk 只使用 **Gateway Relay**:Electron Main 负责 Gateway 协议和 RPC 路由,渲染层不会直接连接 Gateway 或提供商。全局同时只能有一个中继会话。
|
||||
|
||||
直连提供商的音频和转写气泡是临时内容,会在停止 Talk、切换会话、重新加载聊天或重启应用时清除。Agent 咨询则使用选中的 OpenClaw 会话;其持久结果只会通过正常的 **ACP** 历史回放显示。ClawX 不会创建独立的 Talk 转写存储,也不会生成 ACP 条目。
|
||||
|
||||
Talk 会显示并检查目录就绪状态后才可启动。可在 **设置 → Talk** 中配置提供商、模型、语音音色和就绪状态。提供商凭据仍由 Gateway 持有;Main 通过 OpenClaw 配置事务应用允许的 `talk.realtime` 设置。Talk 设置页面不显示密钥、传输方式选择或 VAD 控件。
|
||||
|
||||
手动 macOS 验证仍需要已配置的实时提供商和麦克风权限。直连提供商回复、Agent 咨询回复、打断、停止和 Gateway 重连需要在该已配置环境中检查;自动化覆盖不验证这些依赖提供商的场景。
|
||||
|
||||
### 典型使用场景
|
||||
|
||||
- **🤖 个人 AI 助手**:配置一个通用 AI 智能体,可以回答问题、撰写邮件、总结文档并协助处理日常任务——全部通过简洁的桌面界面完成。
|
||||
|
||||
@@ -0,0 +1,269 @@
|
||||
# Realtime Talk Implementation Plan
|
||||
|
||||
> **For agentic workers:** Use `subagent-driven-development` to implement this plan task-by-task. Use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add OpenClaw Gateway Relay realtime Talk to ClawX Chat while retaining OpenClaw as the only durable source of truth.
|
||||
|
||||
**Architecture:** Renderer owns local microphone capture, playback, and temporary direct-provider transcript display. Electron Main owns every Gateway RPC and `talk.event` route behind a typed Talk host service. ACP remains the only renderer of durable history: direct realtime transcripts are transient, while Agent consults use OpenClaw's existing `chat.send` path and are displayed after ACP reload.
|
||||
|
||||
**Tech Stack:** Electron 40, React 19, TypeScript, Zustand, Web Audio API, OpenClaw Gateway WebSocket, Vitest, Playwright, react-i18next.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Use only `gateway-relay`; do not implement WebRTC, provider WebSocket, video, dictation, recording, or transport selection.
|
||||
- OpenClaw is the only durable source of truth. Do not add a ClawX-owned Talk transcript database, JSON ledger, sidecar history, cache, or direct session transcript file writes.
|
||||
- Direct realtime provider transcripts are in-memory and disappear when Talk ends, Chat reloads, the session changes, or the app restarts.
|
||||
- Agent consults must use the existing OpenClaw `talk.client.toolCall` and `chat.send` flow, never ACP `session/prompt`.
|
||||
- Renderer must use `src/lib/host-api.ts` and `src/lib/host-events.ts`; no direct Renderer IPC, Gateway HTTP, or Gateway WebSocket access.
|
||||
- Start Talk only for the selected non-heartbeat session key. Never fall back to `DEFAULT_SESSION_KEY`.
|
||||
- Allow one active relay globally. Lock the active Chat composer and restore its draft on all terminal paths.
|
||||
- User-facing strings must be present in `en`, `zh`, `ja`, and `ru`. Use the existing design tokens and accessibility conventions.
|
||||
- Update `README.md`, `README.zh-CN.md`, `README.ja-JP.md`, and the Talk harness scenario/rule/task in the same work.
|
||||
- Before implementation review, run `pnpm harness validate --spec harness/specs/tasks/add-realtime-talk.md`; run `pnpm harness run --spec harness/specs/tasks/add-realtime-talk.md --dry-run` when validating the selected flow.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Preflight
|
||||
|
||||
- [ ] Create and validate `harness/specs/tasks/add-realtime-talk.md` before Task 1, because this feature changes Renderer/Main/Gateway communication.
|
||||
- [ ] Run `pnpm harness validate --spec harness/specs/tasks/add-realtime-talk.md` and `pnpm harness run --spec harness/specs/tasks/add-realtime-talk.md --dry-run` before production-code edits.
|
||||
- [ ] Keep the scenario, authority rule, reference document, locale coverage, and README updates in Task 5.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Typed Gateway Talk Boundary
|
||||
|
||||
**Files:**
|
||||
- Create: `shared/talk/types.ts`
|
||||
- Create: `electron/services/talk-api.ts`
|
||||
- Create: `tests/unit/talk-api.test.ts`
|
||||
- Modify: `shared/host-api/contract.ts`
|
||||
- Modify: `shared/host-events/contract.ts`
|
||||
- Modify: `src/lib/host-api.ts`
|
||||
- Modify: `src/lib/host-events.ts`
|
||||
- Modify: `electron/main/ipc-handlers.ts`
|
||||
- Modify: `electron/gateway/event-dispatch.ts`
|
||||
- Modify: `electron/gateway/manager.ts`
|
||||
- Modify: `electron/main/index.ts`
|
||||
- Modify: `tests/unit/host-api-facade.test.ts`
|
||||
- Modify: `tests/unit/host-events.test.ts`
|
||||
- Modify: `tests/unit/gateway-event-dispatch.test.ts`
|
||||
- Modify: `tests/unit/host-services.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `GatewayManager.rpc`, `GatewayManager` notification dispatch, existing typed Host API registry, and OpenClaw `talk.catalog`, `talk.session.*`, `talk.client.toolCall`, and `talk.event` protocol contracts.
|
||||
- Produces: `hostApi.talk`, `hostEvents.onTalkEvent`, `TalkCatalog`, `TalkRelaySession`, `TalkRelayEvent`, and validated Talk request payload types used by Renderer tasks.
|
||||
|
||||
- [ ] **Step 1: Write failing Main-boundary tests**
|
||||
- Add `tests/unit/talk-api.test.ts` cases proving every Talk action calls only the expected Gateway RPC with valid normalized parameters and configured request timeout.
|
||||
- Extend event-dispatch tests to prove `talk.event` becomes a dedicated `talk:event` manager event instead of a generic notification.
|
||||
- Extend Host API and Host Event contract tests to prove Talk actions and events are exposed through typed facades only.
|
||||
- Test that malformed relay ids, empty session keys, invalid base64 audio, invalid marks, and invalid tool call ids are rejected before `GatewayManager.rpc`.
|
||||
|
||||
- [ ] **Step 2: Run the focused tests and verify the expected failures**
|
||||
- Run `pnpm exec vitest run tests/unit/talk-api.test.ts tests/unit/gateway-event-dispatch.test.ts tests/unit/host-api-facade.test.ts tests/unit/host-events.test.ts tests/unit/host-services.test.ts`.
|
||||
- Expected result before implementation: the new Talk tests fail because no Talk service, contract entries, or event route exists.
|
||||
|
||||
- [ ] **Step 3: Implement the typed Talk host service**
|
||||
- Define discriminated result/event types in `shared/talk/types.ts` for catalog readiness, relay creation, PCM16 audio contract, transcript/audio/clear/mark/tool-call/tool-result/error/close events, and terminal reasons.
|
||||
- Add a `talk` module to `HostApiContract`, expose matching `hostApi.talk.*` wrappers, and register `createTalkApi(gatewayManager)` with `HostApiRegistry`.
|
||||
- Implement `talk-api.ts` as the only Main adapter for `talk.catalog`, `talk.session.create`, `talk.session.appendAudio`, `talk.session.cancelOutput`, `talk.session.submitToolResult`, `talk.session.acknowledgeMark`, `talk.session.close`, and `talk.client.toolCall`.
|
||||
- Enforce `mode: 'realtime'`, `transport: 'gateway-relay'`, and `brain: 'agent-consult'` in `startRelay`; do not accept a Renderer transport override.
|
||||
- Teach `event-dispatch.ts`, `GatewayManagerEvents`, and `main/index.ts` to forward `talk.event` through a typed `talk:event` event. Register a corresponding host event and `hostEvents.onTalkEvent` subscription.
|
||||
- Keep generic `hostApi.gateway.rpc` unchanged, but do not consume it from new Renderer Talk code.
|
||||
|
||||
- [ ] **Step 4: Run focused and boundary regressions**
|
||||
- Run the Step 2 command again and expect all tests to pass.
|
||||
- Run `pnpm exec vitest run tests/unit/gateway-events.test.ts tests/unit/gateway-event-dispatch.test.ts tests/unit/host-api-facade.test.ts tests/unit/host-events.test.ts`.
|
||||
- Run `pnpm run typecheck`.
|
||||
|
||||
- [ ] **Step 5: Commit the task**
|
||||
- Commit with `feat(talk): add typed gateway relay boundary`.
|
||||
|
||||
### Task 2: Renderer Talk Controller and Audio Transport
|
||||
|
||||
**Files:**
|
||||
- Create: `src/lib/talk/audio.ts`
|
||||
- Create: `src/lib/talk/audio-worklet.ts`
|
||||
- Create: `src/lib/talk/realtime-talk-controller.ts`
|
||||
- Create: `src/stores/realtime-talk.ts`
|
||||
- Create: `tests/unit/talk-audio.test.ts`
|
||||
- Create: `tests/unit/realtime-talk-controller.test.ts`
|
||||
- Create: `tests/unit/realtime-talk-store.test.ts`
|
||||
- Modify: `src/stores/acp-chat-session.ts`
|
||||
- Modify: `src/lib/host-events.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `hostApi.talk`, `hostEvents.onTalkEvent`, selected ACP session state, and the Task 1 `TalkRelaySession` and `TalkRelayEvent` types.
|
||||
- Produces: `useRealtimeTalkStore`, a single-active-relay controller, input-level state, transient live transcript entries, and explicit terminal/error state consumed by Chat UI.
|
||||
|
||||
- [ ] **Step 1: Write failing controller and audio tests**
|
||||
- Add codec tests for Float32-to-PCM16 conversion, base64 conversion, decoded-byte length checks, and PCM16 playback decoding.
|
||||
- Add controller tests for start, stop, terminal event, session switch, unmount, Gateway disconnect, microphone permission denial, unsupported audio contract, and stale relay event rejection.
|
||||
- Add tests proving direct final and partial transcripts remain only in in-memory live state and are cleared when Talk stops or ACP reloads.
|
||||
- Add tests proving an ACP prompt in flight blocks Talk startup and Talk active state blocks a new ACP prompt.
|
||||
- Add tests proving a consult completion invokes the existing ACP session reload action instead of appending synthetic messages to the ACP timeline.
|
||||
|
||||
- [ ] **Step 2: Run the focused tests and verify the expected failures**
|
||||
- Run `pnpm exec vitest run tests/unit/talk-audio.test.ts tests/unit/realtime-talk-controller.test.ts tests/unit/realtime-talk-store.test.ts`.
|
||||
- Expected result before implementation: test imports fail because the audio/controller/store modules do not exist.
|
||||
|
||||
- [ ] **Step 3: Implement the renderer-owned live transport**
|
||||
- Implement `audio.ts` with PCM16 conversion, a bounded playback queue, input level measurement, and no persisted audio or transcript state.
|
||||
- Implement `audio-worklet.ts` as the mic sample bridge; use `getUserMedia` only in Renderer and terminate every track during controller cleanup.
|
||||
- Implement `realtime-talk-controller.ts` to create exactly one relay from the selected session key, subscribe to typed Talk events, enforce relay-id ownership, and serialize bounded `appendAudio` requests.
|
||||
- Map Talk events to explicit UI states: connecting, listening, thinking, speaking, disconnected, and error.
|
||||
- Implement barge-in by measuring local mic input during playback and calling `hostApi.talk.cancelOutput` once per active output turn.
|
||||
- Implement mark acknowledgement after queued output has played.
|
||||
- For direct transcripts, keep only ordered live entries in Zustand; clear them on all terminal/reset paths.
|
||||
- For Agent consult tool calls, call `hostApi.talk.startAgentConsult`, submit progress/terminal results via the controller, and request the existing ACP store to reload the same session when the consult reaches terminal completion.
|
||||
- Do not add any timeline persistence, synthetic replay, or session transcript write path.
|
||||
|
||||
- [ ] **Step 4: Run focused and ACP regressions**
|
||||
- Run the Step 2 command again and expect all tests to pass.
|
||||
- Run `pnpm exec vitest run tests/unit/acp-chat-store.test.ts tests/unit/chat-acp-inline-timeline.test.tsx tests/unit/chat-history-reply-while-sending.test.tsx`.
|
||||
- Run `pnpm run typecheck`.
|
||||
|
||||
- [ ] **Step 5: Commit the task**
|
||||
- Commit with `feat(talk): add renderer relay controller`.
|
||||
|
||||
### Task 3: Inline Chat Talk Experience
|
||||
|
||||
**Files:**
|
||||
- Create: `src/pages/Chat/LiveTalkTranscript.tsx`
|
||||
- Create: `tests/unit/live-talk-transcript.test.tsx`
|
||||
- Modify: `src/pages/Chat/ChatInput.tsx`
|
||||
- Modify: `src/pages/Chat/index.tsx`
|
||||
- Modify: `src/pages/Chat/ChatToolbar.tsx`
|
||||
- Modify: `tests/unit/chat-input.test.tsx`
|
||||
- Modify: `tests/unit/chat-acp-page.test.tsx`
|
||||
- Modify: `tests/unit/chat-acp-inline-timeline.test.tsx`
|
||||
- Modify: `tests/e2e/chat-acp-inline-timeline.spec.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `useRealtimeTalkStore`, current session key, ACP sending state, and Talk readiness from Task 1.
|
||||
- Produces: accessible Chat controls and transient live transcript UI without changing ACP timeline authority.
|
||||
|
||||
- [ ] **Step 1: Write failing Chat UI tests**
|
||||
- Add component tests for microphone start/stop labels, unavailable tooltip, live status announcement, input-level feedback, transient user/assistant bubbles, and microphone source marker.
|
||||
- Add Chat page tests proving composer text, attachment, model, and agent-target controls lock while Talk is active and the previous draft returns after stop.
|
||||
- Add tests proving the Live Talk area clears on session change and ACP reload, while consult-generated messages arrive only through the existing ACP timeline.
|
||||
- Add an Electron E2E case that stubs typed Talk host calls/events, starts Talk, renders direct transcript bubbles, stops Talk, verifies composer recovery, and verifies the direct bubbles disappear after a Chat reload.
|
||||
|
||||
- [ ] **Step 2: Run the focused tests and verify the expected failures**
|
||||
- Run `pnpm exec vitest run tests/unit/live-talk-transcript.test.tsx tests/unit/chat-input.test.tsx tests/unit/chat-acp-page.test.tsx tests/unit/chat-acp-inline-timeline.test.tsx`.
|
||||
- Run `pnpm exec playwright test tests/e2e/chat-acp-inline-timeline.spec.ts`.
|
||||
- Expected result before implementation: the new Talk controls and test ids are absent.
|
||||
|
||||
- [ ] **Step 3: Implement the inline experience**
|
||||
- Add the microphone button to `ChatInput` beside existing send controls and preserve current responsive behavior.
|
||||
- Render `LiveTalkTranscript` immediately above the composer, not as ACP timeline data. Use the normal user/assistant bubble visual language with a microphone marker only for live direct transcripts.
|
||||
- Use `aria-live` status updates for non-audio state and ensure the start/stop control is keyboard-operable.
|
||||
- Gate start on Gateway readiness, a non-heartbeat selected session, no ACP prompt, and no global active relay. Route configuration failures to Settings > Talk.
|
||||
- Ensure Chat page cleanup stops Talk on session switch/unmount and scrolls new transient entries into view without disturbing ACP item ordering.
|
||||
- Keep `AcpTimeline` unchanged as the durable text/tool/permission renderer; do not append Talk entries to its snapshot.
|
||||
|
||||
- [ ] **Step 4: Run focused and full Chat regressions**
|
||||
- Re-run the Step 2 commands and expect all tests to pass.
|
||||
- Run `pnpm exec vitest run tests/unit/chat-input.test.tsx tests/unit/chat-acp-page.test.tsx tests/unit/chat-acp-inline-timeline.test.tsx tests/unit/chat-question-directory.test.tsx`.
|
||||
- Run `pnpm exec playwright test tests/e2e/chat-acp-inline-timeline.spec.ts tests/e2e/chat-run-state-events.spec.ts`.
|
||||
|
||||
- [ ] **Step 5: Commit the task**
|
||||
- Commit with `feat(talk): add inline chat controls`.
|
||||
|
||||
### Task 4: Talk Settings and Gateway Config Delivery
|
||||
|
||||
**Files:**
|
||||
- Create: `src/components/settings/TalkSettings.tsx`
|
||||
- Create: `tests/unit/talk-settings.test.tsx`
|
||||
- Modify: `electron/services/talk-api.ts`
|
||||
- Modify: `shared/host-api/contract.ts`
|
||||
- Modify: `src/lib/host-api.ts`
|
||||
- Modify: `src/pages/Settings/index.tsx`
|
||||
- Modify: `tests/unit/talk-api.test.ts`
|
||||
- Modify: `tests/unit/host-services.test.ts`
|
||||
- Modify: `tests/e2e/chat-model-picker.spec.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `talk.catalog`, Main-owned config delivery mutation APIs, and Talk config request types from Task 1.
|
||||
- Produces: provider/model/speaker-voice selection controls that update OpenClaw `talk.realtime` configuration without exposing secrets or transport selection.
|
||||
|
||||
- [ ] **Step 1: Write failing Settings and config delivery tests**
|
||||
- Add tests proving catalog provider/model/voice options are shown only from Gateway data and unavailable providers cannot be selected.
|
||||
- Add tests proving writes alter only `talk.realtime.provider`, `talk.realtime.model`, and `talk.realtime.speakerVoice` through the existing config mutation transaction.
|
||||
- Add tests proving no API key, raw provider config, VAD value, or transport selector is rendered in basic Settings.
|
||||
- Add an E2E host-invoke mock case for selecting a provider/model/voice and observing readiness state.
|
||||
|
||||
- [ ] **Step 2: Run the focused tests and verify the expected failures**
|
||||
- Run `pnpm exec vitest run tests/unit/talk-settings.test.tsx tests/unit/talk-api.test.ts tests/unit/host-services.test.ts`.
|
||||
- Expected result before implementation: Talk Settings and config mutation action are unavailable.
|
||||
|
||||
- [ ] **Step 3: Implement Settings and config mutation**
|
||||
- Extend the Talk Main service with a narrow config update action that delegates to `mutateOpenClawConfig`; preserve the existing `config.get`/`config.set` base-hash and persisted-commit recovery behavior.
|
||||
- Build `TalkSettings` with localized provider, model, voice, readiness, unavailable reason, and Advanced Developer guidance.
|
||||
- Render `TalkSettings` from the existing Settings page using the established surface, form, loading, and toast patterns.
|
||||
- Refresh catalog state after a successful config write and keep runtime transport fixed to Gateway Relay in all Talk start calls.
|
||||
|
||||
- [ ] **Step 4: Run focused config and Settings regressions**
|
||||
- Re-run the Step 2 command and expect all tests to pass.
|
||||
- Run `pnpm exec vitest run tests/unit/host-services.test.ts tests/unit/gateway-config-delivery.test.ts tests/unit/settings-store.test.ts`.
|
||||
- Run `pnpm exec playwright test tests/e2e/chat-model-picker.spec.ts`.
|
||||
- Run `pnpm run typecheck`.
|
||||
|
||||
- [ ] **Step 5: Commit the task**
|
||||
- Commit with `feat(talk): add realtime talk settings`.
|
||||
|
||||
### Task 5: Localization, Harness, Documentation, and Release Validation
|
||||
|
||||
**Files:**
|
||||
- Create: `harness/specs/scenarios/realtime-talk.md`
|
||||
- Create: `harness/specs/rules/realtime-talk-openclaw-authority.md`
|
||||
- Create: `harness/specs/tasks/add-realtime-talk.md`
|
||||
- Create: `harness/reference/realtime-talk.md`
|
||||
- Modify: `shared/i18n/locales/en/chat.json`
|
||||
- Modify: `shared/i18n/locales/zh/chat.json`
|
||||
- Modify: `shared/i18n/locales/ja/chat.json`
|
||||
- Modify: `shared/i18n/locales/ru/chat.json`
|
||||
- Modify: `shared/i18n/locales/en/settings.json`
|
||||
- Modify: `shared/i18n/locales/zh/settings.json`
|
||||
- Modify: `shared/i18n/locales/ja/settings.json`
|
||||
- Modify: `shared/i18n/locales/ru/settings.json`
|
||||
- Modify: `README.md`
|
||||
- Modify: `README.zh-CN.md`
|
||||
- Modify: `README.ja-JP.md`
|
||||
- Modify: `tests/unit/harness-specs.test.ts`
|
||||
- Modify: `tests/unit/e2e-parallel-policy.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: completed Talk host APIs, renderer controller, Chat UI, and Settings behavior from Tasks 1-4.
|
||||
- Produces: complete locale coverage, executable harness guardrails, durable architecture reference, user documentation, and release-ready validation evidence.
|
||||
|
||||
- [ ] **Step 1: Write the harness specs and failing structural tests**
|
||||
- Define the scenario ownership for Talk Renderer/Main/Gateway paths and require `gateway-backend-communication`, `renderer-main-boundary`, `host-events-fallback-policy`, `openclaw-config-delivery`, `acp-chat-state-and-history`, `ui-i18n-design-tokens`, `e2e-parallel-isolation`, `comms-regression`, and `docs-sync`.
|
||||
- Define a rule forbidding Talk transcript persistence outside OpenClaw, direct Renderer Gateway access, direct transcript-file writes, and synthetic ACP history for direct realtime responses.
|
||||
- Define the task spec's user behavior, validation profile, required tests, and acceptance criteria.
|
||||
- Extend harness structural tests to validate the new spec references and required locale/documentation paths.
|
||||
|
||||
- [ ] **Step 2: Run harness structural checks and verify expected failures**
|
||||
- Run `pnpm exec vitest run tests/unit/harness-specs.test.ts tests/unit/harness-git.test.ts`.
|
||||
- Expected result before implementation: new harness references or required paths are absent.
|
||||
|
||||
- [ ] **Step 3: Finish localization and documentation**
|
||||
- Add identical key coverage for Talk controls, statuses, permission errors, provider readiness, Settings labels, and retry messages in all four supported languages.
|
||||
- Update all three READMEs with Gateway Relay Talk availability, the direct-versus-consult history distinction, macOS validation status, and the fact that provider credentials remain Gateway-owned.
|
||||
- Add `harness/reference/realtime-talk.md` documenting session ownership, relay lifecycle, typed host event boundary, ACP authority, and the explicit no-local-transcript-storage rule.
|
||||
|
||||
- [ ] **Step 4: Run full validation**
|
||||
- Run `pnpm harness validate --spec harness/specs/tasks/add-realtime-talk.md`.
|
||||
- Run `pnpm harness run --spec harness/specs/tasks/add-realtime-talk.md --dry-run`.
|
||||
- Run `pnpm run harness:ci`.
|
||||
- Run `pnpm run lint:check`.
|
||||
- Run `pnpm run typecheck`.
|
||||
- Run `pnpm test`.
|
||||
- Run `pnpm run build:vite`.
|
||||
- Run `pnpm exec playwright test tests/e2e/chat-acp-inline-timeline.spec.ts tests/e2e/chat-model-picker.spec.ts`.
|
||||
- Run `pnpm run comms:replay` and `pnpm run comms:compare` because this changes Gateway event and RPC communication paths.
|
||||
- Manually validate macOS microphone permission, direct provider response, Agent consult response, barge-in, stopping Talk, and Gateway reconnect against a configured realtime provider.
|
||||
|
||||
- [ ] **Step 5: Commit the task**
|
||||
- Commit with `docs(talk): document realtime talk support`.
|
||||
@@ -13,6 +13,11 @@ export type OpenClawConfig = Record<string, unknown>;
|
||||
export type OpenClawConfigMutator = (
|
||||
config: OpenClawConfig,
|
||||
) => void | Promise<void>;
|
||||
/** Runs inside the serialized transaction before each pure mutator application. */
|
||||
export type OpenClawConfigBeforeApply = () => void | Promise<void>;
|
||||
export type OpenClawConfigMutationOptions = {
|
||||
beforeApply?: OpenClawConfigBeforeApply;
|
||||
};
|
||||
|
||||
type ConfigDeliveryGatewayManager = Pick<GatewayManager, 'getStatus' | 'rpc'>;
|
||||
|
||||
@@ -87,6 +92,7 @@ async function acceptPersistedConfigSetCommitIfMatched(config: OpenClawConfig):
|
||||
async function mutateRunningConfig(
|
||||
manager: ConfigDeliveryGatewayManager,
|
||||
mutator: OpenClawConfigMutator,
|
||||
options: OpenClawConfigMutationOptions,
|
||||
): Promise<boolean> {
|
||||
for (let attempt = 0; attempt < 2; attempt += 1) {
|
||||
const snapshot = await manager.rpc<ConfigSnapshot>('config.get', {});
|
||||
@@ -96,6 +102,7 @@ async function mutateRunningConfig(
|
||||
}
|
||||
|
||||
const config = parseRunningConfigSnapshot(snapshot);
|
||||
await options.beforeApply?.();
|
||||
if (!await applyMutator(config, mutator, true)) return false;
|
||||
|
||||
try {
|
||||
@@ -177,15 +184,17 @@ async function removeTemporaryFile(temporaryPath: string): Promise<void> {
|
||||
async function mutateFileConfig(
|
||||
manager: ConfigDeliveryGatewayManager | undefined,
|
||||
mutator: OpenClawConfigMutator,
|
||||
options: OpenClawConfigMutationOptions,
|
||||
): Promise<boolean> {
|
||||
return await withConfigLock(async () => {
|
||||
const configPath = resolveOpenClawConfigPath();
|
||||
for (let attempt = 0; attempt < 2; attempt += 1) {
|
||||
const snapshot = await readFileConfig(configPath);
|
||||
await options.beforeApply?.();
|
||||
const changed = await applyMutator(snapshot.config, mutator, snapshot.raw !== undefined);
|
||||
|
||||
if (manager?.getStatus().state === 'running') {
|
||||
return await mutateRunningConfig(manager, mutator);
|
||||
return await mutateRunningConfig(manager, mutator, options);
|
||||
}
|
||||
if (!changed) return false;
|
||||
|
||||
@@ -199,12 +208,12 @@ async function mutateFileConfig(
|
||||
|
||||
try {
|
||||
if (manager?.getStatus().state === 'running') {
|
||||
return await mutateRunningConfig(manager, mutator);
|
||||
return await mutateRunningConfig(manager, mutator, options);
|
||||
}
|
||||
|
||||
const currentRaw = await readFileRaw(configPath);
|
||||
if (manager?.getStatus().state === 'running') {
|
||||
return await mutateRunningConfig(manager, mutator);
|
||||
return await mutateRunningConfig(manager, mutator, options);
|
||||
}
|
||||
if (currentRaw !== snapshot.raw) {
|
||||
if (attempt === 0) continue;
|
||||
@@ -222,12 +231,15 @@ async function mutateFileConfig(
|
||||
});
|
||||
}
|
||||
|
||||
async function runMutation(mutator: OpenClawConfigMutator): Promise<boolean> {
|
||||
async function runMutation(
|
||||
mutator: OpenClawConfigMutator,
|
||||
options: OpenClawConfigMutationOptions,
|
||||
): Promise<boolean> {
|
||||
const manager = gatewayManager;
|
||||
if (manager?.getStatus().state === 'running') {
|
||||
return await mutateRunningConfig(manager, mutator);
|
||||
return await mutateRunningConfig(manager, mutator, options);
|
||||
}
|
||||
return await mutateFileConfig(manager, mutator);
|
||||
return await mutateFileConfig(manager, mutator, options);
|
||||
}
|
||||
|
||||
async function runRead(): Promise<OpenClawConfigSnapshot> {
|
||||
@@ -256,6 +268,7 @@ export function registerOpenClawConfigCoordinator(
|
||||
|
||||
export function mutateOpenClawConfig(
|
||||
mutator: OpenClawConfigMutator,
|
||||
options: OpenClawConfigMutationOptions = {},
|
||||
): Promise<boolean> {
|
||||
const context = activeMutation.getStore();
|
||||
if (context?.active) {
|
||||
@@ -263,8 +276,8 @@ export function mutateOpenClawConfig(
|
||||
}
|
||||
|
||||
const transaction = transactionTail.then(
|
||||
() => runMutation(mutator),
|
||||
() => runMutation(mutator),
|
||||
() => runMutation(mutator, options),
|
||||
() => runMutation(mutator, options),
|
||||
);
|
||||
transactionTail = transaction.then(
|
||||
() => undefined,
|
||||
|
||||
@@ -6,6 +6,7 @@ import type {
|
||||
GatewayChatMessageEvent,
|
||||
GatewayRuntimePayload,
|
||||
} from '@shared/host-events/contract';
|
||||
import type { TalkRelayEvent } from '@shared/talk/types';
|
||||
|
||||
type GatewayEventEmitter = {
|
||||
emit: (event: string, payload: unknown) => boolean;
|
||||
@@ -43,6 +44,9 @@ export function dispatchProtocolEvent(
|
||||
case 'presence':
|
||||
emitter.emit('gateway:presence', payload as GatewayRuntimePayload);
|
||||
break;
|
||||
case 'talk.event':
|
||||
emitter.emit('talk:event', payload as TalkRelayEvent);
|
||||
break;
|
||||
default:
|
||||
emitter.emit('notification', { method: event, params: payload });
|
||||
}
|
||||
@@ -57,6 +61,8 @@ export function dispatchJsonRpcNotification(
|
||||
if (normalized) {
|
||||
emitter.emit('chat:runtime-event', normalized);
|
||||
}
|
||||
} else if (notification.method === 'talk.event') {
|
||||
emitter.emit('talk:event', notification.params as TalkRelayEvent);
|
||||
} else {
|
||||
emitter.emit('notification', notification);
|
||||
}
|
||||
|
||||
@@ -79,6 +79,7 @@ import type {
|
||||
GatewayRuntimePayload,
|
||||
} from '@shared/host-events/contract';
|
||||
import type { ChatRuntimeEvent } from '@shared/chat-runtime-events';
|
||||
import type { TalkRelayEvent } from '@shared/talk/types';
|
||||
|
||||
export interface GatewayStatus {
|
||||
state: GatewayLifecycleState;
|
||||
@@ -162,6 +163,7 @@ export interface GatewayManagerEvents {
|
||||
'channel:status': (data: GatewayChannelStatusEvent) => void;
|
||||
'chat:message': (data: GatewayChatMessageEvent) => void;
|
||||
'chat:runtime-event': (data: ChatRuntimeEvent) => void;
|
||||
'talk:event': (data: TalkRelayEvent) => void;
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+29
-2
@@ -54,6 +54,7 @@ import { deviceOAuthManager } from '../utils/device-oauth';
|
||||
import { browserOAuthManager } from '../utils/browser-oauth';
|
||||
import { whatsAppLoginManager } from '../utils/whatsapp-login';
|
||||
import { syncAllProviderAuthToRuntime } from '../services/providers/provider-runtime-sync';
|
||||
import { closeWindowAfterActiveTalk, forwardActiveTalkEvent, type TalkRelayOwnership } from '../services/talk-api';
|
||||
|
||||
const WINDOWS_APP_USER_MODEL_ID = 'app.clawx.desktop';
|
||||
const isE2EMode = process.env.CLAWX_E2E === '1';
|
||||
@@ -119,6 +120,7 @@ const gotTheLock = gotElectronLock && gotFileLock;
|
||||
let mainWindow: BrowserWindow | null = null;
|
||||
let gatewayManager!: GatewayManager;
|
||||
let clawHubService!: ClawHubService;
|
||||
let talkRelayOwnership: TalkRelayOwnership | null = null;
|
||||
const hostApiRegistry = new HostApiRegistry();
|
||||
const webBrowserGuestRegistry = new WebBrowserGuestRegistry();
|
||||
let webBrowserSession!: Session;
|
||||
@@ -257,6 +259,7 @@ function focusMainWindow(): void {
|
||||
|
||||
function createMainWindow(): BrowserWindow {
|
||||
const win = createWindow();
|
||||
let hideAfterTalkCleanupInFlight = false;
|
||||
|
||||
win.once('ready-to-show', () => {
|
||||
if (mainWindow !== win) {
|
||||
@@ -281,7 +284,24 @@ function createMainWindow(): BrowserWindow {
|
||||
win.on('close', (event) => {
|
||||
if (!isQuitting() && !isE2EMode) {
|
||||
event.preventDefault();
|
||||
win.hide();
|
||||
if (hideAfterTalkCleanupInFlight) return;
|
||||
hideAfterTalkCleanupInFlight = true;
|
||||
const ownership = talkRelayOwnership;
|
||||
if (!ownership) {
|
||||
hideAfterTalkCleanupInFlight = false;
|
||||
win.hide();
|
||||
return;
|
||||
}
|
||||
closeWindowAfterActiveTalk({
|
||||
ownership,
|
||||
gatewayManager,
|
||||
sendTalkEvent: (talkEvent) => sendMainWindowEvent('talk:event', talkEvent),
|
||||
hide: () => {
|
||||
hideAfterTalkCleanupInFlight = false;
|
||||
if (!win.isDestroyed() && !isQuitting()) win.hide();
|
||||
},
|
||||
logWarn: (message, error) => logger.warn(message, error),
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
@@ -355,7 +375,7 @@ async function initialize(): Promise<void> {
|
||||
);
|
||||
|
||||
// Register IPC handlers
|
||||
registerIpcHandlers(
|
||||
talkRelayOwnership = registerIpcHandlers(
|
||||
gatewayManager,
|
||||
clawHubService,
|
||||
window,
|
||||
@@ -483,6 +503,13 @@ async function initialize(): Promise<void> {
|
||||
sendMainWindowEvent('chat:runtime-event', data);
|
||||
});
|
||||
|
||||
gatewayManager.on('talk:event', (data) => {
|
||||
if (!talkRelayOwnership) return;
|
||||
forwardActiveTalkEvent(talkRelayOwnership, data, (event) => {
|
||||
sendMainWindowEvent('talk:event', event);
|
||||
});
|
||||
});
|
||||
|
||||
gatewayManager.on('channel:status', (data) => {
|
||||
sendMainWindowEvent('gateway:channel-status', data);
|
||||
});
|
||||
|
||||
@@ -65,6 +65,7 @@ import { createSessionsApi } from '../services/sessions-api';
|
||||
import { createSkillsApi } from '../services/skills-api';
|
||||
import { createUsageApi } from '../services/usage-api';
|
||||
import { createWebBrowserApi } from '../services/web-browser-api';
|
||||
import { createTalkApi, createTalkRelayOwnership, type TalkRelayOwnership } from '../services/talk-api';
|
||||
import type { WebBrowserGuestRegistry } from './web-browser-policy';
|
||||
import {
|
||||
isLaunchAtStartupKey,
|
||||
@@ -85,12 +86,12 @@ export function registerIpcHandlers(
|
||||
hostApiRegistry: HostApiRegistry,
|
||||
browserSession: Session,
|
||||
registry: WebBrowserGuestRegistry,
|
||||
): void {
|
||||
): TalkRelayOwnership {
|
||||
// Unified request protocol (non-breaking: legacy channels remain available)
|
||||
registerUnifiedRequestHandlers(gatewayManager);
|
||||
|
||||
// Typed host invoke handlers (new renderer facade; legacy channels remain available)
|
||||
registerTypedHostHandlers(
|
||||
const talkRelayOwnership = registerTypedHostHandlers(
|
||||
gatewayManager,
|
||||
clawHubService,
|
||||
mainWindow,
|
||||
@@ -134,6 +135,8 @@ export function registerIpcHandlers(
|
||||
|
||||
// File preview handlers (sandboxed read/write/list for inline viewer)
|
||||
registerFilePreviewHandlers();
|
||||
|
||||
return talkRelayOwnership;
|
||||
}
|
||||
|
||||
function registerTypedHostHandlers(
|
||||
@@ -143,7 +146,7 @@ function registerTypedHostHandlers(
|
||||
hostApiRegistry: HostApiRegistry,
|
||||
browserSession: Session,
|
||||
registry: WebBrowserGuestRegistry,
|
||||
): void {
|
||||
): TalkRelayOwnership {
|
||||
const acpSessionAccessRegistry = new AcpSessionAccessRegistry();
|
||||
const stagedAttachments = new StagedAttachmentRegistry();
|
||||
const attachmentOpenWith = createAttachmentOpenWithService();
|
||||
@@ -152,6 +155,7 @@ function registerTypedHostHandlers(
|
||||
stagedAttachments,
|
||||
openWith: attachmentOpenWith,
|
||||
});
|
||||
const talkRelayOwnership = createTalkRelayOwnership();
|
||||
hostApiRegistry.registerCoreServices({
|
||||
app: createAppApi(),
|
||||
openclaw: createOpenClawApi(),
|
||||
@@ -163,6 +167,7 @@ function registerTypedHostHandlers(
|
||||
uv: createUvApi(),
|
||||
settings: createSettingsApi(gatewayManager),
|
||||
gateway: createGatewayApi(gatewayManager),
|
||||
talk: createTalkApi(gatewayManager, talkRelayOwnership),
|
||||
logs: createLogsApi(),
|
||||
channels: createChannelsApi({ gatewayManager, mainWindow }),
|
||||
agents: createAgentsApi({ gatewayManager }),
|
||||
@@ -180,6 +185,7 @@ function registerTypedHostHandlers(
|
||||
usage: createUsageApi(),
|
||||
});
|
||||
registerHostInvokeHandler(hostApiRegistry);
|
||||
return talkRelayOwnership;
|
||||
}
|
||||
|
||||
function registerUnifiedRequestHandlers(gatewayManager: GatewayManager): void {
|
||||
|
||||
@@ -1,96 +0,0 @@
|
||||
/**
|
||||
* Window Management Utilities
|
||||
* Handles window state persistence and multi-window management
|
||||
*/
|
||||
import { BrowserWindow, screen } from 'electron';
|
||||
|
||||
interface WindowState {
|
||||
x?: number;
|
||||
y?: number;
|
||||
width: number;
|
||||
height: number;
|
||||
isMaximized: boolean;
|
||||
}
|
||||
|
||||
// Lazy-load electron-store (ESM module)
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
let windowStateStore: any = null;
|
||||
|
||||
async function getStore() {
|
||||
if (!windowStateStore) {
|
||||
const Store = (await import('electron-store')).default;
|
||||
windowStateStore = new Store<{ windowState: WindowState }>({
|
||||
name: 'window-state',
|
||||
defaults: {
|
||||
windowState: {
|
||||
width: 1280,
|
||||
height: 800,
|
||||
isMaximized: false,
|
||||
},
|
||||
},
|
||||
});
|
||||
}
|
||||
return windowStateStore;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get saved window state with bounds validation
|
||||
*/
|
||||
export async function getWindowState(): Promise<WindowState> {
|
||||
const store = await getStore();
|
||||
const state = store.get('windowState');
|
||||
|
||||
// Validate that the window is visible on a screen
|
||||
if (state.x !== undefined && state.y !== undefined) {
|
||||
const displays = screen.getAllDisplays();
|
||||
const isVisible = displays.some((display) => {
|
||||
const { x, y, width, height } = display.bounds;
|
||||
return (
|
||||
state.x! >= x &&
|
||||
state.x! < x + width &&
|
||||
state.y! >= y &&
|
||||
state.y! < y + height
|
||||
);
|
||||
});
|
||||
|
||||
if (!isVisible) {
|
||||
// Reset position if not visible
|
||||
delete state.x;
|
||||
delete state.y;
|
||||
}
|
||||
}
|
||||
|
||||
return state;
|
||||
}
|
||||
|
||||
/**
|
||||
* Save window state
|
||||
*/
|
||||
export async function saveWindowState(win: BrowserWindow): Promise<void> {
|
||||
const store = await getStore();
|
||||
const isMaximized = win.isMaximized();
|
||||
|
||||
if (!isMaximized) {
|
||||
const bounds = win.getBounds();
|
||||
store.set('windowState', {
|
||||
x: bounds.x,
|
||||
y: bounds.y,
|
||||
width: bounds.width,
|
||||
height: bounds.height,
|
||||
isMaximized,
|
||||
});
|
||||
} else {
|
||||
store.set('windowState.isMaximized', true);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Track window state changes
|
||||
*/
|
||||
export function trackWindowState(win: BrowserWindow): void {
|
||||
// Save state on window events
|
||||
['resize', 'move', 'close'].forEach((event) => {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
win.on(event as any, () => saveWindowState(win));
|
||||
});
|
||||
}
|
||||
@@ -60,6 +60,9 @@ export function createGatewayApi(gatewayManager: GatewayManager): CompleteHostSe
|
||||
if (!method) {
|
||||
throw new Error('Invalid gateway RPC method');
|
||||
}
|
||||
if (method.startsWith('talk.')) {
|
||||
throw new Error('Talk Gateway RPCs must use the typed Talk API');
|
||||
}
|
||||
const timeoutMs = parseTimeoutMs(body.timeoutMs);
|
||||
return gatewayManager.rpc(method, body.params, timeoutMs);
|
||||
},
|
||||
|
||||
@@ -0,0 +1,469 @@
|
||||
import type { GatewayManager } from '../gateway/manager';
|
||||
import type { CompleteHostServiceRegistry } from '../main/ipc/host-contract';
|
||||
import type {
|
||||
TalkCatalog,
|
||||
TalkCatalogProvider,
|
||||
TalkRelayEvent,
|
||||
TalkRealtimeSettingsPayload,
|
||||
} from '@shared/talk/types';
|
||||
import { mutateOpenClawConfig } from '../gateway/config-delivery';
|
||||
import { isRecord } from './payload-utils';
|
||||
|
||||
export const TALK_RPC_TIMEOUT_MS = 8_000;
|
||||
|
||||
export type ActiveTalkRelay = {
|
||||
relaySessionId: string;
|
||||
sessionKey: string;
|
||||
};
|
||||
|
||||
export type TalkRelayOwnership = {
|
||||
activate: (relaySessionId: string, sessionKey: string) => void;
|
||||
clear: (relaySessionId?: string) => void;
|
||||
beginRelayStart: () => number;
|
||||
isRelayStartCurrent: (generation: number) => boolean;
|
||||
beginShutdown: () => ActiveTalkRelay | undefined;
|
||||
finishShutdown: () => void;
|
||||
getActive: () => ActiveTalkRelay | undefined;
|
||||
takeActive: () => ActiveTalkRelay | undefined;
|
||||
isActive: (relaySessionId: string) => boolean;
|
||||
};
|
||||
|
||||
export function createTalkRelayOwnership(): TalkRelayOwnership {
|
||||
let active: ActiveTalkRelay | undefined;
|
||||
let generation = 0;
|
||||
let shuttingDown = false;
|
||||
return {
|
||||
activate: (relaySessionId, sessionKey) => {
|
||||
active = { relaySessionId, sessionKey };
|
||||
},
|
||||
clear: (relaySessionId) => {
|
||||
if (relaySessionId === undefined || active?.relaySessionId === relaySessionId) {
|
||||
active = undefined;
|
||||
}
|
||||
},
|
||||
beginRelayStart: () => {
|
||||
if (shuttingDown) throw new Error('Talk relay startup was cancelled');
|
||||
return generation;
|
||||
},
|
||||
isRelayStartCurrent: (startGeneration) => !shuttingDown && generation === startGeneration,
|
||||
beginShutdown: () => {
|
||||
shuttingDown = true;
|
||||
generation += 1;
|
||||
const claimed = active;
|
||||
active = undefined;
|
||||
return claimed;
|
||||
},
|
||||
finishShutdown: () => {
|
||||
shuttingDown = false;
|
||||
},
|
||||
getActive: () => active,
|
||||
takeActive: () => {
|
||||
const claimed = active;
|
||||
active = undefined;
|
||||
return claimed;
|
||||
},
|
||||
isActive: (relaySessionId) => active?.relaySessionId === relaySessionId,
|
||||
};
|
||||
}
|
||||
|
||||
export function forwardActiveTalkEvent(
|
||||
ownership: TalkRelayOwnership,
|
||||
event: TalkRelayEvent,
|
||||
sendToRenderer: (event: TalkRelayEvent) => void,
|
||||
): void {
|
||||
if (!ownership.isActive(event.relaySessionId)) return;
|
||||
if (event.type === 'close') {
|
||||
ownership.clear(event.relaySessionId);
|
||||
}
|
||||
sendToRenderer(event);
|
||||
}
|
||||
|
||||
type CloseWindowAfterActiveTalkDependencies = {
|
||||
ownership: TalkRelayOwnership;
|
||||
gatewayManager: GatewayManager;
|
||||
sendTalkEvent: (event: TalkRelayEvent) => void;
|
||||
hide: () => void;
|
||||
logWarn?: (message: string, error: unknown) => void;
|
||||
timeoutMs?: number;
|
||||
};
|
||||
|
||||
export function closeWindowAfterActiveTalk({
|
||||
ownership,
|
||||
gatewayManager,
|
||||
sendTalkEvent,
|
||||
hide,
|
||||
logWarn,
|
||||
timeoutMs = TALK_RPC_TIMEOUT_MS,
|
||||
}: CloseWindowAfterActiveTalkDependencies): void {
|
||||
const activeRelay = ownership.beginShutdown();
|
||||
if (!activeRelay) {
|
||||
ownership.finishShutdown();
|
||||
hide();
|
||||
return;
|
||||
}
|
||||
|
||||
// This reaches the renderer through its typed Talk event subscription, releasing microphone tracks first.
|
||||
try {
|
||||
sendTalkEvent({ relaySessionId: activeRelay.relaySessionId, type: 'close', reason: 'cancelled' });
|
||||
} catch (error) {
|
||||
logWarn?.('Failed to request Talk renderer cleanup before hiding the window', error);
|
||||
}
|
||||
void (async () => {
|
||||
let timeout: ReturnType<typeof setTimeout> | undefined;
|
||||
const result = await Promise.race([
|
||||
gatewayManager.rpc('talk.session.close', { sessionId: activeRelay.relaySessionId }, TALK_RPC_TIMEOUT_MS)
|
||||
.then(() => ({ type: 'closed' as const }))
|
||||
.catch((error: unknown) => ({ type: 'failed' as const, error })),
|
||||
new Promise<{ type: 'timed-out' }>((resolve) => {
|
||||
timeout = setTimeout(() => resolve({ type: 'timed-out' }), timeoutMs);
|
||||
}),
|
||||
]);
|
||||
if (timeout) clearTimeout(timeout);
|
||||
if (result.type === 'failed') {
|
||||
logWarn?.('Failed to close active Talk relay before hiding the window', result.error);
|
||||
} else if (result.type === 'timed-out') {
|
||||
logWarn?.('Timed out closing active Talk relay before hiding the window', new Error(`Talk relay close exceeded ${timeoutMs}ms`));
|
||||
}
|
||||
ownership.finishShutdown();
|
||||
hide();
|
||||
})();
|
||||
}
|
||||
|
||||
function requiredString(value: unknown, message: string): string {
|
||||
if (typeof value !== 'string' || !value.trim()) {
|
||||
throw new Error(message);
|
||||
}
|
||||
return value.trim();
|
||||
}
|
||||
|
||||
function stringArray(value: unknown): string[] {
|
||||
return Array.isArray(value)
|
||||
? value.filter((item): item is string => typeof item === 'string' && item.trim().length > 0).map((item) => item.trim())
|
||||
: [];
|
||||
}
|
||||
|
||||
function parseTalkCatalog(value: unknown): TalkCatalog {
|
||||
const catalog = isRecord(value) ? value : {};
|
||||
const realtime = isRecord(catalog.realtime) ? catalog.realtime : {};
|
||||
const providers: TalkCatalogProvider[] = Array.isArray(realtime.providers)
|
||||
? realtime.providers.flatMap((value) => {
|
||||
if (!isRecord(value) || typeof value.id !== 'string' || !value.id.trim()) return [];
|
||||
return [{
|
||||
id: value.id.trim(),
|
||||
label: typeof value.label === 'string' && value.label.trim() ? value.label.trim() : value.id.trim(),
|
||||
configured: value.configured === true,
|
||||
models: stringArray(value.models),
|
||||
...(typeof value.defaultModel === 'string' && value.defaultModel.trim()
|
||||
? { defaultModel: value.defaultModel.trim() }
|
||||
: {}),
|
||||
voices: stringArray(value.voices),
|
||||
transports: stringArray(value.transports),
|
||||
brains: stringArray(value.brains),
|
||||
}];
|
||||
})
|
||||
: [];
|
||||
const readiness = realtime.ready === true
|
||||
? { ready: true as const }
|
||||
: realtime.ready === false
|
||||
? {
|
||||
ready: false as const,
|
||||
...(typeof realtime.reason === 'string' && realtime.reason.trim() ? { reason: realtime.reason.trim() } : {}),
|
||||
}
|
||||
: {};
|
||||
|
||||
return {
|
||||
...(Array.isArray(catalog.modes) ? { modes: stringArray(catalog.modes) } : {}),
|
||||
...(Array.isArray(catalog.transports) ? { transports: stringArray(catalog.transports) } : {}),
|
||||
...(Array.isArray(catalog.brains) ? { brains: stringArray(catalog.brains) } : {}),
|
||||
realtime: {
|
||||
...readiness,
|
||||
...(typeof realtime.activeProvider === 'string' && realtime.activeProvider.trim()
|
||||
? { activeProvider: realtime.activeProvider.trim() }
|
||||
: {}),
|
||||
providers,
|
||||
},
|
||||
} as TalkCatalog;
|
||||
}
|
||||
|
||||
function parseRealtimeSettings(value: unknown): TalkRealtimeSettingsPayload {
|
||||
if (
|
||||
!isRecord(value)
|
||||
|| Object.keys(value).some((key) => key !== 'provider' && key !== 'model' && key !== 'speakerVoice')
|
||||
) {
|
||||
throw new Error('Invalid Talk realtime settings');
|
||||
}
|
||||
const hasSpeakerVoice = Object.prototype.hasOwnProperty.call(value, 'speakerVoice');
|
||||
return {
|
||||
provider: requiredString(value.provider, 'Invalid Talk realtime provider'),
|
||||
model: requiredString(value.model, 'Invalid Talk realtime model'),
|
||||
...(hasSpeakerVoice
|
||||
? { speakerVoice: requiredString(value.speakerVoice, 'Invalid Talk realtime speaker voice') }
|
||||
: {}),
|
||||
};
|
||||
}
|
||||
|
||||
function modelChoices(provider: TalkCatalogProvider): string[] {
|
||||
return provider.models?.length ? provider.models : provider.defaultModel ? [provider.defaultModel] : [];
|
||||
}
|
||||
|
||||
function validateRealtimeSettings(catalog: TalkCatalog, settings: TalkRealtimeSettingsPayload): boolean {
|
||||
const provider = catalog.realtime.providers.find((candidate) => candidate.id === settings.provider);
|
||||
if (!provider || !provider.configured) {
|
||||
throw new Error('Talk realtime provider is not available');
|
||||
}
|
||||
if (!modelChoices(provider).includes(settings.model)) {
|
||||
throw new Error('Talk realtime model is not available for the provider');
|
||||
}
|
||||
if (!provider.voices?.length) {
|
||||
if (settings.speakerVoice !== undefined) {
|
||||
throw new Error('Talk realtime speaker voice is not available for the provider');
|
||||
}
|
||||
return false;
|
||||
}
|
||||
if (!settings.speakerVoice || !provider.voices.includes(settings.speakerVoice)) {
|
||||
throw new Error('Talk realtime speaker voice is not available for the provider');
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
function updateRealtimeConfig(
|
||||
config: Record<string, unknown>,
|
||||
settings: TalkRealtimeSettingsPayload,
|
||||
updateSpeakerVoice: boolean,
|
||||
): void {
|
||||
const talk = config.talk;
|
||||
if (talk !== undefined && !isRecord(talk)) {
|
||||
throw new Error('Invalid OpenClaw Talk config');
|
||||
}
|
||||
const nextTalk = talk ?? {};
|
||||
const realtime = nextTalk.realtime;
|
||||
if (realtime !== undefined && !isRecord(realtime)) {
|
||||
throw new Error('Invalid OpenClaw Talk realtime config');
|
||||
}
|
||||
const nextRealtime = realtime ?? {};
|
||||
nextRealtime.provider = settings.provider;
|
||||
nextRealtime.model = settings.model;
|
||||
if (updateSpeakerVoice) nextRealtime.speakerVoice = settings.speakerVoice;
|
||||
nextTalk.realtime = nextRealtime;
|
||||
config.talk = nextTalk;
|
||||
}
|
||||
|
||||
async function validateRealtimeCatalogForMutation(
|
||||
gatewayManager: GatewayManager,
|
||||
settings: TalkRealtimeSettingsPayload,
|
||||
): Promise<boolean> {
|
||||
if (gatewayManager.getStatus().state !== 'running') {
|
||||
throw new Error('Talk realtime catalog is unavailable; reconnect the Gateway and try again');
|
||||
}
|
||||
try {
|
||||
const catalog = parseTalkCatalog(await gatewayManager.rpc('talk.catalog', {}, TALK_RPC_TIMEOUT_MS));
|
||||
return validateRealtimeSettings(catalog, settings);
|
||||
} catch (error) {
|
||||
if (error instanceof Error && error.message.startsWith('Talk realtime ')) throw error;
|
||||
throw new Error('Talk realtime catalog is unavailable; reconnect the Gateway and try again', { cause: error });
|
||||
}
|
||||
}
|
||||
|
||||
function parseRelayId(value: unknown): string {
|
||||
return requiredString(value, 'Invalid Talk relay session id');
|
||||
}
|
||||
|
||||
function requireActiveRelay(ownership: TalkRelayOwnership, relaySessionId: string): ActiveTalkRelay {
|
||||
const active = ownership.getActive();
|
||||
if (!active || active.relaySessionId !== relaySessionId) {
|
||||
throw new Error('Talk relay session is not active');
|
||||
}
|
||||
return active;
|
||||
}
|
||||
|
||||
function parsePcm16Base64(value: unknown): string {
|
||||
if (
|
||||
typeof value !== 'string'
|
||||
|| !/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/.test(value)
|
||||
) {
|
||||
throw new Error('Invalid Talk PCM16 audio');
|
||||
}
|
||||
const bytes = Buffer.from(value, 'base64');
|
||||
if (bytes.length === 0 || bytes.length % 2 !== 0) {
|
||||
throw new Error('Invalid Talk PCM16 audio');
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function parseTimestamp(value: unknown): number | undefined {
|
||||
if (value === undefined) return undefined;
|
||||
if (typeof value !== 'number' || !Number.isFinite(value) || value < 0) {
|
||||
throw new Error('Invalid Talk audio timestamp');
|
||||
}
|
||||
return Math.floor(value);
|
||||
}
|
||||
|
||||
function parseToolResultOptions(value: unknown): { suppressResponse?: boolean; willContinue?: boolean } | undefined {
|
||||
if (value === undefined) return undefined;
|
||||
if (!isRecord(value)
|
||||
|| (value.suppressResponse !== undefined && typeof value.suppressResponse !== 'boolean')
|
||||
|| (value.willContinue !== undefined && typeof value.willContinue !== 'boolean')) {
|
||||
throw new Error('Invalid Talk tool result options');
|
||||
}
|
||||
return {
|
||||
...(typeof value.suppressResponse === 'boolean' ? { suppressResponse: value.suppressResponse } : {}),
|
||||
...(typeof value.willContinue === 'boolean' ? { willContinue: value.willContinue } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
export function createTalkApi(
|
||||
gatewayManager: GatewayManager,
|
||||
ownership: TalkRelayOwnership = createTalkRelayOwnership(),
|
||||
): CompleteHostServiceRegistry['talk'] {
|
||||
let pendingRelayOperation = Promise.resolve();
|
||||
|
||||
const serializeRelayOperation = async <T>(operation: () => Promise<T>): Promise<T> => {
|
||||
const previousOperation = pendingRelayOperation;
|
||||
let releaseOperation!: () => void;
|
||||
pendingRelayOperation = new Promise<void>((resolve) => {
|
||||
releaseOperation = resolve;
|
||||
});
|
||||
await previousOperation;
|
||||
try {
|
||||
return await operation();
|
||||
} finally {
|
||||
releaseOperation();
|
||||
}
|
||||
};
|
||||
|
||||
return {
|
||||
catalog: async () => parseTalkCatalog(await gatewayManager.rpc('talk.catalog', {}, TALK_RPC_TIMEOUT_MS)),
|
||||
updateRealtimeSettings: async (payload) => {
|
||||
const settings = parseRealtimeSettings(payload);
|
||||
let updateSpeakerVoice = false;
|
||||
await mutateOpenClawConfig((config) => {
|
||||
updateRealtimeConfig(config, settings, updateSpeakerVoice);
|
||||
}, {
|
||||
beforeApply: async () => {
|
||||
updateSpeakerVoice = await validateRealtimeCatalogForMutation(gatewayManager, settings);
|
||||
},
|
||||
});
|
||||
return { ok: true };
|
||||
},
|
||||
startRelay: async (payload) => {
|
||||
const body: Record<string, unknown> = isRecord(payload) ? payload : {};
|
||||
const sessionKey = requiredString(body.sessionKey, 'Invalid Talk session key');
|
||||
const startGeneration = ownership.beginRelayStart();
|
||||
return await serializeRelayOperation(async () => {
|
||||
if (!ownership.isRelayStartCurrent(startGeneration)) {
|
||||
throw new Error('Talk relay startup was cancelled');
|
||||
}
|
||||
const activeRelay = ownership.getActive();
|
||||
if (activeRelay) {
|
||||
try {
|
||||
await gatewayManager.rpc('talk.session.close', { sessionId: activeRelay.relaySessionId }, TALK_RPC_TIMEOUT_MS);
|
||||
} catch {
|
||||
// A failed replacement close must not leave global relay ownership locked forever.
|
||||
} finally {
|
||||
ownership.clear(activeRelay.relaySessionId);
|
||||
}
|
||||
}
|
||||
const relay = await gatewayManager.rpc('talk.session.create', {
|
||||
sessionKey,
|
||||
mode: 'realtime',
|
||||
transport: 'gateway-relay',
|
||||
brain: 'agent-consult',
|
||||
}, TALK_RPC_TIMEOUT_MS);
|
||||
if (!isRecord(relay)) {
|
||||
throw new Error('Invalid Talk relay session');
|
||||
}
|
||||
const relaySessionId = parseRelayId(relay.relaySessionId);
|
||||
if (!ownership.isRelayStartCurrent(startGeneration)) {
|
||||
try {
|
||||
await gatewayManager.rpc('talk.session.close', { sessionId: relaySessionId }, TALK_RPC_TIMEOUT_MS);
|
||||
} catch {
|
||||
// The relay is already invalidated by shutdown; cancellation is still authoritative.
|
||||
}
|
||||
throw new Error('Talk relay startup was cancelled');
|
||||
}
|
||||
ownership.activate(relaySessionId, sessionKey);
|
||||
return relay as never;
|
||||
});
|
||||
},
|
||||
appendAudio: async (payload) => {
|
||||
const body: Record<string, unknown> = isRecord(payload) ? payload : {};
|
||||
const relaySessionId = parseRelayId(body.relaySessionId);
|
||||
const timestamp = parseTimestamp(body.timestamp);
|
||||
const audioBase64 = parsePcm16Base64(body.audioBase64);
|
||||
requireActiveRelay(ownership, relaySessionId);
|
||||
return await gatewayManager.rpc('talk.session.appendAudio', {
|
||||
sessionId: relaySessionId,
|
||||
audioBase64,
|
||||
...(timestamp === undefined ? {} : { timestamp }),
|
||||
}, TALK_RPC_TIMEOUT_MS);
|
||||
},
|
||||
cancelOutput: async (payload) => {
|
||||
const body: Record<string, unknown> = isRecord(payload) ? payload : {};
|
||||
const relaySessionId = parseRelayId(body.relaySessionId);
|
||||
requireActiveRelay(ownership, relaySessionId);
|
||||
return await gatewayManager.rpc('talk.session.cancelOutput', {
|
||||
sessionId: relaySessionId,
|
||||
}, TALK_RPC_TIMEOUT_MS);
|
||||
},
|
||||
submitToolResult: async (payload) => {
|
||||
const body: Record<string, unknown> = isRecord(payload) ? payload : {};
|
||||
if (!('result' in body)) {
|
||||
throw new Error('Invalid Talk tool result');
|
||||
}
|
||||
const relaySessionId = parseRelayId(body.relaySessionId);
|
||||
const callId = requiredString(body.callId, 'Invalid Talk tool call id');
|
||||
const options = parseToolResultOptions(body.options);
|
||||
requireActiveRelay(ownership, relaySessionId);
|
||||
return await gatewayManager.rpc('talk.session.submitToolResult', {
|
||||
sessionId: relaySessionId,
|
||||
callId,
|
||||
result: body.result,
|
||||
...(options && Object.keys(options).length > 0 ? { options } : {}),
|
||||
}, TALK_RPC_TIMEOUT_MS);
|
||||
},
|
||||
acknowledgeMark: async (payload) => {
|
||||
const body: Record<string, unknown> = isRecord(payload) ? payload : {};
|
||||
const relaySessionId = parseRelayId(body.relaySessionId);
|
||||
const markName = requiredString(body.markName, 'Invalid Talk mark name');
|
||||
requireActiveRelay(ownership, relaySessionId);
|
||||
return await gatewayManager.rpc('talk.session.acknowledgeMark', {
|
||||
sessionId: relaySessionId,
|
||||
markName,
|
||||
}, TALK_RPC_TIMEOUT_MS);
|
||||
},
|
||||
stopRelay: async (payload) => {
|
||||
const body: Record<string, unknown> = isRecord(payload) ? payload : {};
|
||||
const relaySessionId = parseRelayId(body.relaySessionId);
|
||||
return await serializeRelayOperation(async () => {
|
||||
requireActiveRelay(ownership, relaySessionId);
|
||||
try {
|
||||
return await gatewayManager.rpc('talk.session.close', {
|
||||
sessionId: relaySessionId,
|
||||
}, TALK_RPC_TIMEOUT_MS);
|
||||
} finally {
|
||||
ownership.clear(relaySessionId);
|
||||
}
|
||||
});
|
||||
},
|
||||
startAgentConsult: async (payload) => {
|
||||
const body: Record<string, unknown> = isRecord(payload) ? payload : {};
|
||||
if (!isRecord(body.args)) {
|
||||
throw new Error('Invalid Talk tool arguments');
|
||||
}
|
||||
const relaySessionId = parseRelayId(body.relaySessionId);
|
||||
const sessionKey = requiredString(body.sessionKey, 'Invalid Talk session key');
|
||||
const callId = requiredString(body.callId, 'Invalid Talk tool call id');
|
||||
const activeRelay = requireActiveRelay(ownership, relaySessionId);
|
||||
if (activeRelay.sessionKey !== sessionKey) {
|
||||
throw new Error('Talk relay session does not match the active session');
|
||||
}
|
||||
return await gatewayManager.rpc('talk.client.toolCall', {
|
||||
relaySessionId,
|
||||
sessionKey,
|
||||
callId,
|
||||
name: 'openclaw_agent_consult',
|
||||
args: body.args,
|
||||
}, TALK_RPC_TIMEOUT_MS);
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
# Realtime Talk
|
||||
|
||||
Status: current architecture reference, reviewed 2026-08-16.
|
||||
|
||||
Related scenarios: `realtime-talk`, `gateway-backend-communication`, `acp-chat-experience`
|
||||
|
||||
Related rule: `realtime-talk-openclaw-authority`
|
||||
|
||||
Related task: `add-realtime-talk`
|
||||
|
||||
## Authority
|
||||
|
||||
Talk uses OpenClaw Gateway Relay only. There is one global relay: it is created for the selected non-heartbeat Chat session and is released on stop, terminal events, session changes, unmount, and Gateway loss. Electron Main owns provider-facing protocol and RPC calls, validates the relay session id, and routes typed `talk.event` notifications. Renderer reaches Main through `hostApi.talk` and `hostEvents.onTalkEvent`; it does not open a Gateway or provider transport.
|
||||
|
||||
## Transient Direct Conversation
|
||||
|
||||
Renderer owns microphone capture, playback, input level, and direct-provider transcript presentation for an active relay. Direct text and audio are transient. They clear when the relay ends, the selected session changes, ACP reloads, or the application restarts.
|
||||
|
||||
There is no ClawX transcript persistence for Talk: no ledger, sidecar, cache, persistent store, direct OpenClaw transcript write, or synthetic ACP entry. The ACP timeline remains unchanged for direct realtime responses.
|
||||
|
||||
## Agent Consult History
|
||||
|
||||
An Agent consult is an OpenClaw operation for the selected session, not an ACP `session/prompt` and not a Renderer timeline projection. Its durable result is owned by OpenClaw and appears only when ACP replays that existing history. The controller can ask the ACP store to reload after consult completion, but cannot append a replacement message or tool event.
|
||||
|
||||
## Configuration
|
||||
|
||||
Talk provider credentials remain Gateway-owned and never enter Renderer state or basic Settings. Main reads and writes `talk.realtime` through the existing catalog and OpenClaw config transaction. The settings surface is limited to a catalog-configured provider, model, speaker voice, and readiness; it excludes secrets, transports, VAD, recording, video, and dictation controls.
|
||||
|
||||
## Validation Anchors
|
||||
|
||||
Primary implementation anchors are `electron/services/talk-api.ts`, `electron/gateway/event-dispatch.ts`, `src/lib/talk/realtime-talk-controller.ts`, `src/stores/realtime-talk.ts`, `src/pages/Chat/ChatInput.tsx`, `src/pages/Chat/LiveTalkTranscript.tsx`, and `src/components/settings/TalkSettings.tsx`.
|
||||
|
||||
Focused coverage is in `tests/unit/talk-api.test.ts`, `tests/unit/talk-audio.test.ts`, `tests/unit/realtime-talk-controller.test.ts`, `tests/unit/realtime-talk-store.test.ts`, `tests/unit/live-talk-transcript.test.tsx`, `tests/unit/talk-settings.test.tsx`, and the Chat Electron E2E specs. Gateway changes require Harness validation, communication replay and comparison, and the applicable unit, E2E, lint, type, and build checks.
|
||||
@@ -4,6 +4,7 @@ title: E2E Parallel Isolation
|
||||
type: ai-coding-rule
|
||||
appliesTo:
|
||||
- gateway-backend-communication
|
||||
- realtime-talk
|
||||
requiredProfiles:
|
||||
- fast
|
||||
- e2e
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
id: realtime-talk-openclaw-authority
|
||||
title: Realtime Talk OpenClaw Authority
|
||||
type: ai-coding-rule
|
||||
appliesTo:
|
||||
- realtime-talk
|
||||
- gateway-backend-communication
|
||||
- acp-chat-experience
|
||||
requiredProfiles:
|
||||
- comms
|
||||
---
|
||||
|
||||
Talk supports only OpenClaw Gateway Relay. Electron Main owns every provider-facing Gateway RPC, typed `talk.event` routing, relay identity validation, and the single global relay lifecycle. Renderer uses the typed Host API and host events only; it must not open a Gateway HTTP, WebSocket, provider WebSocket, WebRTC, or other direct provider path.
|
||||
|
||||
Direct Talk text and audio are transient Renderer state. Do not create a ClawX transcript ledger, sidecar, cache, persistence key, direct transcript-file write, synthetic ACP entry, or custom durable timeline projection. OpenClaw and ACP own durable Agent-consult history: a consult uses OpenClaw's existing path and is visible only through normal ACP replay.
|
||||
|
||||
Talk configuration is Main-owned and catalog-validated. Basic Settings may select only the configured realtime provider, model, speaker voice, and readiness state through the existing OpenClaw configuration transaction. It must not expose or persist provider secrets, transports, VAD controls, recordings, video, dictation, or a transport picker.
|
||||
@@ -46,6 +46,7 @@ conditionalProfiles:
|
||||
requiredRules:
|
||||
- renderer-main-boundary
|
||||
- acp-chat-state-and-history
|
||||
- realtime-talk-openclaw-authority
|
||||
- acp-compatibility-content-safety
|
||||
- attachment-access-safety
|
||||
- diagnostics-trace-safety
|
||||
@@ -64,3 +65,5 @@ ACP Chat covers session load, prompt, cancel, permission, replay, timeline reduc
|
||||
Main owns ACP transport, routing, transcript retrieval and timing extraction, workspace grants, and session/generation-scoped attachment authorization. Renderer owns the in-memory timeline, bounded compatibility and timing alignment, attachment presentation, and display grouping, including user-image thumbnails and user-selected source-path labels. ACP replay remains authoritative for historical turns and content; transcript-derived timing may only annotate an unambiguously matched ACP turn. Standard ACP content remains preferred over compatibility projections, and incidental tool paths never enter the attachment pipeline.
|
||||
|
||||
The durable architecture, exceptions, access boundary, file-activity separation, Office preview behavior, Markdown rendering, Electron rendering performance policy, and validation anchors are documented in `harness/reference/acp-chat.md`, `harness/reference/acp-generated-media-and-diagnostics.md`, `harness/reference/acp-attachment-access-control.md`, `harness/reference/openclaw-file-activity.md`, `harness/reference/office-document-preview.md`, `harness/reference/markdown-rendering.md`, and `harness/reference/electron-rendering-performance.md`.
|
||||
|
||||
Realtime Talk direct-provider content is not ACP timeline content and remains transient. Agent consult history is OpenClaw-owned and reaches the timeline only through ACP replay, as specified in `harness/reference/realtime-talk.md`.
|
||||
|
||||
@@ -59,6 +59,7 @@ requiredRules:
|
||||
- provider-model-metadata-preservation
|
||||
- provider-model-selection-authority
|
||||
- sidebar-session-attention-authority
|
||||
- realtime-talk-openclaw-authority
|
||||
- web-browser-security-and-lifecycle
|
||||
- e2e-parallel-isolation
|
||||
- comms-regression
|
||||
@@ -99,4 +100,6 @@ The local HTML Preview privileged bridge is also Main-owned: Renderer may load a
|
||||
|
||||
Gateway session-catalog subscription, normalization, ordered list/event replay, attention transitions, and reconnect recovery are documented in `harness/reference/sidebar-session-attention.md`. Electron test-process isolation and global-resource scheduling are documented in `harness/reference/e2e-parallelism.md`.
|
||||
|
||||
Realtime Talk uses one Main-owned OpenClaw Gateway Relay and typed Talk host events. Renderer direct-provider audio and text are transient, while OpenClaw/ACP remains the durable authority for Agent consult history; its full boundary is `harness/reference/realtime-talk.md`.
|
||||
|
||||
Gateway WebSocket heartbeat misses are diagnostic availability signals for the first three consecutive misses and must not interrupt long-running work during that window. A pong or any incoming Gateway message resets the sequence. On the fourth consecutive miss, Main may request the guarded Gateway restart path when auto-recovery is enabled and lifecycle state is still running; the heartbeat callback must not directly terminate the socket or process. Authoritative process-exit and socket-close signals retain their existing automatic lifecycle paths.
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
---
|
||||
id: realtime-talk
|
||||
title: Realtime Talk
|
||||
type: runtime-bridge
|
||||
ownedPaths:
|
||||
- shared/talk/**
|
||||
- shared/host-api/contract.ts
|
||||
- shared/host-events/contract.ts
|
||||
- electron/services/talk-api.ts
|
||||
- electron/main/ipc-handlers.ts
|
||||
- electron/gateway/event-dispatch.ts
|
||||
- electron/gateway/config-delivery.ts
|
||||
- electron/gateway/manager.ts
|
||||
- electron/main/index.ts
|
||||
- src/lib/talk/**
|
||||
- src/lib/host-api.ts
|
||||
- src/lib/host-events.ts
|
||||
- src/stores/realtime-talk.ts
|
||||
- src/stores/acp-chat-session.ts
|
||||
- src/pages/Chat/ChatInput.tsx
|
||||
- src/pages/Chat/LiveTalkTranscript.tsx
|
||||
- src/pages/Chat/index.tsx
|
||||
- src/components/settings/TalkSettings.tsx
|
||||
- src/pages/Settings/index.tsx
|
||||
- shared/i18n/locales/**/chat.json
|
||||
- shared/i18n/locales/**/settings.json
|
||||
- tests/unit/talk-*.test.ts
|
||||
- tests/unit/talk-*.test.tsx
|
||||
- tests/unit/realtime-talk-*.test.ts
|
||||
- tests/unit/live-talk-transcript.test.tsx
|
||||
- tests/unit/acp-chat-store.test.ts
|
||||
- tests/unit/chat-acp-inline-timeline.test.tsx
|
||||
- tests/unit/chat-acp-page.test.tsx
|
||||
- tests/unit/chat-input.test.tsx
|
||||
- tests/unit/gateway-config-delivery.test.ts
|
||||
- tests/unit/gateway-event-dispatch.test.ts
|
||||
- tests/unit/harness-specs.test.ts
|
||||
- tests/unit/harness-git.test.ts
|
||||
- tests/unit/host-api-facade.test.ts
|
||||
- tests/unit/host-events.test.ts
|
||||
- tests/unit/host-services.test.ts
|
||||
- tests/e2e/chat-acp-inline-timeline.spec.ts
|
||||
- tests/e2e/chat-model-picker.spec.ts
|
||||
- tests/e2e/developer-mode.spec.ts
|
||||
- README.md
|
||||
- README.zh-CN.md
|
||||
- README.ja-JP.md
|
||||
- harness/specs/tasks/add-realtime-talk.md
|
||||
- harness/specs/scenarios/acp-chat-experience.md
|
||||
- harness/specs/scenarios/gateway-backend-communication.md
|
||||
- harness/specs/scenarios/realtime-talk.md
|
||||
- harness/specs/rules/realtime-talk-openclaw-authority.md
|
||||
- harness/specs/rules/e2e-parallel-isolation.md
|
||||
- harness/reference/realtime-talk.md
|
||||
- harness/src/cli.mjs
|
||||
- harness/src/git.mjs
|
||||
- harness/src/rules.mjs
|
||||
requiredProfiles:
|
||||
- fast
|
||||
- comms
|
||||
- e2e
|
||||
requiredRules:
|
||||
- renderer-main-boundary
|
||||
- backend-communication-boundary
|
||||
- api-client-transport-policy
|
||||
- host-api-fallback-policy
|
||||
- host-events-fallback-policy
|
||||
- gateway-readiness-policy
|
||||
- gateway-heartbeat-safety
|
||||
- openclaw-config-delivery
|
||||
- acp-chat-state-and-history
|
||||
- ui-i18n-design-tokens
|
||||
- realtime-talk-openclaw-authority
|
||||
- e2e-parallel-isolation
|
||||
- comms-regression
|
||||
- docs-sync
|
||||
---
|
||||
|
||||
Realtime Talk provides one global, Main-owned OpenClaw Gateway Relay session for the selected non-heartbeat Chat session. Renderer captures and plays transient audio, presents transient direct-provider text, and reaches Main only through the typed Host API and host-event interfaces.
|
||||
|
||||
Gateway Relay, relay lifecycle, configuration authority, and the direct-versus-Agent-consult history boundary are defined in `harness/reference/realtime-talk.md`. The ACP interaction remains subject to `harness/reference/acp-chat.md` and `harness/reference/openclaw-config-delivery.md`.
|
||||
|
||||
The Talk E2E uses test-local AudioContext and media-device mocks, so it does not require E2E_EXCLUSIVE_TAG because its audio/media mocks are local and do not mutate an OS-global resource.
|
||||
@@ -0,0 +1,126 @@
|
||||
---
|
||||
id: add-realtime-talk
|
||||
title: Add OpenClaw Gateway Relay realtime Talk to ClawX Chat
|
||||
scenario: gateway-backend-communication
|
||||
scenarios:
|
||||
- realtime-talk
|
||||
taskType: runtime-bridge
|
||||
intent: Add a Main-owned OpenClaw Gateway Relay Talk transport, transient Renderer audio and direct-transcript UI, OpenClaw-backed Agent consult handling, and basic Talk settings without adding a ClawX-owned durable transcript source.
|
||||
touchedAreas:
|
||||
- harness/specs/tasks/add-realtime-talk.md
|
||||
- harness/specs/scenarios/acp-chat-experience.md
|
||||
- harness/specs/scenarios/gateway-backend-communication.md
|
||||
- harness/specs/scenarios/realtime-talk.md
|
||||
- harness/specs/rules/realtime-talk-openclaw-authority.md
|
||||
- harness/specs/rules/e2e-parallel-isolation.md
|
||||
- harness/reference/realtime-talk.md
|
||||
- harness/src/cli.mjs
|
||||
- harness/src/git.mjs
|
||||
- harness/src/rules.mjs
|
||||
- shared/talk/**
|
||||
- shared/host-api/contract.ts
|
||||
- shared/host-events/contract.ts
|
||||
- electron/services/talk-api.ts
|
||||
- electron/main/ipc-handlers.ts
|
||||
- electron/gateway/event-dispatch.ts
|
||||
- electron/gateway/config-delivery.ts
|
||||
- electron/gateway/manager.ts
|
||||
- electron/main/index.ts
|
||||
- src/lib/talk/**
|
||||
- src/lib/host-api.ts
|
||||
- src/lib/host-events.ts
|
||||
- src/stores/realtime-talk.ts
|
||||
- src/stores/acp-chat-session.ts
|
||||
- src/pages/Chat/**
|
||||
- src/components/settings/TalkSettings.tsx
|
||||
- src/pages/Settings/index.tsx
|
||||
- shared/i18n/locales/**/chat.json
|
||||
- shared/i18n/locales/**/settings.json
|
||||
- tests/unit/talk-*.test.ts
|
||||
- tests/unit/talk-*.test.tsx
|
||||
- tests/unit/realtime-talk-*.test.ts
|
||||
- tests/unit/live-talk-transcript.test.tsx
|
||||
- tests/unit/acp-chat-store.test.ts
|
||||
- tests/unit/chat-input.test.tsx
|
||||
- tests/unit/chat-acp-page.test.tsx
|
||||
- tests/unit/chat-acp-inline-timeline.test.tsx
|
||||
- tests/unit/host-api-facade.test.ts
|
||||
- tests/unit/host-events.test.ts
|
||||
- tests/unit/host-services.test.ts
|
||||
- tests/unit/gateway-event-dispatch.test.ts
|
||||
- tests/unit/gateway-config-delivery.test.ts
|
||||
- tests/unit/harness-specs.test.ts
|
||||
- tests/unit/harness-git.test.ts
|
||||
- tests/e2e/chat-acp-inline-timeline.spec.ts
|
||||
- tests/e2e/chat-model-picker.spec.ts
|
||||
- tests/e2e/developer-mode.spec.ts
|
||||
- README.md
|
||||
- README.zh-CN.md
|
||||
- README.ja-JP.md
|
||||
expectedUserBehavior:
|
||||
- A user starts and stops one Gateway Relay realtime Talk session from the selected non-heartbeat Chat session.
|
||||
- The Renderer never communicates with the OpenClaw Gateway directly; Main owns all Talk RPCs and Talk event routing.
|
||||
- Direct realtime transcripts are visible only while Talk is active and never become ClawX-owned durable history.
|
||||
- Agent consults use OpenClaw's existing chat.send path for the selected session and reappear only through ACP replay.
|
||||
- Talk locks ordinary text composition while active and restores the user's draft on every terminal path.
|
||||
- Settings exposes only catalog-driven realtime provider, model, speaker voice, and readiness controls.
|
||||
requiredProfiles:
|
||||
- fast
|
||||
- comms
|
||||
- e2e
|
||||
requiredRules:
|
||||
- renderer-main-boundary
|
||||
- backend-communication-boundary
|
||||
- api-client-transport-policy
|
||||
- host-api-fallback-policy
|
||||
- host-events-fallback-policy
|
||||
- gateway-readiness-policy
|
||||
- gateway-heartbeat-safety
|
||||
- openclaw-config-delivery
|
||||
- channel-plugin-migration-guards
|
||||
- capability-owner-resolution
|
||||
- active-config-guards
|
||||
- provider-default-invariant
|
||||
- provider-model-metadata-preservation
|
||||
- provider-model-selection-authority
|
||||
- sidebar-session-attention-authority
|
||||
- web-browser-security-and-lifecycle
|
||||
- acp-chat-state-and-history
|
||||
- ui-i18n-design-tokens
|
||||
- realtime-talk-openclaw-authority
|
||||
- e2e-parallel-isolation
|
||||
- comms-regression
|
||||
- docs-sync
|
||||
requiredTests:
|
||||
- pnpm exec vitest run tests/unit/harness-specs.test.ts tests/unit/harness-git.test.ts
|
||||
- pnpm harness validate --spec harness/specs/tasks/add-realtime-talk.md --since bd1aac8e
|
||||
- pnpm harness run --spec harness/specs/tasks/add-realtime-talk.md --since bd1aac8e --dry-run
|
||||
- pnpm run harness:ci
|
||||
- pnpm run lint:check
|
||||
- pnpm run typecheck
|
||||
- pnpm test
|
||||
- pnpm run build:vite
|
||||
- pnpm exec vitest run tests/unit/talk-api.test.ts tests/unit/talk-audio.test.ts tests/unit/realtime-talk-controller.test.ts tests/unit/realtime-talk-store.test.ts
|
||||
- pnpm exec vitest run tests/unit/host-api-facade.test.ts tests/unit/host-events.test.ts tests/unit/host-services.test.ts tests/unit/gateway-event-dispatch.test.ts
|
||||
- pnpm exec vitest run tests/unit/chat-input.test.tsx tests/unit/chat-acp-page.test.tsx tests/unit/chat-acp-inline-timeline.test.tsx tests/unit/talk-settings.test.tsx
|
||||
- pnpm exec playwright test tests/e2e/chat-acp-inline-timeline.spec.ts tests/e2e/chat-model-picker.spec.ts
|
||||
- pnpm run comms:replay
|
||||
- pnpm run comms:compare
|
||||
acceptance:
|
||||
- Only gateway-relay is supported; client-owned WebRTC/provider WebSocket, video, dictation, recording, and a transport picker are absent.
|
||||
- Main validates Talk request payloads, owns provider-facing Gateway protocol calls, and forwards only active relaySessionId events to Renderer.
|
||||
- No ClawX Talk transcript ledger, sidecar history, cache, direct OpenClaw transcript write, or synthetic ACP history exists.
|
||||
- Direct-provider transcript UI is transient and clears on stop, session switch, ACP reload, and application restart.
|
||||
- Agent consult is started with talk.client.toolCall and shown after ACP reload from the existing OpenClaw transcript, never by a custom timeline projection.
|
||||
- New text is localized in en, zh, ja, and ru; the UI has an Electron E2E interaction test.
|
||||
- README English, Simplified Chinese, and Japanese documentation describe the feature and its history semantics.
|
||||
- The Talk scenario, authority rule, and reference define Gateway Relay-only transport, Main-owned configuration, and no ClawX-owned durable Talk history.
|
||||
- Talk readiness is displayed and checked before start; configured providers expose provider, model, and speaker voice selection only.
|
||||
- Release validation records configured-provider macOS microphone permission, direct response, Agent consult, barge-in, stop, and Gateway reconnect results without claiming unperformed manual checks passed.
|
||||
docs:
|
||||
required: true
|
||||
---
|
||||
|
||||
`gateway-backend-communication` is the primary scenario so its task validation and changed-file ownership apply. `realtime-talk` is supplemental scope for the Talk-specific authority rule and renderer lifecycle contract.
|
||||
|
||||
Use `bd1aac8e`, the approved design commit immediately before Talk implementation, for this task's focused Harness diff validation. Normal `origin/main` comparisons and `pnpm run harness:ci` are baseline checks only: unrelated branch/worktree history can contaminate their changed-file range or structural result and must be reported separately.
|
||||
@@ -14,6 +14,7 @@ import {
|
||||
} from './specs.mjs';
|
||||
import {
|
||||
scanBackendCommunicationBoundary,
|
||||
scanRealtimeTalkAuthority,
|
||||
touchesCommunicationPath,
|
||||
validateGatewayTaskSpec,
|
||||
validatePluginLifecycleTaskSpec,
|
||||
@@ -153,6 +154,18 @@ async function run(specPath, options = {}) {
|
||||
durationMs: 0,
|
||||
});
|
||||
|
||||
if (toArray(validation.spec.data.requiredRules).includes('realtime-talk-openclaw-authority')) {
|
||||
const talkAuthorityFailures = await scanRealtimeTalkAuthority(scanFiles);
|
||||
failures.push(...talkAuthorityFailures);
|
||||
steps.push({
|
||||
profile: 'rules',
|
||||
name: 'Realtime Talk authority scan',
|
||||
status: talkAuthorityFailures.length === 0 ? 'pass' : 'fail',
|
||||
exitCode: talkAuthorityFailures.length === 0 ? 0 : 1,
|
||||
durationMs: 0,
|
||||
});
|
||||
}
|
||||
|
||||
const selectedSteps = selectSteps(profiles);
|
||||
if (failures.length === 0) {
|
||||
for (const step of selectedSteps) {
|
||||
|
||||
+4
-1
@@ -18,6 +18,9 @@ export async function getChangedFiles(since = 'origin/main', cwd = ROOT) {
|
||||
for (const line of await gitLines(['diff', '--name-only', `${since}...HEAD`], cwd)) files.add(line);
|
||||
for (const line of await gitLines(['diff', '--cached', '--name-only'], cwd)) files.add(line);
|
||||
for (const line of await gitLines(['diff', '--name-only'], cwd)) files.add(line);
|
||||
for (const line of await gitLines(['ls-files', '--others', '--exclude-standard'], cwd)) files.add(line);
|
||||
// User-authored plans are non-executable workspace notes, not task implementation changes.
|
||||
for (const line of await gitLines(['ls-files', '--others', '--exclude-standard'], cwd)) {
|
||||
if (!line.startsWith('docs/plans/')) files.add(line);
|
||||
}
|
||||
return [...files].sort();
|
||||
}
|
||||
|
||||
@@ -7,6 +7,22 @@ const DIRECT_GATEWAY_HTTP_PATTERN = /fetch\s*\(\s*['"`]http:\/\/(?:127\.0\.0\.1|
|
||||
const DIRECT_GATEWAY_WS_PATTERN = /new\s+WebSocket\s*\(\s*['"`]ws:\/\/(?:127\.0\.0\.1|localhost):18789|ws:\/\/(?:127\.0\.0\.1|localhost):18789/;
|
||||
const HOST_API_LOCAL_HTTP_PATTERN = /fetch\s*\(\s*['"`]http:\/\/(?:127\.0\.0\.1|localhost):13210|HOST_API_BASE\s*=\s*`?http:\/\/127\.0\.0\.1:\$\{HOST_API_PORT\}`?/;
|
||||
const GATEWAY_READY_MUTATION_PATTERN = /gatewayReady\s*[:=]\s*(?:true|false)|setStatus\s*\([^)]*gatewayReady|setState\s*\([^)]*gatewayReady/s;
|
||||
const DIRECT_WEBSOCKET_PATTERN = /new\s+WebSocket\s*\(/;
|
||||
const WEBRTC_PATTERN = /\b(?:RTCPeerConnection|webkitRTCPeerConnection)\b/;
|
||||
const BROWSER_PERSISTENCE_PATTERN = /\b(?:localStorage|sessionStorage)\.(?:setItem|removeItem|clear)\s*\(|\bpersist\s*\(/;
|
||||
const FILE_WRITE_PATTERN = /\b(?:writeFile|appendFile|writeFileSync|appendFileSync)\s*\(/;
|
||||
const SYNTHETIC_ACP_TALK_PATTERN = /(?:appendSyntheticAssistantMessage|upsertSyntheticTurnAttachments)\s*\([^)]*\b(?:talk|relay|transcript)/i;
|
||||
const REALTIME_TALK_AUTHORITY_PATHS = [
|
||||
'shared/talk/**',
|
||||
'src/lib/talk/**',
|
||||
'src/lib/host-api.ts',
|
||||
'src/lib/host-events.ts',
|
||||
'src/stores/realtime-talk.ts',
|
||||
'src/stores/acp-chat-session.ts',
|
||||
'src/pages/Chat/**',
|
||||
'src/components/settings/TalkSettings.tsx',
|
||||
'src/pages/Settings/index.tsx',
|
||||
];
|
||||
const COMMUNICATION_PATHS = [
|
||||
'src/lib/api-client.ts',
|
||||
'src/lib/host-api.ts',
|
||||
@@ -71,6 +87,48 @@ export async function scanBackendCommunicationBoundary(files) {
|
||||
return failures;
|
||||
}
|
||||
|
||||
export function scanRealtimeTalkAuthorityText(file, text) {
|
||||
const failures = [];
|
||||
if (DIRECT_IPC_PATTERN.test(text)) {
|
||||
failures.push(`${file}: Talk renderer must not call window.electron.ipcRenderer.invoke directly`);
|
||||
}
|
||||
if (DIRECT_GATEWAY_HTTP_PATTERN.test(text)) {
|
||||
failures.push(`${file}: Talk renderer must not fetch Gateway HTTP directly`);
|
||||
}
|
||||
if (DIRECT_WEBSOCKET_PATTERN.test(text)) {
|
||||
failures.push(`${file}: Talk renderer must not open Gateway or provider WebSocket connections directly`);
|
||||
}
|
||||
if (WEBRTC_PATTERN.test(text)) {
|
||||
failures.push(`${file}: Talk renderer must not create WebRTC connections`);
|
||||
}
|
||||
if (BROWSER_PERSISTENCE_PATTERN.test(text)) {
|
||||
failures.push(`${file}: Talk transcripts must not use browser persistence`);
|
||||
}
|
||||
if (FILE_WRITE_PATTERN.test(text)) {
|
||||
failures.push(`${file}: Talk transcripts must not write local files`);
|
||||
}
|
||||
if (SYNTHETIC_ACP_TALK_PATTERN.test(text)) {
|
||||
failures.push(`${file}: Talk must not project direct transcripts into ACP history`);
|
||||
}
|
||||
return failures;
|
||||
}
|
||||
|
||||
export function selectRealtimeTalkAuthorityFiles(files) {
|
||||
return unique(files).filter((file) => (
|
||||
/\.(ts|tsx)$/.test(file) && pathMatchesAny(file, REALTIME_TALK_AUTHORITY_PATHS)
|
||||
));
|
||||
}
|
||||
|
||||
export async function scanRealtimeTalkAuthority(files) {
|
||||
const failures = [];
|
||||
const scanFiles = selectRealtimeTalkAuthorityFiles(files);
|
||||
for (const file of scanFiles) {
|
||||
const text = await readTextIfExists(file);
|
||||
if (text) failures.push(...scanRealtimeTalkAuthorityText(file, text));
|
||||
}
|
||||
return failures;
|
||||
}
|
||||
|
||||
export function validateGatewayTaskSpec(taskSpec, scenarioSpec, changedFiles = []) {
|
||||
return validateTaskSpec(taskSpec, scenarioSpec, changedFiles, {
|
||||
scenarioId: 'gateway-backend-communication',
|
||||
|
||||
@@ -95,6 +95,7 @@
|
||||
"electron-updater": "^6.8.3",
|
||||
"json5": "2.2.3",
|
||||
"node-machine-id": "^1.1.12",
|
||||
"pino": "^9.14.0",
|
||||
"posthog-node": "^5.28.0",
|
||||
"tar": "^6.2.1",
|
||||
"timeago.js": "^4.0.2",
|
||||
|
||||
Generated
+7
-4
@@ -31,6 +31,9 @@ importers:
|
||||
node-machine-id:
|
||||
specifier: ^1.1.12
|
||||
version: 1.1.12
|
||||
pino:
|
||||
specifier: ^9.14.0
|
||||
version: 9.14.0
|
||||
posthog-node:
|
||||
specifier: ^5.28.0
|
||||
version: 5.28.5
|
||||
@@ -4677,8 +4680,8 @@ packages:
|
||||
resolution: {integrity: sha512-+bT2uH4E5LGE7h/n3evcS/sQlJXCpIp6ym8OWJ5eV6+67Dsql/LaaT7qJBAt2rzfoa/5QBGBhxDix1dMt2kQKQ==}
|
||||
engines: {node: '>= 0.8.0'}
|
||||
|
||||
libsignal@git+https://git@github.com:whiskeysockets/libsignal-node.git#bcea72df9ec34d9d9140ab30619cf479c7c144c7:
|
||||
resolution: {commit: bcea72df9ec34d9d9140ab30619cf479c7c144c7, repo: git@github.com:whiskeysockets/libsignal-node.git, type: git}
|
||||
libsignal@https://codeload.github.com/whiskeysockets/libsignal-node/tar.gz/bcea72df9ec34d9d9140ab30619cf479c7c144c7:
|
||||
resolution: {gitHosted: true, tarball: https://codeload.github.com/whiskeysockets/libsignal-node/tar.gz/bcea72df9ec34d9d9140ab30619cf479c7c144c7}
|
||||
version: 6.0.0
|
||||
|
||||
lie@3.3.0:
|
||||
@@ -9571,7 +9574,7 @@ snapshots:
|
||||
'@cacheable/node-cache': 1.7.6
|
||||
'@hapi/boom': 9.1.4
|
||||
async-mutex: 0.5.0
|
||||
libsignal: git+https://git@github.com:whiskeysockets/libsignal-node.git#bcea72df9ec34d9d9140ab30619cf479c7c144c7
|
||||
libsignal: https://codeload.github.com/whiskeysockets/libsignal-node/tar.gz/bcea72df9ec34d9d9140ab30619cf479c7c144c7
|
||||
lru-cache: 11.2.7
|
||||
music-metadata: 11.12.3
|
||||
p-queue: 9.1.0
|
||||
@@ -11829,7 +11832,7 @@ snapshots:
|
||||
prelude-ls: 1.2.1
|
||||
type-check: 0.4.0
|
||||
|
||||
libsignal@git+https://git@github.com:whiskeysockets/libsignal-node.git#bcea72df9ec34d9d9140ab30619cf479c7c144c7:
|
||||
libsignal@https://codeload.github.com/whiskeysockets/libsignal-node/tar.gz/bcea72df9ec34d9d9140ab30619cf479c7c144c7:
|
||||
dependencies:
|
||||
curve25519-js: 0.0.4
|
||||
protobufjs: 7.5.8
|
||||
|
||||
@@ -11,6 +11,19 @@ import type { CronJob, CronJobCreateInput, CronJobUpdateInput } from '../types/c
|
||||
import type { GatewayHealth, GatewayStatus } from '../types/gateway';
|
||||
import type { MarketplaceSkill, QuickAccessSkill, Skill } from '../types/skill';
|
||||
import type { WebBrowserNavigatePayload } from '../web-browser';
|
||||
import type {
|
||||
TalkAcknowledgeMarkPayload,
|
||||
TalkAgentConsultResult,
|
||||
TalkAppendAudioPayload,
|
||||
TalkCatalog,
|
||||
TalkOperationResult,
|
||||
TalkRelayIdPayload,
|
||||
TalkRelaySession,
|
||||
TalkRealtimeSettingsPayload,
|
||||
TalkStartAgentConsultPayload,
|
||||
TalkStartRelayPayload,
|
||||
TalkSubmitToolResultPayload,
|
||||
} from '../talk/types';
|
||||
|
||||
export type JsonRecord = Record<string, unknown>;
|
||||
export type HostSuccess = { success: boolean; error?: string };
|
||||
@@ -865,6 +878,17 @@ export type HostApiContract = {
|
||||
controlUi: () => GatewayControlUiResult;
|
||||
rpc: (payload: GatewayRpcPayload) => unknown;
|
||||
};
|
||||
talk: {
|
||||
catalog: () => TalkCatalog;
|
||||
updateRealtimeSettings: (payload: TalkRealtimeSettingsPayload) => TalkOperationResult;
|
||||
startRelay: (payload: TalkStartRelayPayload) => TalkRelaySession;
|
||||
appendAudio: (payload: TalkAppendAudioPayload) => TalkOperationResult;
|
||||
cancelOutput: (payload: TalkRelayIdPayload) => TalkOperationResult;
|
||||
submitToolResult: (payload: TalkSubmitToolResultPayload) => TalkOperationResult;
|
||||
acknowledgeMark: (payload: TalkAcknowledgeMarkPayload) => TalkOperationResult;
|
||||
stopRelay: (payload: TalkRelayIdPayload) => TalkOperationResult;
|
||||
startAgentConsult: (payload: TalkStartAgentConsultPayload) => TalkAgentConsultResult;
|
||||
};
|
||||
logs: {
|
||||
recent: (payload?: LogRecentPayload) => LogContentResult;
|
||||
memory: (payload?: LogMemoryPayload) => string[];
|
||||
|
||||
@@ -4,6 +4,7 @@ import type {
|
||||
} from '../acp-chat/types';
|
||||
import type { UpdateStatusSnapshot } from '../host-api/contract';
|
||||
import type { ChatRuntimeEvent } from '../chat-runtime-events';
|
||||
import type { TalkRelayEvent } from '../talk/types';
|
||||
import type {
|
||||
GatewayNotification,
|
||||
GatewayRuntimePayload,
|
||||
@@ -82,6 +83,9 @@ export type HostEventContract = {
|
||||
acpSessionUpdate: (payload: AcpSessionUpdateEnvelope) => void;
|
||||
acpPermissionRequest: (payload: AcpPermissionRequestEnvelope) => void;
|
||||
};
|
||||
talk: {
|
||||
event: (payload: TalkRelayEvent) => void;
|
||||
};
|
||||
oauth: {
|
||||
code: (payload: OAuthCodeEvent) => void;
|
||||
success: (payload: OAuthSuccessEvent) => void;
|
||||
@@ -131,6 +135,9 @@ export const HOST_EVENT_CHANNELS = {
|
||||
acpSessionUpdate: 'chat:acp-session-update',
|
||||
acpPermissionRequest: 'chat:acp-permission-request',
|
||||
},
|
||||
talk: {
|
||||
event: 'talk:event',
|
||||
},
|
||||
oauth: {
|
||||
code: 'oauth:code',
|
||||
success: 'oauth:success',
|
||||
|
||||
@@ -240,6 +240,32 @@
|
||||
"skillPreviewTooltip": "Preview SKILL.md",
|
||||
"skillPreviewNotFound": "Could not find this skill. Open the skill picker to refresh the list."
|
||||
},
|
||||
"talk": {
|
||||
"start": "Start Talk",
|
||||
"stop": "Stop Talk",
|
||||
"inputLevel": "Microphone input level",
|
||||
"status": {
|
||||
"idle": "Talk is ready",
|
||||
"connecting": "Connecting Talk",
|
||||
"listening": "Listening",
|
||||
"thinking": "Thinking",
|
||||
"speaking": "Speaking",
|
||||
"disconnected": "Talk disconnected",
|
||||
"error": "Talk error"
|
||||
},
|
||||
"unavailable": {
|
||||
"gateway": "Talk requires a connected Gateway",
|
||||
"sending": "Talk is unavailable while a message is sending",
|
||||
"session": "Select a chat session to start Talk",
|
||||
"configuration": "Talk is unavailable: {{reason}}",
|
||||
"loading": "Checking Talk availability",
|
||||
"default": "Talk is unavailable"
|
||||
},
|
||||
"transcript": {
|
||||
"label": "Live Talk transcript",
|
||||
"microphone": "Microphone"
|
||||
}
|
||||
},
|
||||
"sessionList": {
|
||||
"title": "Sessions",
|
||||
"collapseAll": "Collapse all",
|
||||
|
||||
@@ -157,6 +157,25 @@
|
||||
"proxySaved": "Proxy settings saved",
|
||||
"proxySaveFailed": "Failed to save proxy settings"
|
||||
},
|
||||
"talk": {
|
||||
"title": "Talk",
|
||||
"description": "Configure realtime Talk through the OpenClaw Gateway Relay.",
|
||||
"refresh": "Refresh Talk availability",
|
||||
"provider": "Realtime provider",
|
||||
"model": "Realtime model",
|
||||
"speakerVoice": "Speaker voice",
|
||||
"voiceUnavailable": "No speaker voices are available for this provider.",
|
||||
"readiness": {
|
||||
"ready": "Talk is ready",
|
||||
"unavailable": "Talk is unavailable"
|
||||
},
|
||||
"developerGuidance": "Configure provider credentials in OpenClaw Developer settings",
|
||||
"save": "Save Talk settings",
|
||||
"saving": "Saving Talk settings...",
|
||||
"saved": "Talk settings saved",
|
||||
"loadFailed": "Could not load Talk settings",
|
||||
"saveFailed": "Could not save Talk settings"
|
||||
},
|
||||
"updates": {
|
||||
"title": "Updates",
|
||||
"description": "Keep ClawX up to date",
|
||||
|
||||
@@ -240,6 +240,32 @@
|
||||
"skillPreviewTooltip": "SKILL.md をプレビュー",
|
||||
"skillPreviewNotFound": "この Skill が見つかりません。Skill ピッカーを開いて一覧を更新してください。"
|
||||
},
|
||||
"talk": {
|
||||
"start": "Talk を開始",
|
||||
"stop": "Talk を停止",
|
||||
"inputLevel": "マイク入力レベル",
|
||||
"status": {
|
||||
"idle": "Talk の準備ができました",
|
||||
"connecting": "Talk に接続中",
|
||||
"listening": "聞いています",
|
||||
"thinking": "考えています",
|
||||
"speaking": "話しています",
|
||||
"disconnected": "Talk が切断されました",
|
||||
"error": "Talk エラー"
|
||||
},
|
||||
"unavailable": {
|
||||
"gateway": "Talk には接続済みの Gateway が必要です",
|
||||
"sending": "メッセージ送信中は Talk を利用できません",
|
||||
"session": "Talk を開始するチャットセッションを選択してください",
|
||||
"configuration": "Talk は利用できません: {{reason}}",
|
||||
"loading": "Talk の利用可否を確認中",
|
||||
"default": "Talk は利用できません"
|
||||
},
|
||||
"transcript": {
|
||||
"label": "ライブ Talk の文字起こし",
|
||||
"microphone": "マイク"
|
||||
}
|
||||
},
|
||||
"sessionList": {
|
||||
"title": "セッション一覧",
|
||||
"collapseAll": "すべて折りたたむ",
|
||||
|
||||
@@ -157,6 +157,25 @@
|
||||
"proxySaved": "プロキシ設定を保存しました",
|
||||
"proxySaveFailed": "プロキシ設定の保存に失敗しました"
|
||||
},
|
||||
"talk": {
|
||||
"title": "Talk",
|
||||
"description": "OpenClaw Gateway Relay 経由でリアルタイム Talk を設定します。",
|
||||
"refresh": "Talk の利用可否を更新",
|
||||
"provider": "リアルタイムプロバイダー",
|
||||
"model": "リアルタイムモデル",
|
||||
"speakerVoice": "話者の音声",
|
||||
"voiceUnavailable": "このプロバイダーで利用できる話者の音声はありません。",
|
||||
"readiness": {
|
||||
"ready": "Talk の準備ができました",
|
||||
"unavailable": "Talk は利用できません"
|
||||
},
|
||||
"developerGuidance": "OpenClaw の開発者設定でプロバイダー認証情報を設定",
|
||||
"save": "Talk 設定を保存",
|
||||
"saving": "Talk 設定を保存中...",
|
||||
"saved": "Talk 設定を保存しました",
|
||||
"loadFailed": "Talk 設定を読み込めませんでした",
|
||||
"saveFailed": "Talk 設定を保存できませんでした"
|
||||
},
|
||||
"updates": {
|
||||
"title": "アップデート",
|
||||
"description": "ClawX を最新に保つ",
|
||||
|
||||
@@ -240,6 +240,32 @@
|
||||
"skillPreviewTooltip": "Предпросмотр SKILL.md",
|
||||
"skillPreviewNotFound": "Skill не найден. Откройте список Skill, чтобы обновить его."
|
||||
},
|
||||
"talk": {
|
||||
"start": "Начать Talk",
|
||||
"stop": "Остановить Talk",
|
||||
"inputLevel": "Уровень входного сигнала микрофона",
|
||||
"status": {
|
||||
"idle": "Talk готов",
|
||||
"connecting": "Подключение Talk",
|
||||
"listening": "Слушаю",
|
||||
"thinking": "Думаю",
|
||||
"speaking": "Говорю",
|
||||
"disconnected": "Talk отключён",
|
||||
"error": "Ошибка Talk"
|
||||
},
|
||||
"unavailable": {
|
||||
"gateway": "Для Talk требуется подключённый шлюз",
|
||||
"sending": "Talk недоступен во время отправки сообщения",
|
||||
"session": "Выберите сеанс чата, чтобы начать Talk",
|
||||
"configuration": "Talk недоступен: {{reason}}",
|
||||
"loading": "Проверка доступности Talk",
|
||||
"default": "Talk недоступен"
|
||||
},
|
||||
"transcript": {
|
||||
"label": "Расшифровка Talk в реальном времени",
|
||||
"microphone": "Микрофон"
|
||||
}
|
||||
},
|
||||
"sessionList": {
|
||||
"title": "Список сессий",
|
||||
"collapseAll": "Свернуть все",
|
||||
|
||||
@@ -157,6 +157,25 @@
|
||||
"proxySaved": "Настройки прокси сохранены",
|
||||
"proxySaveFailed": "Не удалось сохранить настройки прокси"
|
||||
},
|
||||
"talk": {
|
||||
"title": "Talk",
|
||||
"description": "Настройте Talk в реальном времени через OpenClaw Gateway Relay.",
|
||||
"refresh": "Обновить доступность Talk",
|
||||
"provider": "Провайдер реального времени",
|
||||
"model": "Модель реального времени",
|
||||
"speakerVoice": "Голос",
|
||||
"voiceUnavailable": "Для этого провайдера нет доступных голосов.",
|
||||
"readiness": {
|
||||
"ready": "Talk готов",
|
||||
"unavailable": "Talk недоступен"
|
||||
},
|
||||
"developerGuidance": "Настройте учётные данные провайдера в настройках разработчика OpenClaw",
|
||||
"save": "Сохранить настройки Talk",
|
||||
"saving": "Сохранение настроек Talk...",
|
||||
"saved": "Настройки Talk сохранены",
|
||||
"loadFailed": "Не удалось загрузить настройки Talk",
|
||||
"saveFailed": "Не удалось сохранить настройки Talk"
|
||||
},
|
||||
"updates": {
|
||||
"title": "Обновления",
|
||||
"description": "Поддерживайте ClawX в актуальном состоянии",
|
||||
|
||||
@@ -240,6 +240,32 @@
|
||||
"skillPreviewTooltip": "预览 SKILL.md",
|
||||
"skillPreviewNotFound": "找不到该技能。请打开技能列表以刷新。"
|
||||
},
|
||||
"talk": {
|
||||
"start": "开始语音对话",
|
||||
"stop": "停止语音对话",
|
||||
"inputLevel": "麦克风输入音量",
|
||||
"status": {
|
||||
"idle": "语音对话已就绪",
|
||||
"connecting": "正在连接语音对话",
|
||||
"listening": "正在聆听",
|
||||
"thinking": "正在思考",
|
||||
"speaking": "正在讲话",
|
||||
"disconnected": "语音对话已断开",
|
||||
"error": "语音对话出错"
|
||||
},
|
||||
"unavailable": {
|
||||
"gateway": "语音对话需要已连接的网关",
|
||||
"sending": "发送消息时无法使用语音对话",
|
||||
"session": "请选择聊天会话后再开始语音对话",
|
||||
"configuration": "语音对话不可用:{{reason}}",
|
||||
"loading": "正在检查语音对话可用性",
|
||||
"default": "语音对话不可用"
|
||||
},
|
||||
"transcript": {
|
||||
"label": "实时语音转写",
|
||||
"microphone": "麦克风"
|
||||
}
|
||||
},
|
||||
"sessionList": {
|
||||
"title": "会话列表",
|
||||
"collapseAll": "一键折叠",
|
||||
|
||||
@@ -157,6 +157,25 @@
|
||||
"proxySaved": "代理设置已保存",
|
||||
"proxySaveFailed": "保存代理设置失败"
|
||||
},
|
||||
"talk": {
|
||||
"title": "语音对话",
|
||||
"description": "通过 OpenClaw 网关中继配置实时语音对话。",
|
||||
"refresh": "刷新语音对话可用性",
|
||||
"provider": "实时提供商",
|
||||
"model": "实时模型",
|
||||
"speakerVoice": "语音音色",
|
||||
"voiceUnavailable": "此提供商没有可用的语音音色。",
|
||||
"readiness": {
|
||||
"ready": "语音对话已就绪",
|
||||
"unavailable": "语音对话不可用"
|
||||
},
|
||||
"developerGuidance": "在 OpenClaw 开发者设置中配置提供商凭据",
|
||||
"save": "保存语音对话设置",
|
||||
"saving": "正在保存语音对话设置...",
|
||||
"saved": "语音对话设置已保存",
|
||||
"loadFailed": "无法加载语音对话设置",
|
||||
"saveFailed": "无法保存语音对话设置"
|
||||
},
|
||||
"updates": {
|
||||
"title": "更新",
|
||||
"description": "保持 ClawX 最新",
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
export type TalkTerminalReason = 'completed' | 'cancelled' | 'error' | 'disconnected' | (string & {});
|
||||
|
||||
export type TalkCatalogReadiness =
|
||||
| { ready: true }
|
||||
| { ready: false; reason?: string }
|
||||
| { ready?: undefined };
|
||||
|
||||
export type TalkCatalogProvider = {
|
||||
id: string;
|
||||
label: string;
|
||||
configured: boolean;
|
||||
models?: string[];
|
||||
defaultModel?: string;
|
||||
voices?: string[];
|
||||
transports?: string[];
|
||||
brains?: string[];
|
||||
};
|
||||
|
||||
export type TalkCatalog = {
|
||||
modes: string[];
|
||||
transports: string[];
|
||||
brains: string[];
|
||||
realtime: TalkCatalogReadiness & {
|
||||
activeProvider?: string;
|
||||
providers: TalkCatalogProvider[];
|
||||
};
|
||||
};
|
||||
|
||||
export type TalkPcm16AudioContract = {
|
||||
inputEncoding: 'pcm16';
|
||||
inputSampleRateHz: number;
|
||||
outputEncoding: 'pcm16';
|
||||
outputSampleRateHz: number;
|
||||
};
|
||||
|
||||
export type TalkRelaySession = {
|
||||
relaySessionId: string;
|
||||
provider: string;
|
||||
transport: 'gateway-relay';
|
||||
audio: TalkPcm16AudioContract;
|
||||
model?: string;
|
||||
voice?: string;
|
||||
expiresAt?: number;
|
||||
};
|
||||
|
||||
export type TalkRelayEventBase = {
|
||||
relaySessionId: string;
|
||||
talkEvent?: unknown;
|
||||
};
|
||||
|
||||
export type TalkRelayEvent = TalkRelayEventBase & (
|
||||
| { type: 'ready' }
|
||||
| { type: 'audio'; audioBase64: string }
|
||||
| { type: 'clear'; reason?: 'barge-in' }
|
||||
| { type: 'mark'; markName: string }
|
||||
| { type: 'transcript'; role: 'user' | 'assistant'; text: string; final: boolean }
|
||||
| { type: 'toolCall'; callId: string; name: string; args: Record<string, unknown>; forced?: boolean }
|
||||
| { type: 'toolCallCancelled'; callId: string }
|
||||
| { type: 'toolResult'; callId: string; final?: boolean }
|
||||
| { type: 'error'; message: string }
|
||||
| { type: 'close'; reason: TalkTerminalReason }
|
||||
);
|
||||
|
||||
export type TalkStartRelayPayload = { sessionKey: string };
|
||||
export type TalkRealtimeSettingsPayload = {
|
||||
provider: string;
|
||||
model: string;
|
||||
speakerVoice?: string;
|
||||
};
|
||||
export type TalkRelayIdPayload = { relaySessionId: string };
|
||||
export type TalkAppendAudioPayload = TalkRelayIdPayload & {
|
||||
audioBase64: string;
|
||||
timestamp?: number;
|
||||
};
|
||||
export type TalkSubmitToolResultPayload = TalkRelayIdPayload & {
|
||||
callId: string;
|
||||
result: unknown;
|
||||
options?: { suppressResponse?: boolean; willContinue?: boolean };
|
||||
};
|
||||
export type TalkAcknowledgeMarkPayload = TalkRelayIdPayload & { markName: string };
|
||||
export type TalkStartAgentConsultPayload = TalkRelayIdPayload & {
|
||||
sessionKey: string;
|
||||
callId: string;
|
||||
args: Record<string, unknown>;
|
||||
};
|
||||
export type TalkOperationResult = { ok: true };
|
||||
export type TalkAgentConsultResult = { runId: string; idempotencyKey: string };
|
||||
@@ -1,74 +0,0 @@
|
||||
/**
|
||||
* Error Boundary Component
|
||||
* Catches and displays errors in the component tree
|
||||
*/
|
||||
import { Component, ReactNode } from 'react';
|
||||
import { AlertTriangle, RefreshCw } from 'lucide-react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from '@/components/ui/card';
|
||||
|
||||
interface Props {
|
||||
children: ReactNode;
|
||||
fallback?: ReactNode;
|
||||
}
|
||||
|
||||
interface State {
|
||||
hasError: boolean;
|
||||
error: Error | null;
|
||||
}
|
||||
|
||||
export class ErrorBoundary extends Component<Props, State> {
|
||||
constructor(props: Props) {
|
||||
super(props);
|
||||
this.state = { hasError: false, error: null };
|
||||
}
|
||||
|
||||
static getDerivedStateFromError(error: Error): State {
|
||||
return { hasError: true, error };
|
||||
}
|
||||
|
||||
componentDidCatch(error: Error, errorInfo: React.ErrorInfo) {
|
||||
console.error('Error caught by boundary:', error, errorInfo);
|
||||
}
|
||||
|
||||
handleReset = () => {
|
||||
this.setState({ hasError: false, error: null });
|
||||
};
|
||||
|
||||
render() {
|
||||
if (this.state.hasError) {
|
||||
if (this.props.fallback) {
|
||||
return this.props.fallback;
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="flex h-full items-center justify-center p-6">
|
||||
<Card className="max-w-md">
|
||||
<CardHeader>
|
||||
<div className="flex items-center gap-2">
|
||||
<AlertTriangle className="h-6 w-6 text-destructive" />
|
||||
<CardTitle>Something went wrong</CardTitle>
|
||||
</div>
|
||||
<CardDescription>
|
||||
An unexpected error occurred. Please try again.
|
||||
</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent className="space-y-4">
|
||||
{this.state.error && (
|
||||
<pre className="rounded-lg bg-surface-input p-4 text-sm overflow-auto max-h-40">
|
||||
{this.state.error.message}
|
||||
</pre>
|
||||
)}
|
||||
<Button onClick={this.handleReset} className="w-full">
|
||||
<RefreshCw className="mr-2 h-4 w-4" />
|
||||
Try Again
|
||||
</Button>
|
||||
</CardContent>
|
||||
</Card>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
return this.props.children;
|
||||
}
|
||||
}
|
||||
@@ -1,47 +0,0 @@
|
||||
/**
|
||||
* Status Badge Component
|
||||
* Displays connection/state status with color coding
|
||||
*/
|
||||
import { cn } from '@/lib/utils';
|
||||
import { Badge } from '@/components/ui/badge';
|
||||
|
||||
export type Status = 'connected' | 'disconnected' | 'connecting' | 'error' | 'running' | 'stopped' | 'starting' | 'reconnecting';
|
||||
|
||||
interface StatusBadgeProps {
|
||||
status: Status;
|
||||
label?: string;
|
||||
showDot?: boolean;
|
||||
}
|
||||
|
||||
const statusConfig: Record<Status, { label: string; variant: 'success' | 'secondary' | 'warning' | 'destructive' }> = {
|
||||
connected: { label: 'Connected', variant: 'success' },
|
||||
running: { label: 'Running', variant: 'success' },
|
||||
disconnected: { label: 'Disconnected', variant: 'secondary' },
|
||||
stopped: { label: 'Stopped', variant: 'secondary' },
|
||||
connecting: { label: 'Connecting', variant: 'warning' },
|
||||
starting: { label: 'Starting', variant: 'warning' },
|
||||
reconnecting: { label: 'Reconnecting', variant: 'warning' },
|
||||
error: { label: 'Error', variant: 'destructive' },
|
||||
};
|
||||
|
||||
export function StatusBadge({ status, label, showDot = true }: StatusBadgeProps) {
|
||||
const config = statusConfig[status];
|
||||
const displayLabel = label || config.label;
|
||||
|
||||
return (
|
||||
<Badge variant={config.variant} className="gap-1.5">
|
||||
{showDot && (
|
||||
<span
|
||||
className={cn(
|
||||
'h-1.5 w-1.5 rounded-full',
|
||||
config.variant === 'success' && 'bg-green-600',
|
||||
config.variant === 'secondary' && 'bg-muted-foreground',
|
||||
config.variant === 'warning' && 'bg-yellow-600 animate-pulse',
|
||||
config.variant === 'destructive' && 'bg-red-600'
|
||||
)}
|
||||
/>
|
||||
)}
|
||||
{displayLabel}
|
||||
</Badge>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,195 @@
|
||||
import { useEffect, useState } from 'react';
|
||||
import { Loader2, RefreshCw } from 'lucide-react';
|
||||
import { useTranslation } from 'react-i18next';
|
||||
import { toast } from 'sonner';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Label } from '@/components/ui/label';
|
||||
import { Select } from '@/components/ui/select';
|
||||
import { hostApi, type TalkCatalog } from '@/lib/host-api';
|
||||
import { toUserMessage } from '@/lib/error-message';
|
||||
|
||||
type Selection = { provider: string; model: string; speakerVoice?: string };
|
||||
|
||||
function modelChoices(provider: TalkCatalog['realtime']['providers'][number] | undefined): string[] {
|
||||
if (!provider) return [];
|
||||
return provider.models?.length ? provider.models : provider.defaultModel ? [provider.defaultModel] : [];
|
||||
}
|
||||
|
||||
function selectionFromCatalog(catalog: TalkCatalog, current?: Selection): Selection {
|
||||
const providers = catalog.realtime.providers.filter((provider) => provider.configured);
|
||||
const provider = providers.find((candidate) => candidate.id === current?.provider)
|
||||
?? providers.find((candidate) => candidate.id === catalog.realtime.activeProvider)
|
||||
?? providers[0];
|
||||
const models = modelChoices(provider);
|
||||
const model = models.includes(current?.model ?? '')
|
||||
? current!.model
|
||||
: models[0] ?? '';
|
||||
const speakerVoice = provider?.voices?.includes(current?.speakerVoice ?? '')
|
||||
? current!.speakerVoice
|
||||
: provider?.voices?.[0];
|
||||
return {
|
||||
provider: provider?.id ?? '',
|
||||
model,
|
||||
...(speakerVoice ? { speakerVoice } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
export function TalkSettings() {
|
||||
const { t } = useTranslation('settings');
|
||||
const [catalog, setCatalog] = useState<TalkCatalog | null>(null);
|
||||
const [selection, setSelection] = useState<Selection>({ provider: '', model: '', speakerVoice: '' });
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [saving, setSaving] = useState(false);
|
||||
|
||||
const load = async () => {
|
||||
setLoading(true);
|
||||
try {
|
||||
const nextCatalog = await hostApi.talk.catalog();
|
||||
setCatalog(nextCatalog);
|
||||
setSelection((current) => selectionFromCatalog(nextCatalog, current));
|
||||
} catch (error) {
|
||||
toast.error(`${t('talk.loadFailed')}: ${toUserMessage(error)}`);
|
||||
setCatalog(null);
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
};
|
||||
|
||||
useEffect(() => {
|
||||
void load();
|
||||
}, []);
|
||||
|
||||
const provider = catalog?.realtime.providers.find((candidate) => candidate.id === selection.provider);
|
||||
const configuredProviders = catalog?.realtime.providers.filter((candidate) => candidate.configured) ?? [];
|
||||
const models = modelChoices(provider);
|
||||
const voiceChoices = provider?.voices ?? [];
|
||||
const canSave = Boolean(selection.provider && selection.model && (!voiceChoices.length || selection.speakerVoice));
|
||||
|
||||
const handleProviderChange = (providerId: string) => {
|
||||
const nextProvider = configuredProviders.find((candidate) => candidate.id === providerId);
|
||||
setSelection({
|
||||
provider: providerId,
|
||||
model: modelChoices(nextProvider)[0] ?? '',
|
||||
...(nextProvider?.voices?.length ? { speakerVoice: nextProvider.voices[0] } : {}),
|
||||
});
|
||||
};
|
||||
|
||||
const handleSave = async () => {
|
||||
if (!canSave) return;
|
||||
setSaving(true);
|
||||
try {
|
||||
await hostApi.talk.updateRealtimeSettings(selection);
|
||||
toast.success(t('talk.saved'));
|
||||
await load();
|
||||
} catch (error) {
|
||||
toast.error(`${t('talk.saveFailed')}: ${toUserMessage(error)}`);
|
||||
} finally {
|
||||
setSaving(false);
|
||||
}
|
||||
};
|
||||
|
||||
return (
|
||||
<section id="talk" tabIndex={-1} data-testid="talk-settings" className="space-y-6 scroll-mt-6">
|
||||
<div className="flex items-start justify-between gap-4">
|
||||
<div>
|
||||
<h2 className="text-3xl font-serif text-foreground font-normal tracking-tight">{t('talk.title')}</h2>
|
||||
<p className="text-meta text-muted-foreground mt-2">{t('talk.description')}</p>
|
||||
</div>
|
||||
<Button
|
||||
type="button"
|
||||
variant="outline"
|
||||
size="sm"
|
||||
className="rounded-full shrink-0"
|
||||
onClick={() => void load()}
|
||||
disabled={loading || saving}
|
||||
data-testid="talk-settings-refresh"
|
||||
aria-label={t('talk.refresh')}
|
||||
>
|
||||
<RefreshCw className={`h-4 w-4${loading ? ' animate-spin' : ''}`} />
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{loading ? (
|
||||
<div className="flex items-center justify-center py-10 text-muted-foreground bg-black/5 dark:bg-white/5 rounded-2xl">
|
||||
<Loader2 className="h-5 w-5 animate-spin" />
|
||||
</div>
|
||||
) : (
|
||||
<div className="space-y-5 rounded-2xl border border-black/10 dark:border-white/10 bg-surface-modal p-5">
|
||||
<p className="text-meta text-muted-foreground" data-testid="talk-settings-readiness">
|
||||
{catalog?.realtime.ready === true ? t('talk.readiness.ready') : t('talk.readiness.unavailable')}
|
||||
</p>
|
||||
{catalog?.realtime.ready === false && catalog.realtime.reason ? (
|
||||
<p className="text-meta text-amber-700 dark:text-amber-400" data-testid="talk-settings-unavailable-reason">
|
||||
{catalog.realtime.reason}
|
||||
</p>
|
||||
) : null}
|
||||
|
||||
<div className="grid gap-4 md:grid-cols-3">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="talk-provider">{t('talk.provider')}</Label>
|
||||
<Select
|
||||
id="talk-provider"
|
||||
value={selection.provider}
|
||||
onChange={(event) => handleProviderChange(event.target.value)}
|
||||
disabled={configuredProviders.length === 0 || saving}
|
||||
data-testid="talk-settings-provider"
|
||||
>
|
||||
{configuredProviders.map((candidate) => <option key={candidate.id} value={candidate.id}>{candidate.label}</option>)}
|
||||
</Select>
|
||||
</div>
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="talk-model">{t('talk.model')}</Label>
|
||||
<Select
|
||||
id="talk-model"
|
||||
value={selection.model}
|
||||
onChange={(event) => setSelection((current) => ({ ...current, model: event.target.value }))}
|
||||
disabled={!models.length || saving}
|
||||
data-testid="talk-settings-model"
|
||||
>
|
||||
{models.map((model) => <option key={model} value={model}>{model}</option>)}
|
||||
</Select>
|
||||
</div>
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="talk-speaker-voice">{t('talk.speakerVoice')}</Label>
|
||||
{voiceChoices.length ? (
|
||||
<Select
|
||||
id="talk-speaker-voice"
|
||||
value={selection.speakerVoice ?? ''}
|
||||
onChange={(event) => setSelection((current) => ({ ...current, speakerVoice: event.target.value }))}
|
||||
disabled={saving}
|
||||
data-testid="talk-settings-voice"
|
||||
>
|
||||
{voiceChoices.map((voice) => <option key={voice} value={voice}>{voice}</option>)}
|
||||
</Select>
|
||||
) : (
|
||||
<p className="text-meta text-muted-foreground" data-testid="talk-settings-voice-unavailable">
|
||||
{t('talk.voiceUnavailable')}
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="flex flex-wrap items-center justify-between gap-3 border-t border-black/5 dark:border-white/5 pt-4">
|
||||
<a
|
||||
href="#/settings?section=developer"
|
||||
className="text-tiny text-muted-foreground underline underline-offset-4 hover:text-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring rounded-sm"
|
||||
data-testid="talk-settings-developer-link"
|
||||
>
|
||||
{t('talk.developerGuidance')}
|
||||
</a>
|
||||
<Button
|
||||
type="button"
|
||||
className="rounded-full"
|
||||
onClick={() => void handleSave()}
|
||||
disabled={!canSave || saving}
|
||||
data-testid="talk-settings-save"
|
||||
>
|
||||
{saving ? <Loader2 className="h-4 w-4 mr-2 animate-spin" /> : null}
|
||||
{saving ? t('talk.saving') : t('talk.save')}
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</section>
|
||||
);
|
||||
}
|
||||
@@ -1,49 +0,0 @@
|
||||
import * as React from 'react';
|
||||
import * as DropdownMenuPrimitive from '@radix-ui/react-dropdown-menu';
|
||||
import { cn } from '@/lib/utils';
|
||||
|
||||
const DropdownMenu = DropdownMenuPrimitive.Root;
|
||||
const DropdownMenuTrigger = DropdownMenuPrimitive.Trigger;
|
||||
|
||||
const DropdownMenuContent = React.forwardRef<
|
||||
React.ComponentRef<typeof DropdownMenuPrimitive.Content>,
|
||||
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Content>
|
||||
>(({ className, sideOffset = 4, ...props }, ref) => (
|
||||
<DropdownMenuPrimitive.Portal>
|
||||
<DropdownMenuPrimitive.Content
|
||||
ref={ref}
|
||||
sideOffset={sideOffset}
|
||||
className={cn(
|
||||
'z-50 min-w-44 overflow-hidden rounded-md border border-border bg-surface-modal p-1 text-popover-foreground shadow-md',
|
||||
'data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95',
|
||||
className,
|
||||
)}
|
||||
{...props}
|
||||
/>
|
||||
</DropdownMenuPrimitive.Portal>
|
||||
));
|
||||
DropdownMenuContent.displayName = DropdownMenuPrimitive.Content.displayName;
|
||||
|
||||
const DropdownMenuItem = React.forwardRef<
|
||||
React.ComponentRef<typeof DropdownMenuPrimitive.Item>,
|
||||
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Item>
|
||||
>(({ className, ...props }, ref) => (
|
||||
<DropdownMenuPrimitive.Item
|
||||
ref={ref}
|
||||
className={cn(
|
||||
'relative flex cursor-default select-none items-center rounded-sm px-2 py-1.5 text-sm outline-none transition-colors',
|
||||
'focus:bg-black/5 focus:text-foreground dark:focus:bg-white/10',
|
||||
'data-[disabled]:pointer-events-none data-[disabled]:opacity-50',
|
||||
className,
|
||||
)}
|
||||
{...props}
|
||||
/>
|
||||
));
|
||||
DropdownMenuItem.displayName = DropdownMenuPrimitive.Item.displayName;
|
||||
|
||||
export {
|
||||
DropdownMenu,
|
||||
DropdownMenuContent,
|
||||
DropdownMenuItem,
|
||||
DropdownMenuTrigger,
|
||||
};
|
||||
@@ -1,5 +1,4 @@
|
||||
export { rendererExtensionRegistry } from './registry';
|
||||
export { registerRendererExtensionModule, loadRendererExtensions } from './loader';
|
||||
export type {
|
||||
RendererExtension,
|
||||
NavItemDef,
|
||||
|
||||
@@ -1,34 +0,0 @@
|
||||
import { rendererExtensionRegistry } from './registry';
|
||||
import type { RendererExtension } from './types';
|
||||
|
||||
interface RendererExtensionManifest {
|
||||
extensions?: {
|
||||
renderer?: string[];
|
||||
};
|
||||
}
|
||||
|
||||
const registeredModules = new Map<string, () => RendererExtension>();
|
||||
|
||||
export function registerRendererExtensionModule(id: string, factory: () => RendererExtension): void {
|
||||
registeredModules.set(id, factory);
|
||||
}
|
||||
|
||||
export function loadRendererExtensions(manifest?: RendererExtensionManifest): void {
|
||||
const extensionIds = manifest?.extensions?.renderer;
|
||||
|
||||
if (!extensionIds || extensionIds.length === 0) {
|
||||
for (const [, factory] of registeredModules) {
|
||||
rendererExtensionRegistry.register(factory());
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
for (const id of extensionIds) {
|
||||
const factory = registeredModules.get(id);
|
||||
if (factory) {
|
||||
rendererExtensionRegistry.register(factory());
|
||||
} else {
|
||||
console.warn(`[extensions] Renderer extension "${id}" not found in registered modules`);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,45 +0,0 @@
|
||||
import { useState, useRef, useEffect } from 'react';
|
||||
|
||||
/**
|
||||
* A hook that ensures a loading state remains true for at least a minimum duration (e.g., 1000ms),
|
||||
* preventing flickering for very fast loading times.
|
||||
*
|
||||
* @param isLoading - The actual loading state from the data source
|
||||
* @param minDurationMs - Minimum duration to show loading (default: 1000)
|
||||
*/
|
||||
export function useMinLoading(isLoading: boolean, minDurationMs: number = 500) {
|
||||
const [showLoading, setShowLoading] = useState(isLoading);
|
||||
const startTime = useRef<number>(0);
|
||||
|
||||
// Guarantee that the loading UI activates immediately without any asynchronous delay
|
||||
if (isLoading && !showLoading) {
|
||||
setShowLoading(true);
|
||||
}
|
||||
|
||||
// Record the actual timestamp in an effect to respect React purity rules
|
||||
useEffect(() => {
|
||||
if (isLoading && startTime.current === 0) {
|
||||
startTime.current = Date.now();
|
||||
}
|
||||
}, [isLoading]);
|
||||
|
||||
useEffect(() => {
|
||||
let timeout: NodeJS.Timeout;
|
||||
|
||||
if (!isLoading && showLoading) {
|
||||
const elapsed = startTime.current > 0 ? Date.now() - startTime.current : 0;
|
||||
const remaining = Math.max(0, minDurationMs - elapsed);
|
||||
|
||||
timeout = setTimeout(() => {
|
||||
setShowLoading(false);
|
||||
startTime.current = 0;
|
||||
}, remaining);
|
||||
}
|
||||
|
||||
return () => {
|
||||
if (timeout) clearTimeout(timeout);
|
||||
};
|
||||
}, [isLoading, showLoading, minDurationMs]);
|
||||
|
||||
return isLoading || showLoading;
|
||||
}
|
||||
@@ -47,6 +47,15 @@ import type {
|
||||
AcpChatRespondPermissionPayload,
|
||||
} from '@shared/acp-chat/types';
|
||||
import type { CronJobCreateInput, CronJobUpdateInput } from '@shared/types/cron';
|
||||
import type {
|
||||
TalkAcknowledgeMarkPayload,
|
||||
TalkAppendAudioPayload,
|
||||
TalkRelayIdPayload,
|
||||
TalkRealtimeSettingsPayload,
|
||||
TalkStartAgentConsultPayload,
|
||||
TalkStartRelayPayload,
|
||||
TalkSubmitToolResultPayload,
|
||||
} from '@shared/talk/types';
|
||||
import { invokeHost } from './host-api-client';
|
||||
|
||||
export type {
|
||||
@@ -102,6 +111,13 @@ export type {
|
||||
WorkspaceNativeFileResult,
|
||||
WorkspaceOpenHandlersResult,
|
||||
} from '@shared/host-api/contract';
|
||||
export type {
|
||||
TalkAgentConsultResult,
|
||||
TalkCatalog,
|
||||
TalkOperationResult,
|
||||
TalkRelayEvent,
|
||||
TalkRelaySession,
|
||||
} from '@shared/talk/types';
|
||||
|
||||
export const hostApi = {
|
||||
app: {
|
||||
@@ -172,6 +188,19 @@ export const hostApi = {
|
||||
invokeHost('gateway', 'rpc', { method, params, timeoutMs }) as Promise<T>
|
||||
),
|
||||
},
|
||||
talk: {
|
||||
catalog: () => invokeHost('talk', 'catalog'),
|
||||
updateRealtimeSettings: (input: TalkRealtimeSettingsPayload) => (
|
||||
invokeHost('talk', 'updateRealtimeSettings', input)
|
||||
),
|
||||
startRelay: (input: TalkStartRelayPayload) => invokeHost('talk', 'startRelay', input),
|
||||
appendAudio: (input: TalkAppendAudioPayload) => invokeHost('talk', 'appendAudio', input),
|
||||
cancelOutput: (input: TalkRelayIdPayload) => invokeHost('talk', 'cancelOutput', input),
|
||||
submitToolResult: (input: TalkSubmitToolResultPayload) => invokeHost('talk', 'submitToolResult', input),
|
||||
acknowledgeMark: (input: TalkAcknowledgeMarkPayload) => invokeHost('talk', 'acknowledgeMark', input),
|
||||
stopRelay: (input: TalkRelayIdPayload) => invokeHost('talk', 'stopRelay', input),
|
||||
startAgentConsult: (input: TalkStartAgentConsultPayload) => invokeHost('talk', 'startAgentConsult', input),
|
||||
},
|
||||
logs: {
|
||||
recent: (tailLines = 100) => invokeHost('logs', 'recent', { tailLines }),
|
||||
dir: () => invokeHost('logs', 'dir'),
|
||||
|
||||
@@ -40,6 +40,11 @@ const onChatEvent = <E extends HostEventName<'chat'>>(
|
||||
handler: HostEventHandler<'chat', E>,
|
||||
) => onIpc(HOST_EVENT_CHANNELS.chat[event], handler);
|
||||
|
||||
const onTalkEvent = <E extends HostEventName<'talk'>>(
|
||||
event: E,
|
||||
handler: HostEventHandler<'talk', E>,
|
||||
) => onIpc(HOST_EVENT_CHANNELS.talk[event], handler);
|
||||
|
||||
const onOAuthEvent = <E extends HostEventName<'oauth'>>(
|
||||
event: E,
|
||||
handler: HostEventHandler<'oauth', E>,
|
||||
@@ -98,6 +103,7 @@ export const hostEvents = {
|
||||
onAcpPermissionRequest: (handler: HostEventHandler<'chat', 'acpPermissionRequest'>) => (
|
||||
onChatEvent('acpPermissionRequest', handler)
|
||||
),
|
||||
onTalkEvent: (handler: HostEventHandler<'talk', 'event'>) => onTalkEvent('event', handler),
|
||||
onOAuthCode: (handler: HostEventHandler<'oauth', 'code'>) => onOAuthEvent('code', handler),
|
||||
onOAuthSuccess: (handler: HostEventHandler<'oauth', 'success'>) => onOAuthEvent('success', handler),
|
||||
onOAuthError: (handler: HostEventHandler<'oauth', 'error'>) => onOAuthEvent('error', handler),
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
export type MicrophoneBridge = {
|
||||
stop: () => void;
|
||||
};
|
||||
|
||||
const processorSource = `
|
||||
class ClawXTalkMicProcessor extends AudioWorkletProcessor {
|
||||
process(inputs) {
|
||||
const samples = inputs[0] && inputs[0][0];
|
||||
if (samples) {
|
||||
const copy = samples.slice();
|
||||
this.port.postMessage(copy, [copy.buffer]);
|
||||
}
|
||||
return true;
|
||||
}
|
||||
}
|
||||
registerProcessor('clawx-talk-mic', ClawXTalkMicProcessor);
|
||||
`;
|
||||
|
||||
export async function createMicrophoneBridge(
|
||||
context: AudioContext,
|
||||
onSamples: (samples: Float32Array) => void,
|
||||
): Promise<MicrophoneBridge> {
|
||||
if (!navigator.mediaDevices?.getUserMedia || !context.audioWorklet) {
|
||||
throw new Error('Realtime microphone audio is not supported');
|
||||
}
|
||||
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
|
||||
const url = URL.createObjectURL(new Blob([processorSource], { type: 'text/javascript' }));
|
||||
try {
|
||||
await context.audioWorklet.addModule(url);
|
||||
const source = context.createMediaStreamSource(stream);
|
||||
const worklet = new AudioWorkletNode(context, 'clawx-talk-mic');
|
||||
const silentOutput = context.createGain();
|
||||
silentOutput.gain.value = 0;
|
||||
worklet.port.onmessage = (event: MessageEvent<Float32Array>) => onSamples(event.data);
|
||||
source.connect(worklet);
|
||||
worklet.connect(silentOutput);
|
||||
silentOutput.connect(context.destination);
|
||||
return {
|
||||
stop: () => {
|
||||
source.disconnect();
|
||||
worklet.disconnect();
|
||||
silentOutput.disconnect();
|
||||
for (const track of stream.getTracks()) track.stop();
|
||||
},
|
||||
};
|
||||
} catch (error) {
|
||||
for (const track of stream.getTracks()) track.stop();
|
||||
throw error;
|
||||
} finally {
|
||||
URL.revokeObjectURL(url);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,173 @@
|
||||
export function float32ToPcm16(samples: Float32Array): Int16Array {
|
||||
const pcm = new Int16Array(samples.length);
|
||||
for (let index = 0; index < samples.length; index += 1) {
|
||||
const sample = Math.max(-1, Math.min(1, samples[index] ?? 0));
|
||||
pcm[index] = Math.trunc(sample < 0 ? sample * 0x8000 : sample * 0x7fff);
|
||||
}
|
||||
return pcm;
|
||||
}
|
||||
|
||||
function pcm16ToBytes(samples: Int16Array): Uint8Array {
|
||||
const bytes = new Uint8Array(samples.length * 2);
|
||||
const view = new DataView(bytes.buffer);
|
||||
for (let index = 0; index < samples.length; index += 1) {
|
||||
view.setInt16(index * 2, samples[index] ?? 0, true);
|
||||
}
|
||||
return bytes;
|
||||
}
|
||||
|
||||
export function pcm16ToBase64(samples: Int16Array): string {
|
||||
const bytes = pcm16ToBytes(samples);
|
||||
let binary = '';
|
||||
for (const byte of bytes) binary += String.fromCharCode(byte);
|
||||
return btoa(binary);
|
||||
}
|
||||
|
||||
export function decodePcm16Base64(audioBase64: string): Int16Array {
|
||||
if (
|
||||
typeof audioBase64 !== 'string'
|
||||
|| !/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/.test(audioBase64)
|
||||
) throw new Error('Invalid PCM16 base64 audio');
|
||||
|
||||
const paddingLength = audioBase64.endsWith('==') ? 2 : audioBase64.endsWith('=') ? 1 : 0;
|
||||
const decodedByteLength = (audioBase64.length / 4) * 3 - paddingLength;
|
||||
if (decodedByteLength > MAX_TALK_PLAYBACK_DECODED_BYTES) {
|
||||
throw new Error('Talk PCM chunk exceeds decoded size limit');
|
||||
}
|
||||
|
||||
let binary: string;
|
||||
try {
|
||||
binary = atob(audioBase64);
|
||||
} catch {
|
||||
throw new Error('Invalid PCM16 base64 audio');
|
||||
}
|
||||
if (binary.length === 0 || binary.length % 2 !== 0) throw new Error('Invalid PCM16 byte length');
|
||||
|
||||
const pcm = new Int16Array(binary.length / 2);
|
||||
for (let index = 0; index < pcm.length; index += 1) {
|
||||
const low = binary.charCodeAt(index * 2);
|
||||
const high = binary.charCodeAt(index * 2 + 1);
|
||||
pcm[index] = (high << 8) | low;
|
||||
}
|
||||
return pcm;
|
||||
}
|
||||
|
||||
export function pcm16ToFloat32(samples: Int16Array): Float32Array {
|
||||
const output = new Float32Array(samples.length);
|
||||
for (let index = 0; index < samples.length; index += 1) output[index] = (samples[index] ?? 0) / 0x8000;
|
||||
return output;
|
||||
}
|
||||
|
||||
export function inputLevel(samples: Float32Array): number {
|
||||
if (samples.length === 0) return 0;
|
||||
let sum = 0;
|
||||
for (const sample of samples) sum += sample * sample;
|
||||
return Math.min(1, Math.sqrt(sum / samples.length));
|
||||
}
|
||||
|
||||
function assertSupportedSampleRates(sourceSampleRateHz: number, targetSampleRateHz: number): void {
|
||||
if (
|
||||
!Number.isInteger(sourceSampleRateHz)
|
||||
|| !Number.isInteger(targetSampleRateHz)
|
||||
|| sourceSampleRateHz < 8_000
|
||||
|| sourceSampleRateHz > 192_000
|
||||
|| targetSampleRateHz < 8_000
|
||||
|| targetSampleRateHz > 192_000
|
||||
) throw new Error('Unsupported Talk audio sample rate');
|
||||
}
|
||||
|
||||
export class StreamingResampler {
|
||||
private buffer = new Float32Array(0);
|
||||
private position = 0;
|
||||
private readonly step: number;
|
||||
|
||||
constructor(
|
||||
private readonly sourceSampleRateHz: number,
|
||||
private readonly targetSampleRateHz: number,
|
||||
) {
|
||||
assertSupportedSampleRates(sourceSampleRateHz, targetSampleRateHz);
|
||||
this.step = sourceSampleRateHz / targetSampleRateHz;
|
||||
}
|
||||
|
||||
process(samples: Float32Array): Float32Array {
|
||||
if (samples.length === 0) return new Float32Array(0);
|
||||
const buffered = new Float32Array(this.buffer.length + samples.length);
|
||||
buffered.set(this.buffer);
|
||||
buffered.set(samples, this.buffer.length);
|
||||
if (this.sourceSampleRateHz === this.targetSampleRateHz) {
|
||||
this.buffer = new Float32Array(0);
|
||||
return buffered;
|
||||
}
|
||||
|
||||
const output: number[] = [];
|
||||
while (this.position + 1 < buffered.length) {
|
||||
const before = Math.floor(this.position);
|
||||
const fraction = this.position - before;
|
||||
output.push(
|
||||
(buffered[before] ?? 0) * (1 - fraction) + (buffered[before + 1] ?? 0) * fraction,
|
||||
);
|
||||
this.position += this.step;
|
||||
}
|
||||
// Keep the phase relative to the next frame even when it advances past this buffer.
|
||||
const consumed = Math.min(Math.floor(this.position), buffered.length);
|
||||
this.buffer = buffered.slice(consumed);
|
||||
this.position -= consumed;
|
||||
return new Float32Array(output);
|
||||
}
|
||||
}
|
||||
|
||||
// Relay PCM arrives in short frames. These caps prevent one malformed event from allocating
|
||||
// unbounded renderer memory while leaving ample room for normal OpenClaw speech responses.
|
||||
export const MAX_TALK_PLAYBACK_CHUNKS = 24;
|
||||
export const MAX_TALK_PLAYBACK_DURATION_SECONDS = 30;
|
||||
export const MAX_TALK_PLAYBACK_DECODED_BYTES = 2 * 1024 * 1024;
|
||||
|
||||
export class Pcm16PlaybackQueue {
|
||||
private nextStartAt = 0;
|
||||
private sources = new Set<AudioBufferSourceNode>();
|
||||
private queuedDurationSeconds = 0;
|
||||
|
||||
constructor(
|
||||
private readonly context: AudioContext,
|
||||
private readonly maxQueuedItems = MAX_TALK_PLAYBACK_CHUNKS,
|
||||
) {}
|
||||
|
||||
enqueue(samples: Int16Array, sampleRate: number): Promise<void> {
|
||||
if (samples.byteLength > MAX_TALK_PLAYBACK_DECODED_BYTES) {
|
||||
return Promise.reject(new Error('Talk PCM chunk exceeds decoded size limit'));
|
||||
}
|
||||
if (!Number.isInteger(sampleRate) || sampleRate < 8_000 || sampleRate > 192_000) {
|
||||
return Promise.reject(new Error('Unsupported Talk playback sample rate'));
|
||||
}
|
||||
if (this.sources.size >= this.maxQueuedItems) return Promise.reject(new Error('Talk playback queue is full'));
|
||||
const duration = samples.length / sampleRate;
|
||||
if (this.queuedDurationSeconds + duration > MAX_TALK_PLAYBACK_DURATION_SECONDS) {
|
||||
return Promise.reject(new Error('Talk playback duration queue is full'));
|
||||
}
|
||||
const buffer = this.context.createBuffer(1, samples.length, sampleRate);
|
||||
buffer.copyToChannel(new Float32Array(pcm16ToFloat32(samples)), 0);
|
||||
const source = this.context.createBufferSource();
|
||||
source.buffer = buffer;
|
||||
source.connect(this.context.destination);
|
||||
const startAt = Math.max(this.context.currentTime, this.nextStartAt);
|
||||
this.nextStartAt = startAt + buffer.duration;
|
||||
this.sources.add(source);
|
||||
this.queuedDurationSeconds += duration;
|
||||
return new Promise((resolve) => {
|
||||
source.onended = () => {
|
||||
if (this.sources.delete(source)) {
|
||||
this.queuedDurationSeconds = Math.max(0, this.queuedDurationSeconds - duration);
|
||||
}
|
||||
resolve();
|
||||
};
|
||||
source.start(startAt);
|
||||
});
|
||||
}
|
||||
|
||||
clear(): void {
|
||||
for (const source of this.sources) source.stop();
|
||||
this.sources.clear();
|
||||
this.queuedDurationSeconds = 0;
|
||||
this.nextStartAt = this.context.currentTime;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,522 @@
|
||||
import type { TalkCatalog, TalkRelayEvent, TalkRelaySession } from '@shared/talk/types';
|
||||
import { DEFAULT_SESSION_KEY, type ChatSession } from '@shared/chat/types';
|
||||
import type { GatewayStatus } from '@shared/types/gateway';
|
||||
import { hostApi } from '@/lib/host-api';
|
||||
import { hostEvents } from '@/lib/host-events';
|
||||
import { useAcpChatSessionStore } from '@/stores/acp-chat-session';
|
||||
import { useChatStore } from '@/stores/chat';
|
||||
import { isOpenClawHeartbeatOnlySession } from '@/stores/chat/session-key-utils';
|
||||
import {
|
||||
isRealtimeTalkCloseInFlight,
|
||||
registerRealtimeTalkCleanup,
|
||||
retainRealtimeTalkReservationUntil,
|
||||
useRealtimeTalkStore,
|
||||
} from '@/stores/realtime-talk';
|
||||
import {
|
||||
decodePcm16Base64,
|
||||
float32ToPcm16,
|
||||
inputLevel,
|
||||
Pcm16PlaybackQueue,
|
||||
pcm16ToBase64,
|
||||
StreamingResampler,
|
||||
} from './audio';
|
||||
import { createMicrophoneBridge, type MicrophoneBridge } from './audio-worklet';
|
||||
|
||||
type TalkApi = Pick<typeof hostApi.talk,
|
||||
'startRelay' | 'appendAudio' | 'cancelOutput' | 'acknowledgeMark' | 'stopRelay' | 'startAgentConsult' | 'submitToolResult'>;
|
||||
|
||||
type TalkAudioTransport = {
|
||||
start: (onSamples: (samples: Float32Array, sampleRateHz: number) => void) => Promise<void>;
|
||||
stop: () => void;
|
||||
enqueueOutput: (samples: Int16Array, sampleRate: number) => Promise<void>;
|
||||
clearOutput: () => void;
|
||||
};
|
||||
|
||||
type ControllerDependencies = {
|
||||
talk: TalkApi;
|
||||
subscribeTalk: (listener: (event: TalkRelayEvent) => void) => () => void;
|
||||
subscribeGatewayStatus: (listener: (status: GatewayStatus) => void) => () => void;
|
||||
createAudio: () => TalkAudioTransport;
|
||||
getAcpState: () => { sending: boolean; activeSessionKey: string | null };
|
||||
// Dependency injection keeps unit tests deterministic; validation remains in the controller.
|
||||
getSelectedSession: (sessionKey: string) => ChatSession | null;
|
||||
reloadAcpSession: () => Promise<boolean>;
|
||||
};
|
||||
|
||||
type StartInput = { sessionKey: string; heartbeatOnly?: boolean };
|
||||
type StartAttempt = { generation: number };
|
||||
type Consult = { callId: string; relaySessionId: string; relayGeneration: number };
|
||||
|
||||
const MAX_PENDING_TALK_EVENTS = 32;
|
||||
|
||||
function isTerminalTalkEvent(event: TalkRelayEvent): boolean {
|
||||
return event.type === 'close' || event.type === 'error';
|
||||
}
|
||||
|
||||
function isSupportedSampleRate(sampleRateHz: number): boolean {
|
||||
return Number.isInteger(sampleRateHz) && sampleRateHz >= 8_000 && sampleRateHz <= 192_000;
|
||||
}
|
||||
|
||||
function isSupportedAudio(session: TalkRelaySession): boolean {
|
||||
const { audio } = session;
|
||||
return audio.inputEncoding === 'pcm16'
|
||||
&& audio.outputEncoding === 'pcm16'
|
||||
&& isSupportedSampleRate(audio.inputSampleRateHz)
|
||||
&& isSupportedSampleRate(audio.outputSampleRateHz);
|
||||
}
|
||||
|
||||
class BrowserTalkAudioTransport implements TalkAudioTransport {
|
||||
private context: AudioContext | null = null;
|
||||
private microphone: MicrophoneBridge | null = null;
|
||||
private playback: Pcm16PlaybackQueue | null = null;
|
||||
|
||||
async start(onSamples: (samples: Float32Array, sampleRateHz: number) => void): Promise<void> {
|
||||
const AudioContextConstructor = window.AudioContext;
|
||||
if (!AudioContextConstructor) throw new Error('Realtime audio is not supported');
|
||||
this.context = new AudioContextConstructor();
|
||||
this.playback = new Pcm16PlaybackQueue(this.context);
|
||||
this.microphone = await createMicrophoneBridge(this.context, (samples) => {
|
||||
onSamples(samples, this.context?.sampleRate ?? 0);
|
||||
});
|
||||
}
|
||||
|
||||
stop(): void {
|
||||
this.microphone?.stop();
|
||||
this.microphone = null;
|
||||
this.playback?.clear();
|
||||
this.playback = null;
|
||||
void this.context?.close();
|
||||
this.context = null;
|
||||
}
|
||||
|
||||
enqueueOutput(samples: Int16Array, sampleRate: number): Promise<void> {
|
||||
if (!this.playback) return Promise.reject(new Error('Talk playback is unavailable'));
|
||||
return this.playback.enqueue(samples, sampleRate);
|
||||
}
|
||||
|
||||
clearOutput(): void {
|
||||
this.playback?.clear();
|
||||
}
|
||||
}
|
||||
|
||||
export class RealtimeTalkController {
|
||||
private relaySessionId: string | null = null;
|
||||
private sessionKey: string | null = null;
|
||||
private unsubscribeTalk: (() => void) | null = null;
|
||||
private unsubscribeGateway: (() => void) | null = null;
|
||||
private unregisterCleanup: (() => void) | null = null;
|
||||
private audio: TalkAudioTransport | null = null;
|
||||
private startGeneration = 0;
|
||||
private relayGeneration = 0;
|
||||
private appendQueue = Promise.resolve();
|
||||
private queuedAppendCount = 0;
|
||||
private inputSampleRateHz = 24_000;
|
||||
private outputSampleRateHz = 24_000;
|
||||
private resampler: StreamingResampler | null = null;
|
||||
private captureSampleRateHz: number | null = null;
|
||||
private outputGeneration = 0;
|
||||
private outputTurnActive = false;
|
||||
private outputCancelled = false;
|
||||
private outputCompletions = new Set<Promise<void>>();
|
||||
private consults = new Map<string, Consult>();
|
||||
private handledToolCallIds = new Set<string>();
|
||||
private cleanupInFlight: Promise<void> | null = null;
|
||||
|
||||
constructor(private readonly dependencies: ControllerDependencies) {}
|
||||
|
||||
async start(input: StartInput): Promise<boolean> {
|
||||
if (useRealtimeTalkStore.getState().isActive || isRealtimeTalkCloseInFlight()) return false;
|
||||
const sessionKey = input.sessionKey.trim();
|
||||
const acp = this.dependencies.getAcpState();
|
||||
const selectedSession = this.dependencies.getSelectedSession(sessionKey);
|
||||
if (
|
||||
!sessionKey
|
||||
|| sessionKey === DEFAULT_SESSION_KEY
|
||||
|| sessionKey !== acp.activeSessionKey
|
||||
|| !selectedSession
|
||||
|| selectedSession.key !== sessionKey
|
||||
|| isOpenClawHeartbeatOnlySession(selectedSession)
|
||||
|| input.heartbeatOnly
|
||||
|| acp.sending
|
||||
) {
|
||||
useRealtimeTalkStore.getState().finish('error', acp.sending
|
||||
? 'An ACP prompt is already in progress'
|
||||
: 'Talk requires the selected non-heartbeat session');
|
||||
return false;
|
||||
}
|
||||
if (!useRealtimeTalkStore.getState().reserve(sessionKey)) return false;
|
||||
|
||||
const attempt: StartAttempt = { generation: this.startGeneration += 1 };
|
||||
this.unregisterCleanup = registerRealtimeTalkCleanup(() => this.stop());
|
||||
let relay: TalkRelaySession | null = null;
|
||||
let audio: TalkAudioTransport | null = null;
|
||||
let unsubscribeTalk: (() => void) | null = null;
|
||||
let unsubscribeGateway: (() => void) | null = null;
|
||||
const pendingTalkEvents: TalkRelayEvent[] = [];
|
||||
try {
|
||||
unsubscribeTalk = this.dependencies.subscribeTalk((event) => {
|
||||
if (this.relaySessionId) {
|
||||
this.handleEvent(event);
|
||||
return;
|
||||
}
|
||||
if (pendingTalkEvents.length >= MAX_PENDING_TALK_EVENTS) {
|
||||
const nonTerminalIndex = pendingTalkEvents.findIndex((pending) => !isTerminalTalkEvent(pending));
|
||||
pendingTalkEvents.splice(nonTerminalIndex >= 0 ? nonTerminalIndex : 0, 1);
|
||||
}
|
||||
pendingTalkEvents.push(event);
|
||||
});
|
||||
relay = await this.dependencies.talk.startRelay({ sessionKey });
|
||||
if (!this.isCurrentAttempt(attempt)) {
|
||||
unsubscribeTalk?.();
|
||||
unsubscribeTalk = null;
|
||||
await this.closeRelayWithReservation(relay.relaySessionId);
|
||||
return false;
|
||||
}
|
||||
if (!isSupportedAudio(relay)) throw new Error('Unsupported Talk audio contract');
|
||||
const relaySessionId = relay.relaySessionId;
|
||||
|
||||
this.relaySessionId = relaySessionId;
|
||||
this.sessionKey = sessionKey;
|
||||
this.relayGeneration += 1;
|
||||
this.inputSampleRateHz = relay.audio.inputSampleRateHz;
|
||||
this.outputSampleRateHz = relay.audio.outputSampleRateHz;
|
||||
this.unsubscribeTalk = unsubscribeTalk;
|
||||
unsubscribeTalk = null;
|
||||
|
||||
const relayEvents = pendingTalkEvents.filter((event) => event.relaySessionId === relaySessionId);
|
||||
const terminalEvent = relayEvents.find(isTerminalTalkEvent);
|
||||
if (terminalEvent) {
|
||||
this.handleEvent(terminalEvent);
|
||||
await this.cleanupInFlight;
|
||||
return false;
|
||||
}
|
||||
|
||||
audio = this.dependencies.createAudio();
|
||||
this.audio = audio;
|
||||
audio = null;
|
||||
unsubscribeGateway = this.dependencies.subscribeGatewayStatus((status) => {
|
||||
if (status.state !== 'running' || status.gatewayReady === false) void this.handleGatewayDisconnect();
|
||||
});
|
||||
this.unsubscribeGateway = unsubscribeGateway;
|
||||
unsubscribeGateway = null;
|
||||
await this.audio.start((samples, sampleRateHz) => this.handleInput(samples, sampleRateHz));
|
||||
if (!this.isCurrentAttempt(attempt)) {
|
||||
await this.cleanupInFlight;
|
||||
return false;
|
||||
}
|
||||
const latestAcp = this.dependencies.getAcpState();
|
||||
const latestSelectedSession = this.dependencies.getSelectedSession(sessionKey);
|
||||
if (
|
||||
!this.isCurrentAttempt(attempt)
|
||||
|| latestAcp.sending
|
||||
|| latestAcp.activeSessionKey !== sessionKey
|
||||
|| !latestSelectedSession
|
||||
|| latestSelectedSession.key !== sessionKey
|
||||
|| isOpenClawHeartbeatOnlySession(latestSelectedSession)
|
||||
) throw new Error('ACP session changed during Talk startup');
|
||||
|
||||
useRealtimeTalkStore.getState().begin(relay.relaySessionId, sessionKey);
|
||||
for (const event of relayEvents) {
|
||||
if (!isTerminalTalkEvent(event)) this.handleEvent(event);
|
||||
}
|
||||
return true;
|
||||
} catch (error) {
|
||||
unsubscribeTalk?.();
|
||||
unsubscribeGateway?.();
|
||||
audio?.stop();
|
||||
if (relay && this.relaySessionId === relay.relaySessionId) this.releaseLocalResources();
|
||||
if (relay) await this.closeRelayWithReservation(relay.relaySessionId);
|
||||
if (this.isCurrentAttempt(attempt)) {
|
||||
this.unregisterCleanup?.();
|
||||
this.unregisterCleanup = null;
|
||||
useRealtimeTalkStore.getState().finish('error', error instanceof Error ? error.message : 'Talk could not start');
|
||||
}
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async stop(): Promise<void> {
|
||||
await this.cleanup('cancelled');
|
||||
}
|
||||
|
||||
async catalog(): Promise<TalkCatalog> {
|
||||
return await hostApi.talk.catalog();
|
||||
}
|
||||
|
||||
async handleSessionChange(sessionKey: string): Promise<void> {
|
||||
const reservedSessionKey = useRealtimeTalkStore.getState().sessionKey;
|
||||
if (reservedSessionKey && reservedSessionKey !== sessionKey) await this.cleanup('cancelled');
|
||||
}
|
||||
|
||||
async handleGatewayDisconnect(): Promise<void> {
|
||||
await this.cleanup('disconnected');
|
||||
}
|
||||
|
||||
private isCurrentAttempt(attempt: StartAttempt): boolean {
|
||||
return this.startGeneration === attempt.generation;
|
||||
}
|
||||
|
||||
private handleInput(samples: Float32Array, captureSampleRateHz: number): void {
|
||||
const relaySessionId = this.relaySessionId;
|
||||
if (!relaySessionId) return;
|
||||
const level = inputLevel(samples);
|
||||
useRealtimeTalkStore.getState().setInputLevel(level);
|
||||
if (this.outputTurnActive && !this.outputCancelled && level >= 0.05) {
|
||||
this.outputCancelled = true;
|
||||
this.audio?.clearOutput();
|
||||
void this.dependencies.talk.cancelOutput({ relaySessionId }).catch(() => undefined);
|
||||
}
|
||||
if (this.queuedAppendCount >= 4) return;
|
||||
try {
|
||||
if (!this.resampler) {
|
||||
this.resampler = new StreamingResampler(captureSampleRateHz, this.inputSampleRateHz);
|
||||
this.captureSampleRateHz = captureSampleRateHz;
|
||||
} else if (this.captureSampleRateHz !== captureSampleRateHz) {
|
||||
throw new Error('Talk microphone sample rate changed');
|
||||
}
|
||||
} catch (error) {
|
||||
void this.cleanup('error', error instanceof Error ? error.message : 'Unsupported Talk audio sample rate');
|
||||
return;
|
||||
}
|
||||
const resampled = this.resampler.process(samples);
|
||||
if (resampled.length === 0) return;
|
||||
this.queuedAppendCount += 1;
|
||||
const audioBase64 = pcm16ToBase64(float32ToPcm16(resampled));
|
||||
const append = this.appendQueue.then(async () => {
|
||||
if (this.relaySessionId !== relaySessionId) return;
|
||||
await this.dependencies.talk.appendAudio({ relaySessionId, audioBase64, timestamp: Date.now() });
|
||||
});
|
||||
this.appendQueue = append
|
||||
.catch((error) => {
|
||||
if (this.relaySessionId === relaySessionId) {
|
||||
void this.cleanup('error', error instanceof Error ? error.message : 'Talk audio upload failed');
|
||||
}
|
||||
})
|
||||
.finally(() => { this.queuedAppendCount = Math.max(0, this.queuedAppendCount - 1); });
|
||||
}
|
||||
|
||||
private handleEvent(event: TalkRelayEvent): void {
|
||||
if (event.relaySessionId !== this.relaySessionId) return;
|
||||
switch (event.type) {
|
||||
case 'ready':
|
||||
useRealtimeTalkStore.getState().setStatus('listening');
|
||||
break;
|
||||
case 'transcript':
|
||||
useRealtimeTalkStore.getState().appendTranscript(event);
|
||||
break;
|
||||
case 'audio':
|
||||
this.playOutput(event.audioBase64);
|
||||
break;
|
||||
case 'clear':
|
||||
this.clearOutputTurn();
|
||||
useRealtimeTalkStore.getState().setStatus('listening');
|
||||
break;
|
||||
case 'mark':
|
||||
this.completeOutputTurn(event.markName);
|
||||
break;
|
||||
case 'toolCall':
|
||||
if (event.name === 'openclaw_agent_consult') void this.startAgentConsult(event);
|
||||
break;
|
||||
case 'toolResult':
|
||||
this.completeAgentConsult(event.callId, Boolean(event.final));
|
||||
break;
|
||||
case 'toolCallCancelled':
|
||||
this.consults.delete(event.callId);
|
||||
break;
|
||||
case 'error':
|
||||
void this.cleanup('error', event.message);
|
||||
break;
|
||||
case 'close':
|
||||
void this.cleanup(event.reason, event.reason === 'error' ? 'Talk relay closed with an error' : undefined);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
private playOutput(audioBase64: string): void {
|
||||
const relaySessionId = this.relaySessionId;
|
||||
if (!relaySessionId || !this.audio || this.outputCancelled) return;
|
||||
let samples: Int16Array;
|
||||
try {
|
||||
samples = decodePcm16Base64(audioBase64);
|
||||
} catch (error) {
|
||||
void this.cleanup('error', error instanceof Error ? error.message : 'Invalid Talk audio');
|
||||
return;
|
||||
}
|
||||
if (!this.outputTurnActive) {
|
||||
this.outputTurnActive = true;
|
||||
this.outputGeneration += 1;
|
||||
useRealtimeTalkStore.getState().setStatus('speaking');
|
||||
}
|
||||
const generation = this.outputGeneration;
|
||||
let completion: Promise<void>;
|
||||
try {
|
||||
completion = this.audio.enqueueOutput(samples, this.outputSampleRateHz);
|
||||
} catch (error) {
|
||||
void this.cleanup('error', error instanceof Error ? error.message : 'Talk playback failed');
|
||||
return;
|
||||
}
|
||||
this.outputCompletions.add(completion);
|
||||
void completion
|
||||
.then(() => undefined)
|
||||
.catch((error) => {
|
||||
if (this.isCurrentOutputTurn(generation, relaySessionId)) {
|
||||
void this.cleanup('error', error instanceof Error ? error.message : 'Talk playback failed');
|
||||
}
|
||||
})
|
||||
.finally(() => this.outputCompletions.delete(completion));
|
||||
}
|
||||
|
||||
private completeOutputTurn(markName: string): void {
|
||||
const relaySessionId = this.relaySessionId;
|
||||
const generation = this.outputGeneration;
|
||||
if (!relaySessionId || !this.outputTurnActive) return;
|
||||
const completions = [...this.outputCompletions];
|
||||
void Promise.all(completions)
|
||||
.then(async () => {
|
||||
if (!this.isCurrentOutputTurn(generation, relaySessionId)) return;
|
||||
await this.acknowledgeMark(markName);
|
||||
if (!this.isCurrentOutputTurn(generation, relaySessionId) || this.outputCancelled) return;
|
||||
this.outputTurnActive = false;
|
||||
useRealtimeTalkStore.getState().setStatus('listening');
|
||||
})
|
||||
.catch(() => undefined);
|
||||
}
|
||||
|
||||
private isCurrentOutputTurn(generation: number, relaySessionId: string): boolean {
|
||||
return this.outputTurnActive
|
||||
&& this.outputGeneration === generation
|
||||
&& this.relaySessionId === relaySessionId;
|
||||
}
|
||||
|
||||
private clearOutputTurn(): void {
|
||||
this.outputGeneration += 1;
|
||||
this.outputTurnActive = false;
|
||||
this.outputCancelled = false;
|
||||
this.outputCompletions.clear();
|
||||
this.audio?.clearOutput();
|
||||
}
|
||||
|
||||
private async acknowledgeMark(markName: string): Promise<void> {
|
||||
const relaySessionId = this.relaySessionId;
|
||||
if (!relaySessionId) return;
|
||||
await this.dependencies.talk.acknowledgeMark({ relaySessionId, markName }).catch(() => undefined);
|
||||
}
|
||||
|
||||
private isCurrentConsult(consult: Consult): boolean {
|
||||
return this.relaySessionId === consult.relaySessionId
|
||||
&& this.relayGeneration === consult.relayGeneration
|
||||
&& this.consults.get(consult.callId) === consult;
|
||||
}
|
||||
|
||||
private async startAgentConsult(event: Extract<TalkRelayEvent, { type: 'toolCall' }>): Promise<void> {
|
||||
const relaySessionId = this.relaySessionId;
|
||||
const sessionKey = this.sessionKey;
|
||||
if (!relaySessionId || !sessionKey || this.handledToolCallIds.has(event.callId)) return;
|
||||
const consult: Consult = { callId: event.callId, relaySessionId, relayGeneration: this.relayGeneration };
|
||||
this.handledToolCallIds.add(event.callId);
|
||||
this.consults.set(event.callId, consult);
|
||||
useRealtimeTalkStore.getState().setStatus('thinking');
|
||||
try {
|
||||
const result = await this.dependencies.talk.startAgentConsult({ relaySessionId, sessionKey, callId: event.callId, args: event.args });
|
||||
if (!this.isCurrentConsult(consult)) return;
|
||||
await this.dependencies.talk.submitToolResult({
|
||||
relaySessionId,
|
||||
callId: event.callId,
|
||||
result,
|
||||
options: { suppressResponse: true, willContinue: true },
|
||||
});
|
||||
} catch (error) {
|
||||
if (!this.isCurrentConsult(consult)) return;
|
||||
this.consults.delete(event.callId);
|
||||
await this.cleanup('error', error instanceof Error ? error.message : 'Agent consult failed');
|
||||
}
|
||||
}
|
||||
|
||||
private completeAgentConsult(callId: string, final: boolean): void {
|
||||
const consult = this.consults.get(callId);
|
||||
if (!final || !consult || !this.isCurrentConsult(consult)) return;
|
||||
this.consults.delete(callId);
|
||||
// The microtask is an identity guard only, never an audio-delay mechanism.
|
||||
void Promise.resolve().then(() => {
|
||||
if (
|
||||
this.relaySessionId === consult.relaySessionId
|
||||
&& this.relayGeneration === consult.relayGeneration
|
||||
&& !this.consults.has(callId)
|
||||
) void this.dependencies.reloadAcpSession();
|
||||
});
|
||||
}
|
||||
|
||||
private async cleanup(reason: 'cancelled' | 'completed' | 'error' | 'disconnected' | (string & {}), error?: string): Promise<void> {
|
||||
if (this.cleanupInFlight) {
|
||||
await this.cleanupInFlight;
|
||||
return;
|
||||
}
|
||||
const cleanup = this.releaseAndClose(reason, error);
|
||||
this.cleanupInFlight = cleanup;
|
||||
try {
|
||||
await cleanup;
|
||||
} finally {
|
||||
if (this.cleanupInFlight === cleanup) this.cleanupInFlight = null;
|
||||
}
|
||||
}
|
||||
|
||||
private async releaseAndClose(reason: 'cancelled' | 'completed' | 'error' | 'disconnected' | (string & {}), error?: string): Promise<void> {
|
||||
this.startGeneration += 1;
|
||||
this.relayGeneration += 1;
|
||||
const relaySessionId = this.relaySessionId;
|
||||
this.releaseLocalResources();
|
||||
useRealtimeTalkStore.getState().finish(reason, error);
|
||||
if (relaySessionId) await this.closeRelayWithReservation(relaySessionId);
|
||||
}
|
||||
|
||||
private async closeRelayWithReservation(relaySessionId: string): Promise<void> {
|
||||
const close = this.closeRelay(relaySessionId);
|
||||
retainRealtimeTalkReservationUntil(close);
|
||||
await close;
|
||||
}
|
||||
|
||||
private async closeRelay(relaySessionId: string): Promise<void> {
|
||||
await this.dependencies.talk.stopRelay({ relaySessionId }).catch(() => undefined);
|
||||
}
|
||||
|
||||
private releaseLocalResources(): void {
|
||||
this.unregisterCleanup?.();
|
||||
this.unregisterCleanup = null;
|
||||
this.unsubscribeTalk?.();
|
||||
this.unsubscribeTalk = null;
|
||||
this.unsubscribeGateway?.();
|
||||
this.unsubscribeGateway = null;
|
||||
this.clearOutputTurn();
|
||||
this.audio?.stop();
|
||||
this.audio = null;
|
||||
this.relaySessionId = null;
|
||||
this.sessionKey = null;
|
||||
this.appendQueue = Promise.resolve();
|
||||
this.queuedAppendCount = 0;
|
||||
this.inputSampleRateHz = 24_000;
|
||||
this.outputSampleRateHz = 24_000;
|
||||
this.resampler = null;
|
||||
this.captureSampleRateHz = null;
|
||||
this.consults.clear();
|
||||
this.handledToolCallIds.clear();
|
||||
}
|
||||
}
|
||||
|
||||
export function createRealtimeTalkController(dependencies: ControllerDependencies): RealtimeTalkController {
|
||||
return new RealtimeTalkController(dependencies);
|
||||
}
|
||||
|
||||
export const realtimeTalkController = createRealtimeTalkController({
|
||||
talk: hostApi.talk,
|
||||
subscribeTalk: hostEvents.onTalkEvent,
|
||||
subscribeGatewayStatus: hostEvents.onGatewayStatus,
|
||||
createAudio: () => new BrowserTalkAudioTransport(),
|
||||
getAcpState: () => {
|
||||
const state = useAcpChatSessionStore.getState();
|
||||
return { sending: state.sending, activeSessionKey: state.activeSessionKey };
|
||||
},
|
||||
getSelectedSession: (sessionKey) => (
|
||||
useChatStore.getState().sessions.find((session) => session.key === sessionKey) ?? null
|
||||
),
|
||||
reloadAcpSession: () => useAcpChatSessionStore.getState().reloadActiveSession(),
|
||||
});
|
||||
+149
-11
@@ -7,7 +7,7 @@
|
||||
* references in the ACP session/prompt request.
|
||||
*/
|
||||
import { useState, useRef, useEffect, useCallback, useMemo } from 'react';
|
||||
import { SendHorizontal, Square, X, Paperclip, FileText, Film, Music, FileArchive, File, FolderOpen, Loader2, AtSign, Search, ChevronDown, Check } from 'lucide-react';
|
||||
import { SendHorizontal, Square, X, Paperclip, FileText, Film, Music, FileArchive, File, FolderOpen, Loader2, AtSign, Search, ChevronDown, Check, Mic } from 'lucide-react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Textarea } from '@/components/ui/textarea';
|
||||
import { Tooltip, TooltipContent, TooltipTrigger } from '@/components/ui/tooltip';
|
||||
@@ -16,6 +16,7 @@ import { cn } from '@/lib/utils';
|
||||
import { useGatewayStore } from '@/stores/gateway';
|
||||
import { useAgentsStore } from '@/stores/agents';
|
||||
import { useChatStore } from '@/stores/chat';
|
||||
import { useRealtimeTalkStore } from '@/stores/realtime-talk';
|
||||
import { useArtifactPanel } from '@/stores/artifact-panel';
|
||||
import { buildPreviewTarget } from '@/components/file-preview/build-preview-target';
|
||||
import { useProviderStore } from '@/stores/providers';
|
||||
@@ -28,6 +29,10 @@ import { rendererExtensionRegistry } from '@/extensions/registry';
|
||||
import { collectDroppedFiles } from '@/lib/collect-dropped-files';
|
||||
import { fetchQuickAccessSkills } from '@/lib/quick-access-skills';
|
||||
import { DEFAULT_WORKSPACE_CWD, isDefaultWorkspacePath, normalizeWorkspacePath } from '@/lib/workspace-context';
|
||||
import { realtimeTalkController } from '@/lib/talk/realtime-talk-controller';
|
||||
import { DEFAULT_SESSION_KEY } from '@shared/chat/types';
|
||||
import { isOpenClawHeartbeatOnlySession } from '@/stores/chat/session-key-utils';
|
||||
import type { TalkCatalog } from '@shared/talk/types';
|
||||
|
||||
// ── Types ────────────────────────────────────────────────────────
|
||||
|
||||
@@ -58,6 +63,8 @@ interface ChatInputProps {
|
||||
workspaceOptions?: ChatWorkspaceOption[];
|
||||
workspaceReadOnly?: boolean;
|
||||
onSelectWorkspace?: (path: string) => void;
|
||||
talkActive?: boolean;
|
||||
onConfigureTalk?: () => void;
|
||||
}
|
||||
|
||||
// ── Helpers ──────────────────────────────────────────────────────
|
||||
@@ -207,6 +214,8 @@ export function ChatInput({
|
||||
workspaceOptions = [],
|
||||
workspaceReadOnly = false,
|
||||
onSelectWorkspace,
|
||||
talkActive,
|
||||
onConfigureTalk,
|
||||
}: ChatInputProps) {
|
||||
const { t } = useTranslation('chat');
|
||||
const [input, setInput] = useState('');
|
||||
@@ -224,6 +233,7 @@ export function ChatInput({
|
||||
const [switchingModelRef, setSwitchingModelRef] = useState<string | null>(null);
|
||||
const [optimisticModelRef, setOptimisticModelRef] = useState<string | null>(null);
|
||||
const [providerSnapshotReady, setProviderSnapshotReady] = useState(false);
|
||||
const [talkCatalog, setTalkCatalog] = useState<TalkCatalog | null>(null);
|
||||
const textareaRef = useRef<HTMLTextAreaElement>(null);
|
||||
const pickerRef = useRef<HTMLDivElement>(null);
|
||||
const skillPickerRef = useRef<HTMLDivElement>(null);
|
||||
@@ -241,6 +251,11 @@ export function ChatInput({
|
||||
const providerError = useProviderStore((s) => s.error);
|
||||
const refreshProviderSnapshot = useProviderStore((s) => s.refreshProviderSnapshot);
|
||||
const currentAgentId = useChatStore((s) => s.currentAgentId);
|
||||
const currentSessionKey = useChatStore((s) => s.currentSessionKey);
|
||||
const sessions = useChatStore((s) => s.sessions);
|
||||
const talkStatus = useRealtimeTalkStore((s) => s.status);
|
||||
const talkInputLevel = useRealtimeTalkStore((s) => s.inputLevel);
|
||||
const storeTalkActive = useRealtimeTalkStore((s) => s.isActive);
|
||||
const currentAgent = useMemo(
|
||||
() => (agents ?? []).find((agent) => agent.id === currentAgentId) ?? null,
|
||||
[agents, currentAgentId],
|
||||
@@ -288,11 +303,45 @@ export function ChatInput({
|
||||
const showModelPicker = modelOptions.length > 1;
|
||||
const chatComposerStatusComponents = rendererExtensionRegistry.getChatComposerStatusComponents();
|
||||
const isGatewayUsable = gatewayStatus.state === 'running' && gatewayStatus.gatewayReady !== false;
|
||||
const inputDisabled = disabled;
|
||||
const talkIsActive = talkActive ?? storeTalkActive;
|
||||
const inputDisabled = disabled || talkIsActive;
|
||||
const attachmentsLocked = inputDisabled;
|
||||
const gatewayUnavailable = !isGatewayUsable;
|
||||
const workspaceSelectorDisabled = workspaceReadOnly || inputDisabled || sending || !onSelectWorkspace;
|
||||
const skillTokenRanges = useMemo(() => findSkillTokenRanges(input), [input]);
|
||||
const openArtifactPreview = useArtifactPanel((s) => s.openPreview);
|
||||
const selectedTalkSession = useMemo(
|
||||
() => sessions.find((session) => session.key === currentSessionKey) ?? null,
|
||||
[currentSessionKey, sessions],
|
||||
);
|
||||
const talkSessionAllowed = !!currentSessionKey
|
||||
&& currentSessionKey !== DEFAULT_SESSION_KEY
|
||||
&& !!selectedTalkSession
|
||||
&& !isOpenClawHeartbeatOnlySession(selectedTalkSession);
|
||||
const talkReady = talkCatalog?.realtime.ready === true;
|
||||
const talkConfigurationUnavailable = talkCatalog?.realtime.ready === false;
|
||||
const talkConfigurationReason = talkCatalog?.realtime.ready === false
|
||||
? talkCatalog.realtime.reason
|
||||
: undefined;
|
||||
const talkStartBlocked = disabled || !isGatewayUsable || sending || !talkSessionAllowed || !talkReady || storeTalkActive;
|
||||
|
||||
useEffect(() => {
|
||||
if (!isGatewayUsable) {
|
||||
setTalkCatalog(null);
|
||||
return;
|
||||
}
|
||||
let cancelled = false;
|
||||
void realtimeTalkController.catalog()
|
||||
.then((catalog) => {
|
||||
if (!cancelled) setTalkCatalog(catalog);
|
||||
})
|
||||
.catch(() => {
|
||||
if (!cancelled) setTalkCatalog({ modes: [], transports: [], brains: [], realtime: { ready: false, providers: [] } });
|
||||
});
|
||||
return () => {
|
||||
cancelled = true;
|
||||
};
|
||||
}, [isGatewayUsable]);
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
void (async () => {
|
||||
@@ -333,6 +382,14 @@ export function ChatInput({
|
||||
}
|
||||
}, [workspaceSelectorDisabled]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!inputDisabled) return;
|
||||
setPickerOpen(false);
|
||||
setSkillPickerOpen(false);
|
||||
setModelPickerOpen(false);
|
||||
setWorkspaceMenuOpen(false);
|
||||
}, [inputDisabled]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!providerSnapshotReady || providerError || !currentAgent || switchingModelRef || optimisticModelRef) return;
|
||||
const override = (currentAgent.overrideModelRef || '').trim();
|
||||
@@ -571,7 +628,7 @@ export function ChatInput({
|
||||
// ── File staging via native dialog / Electron drag-drop paths ──
|
||||
|
||||
const stagePathFiles = useCallback(async (filePaths: string[]) => {
|
||||
if (filePaths.length === 0) return;
|
||||
if (attachmentsLocked || filePaths.length === 0) return;
|
||||
|
||||
const tempIds: string[] = [];
|
||||
for (const filePath of filePaths) {
|
||||
@@ -624,9 +681,10 @@ export function ChatInput({
|
||||
: a,
|
||||
));
|
||||
}
|
||||
}, []);
|
||||
}, [attachmentsLocked]);
|
||||
|
||||
const pickFiles = useCallback(async () => {
|
||||
if (attachmentsLocked) return;
|
||||
try {
|
||||
const result = await hostApi.dialog.open({
|
||||
properties: ['openFile', 'multiSelections'],
|
||||
@@ -636,11 +694,12 @@ export function ChatInput({
|
||||
} catch (err) {
|
||||
console.error('[pickFiles] Failed to open file dialog:', err);
|
||||
}
|
||||
}, [stagePathFiles]);
|
||||
}, [attachmentsLocked, stagePathFiles]);
|
||||
|
||||
// ── Stage browser File objects (paste / drag-drop) ─────────────
|
||||
|
||||
const stageBufferFiles = useCallback(async (files: globalThis.File[]) => {
|
||||
if (attachmentsLocked) return;
|
||||
for (const file of files) {
|
||||
const tempId = crypto.randomUUID();
|
||||
setAttachments(prev => [...prev, {
|
||||
@@ -675,13 +734,14 @@ export function ChatInput({
|
||||
));
|
||||
}
|
||||
}
|
||||
}, []);
|
||||
}, [attachmentsLocked]);
|
||||
|
||||
// ── Attachment management ──────────────────────────────────────
|
||||
|
||||
const removeAttachment = useCallback((id: string) => {
|
||||
if (attachmentsLocked) return;
|
||||
setAttachments(prev => prev.filter(a => a.id !== id));
|
||||
}, []);
|
||||
}, [attachmentsLocked]);
|
||||
|
||||
const allReady = attachments.length === 0 || attachments.every(a => a.status === 'ready');
|
||||
const hasFailedAttachments = attachments.some((a) => a.status === 'error');
|
||||
@@ -740,6 +800,31 @@ export function ChatInput({
|
||||
onStop?.();
|
||||
}, [canStop, onStop]);
|
||||
|
||||
const handleTalk = useCallback(() => {
|
||||
if (talkIsActive) {
|
||||
void realtimeTalkController.stop();
|
||||
return;
|
||||
}
|
||||
if (talkConfigurationUnavailable) {
|
||||
onConfigureTalk?.();
|
||||
return;
|
||||
}
|
||||
if (talkStartBlocked || !currentSessionKey) return;
|
||||
void realtimeTalkController.start({ sessionKey: currentSessionKey });
|
||||
}, [currentSessionKey, disabled, onConfigureTalk, talkConfigurationUnavailable, talkIsActive, talkStartBlocked]);
|
||||
|
||||
const talkUnavailableReason = !isGatewayUsable
|
||||
? t('talk.unavailable.gateway')
|
||||
: sending
|
||||
? t('talk.unavailable.sending')
|
||||
: !talkSessionAllowed
|
||||
? t('talk.unavailable.session')
|
||||
: talkConfigurationUnavailable
|
||||
? t('talk.unavailable.configuration', { reason: talkConfigurationReason })
|
||||
: !talkReady
|
||||
? t('talk.unavailable.loading')
|
||||
: t('talk.unavailable.default');
|
||||
|
||||
const handleKeyDown = useCallback(
|
||||
(e: React.KeyboardEvent) => {
|
||||
if (e.key === 'Backspace') {
|
||||
@@ -827,11 +912,12 @@ export function ChatInput({
|
||||
}
|
||||
}
|
||||
if (pastedFiles.length > 0) {
|
||||
if (attachmentsLocked) return;
|
||||
e.preventDefault();
|
||||
stageBufferFiles(pastedFiles);
|
||||
}
|
||||
},
|
||||
[stageBufferFiles],
|
||||
[attachmentsLocked, stageBufferFiles],
|
||||
);
|
||||
|
||||
// Handle drag & drop
|
||||
@@ -840,8 +926,9 @@ export function ChatInput({
|
||||
const handleDragOver = useCallback((e: React.DragEvent) => {
|
||||
e.preventDefault();
|
||||
e.stopPropagation();
|
||||
if (attachmentsLocked) return;
|
||||
setDragOver(true);
|
||||
}, []);
|
||||
}, [attachmentsLocked]);
|
||||
|
||||
const handleDragLeave = useCallback((e: React.DragEvent) => {
|
||||
e.preventDefault();
|
||||
@@ -854,6 +941,7 @@ export function ChatInput({
|
||||
e.preventDefault();
|
||||
e.stopPropagation();
|
||||
setDragOver(false);
|
||||
if (attachmentsLocked) return;
|
||||
if (!e.dataTransfer) return;
|
||||
|
||||
const { pathFiles, bufferFiles } = collectDroppedFiles(e.dataTransfer);
|
||||
@@ -864,7 +952,7 @@ export function ChatInput({
|
||||
if (pathFiles.length > 0) void stagePathFiles(pathFiles);
|
||||
if (bufferFiles.length > 0) void stageBufferFiles(bufferFiles);
|
||||
},
|
||||
[stageBufferFiles, stagePathFiles, t],
|
||||
[attachmentsLocked, stageBufferFiles, stagePathFiles, t],
|
||||
);
|
||||
|
||||
return (
|
||||
@@ -927,6 +1015,7 @@ export function ChatInput({
|
||||
key={att.id}
|
||||
attachment={att}
|
||||
onRemove={() => removeAttachment(att.id)}
|
||||
disabled={attachmentsLocked}
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
@@ -939,6 +1028,7 @@ export function ChatInput({
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setTargetAgentId(null)}
|
||||
disabled={inputDisabled}
|
||||
className="inline-flex items-center gap-1.5 rounded-lg border border-primary/20 bg-primary/5 px-2.5 py-1 text-meta font-medium text-foreground transition-colors hover:bg-primary/10"
|
||||
title={t('composer.clearTarget')}
|
||||
>
|
||||
@@ -990,7 +1080,10 @@ export function ChatInput({
|
||||
</div>
|
||||
|
||||
{/* Action Row — icons on their own line */}
|
||||
<div className="mt-1.5 flex items-center gap-1">
|
||||
<div className="mt-1.5 flex items-center gap-1">
|
||||
<div data-testid="chat-talk-status" role="status" aria-live="polite" className="sr-only">
|
||||
{t(`talk.status.${talkStatus}`)}
|
||||
</div>
|
||||
{/* Attach Button */}
|
||||
<Button
|
||||
variant="ghost"
|
||||
@@ -1187,6 +1280,44 @@ export function ChatInput({
|
||||
</div>
|
||||
)}
|
||||
|
||||
<Tooltip>
|
||||
<TooltipTrigger asChild>
|
||||
<span className="inline-flex shrink-0">
|
||||
<Button
|
||||
type="button"
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
data-testid="chat-composer-talk"
|
||||
className={cn(
|
||||
'relative h-8 w-8 rounded-lg text-muted-foreground transition-colors hover:bg-black/5 hover:text-foreground dark:hover:bg-white/10',
|
||||
talkIsActive && 'bg-black/5 text-foreground dark:bg-white/10',
|
||||
)}
|
||||
disabled={!talkIsActive && talkStartBlocked && !talkConfigurationUnavailable}
|
||||
aria-label={talkIsActive ? t('talk.stop') : t('talk.start')}
|
||||
aria-pressed={talkIsActive}
|
||||
onClick={handleTalk}
|
||||
>
|
||||
<Mic className="h-3.5 w-3.5" />
|
||||
{talkIsActive && (
|
||||
<span
|
||||
data-testid="chat-talk-input-level"
|
||||
role="meter"
|
||||
aria-label={t('talk.inputLevel')}
|
||||
aria-valuemin={0}
|
||||
aria-valuemax={1}
|
||||
aria-valuenow={talkInputLevel}
|
||||
className="absolute bottom-1 h-0.5 rounded-full bg-primary"
|
||||
style={{ width: `${Math.max(8, talkInputLevel * 100)}%` }}
|
||||
/>
|
||||
)}
|
||||
</Button>
|
||||
</span>
|
||||
</TooltipTrigger>
|
||||
<TooltipContent>
|
||||
<p>{talkIsActive ? t('talk.stop') : talkStartBlocked ? talkUnavailableReason : t('talk.start')}</p>
|
||||
</TooltipContent>
|
||||
</Tooltip>
|
||||
|
||||
{/* Send Button — pushed to the right */}
|
||||
<Button
|
||||
onClick={sending ? handleStop : handleSend}
|
||||
@@ -1320,9 +1451,11 @@ export function ChatInput({
|
||||
size="sm"
|
||||
className="h-auto shrink-0 p-0 text-tiny"
|
||||
onClick={() => {
|
||||
if (attachmentsLocked) return;
|
||||
setAttachments((prev) => prev.filter((att) => att.status !== 'error'));
|
||||
void pickFiles();
|
||||
}}
|
||||
disabled={attachmentsLocked}
|
||||
>
|
||||
{t('composer.retryFailedAttachments')}
|
||||
</Button>
|
||||
@@ -1339,9 +1472,11 @@ export function ChatInput({
|
||||
function AttachmentPreview({
|
||||
attachment,
|
||||
onRemove,
|
||||
disabled,
|
||||
}: {
|
||||
attachment: FileAttachment;
|
||||
onRemove: () => void;
|
||||
disabled: boolean;
|
||||
}) {
|
||||
const { t } = useTranslation('chat');
|
||||
const isImage = attachment.mimeType.startsWith('image/') && attachment.preview;
|
||||
@@ -1390,7 +1525,10 @@ function AttachmentPreview({
|
||||
|
||||
{/* Remove button */}
|
||||
<button
|
||||
type="button"
|
||||
data-testid="chat-attachment-remove"
|
||||
onClick={onRemove}
|
||||
disabled={disabled}
|
||||
className="absolute -top-1 -right-1 bg-destructive text-destructive-foreground rounded-full p-0.5 opacity-0 group-hover:opacity-100 transition-opacity"
|
||||
>
|
||||
<X className="h-3 w-3" />
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
import { Mic } from 'lucide-react';
|
||||
import { useTranslation } from 'react-i18next';
|
||||
import { cn } from '@/lib/utils';
|
||||
import type { LiveTalkTranscript as LiveTalkTranscriptEntry } from '@/stores/realtime-talk';
|
||||
|
||||
type LiveTalkTranscriptProps = {
|
||||
transcripts: LiveTalkTranscriptEntry[];
|
||||
};
|
||||
|
||||
export function LiveTalkTranscript({ transcripts }: LiveTalkTranscriptProps) {
|
||||
const { t } = useTranslation('chat');
|
||||
|
||||
if (transcripts.length === 0) return null;
|
||||
|
||||
return (
|
||||
<section data-testid="live-talk-transcript" aria-label={t('talk.transcript.label')} className="mx-auto w-full max-w-3xl px-4 pb-2">
|
||||
<div className="space-y-3">
|
||||
{transcripts.map((entry, index) => {
|
||||
const isUser = entry.role === 'user';
|
||||
return (
|
||||
<div
|
||||
key={`${entry.role}-${index}`}
|
||||
data-testid={isUser ? 'live-talk-user-message' : 'live-talk-assistant-message'}
|
||||
className={cn('flex gap-3', isUser ? 'justify-end' : 'justify-start')}
|
||||
>
|
||||
<div
|
||||
className={cn(
|
||||
'max-w-[80%] rounded-2xl px-4 py-3 text-sm shadow-sm',
|
||||
isUser ? 'bg-brand text-white' : 'bg-surface-modal text-foreground',
|
||||
)}
|
||||
>
|
||||
<div className="mb-1 flex items-center gap-1.5 text-tiny opacity-70">
|
||||
<Mic aria-label={t('talk.transcript.microphone')} className="h-3 w-3" />
|
||||
<span>{t('talk.transcript.microphone')}</span>
|
||||
</div>
|
||||
<p className="whitespace-pre-wrap break-words">{entry.text}</p>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
</section>
|
||||
);
|
||||
}
|
||||
@@ -2,7 +2,7 @@
|
||||
* Chat Page
|
||||
* ACP-native runtime rendering through the ordered inline timeline.
|
||||
*/
|
||||
import { Suspense, lazy, useDeferredValue, useEffect, useMemo, useRef, useState } from 'react';
|
||||
import { Suspense, lazy, useDeferredValue, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react';
|
||||
import { AlertTriangle, ArrowDownToLine, FolderOpen } from 'lucide-react';
|
||||
import { useTranslation } from 'react-i18next';
|
||||
import { toast } from 'sonner';
|
||||
@@ -33,6 +33,9 @@ import { ChatInput, type ChatWorkspaceOption, type FileAttachment } from './Chat
|
||||
import { ChatToolbar } from './ChatToolbar';
|
||||
import { AcpTimeline } from './AcpTimeline';
|
||||
import { AcpErrorBanner } from './AcpErrorBanner';
|
||||
import { LiveTalkTranscript } from './LiveTalkTranscript';
|
||||
import { useRealtimeTalkStore } from '@/stores/realtime-talk';
|
||||
import { realtimeTalkController } from '@/lib/talk/realtime-talk-controller';
|
||||
|
||||
const ArtifactPanelLazy = lazy(() =>
|
||||
import('@/components/file-preview/ArtifactPanel').then((m) => ({ default: m.ArtifactPanel })),
|
||||
@@ -283,12 +286,20 @@ export function Chat() {
|
||||
const cancelAcp = useAcpChatSessionStore((s) => s.cancel);
|
||||
const respondAcpPermission = useAcpChatSessionStore((s) => s.respondPermission);
|
||||
const clearAcpError = useAcpChatSessionStore((s) => s.clearError);
|
||||
const talkActive = useRealtimeTalkStore((s) => s.isActive);
|
||||
const talkSessionKey = useRealtimeTalkStore((s) => s.sessionKey);
|
||||
const talkTranscripts = useRealtimeTalkStore((s) => s.transcripts);
|
||||
const talkMatchesCurrentSession = talkSessionKey === currentSessionKey;
|
||||
const visibleTalkActive = talkActive && talkMatchesCurrentSession;
|
||||
const visibleTalkTranscripts = talkMatchesCurrentSession ? talkTranscripts : [];
|
||||
|
||||
const panelOpen = useArtifactPanel((s) => s.open);
|
||||
const panelWidthPct = useArtifactPanel((s) => s.widthPct);
|
||||
const closeArtifactPanel = useArtifactPanel((s) => s.close);
|
||||
const splitContainerRef = useRef<HTMLDivElement | null>(null);
|
||||
const acpLoadInFlightKeyRef = useRef<string | null>(null);
|
||||
const liveTalkAreaRef = useRef<HTMLDivElement | null>(null);
|
||||
const liveTranscriptCountRef = useRef(visibleTalkTranscripts.length);
|
||||
const { contentRef, scrollRef, scrollToBottom, isAtBottom } = useStickToBottomInstant(
|
||||
currentSessionKey,
|
||||
acpSending || acpCancelling,
|
||||
@@ -299,6 +310,28 @@ export function Chat() {
|
||||
return () => setVisibleSession(null);
|
||||
}, [currentSessionKey, setVisibleSession]);
|
||||
|
||||
useLayoutEffect(() => {
|
||||
const previousCount = liveTranscriptCountRef.current;
|
||||
liveTranscriptCountRef.current = visibleTalkTranscripts.length;
|
||||
if (visibleTalkTranscripts.length <= previousCount) return;
|
||||
liveTalkAreaRef.current?.scrollIntoView({ block: 'nearest' });
|
||||
}, [visibleTalkTranscripts.length]);
|
||||
|
||||
useEffect(() => {
|
||||
void realtimeTalkController.handleSessionChange(currentSessionKey ?? '');
|
||||
}, [currentSessionKey]);
|
||||
|
||||
useEffect(() => {
|
||||
const stopTalk = () => {
|
||||
void realtimeTalkController.stop();
|
||||
};
|
||||
window.addEventListener('pagehide', stopTalk);
|
||||
return () => {
|
||||
window.removeEventListener('pagehide', stopTalk);
|
||||
stopTalk();
|
||||
};
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
void fetchAgents().catch(() => undefined);
|
||||
}, [fetchAgents]);
|
||||
@@ -544,6 +577,9 @@ export function Chat() {
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div ref={liveTalkAreaRef}>
|
||||
<LiveTalkTranscript transcripts={visibleTalkTranscripts} />
|
||||
</div>
|
||||
<ChatInput
|
||||
onSend={(text: string, attachments?: FileAttachment[], targetAgentId?: string | null) => {
|
||||
if (!currentSessionKey || !cwd || !workspaceContextAvailable) return;
|
||||
@@ -624,6 +660,8 @@ export function Chat() {
|
||||
workspaceOptions={workspaceOptions}
|
||||
workspaceReadOnly={effectiveWorkspace.readOnly}
|
||||
onSelectWorkspace={setChatWorkspacePath}
|
||||
talkActive={visibleTalkActive}
|
||||
onConfigureTalk={() => window.location.assign('#/settings?section=talk')}
|
||||
/>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
* Application configuration
|
||||
*/
|
||||
import { useEffect, useMemo, useState } from 'react';
|
||||
import { useLocation } from 'react-router-dom';
|
||||
import { Sun, Moon, Monitor, RefreshCw, ExternalLink, Copy, FileText } from 'lucide-react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Label } from '@/components/ui/label';
|
||||
@@ -15,6 +16,7 @@ import { useSettingsStore } from '@/stores/settings';
|
||||
import { useGatewayStore } from '@/stores/gateway';
|
||||
import { useUpdateStore } from '@/stores/update';
|
||||
import { UpdateSettings } from '@/components/settings/UpdateSettings';
|
||||
import { TalkSettings } from '@/components/settings/TalkSettings';
|
||||
import { toUserMessage } from '@/lib/error-message';
|
||||
import {
|
||||
clearUiTelemetry,
|
||||
@@ -36,6 +38,7 @@ type ControlUiInfo = {
|
||||
|
||||
export function Settings() {
|
||||
const { t, i18n } = useTranslation('settings');
|
||||
const location = useLocation();
|
||||
const {
|
||||
theme,
|
||||
setTheme,
|
||||
@@ -210,6 +213,14 @@ export function Settings() {
|
||||
};
|
||||
}, [devModeUnlocked, showCliTools]);
|
||||
|
||||
useEffect(() => {
|
||||
const section = new URLSearchParams(location.search).get('section');
|
||||
if (section !== 'talk' && section !== 'developer') return;
|
||||
const target = document.getElementById(section);
|
||||
target?.scrollIntoView({ block: 'start' });
|
||||
target?.focus({ preventScroll: true });
|
||||
}, [location.search]);
|
||||
|
||||
const handleCopyCliCommand = async () => {
|
||||
if (!openclawCliCommand) return;
|
||||
try {
|
||||
@@ -611,18 +622,6 @@ export function Settings() {
|
||||
<Switch checked={gatewayAutoStart} onCheckedChange={setGatewayAutoStart} />
|
||||
</div>
|
||||
|
||||
<div className="flex items-center justify-between">
|
||||
<div>
|
||||
<Label className="text-sm font-medium text-foreground">{t('advanced.devMode')}</Label>
|
||||
<p className="text-meta text-muted-foreground mt-1">{t('advanced.devModeDesc')}</p>
|
||||
</div>
|
||||
<Switch
|
||||
checked={devModeUnlocked}
|
||||
onCheckedChange={setDevModeUnlocked}
|
||||
data-testid="settings-dev-mode-switch"
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div className="flex items-center justify-between">
|
||||
<div>
|
||||
<Label className="text-sm font-medium text-foreground">{t('advanced.telemetry')}</Label>
|
||||
@@ -633,17 +632,38 @@ export function Settings() {
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<Separator className="bg-black/5 dark:bg-white/5" />
|
||||
|
||||
<TalkSettings />
|
||||
|
||||
{/* Developer */}
|
||||
{devModeUnlocked && (
|
||||
<>
|
||||
<Separator className="bg-black/5 dark:bg-white/5" />
|
||||
<div data-testid="settings-developer-section">
|
||||
<h2
|
||||
data-testid="settings-developer-title"
|
||||
className="text-3xl font-serif text-foreground mb-6 font-normal tracking-tight"
|
||||
>
|
||||
{t('developer.title')}
|
||||
</h2>
|
||||
<section id="developer" tabIndex={-1} data-testid="settings-developer-section" className="space-y-6 scroll-mt-6">
|
||||
<h2
|
||||
data-testid="settings-developer-title"
|
||||
className="text-3xl font-serif text-foreground font-normal tracking-tight"
|
||||
>
|
||||
{t('developer.title')}
|
||||
</h2>
|
||||
<div className="flex items-center justify-between">
|
||||
<div>
|
||||
<Label className="text-sm font-medium text-foreground">{t('advanced.devMode')}</Label>
|
||||
<p className="text-meta text-muted-foreground mt-1">{t('advanced.devModeDesc')}</p>
|
||||
</div>
|
||||
<Switch
|
||||
checked={devModeUnlocked}
|
||||
onCheckedChange={setDevModeUnlocked}
|
||||
data-testid="settings-dev-mode-switch"
|
||||
/>
|
||||
</div>
|
||||
{!devModeUnlocked && (
|
||||
<p className="text-meta text-muted-foreground" data-testid="settings-developer-locked-guidance">
|
||||
{t('advanced.devModeDesc')}
|
||||
</p>
|
||||
)}
|
||||
{devModeUnlocked && (
|
||||
<>
|
||||
<Separator className="bg-black/5 dark:bg-white/5" />
|
||||
<div data-testid="settings-developer-advanced">
|
||||
<div className="space-y-8">
|
||||
{/* Gateway Proxy */}
|
||||
<div className="space-y-4" data-testid="settings-proxy-section">
|
||||
@@ -1030,8 +1050,9 @@ export function Settings() {
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
</section>
|
||||
|
||||
<Separator className="bg-black/5 dark:bg-white/5" />
|
||||
|
||||
|
||||
@@ -45,6 +45,7 @@ import { hostApi } from '@/lib/host-api';
|
||||
import { hostEvents } from '@/lib/host-events';
|
||||
import type { AcpTimelineSnapshot, MessageSegmentItem, PermissionItem, RenderPart } from '@/lib/acp/timeline-types';
|
||||
import { isCronSessionKey } from './chat/cron-session-utils';
|
||||
import { stopActiveRealtimeTalk, useRealtimeTalkStore } from './realtime-talk';
|
||||
|
||||
const EMPTY_SESSION_ID = '';
|
||||
const CANCEL_PERMISSION_OPTION_ID = '__cancelled__';
|
||||
@@ -147,6 +148,7 @@ export type AcpChatSessionState = {
|
||||
turnTimingsByUserMessageId: Record<string, AcpTurnTiming>;
|
||||
prepareLocalSession: (input: AcpChatLoadPayload) => void;
|
||||
loadSession: (input: AcpChatLoadPayload) => Promise<boolean>;
|
||||
reloadActiveSession: () => Promise<boolean>;
|
||||
sendPrompt: (input: AcpChatPromptPayload) => Promise<boolean>;
|
||||
cancel: () => Promise<void>;
|
||||
respondPermission: (requestId: string, optionId: string) => Promise<void>;
|
||||
@@ -1016,6 +1018,8 @@ export const useAcpChatSessionStore = create<AcpChatSessionState>((set, get) =>
|
||||
turnTimingsByUserMessageId: {},
|
||||
|
||||
prepareLocalSession(input) {
|
||||
void stopActiveRealtimeTalk();
|
||||
useRealtimeTalkStore.getState().reset();
|
||||
captureLiveSession(get());
|
||||
loadRequestSeq += 1;
|
||||
pendingLoadUpdates.clear();
|
||||
@@ -1061,6 +1065,8 @@ export const useAcpChatSessionStore = create<AcpChatSessionState>((set, get) =>
|
||||
timeline: liveSnapshot?.timeline ?? createEmptyAcpTimeline(input.sessionKey, generation),
|
||||
turnTimingsByUserMessageId: liveSnapshot?.turnTimingsByUserMessageId ?? {},
|
||||
});
|
||||
await stopActiveRealtimeTalk();
|
||||
useRealtimeTalkStore.getState().reset();
|
||||
|
||||
try {
|
||||
let result = await hostApi.chat.loadAcpSession(input);
|
||||
@@ -1223,11 +1229,18 @@ export const useAcpChatSessionStore = create<AcpChatSessionState>((set, get) =>
|
||||
}
|
||||
},
|
||||
|
||||
async reloadActiveSession() {
|
||||
const { activeSessionKey, workspaceRoot, cwd } = get();
|
||||
if (!activeSessionKey || !workspaceRoot || !cwd) return false;
|
||||
return await get().loadSession({ sessionKey: activeSessionKey, workspaceRoot, cwd });
|
||||
},
|
||||
|
||||
async sendPrompt(input) {
|
||||
const startState = get();
|
||||
const sessionKey = input.sessionKey;
|
||||
const generation = startState.generation;
|
||||
if (startState.activeSessionKey !== sessionKey) return false;
|
||||
if (useRealtimeTalkStore.getState().isActive) return false;
|
||||
|
||||
const messageId = input.messageId ?? createOptimisticMessageId();
|
||||
const payload = { ...input, messageId };
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
import { create } from 'zustand';
|
||||
import type { TalkTerminalReason } from '@shared/talk/types';
|
||||
|
||||
let activeRelayCleanup: (() => Promise<void>) | null = null;
|
||||
let relayCloseInFlight: Promise<void> | null = null;
|
||||
|
||||
// ACP owns reload boundaries, but the controller owns the browser resources it must release.
|
||||
export function registerRealtimeTalkCleanup(cleanup: () => Promise<void>): () => void {
|
||||
activeRelayCleanup = cleanup;
|
||||
return () => {
|
||||
if (activeRelayCleanup === cleanup) activeRelayCleanup = null;
|
||||
};
|
||||
}
|
||||
|
||||
export async function stopActiveRealtimeTalk(): Promise<void> {
|
||||
await activeRelayCleanup?.();
|
||||
}
|
||||
|
||||
export function isRealtimeTalkCloseInFlight(): boolean {
|
||||
return relayCloseInFlight !== null;
|
||||
}
|
||||
|
||||
export function retainRealtimeTalkReservationUntil(close: Promise<void>): void {
|
||||
relayCloseInFlight = close;
|
||||
void close.then(
|
||||
() => {
|
||||
if (relayCloseInFlight === close) relayCloseInFlight = null;
|
||||
},
|
||||
() => {
|
||||
if (relayCloseInFlight === close) relayCloseInFlight = null;
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
export type RealtimeTalkStatus = 'idle' | 'connecting' | 'listening' | 'thinking' | 'speaking' | 'disconnected' | 'error';
|
||||
|
||||
export type LiveTalkTranscript = {
|
||||
role: 'user' | 'assistant';
|
||||
text: string;
|
||||
final: boolean;
|
||||
};
|
||||
|
||||
type RealtimeTalkState = {
|
||||
status: RealtimeTalkStatus;
|
||||
relaySessionId: string | null;
|
||||
sessionKey: string | null;
|
||||
inputLevel: number;
|
||||
transcripts: LiveTalkTranscript[];
|
||||
error: string | null;
|
||||
isActive: boolean;
|
||||
reserve: (sessionKey: string) => boolean;
|
||||
begin: (relaySessionId: string, sessionKey: string) => void;
|
||||
setStatus: (status: RealtimeTalkStatus) => void;
|
||||
setInputLevel: (inputLevel: number) => void;
|
||||
appendTranscript: (entry: LiveTalkTranscript) => void;
|
||||
finish: (reason: TalkTerminalReason, error?: string) => void;
|
||||
reset: () => void;
|
||||
};
|
||||
|
||||
const initialState = {
|
||||
status: 'idle' as const,
|
||||
relaySessionId: null,
|
||||
sessionKey: null,
|
||||
inputLevel: 0,
|
||||
transcripts: [] as LiveTalkTranscript[],
|
||||
error: null,
|
||||
isActive: false,
|
||||
};
|
||||
|
||||
export const useRealtimeTalkStore = create<RealtimeTalkState>((set) => ({
|
||||
...initialState,
|
||||
reserve: (sessionKey) => {
|
||||
let reserved = false;
|
||||
set((state) => {
|
||||
if (state.isActive) return {};
|
||||
reserved = true;
|
||||
return {
|
||||
status: 'connecting',
|
||||
sessionKey,
|
||||
relaySessionId: null,
|
||||
inputLevel: 0,
|
||||
transcripts: [],
|
||||
error: null,
|
||||
isActive: true,
|
||||
};
|
||||
});
|
||||
return reserved;
|
||||
},
|
||||
begin: (relaySessionId, sessionKey) => set({
|
||||
relaySessionId,
|
||||
sessionKey,
|
||||
status: 'listening',
|
||||
error: null,
|
||||
inputLevel: 0,
|
||||
transcripts: [],
|
||||
isActive: true,
|
||||
}),
|
||||
setStatus: (status) => set({ status }),
|
||||
setInputLevel: (inputLevel) => set({ inputLevel: Math.max(0, Math.min(1, inputLevel)) }),
|
||||
appendTranscript: (entry) => set((state) => {
|
||||
const last = state.transcripts.at(-1);
|
||||
const transcripts = !entry.final && last && last.role === entry.role && !last.final
|
||||
? [...state.transcripts.slice(0, -1), entry]
|
||||
: entry.final && last && last.role === entry.role && !last.final
|
||||
? [...state.transcripts.slice(0, -1), entry]
|
||||
: [...state.transcripts, entry];
|
||||
return { transcripts };
|
||||
}),
|
||||
finish: (reason, error) => set({
|
||||
...initialState,
|
||||
status: error ? 'error' : reason === 'disconnected' ? 'disconnected' : 'idle',
|
||||
error: error ?? null,
|
||||
}),
|
||||
reset: () => set(initialState),
|
||||
}));
|
||||
@@ -10,6 +10,7 @@ import {
|
||||
import { expandAcpToolCallsGroup } from './fixtures/acp-timeline';
|
||||
|
||||
const MAIN_SESSION_KEY = 'agent:main:main';
|
||||
const TALK_SESSION_KEY = 'agent:main:talk-e2e';
|
||||
const MAIN_WORKSPACE = '/workspace';
|
||||
const DEFAULT_WORKSPACE = '~/.openclaw/workspace';
|
||||
const REVIEWER_SESSION_KEY = 'agent:reviewer:main';
|
||||
@@ -1422,4 +1423,137 @@ test.describe('ClawX ACP inline timeline', () => {
|
||||
await closeElectronApp(app);
|
||||
}
|
||||
});
|
||||
|
||||
test('renders typed Talk events as transient direct bubbles without microphone hardware', async ({ launchElectronApp }) => {
|
||||
const app = await launchElectronApp({ skipSetup: true });
|
||||
|
||||
try {
|
||||
await installIpcMocks(app, {
|
||||
gatewayStatus: { state: 'running', gatewayReady: true, port: 18789, pid: 12345 },
|
||||
gatewayRpc: {
|
||||
[stableStringify(['sessions.list', {}])]: {
|
||||
success: true,
|
||||
result: {
|
||||
sessions: [
|
||||
{ key: MAIN_SESSION_KEY, displayName: 'main', workspacePath: MAIN_WORKSPACE },
|
||||
{ key: TALK_SESSION_KEY, displayName: 'Talk', workspacePath: MAIN_WORKSPACE },
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
hostApi: {
|
||||
...baseHostApiMocks(),
|
||||
[stableStringify(['chat', 'loadAcpSession', {
|
||||
sessionKey: TALK_SESSION_KEY, workspaceRoot: MAIN_WORKSPACE, cwd: MAIN_WORKSPACE,
|
||||
}])]: { success: true, generation: 1 },
|
||||
[stableStringify(['talk', 'catalog', null])]: {
|
||||
modes: ['realtime'],
|
||||
transports: ['gateway-relay'],
|
||||
brains: ['agent-consult'],
|
||||
realtime: { ready: true, providers: [] },
|
||||
},
|
||||
[stableStringify(['talk', 'startRelay', { sessionKey: TALK_SESSION_KEY }])]: {
|
||||
relaySessionId: 'relay-e2e',
|
||||
provider: 'mock',
|
||||
transport: 'gateway-relay',
|
||||
audio: {
|
||||
inputEncoding: 'pcm16',
|
||||
inputSampleRateHz: 24_000,
|
||||
outputEncoding: 'pcm16',
|
||||
outputSampleRateHz: 24_000,
|
||||
},
|
||||
},
|
||||
[stableStringify(['talk', 'stopRelay', { relaySessionId: 'relay-e2e' }])]: { ok: true },
|
||||
},
|
||||
recordHostInvocations: true,
|
||||
});
|
||||
const initialPage = await getStableWindow(app);
|
||||
await initialPage.addInitScript(() => {
|
||||
class FakeAudioContext {
|
||||
sampleRate = 24_000;
|
||||
audioWorklet = { addModule: async () => undefined };
|
||||
destination = {};
|
||||
createMediaStreamSource() { return { connect() {}, disconnect() {} }; }
|
||||
createGain() { return { gain: { value: 0 }, connect() {}, disconnect() {} }; }
|
||||
async close() {}
|
||||
}
|
||||
class FakeAudioWorkletNode {
|
||||
port = { onmessage: null as ((event: MessageEvent<Float32Array>) => void) | null };
|
||||
constructor(_context: AudioContext, _name: string) {}
|
||||
connect() {}
|
||||
disconnect() {}
|
||||
}
|
||||
Object.defineProperty(window, 'AudioContext', { value: FakeAudioContext, configurable: true });
|
||||
Object.defineProperty(window, 'AudioWorkletNode', { value: FakeAudioWorkletNode, configurable: true });
|
||||
Object.defineProperty(navigator, 'mediaDevices', {
|
||||
value: { getUserMedia: async () => ({ getTracks: () => [{ stop() {} }] }) },
|
||||
configurable: true,
|
||||
});
|
||||
});
|
||||
|
||||
const page = await openChat(app);
|
||||
await expect(page.getByTestId('acp-chat-empty-state')).toBeVisible({ timeout: 30_000 });
|
||||
await page.getByTestId(`sidebar-session-${TALK_SESSION_KEY}`).click();
|
||||
await expect(page.getByTestId('chat-composer-talk')).toBeEnabled();
|
||||
await page.getByTestId('chat-composer-input').fill('Preserve this draft');
|
||||
await page.getByTestId('chat-composer-talk').click();
|
||||
await expect(page.getByTestId('chat-composer-input')).toBeDisabled();
|
||||
|
||||
await app.evaluate(async ({ app: _app }) => {
|
||||
const { BrowserWindow } = process.mainModule!.require('electron') as typeof import('electron');
|
||||
for (const window of BrowserWindow.getAllWindows()) {
|
||||
window.webContents.send('talk:event', {
|
||||
relaySessionId: 'relay-e2e', type: 'transcript', role: 'user', text: 'Direct user', final: true,
|
||||
});
|
||||
window.webContents.send('talk:event', {
|
||||
relaySessionId: 'relay-e2e', type: 'transcript', role: 'assistant', text: 'Direct assistant', final: true,
|
||||
});
|
||||
}
|
||||
});
|
||||
await expect(page.getByTestId('live-talk-user-message')).toContainText('Direct user');
|
||||
await expect(page.getByTestId('live-talk-assistant-message')).toContainText('Direct assistant');
|
||||
await expect(page.getByTestId('acp-chat-timeline')).toHaveCount(0);
|
||||
|
||||
await app.evaluate(async ({ app: _app }) => {
|
||||
const { BrowserWindow } = process.mainModule!.require('electron') as typeof import('electron');
|
||||
for (const window of BrowserWindow.getAllWindows()) {
|
||||
window.webContents.send('talk:event', {
|
||||
relaySessionId: 'relay-e2e', type: 'toolCall', callId: 'unknown-tool', name: 'untrusted_tool', args: {},
|
||||
});
|
||||
}
|
||||
});
|
||||
await page.waitForTimeout(50);
|
||||
expect((await getRecordedHostInvocations(app)).some((call) => (
|
||||
call.module === 'talk' && call.action === 'startAgentConsult'
|
||||
))).toBe(false);
|
||||
|
||||
await page.getByTestId('chat-composer-talk').click();
|
||||
await expect(page.getByTestId('chat-composer-input')).toBeEnabled();
|
||||
await expect(page.getByTestId('chat-composer-input')).toHaveValue('Preserve this draft');
|
||||
await expect(page.getByTestId('live-talk-transcript')).toHaveCount(0);
|
||||
|
||||
await page.getByTestId('chat-composer-talk').click();
|
||||
await expect(page.getByTestId('chat-composer-input')).toBeDisabled();
|
||||
await app.evaluate(async ({ app: _app }) => {
|
||||
const { BrowserWindow } = process.mainModule!.require('electron') as typeof import('electron');
|
||||
for (const window of BrowserWindow.getAllWindows()) {
|
||||
window.webContents.send('talk:event', {
|
||||
relaySessionId: 'relay-e2e', type: 'transcript', role: 'assistant', text: 'Direct assistant after restart', final: true,
|
||||
});
|
||||
}
|
||||
});
|
||||
await expect(page.getByTestId('live-talk-assistant-message')).toContainText('Direct assistant after restart');
|
||||
|
||||
await page.reload();
|
||||
await expect(page.getByTestId('live-talk-transcript')).toHaveCount(0);
|
||||
await expect.poll(async () => (await getRecordedHostInvocations(app)).some((call) => (
|
||||
call.module === 'talk' && call.action === 'stopRelay'
|
||||
))).toBe(true);
|
||||
expect((await getRecordedHostInvocations(app)).some((call) => (
|
||||
call.module === 'talk' && call.action === 'startRelay'
|
||||
))).toBe(true);
|
||||
} finally {
|
||||
await closeElectronApp(app);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -28,6 +28,26 @@ test.describe('ClawX chat model picker', () => {
|
||||
});
|
||||
|
||||
const workspacePath = '/tmp/clawx-model-picker-workspace';
|
||||
const talkCatalog = {
|
||||
realtime: {
|
||||
ready: false,
|
||||
reason: 'Configure a realtime provider',
|
||||
activeProvider: 'openai',
|
||||
providers: [{
|
||||
id: 'openai',
|
||||
label: 'OpenAI',
|
||||
configured: true,
|
||||
models: ['gpt-realtime', 'gpt-realtime-mini'],
|
||||
voices: ['alloy', 'verse'],
|
||||
}, {
|
||||
id: 'google',
|
||||
label: 'Google',
|
||||
configured: true,
|
||||
models: ['gemini-realtime'],
|
||||
voices: ['puck'],
|
||||
}],
|
||||
},
|
||||
};
|
||||
const agentsSnapshot = () => ({
|
||||
success: true,
|
||||
agents: [{
|
||||
@@ -98,6 +118,12 @@ test.describe('ClawX chat model picker', () => {
|
||||
if (request?.module === 'chat' && request.action === 'loadAcpSession') {
|
||||
return makeResponse(request.id, { success: true, generation: 1 });
|
||||
}
|
||||
if (request?.module === 'talk' && request.action === 'catalog') {
|
||||
return makeResponse(request.id, talkCatalog);
|
||||
}
|
||||
if (request?.module === 'talk' && request.action === 'updateRealtimeSettings') {
|
||||
return makeResponse(request.id, { ok: true });
|
||||
}
|
||||
if (request?.module === 'gateway' && request.action === 'rpc') {
|
||||
const method = typeof body?.method === 'string' ? body.method : '';
|
||||
const params = body?.params ?? null;
|
||||
@@ -255,6 +281,38 @@ test.describe('ClawX chat model picker', () => {
|
||||
|| request.path === 'gateway:start'
|
||||
|| request.path === 'gateway:config.patch'
|
||||
)).toBe(false);
|
||||
|
||||
await page.evaluate(() => window.location.assign('#/settings?section=talk'));
|
||||
await expect(page.getByTestId('talk-settings')).toBeVisible();
|
||||
await expect(page.getByTestId('talk-settings-readiness')).toContainText('Talk is unavailable');
|
||||
await expect(page.getByTestId('talk-settings-unavailable-reason')).toHaveText('Configure a realtime provider');
|
||||
await page.getByTestId('talk-settings-provider').selectOption('google');
|
||||
await expect(page.getByTestId('talk-settings-model')).toHaveValue('gemini-realtime');
|
||||
await expect(page.getByTestId('talk-settings-voice')).toHaveValue('puck');
|
||||
await page.getByTestId('talk-settings-save').click();
|
||||
|
||||
await expect.poll(async () => app.evaluate(() => (
|
||||
(globalThis as typeof globalThis & {
|
||||
__chatModelPickerRequests?: Array<{ path: string; body: unknown }>;
|
||||
}).__chatModelPickerRequests?.some((request) => (
|
||||
request.path === 'talk:updateRealtimeSettings'
|
||||
&& JSON.stringify(request.body) === JSON.stringify({
|
||||
provider: 'google',
|
||||
model: 'gemini-realtime',
|
||||
speakerVoice: 'puck',
|
||||
})
|
||||
)) ?? false
|
||||
))).toBe(true);
|
||||
expect(await app.evaluate(() => (
|
||||
(globalThis as typeof globalThis & {
|
||||
__chatModelPickerRequests?: Array<{ path: string }>;
|
||||
}).__chatModelPickerRequests?.filter((request) => request.path === 'talk:catalog').length ?? 0
|
||||
))).toBeGreaterThanOrEqual(2);
|
||||
|
||||
await page.getByTestId('settings-dev-mode-switch').click();
|
||||
await expect(page.getByTestId('settings-developer-section')).toBeVisible();
|
||||
await page.evaluate(() => window.location.assign('#/settings?section=developer'));
|
||||
await expect(page.locator('#developer')).toBeFocused();
|
||||
} finally {
|
||||
await closeElectronApp(app);
|
||||
}
|
||||
|
||||
@@ -6,11 +6,16 @@ test.describe('ClawX developer-mode gated UI', () => {
|
||||
|
||||
await page.getByTestId('sidebar-nav-settings').click();
|
||||
await expect(page.getByTestId('settings-page')).toBeVisible();
|
||||
await expect(page.getByTestId('settings-developer-section')).toHaveCount(0);
|
||||
await expect(page.getByTestId('settings-developer-section')).toBeVisible();
|
||||
await expect(page.getByTestId('settings-developer-locked-guidance')).toBeVisible();
|
||||
await expect(page.getByTestId('settings-dev-mode-switch')).toHaveAttribute('data-state', 'unchecked');
|
||||
await expect(page.getByTestId('sidebar-open-dev-console')).toHaveCount(0);
|
||||
await expect(page.getByTestId('sidebar-nav-dreams')).toHaveCount(0);
|
||||
|
||||
await page.evaluate(() => window.location.assign('#/settings?section=developer'));
|
||||
await expect(page.getByTestId('settings-developer-section')).toBeFocused();
|
||||
await expect(page.getByTestId('settings-dev-mode-switch')).toHaveAttribute('data-state', 'unchecked');
|
||||
|
||||
await page.evaluate(() => {
|
||||
window.location.hash = '#/dreams';
|
||||
});
|
||||
@@ -34,6 +39,7 @@ test.describe('ClawX developer-mode gated UI', () => {
|
||||
await page.getByTestId('settings-dev-mode-switch').click();
|
||||
await expect(page.getByTestId('settings-dev-mode-switch')).toHaveAttribute('data-state', 'checked');
|
||||
await expect(page.getByTestId('settings-developer-section')).toBeVisible();
|
||||
await expect(page.getByTestId('settings-developer-locked-guidance')).toHaveCount(0);
|
||||
await expect(page.getByTestId('settings-developer-gateway-token')).toBeVisible();
|
||||
await expect(page.getByTestId('sidebar-open-dev-console')).toBeVisible();
|
||||
await expect(page.getByTestId('sidebar-nav-dreams')).toHaveCount(0);
|
||||
|
||||
@@ -166,6 +166,50 @@ describe('ACP Chat store', () => {
|
||||
hostEventsMock.onChatRuntimeEvent.mockClear();
|
||||
});
|
||||
|
||||
it('does not send an ACP prompt while a realtime Talk relay is active', async () => {
|
||||
const { useAcpChatSessionStore } = await importStore();
|
||||
const { useRealtimeTalkStore } = await import('@/stores/realtime-talk');
|
||||
useAcpChatSessionStore.getState().prepareLocalSession({
|
||||
sessionKey: 'agent:pi:s1', workspaceRoot: '/repo-a', cwd: '/repo-a',
|
||||
});
|
||||
useRealtimeTalkStore.getState().begin('relay-1', 'agent:pi:s1');
|
||||
|
||||
await expect(useAcpChatSessionStore.getState().sendPrompt({
|
||||
sessionKey: 'agent:pi:s1', cwd: '/repo-a', message: 'blocked',
|
||||
})).resolves.toBe(false);
|
||||
expect(hostApiMock.sendAcpPrompt).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('does not send an ACP prompt while realtime Talk is connecting', async () => {
|
||||
const { useAcpChatSessionStore } = await importStore();
|
||||
const { useRealtimeTalkStore } = await import('@/stores/realtime-talk');
|
||||
useAcpChatSessionStore.getState().prepareLocalSession({
|
||||
sessionKey: 'agent:pi:s1', workspaceRoot: '/repo-a', cwd: '/repo-a',
|
||||
});
|
||||
expect(useRealtimeTalkStore.getState().reserve('agent:pi:s1')).toBe(true);
|
||||
|
||||
await expect(useAcpChatSessionStore.getState().sendPrompt({
|
||||
sessionKey: 'agent:pi:s1', cwd: '/repo-a', message: 'blocked while connecting',
|
||||
})).resolves.toBe(false);
|
||||
expect(hostApiMock.sendAcpPrompt).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('stops the active realtime Talk controller before reloading ACP state', async () => {
|
||||
const { useAcpChatSessionStore } = await importStore();
|
||||
const { registerRealtimeTalkCleanup, useRealtimeTalkStore } = await import('@/stores/realtime-talk');
|
||||
const stopRelay = vi.fn().mockResolvedValue(undefined);
|
||||
const unregister = registerRealtimeTalkCleanup(stopRelay);
|
||||
useRealtimeTalkStore.getState().begin('relay-1', 'agent:pi:s1');
|
||||
|
||||
await useAcpChatSessionStore.getState().loadSession({
|
||||
sessionKey: 'agent:pi:s1', workspaceRoot: '/repo-a', cwd: '/repo-a',
|
||||
});
|
||||
|
||||
expect(stopRelay).toHaveBeenCalledOnce();
|
||||
expect(useRealtimeTalkStore.getState().isActive).toBe(false);
|
||||
unregister();
|
||||
});
|
||||
|
||||
it('projects cron history when ACP replay is empty', async () => {
|
||||
hostApiMock.cronSessionHistory.mockResolvedValue({
|
||||
messages: [
|
||||
|
||||
@@ -49,6 +49,13 @@ const { acpState, agentsState, artifactPanelState, chatState, gatewayState, stic
|
||||
},
|
||||
}));
|
||||
|
||||
const talkState = vi.hoisted(() => ({
|
||||
status: 'listening',
|
||||
isActive: true,
|
||||
sessionKey: 'agent:main:main' as string | null,
|
||||
transcripts: [{ role: 'assistant' as const, text: 'Direct Talk response', final: true }],
|
||||
}));
|
||||
|
||||
const ensureAcpChatSubscriptions = vi.hoisted(() => vi.fn());
|
||||
const resolveWorkspaceContext = vi.hoisted(() => vi.fn());
|
||||
|
||||
@@ -67,6 +74,17 @@ vi.mock('@/stores/gateway', () => ({
|
||||
useGatewayStore: (selector: (state: typeof gatewayState) => unknown) => selector(gatewayState),
|
||||
}));
|
||||
|
||||
vi.mock('@/stores/realtime-talk', () => ({
|
||||
useRealtimeTalkStore: (selector: (state: typeof talkState) => unknown) => selector(talkState),
|
||||
}));
|
||||
|
||||
vi.mock('@/lib/talk/realtime-talk-controller', () => ({
|
||||
realtimeTalkController: {
|
||||
handleSessionChange: vi.fn().mockResolvedValue(undefined),
|
||||
stop: vi.fn().mockResolvedValue(undefined),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('@/stores/agents', () => ({
|
||||
useAgentsStore: (selector: (state: typeof agentsState) => unknown) => selector(agentsState),
|
||||
}));
|
||||
@@ -115,10 +133,6 @@ vi.mock('@/hooks/use-stick-to-bottom-instant', () => ({
|
||||
})),
|
||||
}));
|
||||
|
||||
vi.mock('@/hooks/use-min-loading', () => ({
|
||||
useMinLoading: () => false,
|
||||
}));
|
||||
|
||||
vi.mock('@/pages/Chat/ChatToolbar', () => ({
|
||||
ChatToolbar: () => null,
|
||||
}));
|
||||
@@ -249,6 +263,10 @@ describe('ACP Chat page inline timeline lifecycle', () => {
|
||||
gatewayState.status = { state: 'running', gatewayReady: true, port: 18789 };
|
||||
stickState.isAtBottom = true;
|
||||
stickState.scrollToBottom.mockReset();
|
||||
talkState.status = 'listening';
|
||||
talkState.isActive = true;
|
||||
talkState.sessionKey = 'agent:main:main';
|
||||
talkState.transcripts = [{ role: 'assistant', text: 'Direct Talk response', final: true }];
|
||||
});
|
||||
|
||||
it('renders ACP process blocks in timeline order', async () => {
|
||||
@@ -282,6 +300,45 @@ describe('ACP Chat page inline timeline lifecycle', () => {
|
||||
});
|
||||
});
|
||||
|
||||
it('keeps direct Talk transcripts outside the ACP item sequence', async () => {
|
||||
const { Chat } = await import('@/pages/Chat/index');
|
||||
const { container } = render(<Chat />);
|
||||
|
||||
expect(screen.getByText('Direct Talk response')).toBeInTheDocument();
|
||||
expect(Array.from(container.querySelectorAll('[data-acp-item-id]')).map((node) => node.getAttribute('data-acp-item-id'))).toEqual([
|
||||
'msg-user:0',
|
||||
'thought:assistant-run',
|
||||
'tool:read-file',
|
||||
'permission:approve-edit',
|
||||
'plan:current',
|
||||
'msg-assistant:0',
|
||||
]);
|
||||
});
|
||||
|
||||
it('hides transient Talk state immediately when it belongs to a different session', async () => {
|
||||
talkState.sessionKey = 'agent:main:other';
|
||||
const { Chat } = await import('@/pages/Chat/index');
|
||||
|
||||
render(<Chat />);
|
||||
|
||||
expect(screen.queryByText('Direct Talk response')).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('brings a newly received live Talk area into view without scrolling the ACP timeline', async () => {
|
||||
const scrollIntoView = vi.fn();
|
||||
Object.defineProperty(HTMLElement.prototype, 'scrollIntoView', { configurable: true, value: scrollIntoView });
|
||||
talkState.transcripts = [];
|
||||
const { Chat } = await import('@/pages/Chat/index');
|
||||
const { rerender } = render(<Chat />);
|
||||
|
||||
talkState.transcripts = [{ role: 'assistant', text: 'Newest direct Talk response', final: true }];
|
||||
rerender(<Chat />);
|
||||
|
||||
expect(scrollIntoView).toHaveBeenCalledWith({ block: 'nearest' });
|
||||
expect(stickState.scrollToBottom).not.toHaveBeenCalled();
|
||||
expect(screen.getByText('Newest direct Talk response')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('keeps ACP tool status in the inline timeline while the composer is busy', async () => {
|
||||
acpState.sending = true;
|
||||
acpState.timeline = {
|
||||
|
||||
@@ -3,7 +3,7 @@ import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { Chat } from '@/pages/Chat';
|
||||
import type { AcpTimelineSnapshot } from '@/lib/acp/timeline-types';
|
||||
|
||||
const { acpState, agentsState, artifactPanelState, artifactPanelProps, chatState, gatewayState, settingsState } = vi.hoisted(() => ({
|
||||
const { acpState, agentsState, artifactPanelState, artifactPanelProps, chatState, gatewayState, settingsState, talkState, talkController } = vi.hoisted(() => ({
|
||||
acpState: {
|
||||
timeline: {
|
||||
sessionId: 'agent:main:main',
|
||||
@@ -86,6 +86,16 @@ const { acpState, agentsState, artifactPanelState, artifactPanelProps, chatState
|
||||
chatWorkspacePath: '/workspace',
|
||||
setChatWorkspacePath: vi.fn(),
|
||||
},
|
||||
talkState: {
|
||||
status: 'idle',
|
||||
isActive: false,
|
||||
sessionKey: null as string | null,
|
||||
transcripts: [] as Array<{ role: 'user' | 'assistant'; text: string; final: boolean }>,
|
||||
},
|
||||
talkController: {
|
||||
handleSessionChange: vi.fn().mockResolvedValue(undefined),
|
||||
stop: vi.fn().mockResolvedValue(undefined),
|
||||
},
|
||||
}));
|
||||
|
||||
const ensureAcpChatSubscriptions = vi.hoisted(() => vi.fn());
|
||||
@@ -128,6 +138,14 @@ vi.mock('@/stores/gateway', () => ({
|
||||
useGatewayStore: (selector: (state: typeof gatewayState) => unknown) => selector(gatewayState),
|
||||
}));
|
||||
|
||||
vi.mock('@/stores/realtime-talk', () => ({
|
||||
useRealtimeTalkStore: (selector: (state: typeof talkState) => unknown) => selector(talkState),
|
||||
}));
|
||||
|
||||
vi.mock('@/lib/talk/realtime-talk-controller', () => ({
|
||||
realtimeTalkController: talkController,
|
||||
}));
|
||||
|
||||
vi.mock('@/hooks/use-stick-to-bottom-instant', () => ({
|
||||
useStickToBottomInstant: () => ({
|
||||
contentRef: { current: null },
|
||||
@@ -137,10 +155,6 @@ vi.mock('@/hooks/use-stick-to-bottom-instant', () => ({
|
||||
}),
|
||||
}));
|
||||
|
||||
vi.mock('@/hooks/use-min-loading', () => ({
|
||||
useMinLoading: () => false,
|
||||
}));
|
||||
|
||||
vi.mock('@/pages/Chat/ChatToolbar', () => ({
|
||||
ChatToolbar: () => <div data-testid="mock-chat-toolbar" />,
|
||||
}));
|
||||
@@ -154,6 +168,7 @@ vi.mock('@/pages/Chat/ChatInput', () => ({
|
||||
workspaceLabel,
|
||||
workspacePath,
|
||||
workspaceReadOnly,
|
||||
talkActive,
|
||||
}: {
|
||||
disabled?: boolean;
|
||||
onSend: (text: string, attachments?: Array<Record<string, unknown>>, targetAgentId?: string | null) => void;
|
||||
@@ -162,9 +177,10 @@ vi.mock('@/pages/Chat/ChatInput', () => ({
|
||||
workspaceLabel?: string;
|
||||
workspacePath?: string;
|
||||
workspaceReadOnly?: boolean;
|
||||
talkActive?: boolean;
|
||||
onSelectWorkspace?: (path: string) => void;
|
||||
}) => (
|
||||
<div data-testid="mock-chat-input" data-disabled={disabled ? 'true' : 'false'} data-sending={sending ? 'true' : 'false'}>
|
||||
<div data-testid="mock-chat-input" data-disabled={disabled ? 'true' : 'false'} data-sending={sending ? 'true' : 'false'} data-talk-active={talkActive ? 'true' : 'false'}>
|
||||
<span data-testid="mock-workspace-label">{workspaceLabel}</span>
|
||||
<span data-testid="mock-workspace-path">{workspacePath}</span>
|
||||
<span data-testid="mock-workspace-readonly">{workspaceReadOnly ? 'readonly' : 'editable'}</span>
|
||||
@@ -360,6 +376,14 @@ describe('ACP Chat page', () => {
|
||||
settingsState.chatWorkspacePath = '/workspace';
|
||||
settingsState.setChatWorkspacePath.mockReset();
|
||||
gatewayState.status = { state: 'running', gatewayReady: true, port: 18789 };
|
||||
talkState.status = 'idle';
|
||||
talkState.isActive = false;
|
||||
talkState.sessionKey = null;
|
||||
talkState.transcripts = [];
|
||||
talkController.handleSessionChange.mockReset();
|
||||
talkController.handleSessionChange.mockResolvedValue(undefined);
|
||||
talkController.stop.mockReset();
|
||||
talkController.stop.mockResolvedValue(undefined);
|
||||
});
|
||||
|
||||
it('renders ACP inline timeline content', async () => {
|
||||
@@ -384,6 +408,26 @@ describe('ACP Chat page', () => {
|
||||
});
|
||||
});
|
||||
|
||||
it('locks the Chat composer and best-effort stops Talk on session switch and unmount', async () => {
|
||||
talkState.status = 'connecting';
|
||||
talkState.isActive = true;
|
||||
talkState.sessionKey = 'agent:main:main';
|
||||
const { rerender, unmount } = render(<Chat />);
|
||||
|
||||
expect(screen.getByTestId('mock-chat-input')).toHaveAttribute('data-talk-active', 'true');
|
||||
|
||||
chatState.currentSessionKey = 'agent:main:session-2';
|
||||
chatState.sessions = [{ key: 'agent:main:session-2', workspacePath: '/workspace' }];
|
||||
acpState.activeSessionKey = 'agent:main:session-2';
|
||||
acpState.timeline = { ...emptyTimeline(), sessionId: 'agent:main:session-2' };
|
||||
rerender(<Chat />);
|
||||
|
||||
await waitFor(() => expect(talkController.handleSessionChange).toHaveBeenCalledWith('agent:main:session-2'));
|
||||
expect(screen.getByTestId('mock-chat-input')).toHaveAttribute('data-talk-active', 'false');
|
||||
unmount();
|
||||
expect(talkController.stop).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('sends ready staged attachments and cancels through the ACP session store', async () => {
|
||||
acpState.workspaceRoot = '/workspace';
|
||||
acpState.cwd = '/workspace';
|
||||
|
||||
@@ -71,10 +71,6 @@ vi.mock('@/hooks/use-stick-to-bottom-instant', () => ({
|
||||
}),
|
||||
}));
|
||||
|
||||
vi.mock('@/hooks/use-min-loading', () => ({
|
||||
useMinLoading: (value: boolean) => value,
|
||||
}));
|
||||
|
||||
vi.mock('@/pages/Chat/ChatInput', () => ({
|
||||
ChatInput: () => <div data-testid="chat-input" />,
|
||||
}));
|
||||
|
||||
@@ -97,10 +97,6 @@ vi.mock('@/hooks/use-stick-to-bottom-instant', () => ({
|
||||
})),
|
||||
}));
|
||||
|
||||
vi.mock('@/hooks/use-min-loading', () => ({
|
||||
useMinLoading: () => false,
|
||||
}));
|
||||
|
||||
vi.mock('@/pages/Chat/ChatToolbar', () => ({ ChatToolbar: () => null }));
|
||||
vi.mock('@/pages/Chat/ChatInput', () => ({ ChatInput: () => null }));
|
||||
|
||||
|
||||
@@ -2,8 +2,11 @@ import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { fireEvent, render, screen, waitFor } from '@testing-library/react';
|
||||
import { ChatInput } from '@/pages/Chat/ChatInput';
|
||||
import { TooltipProvider } from '@/components/ui/tooltip';
|
||||
import { useRealtimeTalkStore } from '@/stores/realtime-talk';
|
||||
import { realtimeTalkController } from '@/lib/talk/realtime-talk-controller';
|
||||
const hostApiFetchMock = vi.hoisted(() => vi.fn());
|
||||
const hostApiDialogOpenMock = vi.hoisted(() => vi.fn());
|
||||
const hostApiTalkCatalogMock = vi.hoisted(() => vi.fn());
|
||||
const toastErrorMock = vi.hoisted(() => vi.fn());
|
||||
const { agentsState, chatState, gatewayState, providersState, artifactPanelMocks } = vi.hoisted(() => ({
|
||||
agentsState: {
|
||||
@@ -13,6 +16,8 @@ const { agentsState, chatState, gatewayState, providersState, artifactPanelMocks
|
||||
},
|
||||
chatState: {
|
||||
currentAgentId: 'main',
|
||||
currentSessionKey: 'agent:main:session-1',
|
||||
sessions: [{ key: 'agent:main:session-1' }],
|
||||
},
|
||||
gatewayState: {
|
||||
status: { state: 'running', port: 18789 },
|
||||
@@ -71,6 +76,9 @@ vi.mock('@/lib/host-api', () => ({
|
||||
dialog: {
|
||||
open: hostApiDialogOpenMock,
|
||||
},
|
||||
talk: {
|
||||
catalog: hostApiTalkCatalogMock,
|
||||
},
|
||||
},
|
||||
}));
|
||||
|
||||
@@ -138,12 +146,25 @@ function translate(key: string, vars?: Record<string, unknown>): string {
|
||||
return 'Preview SKILL.md';
|
||||
case 'composer.skillPreviewNotFound':
|
||||
return 'Skill not found';
|
||||
case 'talk.start':
|
||||
return 'Start Talk';
|
||||
case 'talk.stop':
|
||||
return 'Stop Talk';
|
||||
case 'talk.status.connecting':
|
||||
return 'Connecting Talk';
|
||||
case 'talk.status.listening':
|
||||
return 'Listening';
|
||||
case 'talk.unavailable':
|
||||
return 'Talk unavailable';
|
||||
case 'talk.unavailable.configuration':
|
||||
return 'Configure Talk in Settings';
|
||||
default:
|
||||
return key;
|
||||
}
|
||||
}
|
||||
|
||||
vi.mock('react-i18next', () => ({
|
||||
initReactI18next: { type: '3rdParty', init: vi.fn() },
|
||||
useTranslation: () => ({
|
||||
t: translate,
|
||||
}),
|
||||
@@ -224,6 +245,9 @@ describe('ChatInput agent targeting', () => {
|
||||
agentsState.defaultModelRef = null;
|
||||
agentsState.updateAgentModel.mockReset();
|
||||
chatState.currentAgentId = 'main';
|
||||
chatState.currentSessionKey = 'agent:main:session-1';
|
||||
chatState.sessions = [{ key: 'agent:main:session-1' }];
|
||||
useRealtimeTalkStore.getState().reset();
|
||||
gatewayState.status = { state: 'running', port: 18789 };
|
||||
providersState.accounts = [];
|
||||
providersState.statuses = [];
|
||||
@@ -232,6 +256,13 @@ describe('ChatInput agent targeting', () => {
|
||||
providersState.refreshProviderSnapshot.mockReset();
|
||||
vi.mocked(hostApiFetchMock).mockReset();
|
||||
vi.mocked(hostApiDialogOpenMock).mockReset();
|
||||
hostApiTalkCatalogMock.mockReset();
|
||||
hostApiTalkCatalogMock.mockResolvedValue({
|
||||
modes: ['realtime'],
|
||||
transports: ['gateway-relay'],
|
||||
brains: ['agent-consult'],
|
||||
realtime: { ready: true, providers: [] },
|
||||
});
|
||||
toastErrorMock.mockReset();
|
||||
artifactPanelMocks.openPreview.mockReset();
|
||||
});
|
||||
@@ -252,6 +283,122 @@ describe('ChatInput agent targeting', () => {
|
||||
expect(screen.queryByTestId('chat-composer-zoomies')).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('locks composer controls while Talk is connecting and restores the exact local draft after cleanup', () => {
|
||||
const { rerender } = renderChatInput();
|
||||
const input = screen.getByTestId('chat-composer-input');
|
||||
fireEvent.change(input, { target: { value: 'Keep this draft exactly' } });
|
||||
|
||||
useRealtimeTalkStore.getState().reserve('agent:main:session-1');
|
||||
useRealtimeTalkStore.getState().setInputLevel(0.4);
|
||||
rerender(
|
||||
<TooltipProvider>
|
||||
<ChatInput onSend={vi.fn()} />
|
||||
</TooltipProvider>,
|
||||
);
|
||||
|
||||
expect(screen.getByTestId('chat-composer-input')).toBeDisabled();
|
||||
expect(screen.getByTitle('Attach files')).toBeDisabled();
|
||||
expect(screen.getByTestId('chat-composer-send')).toBeDisabled();
|
||||
expect(screen.getByTestId('chat-composer-talk')).toHaveAttribute('aria-label', 'Stop Talk');
|
||||
expect(screen.getByTestId('chat-talk-status')).toHaveTextContent('Connecting Talk');
|
||||
expect(screen.getByTestId('chat-talk-input-level')).toHaveAttribute('aria-valuenow', '0.4');
|
||||
|
||||
useRealtimeTalkStore.getState().finish('cancelled');
|
||||
rerender(
|
||||
<TooltipProvider>
|
||||
<ChatInput onSend={vi.fn()} />
|
||||
</TooltipProvider>,
|
||||
);
|
||||
|
||||
expect(screen.getByTestId('chat-composer-input')).toHaveValue('Keep this draft exactly');
|
||||
expect(screen.getByTestId('chat-composer-input')).not.toBeDisabled();
|
||||
expect(screen.getByTestId('chat-composer-talk')).toHaveAttribute('aria-label', 'Start Talk');
|
||||
});
|
||||
|
||||
it('preserves attachment drafts and blocks attachment mutations while Talk is active', async () => {
|
||||
vi.mocked(hostApiFetchMock).mockRejectedValueOnce(new Error('staging failed'));
|
||||
const { container, rerender } = renderChatInput();
|
||||
const attachment = new File(['attachment'], 'draft.txt', { type: 'text/plain' });
|
||||
Object.defineProperty(attachment, 'path', { value: '/tmp/draft.txt' });
|
||||
|
||||
fireEvent.drop(container.firstElementChild as Element, {
|
||||
dataTransfer: {
|
||||
items: [{
|
||||
kind: 'file',
|
||||
getAsFile: () => attachment,
|
||||
webkitGetAsEntry: () => ({ isDirectory: false, isFile: true }),
|
||||
}],
|
||||
files: [attachment],
|
||||
},
|
||||
});
|
||||
await screen.findByText('draft.txt');
|
||||
await screen.findByText('Retry failed attachments');
|
||||
|
||||
useRealtimeTalkStore.getState().reserve('agent:main:session-1');
|
||||
rerender(
|
||||
<TooltipProvider>
|
||||
<ChatInput onSend={vi.fn()} />
|
||||
</TooltipProvider>,
|
||||
);
|
||||
|
||||
const retry = screen.getByText('Retry failed attachments').closest('button');
|
||||
expect(retry).toBeDisabled();
|
||||
expect(screen.getByTestId('chat-attachment-remove')).toBeDisabled();
|
||||
expect(screen.getByTitle('Attach files')).toBeDisabled();
|
||||
|
||||
fireEvent.click(retry!);
|
||||
fireEvent.drop(container.firstElementChild as Element, {
|
||||
dataTransfer: {
|
||||
items: [{
|
||||
kind: 'file',
|
||||
getAsFile: () => attachment,
|
||||
webkitGetAsEntry: () => ({ isDirectory: false, isFile: true }),
|
||||
}],
|
||||
files: [attachment],
|
||||
},
|
||||
});
|
||||
|
||||
expect(hostApiFetchMock).toHaveBeenCalledTimes(1);
|
||||
expect(screen.getByText('draft.txt')).toBeInTheDocument();
|
||||
expect(hostApiDialogOpenMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('routes catalog configuration failures to the Talk Settings entry point', async () => {
|
||||
const onConfigureTalk = vi.fn();
|
||||
hostApiTalkCatalogMock.mockResolvedValue({
|
||||
modes: ['realtime'],
|
||||
transports: ['gateway-relay'],
|
||||
brains: ['agent-consult'],
|
||||
realtime: { ready: false, reason: 'not configured', providers: [] },
|
||||
});
|
||||
render(
|
||||
<TooltipProvider>
|
||||
<ChatInput onSend={vi.fn()} onConfigureTalk={onConfigureTalk} />
|
||||
</TooltipProvider>,
|
||||
);
|
||||
|
||||
const talkButton = await screen.findByTestId('chat-composer-talk');
|
||||
await waitFor(() => expect(talkButton).toBeEnabled());
|
||||
fireEvent.click(talkButton);
|
||||
|
||||
expect(onConfigureTalk).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it('does not start Talk while the composer is disabled by ACP lifecycle state', async () => {
|
||||
const start = vi.spyOn(realtimeTalkController, 'start').mockResolvedValue(false);
|
||||
render(
|
||||
<TooltipProvider>
|
||||
<ChatInput onSend={vi.fn()} disabled />
|
||||
</TooltipProvider>,
|
||||
);
|
||||
|
||||
const talkButton = screen.getByTestId('chat-composer-talk');
|
||||
expect(talkButton).toBeDisabled();
|
||||
fireEvent.click(talkButton);
|
||||
expect(start).not.toHaveBeenCalled();
|
||||
start.mockRestore();
|
||||
});
|
||||
|
||||
it('shows an image-generation indicator without locking the composer for background work', () => {
|
||||
render(
|
||||
<TooltipProvider>
|
||||
|
||||
@@ -102,10 +102,6 @@ vi.mock('@/hooks/use-stick-to-bottom-instant', () => ({
|
||||
})),
|
||||
}));
|
||||
|
||||
vi.mock('@/hooks/use-min-loading', () => ({
|
||||
useMinLoading: () => false,
|
||||
}));
|
||||
|
||||
vi.mock('@/pages/Chat/ChatToolbar', () => ({ ChatToolbar: () => null }));
|
||||
vi.mock('@/pages/Chat/ChatInput', () => ({ ChatInput: () => null }));
|
||||
|
||||
|
||||
@@ -102,10 +102,6 @@ vi.mock('@/hooks/use-stick-to-bottom-instant', () => ({
|
||||
}),
|
||||
}));
|
||||
|
||||
vi.mock('@/hooks/use-min-loading', () => ({
|
||||
useMinLoading: () => false,
|
||||
}));
|
||||
|
||||
vi.mock('@/pages/Chat/ChatInput', () => ({
|
||||
ChatInput: () => null,
|
||||
}));
|
||||
|
||||
@@ -102,10 +102,6 @@ vi.mock('@/hooks/use-stick-to-bottom-instant', () => ({
|
||||
})),
|
||||
}));
|
||||
|
||||
vi.mock('@/hooks/use-min-loading', () => ({
|
||||
useMinLoading: () => false,
|
||||
}));
|
||||
|
||||
vi.mock('@/pages/Chat/ChatToolbar', () => ({
|
||||
ChatToolbar: () => null,
|
||||
}));
|
||||
|
||||
@@ -339,6 +339,34 @@ describe('OpenClaw config delivery coordinator', () => {
|
||||
expect(JSON.parse((gatewayManager.rpc.mock.calls[3][1] as { raw: string }).raw)).toEqual({ value: 5 });
|
||||
});
|
||||
|
||||
it('runs transaction-serialized before-apply validation before each pure mutator replay', async () => {
|
||||
const gatewayManager = createGatewayManager();
|
||||
gatewayManager.rpc
|
||||
.mockResolvedValueOnce({ raw: '{ value: 1 }', hash: 'hash-1' })
|
||||
.mockRejectedValueOnce(new Error('config changed since last load; re-run config.get and retry'))
|
||||
.mockResolvedValueOnce({ raw: '{ value: 4 }', hash: 'hash-2' })
|
||||
.mockResolvedValueOnce({ ok: true });
|
||||
registerOpenClawConfigCoordinator(gatewayManager);
|
||||
const events: string[] = [];
|
||||
|
||||
await mutateOpenClawConfig((config) => {
|
||||
events.push(`mutate:${String(config.value)}`);
|
||||
config.value = Number(config.value) + 1;
|
||||
}, {
|
||||
beforeApply: async () => {
|
||||
events.push('validate');
|
||||
},
|
||||
});
|
||||
|
||||
expect(events).toEqual(['validate', 'mutate:1', 'validate', 'mutate:4']);
|
||||
expect(gatewayManager.rpc.mock.calls.map(([method]) => method)).toEqual([
|
||||
'config.get',
|
||||
'config.set',
|
||||
'config.get',
|
||||
'config.set',
|
||||
]);
|
||||
});
|
||||
|
||||
it('does not commit a no-op mutation', async () => {
|
||||
const gatewayManager = createGatewayManager();
|
||||
gatewayManager.rpc.mockResolvedValueOnce({ raw: '{ value: 1 }', hash: 'hash-1' });
|
||||
|
||||
@@ -48,6 +48,16 @@ describe('dispatchProtocolEvent', () => {
|
||||
expect(emitter.emit).toHaveBeenCalledWith('chat:message', { message: { text: 'hello' } });
|
||||
});
|
||||
|
||||
it('dispatches talk.event to talk:event without the generic notification path', () => {
|
||||
const emitter = createMockEmitter();
|
||||
const payload = { relaySessionId: 'relay-1', type: 'transcript', role: 'assistant', text: 'Hello' };
|
||||
|
||||
dispatchProtocolEvent(emitter, 'talk.event', payload);
|
||||
|
||||
expect(emitter.emit).toHaveBeenCalledWith('talk:event', payload);
|
||||
expect(emitter.emit).not.toHaveBeenCalledWith('notification', expect.anything());
|
||||
});
|
||||
|
||||
it('does not normalize non-terminal lifecycle phase=end as run.ended', () => {
|
||||
const emitter = createMockEmitter();
|
||||
const payload = {
|
||||
|
||||
@@ -37,4 +37,23 @@ describe('harness git changed files', () => {
|
||||
await rm(repo, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('does not treat untracked implementation plans as task changes', async () => {
|
||||
const repo = await mkdtemp(path.join(tmpdir(), 'clawx-harness-git-'));
|
||||
|
||||
try {
|
||||
await mkdir(path.join(repo, 'docs', 'plans'), { recursive: true });
|
||||
await writeFile(path.join(repo, 'tracked.txt'), 'tracked\n');
|
||||
await writeFile(path.join(repo, 'docs', 'plans', 'user-plan.md'), '# User plan\n');
|
||||
await git(repo, ['init']);
|
||||
await git(repo, ['config', 'user.email', 'test@example.com']);
|
||||
await git(repo, ['config', 'user.name', 'Test']);
|
||||
await git(repo, ['add', 'tracked.txt']);
|
||||
await git(repo, ['commit', '-m', 'init']);
|
||||
|
||||
expect(await getChangedFiles('HEAD', repo)).not.toContain('docs/plans/user-plan.md');
|
||||
} finally {
|
||||
await rm(repo, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -4,6 +4,7 @@ import path from 'node:path';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import {
|
||||
isGatewayBackendCommunicationTask,
|
||||
isPluginLifecycleTask,
|
||||
loadRuleSpecs,
|
||||
loadScenarioSpecs,
|
||||
@@ -13,6 +14,8 @@ import {
|
||||
} from '../../harness/src/specs.mjs';
|
||||
import {
|
||||
scanBackendCommunicationBoundary,
|
||||
scanRealtimeTalkAuthorityText,
|
||||
selectRealtimeTalkAuthorityFiles,
|
||||
touchesCommunicationPath,
|
||||
validateGatewayTaskSpec,
|
||||
validatePluginLifecycleTaskSpec,
|
||||
@@ -30,6 +33,227 @@ async function readMarkdownTree(directory: string): Promise<Array<{ file: string
|
||||
}
|
||||
|
||||
describe('harness specs', () => {
|
||||
it('defines the realtime Talk authority, locale, and documentation contract', async () => {
|
||||
const expectedRules = [
|
||||
'renderer-main-boundary',
|
||||
'backend-communication-boundary',
|
||||
'api-client-transport-policy',
|
||||
'host-api-fallback-policy',
|
||||
'host-events-fallback-policy',
|
||||
'gateway-readiness-policy',
|
||||
'gateway-heartbeat-safety',
|
||||
'openclaw-config-delivery',
|
||||
'channel-plugin-migration-guards',
|
||||
'capability-owner-resolution',
|
||||
'active-config-guards',
|
||||
'provider-default-invariant',
|
||||
'provider-model-metadata-preservation',
|
||||
'provider-model-selection-authority',
|
||||
'sidebar-session-attention-authority',
|
||||
'web-browser-security-and-lifecycle',
|
||||
'acp-chat-state-and-history',
|
||||
'ui-i18n-design-tokens',
|
||||
'realtime-talk-openclaw-authority',
|
||||
'e2e-parallel-isolation',
|
||||
'comms-regression',
|
||||
'docs-sync',
|
||||
];
|
||||
const [task, rules, scenarios, reference, chatSource, transcriptSource, settingsSource, talkStoreSource, readmes, locales] = await Promise.all([
|
||||
loadSpec('harness/specs/tasks/add-realtime-talk.md'),
|
||||
loadRuleSpecs(),
|
||||
loadScenarioSpecs(),
|
||||
readFile('harness/reference/realtime-talk.md', 'utf8'),
|
||||
readFile('src/pages/Chat/ChatInput.tsx', 'utf8'),
|
||||
readFile('src/pages/Chat/LiveTalkTranscript.tsx', 'utf8'),
|
||||
readFile('src/components/settings/TalkSettings.tsx', 'utf8'),
|
||||
readFile('src/stores/realtime-talk.ts', 'utf8'),
|
||||
Promise.all(['README.md', 'README.zh-CN.md', 'README.ja-JP.md'].map((file) => readFile(file, 'utf8'))),
|
||||
Promise.all(['en', 'zh', 'ja', 'ru'].flatMap((locale) => [
|
||||
readFile(`shared/i18n/locales/${locale}/chat.json`, 'utf8'),
|
||||
readFile(`shared/i18n/locales/${locale}/settings.json`, 'utf8'),
|
||||
])),
|
||||
]);
|
||||
const scenarioById = new Map(scenarios.map((scenario) => [scenario.data.id, scenario]));
|
||||
const ruleIds = new Set(rules.map((rule) => rule.data.id));
|
||||
const talkRule = rules.find((rule) => rule.data.id === 'realtime-talk-openclaw-authority');
|
||||
const staticTalkKeys = (source: string) => [...source.matchAll(/t\('((?:talk\.)[^']+)'/g)]
|
||||
.map((match) => match[1]);
|
||||
const chatTalkKeys = staticTalkKeys(`${chatSource}\n${transcriptSource}`);
|
||||
const settingsTalkKeys = staticTalkKeys(settingsSource);
|
||||
const statuses = [...(talkStoreSource.match(/export type RealtimeTalkStatus = ([^;]+);/)?.[1] ?? '')
|
||||
.matchAll(/'([^']+)'/g)].map((match) => match[1]);
|
||||
const keyShape = (value: unknown): unknown => (
|
||||
value && typeof value === 'object'
|
||||
? Object.fromEntries(Object.entries(value as Record<string, unknown>)
|
||||
.sort(([left], [right]) => left.localeCompare(right))
|
||||
.map(([key, nested]) => [key, keyShape(nested)]))
|
||||
: typeof value
|
||||
);
|
||||
|
||||
expect(task.data).toMatchObject({
|
||||
id: 'add-realtime-talk',
|
||||
scenario: 'gateway-backend-communication',
|
||||
scenarios: ['realtime-talk'],
|
||||
taskType: 'runtime-bridge',
|
||||
requiredProfiles: ['fast', 'comms', 'e2e'],
|
||||
requiredRules: expectedRules,
|
||||
docs: { required: true },
|
||||
});
|
||||
expect(isGatewayBackendCommunicationTask(task)).toBe(true);
|
||||
expect([...task.data.requiredRules as string[]].filter((ruleId) => !ruleIds.has(ruleId))).toEqual([]);
|
||||
expect(task.data.requiredRules).toEqual(expect.arrayContaining(
|
||||
scenarioById.get('gateway-backend-communication')?.data.requiredRules as string[],
|
||||
));
|
||||
expect(task.data.requiredRules).toEqual(expect.arrayContaining(
|
||||
scenarioById.get('realtime-talk')?.data.requiredRules as string[],
|
||||
));
|
||||
expect(scenarioById.get('gateway-backend-communication')?.data.requiredRules)
|
||||
.toContain('realtime-talk-openclaw-authority');
|
||||
expect(scenarioById.get('acp-chat-experience')?.data.requiredRules)
|
||||
.toContain('realtime-talk-openclaw-authority');
|
||||
expect(talkRule?.data.appliesTo).toEqual(expect.arrayContaining([
|
||||
'realtime-talk',
|
||||
'gateway-backend-communication',
|
||||
'acp-chat-experience',
|
||||
]));
|
||||
expect(rules.find((rule) => rule.data.id === 'e2e-parallel-isolation')?.data.appliesTo)
|
||||
.toContain('realtime-talk');
|
||||
for (const anchor of [
|
||||
'Gateway Relay',
|
||||
'one global relay',
|
||||
'Main owns',
|
||||
'Renderer',
|
||||
'transient',
|
||||
'OpenClaw',
|
||||
'ACP',
|
||||
'no ClawX transcript persistence',
|
||||
'config',
|
||||
'secrets',
|
||||
'VAD',
|
||||
]) expect(reference).toContain(anchor);
|
||||
|
||||
expect(chatTalkKeys).not.toEqual([]);
|
||||
expect(settingsTalkKeys).not.toEqual([]);
|
||||
expect(statuses).not.toEqual([]);
|
||||
expect(chatSource).toContain('`talk.status.${talkStatus}`');
|
||||
const [englishChat, englishSettings] = locales.map((locale) => JSON.parse(locale) as { talk: unknown });
|
||||
for (let index = 0; index < locales.length; index += 2) {
|
||||
const chat = JSON.parse(locales[index]) as Record<string, unknown>;
|
||||
const settings = JSON.parse(locales[index + 1]) as Record<string, unknown>;
|
||||
for (const key of chatTalkKeys) expect(key.split('.').reduce<unknown>((value, part) => (
|
||||
value && typeof value === 'object' ? (value as Record<string, unknown>)[part] : undefined
|
||||
), chat), `${key} must resolve without rendering its raw key`).toEqual(expect.any(String));
|
||||
for (const key of settingsTalkKeys) expect(key.split('.').reduce<unknown>((value, part) => (
|
||||
value && typeof value === 'object' ? (value as Record<string, unknown>)[part] : undefined
|
||||
), settings), `${key} must resolve without rendering its raw key`).toEqual(expect.any(String));
|
||||
for (const status of statuses) expect((chat.talk as Record<string, unknown>).status)
|
||||
.toHaveProperty(status, expect.any(String));
|
||||
expect(keyShape(chat.talk)).toEqual(keyShape(englishChat.talk));
|
||||
expect(keyShape(settings.talk)).toEqual(keyShape(englishSettings.talk));
|
||||
}
|
||||
expect((JSON.parse(locales[0]) as { talk: { unavailable: { configuration: string } } }).talk.unavailable.configuration)
|
||||
.toContain('{{reason}}');
|
||||
for (const locale of locales.filter((_, index) => index % 2 === 0).slice(1)) {
|
||||
expect((JSON.parse(locale) as { talk: { unavailable: { configuration: string } } }).talk.unavailable.configuration)
|
||||
.toContain('{{reason}}');
|
||||
}
|
||||
for (const readme of readmes) {
|
||||
expect(readme).toContain('Talk');
|
||||
expect(readme).toContain('Gateway Relay');
|
||||
expect(readme).toContain('ACP');
|
||||
expect(readme).toContain('macOS');
|
||||
}
|
||||
expect(readmes[0]).toContain('readiness');
|
||||
expect((task.data.requiredTests as string[])).toEqual(expect.arrayContaining([
|
||||
'pnpm harness validate --spec harness/specs/tasks/add-realtime-talk.md --since bd1aac8e',
|
||||
'pnpm harness run --spec harness/specs/tasks/add-realtime-talk.md --since bd1aac8e --dry-run',
|
||||
]));
|
||||
expect((task.data.acceptance as string[]).join('\n')).toContain('readiness is displayed and checked');
|
||||
for (const path of [
|
||||
'electron/gateway/config-delivery.ts',
|
||||
'src/stores/acp-chat-session.ts',
|
||||
'tests/unit/acp-chat-store.test.ts',
|
||||
'tests/unit/gateway-config-delivery.test.ts',
|
||||
'tests/e2e/developer-mode.spec.ts',
|
||||
'tests/unit/harness-specs.test.ts',
|
||||
'README.md',
|
||||
'shared/i18n/locales/**/chat.json',
|
||||
]) {
|
||||
expect(task.data.touchedAreas).toContain(path);
|
||||
expect(scenarioById.get('realtime-talk')?.data.ownedPaths).toContain(path);
|
||||
}
|
||||
expect(scenarioById.get('realtime-talk')?.body)
|
||||
.toContain('does not require E2E_EXCLUSIVE_TAG because its audio/media mocks are local');
|
||||
});
|
||||
|
||||
it('rejects direct renderer Talk transports, persistence, and synthetic ACP projections', () => {
|
||||
const failures = scanRealtimeTalkAuthorityText('src/lib/talk/unsafe.ts', `
|
||||
window.electron.ipcRenderer.invoke('talk:start');
|
||||
fetch('http://127.0.0.1:18789/talk');
|
||||
new WebSocket('ws://127.0.0.1:18789');
|
||||
new WebSocket('wss://provider.example/realtime');
|
||||
new RTCPeerConnection();
|
||||
localStorage.setItem('talk-transcript', text);
|
||||
appendFile('talk.jsonl', text);
|
||||
appendSyntheticAssistantMessage(talkTranscript);
|
||||
`);
|
||||
|
||||
expect(failures).toEqual(expect.arrayContaining([
|
||||
'src/lib/talk/unsafe.ts: Talk renderer must not call window.electron.ipcRenderer.invoke directly',
|
||||
'src/lib/talk/unsafe.ts: Talk renderer must not fetch Gateway HTTP directly',
|
||||
'src/lib/talk/unsafe.ts: Talk renderer must not open Gateway or provider WebSocket connections directly',
|
||||
'src/lib/talk/unsafe.ts: Talk renderer must not create WebRTC connections',
|
||||
'src/lib/talk/unsafe.ts: Talk transcripts must not use browser persistence',
|
||||
'src/lib/talk/unsafe.ts: Talk transcripts must not write local files',
|
||||
'src/lib/talk/unsafe.ts: Talk must not project direct transcripts into ACP history',
|
||||
]));
|
||||
expect(scanRealtimeTalkAuthorityText('src/lib/talk/allowed.ts', `
|
||||
hostApi.talk.startRelay({ sessionKey });
|
||||
hostEvents.onTalkEvent(handleEvent);
|
||||
useRealtimeTalkStore.getState().appendTranscript(entry);
|
||||
`)).toEqual([]);
|
||||
});
|
||||
|
||||
it('selects every Talk authority path category without scanning unrelated files', () => {
|
||||
const selected = selectRealtimeTalkAuthorityFiles([
|
||||
'shared/talk/types.ts',
|
||||
'src/lib/talk/audio.ts',
|
||||
'src/lib/host-api.ts',
|
||||
'src/lib/host-events.ts',
|
||||
'src/stores/realtime-talk.ts',
|
||||
'src/stores/acp-chat-session.ts',
|
||||
'src/pages/Chat/TalkPanel.tsx',
|
||||
'src/components/settings/TalkSettings.tsx',
|
||||
'src/pages/Settings/index.tsx',
|
||||
'src/pages/Agents/index.tsx',
|
||||
'src/components/settings/ProviderSettings.tsx',
|
||||
'shared/acp-chat/types.ts',
|
||||
'electron/services/talk-api.ts',
|
||||
]);
|
||||
|
||||
expect(selected).toEqual([
|
||||
'shared/talk/types.ts',
|
||||
'src/lib/talk/audio.ts',
|
||||
'src/lib/host-api.ts',
|
||||
'src/lib/host-events.ts',
|
||||
'src/stores/realtime-talk.ts',
|
||||
'src/stores/acp-chat-session.ts',
|
||||
'src/pages/Chat/TalkPanel.tsx',
|
||||
'src/components/settings/TalkSettings.tsx',
|
||||
'src/pages/Settings/index.tsx',
|
||||
].sort());
|
||||
|
||||
for (const file of selected) {
|
||||
expect(scanRealtimeTalkAuthorityText(file, `
|
||||
new WebSocket('wss://provider.example/realtime');
|
||||
localStorage.setItem('talk-transcript', text);
|
||||
`)).toEqual(expect.arrayContaining([
|
||||
`${file}: Talk renderer must not open Gateway or provider WebSocket connections directly`,
|
||||
`${file}: Talk transcripts must not use browser persistence`,
|
||||
]));
|
||||
}
|
||||
});
|
||||
|
||||
it('defines the sidebar session attention harness contract', async () => {
|
||||
const expectedRules = [
|
||||
'renderer-main-boundary',
|
||||
|
||||
@@ -472,6 +472,65 @@ describe('hostApi facade', () => {
|
||||
}));
|
||||
});
|
||||
|
||||
it('routes Talk actions through the typed host facade', async () => {
|
||||
hostInvoke.mockResolvedValue({ id: 'req', ok: true, data: { ok: true } });
|
||||
const { hostApi } = await import('@/lib/host-api');
|
||||
|
||||
await hostApi.talk.catalog();
|
||||
await hostApi.talk.updateRealtimeSettings({
|
||||
provider: 'openai',
|
||||
model: 'gpt-realtime',
|
||||
speakerVoice: 'alloy',
|
||||
});
|
||||
await hostApi.talk.startRelay({ sessionKey: 'agent:main:session-1' });
|
||||
await hostApi.talk.appendAudio({ relaySessionId: 'relay-1', audioBase64: 'AQI=' });
|
||||
await hostApi.talk.cancelOutput({ relaySessionId: 'relay-1' });
|
||||
await hostApi.talk.submitToolResult({ relaySessionId: 'relay-1', callId: 'call-1', result: { ok: true } });
|
||||
await hostApi.talk.acknowledgeMark({ relaySessionId: 'relay-1', markName: 'mark-1' });
|
||||
await hostApi.talk.stopRelay({ relaySessionId: 'relay-1' });
|
||||
await hostApi.talk.startAgentConsult({
|
||||
relaySessionId: 'relay-1',
|
||||
sessionKey: 'agent:main:session-1',
|
||||
callId: 'call-1',
|
||||
args: {},
|
||||
});
|
||||
|
||||
expect(hostInvoke.mock.calls.map(([request]) => request)).toEqual(expect.arrayContaining([
|
||||
expect.objectContaining({ module: 'talk', action: 'catalog' }),
|
||||
expect.objectContaining({
|
||||
module: 'talk',
|
||||
action: 'updateRealtimeSettings',
|
||||
payload: { provider: 'openai', model: 'gpt-realtime', speakerVoice: 'alloy' },
|
||||
}),
|
||||
expect.objectContaining({ module: 'talk', action: 'startRelay', payload: { sessionKey: 'agent:main:session-1' } }),
|
||||
expect.objectContaining({ module: 'talk', action: 'appendAudio', payload: { relaySessionId: 'relay-1', audioBase64: 'AQI=' } }),
|
||||
expect.objectContaining({ module: 'talk', action: 'cancelOutput' }),
|
||||
expect.objectContaining({ module: 'talk', action: 'submitToolResult' }),
|
||||
expect.objectContaining({ module: 'talk', action: 'acknowledgeMark' }),
|
||||
expect.objectContaining({ module: 'talk', action: 'stopRelay' }),
|
||||
expect.objectContaining({ module: 'talk', action: 'startAgentConsult' }),
|
||||
]));
|
||||
});
|
||||
|
||||
it('keeps generic gateway RPC requests on the gateway facade for Main-side method validation', async () => {
|
||||
hostInvoke
|
||||
.mockResolvedValueOnce({ id: 'non-talk', ok: true, data: { ok: true } })
|
||||
.mockResolvedValueOnce({
|
||||
id: 'talk', ok: false, error: { code: 'VALIDATION', message: 'Talk Gateway RPCs must use the typed Talk API' },
|
||||
});
|
||||
const { hostApi } = await import('@/lib/host-api');
|
||||
|
||||
await expect(hostApi.gateway.rpc('sessions.list', { limit: 10 }, 500)).resolves.toEqual({ ok: true });
|
||||
await expect(hostApi.gateway.rpc('talk.session.create', {})).rejects.toThrow('Talk Gateway RPCs must use the typed Talk API');
|
||||
|
||||
expect(hostInvoke).toHaveBeenNthCalledWith(1, expect.objectContaining({
|
||||
module: 'gateway', action: 'rpc', payload: { method: 'sessions.list', params: { limit: 10 }, timeoutMs: 500 },
|
||||
}));
|
||||
expect(hostInvoke).toHaveBeenNthCalledWith(2, expect.objectContaining({
|
||||
module: 'gateway', action: 'rpc', payload: { method: 'talk.session.create', params: {}, timeoutMs: undefined },
|
||||
}));
|
||||
});
|
||||
|
||||
it('calls sessions.summaries through hostInvoke', async () => {
|
||||
hostInvoke.mockResolvedValueOnce({ id: 'req', ok: true, data: { success: true, summaries: [] } });
|
||||
const { hostApi } = await import('@/lib/host-api');
|
||||
|
||||
@@ -74,6 +74,18 @@ describe('hostEvents', () => {
|
||||
expect(on).toHaveBeenCalledWith('channel:wechat-qr', expect.any(Function));
|
||||
});
|
||||
|
||||
it('subscribes to Talk relay events over the dedicated IPC channel', async () => {
|
||||
const { hostEvents } = await import('@/lib/host-events');
|
||||
const handler = vi.fn();
|
||||
|
||||
hostEvents.onTalkEvent(handler);
|
||||
const callback = on.mock.calls[0]?.[1] as ((payload: unknown) => void) | undefined;
|
||||
callback?.({ relaySessionId: 'relay-1', type: 'ready' });
|
||||
|
||||
expect(on).toHaveBeenCalledWith('talk:event', expect.any(Function));
|
||||
expect(handler).toHaveBeenCalledWith({ relaySessionId: 'relay-1', type: 'ready' });
|
||||
});
|
||||
|
||||
it('does not create EventSource fallback', async () => {
|
||||
const eventSource = vi.fn();
|
||||
vi.stubGlobal('EventSource', eventSource);
|
||||
|
||||
@@ -404,7 +404,7 @@ describe('host services', () => {
|
||||
expect(setSettingMock).toHaveBeenCalledWith('recentWorkspacePaths', ['/Users/alex/workspace/ClawX']);
|
||||
});
|
||||
|
||||
it('routes validated generic gateway rpc directly to the manager', async () => {
|
||||
it('routes validated non-Talk gateway rpc directly to the manager and blocks Talk methods', async () => {
|
||||
const gatewayManager = {
|
||||
rpc: vi.fn(async () => ({ ok: true })),
|
||||
};
|
||||
@@ -426,6 +426,9 @@ describe('host services', () => {
|
||||
await expect(gatewayApi.rpc({ method: 'status', timeoutMs: 0 })).rejects.toThrow(
|
||||
'Invalid gateway RPC timeout',
|
||||
);
|
||||
await expect(gatewayApi.rpc({ method: ' talk.session.create ' })).rejects.toThrow(
|
||||
'Talk Gateway RPCs must use the typed Talk API',
|
||||
);
|
||||
expect(gatewayManager.rpc).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
@@ -1278,6 +1281,40 @@ describe('host services', () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it('registers the typed Talk service with Main-owned relay ownership and no legacy direct IPC channel', () => {
|
||||
const source = readFileSync(join(process.cwd(), 'electron/main/ipc-handlers.ts'), 'utf8');
|
||||
const mainSource = readFileSync(join(process.cwd(), 'electron/main/index.ts'), 'utf8');
|
||||
const registerIpcHandlersSource = source.slice(
|
||||
source.indexOf('export function registerIpcHandlers('),
|
||||
source.indexOf('function registerTypedHostHandlers('),
|
||||
);
|
||||
|
||||
expect(source).toContain('const talkRelayOwnership = createTalkRelayOwnership();');
|
||||
expect(source).toContain('talk: createTalkApi(gatewayManager, talkRelayOwnership)');
|
||||
expect(source).not.toMatch(/ipcMain\.handle\(\s*['"]talk:/);
|
||||
expect(mainSource).toContain('forwardActiveTalkEvent(talkRelayOwnership, data');
|
||||
const returnIndex = registerIpcHandlersSource.indexOf('return talkRelayOwnership;');
|
||||
expect(returnIndex).toBeGreaterThan(-1);
|
||||
[
|
||||
'registerGatewayHandlers(gatewayManager);',
|
||||
'registerOpenClawHandlers();',
|
||||
'registerProviderHandlers(gatewayManager);',
|
||||
'registerShellHandlers();',
|
||||
'registerDialogHandlers();',
|
||||
'registerAppHandlers();',
|
||||
'registerSettingsHandlers(gatewayManager);',
|
||||
'registerUsageHandlers();',
|
||||
'registerCronHandlers(gatewayManager);',
|
||||
'registerWindowHandlers(mainWindow);',
|
||||
'registerWhatsAppHandlers(mainWindow);',
|
||||
'registerFilePreviewHandlers();',
|
||||
].forEach((registration) => {
|
||||
const registrationIndex = registerIpcHandlersSource.indexOf(registration);
|
||||
expect(registrationIndex).toBeGreaterThan(-1);
|
||||
expect(registrationIndex).toBeLessThan(returnIndex);
|
||||
});
|
||||
});
|
||||
|
||||
it('loads session summaries and transcript history through the typed sessions service', async () => {
|
||||
const sessionsDir = join(testOpenClawConfigDir, 'agents', 'main', 'sessions');
|
||||
mkdirSync(sessionsDir, { recursive: true });
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
import { render, screen } from '@testing-library/react';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { LiveTalkTranscript } from '@/pages/Chat/LiveTalkTranscript';
|
||||
|
||||
vi.mock('react-i18next', () => ({
|
||||
initReactI18next: { type: '3rdParty', init: vi.fn() },
|
||||
useTranslation: () => ({
|
||||
t: (key: string) => ({
|
||||
'talk.transcript.label': 'Live Talk',
|
||||
'talk.transcript.microphone': 'Microphone',
|
||||
})[key] ?? key,
|
||||
}),
|
||||
}));
|
||||
|
||||
describe('LiveTalkTranscript', () => {
|
||||
it('renders transient direct Talk user and assistant bubbles with microphone markers', () => {
|
||||
render(
|
||||
<LiveTalkTranscript
|
||||
transcripts={[
|
||||
{ role: 'user', text: 'Can you hear me?', final: true },
|
||||
{ role: 'assistant', text: 'Yes, I can hear you.', final: false },
|
||||
]}
|
||||
/>,
|
||||
);
|
||||
|
||||
expect(screen.getByTestId('live-talk-transcript')).toBeInTheDocument();
|
||||
expect(screen.getByTestId('live-talk-user-message')).toHaveTextContent('Can you hear me?');
|
||||
expect(screen.getByTestId('live-talk-assistant-message')).toHaveTextContent('Yes, I can hear you.');
|
||||
expect(screen.getAllByLabelText('Microphone')).toHaveLength(2);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,439 @@
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import type { TalkRelayEvent, TalkRelaySession } from '@shared/talk/types';
|
||||
import { createRealtimeTalkController } from '@/lib/talk/realtime-talk-controller';
|
||||
import { stopActiveRealtimeTalk, useRealtimeTalkStore } from '@/stores/realtime-talk';
|
||||
|
||||
const relay: TalkRelaySession = {
|
||||
relaySessionId: 'relay-1',
|
||||
provider: 'openai',
|
||||
transport: 'gateway-relay',
|
||||
audio: {
|
||||
inputEncoding: 'pcm16',
|
||||
inputSampleRateHz: 24_000,
|
||||
outputEncoding: 'pcm16',
|
||||
outputSampleRateHz: 24_000,
|
||||
},
|
||||
};
|
||||
|
||||
function setup(overrides: Partial<{
|
||||
sending: boolean;
|
||||
activeSessionKey: string | null;
|
||||
selectedSession: { key: string; lastMessagePreview?: string } | null;
|
||||
startAudio: () => Promise<void>;
|
||||
relay: TalkRelaySession;
|
||||
}> = {}) {
|
||||
const eventListeners = new Set<(event: TalkRelayEvent) => void>();
|
||||
let gatewayStatusListener: ((status: { state: string; gatewayReady?: boolean }) => void) | null = null;
|
||||
let activeSessionKey = overrides.activeSessionKey ?? 'agent:main:session-1';
|
||||
let sending = overrides.sending ?? false;
|
||||
let selectedSession = 'selectedSession' in overrides
|
||||
? overrides.selectedSession
|
||||
: { key: activeSessionKey ?? '' };
|
||||
const talk = {
|
||||
startRelay: vi.fn().mockResolvedValue(overrides.relay ?? relay),
|
||||
appendAudio: vi.fn().mockResolvedValue({ ok: true }),
|
||||
cancelOutput: vi.fn().mockResolvedValue({ ok: true }),
|
||||
acknowledgeMark: vi.fn().mockResolvedValue({ ok: true }),
|
||||
stopRelay: vi.fn().mockResolvedValue({ ok: true }),
|
||||
startAgentConsult: vi.fn().mockResolvedValue({ runId: 'run-1' }),
|
||||
submitToolResult: vi.fn().mockResolvedValue({ ok: true }),
|
||||
};
|
||||
const audio = {
|
||||
start: vi.fn(overrides.startAudio ?? (async () => {})),
|
||||
stop: vi.fn(),
|
||||
enqueueOutput: vi.fn().mockResolvedValue(undefined),
|
||||
clearOutput: vi.fn(),
|
||||
};
|
||||
const unsubscribe = vi.fn();
|
||||
const reload = vi.fn().mockResolvedValue(true);
|
||||
const controller = createRealtimeTalkController({
|
||||
talk,
|
||||
subscribeTalk: (listener) => {
|
||||
eventListeners.add(listener);
|
||||
return () => {
|
||||
eventListeners.delete(listener);
|
||||
unsubscribe();
|
||||
};
|
||||
},
|
||||
subscribeGatewayStatus: (listener) => {
|
||||
gatewayStatusListener = listener;
|
||||
return vi.fn();
|
||||
},
|
||||
createAudio: () => audio,
|
||||
getAcpState: () => ({
|
||||
sending,
|
||||
activeSessionKey,
|
||||
}),
|
||||
getSelectedSession: () => selectedSession,
|
||||
reloadAcpSession: reload,
|
||||
});
|
||||
return {
|
||||
audio,
|
||||
controller,
|
||||
emit: (event: TalkRelayEvent) => {
|
||||
for (const listener of eventListeners) listener(event);
|
||||
},
|
||||
emitGatewayStatus: (status: { state: string; gatewayReady?: boolean }) => gatewayStatusListener?.(status),
|
||||
reload,
|
||||
setActiveSessionKey: (sessionKey: string | null) => {
|
||||
const wasDefaultSelection = selectedSession?.key === activeSessionKey;
|
||||
activeSessionKey = sessionKey;
|
||||
if (wasDefaultSelection) selectedSession = { key: sessionKey ?? '' };
|
||||
},
|
||||
setSelectedSession: (session: { key: string; lastMessagePreview?: string } | null) => { selectedSession = session; },
|
||||
setSending: (value: boolean) => { sending = value; },
|
||||
talk,
|
||||
unsubscribe,
|
||||
};
|
||||
}
|
||||
|
||||
describe('realtime Talk controller', () => {
|
||||
beforeEach(() => {
|
||||
useRealtimeTalkStore.getState().reset();
|
||||
});
|
||||
|
||||
it('starts one selected relay and serializes local PCM input through the typed Talk API', async () => {
|
||||
const { audio, controller, talk } = setup();
|
||||
|
||||
await expect(controller.start({ sessionKey: 'agent:main:session-1' })).resolves.toBe(true);
|
||||
audio.start.mock.calls[0]?.[0](new Float32Array([0, 1, -1, 0]), 48_000);
|
||||
await vi.waitFor(() => expect(talk.appendAudio).toHaveBeenCalledWith(expect.objectContaining({
|
||||
relaySessionId: 'relay-1',
|
||||
audioBase64: 'AAAAgA==',
|
||||
})));
|
||||
|
||||
expect(useRealtimeTalkStore.getState()).toMatchObject({
|
||||
status: 'listening',
|
||||
relaySessionId: 'relay-1',
|
||||
sessionKey: 'agent:main:session-1',
|
||||
});
|
||||
});
|
||||
|
||||
it('rejects stale relay events and clears direct transcript state on terminal paths', async () => {
|
||||
const { controller, emit, talk } = setup();
|
||||
await controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
|
||||
emit({ relaySessionId: 'stale', type: 'transcript', role: 'assistant', text: 'ignore', final: true });
|
||||
emit({ relaySessionId: 'relay-1', type: 'transcript', role: 'assistant', text: 'live', final: true });
|
||||
expect(useRealtimeTalkStore.getState().transcripts).toHaveLength(1);
|
||||
|
||||
emit({ relaySessionId: 'relay-1', type: 'close', reason: 'completed' });
|
||||
await vi.waitFor(() => expect(talk.stopRelay).toHaveBeenCalledWith({ relaySessionId: 'relay-1' }));
|
||||
expect(useRealtimeTalkStore.getState().transcripts).toEqual([]);
|
||||
});
|
||||
|
||||
it('cleans tracks, events, and relay ownership on session switch and gateway disconnect', async () => {
|
||||
const { audio, controller, emitGatewayStatus, setActiveSessionKey, talk, unsubscribe } = setup();
|
||||
await controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
|
||||
await controller.handleSessionChange('agent:main:session-2');
|
||||
expect(audio.stop).toHaveBeenCalledOnce();
|
||||
expect(unsubscribe).toHaveBeenCalledOnce();
|
||||
expect(talk.stopRelay).toHaveBeenCalledWith({ relaySessionId: 'relay-1' });
|
||||
|
||||
setActiveSessionKey('agent:main:session-2');
|
||||
await controller.start({ sessionKey: 'agent:main:session-2' });
|
||||
emitGatewayStatus({ state: 'stopped' });
|
||||
await vi.waitFor(() => expect(talk.stopRelay).toHaveBeenCalledTimes(2));
|
||||
expect(useRealtimeTalkStore.getState().status).toBe('disconnected');
|
||||
});
|
||||
|
||||
it('keeps the global Talk reservation until a session-switch relay close settles', async () => {
|
||||
let resolveStop!: () => void;
|
||||
const stopRelay = new Promise<void>((resolve) => { resolveStop = resolve; });
|
||||
const first = setup();
|
||||
const second = setup();
|
||||
first.talk.stopRelay.mockReturnValueOnce(stopRelay);
|
||||
await first.controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
|
||||
const closing = first.controller.handleSessionChange('agent:main:session-2');
|
||||
await vi.waitFor(() => expect(first.talk.stopRelay).toHaveBeenCalledWith({ relaySessionId: 'relay-1' }));
|
||||
expect(useRealtimeTalkStore.getState().isActive).toBe(false);
|
||||
|
||||
second.setActiveSessionKey('agent:main:session-2');
|
||||
await expect(second.controller.start({ sessionKey: 'agent:main:session-2' })).resolves.toBe(false);
|
||||
expect(second.talk.startRelay).not.toHaveBeenCalled();
|
||||
|
||||
resolveStop();
|
||||
await closing;
|
||||
await expect(second.controller.start({ sessionKey: 'agent:main:session-2' })).resolves.toBe(true);
|
||||
});
|
||||
|
||||
it('rejects overlapping starts while the first reservation is connecting', async () => {
|
||||
let resolveFirst!: (value: TalkRelaySession) => void;
|
||||
const firstRelay = new Promise<TalkRelaySession>((resolve) => { resolveFirst = resolve; });
|
||||
const { audio, controller, talk } = setup();
|
||||
talk.startRelay.mockReturnValueOnce(firstRelay);
|
||||
|
||||
const first = controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
const second = controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
resolveFirst({ ...relay, relaySessionId: 'relay-old' });
|
||||
|
||||
await expect(first).resolves.toBe(true);
|
||||
await expect(second).resolves.toBe(false);
|
||||
expect(audio.start).toHaveBeenCalledOnce();
|
||||
expect(useRealtimeTalkStore.getState().relaySessionId).toBe('relay-old');
|
||||
});
|
||||
|
||||
it('reserves Talk while connecting so competing starts and ACP prompts stay blocked', async () => {
|
||||
let resolveRelay!: (value: TalkRelaySession) => void;
|
||||
const pendingRelay = new Promise<TalkRelaySession>((resolve) => { resolveRelay = resolve; });
|
||||
const first = setup();
|
||||
const second = setup();
|
||||
first.talk.startRelay.mockReturnValueOnce(pendingRelay);
|
||||
|
||||
const starting = first.controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
expect(useRealtimeTalkStore.getState()).toMatchObject({ status: 'connecting', isActive: true });
|
||||
await expect(second.controller.start({ sessionKey: 'agent:main:session-1' })).resolves.toBe(false);
|
||||
expect(second.talk.startRelay).not.toHaveBeenCalled();
|
||||
resolveRelay(relay);
|
||||
await expect(starting).resolves.toBe(true);
|
||||
});
|
||||
|
||||
it('rechecks ACP state after startup before activating a relay', async () => {
|
||||
let resolveRelay!: (value: TalkRelaySession) => void;
|
||||
const pendingRelay = new Promise<TalkRelaySession>((resolve) => { resolveRelay = resolve; });
|
||||
const { controller, setSending, talk } = setup();
|
||||
talk.startRelay.mockReturnValueOnce(pendingRelay);
|
||||
|
||||
const starting = controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
setSending(true);
|
||||
resolveRelay(relay);
|
||||
|
||||
await expect(starting).resolves.toBe(false);
|
||||
expect(talk.stopRelay).toHaveBeenCalledWith({ relaySessionId: 'relay-1' });
|
||||
});
|
||||
|
||||
it('invalidates a pending start when ACP lifecycle cleanup runs before relay creation resolves', async () => {
|
||||
let resolveRelay!: (value: TalkRelaySession) => void;
|
||||
const pendingRelay = new Promise<TalkRelaySession>((resolve) => { resolveRelay = resolve; });
|
||||
const { audio, controller, talk } = setup();
|
||||
talk.startRelay.mockReturnValueOnce(pendingRelay);
|
||||
|
||||
const starting = controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
await stopActiveRealtimeTalk();
|
||||
resolveRelay(relay);
|
||||
|
||||
await expect(starting).resolves.toBe(false);
|
||||
expect(talk.stopRelay).toHaveBeenCalledWith({ relaySessionId: 'relay-1' });
|
||||
expect(audio.start).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('releases an early listener when a pending start is cancelled so only the next relay handles events', async () => {
|
||||
let resolveFirstRelay!: (value: TalkRelaySession) => void;
|
||||
const firstRelay = new Promise<TalkRelaySession>((resolve) => { resolveFirstRelay = resolve; });
|
||||
const { audio, controller, emit, talk, unsubscribe } = setup();
|
||||
talk.startRelay.mockReturnValueOnce(firstRelay).mockResolvedValueOnce({ ...relay, relaySessionId: 'relay-2' });
|
||||
|
||||
const firstStart = controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
await vi.waitFor(() => expect(talk.startRelay).toHaveBeenCalledOnce());
|
||||
await stopActiveRealtimeTalk();
|
||||
resolveFirstRelay(relay);
|
||||
await expect(firstStart).resolves.toBe(false);
|
||||
expect(unsubscribe).toHaveBeenCalledOnce();
|
||||
|
||||
await expect(controller.start({ sessionKey: 'agent:main:session-1' })).resolves.toBe(true);
|
||||
emit({ relaySessionId: 'relay-2', type: 'audio', audioBase64: 'AAD/fw==' });
|
||||
|
||||
await vi.waitFor(() => expect(audio.enqueueOutput).toHaveBeenCalledOnce());
|
||||
});
|
||||
|
||||
it('buffers a terminal Talk event received during relay startup and releases it before microphone activation', async () => {
|
||||
let resolveRelay!: (value: TalkRelaySession) => void;
|
||||
const pendingRelay = new Promise<TalkRelaySession>((resolve) => { resolveRelay = resolve; });
|
||||
const { audio, controller, emit, talk, unsubscribe } = setup();
|
||||
talk.startRelay.mockReturnValueOnce(pendingRelay);
|
||||
|
||||
const starting = controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
await vi.waitFor(() => expect(talk.startRelay).toHaveBeenCalledOnce());
|
||||
emit({ relaySessionId: 'relay-1', type: 'close', reason: 'cancelled' });
|
||||
resolveRelay(relay);
|
||||
|
||||
await expect(starting).resolves.toBe(false);
|
||||
expect(audio.start).not.toHaveBeenCalled();
|
||||
expect(talk.stopRelay).toHaveBeenCalledWith({ relaySessionId: 'relay-1' });
|
||||
expect(unsubscribe).toHaveBeenCalledOnce();
|
||||
expect(useRealtimeTalkStore.getState()).toMatchObject({ isActive: false, relaySessionId: null });
|
||||
});
|
||||
|
||||
it('cancels once for barge-in and acknowledges marks after queued playback completes', async () => {
|
||||
let finishPlayback!: () => void;
|
||||
const playback = new Promise<void>((resolve) => { finishPlayback = resolve; });
|
||||
const { audio, controller, emit, talk } = setup();
|
||||
audio.enqueueOutput.mockReturnValue(playback);
|
||||
await controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
|
||||
emit({ relaySessionId: 'relay-1', type: 'audio', audioBase64: 'AAD/fw==' });
|
||||
await vi.waitFor(() => expect(audio.enqueueOutput).toHaveBeenCalledOnce());
|
||||
audio.start.mock.calls[0]?.[0](new Float32Array([0.2]), 24_000);
|
||||
audio.start.mock.calls[0]?.[0](new Float32Array([0.3]), 24_000);
|
||||
expect(talk.cancelOutput).toHaveBeenCalledOnce();
|
||||
|
||||
emit({ relaySessionId: 'relay-1', type: 'mark', markName: 'end-turn' });
|
||||
expect(talk.acknowledgeMark).not.toHaveBeenCalled();
|
||||
finishPlayback();
|
||||
await vi.waitFor(() => expect(talk.acknowledgeMark).toHaveBeenCalledWith({
|
||||
relaySessionId: 'relay-1', markName: 'end-turn',
|
||||
}));
|
||||
});
|
||||
|
||||
it('drops queued output continuations after local barge-in', async () => {
|
||||
let finishFirst!: () => void;
|
||||
const firstPlayback = new Promise<void>((resolve) => { finishFirst = resolve; });
|
||||
const { audio, controller, emit, talk } = setup();
|
||||
audio.enqueueOutput.mockReturnValueOnce(firstPlayback).mockResolvedValue(undefined);
|
||||
await controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
|
||||
emit({ relaySessionId: 'relay-1', type: 'audio', audioBase64: 'AAD/fw==' });
|
||||
emit({ relaySessionId: 'relay-1', type: 'audio', audioBase64: 'AAD/fw==' });
|
||||
await vi.waitFor(() => expect(audio.enqueueOutput).toHaveBeenCalledTimes(2));
|
||||
audio.start.mock.calls[0]?.[0](new Float32Array([0.2]), 24_000);
|
||||
expect(talk.cancelOutput).toHaveBeenCalledOnce();
|
||||
finishFirst();
|
||||
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
expect(audio.enqueueOutput).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it('discards late audio after barge-in until the provider clears the interrupted turn', async () => {
|
||||
const { audio, controller, emit, talk } = setup();
|
||||
await controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
emit({ relaySessionId: 'relay-1', type: 'audio', audioBase64: 'AAD/fw==' });
|
||||
await vi.waitFor(() => expect(audio.enqueueOutput).toHaveBeenCalledOnce());
|
||||
audio.start.mock.calls[0]?.[0](new Float32Array([0.2]), 24_000);
|
||||
emit({ relaySessionId: 'relay-1', type: 'audio', audioBase64: 'AAD/fw==' });
|
||||
|
||||
expect(talk.cancelOutput).toHaveBeenCalledOnce();
|
||||
expect(audio.enqueueOutput).toHaveBeenCalledOnce();
|
||||
emit({ relaySessionId: 'relay-1', type: 'clear', reason: 'barge-in' });
|
||||
emit({ relaySessionId: 'relay-1', type: 'audio', audioBase64: 'AAD/fw==' });
|
||||
expect(audio.enqueueOutput).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it('enqueues playback eagerly and terminates Talk when the bounded audio transport rejects', async () => {
|
||||
const { audio, controller, emit, talk } = setup();
|
||||
const full = Promise.reject(new Error('Talk playback queue is full'));
|
||||
audio.enqueueOutput.mockReturnValueOnce(Promise.resolve()).mockReturnValueOnce(full);
|
||||
await controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
emit({ relaySessionId: 'relay-1', type: 'audio', audioBase64: 'AAD/fw==' });
|
||||
emit({ relaySessionId: 'relay-1', type: 'audio', audioBase64: 'AAD/fw==' });
|
||||
|
||||
expect(audio.enqueueOutput).toHaveBeenCalledTimes(2);
|
||||
await vi.waitFor(() => expect(useRealtimeTalkStore.getState().status).toBe('error'));
|
||||
expect(talk.stopRelay).toHaveBeenCalledWith({ relaySessionId: 'relay-1' });
|
||||
});
|
||||
|
||||
it('terminates Talk when an oversized valid audio event exceeds the decoded byte limit', async () => {
|
||||
const { controller, emit, talk } = setup();
|
||||
await controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
const oversized = 'AAAA'.repeat(Math.ceil((2 * 1024 * 1024 + 1) / 3));
|
||||
|
||||
emit({ relaySessionId: 'relay-1', type: 'audio', audioBase64: oversized });
|
||||
|
||||
await vi.waitFor(() => expect(useRealtimeTalkStore.getState().status).toBe('error'));
|
||||
expect(talk.stopRelay).toHaveBeenCalledWith({ relaySessionId: 'relay-1' });
|
||||
});
|
||||
|
||||
it('terminates Talk when serialized microphone append fails', async () => {
|
||||
const { audio, controller, talk } = setup();
|
||||
talk.appendAudio.mockRejectedValueOnce(new Error('append failed'));
|
||||
await controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
audio.start.mock.calls[0]?.[0](new Float32Array([0.2]), 24_000);
|
||||
|
||||
await vi.waitFor(() => expect(useRealtimeTalkStore.getState().status).toBe('error'));
|
||||
expect(talk.stopRelay).toHaveBeenCalledWith({ relaySessionId: 'relay-1' });
|
||||
});
|
||||
|
||||
it('fails safely for a pending ACP prompt, microphone denial, and unsupported audio', async () => {
|
||||
const pending = setup({ sending: true });
|
||||
await expect(pending.controller.start({ sessionKey: 'agent:main:session-1' })).resolves.toBe(false);
|
||||
expect(pending.talk.startRelay).not.toHaveBeenCalled();
|
||||
|
||||
const denied = setup({ startAudio: async () => { throw new Error('Permission denied'); } });
|
||||
await expect(denied.controller.start({ sessionKey: 'agent:main:session-1' })).resolves.toBe(false);
|
||||
expect(denied.talk.stopRelay).toHaveBeenCalledWith({ relaySessionId: 'relay-1' });
|
||||
|
||||
const unsupported = setup({ relay: { ...relay, audio: { ...relay.audio, inputEncoding: 'pcm16', outputEncoding: 'pcm16', inputSampleRateHz: 0 } } });
|
||||
await expect(unsupported.controller.start({ sessionKey: 'agent:main:session-1' })).resolves.toBe(false);
|
||||
expect(unsupported.talk.stopRelay).toHaveBeenCalledWith({ relaySessionId: 'relay-1' });
|
||||
});
|
||||
|
||||
it('requires the ACP-selected non-heartbeat session and never accepts the default fallback', async () => {
|
||||
const arbitrary = setup({ activeSessionKey: 'agent:main:selected' });
|
||||
await expect(arbitrary.controller.start({ sessionKey: 'agent:main:other' })).resolves.toBe(false);
|
||||
expect(arbitrary.talk.startRelay).not.toHaveBeenCalled();
|
||||
|
||||
const heartbeat = setup({
|
||||
selectedSession: { key: 'agent:main:session-1', lastMessagePreview: '[OpenClaw heartbeat poll]' },
|
||||
});
|
||||
await expect(heartbeat.controller.start({ sessionKey: 'agent:main:session-1' })).resolves.toBe(false);
|
||||
expect(heartbeat.talk.startRelay).not.toHaveBeenCalled();
|
||||
|
||||
const fallback = setup({ activeSessionKey: 'agent:main:main' });
|
||||
await expect(fallback.controller.start({ sessionKey: 'agent:main:main' })).resolves.toBe(false);
|
||||
expect(fallback.talk.startRelay).not.toHaveBeenCalled();
|
||||
|
||||
const unknown = setup({ selectedSession: null });
|
||||
await expect(unknown.controller.start({ sessionKey: 'agent:main:session-1' })).resolves.toBe(false);
|
||||
expect(unknown.talk.startRelay).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('maps an error close event to explicit error state without a separate error message', async () => {
|
||||
const { controller, emit } = setup();
|
||||
await controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
|
||||
emit({ relaySessionId: 'relay-1', type: 'close', reason: 'error' });
|
||||
await vi.waitFor(() => expect(useRealtimeTalkStore.getState().status).toBe('error'));
|
||||
expect(useRealtimeTalkStore.getState().error).toBeTruthy();
|
||||
});
|
||||
|
||||
it('starts Agent consults through Talk and reloads the same ACP session only after terminal completion', async () => {
|
||||
const { controller, emit, reload, talk } = setup();
|
||||
await controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
|
||||
emit({ relaySessionId: 'relay-1', type: 'toolCall', callId: 'call-1', name: 'openclaw_agent_consult', args: { prompt: 'review' } });
|
||||
await vi.waitFor(() => expect(talk.startAgentConsult).toHaveBeenCalledWith({
|
||||
relaySessionId: 'relay-1', sessionKey: 'agent:main:session-1', callId: 'call-1', args: { prompt: 'review' },
|
||||
}));
|
||||
expect(reload).not.toHaveBeenCalled();
|
||||
|
||||
emit({ relaySessionId: 'relay-1', type: 'toolResult', callId: 'call-1', final: true });
|
||||
await vi.waitFor(() => expect(reload).toHaveBeenCalledOnce());
|
||||
});
|
||||
|
||||
it('deduplicates tool calls and suppresses late results after cancellation or cleanup', async () => {
|
||||
let resolveConsult!: (value: { runId: string }) => void;
|
||||
const pendingConsult = new Promise<{ runId: string }>((resolve) => { resolveConsult = resolve; });
|
||||
const { controller, emit, reload, talk } = setup();
|
||||
talk.startAgentConsult.mockReturnValueOnce(pendingConsult);
|
||||
await controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
|
||||
const toolCall = { relaySessionId: 'relay-1', type: 'toolCall' as const, callId: 'call-1', name: 'openclaw_agent_consult', args: {} };
|
||||
emit(toolCall);
|
||||
emit(toolCall);
|
||||
expect(talk.startAgentConsult).toHaveBeenCalledOnce();
|
||||
emit({ relaySessionId: 'relay-1', type: 'toolCallCancelled', callId: 'call-1' });
|
||||
resolveConsult({ runId: 'run-1' });
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
expect(talk.submitToolResult).not.toHaveBeenCalled();
|
||||
expect(reload).not.toHaveBeenCalled();
|
||||
|
||||
emit({ relaySessionId: 'relay-1', type: 'toolCall', callId: 'call-2', name: 'openclaw_agent_consult', args: {} });
|
||||
await controller.stop();
|
||||
emit({ relaySessionId: 'relay-1', type: 'toolResult', callId: 'call-2', final: true });
|
||||
expect(reload).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('ignores unknown tool calls without starting an Agent consult or ACP history reload', async () => {
|
||||
const { controller, emit, reload, talk } = setup();
|
||||
await controller.start({ sessionKey: 'agent:main:session-1' });
|
||||
|
||||
emit({ relaySessionId: 'relay-1', type: 'toolCall', callId: 'call-unknown', name: 'untrusted_tool', args: {} });
|
||||
emit({ relaySessionId: 'relay-1', type: 'toolResult', callId: 'call-unknown', final: true });
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
|
||||
expect(talk.startAgentConsult).not.toHaveBeenCalled();
|
||||
expect(talk.submitToolResult).not.toHaveBeenCalled();
|
||||
expect(reload).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,40 @@
|
||||
import { beforeEach, describe, expect, it } from 'vitest';
|
||||
import { useRealtimeTalkStore } from '@/stores/realtime-talk';
|
||||
|
||||
describe('realtime Talk store', () => {
|
||||
beforeEach(() => {
|
||||
useRealtimeTalkStore.getState().reset();
|
||||
});
|
||||
|
||||
it('keeps direct partial and final transcripts only in ordered live memory', () => {
|
||||
const store = useRealtimeTalkStore.getState();
|
||||
store.begin('relay-1', 'agent:main:session-1');
|
||||
store.appendTranscript({ role: 'user', text: 'hel', final: false });
|
||||
store.appendTranscript({ role: 'user', text: 'hello', final: true });
|
||||
store.appendTranscript({ role: 'assistant', text: 'hi', final: true });
|
||||
|
||||
expect(useRealtimeTalkStore.getState().transcripts).toEqual([
|
||||
{ role: 'user', text: 'hello', final: true },
|
||||
{ role: 'assistant', text: 'hi', final: true },
|
||||
]);
|
||||
});
|
||||
|
||||
it('clears live transcript state on terminal, session reset, and app reset paths', () => {
|
||||
const store = useRealtimeTalkStore.getState();
|
||||
store.begin('relay-1', 'agent:main:session-1');
|
||||
store.appendTranscript({ role: 'assistant', text: 'temporary', final: true });
|
||||
store.finish('completed');
|
||||
|
||||
expect(useRealtimeTalkStore.getState()).toMatchObject({
|
||||
status: 'idle',
|
||||
relaySessionId: null,
|
||||
transcripts: [],
|
||||
});
|
||||
|
||||
store.begin('relay-2', 'agent:main:session-2');
|
||||
store.appendTranscript({ role: 'assistant', text: 'temporary again', final: true });
|
||||
store.reset();
|
||||
|
||||
expect(useRealtimeTalkStore.getState().transcripts).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,578 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import {
|
||||
closeWindowAfterActiveTalk,
|
||||
createTalkApi,
|
||||
createTalkRelayOwnership,
|
||||
forwardActiveTalkEvent,
|
||||
} from '@electron/services/talk-api';
|
||||
import {
|
||||
registerOpenClawConfigCoordinator,
|
||||
resetOpenClawConfigCoordinatorForTests,
|
||||
} from '@electron/gateway/config-delivery';
|
||||
import { afterEach } from 'vitest';
|
||||
|
||||
const TALK_RPC_TIMEOUT_MS = 8_000;
|
||||
|
||||
afterEach(() => {
|
||||
resetOpenClawConfigCoordinatorForTests();
|
||||
});
|
||||
|
||||
describe('talk API', () => {
|
||||
it('uses only the expected Gateway RPCs with normalized relay parameters', async () => {
|
||||
const gatewayManager = {
|
||||
rpc: vi.fn()
|
||||
.mockResolvedValueOnce({
|
||||
rawConfig: { apiKey: 'secret' },
|
||||
realtime: { ready: true, providers: [], apiKey: 'secret' },
|
||||
})
|
||||
.mockResolvedValueOnce({
|
||||
relaySessionId: 'relay-1',
|
||||
provider: 'openai',
|
||||
transport: 'gateway-relay',
|
||||
audio: {
|
||||
inputEncoding: 'pcm16',
|
||||
inputSampleRateHz: 24000,
|
||||
outputEncoding: 'pcm16',
|
||||
outputSampleRateHz: 24000,
|
||||
},
|
||||
})
|
||||
.mockResolvedValue({ ok: true }),
|
||||
};
|
||||
const talkApi = createTalkApi(gatewayManager as never);
|
||||
|
||||
await expect(talkApi.catalog()).resolves.toEqual({ realtime: { ready: true, providers: [] } });
|
||||
await expect(talkApi.startRelay({ sessionKey: ' agent:main:session-1 ' })).resolves.toMatchObject({
|
||||
relaySessionId: 'relay-1',
|
||||
});
|
||||
await talkApi.appendAudio({ relaySessionId: ' relay-1 ', audioBase64: 'AQI=', timestamp: 12.8 });
|
||||
await talkApi.cancelOutput({ relaySessionId: ' relay-1 ' });
|
||||
await talkApi.submitToolResult({ relaySessionId: ' relay-1 ', callId: ' call-1 ', result: { ok: true } });
|
||||
await talkApi.acknowledgeMark({ relaySessionId: ' relay-1 ', markName: ' mark-1 ' });
|
||||
await expect(talkApi.startAgentConsult({
|
||||
relaySessionId: ' relay-1 ',
|
||||
sessionKey: ' agent:main:session-1 ',
|
||||
callId: ' call-2 ',
|
||||
args: { prompt: 'hello' },
|
||||
})).resolves.toEqual({ ok: true });
|
||||
await talkApi.stopRelay({ relaySessionId: ' relay-1 ' });
|
||||
|
||||
expect(gatewayManager.rpc.mock.calls).toEqual([
|
||||
['talk.catalog', {}, TALK_RPC_TIMEOUT_MS],
|
||||
['talk.session.create', {
|
||||
sessionKey: 'agent:main:session-1',
|
||||
mode: 'realtime',
|
||||
transport: 'gateway-relay',
|
||||
brain: 'agent-consult',
|
||||
}, TALK_RPC_TIMEOUT_MS],
|
||||
['talk.session.appendAudio', { sessionId: 'relay-1', audioBase64: 'AQI=', timestamp: 12 }, TALK_RPC_TIMEOUT_MS],
|
||||
['talk.session.cancelOutput', { sessionId: 'relay-1' }, TALK_RPC_TIMEOUT_MS],
|
||||
['talk.session.submitToolResult', { sessionId: 'relay-1', callId: 'call-1', result: { ok: true } }, TALK_RPC_TIMEOUT_MS],
|
||||
['talk.session.acknowledgeMark', { sessionId: 'relay-1', markName: 'mark-1' }, TALK_RPC_TIMEOUT_MS],
|
||||
['talk.client.toolCall', {
|
||||
relaySessionId: 'relay-1',
|
||||
sessionKey: 'agent:main:session-1',
|
||||
callId: 'call-2',
|
||||
name: 'openclaw_agent_consult',
|
||||
args: { prompt: 'hello' },
|
||||
}, TALK_RPC_TIMEOUT_MS],
|
||||
['talk.session.close', { sessionId: 'relay-1' }, TALK_RPC_TIMEOUT_MS],
|
||||
]);
|
||||
});
|
||||
|
||||
it('rejects malformed relay requests before invoking the Gateway', async () => {
|
||||
const gatewayManager = { rpc: vi.fn() };
|
||||
const talkApi = createTalkApi(gatewayManager as never);
|
||||
|
||||
await expect(talkApi.startRelay({ sessionKey: ' ' })).rejects.toThrow('Invalid Talk session key');
|
||||
await expect(talkApi.appendAudio({ relaySessionId: ' ', audioBase64: 'AQI=' })).rejects.toThrow('Invalid Talk relay session id');
|
||||
await expect(talkApi.appendAudio({ relaySessionId: 'relay-1', audioBase64: 'not base64' })).rejects.toThrow('Invalid Talk PCM16 audio');
|
||||
await expect(talkApi.acknowledgeMark({ relaySessionId: 'relay-1', markName: ' ' })).rejects.toThrow('Invalid Talk mark name');
|
||||
await expect(talkApi.startAgentConsult({
|
||||
relaySessionId: 'relay-1',
|
||||
sessionKey: 'agent:main:session-1',
|
||||
callId: ' ',
|
||||
args: {},
|
||||
})).rejects.toThrow('Invalid Talk tool call id');
|
||||
|
||||
expect(gatewayManager.rpc).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects every stale relay operation and a mismatched Agent consult session before Gateway RPC', async () => {
|
||||
const gatewayManager = {
|
||||
rpc: vi.fn()
|
||||
.mockResolvedValueOnce({ relaySessionId: 'relay-active' })
|
||||
.mockResolvedValue({ ok: true }),
|
||||
};
|
||||
const talkApi = createTalkApi(gatewayManager as never);
|
||||
await talkApi.startRelay({ sessionKey: 'agent:main:session-active' });
|
||||
gatewayManager.rpc.mockClear();
|
||||
|
||||
await expect(talkApi.appendAudio({ relaySessionId: 'relay-stale', audioBase64: 'AQI=' }))
|
||||
.rejects.toThrow('Talk relay session is not active');
|
||||
await expect(talkApi.cancelOutput({ relaySessionId: 'relay-stale' }))
|
||||
.rejects.toThrow('Talk relay session is not active');
|
||||
await expect(talkApi.submitToolResult({ relaySessionId: 'relay-stale', callId: 'call-1', result: {} }))
|
||||
.rejects.toThrow('Talk relay session is not active');
|
||||
await expect(talkApi.acknowledgeMark({ relaySessionId: 'relay-stale', markName: 'mark-1' }))
|
||||
.rejects.toThrow('Talk relay session is not active');
|
||||
await expect(talkApi.stopRelay({ relaySessionId: 'relay-stale' }))
|
||||
.rejects.toThrow('Talk relay session is not active');
|
||||
await expect(talkApi.startAgentConsult({
|
||||
relaySessionId: 'relay-stale', sessionKey: 'agent:main:session-active', callId: 'call-1', args: {},
|
||||
})).rejects.toThrow('Talk relay session is not active');
|
||||
await expect(talkApi.startAgentConsult({
|
||||
relaySessionId: 'relay-active', sessionKey: 'agent:main:session-other', callId: 'call-1', args: {},
|
||||
})).rejects.toThrow('Talk relay session does not match the active session');
|
||||
|
||||
expect(gatewayManager.rpc).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('keeps the selected session key with Main relay ownership', () => {
|
||||
const ownership = createTalkRelayOwnership();
|
||||
|
||||
ownership.activate('relay-1', 'agent:main:session-1');
|
||||
|
||||
expect(ownership.getActive()).toEqual({
|
||||
relaySessionId: 'relay-1',
|
||||
sessionKey: 'agent:main:session-1',
|
||||
});
|
||||
expect(ownership.isActive('relay-1')).toBe(true);
|
||||
expect(ownership.isActive('relay-2')).toBe(false);
|
||||
});
|
||||
|
||||
it('updates only the catalog-selected realtime provider, model, and speaker voice', async () => {
|
||||
const gatewayManager = {
|
||||
getStatus: vi.fn(() => ({ state: 'running' })),
|
||||
rpc: vi.fn(async (method: string) => {
|
||||
if (method === 'talk.catalog') {
|
||||
return {
|
||||
realtime: {
|
||||
ready: true,
|
||||
providers: [{
|
||||
id: 'openai',
|
||||
label: 'OpenAI',
|
||||
configured: true,
|
||||
models: ['gpt-realtime'],
|
||||
voices: ['alloy'],
|
||||
}],
|
||||
},
|
||||
};
|
||||
}
|
||||
if (method === 'config.get') {
|
||||
return {
|
||||
hash: 'talk-hash',
|
||||
config: {
|
||||
channels: { telegram: { botToken: '__OPENCLAW_REDACTED__' } },
|
||||
talk: {
|
||||
realtime: {
|
||||
provider: 'old-provider',
|
||||
model: 'old-model',
|
||||
speakerVoice: 'old-voice',
|
||||
vad: { enabled: true },
|
||||
},
|
||||
otherTalkSetting: true,
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
if (method === 'config.set') return { ok: true };
|
||||
throw new Error(`Unexpected RPC: ${method}`);
|
||||
}),
|
||||
};
|
||||
registerOpenClawConfigCoordinator(gatewayManager as never);
|
||||
|
||||
await expect(createTalkApi(gatewayManager as never).updateRealtimeSettings({
|
||||
provider: 'openai',
|
||||
model: 'gpt-realtime',
|
||||
speakerVoice: 'alloy',
|
||||
})).resolves.toEqual({ ok: true });
|
||||
|
||||
expect(gatewayManager.rpc.mock.calls.map(([method]) => method)).toEqual([
|
||||
'config.get',
|
||||
'talk.catalog',
|
||||
'config.set',
|
||||
]);
|
||||
const configSet = gatewayManager.rpc.mock.calls[2][1] as { raw: string; baseHash: string };
|
||||
expect(configSet.baseHash).toBe('talk-hash');
|
||||
expect(JSON.parse(configSet.raw)).toEqual({
|
||||
channels: { telegram: { botToken: '__OPENCLAW_REDACTED__' } },
|
||||
talk: {
|
||||
realtime: {
|
||||
provider: 'openai',
|
||||
model: 'gpt-realtime',
|
||||
speakerVoice: 'alloy',
|
||||
vad: { enabled: true },
|
||||
},
|
||||
otherTalkSetting: true,
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it('uses a bundled catalog default model and preserves speaker voice when no voice choices are exposed', async () => {
|
||||
const gatewayManager = {
|
||||
getStatus: vi.fn(() => ({ state: 'running' })),
|
||||
rpc: vi.fn(async (method: string) => {
|
||||
if (method === 'config.get') {
|
||||
return {
|
||||
hash: 'bundled-hash',
|
||||
config: {
|
||||
talk: {
|
||||
realtime: {
|
||||
speakerVoice: 'existing-voice',
|
||||
vad: { enabled: true },
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
if (method === 'talk.catalog') {
|
||||
return {
|
||||
realtime: {
|
||||
providers: [{
|
||||
id: 'openai',
|
||||
label: 'OpenAI',
|
||||
configured: true,
|
||||
defaultModel: 'gpt-realtime',
|
||||
}],
|
||||
},
|
||||
};
|
||||
}
|
||||
if (method === 'config.set') return { ok: true };
|
||||
throw new Error(`Unexpected RPC: ${method}`);
|
||||
}),
|
||||
};
|
||||
registerOpenClawConfigCoordinator(gatewayManager as never);
|
||||
const talkApi = createTalkApi(gatewayManager as never);
|
||||
|
||||
await expect(talkApi.catalog()).resolves.toEqual({
|
||||
realtime: {
|
||||
providers: [{
|
||||
id: 'openai',
|
||||
label: 'OpenAI',
|
||||
configured: true,
|
||||
models: [],
|
||||
voices: [],
|
||||
transports: [],
|
||||
brains: [],
|
||||
defaultModel: 'gpt-realtime',
|
||||
}],
|
||||
},
|
||||
});
|
||||
await expect(talkApi.updateRealtimeSettings({ provider: 'openai', model: 'gpt-realtime' })).resolves.toEqual({ ok: true });
|
||||
|
||||
const configSet = gatewayManager.rpc.mock.calls.at(-1)?.[1] as { raw: string };
|
||||
expect(JSON.parse(configSet.raw)).toEqual({
|
||||
talk: {
|
||||
realtime: {
|
||||
provider: 'openai',
|
||||
model: 'gpt-realtime',
|
||||
speakerVoice: 'existing-voice',
|
||||
vad: { enabled: true },
|
||||
},
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it('revalidates the catalog inside each config mutation retry', async () => {
|
||||
let catalogCalls = 0;
|
||||
const gatewayManager = {
|
||||
getStatus: vi.fn(() => ({ state: 'running' })),
|
||||
rpc: vi.fn(async (method: string) => {
|
||||
if (method === 'config.get') {
|
||||
return { hash: `hash-${catalogCalls}`, config: { talk: { realtime: {} } } };
|
||||
}
|
||||
if (method === 'talk.catalog') {
|
||||
catalogCalls += 1;
|
||||
return {
|
||||
realtime: {
|
||||
providers: [{
|
||||
id: 'openai',
|
||||
label: 'OpenAI',
|
||||
configured: true,
|
||||
models: catalogCalls === 1 ? ['gpt-realtime'] : ['gpt-realtime-new'],
|
||||
voices: ['alloy'],
|
||||
}],
|
||||
},
|
||||
};
|
||||
}
|
||||
if (method === 'config.set') {
|
||||
throw new Error('config changed since last load; re-run config.get and retry');
|
||||
}
|
||||
throw new Error(`Unexpected RPC: ${method}`);
|
||||
}),
|
||||
};
|
||||
registerOpenClawConfigCoordinator(gatewayManager as never);
|
||||
|
||||
await expect(createTalkApi(gatewayManager as never).updateRealtimeSettings({
|
||||
provider: 'openai',
|
||||
model: 'gpt-realtime',
|
||||
speakerVoice: 'alloy',
|
||||
})).rejects.toThrow('Talk realtime model is not available for the provider');
|
||||
|
||||
expect(gatewayManager.rpc.mock.calls.map(([method]) => method)).toEqual([
|
||||
'config.get',
|
||||
'talk.catalog',
|
||||
'config.set',
|
||||
'config.get',
|
||||
'talk.catalog',
|
||||
]);
|
||||
});
|
||||
|
||||
it('fails closed when the Gateway catalog cannot be authoritatively fetched', async () => {
|
||||
const gatewayManager = {
|
||||
getStatus: vi.fn(() => ({ state: 'stopped' })),
|
||||
rpc: vi.fn(),
|
||||
};
|
||||
|
||||
await expect(createTalkApi(gatewayManager as never).updateRealtimeSettings({
|
||||
provider: 'openai',
|
||||
model: 'gpt-realtime',
|
||||
})).rejects.toThrow('Talk realtime catalog is unavailable; reconnect the Gateway and try again');
|
||||
expect(gatewayManager.rpc).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects blank and catalog-invalid realtime selections before mutating config', async () => {
|
||||
const gatewayManager = {
|
||||
getStatus: vi.fn(() => ({ state: 'running' })),
|
||||
rpc: vi.fn(async (method: string) => {
|
||||
if (method === 'config.get') {
|
||||
return { hash: 'invalid-selection-hash', config: { talk: { realtime: {} } } };
|
||||
}
|
||||
if (method === 'talk.catalog') {
|
||||
return {
|
||||
realtime: {
|
||||
providers: [{
|
||||
id: 'openai',
|
||||
label: 'OpenAI',
|
||||
configured: true,
|
||||
models: ['gpt-realtime'],
|
||||
voices: ['alloy'],
|
||||
}, {
|
||||
id: 'unconfigured',
|
||||
label: 'Unavailable',
|
||||
configured: false,
|
||||
models: ['model'],
|
||||
voices: ['voice'],
|
||||
}],
|
||||
},
|
||||
};
|
||||
}
|
||||
throw new Error(`Unexpected RPC: ${method}`);
|
||||
}),
|
||||
};
|
||||
registerOpenClawConfigCoordinator(gatewayManager as never);
|
||||
const talkApi = createTalkApi(gatewayManager as never);
|
||||
|
||||
await expect(talkApi.updateRealtimeSettings({
|
||||
provider: ' ', model: 'gpt-realtime', speakerVoice: 'alloy',
|
||||
})).rejects.toThrow('Invalid Talk realtime provider');
|
||||
await expect(talkApi.updateRealtimeSettings({
|
||||
provider: 'openai', model: 'other-model', speakerVoice: 'alloy',
|
||||
})).rejects.toThrow('Talk realtime model is not available for the provider');
|
||||
await expect(talkApi.updateRealtimeSettings({
|
||||
provider: 'openai', model: 'gpt-realtime',
|
||||
})).rejects.toThrow('Talk realtime speaker voice is not available for the provider');
|
||||
await expect(talkApi.updateRealtimeSettings({
|
||||
provider: 'unconfigured', model: 'model', speakerVoice: 'voice',
|
||||
})).rejects.toThrow('Talk realtime provider is not available');
|
||||
expect(gatewayManager.rpc.mock.calls.map(([method]) => method)).toEqual([
|
||||
'config.get',
|
||||
'talk.catalog',
|
||||
'config.get',
|
||||
'talk.catalog',
|
||||
'config.get',
|
||||
'talk.catalog',
|
||||
]);
|
||||
});
|
||||
|
||||
it('does not forward late events from stopped or replaced relays', async () => {
|
||||
const gatewayManager = {
|
||||
rpc: vi.fn()
|
||||
.mockResolvedValueOnce({ relaySessionId: 'relay-old' })
|
||||
.mockResolvedValueOnce({ ok: true })
|
||||
.mockResolvedValueOnce({ relaySessionId: 'relay-old' })
|
||||
.mockResolvedValueOnce({ ok: true })
|
||||
.mockResolvedValueOnce({ relaySessionId: 'relay-new' }),
|
||||
};
|
||||
const ownership = createTalkRelayOwnership();
|
||||
const talkApi = createTalkApi(gatewayManager as never, ownership);
|
||||
const sendToRenderer = vi.fn();
|
||||
const oldEvent = {
|
||||
relaySessionId: 'relay-old',
|
||||
type: 'transcript' as const,
|
||||
role: 'assistant' as const,
|
||||
text: 'late',
|
||||
final: true,
|
||||
};
|
||||
const newEvent = { ...oldEvent, relaySessionId: 'relay-new' };
|
||||
|
||||
await talkApi.startRelay({ sessionKey: 'agent:main:session-1' });
|
||||
forwardActiveTalkEvent(ownership, oldEvent, sendToRenderer);
|
||||
await talkApi.stopRelay({ relaySessionId: 'relay-old' });
|
||||
forwardActiveTalkEvent(ownership, oldEvent, sendToRenderer);
|
||||
|
||||
await talkApi.startRelay({ sessionKey: 'agent:main:session-1' });
|
||||
await talkApi.startRelay({ sessionKey: 'agent:main:session-2' });
|
||||
forwardActiveTalkEvent(ownership, oldEvent, sendToRenderer);
|
||||
forwardActiveTalkEvent(ownership, newEvent, sendToRenderer);
|
||||
|
||||
expect(sendToRenderer).toHaveBeenCalledTimes(2);
|
||||
expect(sendToRenderer).toHaveBeenNthCalledWith(1, oldEvent);
|
||||
expect(sendToRenderer).toHaveBeenNthCalledWith(2, newEvent);
|
||||
});
|
||||
|
||||
it('keeps the latest relay active when concurrent starts are serialized', async () => {
|
||||
let resolveFirst!: (value: unknown) => void;
|
||||
const firstResponse = new Promise<unknown>((resolve) => {
|
||||
resolveFirst = resolve;
|
||||
});
|
||||
const gatewayManager = {
|
||||
rpc: vi.fn()
|
||||
.mockReturnValueOnce(firstResponse)
|
||||
.mockResolvedValueOnce({ ok: true })
|
||||
.mockResolvedValueOnce({ relaySessionId: 'relay-new' }),
|
||||
};
|
||||
const ownership = createTalkRelayOwnership();
|
||||
const talkApi = createTalkApi(gatewayManager as never, ownership);
|
||||
const sendToRenderer = vi.fn();
|
||||
const oldEvent = {
|
||||
relaySessionId: 'relay-old',
|
||||
type: 'transcript' as const,
|
||||
role: 'assistant' as const,
|
||||
text: 'late',
|
||||
final: true,
|
||||
};
|
||||
const newEvent = { ...oldEvent, relaySessionId: 'relay-new' };
|
||||
|
||||
const firstStart = talkApi.startRelay({ sessionKey: 'agent:main:session-1' });
|
||||
const secondStart = talkApi.startRelay({ sessionKey: 'agent:main:session-2' });
|
||||
resolveFirst({ relaySessionId: 'relay-old' });
|
||||
await firstStart;
|
||||
await secondStart;
|
||||
|
||||
forwardActiveTalkEvent(ownership, oldEvent, sendToRenderer);
|
||||
forwardActiveTalkEvent(ownership, newEvent, sendToRenderer);
|
||||
|
||||
expect(sendToRenderer).toHaveBeenCalledTimes(1);
|
||||
expect(sendToRenderer).toHaveBeenCalledWith(newEvent);
|
||||
expect(gatewayManager.rpc).toHaveBeenNthCalledWith(
|
||||
2,
|
||||
'talk.session.close',
|
||||
{ sessionId: 'relay-old' },
|
||||
TALK_RPC_TIMEOUT_MS,
|
||||
);
|
||||
});
|
||||
|
||||
it('serializes relay replacement until a pending close settles and keeps old-event filtering intact', async () => {
|
||||
let resolveClose!: (value: unknown) => void;
|
||||
const close = new Promise<unknown>((resolve) => { resolveClose = resolve; });
|
||||
const gatewayManager = {
|
||||
rpc: vi.fn()
|
||||
.mockResolvedValueOnce({ relaySessionId: 'relay-old' })
|
||||
.mockReturnValueOnce(close)
|
||||
.mockResolvedValueOnce({ relaySessionId: 'relay-new' }),
|
||||
};
|
||||
const ownership = createTalkRelayOwnership();
|
||||
const talkApi = createTalkApi(gatewayManager as never, ownership);
|
||||
const sendToRenderer = vi.fn();
|
||||
const oldEvent = {
|
||||
relaySessionId: 'relay-old', type: 'transcript' as const, role: 'assistant' as const, text: 'old', final: true,
|
||||
};
|
||||
|
||||
await talkApi.startRelay({ sessionKey: 'agent:main:session-1' });
|
||||
const stopping = talkApi.stopRelay({ relaySessionId: 'relay-old' });
|
||||
const replacing = talkApi.startRelay({ sessionKey: 'agent:main:session-2' });
|
||||
|
||||
await vi.waitFor(() => expect(gatewayManager.rpc).toHaveBeenCalledTimes(2));
|
||||
forwardActiveTalkEvent(ownership, oldEvent, sendToRenderer);
|
||||
expect(sendToRenderer).toHaveBeenCalledWith(oldEvent);
|
||||
|
||||
resolveClose({ ok: true });
|
||||
await stopping;
|
||||
await expect(replacing).resolves.toMatchObject({ relaySessionId: 'relay-new' });
|
||||
forwardActiveTalkEvent(ownership, oldEvent, sendToRenderer);
|
||||
|
||||
expect(sendToRenderer).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('releases relay ownership after a failed close so a replacement cannot deadlock', async () => {
|
||||
const gatewayManager = {
|
||||
rpc: vi.fn()
|
||||
.mockResolvedValueOnce({ relaySessionId: 'relay-old' })
|
||||
.mockRejectedValueOnce(new Error('close timed out'))
|
||||
.mockResolvedValueOnce({ relaySessionId: 'relay-new' }),
|
||||
};
|
||||
const talkApi = createTalkApi(gatewayManager as never);
|
||||
|
||||
await talkApi.startRelay({ sessionKey: 'agent:main:session-1' });
|
||||
await expect(talkApi.stopRelay({ relaySessionId: 'relay-old' })).rejects.toThrow('close timed out');
|
||||
await expect(talkApi.startRelay({ sessionKey: 'agent:main:session-2' })).resolves.toMatchObject({ relaySessionId: 'relay-new' });
|
||||
});
|
||||
|
||||
it('tears down a relay created after window shutdown began without activating it, then permits a later start', async () => {
|
||||
let resolveCreate!: (value: unknown) => void;
|
||||
const pendingCreate = new Promise<unknown>((resolve) => { resolveCreate = resolve; });
|
||||
const gatewayManager = {
|
||||
rpc: vi.fn()
|
||||
.mockReturnValueOnce(pendingCreate)
|
||||
.mockResolvedValueOnce({ ok: true })
|
||||
.mockResolvedValueOnce({ relaySessionId: 'relay-reopened' }),
|
||||
};
|
||||
const ownership = createTalkRelayOwnership();
|
||||
const talkApi = createTalkApi(gatewayManager as never, ownership);
|
||||
const hide = vi.fn();
|
||||
const starting = talkApi.startRelay({ sessionKey: 'agent:main:session-closing' });
|
||||
|
||||
await vi.waitFor(() => expect(gatewayManager.rpc).toHaveBeenCalledWith(
|
||||
'talk.session.create',
|
||||
expect.any(Object),
|
||||
TALK_RPC_TIMEOUT_MS,
|
||||
));
|
||||
|
||||
closeWindowAfterActiveTalk({
|
||||
ownership,
|
||||
gatewayManager: gatewayManager as never,
|
||||
sendTalkEvent: vi.fn(),
|
||||
hide,
|
||||
});
|
||||
expect(hide).toHaveBeenCalledOnce();
|
||||
|
||||
resolveCreate({ relaySessionId: 'relay-late' });
|
||||
await expect(starting).rejects.toThrow('Talk relay startup was cancelled');
|
||||
expect(ownership.getActive()).toBeUndefined();
|
||||
expect(gatewayManager.rpc).toHaveBeenNthCalledWith(
|
||||
2,
|
||||
'talk.session.close',
|
||||
{ sessionId: 'relay-late' },
|
||||
TALK_RPC_TIMEOUT_MS,
|
||||
);
|
||||
await expect(talkApi.startRelay({ sessionKey: 'agent:main:session-reopened' })).resolves.toMatchObject({
|
||||
relaySessionId: 'relay-reopened',
|
||||
});
|
||||
});
|
||||
|
||||
it('closes a newly owned relay and sends a terminal event before the renderer adopts it', async () => {
|
||||
let resolveClose!: (value: unknown) => void;
|
||||
const pendingClose = new Promise<unknown>((resolve) => { resolveClose = resolve; });
|
||||
const gatewayManager = {
|
||||
rpc: vi.fn()
|
||||
.mockResolvedValueOnce({ relaySessionId: 'relay-created' })
|
||||
.mockReturnValueOnce(pendingClose),
|
||||
};
|
||||
const ownership = createTalkRelayOwnership();
|
||||
const talkApi = createTalkApi(gatewayManager as never, ownership);
|
||||
const sendTalkEvent = vi.fn();
|
||||
const hide = vi.fn();
|
||||
const starting = talkApi.startRelay({ sessionKey: 'agent:main:session-1' });
|
||||
|
||||
await vi.waitFor(() => expect(ownership.getActive()).toMatchObject({ relaySessionId: 'relay-created' }));
|
||||
closeWindowAfterActiveTalk({ ownership, gatewayManager: gatewayManager as never, sendTalkEvent, hide });
|
||||
|
||||
expect(sendTalkEvent).toHaveBeenCalledWith({ relaySessionId: 'relay-created', type: 'close', reason: 'cancelled' });
|
||||
expect(ownership.getActive()).toBeUndefined();
|
||||
await expect(starting).resolves.toMatchObject({ relaySessionId: 'relay-created' });
|
||||
expect(hide).not.toHaveBeenCalled();
|
||||
resolveClose({ ok: true });
|
||||
await vi.waitFor(() => expect(hide).toHaveBeenCalledOnce());
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,97 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import {
|
||||
decodePcm16Base64,
|
||||
float32ToPcm16,
|
||||
MAX_TALK_PLAYBACK_DECODED_BYTES,
|
||||
MAX_TALK_PLAYBACK_DURATION_SECONDS,
|
||||
Pcm16PlaybackQueue,
|
||||
pcm16ToBase64,
|
||||
pcm16ToFloat32,
|
||||
StreamingResampler,
|
||||
} from '@/lib/talk/audio';
|
||||
|
||||
describe('Talk audio codec', () => {
|
||||
it('converts Float32 microphone samples to clipped little-endian PCM16', () => {
|
||||
expect([...float32ToPcm16(new Float32Array([-1, -0.5, 0, 0.5, 1, 2]))])
|
||||
.toEqual([-32768, -16384, 0, 16383, 32767, 32767]);
|
||||
});
|
||||
|
||||
it('round-trips PCM16 bytes through base64', () => {
|
||||
const pcm = new Int16Array([-32768, 0, 32767]);
|
||||
|
||||
expect(pcm16ToBase64(pcm)).toBe('AIAAAP9/');
|
||||
});
|
||||
|
||||
it('rejects malformed and odd-length PCM16 base64 payloads', () => {
|
||||
expect(() => decodePcm16Base64('not base64')).toThrow('Invalid PCM16 base64 audio');
|
||||
expect(() => decodePcm16Base64('AQ==')).toThrow('Invalid PCM16 byte length');
|
||||
});
|
||||
|
||||
it('rejects oversized valid base64 before invoking atob or allocating PCM', () => {
|
||||
const atob = vi.fn();
|
||||
vi.stubGlobal('atob', atob);
|
||||
const oversized = 'AAAA'.repeat(Math.ceil((MAX_TALK_PLAYBACK_DECODED_BYTES + 1) / 3));
|
||||
|
||||
try {
|
||||
expect(() => decodePcm16Base64(oversized)).toThrow('Talk PCM chunk exceeds decoded size limit');
|
||||
expect(atob).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
vi.unstubAllGlobals();
|
||||
}
|
||||
});
|
||||
|
||||
it('decodes little-endian PCM16 output for playback', () => {
|
||||
const decoded = pcm16ToFloat32(decodePcm16Base64('AIAAAP9/'));
|
||||
|
||||
expect([...decoded]).toEqual([-1, 0, 32767 / 32768]);
|
||||
});
|
||||
|
||||
it('preserves resampling phase across microphone frames at the negotiated rate', () => {
|
||||
const resampler = new StreamingResampler(48_000, 24_000);
|
||||
const output = Array.from({ length: 100 }, (_, chunk) => (
|
||||
resampler.process(new Float32Array(480).fill(chunk / 100))
|
||||
)).flatMap((samples) => [...samples]);
|
||||
|
||||
expect(output).toHaveLength(24_000);
|
||||
expect(output[0]).toBe(0);
|
||||
expect(output.at(-1)).toBeCloseTo(0.99, 4);
|
||||
});
|
||||
|
||||
it('matches continuous 48kHz-to-16kHz conversion across arbitrary 128-sample frames', () => {
|
||||
const source = Float32Array.from({ length: 12_800 }, (_, index) => Math.sin(index / 19));
|
||||
const continuous = new StreamingResampler(48_000, 16_000).process(source);
|
||||
const chunkedResampler = new StreamingResampler(48_000, 16_000);
|
||||
const chunked = Array.from({ length: 100 }, (_, index) => (
|
||||
chunkedResampler.process(source.slice(index * 128, (index + 1) * 128))
|
||||
)).flatMap((samples) => [...samples]);
|
||||
|
||||
expect(chunked).toEqual([...continuous]);
|
||||
});
|
||||
|
||||
it('rejects oversized PCM before allocating playback buffers and releases duration budget on clear', async () => {
|
||||
const sources: Array<{ onended: (() => void) | null; stop: ReturnType<typeof vi.fn> }> = [];
|
||||
const createBuffer = vi.fn(() => ({ duration: 20, copyToChannel: vi.fn() }));
|
||||
const context = {
|
||||
currentTime: 0,
|
||||
destination: {},
|
||||
createBuffer,
|
||||
createBufferSource: vi.fn(() => {
|
||||
const source = { onended: null as (() => void) | null, stop: vi.fn(), connect: vi.fn(), start: vi.fn(), buffer: null };
|
||||
sources.push(source);
|
||||
return source;
|
||||
}),
|
||||
} as unknown as AudioContext;
|
||||
const queue = new Pcm16PlaybackQueue(context);
|
||||
const oversized = new Int16Array((Math.max(MAX_TALK_PLAYBACK_DECODED_BYTES ?? 0, 2_000_000) / 2) + 1);
|
||||
const twentySeconds = new Int16Array(24_000 * 20);
|
||||
|
||||
await expect(queue.enqueue(oversized, 24_000)).rejects.toThrow('Talk PCM chunk exceeds decoded size limit');
|
||||
expect(createBuffer).not.toHaveBeenCalled();
|
||||
void queue.enqueue(twentySeconds, 24_000);
|
||||
await expect(queue.enqueue(twentySeconds, 24_000)).rejects.toThrow('Talk playback duration queue is full');
|
||||
queue.clear();
|
||||
void queue.enqueue(twentySeconds, 24_000);
|
||||
expect(MAX_TALK_PLAYBACK_DURATION_SECONDS).toBeGreaterThanOrEqual(20);
|
||||
expect(sources).toHaveLength(2);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,109 @@
|
||||
import { fireEvent, render, screen, waitFor } from '@testing-library/react';
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
const { catalogMock, updateRealtimeSettingsMock } = vi.hoisted(() => ({
|
||||
catalogMock: vi.fn(),
|
||||
updateRealtimeSettingsMock: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('@/lib/host-api', () => ({
|
||||
hostApi: {
|
||||
talk: {
|
||||
catalog: (...args: unknown[]) => catalogMock(...args),
|
||||
updateRealtimeSettings: (...args: unknown[]) => updateRealtimeSettingsMock(...args),
|
||||
},
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('react-i18next', () => ({
|
||||
useTranslation: () => ({ t: (key: string) => key }),
|
||||
}));
|
||||
|
||||
import { TalkSettings } from '@/components/settings/TalkSettings';
|
||||
|
||||
const catalog = {
|
||||
realtime: {
|
||||
ready: false,
|
||||
reason: 'Configure a realtime provider',
|
||||
activeProvider: 'openai',
|
||||
providers: [{
|
||||
id: 'openai',
|
||||
label: 'OpenAI',
|
||||
configured: true,
|
||||
models: ['gpt-realtime', 'gpt-realtime-mini'],
|
||||
voices: ['alloy', 'verse'],
|
||||
}, {
|
||||
id: 'unavailable',
|
||||
label: 'Unavailable',
|
||||
configured: false,
|
||||
models: ['private-model'],
|
||||
voices: ['private-voice'],
|
||||
}],
|
||||
},
|
||||
};
|
||||
|
||||
describe('TalkSettings', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
catalogMock.mockResolvedValue(catalog);
|
||||
updateRealtimeSettingsMock.mockResolvedValue({ ok: true });
|
||||
});
|
||||
|
||||
it('uses only configured catalog options and refreshes after saving a selection', async () => {
|
||||
render(<TalkSettings />);
|
||||
|
||||
await waitFor(() => expect(screen.getByTestId('talk-settings-provider')).toHaveValue('openai'));
|
||||
expect(screen.getByTestId('talk-settings-provider')).toContainHTML('OpenAI');
|
||||
expect(screen.getByTestId('talk-settings-provider')).not.toContainHTML('Unavailable');
|
||||
expect(screen.getByTestId('talk-settings-model')).toContainHTML('gpt-realtime-mini');
|
||||
expect(screen.getByTestId('talk-settings-voice')).toContainHTML('verse');
|
||||
expect(screen.getByTestId('talk-settings-unavailable-reason')).toHaveTextContent('Configure a realtime provider');
|
||||
expect(screen.queryByText(/api key|transport|vad/i)).not.toBeInTheDocument();
|
||||
|
||||
fireEvent.change(screen.getByTestId('talk-settings-model'), { target: { value: 'gpt-realtime-mini' } });
|
||||
fireEvent.change(screen.getByTestId('talk-settings-voice'), { target: { value: 'verse' } });
|
||||
fireEvent.click(screen.getByTestId('talk-settings-save'));
|
||||
|
||||
await waitFor(() => expect(updateRealtimeSettingsMock).toHaveBeenCalledWith({
|
||||
provider: 'openai',
|
||||
model: 'gpt-realtime-mini',
|
||||
speakerVoice: 'verse',
|
||||
}));
|
||||
await waitFor(() => expect(catalogMock).toHaveBeenCalledTimes(2));
|
||||
});
|
||||
|
||||
it('uses defaultModel without rendering an empty voice selector when the catalog has no voice choices', async () => {
|
||||
catalogMock.mockResolvedValue({
|
||||
realtime: {
|
||||
ready: true,
|
||||
activeProvider: 'bundled',
|
||||
providers: [{
|
||||
id: 'bundled',
|
||||
label: 'Bundled provider',
|
||||
configured: true,
|
||||
defaultModel: 'bundled-realtime',
|
||||
}],
|
||||
},
|
||||
});
|
||||
render(<TalkSettings />);
|
||||
|
||||
await waitFor(() => expect(screen.getByTestId('talk-settings-model')).toHaveValue('bundled-realtime'));
|
||||
expect(screen.queryByTestId('talk-settings-voice')).not.toBeInTheDocument();
|
||||
expect(screen.getByTestId('talk-settings-voice-unavailable')).toHaveTextContent('talk.voiceUnavailable');
|
||||
fireEvent.click(screen.getByTestId('talk-settings-save'));
|
||||
|
||||
await waitFor(() => expect(updateRealtimeSettingsMock).toHaveBeenCalledWith({
|
||||
provider: 'bundled',
|
||||
model: 'bundled-realtime',
|
||||
}));
|
||||
});
|
||||
|
||||
it('links developer guidance to the Developer settings anchor', async () => {
|
||||
render(<TalkSettings />);
|
||||
|
||||
await waitFor(() => expect(screen.getByTestId('talk-settings-developer-link')).toHaveAttribute(
|
||||
'href',
|
||||
'#/settings?section=developer',
|
||||
));
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,79 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { closeWindowAfterActiveTalk, createTalkRelayOwnership } from '@electron/services/talk-api';
|
||||
|
||||
describe('normal window close Talk cleanup', () => {
|
||||
it('requests typed renderer cleanup, closes the Main-owned relay, then hides', async () => {
|
||||
let resolveClose!: (value: { ok: true }) => void;
|
||||
const close = new Promise<{ ok: true }>((resolve) => { resolveClose = resolve; });
|
||||
const ownership = createTalkRelayOwnership();
|
||||
ownership.activate('relay-1', 'agent:main:session-1');
|
||||
const gatewayManager = { rpc: vi.fn().mockReturnValue(close) };
|
||||
const sendTalkEvent = vi.fn();
|
||||
const hide = vi.fn();
|
||||
|
||||
closeWindowAfterActiveTalk({ ownership, gatewayManager: gatewayManager as never, sendTalkEvent, hide });
|
||||
|
||||
expect(sendTalkEvent).toHaveBeenCalledWith({ relaySessionId: 'relay-1', type: 'close', reason: 'cancelled' });
|
||||
expect(gatewayManager.rpc).toHaveBeenCalledWith('talk.session.close', { sessionId: 'relay-1' }, 8_000);
|
||||
expect(hide).not.toHaveBeenCalled();
|
||||
resolveClose({ ok: true });
|
||||
await vi.waitFor(() => expect(hide).toHaveBeenCalledOnce());
|
||||
});
|
||||
|
||||
it('still hides deterministically after an active relay close failure', async () => {
|
||||
const ownership = createTalkRelayOwnership();
|
||||
ownership.activate('relay-1', 'agent:main:session-1');
|
||||
const logWarn = vi.fn();
|
||||
const hide = vi.fn();
|
||||
|
||||
closeWindowAfterActiveTalk({
|
||||
ownership,
|
||||
gatewayManager: { rpc: vi.fn().mockRejectedValue(new Error('Gateway unavailable')) } as never,
|
||||
sendTalkEvent: vi.fn(),
|
||||
hide,
|
||||
logWarn,
|
||||
});
|
||||
|
||||
await vi.waitFor(() => expect(hide).toHaveBeenCalledOnce());
|
||||
expect(logWarn).toHaveBeenCalledWith('Failed to close active Talk relay before hiding the window', expect.any(Error));
|
||||
});
|
||||
|
||||
it('still closes and hides when renderer terminal cleanup delivery fails', async () => {
|
||||
const ownership = createTalkRelayOwnership();
|
||||
ownership.activate('relay-1', 'agent:main:session-1');
|
||||
const gatewayManager = { rpc: vi.fn().mockResolvedValue({ ok: true }) };
|
||||
const hide = vi.fn();
|
||||
const logWarn = vi.fn();
|
||||
|
||||
closeWindowAfterActiveTalk({
|
||||
ownership,
|
||||
gatewayManager: gatewayManager as never,
|
||||
sendTalkEvent: () => { throw new Error('Renderer unavailable'); },
|
||||
hide,
|
||||
logWarn,
|
||||
});
|
||||
|
||||
await vi.waitFor(() => expect(hide).toHaveBeenCalledOnce());
|
||||
expect(gatewayManager.rpc).toHaveBeenCalledWith('talk.session.close', { sessionId: 'relay-1' }, 8_000);
|
||||
expect(logWarn).toHaveBeenCalledWith('Failed to request Talk renderer cleanup before hiding the window', expect.any(Error));
|
||||
});
|
||||
|
||||
it('hides after the bounded close deadline when the Gateway never settles', async () => {
|
||||
const ownership = createTalkRelayOwnership();
|
||||
ownership.activate('relay-1', 'agent:main:session-1');
|
||||
const hide = vi.fn();
|
||||
const logWarn = vi.fn();
|
||||
|
||||
closeWindowAfterActiveTalk({
|
||||
ownership,
|
||||
gatewayManager: { rpc: vi.fn().mockReturnValue(new Promise(() => {})) } as never,
|
||||
sendTalkEvent: vi.fn(),
|
||||
hide,
|
||||
logWarn,
|
||||
timeoutMs: 1,
|
||||
});
|
||||
|
||||
await vi.waitFor(() => expect(hide).toHaveBeenCalledOnce());
|
||||
expect(logWarn).toHaveBeenCalledWith('Timed out closing active Talk relay before hiding the window', expect.any(Error));
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user