mirror of
https://github.com/Purple-Horizons/openclaw-voice.git
synced 2026-08-14 00:58:01 +00:00
Initial commit - OpenClaw Voice
Self-hosted browser-based voice interface for AI assistants. Features: - FastAPI WebSocket server - Whisper STT (faster-whisper or openai-whisper) - Chatterbox TTS (MIT license, ElevenLabs quality) - OpenAI backend (pluggable) - Browser voice widget (push-to-talk) - Mock mode for testing without models Stack: - Python 3.10+ - FastAPI + Uvicorn - WebSocket for real-time audio - React/vanilla JS client Tested: Server runs, serves HTML, OpenAI backend connects. TODO: Install Whisper, Chatterbox for full functionality.
This commit is contained in:
gianni-dalerta
@@ -0,0 +1,20 @@
|
||||
# OpenClaw Voice Configuration
|
||||
|
||||
# OpenAI API Key (for AI backend)
|
||||
OPENAI_API_KEY=sk-your-key-here
|
||||
|
||||
# Server settings
|
||||
OPENCLAW_HOST=0.0.0.0
|
||||
OPENCLAW_PORT=8765
|
||||
|
||||
# STT settings
|
||||
OPENCLAW_STT_MODEL=base # tiny, base, small, medium, large-v3-turbo
|
||||
OPENCLAW_STT_DEVICE=auto # auto, cpu, cuda, mps
|
||||
|
||||
# TTS settings
|
||||
OPENCLAW_TTS_MODEL=chatterbox
|
||||
# OPENCLAW_TTS_VOICE=/path/to/voice-sample.wav # For voice cloning
|
||||
|
||||
# AI Backend
|
||||
OPENCLAW_BACKEND_TYPE=openai
|
||||
OPENCLAW_BACKEND_MODEL=gpt-4o-mini
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
.venv/
|
||||
venv/
|
||||
.env
|
||||
*.egg-info/
|
||||
dist/
|
||||
build/
|
||||
.pytest_cache/
|
||||
.ruff_cache/
|
||||
*.log
|
||||
.DS_Store
|
||||
@@ -0,0 +1,191 @@
|
||||
# OpenClaw Voice
|
||||
|
||||
**Open-source browser-based voice interface for AI assistants.**
|
||||
|
||||
Talk to your AI like you talk to Alexa — but self-hosted, private, and free from subscription fees.
|
||||
|
||||
|
||||
|
||||
|
||||
## Why?
|
||||
|
||||
Voice AI platforms like ElevenLabs Agents ($0.08-0.12/min) and Retell.ai ($0.13-0.31/min) are expensive. OpenClaw Voice runs entirely on your own hardware for ~$0.003/min at scale.
|
||||
|
||||
## Features
|
||||
|
||||
- 🎙️ **Browser voice widget** — Push-to-talk or hands-free VAD
|
||||
- 🔊 **Self-hosted STT** — Whisper Large V3 Turbo (runs on Mac/Linux/GPU)
|
||||
- 🗣️ **Self-hosted TTS** — Chatterbox (MIT license, ElevenLabs quality)
|
||||
- 🔌 **Pluggable backend** — Connect to any AI (OpenAI, Claude, Clawdbot, etc.)
|
||||
- 🌐 **WebRTC audio** — Low latency (<500ms end-to-end achievable)
|
||||
- 🏠 **Fully self-hosted** — Your data stays on your servers
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.10+
|
||||
- Node.js 18+ (for client dev)
|
||||
- CUDA GPU recommended (CPU works but slower)
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
# Clone the repo
|
||||
git clone https://github.com/Purple-Horizons/openclaw-voice.git
|
||||
cd openclaw-voice
|
||||
|
||||
# Install Python dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Download models (first run only)
|
||||
python scripts/download_models.py
|
||||
|
||||
# Start the voice server
|
||||
python -m src.server.main
|
||||
|
||||
# Open http://localhost:8765 in your browser
|
||||
```
|
||||
|
||||
### Docker (Recommended)
|
||||
|
||||
```bash
|
||||
docker compose up
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ WebRTC ┌─────────────────┐
|
||||
│ Browser │ ◄─────────────► │ Voice Gateway │
|
||||
│ (Voice Widget) │ Audio/Text │ (Python) │
|
||||
└─────────────────┘ └────────┬────────┘
|
||||
│
|
||||
┌────────────────────────┼────────────────────────┐
|
||||
│ │ │
|
||||
┌─────▼─────┐ ┌───────▼───────┐ ┌──────▼──────┐
|
||||
│ Whisper │ │ Your AI │ │ Chatterbox │
|
||||
│ (STT) │ │ Backend │ │ (TTS) │
|
||||
└───────────┘ └───────────────┘ └─────────────┘
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
stt:
|
||||
model: "whisper-large-v3-turbo"
|
||||
device: "cuda" # or "cpu", "mps" (Mac)
|
||||
|
||||
tts:
|
||||
model: "chatterbox"
|
||||
voice: "default" # or path to voice sample for cloning
|
||||
|
||||
backend:
|
||||
type: "openai" # or "clawdbot", "custom"
|
||||
url: "https://api.openai.com/v1"
|
||||
model: "gpt-4o"
|
||||
|
||||
server:
|
||||
host: "0.0.0.0"
|
||||
port: 8765
|
||||
ssl: false # Set true + provide certs for production
|
||||
```
|
||||
|
||||
## Supported Models
|
||||
|
||||
### Speech-to-Text (STT)
|
||||
| Model | Speed | Quality | VRAM |
|
||||
|-------|-------|---------|------|
|
||||
| Whisper Large V3 Turbo | 216x realtime | Best | ~6GB |
|
||||
| Distil-Whisper | 6x faster | Good | ~3GB |
|
||||
| Whisper.cpp (CPU) | Slower | Best | N/A |
|
||||
|
||||
### Text-to-Speech (TTS)
|
||||
| Model | Speed | Quality | Voice Cloning |
|
||||
|-------|-------|---------|---------------|
|
||||
| Chatterbox | ~1s | Excellent | 5-second samples |
|
||||
| Kokoro-82M | <0.3s | Very Good | No |
|
||||
| XTTS-v2 | ~1s | Excellent | 6-second samples |
|
||||
|
||||
## Browser Widget
|
||||
|
||||
Embed the voice widget in any webpage:
|
||||
|
||||
```html
|
||||
<script src="https://unpkg.com/@openclaw/voice-widget"></script>
|
||||
<openclaw-voice server="wss://your-server:8765"></openclaw-voice>
|
||||
```
|
||||
|
||||
Or use React:
|
||||
|
||||
```jsx
|
||||
import { VoiceWidget } from '@openclaw/voice-widget-react';
|
||||
|
||||
<VoiceWidget serverUrl="wss://your-server:8765" />
|
||||
```
|
||||
|
||||
## API
|
||||
|
||||
### WebSocket Protocol
|
||||
|
||||
Connect to `ws://localhost:8765/ws` and send/receive JSON messages:
|
||||
|
||||
```javascript
|
||||
// Start listening
|
||||
{ "type": "start_listening" }
|
||||
|
||||
// Audio data (base64 PCM)
|
||||
{ "type": "audio", "data": "base64..." }
|
||||
|
||||
// Stop listening
|
||||
{ "type": "stop_listening" }
|
||||
|
||||
// Receive transcription
|
||||
{ "type": "transcript", "text": "Hello world", "final": true }
|
||||
|
||||
// Receive AI response audio
|
||||
{ "type": "audio_response", "data": "base64...", "text": "Hi there!" }
|
||||
```
|
||||
|
||||
## Roadmap
|
||||
|
||||
- [x] Basic WebSocket voice gateway
|
||||
- [x] Whisper STT integration
|
||||
- [x] Chatterbox TTS integration
|
||||
- [ ] WebRTC for lower latency
|
||||
- [ ] Voice Activity Detection (VAD)
|
||||
- [ ] Streaming responses
|
||||
- [ ] Voice cloning UI
|
||||
- [ ] Browser widget npm package
|
||||
- [ ] React/Vue components
|
||||
- [ ] Docker GPU support
|
||||
- [ ] Kubernetes Helm chart
|
||||
|
||||
## Cost Comparison
|
||||
|
||||
| Platform | Cost/Minute |
|
||||
|----------|-------------|
|
||||
| ElevenLabs Conversational AI | $0.08-0.12 |
|
||||
| Retell.ai | $0.13-0.31 |
|
||||
| Vapi.ai | $0.05-0.15 |
|
||||
| **OpenClaw Voice (self-hosted)** | **~$0.003** |
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions welcome! See [CONTRIBUTING.md](CONTRIBUTING.md).
|
||||
|
||||
## License
|
||||
|
||||
MIT License — see [LICENSE](LICENSE).
|
||||
|
||||
## Credits
|
||||
|
||||
- [Whisper](https://github.com/openai/whisper) — OpenAI
|
||||
- [Chatterbox](https://github.com/resemble-ai/chatterbox) — Resemble AI
|
||||
- [Silero VAD](https://github.com/snakers4/silero-vad) — Silero
|
||||
- Built for [Clawdbot](https://github.com/clawdbot/clawdbot)
|
||||
|
||||
---
|
||||
|
||||
**Made with 🦀 by [Purple Horizons](https://purplehorizons.io)**
|
||||
@@ -0,0 +1,42 @@
|
||||
# Core
|
||||
fastapi>=0.109.0
|
||||
uvicorn[standard]>=0.27.0
|
||||
websockets>=12.0
|
||||
pydantic>=2.5.0
|
||||
pydantic-settings>=2.1.0
|
||||
python-multipart>=0.0.6
|
||||
|
||||
# Audio processing
|
||||
numpy>=1.26.0
|
||||
soundfile>=0.12.1
|
||||
librosa>=0.10.1
|
||||
webrtcvad>=2.0.10
|
||||
|
||||
# Speech-to-Text
|
||||
openai-whisper>=20231117
|
||||
faster-whisper>=1.0.0
|
||||
# whisper.cpp via ctypes (optional, for CPU)
|
||||
|
||||
# Text-to-Speech
|
||||
# chatterbox-tts # Install from source for now
|
||||
torch>=2.1.0
|
||||
torchaudio>=2.1.0
|
||||
transformers>=4.36.0
|
||||
|
||||
# Voice Activity Detection
|
||||
silero-vad>=4.0.0
|
||||
|
||||
# AI Backend
|
||||
openai>=1.6.0
|
||||
httpx>=0.26.0
|
||||
|
||||
# Utilities
|
||||
pyyaml>=6.0.1
|
||||
python-dotenv>=1.0.0
|
||||
loguru>=0.7.2
|
||||
|
||||
# Dev
|
||||
pytest>=7.4.0
|
||||
pytest-asyncio>=0.23.0
|
||||
black>=23.12.0
|
||||
ruff>=0.1.0
|
||||
@@ -0,0 +1,397 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>OpenClaw Voice</title>
|
||||
<style>
|
||||
* {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
background: linear-gradient(135deg, #1a1a2e 0%, #16213e 100%);
|
||||
color: #fff;
|
||||
min-height: 100vh;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
padding: 20px;
|
||||
}
|
||||
|
||||
.container {
|
||||
max-width: 600px;
|
||||
width: 100%;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
h1 {
|
||||
font-size: 2.5rem;
|
||||
margin-bottom: 10px;
|
||||
background: linear-gradient(90deg, #ff6b35, #f7c94b);
|
||||
-webkit-background-clip: text;
|
||||
-webkit-text-fill-color: transparent;
|
||||
}
|
||||
|
||||
.subtitle {
|
||||
color: #888;
|
||||
margin-bottom: 40px;
|
||||
}
|
||||
|
||||
.voice-button {
|
||||
width: 150px;
|
||||
height: 150px;
|
||||
border-radius: 50%;
|
||||
border: none;
|
||||
background: linear-gradient(135deg, #ff6b35 0%, #e55a2b 100%);
|
||||
color: white;
|
||||
font-size: 1.2rem;
|
||||
font-weight: 600;
|
||||
cursor: pointer;
|
||||
transition: all 0.3s ease;
|
||||
box-shadow: 0 10px 40px rgba(255, 107, 53, 0.3);
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
margin: 0 auto 30px;
|
||||
}
|
||||
|
||||
.voice-button:hover {
|
||||
transform: scale(1.05);
|
||||
box-shadow: 0 15px 50px rgba(255, 107, 53, 0.4);
|
||||
}
|
||||
|
||||
.voice-button:active,
|
||||
.voice-button.listening {
|
||||
transform: scale(0.95);
|
||||
background: linear-gradient(135deg, #e55a2b 0%, #c44a22 100%);
|
||||
}
|
||||
|
||||
.voice-button.listening {
|
||||
animation: pulse 1.5s infinite;
|
||||
}
|
||||
|
||||
@keyframes pulse {
|
||||
0%, 100% { box-shadow: 0 10px 40px rgba(255, 107, 53, 0.3); }
|
||||
50% { box-shadow: 0 10px 60px rgba(255, 107, 53, 0.6); }
|
||||
}
|
||||
|
||||
.status {
|
||||
font-size: 1.1rem;
|
||||
color: #888;
|
||||
margin-bottom: 20px;
|
||||
min-height: 30px;
|
||||
}
|
||||
|
||||
.status.active {
|
||||
color: #ff6b35;
|
||||
}
|
||||
|
||||
.transcript-box {
|
||||
background: rgba(255, 255, 255, 0.05);
|
||||
border-radius: 12px;
|
||||
padding: 20px;
|
||||
margin-bottom: 20px;
|
||||
text-align: left;
|
||||
min-height: 100px;
|
||||
}
|
||||
|
||||
.transcript-box h3 {
|
||||
font-size: 0.9rem;
|
||||
color: #666;
|
||||
margin-bottom: 10px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.1em;
|
||||
}
|
||||
|
||||
.transcript-box p {
|
||||
font-size: 1rem;
|
||||
line-height: 1.6;
|
||||
color: #ccc;
|
||||
}
|
||||
|
||||
.transcript-box p.user {
|
||||
color: #ff6b35;
|
||||
}
|
||||
|
||||
.transcript-box p.assistant {
|
||||
color: #4fc3f7;
|
||||
}
|
||||
|
||||
.controls {
|
||||
display: flex;
|
||||
gap: 10px;
|
||||
justify-content: center;
|
||||
margin-top: 20px;
|
||||
}
|
||||
|
||||
.controls button {
|
||||
padding: 10px 20px;
|
||||
border: 1px solid #444;
|
||||
background: transparent;
|
||||
color: #888;
|
||||
border-radius: 8px;
|
||||
cursor: pointer;
|
||||
font-size: 0.9rem;
|
||||
transition: all 0.2s;
|
||||
}
|
||||
|
||||
.controls button:hover {
|
||||
border-color: #ff6b35;
|
||||
color: #ff6b35;
|
||||
}
|
||||
|
||||
.error {
|
||||
color: #ff4444;
|
||||
margin-top: 20px;
|
||||
}
|
||||
|
||||
.instructions {
|
||||
margin-top: 40px;
|
||||
padding: 20px;
|
||||
background: rgba(255, 255, 255, 0.03);
|
||||
border-radius: 12px;
|
||||
color: #666;
|
||||
font-size: 0.9rem;
|
||||
}
|
||||
|
||||
.instructions code {
|
||||
background: rgba(255, 107, 53, 0.1);
|
||||
color: #ff6b35;
|
||||
padding: 2px 6px;
|
||||
border-radius: 4px;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="container">
|
||||
<h1>🎙️ OpenClaw Voice</h1>
|
||||
<p class="subtitle">Self-hosted voice AI interface</p>
|
||||
|
||||
<button class="voice-button" id="voiceBtn">
|
||||
Hold to Talk
|
||||
</button>
|
||||
|
||||
<p class="status" id="status">Ready</p>
|
||||
|
||||
<div class="transcript-box">
|
||||
<h3>Conversation</h3>
|
||||
<div id="transcript"></div>
|
||||
</div>
|
||||
|
||||
<div class="controls">
|
||||
<button id="clearBtn">Clear History</button>
|
||||
</div>
|
||||
|
||||
<p class="error" id="error"></p>
|
||||
|
||||
<div class="instructions">
|
||||
<p><strong>Instructions:</strong> Hold the button and speak. Release to send.</p>
|
||||
<p style="margin-top: 10px;">Press <code>Space</code> to toggle recording.</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const voiceBtn = document.getElementById('voiceBtn');
|
||||
const statusEl = document.getElementById('status');
|
||||
const transcriptEl = document.getElementById('transcript');
|
||||
const errorEl = document.getElementById('error');
|
||||
const clearBtn = document.getElementById('clearBtn');
|
||||
|
||||
let ws = null;
|
||||
let mediaRecorder = null;
|
||||
let audioContext = null;
|
||||
let isRecording = false;
|
||||
|
||||
// Connect to WebSocket
|
||||
function connect() {
|
||||
const protocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:';
|
||||
const wsUrl = `${protocol}//${window.location.host}/ws`;
|
||||
|
||||
ws = new WebSocket(wsUrl);
|
||||
|
||||
ws.onopen = () => {
|
||||
setStatus('Connected');
|
||||
errorEl.textContent = '';
|
||||
};
|
||||
|
||||
ws.onclose = () => {
|
||||
setStatus('Disconnected - reconnecting...');
|
||||
setTimeout(connect, 2000);
|
||||
};
|
||||
|
||||
ws.onerror = (e) => {
|
||||
errorEl.textContent = 'WebSocket error. Is the server running?';
|
||||
};
|
||||
|
||||
ws.onmessage = (event) => {
|
||||
const msg = JSON.parse(event.data);
|
||||
handleMessage(msg);
|
||||
};
|
||||
}
|
||||
|
||||
function handleMessage(msg) {
|
||||
switch (msg.type) {
|
||||
case 'listening_started':
|
||||
setStatus('Listening...', true);
|
||||
break;
|
||||
case 'listening_stopped':
|
||||
setStatus('Processing...');
|
||||
break;
|
||||
case 'transcript':
|
||||
addTranscript('You', msg.text, 'user');
|
||||
setStatus('Getting response...');
|
||||
break;
|
||||
case 'response_text':
|
||||
addTranscript('AI', msg.text, 'assistant');
|
||||
break;
|
||||
case 'audio_response':
|
||||
playAudio(msg.data, msg.sample_rate);
|
||||
setStatus('Ready');
|
||||
break;
|
||||
case 'pong':
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
function setStatus(text, active = false) {
|
||||
statusEl.textContent = text;
|
||||
statusEl.className = active ? 'status active' : 'status';
|
||||
}
|
||||
|
||||
function addTranscript(speaker, text, className) {
|
||||
const p = document.createElement('p');
|
||||
p.className = className;
|
||||
p.innerHTML = `<strong>${speaker}:</strong> ${text}`;
|
||||
transcriptEl.appendChild(p);
|
||||
transcriptEl.scrollTop = transcriptEl.scrollHeight;
|
||||
}
|
||||
|
||||
// Audio recording
|
||||
async function startRecording() {
|
||||
if (isRecording) return;
|
||||
|
||||
try {
|
||||
const stream = await navigator.mediaDevices.getUserMedia({
|
||||
audio: {
|
||||
sampleRate: 16000,
|
||||
channelCount: 1,
|
||||
echoCancellation: true,
|
||||
noiseSuppression: true,
|
||||
}
|
||||
});
|
||||
|
||||
audioContext = new AudioContext({ sampleRate: 16000 });
|
||||
const source = audioContext.createMediaStreamSource(stream);
|
||||
const processor = audioContext.createScriptProcessor(4096, 1, 1);
|
||||
|
||||
processor.onaudioprocess = (e) => {
|
||||
if (isRecording && ws && ws.readyState === WebSocket.OPEN) {
|
||||
const audioData = e.inputBuffer.getChannelData(0);
|
||||
const base64 = float32ToBase64(audioData);
|
||||
ws.send(JSON.stringify({ type: 'audio', data: base64 }));
|
||||
}
|
||||
};
|
||||
|
||||
source.connect(processor);
|
||||
processor.connect(audioContext.destination);
|
||||
|
||||
isRecording = true;
|
||||
voiceBtn.classList.add('listening');
|
||||
voiceBtn.textContent = 'Listening...';
|
||||
|
||||
ws.send(JSON.stringify({ type: 'start_listening' }));
|
||||
|
||||
} catch (err) {
|
||||
errorEl.textContent = `Microphone error: ${err.message}`;
|
||||
}
|
||||
}
|
||||
|
||||
function stopRecording() {
|
||||
if (!isRecording) return;
|
||||
|
||||
isRecording = false;
|
||||
voiceBtn.classList.remove('listening');
|
||||
voiceBtn.textContent = 'Hold to Talk';
|
||||
|
||||
if (audioContext) {
|
||||
audioContext.close();
|
||||
audioContext = null;
|
||||
}
|
||||
|
||||
if (ws && ws.readyState === WebSocket.OPEN) {
|
||||
ws.send(JSON.stringify({ type: 'stop_listening' }));
|
||||
}
|
||||
}
|
||||
|
||||
// Audio playback
|
||||
function playAudio(base64Data, sampleRate) {
|
||||
const audioCtx = new AudioContext({ sampleRate: sampleRate });
|
||||
const audioData = base64ToFloat32(base64Data);
|
||||
const buffer = audioCtx.createBuffer(1, audioData.length, sampleRate);
|
||||
buffer.getChannelData(0).set(audioData);
|
||||
|
||||
const source = audioCtx.createBufferSource();
|
||||
source.buffer = buffer;
|
||||
source.connect(audioCtx.destination);
|
||||
source.start();
|
||||
}
|
||||
|
||||
// Utilities
|
||||
function float32ToBase64(float32Array) {
|
||||
const bytes = new Uint8Array(float32Array.buffer);
|
||||
let binary = '';
|
||||
for (let i = 0; i < bytes.length; i++) {
|
||||
binary += String.fromCharCode(bytes[i]);
|
||||
}
|
||||
return btoa(binary);
|
||||
}
|
||||
|
||||
function base64ToFloat32(base64) {
|
||||
const binary = atob(base64);
|
||||
const bytes = new Uint8Array(binary.length);
|
||||
for (let i = 0; i < binary.length; i++) {
|
||||
bytes[i] = binary.charCodeAt(i);
|
||||
}
|
||||
return new Float32Array(bytes.buffer);
|
||||
}
|
||||
|
||||
// Event listeners
|
||||
voiceBtn.addEventListener('mousedown', startRecording);
|
||||
voiceBtn.addEventListener('mouseup', stopRecording);
|
||||
voiceBtn.addEventListener('mouseleave', stopRecording);
|
||||
voiceBtn.addEventListener('touchstart', (e) => { e.preventDefault(); startRecording(); });
|
||||
voiceBtn.addEventListener('touchend', stopRecording);
|
||||
|
||||
// Spacebar to toggle
|
||||
document.addEventListener('keydown', (e) => {
|
||||
if (e.code === 'Space' && !e.repeat) {
|
||||
e.preventDefault();
|
||||
if (isRecording) {
|
||||
stopRecording();
|
||||
} else {
|
||||
startRecording();
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
clearBtn.addEventListener('click', () => {
|
||||
transcriptEl.innerHTML = '';
|
||||
});
|
||||
|
||||
// Connect on load
|
||||
connect();
|
||||
|
||||
// Keep connection alive
|
||||
setInterval(() => {
|
||||
if (ws && ws.readyState === WebSocket.OPEN) {
|
||||
ws.send(JSON.stringify({ type: 'ping' }));
|
||||
}
|
||||
}, 30000);
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1 @@
|
||||
# OpenClaw Voice Server
|
||||
@@ -0,0 +1,104 @@
|
||||
"""
|
||||
AI Backend module - connects to OpenAI, Clawdbot, or custom backends.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from typing import Optional, List, Dict
|
||||
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class AIBackend:
|
||||
"""AI backend for processing user messages."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
backend_type: str = "openai",
|
||||
url: str = "https://api.openai.com/v1",
|
||||
model: str = "gpt-4o-mini",
|
||||
api_key: Optional[str] = None,
|
||||
system_prompt: Optional[str] = None,
|
||||
):
|
||||
self.backend_type = backend_type
|
||||
self.url = url
|
||||
self.model = model
|
||||
self.api_key = api_key
|
||||
self.system_prompt = system_prompt or (
|
||||
"You are a helpful voice assistant. Keep responses concise and conversational. "
|
||||
"Aim for 1-2 sentences unless more detail is needed."
|
||||
)
|
||||
self.conversation_history: List[Dict] = []
|
||||
self._client = None
|
||||
self._setup_client()
|
||||
|
||||
def _setup_client(self):
|
||||
"""Set up the API client."""
|
||||
if self.backend_type == "openai":
|
||||
try:
|
||||
from openai import AsyncOpenAI
|
||||
self._client = AsyncOpenAI(
|
||||
api_key=self.api_key,
|
||||
base_url=self.url if self.url != "https://api.openai.com/v1" else None,
|
||||
)
|
||||
logger.info(f"✅ OpenAI client ready (model: {self.model})")
|
||||
except ImportError:
|
||||
logger.error("openai package not installed")
|
||||
elif self.backend_type == "clawdbot":
|
||||
# TODO: Implement Clawdbot gateway connection
|
||||
logger.info("Clawdbot backend (not yet implemented)")
|
||||
else:
|
||||
logger.warning(f"Unknown backend type: {self.backend_type}")
|
||||
|
||||
async def chat(self, user_message: str) -> str:
|
||||
"""
|
||||
Send a message and get a response.
|
||||
|
||||
Args:
|
||||
user_message: The user's transcribed speech
|
||||
|
||||
Returns:
|
||||
AI response text
|
||||
"""
|
||||
if self.backend_type == "openai" and self._client:
|
||||
return await self._chat_openai(user_message)
|
||||
else:
|
||||
# Fallback echo response
|
||||
return f"I heard you say: {user_message}"
|
||||
|
||||
async def _chat_openai(self, user_message: str) -> str:
|
||||
"""Chat via OpenAI API."""
|
||||
# Add user message to history
|
||||
self.conversation_history.append({
|
||||
"role": "user",
|
||||
"content": user_message,
|
||||
})
|
||||
|
||||
# Build messages
|
||||
messages = [{"role": "system", "content": self.system_prompt}]
|
||||
messages.extend(self.conversation_history[-10:]) # Last 10 turns
|
||||
|
||||
try:
|
||||
response = await self._client.chat.completions.create(
|
||||
model=self.model,
|
||||
messages=messages,
|
||||
max_tokens=150, # Keep responses short for voice
|
||||
temperature=0.7,
|
||||
)
|
||||
|
||||
assistant_message = response.choices[0].message.content
|
||||
|
||||
# Add to history
|
||||
self.conversation_history.append({
|
||||
"role": "assistant",
|
||||
"content": assistant_message,
|
||||
})
|
||||
|
||||
return assistant_message
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"OpenAI API error: {e}")
|
||||
return "Sorry, I had trouble processing that. Could you try again?"
|
||||
|
||||
def clear_history(self):
|
||||
"""Clear conversation history."""
|
||||
self.conversation_history = []
|
||||
@@ -0,0 +1,208 @@
|
||||
"""
|
||||
OpenClaw Voice Server
|
||||
|
||||
WebSocket server that handles:
|
||||
- Audio input from browser
|
||||
- Speech-to-Text via Whisper
|
||||
- AI backend communication
|
||||
- Text-to-Speech via Chatterbox
|
||||
- Audio streaming back to browser
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from fastapi.responses import FileResponse
|
||||
from loguru import logger
|
||||
from pydantic_settings import BaseSettings
|
||||
|
||||
from .stt import WhisperSTT
|
||||
from .tts import ChatterboxTTS
|
||||
from .backend import AIBackend
|
||||
from .vad import VoiceActivityDetector
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
"""Server configuration."""
|
||||
|
||||
# Server
|
||||
host: str = "0.0.0.0"
|
||||
port: int = 8765
|
||||
|
||||
# STT
|
||||
stt_model: str = "base" # tiny, base, small, medium, large-v3-turbo
|
||||
stt_device: str = "auto" # auto, cpu, cuda, mps
|
||||
|
||||
# TTS
|
||||
tts_model: str = "chatterbox"
|
||||
tts_voice: Optional[str] = None # Path to voice sample for cloning
|
||||
|
||||
# AI Backend
|
||||
backend_type: str = "openai" # openai, clawdbot, custom
|
||||
backend_url: str = "https://api.openai.com/v1"
|
||||
backend_model: str = "gpt-4o-mini"
|
||||
openai_api_key: Optional[str] = None
|
||||
|
||||
# Audio
|
||||
sample_rate: int = 16000
|
||||
|
||||
class Config:
|
||||
env_prefix = "OPENCLAW_"
|
||||
env_file = ".env"
|
||||
|
||||
|
||||
settings = Settings()
|
||||
app = FastAPI(title="OpenClaw Voice", version="0.1.0")
|
||||
|
||||
# Global instances (initialized on startup)
|
||||
stt: Optional[WhisperSTT] = None
|
||||
tts: Optional[ChatterboxTTS] = None
|
||||
backend: Optional[AIBackend] = None
|
||||
vad: Optional[VoiceActivityDetector] = None
|
||||
|
||||
|
||||
@app.on_event("startup")
|
||||
async def startup():
|
||||
"""Initialize models on server start."""
|
||||
global stt, tts, backend, vad
|
||||
|
||||
logger.info("Initializing OpenClaw Voice server...")
|
||||
|
||||
# Initialize STT
|
||||
logger.info(f"Loading STT model: {settings.stt_model}")
|
||||
stt = WhisperSTT(
|
||||
model_name=settings.stt_model,
|
||||
device=settings.stt_device,
|
||||
)
|
||||
|
||||
# Initialize TTS
|
||||
logger.info(f"Loading TTS model: {settings.tts_model}")
|
||||
tts = ChatterboxTTS(
|
||||
voice_sample=settings.tts_voice,
|
||||
)
|
||||
|
||||
# Initialize AI backend
|
||||
logger.info(f"Connecting to backend: {settings.backend_type}")
|
||||
backend = AIBackend(
|
||||
backend_type=settings.backend_type,
|
||||
url=settings.backend_url,
|
||||
model=settings.backend_model,
|
||||
api_key=settings.openai_api_key or os.getenv("OPENAI_API_KEY"),
|
||||
)
|
||||
|
||||
# Initialize VAD
|
||||
logger.info("Loading VAD model")
|
||||
vad = VoiceActivityDetector()
|
||||
|
||||
logger.info("✅ OpenClaw Voice server ready!")
|
||||
|
||||
|
||||
@app.get("/")
|
||||
async def index():
|
||||
"""Serve the demo page."""
|
||||
return FileResponse("src/client/index.html")
|
||||
|
||||
|
||||
@app.websocket("/ws")
|
||||
async def websocket_endpoint(websocket: WebSocket):
|
||||
"""Handle voice WebSocket connections."""
|
||||
await websocket.accept()
|
||||
logger.info("Client connected")
|
||||
|
||||
audio_buffer = []
|
||||
is_listening = False
|
||||
|
||||
try:
|
||||
while True:
|
||||
data = await websocket.receive_text()
|
||||
msg = json.loads(data)
|
||||
|
||||
if msg["type"] == "start_listening":
|
||||
is_listening = True
|
||||
audio_buffer = []
|
||||
await websocket.send_json({"type": "listening_started"})
|
||||
logger.debug("Started listening")
|
||||
|
||||
elif msg["type"] == "stop_listening":
|
||||
is_listening = False
|
||||
|
||||
if audio_buffer:
|
||||
# Combine audio chunks
|
||||
audio_data = np.concatenate(audio_buffer)
|
||||
|
||||
# Transcribe
|
||||
logger.debug("Transcribing audio...")
|
||||
transcript = await stt.transcribe(audio_data)
|
||||
|
||||
await websocket.send_json({
|
||||
"type": "transcript",
|
||||
"text": transcript,
|
||||
"final": True,
|
||||
})
|
||||
logger.info(f"Transcript: {transcript}")
|
||||
|
||||
if transcript.strip():
|
||||
# Get AI response
|
||||
logger.debug("Getting AI response...")
|
||||
response_text = await backend.chat(transcript)
|
||||
|
||||
await websocket.send_json({
|
||||
"type": "response_text",
|
||||
"text": response_text,
|
||||
})
|
||||
logger.info(f"Response: {response_text}")
|
||||
|
||||
# Generate speech
|
||||
logger.debug("Generating speech...")
|
||||
audio_response = await tts.synthesize(response_text)
|
||||
|
||||
# Send audio back
|
||||
audio_b64 = base64.b64encode(audio_response.tobytes()).decode()
|
||||
await websocket.send_json({
|
||||
"type": "audio_response",
|
||||
"data": audio_b64,
|
||||
"sample_rate": 24000, # TTS output rate
|
||||
"text": response_text,
|
||||
})
|
||||
|
||||
audio_buffer = []
|
||||
await websocket.send_json({"type": "listening_stopped"})
|
||||
logger.debug("Stopped listening")
|
||||
|
||||
elif msg["type"] == "audio" and is_listening:
|
||||
# Decode base64 audio
|
||||
audio_bytes = base64.b64decode(msg["data"])
|
||||
audio_np = np.frombuffer(audio_bytes, dtype=np.float32)
|
||||
audio_buffer.append(audio_np)
|
||||
|
||||
elif msg["type"] == "ping":
|
||||
await websocket.send_json({"type": "pong"})
|
||||
|
||||
except WebSocketDisconnect:
|
||||
logger.info("Client disconnected")
|
||||
except Exception as e:
|
||||
logger.error(f"WebSocket error: {e}")
|
||||
await websocket.close()
|
||||
|
||||
|
||||
# Serve static files for client
|
||||
client_dir = Path(__file__).parent.parent / "client"
|
||||
if client_dir.exists():
|
||||
app.mount("/static", StaticFiles(directory=str(client_dir)), name="static")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(
|
||||
"src.server.main:app",
|
||||
host=settings.host,
|
||||
port=settings.port,
|
||||
reload=True,
|
||||
)
|
||||
@@ -0,0 +1,109 @@
|
||||
"""
|
||||
Speech-to-Text module using Whisper.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class WhisperSTT:
|
||||
"""Whisper-based Speech-to-Text."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model_name: str = "base",
|
||||
device: str = "auto",
|
||||
language: str = "en",
|
||||
):
|
||||
self.model_name = model_name
|
||||
self.device = device
|
||||
self.language = language
|
||||
self.model = None
|
||||
self._backend = "mock"
|
||||
self._load_model()
|
||||
|
||||
def _load_model(self):
|
||||
"""Load the Whisper model."""
|
||||
# Try faster-whisper first
|
||||
try:
|
||||
from faster_whisper import WhisperModel
|
||||
|
||||
if self.device == "auto":
|
||||
import torch
|
||||
if torch.cuda.is_available():
|
||||
self.device = "cuda"
|
||||
compute_type = "float16"
|
||||
elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
|
||||
self.device = "cpu"
|
||||
compute_type = "int8"
|
||||
else:
|
||||
self.device = "cpu"
|
||||
compute_type = "int8"
|
||||
elif self.device == "cuda":
|
||||
compute_type = "float16"
|
||||
else:
|
||||
compute_type = "int8"
|
||||
|
||||
logger.info(f"Loading faster-whisper {self.model_name} on {self.device}")
|
||||
self.model = WhisperModel(
|
||||
self.model_name,
|
||||
device=self.device if self.device != "mps" else "cpu",
|
||||
compute_type=compute_type,
|
||||
)
|
||||
self._backend = "faster-whisper"
|
||||
logger.info("✅ faster-whisper loaded")
|
||||
return
|
||||
except ImportError:
|
||||
logger.warning("faster-whisper not available")
|
||||
except Exception as e:
|
||||
logger.warning(f"faster-whisper failed: {e}")
|
||||
|
||||
# Try openai-whisper
|
||||
try:
|
||||
import whisper
|
||||
|
||||
if self.device == "auto":
|
||||
import torch
|
||||
self.device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
|
||||
logger.info(f"Loading openai-whisper {self.model_name}")
|
||||
self.model = whisper.load_model(self.model_name, device=self.device)
|
||||
self._backend = "openai-whisper"
|
||||
logger.info("✅ openai-whisper loaded")
|
||||
return
|
||||
except ImportError:
|
||||
logger.warning("openai-whisper not available")
|
||||
except Exception as e:
|
||||
logger.warning(f"openai-whisper failed: {e}")
|
||||
|
||||
# Mock mode for testing
|
||||
logger.warning("⚠️ No STT backend - using mock mode")
|
||||
self._backend = "mock"
|
||||
|
||||
async def transcribe(self, audio: np.ndarray) -> str:
|
||||
"""Transcribe audio to text."""
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(None, self._transcribe_sync, audio)
|
||||
|
||||
def _transcribe_sync(self, audio: np.ndarray) -> str:
|
||||
"""Synchronous transcription."""
|
||||
if self._backend == "faster-whisper":
|
||||
segments, info = self.model.transcribe(
|
||||
audio,
|
||||
language=self.language,
|
||||
beam_size=5,
|
||||
vad_filter=True,
|
||||
)
|
||||
return " ".join(segment.text for segment in segments).strip()
|
||||
|
||||
elif self._backend == "openai-whisper":
|
||||
result = self.model.transcribe(audio, language=self.language)
|
||||
return result["text"].strip()
|
||||
|
||||
else:
|
||||
# Mock mode - return placeholder
|
||||
logger.debug(f"Mock STT: received {len(audio)} samples")
|
||||
return "[Mock transcription - install whisper for real STT]"
|
||||
@@ -0,0 +1,97 @@
|
||||
"""
|
||||
Text-to-Speech module using Chatterbox or fallbacks.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from typing import Optional
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class ChatterboxTTS:
|
||||
"""Text-to-Speech using Chatterbox or fallbacks."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
voice_sample: Optional[str] = None,
|
||||
device: str = "auto",
|
||||
):
|
||||
self.voice_sample = voice_sample
|
||||
self.device = device
|
||||
self.model = None
|
||||
self._backend = "mock"
|
||||
self._load_model()
|
||||
|
||||
def _load_model(self):
|
||||
"""Load the TTS model."""
|
||||
# Try Chatterbox
|
||||
try:
|
||||
from chatterbox.tts import ChatterboxTTS as CBModel
|
||||
logger.info("Loading Chatterbox TTS...")
|
||||
self.model = CBModel.from_pretrained(device=self._get_device())
|
||||
self._backend = "chatterbox"
|
||||
logger.info("✅ Chatterbox loaded")
|
||||
return
|
||||
except ImportError:
|
||||
logger.warning("Chatterbox not installed")
|
||||
except Exception as e:
|
||||
logger.warning(f"Chatterbox failed: {e}")
|
||||
|
||||
# Try XTTS
|
||||
try:
|
||||
from TTS.api import TTS
|
||||
logger.info("Loading Coqui XTTS...")
|
||||
self.model = TTS("tts_models/multilingual/multi-dataset/xtts_v2")
|
||||
self._backend = "xtts"
|
||||
logger.info("✅ XTTS loaded")
|
||||
return
|
||||
except ImportError:
|
||||
logger.warning("Coqui TTS not installed")
|
||||
except Exception as e:
|
||||
logger.warning(f"XTTS failed: {e}")
|
||||
|
||||
# Mock mode
|
||||
logger.warning("⚠️ No TTS backend - using mock mode (silence)")
|
||||
self._backend = "mock"
|
||||
|
||||
def _get_device(self) -> str:
|
||||
if self.device != "auto":
|
||||
return self.device
|
||||
try:
|
||||
import torch
|
||||
if torch.cuda.is_available():
|
||||
return "cuda"
|
||||
elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
|
||||
return "mps"
|
||||
except ImportError:
|
||||
pass
|
||||
return "cpu"
|
||||
|
||||
async def synthesize(self, text: str) -> np.ndarray:
|
||||
"""Synthesize speech from text."""
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(None, self._synthesize_sync, text)
|
||||
|
||||
def _synthesize_sync(self, text: str) -> np.ndarray:
|
||||
"""Synchronous synthesis."""
|
||||
if self._backend == "chatterbox":
|
||||
if self.voice_sample:
|
||||
audio = self.model.generate(text, audio_prompt=self.voice_sample)
|
||||
else:
|
||||
audio = self.model.generate(text)
|
||||
return audio.cpu().numpy().astype(np.float32)
|
||||
|
||||
elif self._backend == "xtts":
|
||||
if self.voice_sample:
|
||||
wav = self.model.tts(text=text, speaker_wav=self.voice_sample, language="en")
|
||||
else:
|
||||
wav = self.model.tts(text=text, language="en")
|
||||
return np.array(wav, dtype=np.float32)
|
||||
|
||||
else:
|
||||
# Mock mode - return short silence
|
||||
logger.debug(f"Mock TTS: '{text[:50]}...'")
|
||||
# 0.5 seconds of silence at 24kHz
|
||||
return np.zeros(12000, dtype=np.float32)
|
||||
@@ -0,0 +1,45 @@
|
||||
"""
|
||||
Voice Activity Detection module.
|
||||
"""
|
||||
|
||||
from typing import Optional
|
||||
import numpy as np
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class VoiceActivityDetector:
|
||||
"""Voice Activity Detection."""
|
||||
|
||||
def __init__(self, threshold: float = 0.5):
|
||||
self.threshold = threshold
|
||||
self.model = None
|
||||
self._load_model()
|
||||
|
||||
def _load_model(self):
|
||||
"""Load VAD model."""
|
||||
try:
|
||||
import torch
|
||||
model, utils = torch.hub.load(
|
||||
repo_or_dir='snakers4/silero-vad',
|
||||
model='silero_vad',
|
||||
force_reload=False,
|
||||
)
|
||||
self.model = model
|
||||
self._get_speech_timestamps = utils[0]
|
||||
logger.info("✅ Silero VAD loaded")
|
||||
except Exception as e:
|
||||
logger.warning(f"VAD not available: {e}")
|
||||
self.model = None
|
||||
|
||||
def is_speech(self, audio: np.ndarray, sample_rate: int = 16000) -> bool:
|
||||
"""Check if audio contains speech."""
|
||||
if self.model is None:
|
||||
return True # Assume speech if no VAD
|
||||
try:
|
||||
import torch
|
||||
audio_tensor = torch.from_numpy(audio).float()
|
||||
speech_prob = self.model(audio_tensor, sample_rate).item()
|
||||
return speech_prob > self.threshold
|
||||
except Exception as e:
|
||||
logger.error(f"VAD error: {e}")
|
||||
return True
|
||||
Reference in New Issue
Block a user