admin/openfang
1
0
Fork 0
mirror of https://github.com/RightNow-AI/openfang.git synced 2026-08-14 08:52:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(media): add audio_base_url override for local OpenAI-compat Whisper (#1124)

Browse Source 
Adds an optional `audio_base_url` field to `MediaConfig` that overrides
the hardcoded provider URLs in `media_understanding::transcribe_audio`,
allowing the same OpenAI-compatible multipart wire format to be sent to
a local Whisper service (speaches, faster-whisper-server, LM Studio,
etc.) instead of api.openai.com / api.groq.com.

Closes #1051.

## Why

Self-hosted, sovereignty-conscious, or rate-limited deployments often
need to route audio transcription to a local Whisper backend while
keeping `media_transcribe` / `speech_to_text` working as native tools
(no helper scripts, no shell_exec workarounds). Today the URLs in
`media_understanding.rs:118-128` are literal `&'static str` so neither
`OPENAI_BASE_URL` nor `provider_urls` (which the LLM drivers do
respect) is read for audio. The same problem existed for embeddings
and was already addressable via `provider_urls`, so this change keeps
the pattern symmetric for media at the simplest possible surface area.

## Wire format

The endpoint shape and Authorization header remain identical:

  POST <audio_base_url>/v1/audio/transcriptions
  Authorization: Bearer $<provider>_API_KEY
  Content-Type: multipart/form-data
  fields: file (binary), model, response_format=text

This means **any OpenAI-compatible Whisper server is drop-in**
(Speaches, faster-whisper-server, LM Studio's Whisper server, etc.).
Local servers typically accept any non-empty bearer string, so users
can keep `OPENAI_API_KEY=anything` for the auth header.

## Configuration

```toml
[media]
audio_provider = "openai"
audio_base_url = "http://127.0.0.1:8000"
# → POST http://127.0.0.1:8000/v1/audio/transcriptions
```

Or for Groq-compatible local servers:

```toml
[media]
audio_provider = "groq"
audio_base_url = "http://127.0.0.1:9000"
# → POST http://127.0.0.1:9000/v1/audio/transcriptions
```

Trailing slash on the user-supplied base is stripped to avoid double
slashes in the final URL.

## Backward compatibility

- `MediaConfig` already uses `#[serde(default)]`, so existing
  configs without `audio_base_url` deserialize as `None` and behave
  exactly as before (cloud provider URLs).
- `Default` impl extended; `audio_base_url: None`.
- `parakeet-mlx` provider path unaffected (it's a separate code branch).
- No new dependencies, no breaking changes to public API.

## Tests

- `test_media_config_default` extended to assert `audio_base_url.is_none()`.
- `test_media_config_audio_base_url_serde_roundtrip` — set + JSON roundtrip.
- `test_media_config_backward_compat_no_audio_base_url` — legacy JSON
  parses with the new field as None.
- `test_audio_base_url_override_logic` — pure-function test that
  exercises the URL building branch (default URLs preserved when
  unset, override applied for both providers, trailing-slash strip).

The runtime branch in `transcribe_audio` was kept as a straight
`if Some/else default` rather than a helper function to minimize the
diff and keep the patch obviously safe to review.

## Operational note

This change does not affect anyone running the cloud provider URLs
out of the box. The override is opt-in via a single optional config
field. Useful for users like myself running a local Speaches container
behind a reverse proxy and a chat-only LLM key (z.ai Coding Plan)
that can't satisfy openai.com's audio endpoint.

Linked: #1051 (Configurable STT/TTS/image URLs and local backends).

Co-authored-by: Miguel Guerrero <kortux@gmail.com>
This commit is contained in:
guatoc-ecohub
2026-04-29 14:28:04 +03:00
committed by GitHub
co-authored by Miguel Guerrero
parent e6bab993ae
commit 92f7e996de
2 changed files with 158 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/openfang-runtime/src/media_understanding.rs
+104 -10
View File
@@ -114,16 +114,54 @@ impl MediaEngine {
let model = default_audio_model(provider);
// Build API request
// Build API request.
//
// `audio_base_url` (config.media.audio_base_url) overrides the hardcoded
// provider URL when set, allowing the same OpenAI-compatible multipart
// wire format to be sent to a local Whisper service (speaches,
// faster-whisper-server, LM Studio, etc.) instead of the cloud provider.
// The Authorization header is still built from the provider's standard
// env var (`*_API_KEY`); local services typically accept any non-empty
// bearer token. Closes #1051.
let (api_url, api_key) = match provider {
"groq" => (
"https://api.groq.com/openai/v1/audio/transcriptions",
std::env::var("GROQ_API_KEY").map_err(|_| "GROQ_API_KEY not set")?,
),
"openai" => (
"https://api.openai.com/v1/audio/transcriptions",
std::env::var("OPENAI_API_KEY").map_err(|_| "OPENAI_API_KEY not set")?,
),
"groq" => {
let url = self
.config
.audio_base_url
.as_deref()
.map(|base| {
format!(
"{}/v1/audio/transcriptions",
base.trim_end_matches('/')
)
})
.unwrap_or_else(|| {
"https://api.groq.com/openai/v1/audio/transcriptions".to_string()
});
(
url,
std::env::var("GROQ_API_KEY").map_err(|_| "GROQ_API_KEY not set")?,
)
}
"openai" => {
let url = self
.config
.audio_base_url
.as_deref()
.map(|base| {
format!(
"{}/v1/audio/transcriptions",
base.trim_end_matches('/')
)
})
.unwrap_or_else(|| {
"https://api.openai.com/v1/audio/transcriptions".to_string()
});
(
url,
std::env::var("OPENAI_API_KEY").map_err(|_| "OPENAI_API_KEY not set")?,
)
}
other => return Err(format!("Unsupported audio provider: {}", other)),
};
@@ -141,7 +179,7 @@ impl MediaEngine {
let client = reqwest::Client::new();
let resp = client
.post(api_url)
.post(&api_url)
.bearer_auth(&api_key)
.multipart(form)
.timeout(std::time::Duration::from_secs(60))
@@ -412,6 +450,62 @@ mod tests {
assert!(engine.semaphore.available_permits() <= 8);
}
/// Closes #1051: when `audio_base_url` is set, the URL building logic
/// must use the override (with `/v1/audio/transcriptions` appended) and
/// strip any trailing slash from the user-supplied base. When unset, the
/// hardcoded provider URL is used.
#[test]
fn test_audio_base_url_override_logic() {
// Helper closure mirroring the URL construction in `transcribe_audio`
// for both providers, kept in sync intentionally.
fn build(provider: &str, base: Option<&str>) -> String {
match provider {
"groq" => base
.map(|b| format!("{}/v1/audio/transcriptions", b.trim_end_matches('/')))
.unwrap_or_else(|| {
"https://api.groq.com/openai/v1/audio/transcriptions".to_string()
}),
"openai" => base
.map(|b| format!("{}/v1/audio/transcriptions", b.trim_end_matches('/')))
.unwrap_or_else(|| {
"https://api.openai.com/v1/audio/transcriptions".to_string()
}),
_ => unreachable!(),
}
}
// Default: hardcoded provider URLs preserved (backward compatibility).
assert_eq!(
build("openai", None),
"https://api.openai.com/v1/audio/transcriptions"
);
assert_eq!(
build("groq", None),
"https://api.groq.com/openai/v1/audio/transcriptions"
);
// Override applied for both providers.
assert_eq!(
build("openai", Some("http://127.0.0.1:8000")),
"http://127.0.0.1:8000/v1/audio/transcriptions"
);
assert_eq!(
build("groq", Some("http://localhost:9000")),
"http://localhost:9000/v1/audio/transcriptions"
);
// Trailing slash on the user-supplied base is stripped to avoid
// double slashes in the final URL.
assert_eq!(
build("openai", Some("http://127.0.0.1:8000/")),
"http://127.0.0.1:8000/v1/audio/transcriptions"
);
assert_eq!(
build("openai", Some("https://whisper.example.com/")),
"https://whisper.example.com/v1/audio/transcriptions"
);
}
#[tokio::test]
async fn test_describe_image_wrong_type() {
let engine = MediaEngine::new(MediaConfig::default());
crates/openfang-types/src/media.rs
+54
View File
@@ -75,6 +75,27 @@ pub struct MediaConfig {
pub image_provider: Option<String>,
/// Preferred audio transcription provider (auto-detect if None).
pub audio_provider: Option<String>,
/// Optional override for the audio transcription endpoint base URL.
///
/// When set, replaces the hardcoded provider URL with
/// `<audio_base_url>/v1/audio/transcriptions`. Use this to point to a
/// local OpenAI-compat Whisper service (e.g., speaches, faster-whisper-server,
/// LM Studio) while keeping the same multipart wire format and the
/// `audio_provider` semantics ("openai" or "groq" still selects which
/// `*_API_KEY` env var is used for the Authorization header).
///
/// Closes <https://github.com/RightNow-AI/openfang/issues/1051>.
///
/// Example:
/// ```toml
/// [media]
/// audio_provider = "openai"
/// audio_base_url = "http://127.0.0.1:8000"
/// # → POST http://127.0.0.1:8000/v1/audio/transcriptions
/// # with Authorization: Bearer $OPENAI_API_KEY (any non-empty string
/// # works for most local OpenAI-compat servers).
/// ```
pub audio_base_url: Option<String>,
}
impl Default for MediaConfig {
@@ -86,6 +107,7 @@ impl Default for MediaConfig {
max_concurrency: 2,
image_provider: None,
audio_provider: None,
audio_base_url: None,
}
}
}
@@ -359,6 +381,38 @@ mod tests {
assert!(!config.video_description);
assert_eq!(config.max_concurrency, 2);
assert!(config.image_provider.is_none());
assert!(config.audio_base_url.is_none());
}
#[test]
fn test_media_config_audio_base_url_serde_roundtrip() {
let mut config = MediaConfig::default();
config.audio_base_url = Some("http://127.0.0.1:8000".to_string());
config.audio_provider = Some("openai".to_string());
let json = serde_json::to_string(&config).unwrap();
let parsed: MediaConfig = serde_json::from_str(&json).unwrap();
assert_eq!(
parsed.audio_base_url.as_deref(),
Some("http://127.0.0.1:8000")
);
assert_eq!(parsed.audio_provider.as_deref(), Some("openai"));
}
#[test]
fn test_media_config_backward_compat_no_audio_base_url() {
// Old TOML/JSON without `audio_base_url` must still parse with None,
// thanks to #[serde(default)] on the struct.
let legacy_json = r#"{
"image_description": true,
"audio_transcription": true,
"video_description": false,
"max_concurrency": 2,
"image_provider": null,
"audio_provider": "openai"
}"#;
let parsed: MediaConfig = serde_json::from_str(legacy_json).unwrap();
assert!(parsed.audio_base_url.is_none());
assert_eq!(parsed.audio_provider.as_deref(), Some("openai"));
}
#[test]