mirror of
https://github.com/moeru-ai/airi.git
synced 2026-08-14 08:52:42 +00:00
feat(analytics): add product instrumentation for activation and voice flows (#2023)
This commit is contained in:
Lovehsigure_520
@@ -33,6 +33,10 @@
|
||||
- 全量 metric 目录（按域分组：HTTP / Auth / Engagement / Revenue / GenAI / Email / Rate limit / Runtime），含名字、类型、Labels、落点
|
||||
- `metrics-ownership.md`
|
||||
- 指标分层规则：什么走 Grafana / 什么走 PostHog / 什么是 Postgres truth；含 7 题判定 Checklist、PostHog 事件命名约定、当前指标归属总表、PostHog 接入路线图
|
||||
- `product-analytics-instrumentation.md`
|
||||
- 面向社区 / 产品问题的埋点补充方案：上手激活、Provider 配置、TTS 音色、语音输入、反馈、看板和异常播报
|
||||
- `product-analytics-dashboard-setup.md`
|
||||
- 产品分析看板落地说明：PostHog insights、Grafana 产品事件面板、告警表达式；不含 Discord / QQ 同步和日报 / 周报脚本
|
||||
- `auth-and-oidc.md`
|
||||
- 认证与 OIDC Provider 架构、登录流程、trusted clients、踩坑记录
|
||||
- `email-auth-resend.md`
|
||||
@@ -55,6 +59,8 @@
|
||||
- 70.2K 历史漏账的取证 SQL + Loki query 模板、处理决策框架、修补后的监控建议
|
||||
- `verifications/admin-user-balance-ban.md`
|
||||
- Admin role 鉴权 / 封禁热路径闸 / 改余额：role adminGuard、resolveRequestAuth+userinfo 封禁、setFlux 的真实 PGlite/Hono 执行证据（含 flux-grants 集成测试走 role），及 better-auth admin 端点本身待端到端实测
|
||||
- `verifications/product-analytics-smoke.md`
|
||||
- 产品分析埋点上线冒烟清单：PostHog journey events、Postgres TTS metadata、Grafana Product Analytics row、Prometheus label 安全边界
|
||||
|
||||
## 快速结论
|
||||
|
||||
|
||||
@@ -0,0 +1,305 @@
|
||||
# Product Analytics Dashboard Setup
|
||||
|
||||
This document turns `product-analytics-instrumentation.md` into dashboard setup steps. It intentionally excludes Discord / QQ ingestion and daily / weekly report automation.
|
||||
|
||||
After setting up the dashboards, run `verifications/product-analytics-smoke.md` against the deployed environment.
|
||||
|
||||
## Scope
|
||||
|
||||
In scope:
|
||||
|
||||
- PostHog insights for frontend product journeys.
|
||||
- Grafana panels for server-side product event health.
|
||||
- Alert rules that can be configured directly in PostHog / Grafana.
|
||||
|
||||
Out of scope for this pass:
|
||||
|
||||
- Discord / QQ bot or spreadsheet synchronization.
|
||||
- Daily / weekly report generation scripts.
|
||||
|
||||
## Destination Rules
|
||||
|
||||
| Question | Destination | Reason |
|
||||
|---|---|---|
|
||||
| Can a user start chatting? | PostHog | Frontend journey and distinct user funnels |
|
||||
| Where does provider setup fail? | PostHog | Provider config events are frontend PostHog events |
|
||||
| Which voice is selected or previewed? | PostHog / Postgres metadata | `voice_id` is high-cardinality and must not be a Prometheus label |
|
||||
| Is server TTS healthy right now? | Grafana | Server product events and OTel metrics are Prometheus-safe |
|
||||
| Are users submitting feedback? | PostHog | App feedback is frontend product analytics |
|
||||
|
||||
## PostHog Dashboard
|
||||
|
||||
Create a dashboard named `AIRI Activation And Feedback`.
|
||||
|
||||
Live dashboard created on 2026-06-30:
|
||||
|
||||
- Project: `Project AIRI (Web)` (`90721`)
|
||||
- URL: `https://us.posthog.com/project/90721/dashboard/1779029`
|
||||
- Current cards:
|
||||
- Text card: `AIRI product analytics runbook`
|
||||
- Funnel: `Chat activation funnel`
|
||||
- Trend: `Provider config failures`
|
||||
- Trend: `TTS voice selection and preview`
|
||||
- Trend: `Top selected TTS voices`
|
||||
- Trend: `Voice input friction`
|
||||
- Trend: `Feedback and bug reports`
|
||||
|
||||
### Insight 1: Chat Activation Funnel
|
||||
|
||||
Type: Funnel
|
||||
|
||||
Steps:
|
||||
|
||||
1. `chat_activation_started`
|
||||
2. `chat_activation_succeeded`
|
||||
|
||||
Breakdowns:
|
||||
|
||||
- `provider_mode`
|
||||
- `surface`
|
||||
|
||||
Filters:
|
||||
|
||||
- Date range: last 7 days
|
||||
- Exclude internal users if the project has an internal user cohort.
|
||||
|
||||
Watch for:
|
||||
|
||||
- Official provider conversion lower than custom provider conversion.
|
||||
- Large drop after `chat_activation_started`.
|
||||
|
||||
### Insight 2: Chat Activation Failures
|
||||
|
||||
Type: Trends
|
||||
|
||||
Events:
|
||||
|
||||
- `chat_activation_failed`
|
||||
|
||||
Breakdowns:
|
||||
|
||||
- `failure_stage`
|
||||
- `error_code`
|
||||
- `provider_mode`
|
||||
|
||||
Display:
|
||||
|
||||
- Stacked bar or line chart.
|
||||
|
||||
Watch for:
|
||||
|
||||
- `failure_stage = provider_config`
|
||||
- `failure_stage = model_list`
|
||||
- `failure_stage = llm_response`
|
||||
|
||||
### Insight 3: Provider Configuration Health
|
||||
|
||||
Type: Funnel
|
||||
|
||||
Steps:
|
||||
|
||||
1. `provider_config_started`
|
||||
2. `provider_config_succeeded`
|
||||
|
||||
Breakdowns:
|
||||
|
||||
- `provider_mode`
|
||||
- `step`
|
||||
|
||||
Companion trend:
|
||||
|
||||
- Event: `provider_config_failed`
|
||||
- Breakdown: `error_code`
|
||||
|
||||
Watch for:
|
||||
|
||||
- Official provider failures greater than zero for more than 15 minutes.
|
||||
- `step = manual_chat_ping` failures after auto validation succeeds.
|
||||
|
||||
### Insight 4: Model List Health
|
||||
|
||||
Type: Trends
|
||||
|
||||
Events:
|
||||
|
||||
- `model_list_loaded`
|
||||
- `model_list_failed`
|
||||
|
||||
Breakdowns:
|
||||
|
||||
- `provider_id`
|
||||
- `provider_mode`
|
||||
|
||||
Watch for:
|
||||
|
||||
- `model_list_failed` spikes for one provider.
|
||||
- High failure rate after a release.
|
||||
|
||||
### Insight 5: TTS Voice Selection
|
||||
|
||||
Type: Trends
|
||||
|
||||
Events:
|
||||
|
||||
- `voice_selected`
|
||||
- `voice_preview_played`
|
||||
- `voice_pack_bound`
|
||||
|
||||
Breakdowns:
|
||||
|
||||
- `voice_type`
|
||||
- `tts_provider_id`
|
||||
- `source`
|
||||
|
||||
Do not use:
|
||||
|
||||
- Prometheus labels for `voice_id` or `voice_pack_id`.
|
||||
|
||||
Use PostHog or SQL when grouping by:
|
||||
|
||||
- `voice_id`
|
||||
- `voice_pack_id`
|
||||
|
||||
### Insight 6: Voice Input Friction
|
||||
|
||||
Type: Funnel
|
||||
|
||||
Steps:
|
||||
|
||||
1. `voice_input_started`
|
||||
2. `stt_succeeded`
|
||||
|
||||
Companion trends:
|
||||
|
||||
- `microphone_permission_denied`
|
||||
- `audio_device_unavailable`
|
||||
- `voice_input_cancelled`
|
||||
- `stt_failed`
|
||||
|
||||
Breakdowns:
|
||||
|
||||
- `stt_provider_id`
|
||||
- `error_code`
|
||||
- `surface`
|
||||
|
||||
### Insight 7: Feedback And Bug Reports
|
||||
|
||||
Type: Trends
|
||||
|
||||
Events:
|
||||
|
||||
- `feedback_submitted`
|
||||
- `bug_report_submitted`
|
||||
|
||||
Breakdowns:
|
||||
|
||||
- `category`
|
||||
- `severity`
|
||||
- `entrypoint`
|
||||
- `surface`
|
||||
|
||||
Watch for:
|
||||
|
||||
- `severity = blocker` spikes.
|
||||
- `entrypoint = about_update_error` after desktop releases.
|
||||
|
||||
## Grafana Dashboard
|
||||
|
||||
Source of truth:
|
||||
|
||||
- `apps/server/otel/grafana/dashboards/build.ts`
|
||||
- Generated JSON: `apps/server/otel/grafana/dashboards/airi-server-overview-cloud.json`
|
||||
|
||||
The `Product Analytics` row includes:
|
||||
|
||||
- `Product Events (range)`
|
||||
- `Product Failure %`
|
||||
- `TTS Success %`
|
||||
- `TTS Failed / Blocked (range)`
|
||||
- `Top Product Actions (range)`
|
||||
- `Product Event Rate`
|
||||
- `TTS Event Rate by Source`
|
||||
|
||||
Live import status:
|
||||
|
||||
- Imported on 2026-06-30.
|
||||
- Live URL: `https://projairi.grafana.net/d/ad8qbp5/airi-server-overview`
|
||||
- Dashboard: `AIRI Server Overview - Product Analytics` (`ad8qbp5`)
|
||||
- The live dashboard now shows the full `Product Analytics` row:
|
||||
- `Product Events (range)`
|
||||
- `Product Failure %`
|
||||
- `TTS Success %`
|
||||
- `TTS Failed / Blocked (range)`
|
||||
- `Top Product Actions (range)`
|
||||
- `Product Event Rate`
|
||||
- `TTS Event Rate by Source`
|
||||
- The generated JSON remains the source of truth for the Product Analytics / TTS panel set.
|
||||
|
||||
Permission notes from the import retry:
|
||||
|
||||
- The first import attempt with title `AIRI Server Overview` and UID `rbr55dn` showed duplicate title / UID warnings because it targets the existing dashboard.
|
||||
- A second import attempt with a new title / UID (`AIRI Server Overview - Product Analytics Test`, `airi-product-analytics-test`) removed the duplicate warnings, but still did not import.
|
||||
- API confirmation returned `403 Access denied`: `You'll need additional permissions to perform this action. Permissions needed: any of dashboards:create, dashboards:write`.
|
||||
- The logged-in Grafana user `1260907335@qq.com` has org role `Viewer`; API metadata for `/d/rbr55dn/airi-server-overview` reports `canSave=false`, `canEdit=false`, `canAdmin=false`.
|
||||
- After permissions were updated, the generated dashboard was imported from Microsoft Edge. Grafana assigned the imported dashboard UID `ad8qbp5` instead of overwriting the earlier `rbr55dn` dashboard, so the imported dashboard was renamed to `AIRI Server Overview - Product Analytics` to avoid ambiguity.
|
||||
|
||||
Regenerate after dashboard changes:
|
||||
|
||||
```bash
|
||||
node node_modules/tsx/dist/cli.mjs apps/server/otel/grafana/dashboards/build.ts
|
||||
```
|
||||
|
||||
## Alert Setup
|
||||
|
||||
### PostHog Alerts
|
||||
|
||||
Configure these as insight subscriptions or monitor-style alerts.
|
||||
|
||||
| Alert | Insight | Trigger |
|
||||
|---|---|---|
|
||||
| Activation drop | Chat Activation Funnel | `chat_activation_succeeded / chat_activation_started` drops by 15% vs previous 24h |
|
||||
| Provider config regression | Provider Configuration Health | Official provider `provider_config_failed` is greater than 0 for 15 minutes |
|
||||
| Voice input spike | Voice Input Friction | `stt_failed / voice_input_started` exceeds 20% over 1h |
|
||||
| Feedback spike | Feedback And Bug Reports | `bug_report_submitted` doubles vs previous 24h |
|
||||
|
||||
### Grafana Alerts
|
||||
|
||||
Use these PromQL expressions from the server dashboard context.
|
||||
|
||||
TTS success below 95% over 15 minutes:
|
||||
|
||||
```promql
|
||||
100 * sum(increase(airi_product_events_total{feature="tts", action="speech_succeeded", status="succeeded"}[15m]))
|
||||
/
|
||||
clamp_min(sum(increase(airi_product_events_total{feature="tts", action="speech_requested", status="started"}[15m])), 1)
|
||||
< 95
|
||||
```
|
||||
|
||||
TTS blocked spike over 15 minutes:
|
||||
|
||||
```promql
|
||||
sum(increase(airi_product_events_total{feature="tts", action="speech_blocked", status="blocked"}[15m])) > 10
|
||||
```
|
||||
|
||||
TTS failed spike over 15 minutes:
|
||||
|
||||
```promql
|
||||
sum(increase(airi_product_events_total{feature="tts", action="speech_failed", status="failed"}[15m])) > 5
|
||||
```
|
||||
|
||||
Product failure ratio above 10% over 15 minutes:
|
||||
|
||||
```promql
|
||||
100 * sum(increase(airi_product_events_total{feature!="", action!="", status="failed"}[15m]))
|
||||
/
|
||||
clamp_min(sum(increase(airi_product_events_total{feature!="", action!=""}[15m])), 1)
|
||||
> 10
|
||||
```
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
- PostHog can show `chat_activation_started -> chat_activation_succeeded` by `provider_mode`.
|
||||
- PostHog can show `voice_selected` by `voice_type` and `tts_provider_id`.
|
||||
- PostHog can show `feedback_submitted` and `bug_report_submitted`.
|
||||
- Grafana dashboard JSON contains `TTS Success %`, `TTS Failed / Blocked (range)`, and `TTS Event Rate by Source`.
|
||||
- Grafana product analytics panels use only bounded labels: `feature`, `action`, `status`, `source`.
|
||||
@@ -0,0 +1,891 @@
|
||||
# Product Analytics Instrumentation Plan（产品埋点与数据播报方案）
|
||||
|
||||
这份文档把 AIRI 当前埋点现状、社区侧最关心的问题、缺口、事件 schema、看板和异常播报整理到一处。它补充 [`metrics-ownership.md`](./metrics-ownership.md)：后者定义指标归属和命名规则，本文定义“为了回答产品/社区问题，接下来要补什么”。
|
||||
|
||||
## TL;DR
|
||||
|
||||
- 现在已经能看 activation、Provider 配置次数、国家来源、付费漏斗、LLM 请求和服务端 TTS 请求健康。
|
||||
- 现在还不能可靠回答“用户最常用哪个 TTS 音色 / Voice Pack”，因为前端和服务端事件都缺稳定的 `voice_id` / `voice_type` / `voice_pack_id`。
|
||||
- 最优先补的不是更多点击，而是“能不能开始聊天”“卡在哪个配置步骤”“语音 / TTS 为什么失败”。
|
||||
- 官方 Provider 要和自配置 Provider 分开看，核心判断是官方路径是否提高 activation、降低配置失败和缩短首次聊天时间。
|
||||
- 自配置用户可能更高价值，不能只看失败率；需要同时看 retention、paid conversion 和 feedback rate。
|
||||
- PostHog 负责用户路径、漏斗、留存和分群；Grafana 负责服务端健康和异常；Postgres / SQL 负责高基数字段聚合，例如 voice / voice pack。
|
||||
- Prometheus 不要加 `voice_id`、`voice_pack_id`、用户自定义模型名等高基数字段。
|
||||
- 日报先做“指标 + 异常提醒”，周报再加入社区侧解释和下周行动建议。
|
||||
- Discord / QQ 反馈要有轻量标签，不能完全靠埋点替代社区观察。
|
||||
|
||||
## 背景
|
||||
|
||||
社区反馈里最常见的劝退点集中在：
|
||||
|
||||
- 性能问题
|
||||
- 模型 / Provider 配置复杂
|
||||
- 配置失败或模型列表加载失败
|
||||
- Bug 多，用户不知道卡在哪
|
||||
- 语音输入体验不稳定
|
||||
|
||||
产品方向是降低上手门槛，让 AIRI 更开箱即用；官方 Provider 已经上线，但仍有一部分用户喜欢自配置模型和音色。社区侧同时关心海外用户、付费用户、二次开发用户、角色聊天用户，以及愿意反馈问题的用户。
|
||||
|
||||
因此埋点目标不是单纯统计点击，而是回答：
|
||||
|
||||
1. 新用户有没有正常开始聊天？
|
||||
2. 用户卡在 Provider / 模型配置的哪一步？
|
||||
3. 官方 Provider 是否真的降低了上手门槛？
|
||||
4. 自配置用户和官方 Provider 用户在留存、付费、反馈上有什么差异？
|
||||
5. TTS 音色和提供商到底怎么被选择、试听和实际使用？
|
||||
6. Bug、性能、语音输入失败分别造成了多少流失？
|
||||
|
||||
## 现状审计
|
||||
|
||||
### PostHog
|
||||
|
||||
线上 `Project AIRI (Web)` 已有核心 dashboard：`My App Dashboard`。
|
||||
|
||||
已覆盖：
|
||||
|
||||
- first message 趋势
|
||||
- Provider 配置次数
|
||||
- 国家来源
|
||||
- Signup activation funnel
|
||||
- 7-day activation retention
|
||||
- Paywall conversion
|
||||
- LLM request volume / error rate
|
||||
- rage-click friction
|
||||
|
||||
近 7 天仍活跃的主要产品事件包括：
|
||||
|
||||
- `app_loaded`
|
||||
- `first_message_sent`
|
||||
- `first_model_selected`
|
||||
- `model_switched`
|
||||
- `message_send_started`
|
||||
- `llm_request_started`
|
||||
- `llm_first_token`
|
||||
- `message_round`
|
||||
- `chat_session_started`
|
||||
- `chat_session_selected`
|
||||
- `chat_message_deleted`
|
||||
- `chat_messages_cleared`
|
||||
- `pricing_page_viewed`
|
||||
- `plan_selected`
|
||||
- `checkout_started`
|
||||
- `provider_card_clicked`
|
||||
- `stt_started`
|
||||
- `stt_succeeded`
|
||||
- `stt_failed`
|
||||
- `tts_stop_clicked`
|
||||
- `character_switched`
|
||||
- `character_deleted`
|
||||
|
||||
PostHog 当前缺口：
|
||||
|
||||
- 没有 `voice_id`、`voice_pack_id`、`voice_type` 等 TTS 音色维度。
|
||||
- 近 30 天匹配 `tts` / `speech` / `voice` 的客户端事件里，`voice`、`voice_id`、`voiceId`、`voice_pack_id`、`voice_type` 均未出现有效值。
|
||||
- `model` / `model_id` 仍有自由文本风险，需要收敛白名单和脱敏策略。
|
||||
- PostHog 漏斗主要覆盖前端旅程，服务端 TTS 真实请求没有同步进入 PostHog。
|
||||
|
||||
### Grafana / Prometheus
|
||||
|
||||
线上 `AIRI Server Overview` 已覆盖：
|
||||
|
||||
- DAU / WAU / MAU
|
||||
- Active sessions
|
||||
- HTTP / WS health
|
||||
- Product Analytics：event volume、failure rate、top product actions、event rate
|
||||
- LLM Gateway：request rate by model、latency、provider failure
|
||||
- TTS Characters/s by Model
|
||||
- Stripe / revenue
|
||||
|
||||
Grafana 当前能回答：
|
||||
|
||||
- TTS 请求量、成功量、失败量、余额挡住量。
|
||||
- TTS 字符消耗按 model 聚合，例如 `stepfun/stepaudio-2.5-tts`、`volcengine/seed-tts-2.0`、`alibaba/cosyvoice-v1`。
|
||||
- 服务端 `product_events` 的 Prometheus label 当前包括 `feature`、`action`、`status`、`source`，再加部分 metric 的 `model` / `provider`。
|
||||
|
||||
Grafana 当前不能回答：
|
||||
|
||||
- 哪个 TTS 音色使用最多。
|
||||
- 哪个 Voice Pack 使用最多。
|
||||
- 官方默认音色与用户自选音色的转化差异。
|
||||
- 试听音色后是否真的绑定 / 使用。
|
||||
|
||||
## 用户分群
|
||||
|
||||
第一版不要求用户手动选 persona，先从行为推断。
|
||||
|
||||
| Segment | 初始推断规则 | 用途 |
|
||||
|---|---|---|
|
||||
| `new_user` | 账号创建后 7 天内或首次 `app_loaded` 后 7 天内 | 新手引导、激活漏斗 |
|
||||
| `official_provider_user` | 关键事件里 `provider_mode = official` | 衡量开箱即用效果 |
|
||||
| `custom_provider_user` | 关键事件里 `provider_mode = custom` | 衡量自配置门槛和失败率 |
|
||||
| `role_chat_user` | 导入 / 切换角色、绑定音色、频繁角色聊天 | 角色陪伴体验 |
|
||||
| `developer_user` | 使用插件、API、Devtools、二开相关入口 | 二次开发群体 |
|
||||
| `paid_user` | Stripe / Postgres 付费状态为真 | 付费转化与保留 |
|
||||
| `feedback_user` | 提交 bug report / feedback | 高价值社区用户 |
|
||||
|
||||
公共字段建议：
|
||||
|
||||
| Field | Values | Notes |
|
||||
|---|---|---|
|
||||
| `surface` | `web` / `desktop` / `mobile` | 所有关键前端事件必带 |
|
||||
| `provider_mode` | `official` / `custom` / `unknown` | 官方开箱即用 vs 用户自配置 |
|
||||
| `provider_id` | 白名单 ID | 不传 raw URL / raw key / 用户输入 |
|
||||
| `model_id` | 白名单或归一化后的 ID | 自定义模型用 `is_custom_model = true` |
|
||||
| `is_custom_model` | boolean | 避免把任意文本当 group-by |
|
||||
| `is_paid_user` | boolean | 从服务端或 PostHog person profile 派生 |
|
||||
| `setup_completed` | boolean | 是否完成基础配置 |
|
||||
| `region` | PostHog geo | 不由客户端手动传 |
|
||||
|
||||
## P0 埋点
|
||||
|
||||
### Chat Activation
|
||||
|
||||
目标：社区侧判断“能正常开始聊天”。
|
||||
|
||||
| Event | Owner | Truth | When |
|
||||
|---|---|---|---|
|
||||
| `chat_activation_started` | frontend | PostHog | 用户进入首次聊天路径或点击发送第一条消息前 |
|
||||
| `chat_activation_succeeded` | frontend | PostHog | 首次消息完成并看到 assistant response |
|
||||
| `chat_activation_failed` | frontend | PostHog | 首次消息未完成，包含配置、网络、鉴权、余额、模型等失败 |
|
||||
|
||||
字段：
|
||||
|
||||
| Field | Required | Notes |
|
||||
|---|---|---|
|
||||
| `provider_mode` | yes | `official` / `custom` |
|
||||
| `provider_id` | yes | 归一化 ID |
|
||||
| `model_id` | yes | 归一化 ID |
|
||||
| `surface` | yes | web / desktop / mobile |
|
||||
| `time_to_first_message_ms` | success only | 从 app start 或 onboarding complete 到首次成功 |
|
||||
| `error_code` | failed only | 稳定错误码 |
|
||||
| `failure_stage` | failed only | `provider_config` / `model_list` / `message_send` / `llm_response` / `tts` |
|
||||
|
||||
推荐看板：
|
||||
|
||||
- 新用户 `chat_activation_started -> chat_activation_succeeded` 漏斗。
|
||||
- 按 `provider_mode` 拆分 activation conversion。
|
||||
- `chat_activation_failed` 按 `failure_stage` / `provider_id` 排名。
|
||||
|
||||
异常提醒：
|
||||
|
||||
- 新用户 activation conversion 24h 环比下降超过 15%。
|
||||
- `provider_mode = official` 的 activation failure 上升，优先排查官方 Provider。
|
||||
|
||||
### Provider And Model Configuration
|
||||
|
||||
目标：定位配置复杂和失败劝退。
|
||||
|
||||
| Event | Owner | Truth | When |
|
||||
|---|---|---|---|
|
||||
| `provider_config_started` | frontend | PostHog | 打开 Provider 配置表单 |
|
||||
| `provider_config_succeeded` | frontend | PostHog | 配置保存并通过最小校验 |
|
||||
| `provider_config_failed` | frontend | PostHog | 保存、校验、鉴权或连接测试失败 |
|
||||
| `model_list_loaded` | frontend | PostHog | 模型列表加载成功 |
|
||||
| `model_list_failed` | frontend | PostHog | 模型列表加载失败 |
|
||||
|
||||
字段：
|
||||
|
||||
| Field | Required | Notes |
|
||||
|---|---|---|
|
||||
| `provider_id` | yes | 归一化 ID |
|
||||
| `provider_mode` | yes | 官方 Provider 也要记录 |
|
||||
| `step` | yes | `open_form` / `save` / `validate` / `load_models` / `select_model` |
|
||||
| `duration_ms` | no | 保存或加载耗时 |
|
||||
| `error_code` | failed only | 不能写 raw error message |
|
||||
| `http_status` | failed only | 有 HTTP 边界时记录 |
|
||||
|
||||
推荐看板：
|
||||
|
||||
- Provider 配置成功率。
|
||||
- 模型列表加载成功率。
|
||||
- Top failing providers。
|
||||
- 官方 Provider vs 自配置 Provider 配置耗时和失败率。
|
||||
|
||||
异常提醒：
|
||||
|
||||
- `model_list_failed` 任一 Provider 1h 内增长超过 2 倍。
|
||||
- 官方 Provider `provider_config_failed` 非零连续 15 分钟。
|
||||
|
||||
### TTS Voice And Provider
|
||||
|
||||
目标：回答“用户选择了哪种音色和提供商”。
|
||||
|
||||
| Event | Owner | Truth | When |
|
||||
|---|---|---|---|
|
||||
| `tts_provider_selected` | frontend | PostHog | 用户选择或切换 TTS Provider |
|
||||
| `voice_selected` | frontend | PostHog | 用户选择音色，包括官方默认落地时的 baseline |
|
||||
| `voice_preview_played` | frontend | PostHog | 用户试听音色 |
|
||||
| `voice_pack_bound` | frontend | PostHog | 用户把 Voice Pack 绑定到角色 |
|
||||
| `speech_requested` | server | Postgres/Grafana | REST / WS TTS 请求开始；沿用现有 `feature = tts` action |
|
||||
| `speech_succeeded` | server | Postgres/Grafana | REST / WS TTS 交付成功；沿用现有 `feature = tts` action |
|
||||
| `speech_failed` | server | Postgres/Grafana | TTS 上游、配置、路由失败；沿用现有 `feature = tts` action |
|
||||
| `speech_blocked` | server | Postgres/Grafana | Flux 不足等业务阻断；沿用现有 `feature = tts` action |
|
||||
|
||||
字段：
|
||||
|
||||
| Field | Required | Notes |
|
||||
|---|---|---|
|
||||
| `tts_provider_id` | yes | 归一化 ID |
|
||||
| `tts_model_id` | yes | 归一化 ID |
|
||||
| `voice_id` | yes | 官方 catalog ID；自定义音色见数据卫生 |
|
||||
| `voice_type` | yes | `official_default` / `official_selected` / `custom_configured` / `voice_pack` |
|
||||
| `voice_pack_id` | no | Voice Pack 绑定和服务端使用时记录 |
|
||||
| `source` | yes | `settings` / `onboarding` / `chat_auto_tts` / `manual_preview` |
|
||||
| `trigger` | server TTS | `auto` / `manual` |
|
||||
| `input_chars` | server TTS | 字数 |
|
||||
| `duration_ms` | server TTS | 端到端耗时 |
|
||||
| `error_code` | failed only | 稳定错误码 |
|
||||
|
||||
推荐看板：
|
||||
|
||||
- Top voices by `voice_selected`。
|
||||
- Top voices by server `feature = tts` + `action = speech_succeeded`。
|
||||
- Voice preview to selection conversion。
|
||||
- Official default vs official selected vs custom configured。
|
||||
- TTS failure / blocked rate by provider, model, voice type。
|
||||
|
||||
异常提醒：
|
||||
|
||||
- 某个官方音色的 server `speech_failed` 持续上升。
|
||||
- `voice_preview_played -> voice_selected` 转化下降。
|
||||
- server `speech_blocked` 突增，提示 Flux 或默认策略可能影响体验。
|
||||
|
||||
### Voice Input
|
||||
|
||||
目标：语音输入是明确社区痛点，需要把权限、设备和 Provider 失败拆开。
|
||||
|
||||
当前已有：
|
||||
|
||||
- `stt_started`
|
||||
- `stt_succeeded`
|
||||
- `stt_failed`
|
||||
|
||||
补充：
|
||||
|
||||
| Event | Owner | Truth | When |
|
||||
|---|---|---|---|
|
||||
| `voice_input_started` | frontend | PostHog | 用户开始语音输入 |
|
||||
| `microphone_permission_requested` | frontend | PostHog | 首次或重新请求麦克风权限 |
|
||||
| `microphone_permission_denied` | frontend | PostHog | 权限被拒绝 |
|
||||
| `audio_device_unavailable` | frontend | PostHog | 无设备、设备被占用、采样失败 |
|
||||
| `voice_input_cancelled` | frontend | PostHog | 用户主动取消或超时 |
|
||||
|
||||
字段：
|
||||
|
||||
| Field | Required | Notes |
|
||||
|---|---|---|
|
||||
| `stt_provider_id` | yes | 归一化 ID |
|
||||
| `surface` | yes | web / desktop / mobile |
|
||||
| `duration_ms` | no | 用户按住或录音时长 |
|
||||
| `error_code` | failed only | `permission_denied` / `device_unavailable` / `provider_error` / `timeout` |
|
||||
|
||||
推荐看板：
|
||||
|
||||
- Voice input start -> STT success funnel。
|
||||
- Permission denied rate by browser / surface。
|
||||
- STT failure rate by Provider。
|
||||
|
||||
### Feedback And Bug Reports
|
||||
|
||||
目标：社区侧认为“愿意反馈问题”是高价值行为，应进入核心指标。
|
||||
|
||||
| Event | Owner | Truth | When |
|
||||
|---|---|---|---|
|
||||
| `bug_report_opened` | frontend | PostHog | 打开 bug report dialog |
|
||||
| `bug_report_submitted` | frontend/server | PostHog + Postgres optional | 成功提交 |
|
||||
| `feedback_submitted` | frontend/server | PostHog + Postgres optional | 非 bug 的反馈 |
|
||||
| `community_feedback_tagged` | manual/job | Postgres optional | Discord / QQ 反馈被人工归类 |
|
||||
|
||||
字段：
|
||||
|
||||
| Field | Required | Notes |
|
||||
|---|---|---|
|
||||
| `source` | yes | `app` / `discord` / `qq` / `github` / `email` / `other` |
|
||||
| `category` | no | 见下方社区反馈标签 |
|
||||
| `severity` | yes | `blocker` / `major` / `minor` / `suggestion` |
|
||||
| `user_type` | yes | `new_user` / `paid_user` / `overseas_user` / `developer_user` / `role_chat_user` / `unknown` |
|
||||
| `entrypoint` | yes | `about_update_error` / `community_manual_tag` 等低基数入口 |
|
||||
| `surface` | in-app | web / desktop / mobile |
|
||||
| `provider_mode` | no | 可从最近一次配置状态补齐 |
|
||||
| `description_length_bucket` | bug report | `empty` / `short` / `medium` / `long`，不要上传正文 |
|
||||
| `include_triage_context` | bug report | 是否附带页面上下文 |
|
||||
| `screenshot_attached` | bug report | 是否附带截图或录屏 |
|
||||
|
||||
推荐看板：
|
||||
|
||||
- Feedback users count。
|
||||
- Bug report trend by category。
|
||||
- 配置失败后 24h 内是否反馈。
|
||||
|
||||
社区反馈标签建议：
|
||||
|
||||
| Category | 适用反馈 |
|
||||
|---|---|
|
||||
| `performance` | 卡顿、慢、首 token 慢、内存 / CPU 异常 |
|
||||
| `provider_config` | Provider 配置复杂、保存失败、Key / Endpoint 不知道怎么填 |
|
||||
| `model_list` | 模型列表加载失败、模型不可选、模型名不符合预期 |
|
||||
| `crash` | 崩溃、白屏、不可恢复异常 |
|
||||
| `ui_ux` | UI 操作不顺、按钮找不到、信息表达不清 |
|
||||
| `voice_input` | 麦克风权限、录音、STT、语音输入失败 |
|
||||
| `tts` | TTS 音色、试听、Voice Pack、音色绑定问题 |
|
||||
| `payment` | Flux、余额不足、付费、checkout、扣费疑问 |
|
||||
| `chat_activation` | 新手无法开始聊天、首次发送失败、开箱即用失败 |
|
||||
| `live2d` | Live2D / 模型显示 / 角色舞台问题 |
|
||||
| `desktop_window` | 桌宠窗口、置顶、显示器、多屏问题 |
|
||||
| `mobile` | iOS / Android / 移动端体验 |
|
||||
| `unknown` | 信息不足，待社区负责人二次归类 |
|
||||
|
||||
社区侧现在可以这样做：
|
||||
|
||||
1. Discord / QQ 反馈先人工记录，不等 App 内入口完善。
|
||||
2. 每条反馈只需要填：日期、来源、用户类型、分类、严重程度、原文链接、简短摘要、是否已转 issue。
|
||||
3. 原文和截图留在 Discord / QQ / issue，不进 PostHog；PostHog 只放标签和计数。
|
||||
4. 每周把 `category` + `severity` 聚合进周报，用来解释为什么某个指标变差。
|
||||
|
||||
## P1 埋点
|
||||
|
||||
### Onboarding
|
||||
|
||||
当前 `onboarding_step_completed` 不足以定位用户卡点。
|
||||
|
||||
补充：
|
||||
|
||||
- `onboarding_started`
|
||||
- `onboarding_step_viewed`
|
||||
- `onboarding_step_completed`
|
||||
- `onboarding_skipped`
|
||||
- `onboarding_failed`
|
||||
|
||||
字段：
|
||||
|
||||
- `step`
|
||||
- `time_spent_ms`
|
||||
- `provider_mode`
|
||||
- `provider_id`
|
||||
- `error_code`
|
||||
|
||||
看板：
|
||||
|
||||
- Onboarding step drop-off。
|
||||
- 官方 Provider onboarding conversion。
|
||||
- Skip 后是否仍完成 `chat_activation_succeeded`。
|
||||
|
||||
### Chat Reliability
|
||||
|
||||
补充：
|
||||
|
||||
- `message_send_failed`
|
||||
- `assistant_response_failed`
|
||||
- `generation_cancelled`
|
||||
- `generation_retried`
|
||||
- `response_regenerated`
|
||||
|
||||
字段：
|
||||
|
||||
- `model_id`
|
||||
- `provider_id`
|
||||
- `provider_mode`
|
||||
- `has_voice`
|
||||
- `latency_ms`
|
||||
- `error_code`
|
||||
- `surface`
|
||||
|
||||
看板：
|
||||
|
||||
- Message send success rate。
|
||||
- Retry / cancel rate。
|
||||
- Failure by provider and model。
|
||||
|
||||
### Character And Role Usage
|
||||
|
||||
补充：
|
||||
|
||||
- `character_created`
|
||||
- `character_imported`
|
||||
- `character_edited`
|
||||
- `character_switched`
|
||||
- `character_deleted`
|
||||
- `display_model_changed`
|
||||
- `voice_pack_bound`
|
||||
|
||||
字段：
|
||||
|
||||
- `character_type`: `built_in` / `imported` / `custom`
|
||||
- `has_voice`
|
||||
- `voice_type`
|
||||
- `surface`
|
||||
|
||||
看板：
|
||||
|
||||
- Character adoption。
|
||||
- 角色用户与普通聊天用户 retention / payment 差异。
|
||||
|
||||
### Payment And Flux Friction
|
||||
|
||||
当前已有 `pricing_page_viewed`、`plan_selected`、`checkout_started`，服务端有 payment truth。补充：
|
||||
|
||||
- `flux_low_warning_shown`
|
||||
- `flux_topup_clicked`
|
||||
- `checkout_failed`
|
||||
- `payment_completed_imported`
|
||||
|
||||
字段：
|
||||
|
||||
- `surface`
|
||||
- `balance_state`
|
||||
- `plan_id`
|
||||
- `currency`
|
||||
- `error_code`
|
||||
|
||||
注意：
|
||||
|
||||
- `payment_completed` 真相仍在 Postgres / Stripe webhook。
|
||||
- PostHog 只用于漏斗展示，优先用 Stripe connector 或离线导入。
|
||||
|
||||
## 事件复用关系
|
||||
|
||||
新增埋点时先复用现有事件，不要把相同事实拆成多个名字。
|
||||
|
||||
| Existing / New | Relationship | Notes |
|
||||
|---|---|---|
|
||||
| `first_message_sent` | 保留历史指标 | 继续用于老 dashboard；新激活口径用 `chat_activation_succeeded` |
|
||||
| `chat_activation_started` / `chat_activation_succeeded` / `chat_activation_failed` | 新核心 activation 口径 | 用来回答“用户能不能正常开始聊天” |
|
||||
| `provider_card_clicked` | 保留入口点击 | 不等于配置成功；成功 / 失败看 `provider_config_succeeded` / `provider_config_failed` |
|
||||
| `first_model_selected` / `model_switched` | 保留模型选择行为 | 配置链路和模型列表健康看 `model_list_loaded` / `model_list_failed` |
|
||||
| `stt_started` / `stt_succeeded` / `stt_failed` | 保留 STT Provider 结果 | 权限和设备问题用新增 `microphone_*` / `audio_device_unavailable` 拆开 |
|
||||
| `tts_stop_clicked` | 保留用户停止行为 | 不代表音色选择；音色选择用 `voice_selected` |
|
||||
| `speech_requested` / `speech_succeeded` / `speech_failed` / `speech_blocked` | 服务端 TTS truth | 继续沿用 `feature = tts`，只补 metadata 字段 |
|
||||
| `pricing_page_viewed` / `plan_selected` / `checkout_started` | 保留付费漏斗前段 | 真正 payment completed 仍以 Stripe / Postgres 为准 |
|
||||
|
||||
## 数据卫生
|
||||
|
||||
### 不要把自由文本直接作为分析维度
|
||||
|
||||
PostHog 线上已经能看到 `model` / `model_id` 存在自由文本风险。后续新增字段必须遵循：
|
||||
|
||||
- Provider、model、voice 使用稳定 ID。
|
||||
- 自定义值不要直接 group-by。
|
||||
- 自定义模型传：
|
||||
- `provider_id = custom`
|
||||
- `model_family = custom`
|
||||
- `is_custom_model = true`
|
||||
- `custom_model_hash` 可选，必须单向 hash，不能还原原文。
|
||||
- 自定义 voice 传：
|
||||
- `voice_type = custom_configured`
|
||||
- `voice_id = custom`
|
||||
- `custom_voice_hash` 可选，必须单向 hash。
|
||||
- 错误字段传稳定 `error_code`，不要传 raw error message。
|
||||
|
||||
### 字段基数约束
|
||||
|
||||
| Field | Cardinality | Rule |
|
||||
|---|---|---|
|
||||
| `provider_id` | low | 白名单 |
|
||||
| `provider_mode` | low | enum |
|
||||
| `model_id` | medium | 官方 catalog 或归一化 ID |
|
||||
| `voice_id` | medium | 官方 catalog 或 `custom` |
|
||||
| `voice_pack_id` | medium | 只进 PostHog / Postgres，不进 Prometheus label |
|
||||
| `error_code` | low | enum |
|
||||
| `source` | low | enum |
|
||||
| `surface` | low | enum |
|
||||
|
||||
Prometheus label 不放 `user_id`、`session_id`、`voice_pack_id`、自定义模型名、自定义音色名。
|
||||
|
||||
### 不要做什么
|
||||
|
||||
- 不要把用户输入、聊天正文、角色 prompt、raw API key、raw Endpoint、raw model name、raw voice name 发到 PostHog / Grafana。
|
||||
- 不要把 raw error message 作为分析字段；统一映射成稳定 `error_code`。
|
||||
- 不要在 Prometheus label 里加入 `voice_id`、`voice_pack_id`、`session_id`、`user_id`、自定义模型名、自定义音色名。
|
||||
- 不要为了看一个 funnel 同时新增两个语义相同的事件；优先查上面的事件复用关系。
|
||||
- 不要在请求主链路里同步等待 PostHog 发送完成；失败不能影响用户聊天 / TTS。
|
||||
- 不要只看点击量判断用户意图；至少结合 success / failed / blocked 和社区反馈标签。
|
||||
|
||||
## 看板建议
|
||||
|
||||
### PostHog
|
||||
|
||||
新建或扩展 dashboard：`Onboarding and Activation`
|
||||
|
||||
- 新用户 activation funnel：
|
||||
- `app_loaded`
|
||||
- `chat_activation_started`
|
||||
- `provider_config_succeeded`
|
||||
- `model_list_loaded`
|
||||
- `chat_activation_succeeded`
|
||||
- Official vs custom Provider activation split。
|
||||
- Provider configuration failure ranking。
|
||||
- Onboarding step drop-off。
|
||||
|
||||
新建 dashboard：`Voice and TTS Adoption`
|
||||
|
||||
- Top voices by selected users。
|
||||
- Top voices by successful TTS requests。
|
||||
- Voice preview -> selection conversion。
|
||||
- Official default vs selected vs custom configured。
|
||||
- Voice input start -> STT success funnel。
|
||||
|
||||
扩展 `My App Dashboard`
|
||||
|
||||
- 保留现有 activation、paywall、LLM、rage-click。
|
||||
- 增加 `chat_activation_succeeded`、`provider_config_failed`、`stt_failed`、`voice_selected` 摘要卡。
|
||||
|
||||
### Grafana
|
||||
|
||||
扩展 `AIRI Server Overview`
|
||||
|
||||
- TTS request / success / failed / blocked by source。
|
||||
- TTS character total by model over range。
|
||||
- Product action failure rate by feature/action。
|
||||
|
||||
不要在 Prometheus 增加 `voice_id` label。若要看音色排行：
|
||||
|
||||
- Postgres `product_events.metadata.voice_id` 做 SQL / admin API 聚合。
|
||||
- 或离线导入 PostHog，用 PostHog group-by 展示。
|
||||
|
||||
## 播报、分析和可视化方案
|
||||
|
||||
### 分层原则
|
||||
|
||||
播报要分三层，不要把所有指标塞进一个 dashboard。
|
||||
|
||||
| Layer | Frequency | Audience | Goal | Tool |
|
||||
|---|---|---|---|---|
|
||||
| 实时异常 | 5m - 1h | 工程 / on-call / 社区负责人 | 发现“今天是不是坏了” | Grafana alert + PostHog insight alert |
|
||||
| 日报 | daily | 社区 / 产品 / 工程 | 看上手、聊天、语音、付费是否正常 | PostHog + Grafana + 少量 SQL |
|
||||
| 周报 | weekly | 产品 / 战略讨论 | 看趋势、用户意图、投入方向 | PostHog cohort/funnel + SQL 聚合 |
|
||||
|
||||
工具分工：
|
||||
|
||||
| Tool | 用途 | 不适合做什么 |
|
||||
|---|---|---|
|
||||
| PostHog | 用户路径、漏斗、留存、分群、前端行为 | 服务端真实扣费、低延迟 on-call |
|
||||
| Grafana | 服务端健康、TTS/LLM 请求、错误、余额阻断、告警 | 高基数用户行为和音色排行 |
|
||||
| Postgres / SQL | 付费事实、`product_events.metadata`、voice / voice pack 聚合 | 实时看板和复杂前端路径 |
|
||||
| Community tags | Discord / QQ 反馈归类 | 自动替代埋点 |
|
||||
|
||||
### 日报模板
|
||||
|
||||
日报回答“今天是否健康，有没有需要马上处理的问题”。
|
||||
|
||||
```md
|
||||
# AIRI 数据日报 YYYY-MM-DD
|
||||
|
||||
## 核心状态
|
||||
|
||||
- DAU: <value> (<day-over-day>)
|
||||
- New users: <value>
|
||||
- Chat activation: <chat_activation_succeeded / chat_activation_started>
|
||||
- Official provider activation: <value>
|
||||
- Custom provider activation: <value>
|
||||
- Paid conversion proxy: pricing -> plan -> checkout = <value>
|
||||
|
||||
## 上手和配置
|
||||
|
||||
- Top provider config failures:
|
||||
1. <provider_id> / <error_code> / <count>
|
||||
2. <provider_id> / <error_code> / <count>
|
||||
- Model list failure rate: <value>
|
||||
- New-user first-message median time: <value>
|
||||
|
||||
## 聊天与性能
|
||||
|
||||
- Message round success: <value>
|
||||
- LLM first-token p95: <value>
|
||||
- LLM / chat failures by provider:
|
||||
1. <provider_id> / <error_code> / <count>
|
||||
|
||||
## 语音与 TTS
|
||||
|
||||
- STT success rate: <value>
|
||||
- Microphone permission denied: <value>
|
||||
- TTS success / failed / blocked: <succeeded>/<failed>/<blocked>
|
||||
- Top voices selected:
|
||||
1. <voice_id> / <users>
|
||||
2. <voice_id> / <users>
|
||||
- Top voices used:
|
||||
1. <voice_id> / <successful_requests>
|
||||
2. <voice_id> / <successful_requests>
|
||||
|
||||
## 反馈与异常
|
||||
|
||||
- Bug reports: <value>
|
||||
- Feedback submitted: <value>
|
||||
- Discord / QQ 高关键词:
|
||||
- <category>: <count>
|
||||
- 异常提醒:
|
||||
- <alert_name>: <current> vs <baseline>, 建议动作 <action>
|
||||
```
|
||||
|
||||
第一版日报可以先不追求自动生成完整解释，只要自动填指标，并把异常规则命中的项放到“异常提醒”即可。
|
||||
|
||||
### 周报模板
|
||||
|
||||
周报回答“用户意图有什么变化，战略上要改什么”。
|
||||
|
||||
```md
|
||||
# AIRI 数据周报 YYYY-WW
|
||||
|
||||
## 本周结论
|
||||
|
||||
1. <最重要趋势，例如 official provider 激活率上升>
|
||||
2. <最大风险，例如 custom provider 配置失败仍高>
|
||||
3. <建议动作，例如下周优先修 model list failed>
|
||||
|
||||
## 新手上手
|
||||
|
||||
- New users: <value>
|
||||
- Activation funnel:
|
||||
- app_loaded -> chat_activation_started: <value>
|
||||
- chat_activation_started -> provider_config_succeeded: <value>
|
||||
- provider_config_succeeded -> chat_activation_succeeded: <value>
|
||||
- Official vs custom:
|
||||
- official activation: <value>
|
||||
- custom activation: <value>
|
||||
- conclusion: <which path is healthier>
|
||||
|
||||
## 用户意图
|
||||
|
||||
- Role chat users: <value>
|
||||
- Developer users: <value>
|
||||
- Voice users: <value>
|
||||
- Paid users: <value>
|
||||
- Feedback users: <value>
|
||||
- 海外用户占比: <value>
|
||||
|
||||
## 语音和音色
|
||||
|
||||
- Top selected voices: <voice_id list>
|
||||
- Top used voices: <voice_id list>
|
||||
- Default voice adoption: <value>
|
||||
- Custom configured voice adoption: <value>
|
||||
- Voice preview -> selected conversion: <value>
|
||||
- TTS blocked / failed trend: <value>
|
||||
|
||||
## 配置和 Bug
|
||||
|
||||
- Top failing providers: <list>
|
||||
- Top failing error codes: <list>
|
||||
- Rage-click pages / surfaces: <list>
|
||||
- Discord / QQ feedback categories:
|
||||
- performance: <count>
|
||||
- config: <count>
|
||||
- bug: <count>
|
||||
- voice_input: <count>
|
||||
|
||||
## 下周建议
|
||||
|
||||
- Product: <one action>
|
||||
- Engineering: <one action>
|
||||
- Community: <one action>
|
||||
```
|
||||
|
||||
周报需要人工写“结论”和“建议动作”。指标只负责提示方向，社区反馈负责解释为什么。
|
||||
|
||||
### 可视化布局
|
||||
|
||||
#### Executive Overview
|
||||
|
||||
给产品 / 社区快速看：
|
||||
|
||||
- Activation conversion
|
||||
- Official vs custom activation
|
||||
- Provider config failures
|
||||
- STT success rate
|
||||
- TTS success / blocked
|
||||
- Top voices selected / used
|
||||
- Bug reports / feedback
|
||||
- Paywall funnel
|
||||
|
||||
#### Onboarding And Activation
|
||||
|
||||
给负责上手体验的人看：
|
||||
|
||||
- Funnel：`app_loaded -> chat_activation_started -> provider_config_succeeded -> model_list_loaded -> chat_activation_succeeded`
|
||||
- Breakdown：`provider_mode`、`surface`、`region`
|
||||
- Table：Top `provider_config_failed` by `provider_id` / `error_code`
|
||||
- Timeseries：`time_to_first_message_ms` p50 / p95
|
||||
|
||||
#### Voice And TTS Adoption
|
||||
|
||||
给语音和角色体验看：
|
||||
|
||||
- Bar：Top voices by selected users
|
||||
- Bar：Top voices by successful TTS requests
|
||||
- Funnel：`voice_preview_played -> voice_selected -> speech_succeeded`
|
||||
- Timeseries：`speech_succeeded` / `speech_failed` / `speech_blocked`
|
||||
- Breakdown：`voice_type` = official default / official selected / custom configured / voice pack
|
||||
|
||||
#### Reliability And Friction
|
||||
|
||||
给工程和社区排障看：
|
||||
|
||||
- Grafana：5xx、LLM latency、provider failure、TTS blocked
|
||||
- PostHog：rage-click trend、failed frontend events
|
||||
- Table：Top error_code by surface / provider
|
||||
- Community tags：Discord / QQ 反馈分类趋势
|
||||
|
||||
### 分析方法
|
||||
|
||||
#### Official provider 是否降低门槛
|
||||
|
||||
看：
|
||||
|
||||
- `chat_activation_succeeded / chat_activation_started` by `provider_mode`
|
||||
- `time_to_first_message_ms` by `provider_mode`
|
||||
- `provider_config_failed` by `provider_mode`
|
||||
- D7 retention by `provider_mode`
|
||||
|
||||
如果 official 激活率高、耗时短、失败率低，说明开箱即用策略有效。若 official 使用率高但失败率也高，优先修官方 Provider 稳定性。
|
||||
|
||||
#### 自配置用户是不是更高价值
|
||||
|
||||
看：
|
||||
|
||||
- `custom_provider_user` 的 D7 / D30 retention
|
||||
- `custom_provider_user` 的 feedback rate
|
||||
- `custom_provider_user` 的 paid conversion
|
||||
- `custom_provider_user` 的 config failure rate
|
||||
|
||||
如果自配置用户付费和反馈更高，但失败率也高，可以把高级配置保留，但需要更好的错误提示和导入模板。
|
||||
|
||||
#### 哪些 Bug 最劝退
|
||||
|
||||
看：
|
||||
|
||||
- `chat_activation_failed` by `failure_stage`
|
||||
- `provider_config_failed` by `error_code`
|
||||
- `model_list_failed` by `provider_id`
|
||||
- `$rageclick` by page / surface
|
||||
- Discord / QQ `category = bug` 的高频词
|
||||
|
||||
日报只报异常；周报把异常和社区反馈合并成“优先修复建议”。
|
||||
|
||||
#### TTS 音色策略
|
||||
|
||||
看：
|
||||
|
||||
- `voice_selected` users by `voice_id`
|
||||
- `speech_succeeded` count by `voice_id`
|
||||
- `voice_preview_played -> voice_selected` conversion by `voice_id`
|
||||
- `speech_failed / speech_requested` by `voice_id`
|
||||
- retention / paid conversion by `voice_type`
|
||||
|
||||
用法：
|
||||
|
||||
- 选择多但使用少：可能试听不错，实际聊天不合适。
|
||||
- 使用多但失败高：优先修该音色或 Provider。
|
||||
- 默认音色使用高但切换少：默认可能足够好，也可能用户没发现入口，需要结合 `voice_preview_played` 看。
|
||||
- 自定义音色用户留存高：说明高阶用户重视声音个性化。
|
||||
|
||||
### 自动化路线
|
||||
|
||||
第一阶段：半自动日报。
|
||||
|
||||
- Grafana 提供服务端健康和 TTS/LLM 指标。
|
||||
- PostHog 提供 activation、provider、voice、STT、feedback 指标。
|
||||
- SQL 提供 voice / voice pack 聚合。
|
||||
- 由脚本生成 Markdown，发到 Discord / QQ / 飞书其中一个固定频道。
|
||||
|
||||
第二阶段：异常驱动播报。
|
||||
|
||||
- 每小时检查 activation、provider config、model list、STT、TTS blocked、bug report。
|
||||
- 只有超过阈值才发提醒。
|
||||
- 提醒里必须带“建议查看哪个 dashboard / query”。
|
||||
|
||||
第三阶段：周报带人工结论。
|
||||
|
||||
- 自动填指标和 Top lists。
|
||||
- 社区负责人补充 Discord / QQ 反馈解释。
|
||||
- 产品 / 工程共同确认下周行动。
|
||||
|
||||
## 异常播报
|
||||
|
||||
第一版日报 / 周报走“指标 + 异常提醒”。日报不要做复杂归因；周报允许加入人工结论。
|
||||
|
||||
### 每日必看指标
|
||||
|
||||
建议内容：
|
||||
|
||||
- DAU / WAU / MAU。
|
||||
- New users。
|
||||
- `chat_activation_succeeded` 转化率。
|
||||
- Official vs custom Provider activation conversion。
|
||||
- Top Provider config failures。
|
||||
- STT success rate / failure rate。
|
||||
- TTS success / failed / blocked。
|
||||
- Top voices selected / used。
|
||||
- Bug reports / feedback count。
|
||||
- Paywall funnel：pricing -> plan -> checkout。
|
||||
|
||||
### 异常规则
|
||||
|
||||
| Alert | Trigger | Action |
|
||||
|---|---|---|
|
||||
| Activation drop | 24h `chat_activation_succeeded / chat_activation_started` 环比下降 15% | 查 Provider config / model_list / LLM failures |
|
||||
| Official Provider regression | 官方 Provider `provider_config_failed` 连续 15 分钟非零或 24h 明显上升 | 优先排官方配置 |
|
||||
| Model list failure spike | 任一 Provider `model_list_failed` 1h 翻倍 | 查 Provider API / auth / CORS |
|
||||
| STT failure spike | `stt_failed / stt_started` 超过阈值 | 查权限、设备、Provider |
|
||||
| TTS blocked spike | `feature = tts` + `action = speech_blocked` 1h 翻倍 | 查 Flux、默认音色成本、余额提示 |
|
||||
| Bug report spike | `bug_report_submitted` 24h 翻倍 | 社区同步归类 |
|
||||
| Rage-click spike | `$rageclick` 7d trend 异常 | 结合页面和 session replay |
|
||||
|
||||
## 实施顺序
|
||||
|
||||
### 分阶段落地
|
||||
|
||||
| Phase | Scope | Owner | 产出 |
|
||||
|---|---|---|---|
|
||||
| Phase 0 | 字段归一化、事件复用确认、敏感字段拦截 | frontend / server | 公共 helper、事件字典、测试用例 |
|
||||
| Phase 1 | Chat activation、Provider / model 配置、TTS voice 字段 | frontend / server | 能回答“能不能开始聊天”和“哪个音色常用” |
|
||||
| Phase 2 | Voice input、feedback、community tags | frontend / community | 能定位语音输入和社区反馈高频问题 |
|
||||
| Phase 3 | PostHog / Grafana dashboard、半自动日报、异常提醒 | data / server / community | 每日播报和异常告警可用 |
|
||||
| Phase 4 | 周报、cohort、retention、付费 / 反馈关联分析 | product / community / data | 支持战略复盘和下周优先级 |
|
||||
|
||||
### 任务顺序
|
||||
|
||||
1. 先补字段归一化 helpers，尤其是 Provider / model / voice。
|
||||
2. 补 Chat Activation 三个事件。
|
||||
3. 补 Provider / model 配置成功失败事件。
|
||||
4. 补 TTS voice 选择、试听、Voice Pack 绑定事件。
|
||||
5. 服务端 `product_events.metadata` 补 TTS `voice_id`、`voice_type`、`voice_pack_id`。
|
||||
6. 补语音输入权限 / 设备事件。
|
||||
7. 扩 PostHog dashboard。
|
||||
8. 扩 Grafana dashboard，但不把 voice 放进 Prometheus label。
|
||||
9. 建半自动日报 Markdown：PostHog + Grafana + SQL 聚合，先发固定频道。
|
||||
10. 建异常检查：activation、provider config、model list、STT、TTS blocked、bug report。
|
||||
11. 建周报模板：自动填指标，社区负责人补充 Discord / QQ 反馈解释和下周建议。
|
||||
|
||||
### 当前接入状态（2026-06-30）
|
||||
|
||||
已接入代码：
|
||||
|
||||
- Chat activation：`chat_activation_started`、`chat_activation_succeeded`、`chat_activation_failed`。
|
||||
- Model list：`model_list_loaded`、`model_list_failed`。
|
||||
- Provider config：`provider_config_started`、`provider_config_succeeded`、`provider_config_failed`。
|
||||
- TTS voice：`tts_provider_selected`、`voice_selected`、`voice_preview_played`、`voice_pack_bound`。
|
||||
- TTS 服务端 metadata：REST / WS TTS `product_events.metadata` 已补 `voice_id`、`voice_type`、`voice_pack_id`。
|
||||
- Voice input：`voice_input_started`、`microphone_permission_requested`、`microphone_permission_denied`、`audio_device_unavailable`、`voice_input_cancelled`。
|
||||
- STT：保留 `stt_started`、`stt_succeeded`、`stt_failed`，并将失败码收敛到稳定枚举，避免上报 raw error。
|
||||
- Feedback：`feedback_submitted` / `bug_report_submitted` 的低基数字段与 analytics API 已定义；产品内反馈提交入口与服务端收件流程拆到单独 PR。
|
||||
- Grafana Dashboard：`Product Analytics` 行已补 TTS success、TTS failed / blocked、TTS event rate by source 面板，并保留 voice drilldown 在 Postgres metadata / PostHog，不进入 Prometheus labels。
|
||||
- Dashboard setup 文档：`product-analytics-dashboard-setup.md` 已补 PostHog insights、Grafana panels、PostHog / Grafana alert 配置建议。
|
||||
- 上线冒烟文档：`verifications/product-analytics-smoke.md` 已补 PostHog、Postgres、Grafana 三层验证步骤。
|
||||
|
||||
待接入或待产品确认：
|
||||
|
||||
- Discord / QQ 社区标签的数据入口，例如人工表格、bot 或 issue 同步。
|
||||
- PostHog Dashboard 需要在 PostHog 账号里按 setup 文档创建 insights / alerts。
|
||||
- Grafana Dashboard JSON 已更新，仍需部署 / import 到线上 Grafana。
|
||||
- 日报 / 周报自动拉取脚本与提醒频道。
|
||||
|
||||
## 验证清单
|
||||
|
||||
- 新用户完成一次官方 Provider 聊天后，PostHog 能看到 `chat_activation_succeeded`。
|
||||
- 自配置 Provider 失败时，PostHog 能按 `provider_id` + `error_code` 聚合。
|
||||
- 选择官方默认音色后，PostHog 能看到 `voice_selected` 且 `voice_type = official_default`。
|
||||
- 试听音色后，PostHog 能看到 `voice_preview_played`。
|
||||
- REST 和 WS TTS 成功后，Postgres `product_events.metadata` 能看到 `voice_id`。
|
||||
- Grafana 继续只按低基数字段聚合，不新增高基数 voice label。
|
||||
- 日报能输出：activation conversion、Top failing providers、STT success rate、TTS success/blocked、Top voices、feedback count。
|
||||
- 异常提醒命中时能带上：当前值、基线值、影响范围、建议查看的 dashboard / query。
|
||||
- 周报能输出：本周结论、用户意图变化、Discord / QQ 反馈分类、下周 Product / Engineering / Community 行动建议。
|
||||
@@ -0,0 +1,275 @@
|
||||
# Verification: Product Analytics Smoke Test
|
||||
|
||||
Status: **code-level instrumentation verified; live PostHog dashboard created; Grafana dashboard imported; alert setup pending**
|
||||
Owner: Community / Product Analytics
|
||||
Last updated: 2026-06-30
|
||||
Related:
|
||||
- [`product-analytics-instrumentation.md`](../product-analytics-instrumentation.md)
|
||||
- [`product-analytics-dashboard-setup.md`](../product-analytics-dashboard-setup.md)
|
||||
- [`airi-server-overview-cloud.json`](../../../otel/grafana/dashboards/airi-server-overview-cloud.json)
|
||||
|
||||
## 用户路径
|
||||
|
||||
- **场景**：验证新增埋点能回答“用户是否能正常开始聊天”“Provider 配置卡在哪里”“哪个 TTS 音色被选择 / 实际播放”“语音输入卡在哪里”“用户是否提交反馈”。
|
||||
- **预期**：PostHog 能看到前端 journey events；Postgres `product_events` 能看到服务端 TTS metadata；Grafana 能看到低基数 server-side product health。
|
||||
- **当前状态**：代码与 dashboard JSON 已验证；线上 PostHog dashboard 已创建；线上 Grafana `AIRI Server Overview - Product Analytics` (`ad8qbp5`) 已导入完整 Product Analytics row；alert 仍需人工配置。
|
||||
|
||||
## 已经由代码验证
|
||||
|
||||
| Area | Evidence |
|
||||
|---|---|
|
||||
| Frontend analytics API | `packages/stage-ui/src/composables/use-analytics.test.ts` 覆盖 activation、model list、provider config、voice selection、voice input、feedback event API |
|
||||
| Chat activation hooks | `packages/core-agent/src/runtime/chat-orchestrator-runtime.test.ts` 覆盖 activation started / succeeded / failed hook |
|
||||
| Voice input failures | `packages/stage-ui/src/composables/audio/audio-device.test.ts` 与 `packages/stage-ui/src/stores/modules/hearing.analytics.test.ts` 覆盖 permission / device / cancel / STT failed |
|
||||
| Server TTS metadata | `apps/server/src/routes/openai/v1/route.test.ts` 与 `apps/server/src/routes/audio-speech-ws/route.test.ts` 覆盖 REST / WS TTS `voice_id`、`voice_type`、`voice_pack_id` metadata |
|
||||
| Grafana product row | `apps/server/otel/grafana/dashboards/build.test.ts` 覆盖 Product Analytics panels、layout references、PromQL 不包含 high-cardinality voice / user fields |
|
||||
|
||||
## Live Smoke Checklist
|
||||
|
||||
Run this after deploying a build with the instrumentation changes. The PostHog dashboard and Grafana dashboard shell are already created, but they still need live event traffic from the deployed build.
|
||||
|
||||
### 1. PostHog: chat activation
|
||||
|
||||
Action:
|
||||
|
||||
1. Use a fresh or test account.
|
||||
2. Start with an official provider.
|
||||
3. Send the first chat message and wait for the assistant response.
|
||||
|
||||
Expected PostHog events:
|
||||
|
||||
```text
|
||||
chat_activation_started
|
||||
chat_activation_succeeded
|
||||
```
|
||||
|
||||
Required properties:
|
||||
|
||||
```text
|
||||
provider_mode = official
|
||||
provider_id = <official provider id>
|
||||
model_id = <selected model id>
|
||||
surface = web | mobile | electron
|
||||
```
|
||||
|
||||
Fail if:
|
||||
|
||||
- `chat_activation_started` appears but `chat_activation_succeeded` never appears for a successful chat.
|
||||
- `provider_mode` is missing or always `unknown`.
|
||||
- `surface` is missing.
|
||||
|
||||
### 2. PostHog: provider config failure
|
||||
|
||||
Action:
|
||||
|
||||
1. Configure a custom provider with an invalid key or invalid endpoint.
|
||||
2. Trigger settings validation or manual chat ping.
|
||||
|
||||
Expected PostHog events:
|
||||
|
||||
```text
|
||||
provider_config_started
|
||||
provider_config_failed
|
||||
```
|
||||
|
||||
Required properties:
|
||||
|
||||
```text
|
||||
provider_mode = custom
|
||||
provider_id = <provider id>
|
||||
step = settings_auto_validate | manual_chat_ping
|
||||
error_code = <bounded error code>
|
||||
```
|
||||
|
||||
Fail if:
|
||||
|
||||
- Raw error text, API key fragments, endpoint secrets, or stack traces appear in event properties.
|
||||
- `provider_config_failed` has no `error_code`.
|
||||
|
||||
### 3. PostHog: TTS voice selection
|
||||
|
||||
Action:
|
||||
|
||||
1. Open speech settings.
|
||||
2. Select an official TTS provider.
|
||||
3. Select or keep an official voice.
|
||||
4. Play voice preview once.
|
||||
|
||||
Expected PostHog events:
|
||||
|
||||
```text
|
||||
tts_provider_selected
|
||||
voice_selected
|
||||
voice_preview_played
|
||||
```
|
||||
|
||||
Required properties:
|
||||
|
||||
```text
|
||||
tts_provider_id = <provider id>
|
||||
tts_model_id = <model id>
|
||||
voice_id = <catalog voice id or custom>
|
||||
voice_type = official_default | official_selected | custom_configured | voice_pack | unknown
|
||||
source = settings | manual_preview
|
||||
```
|
||||
|
||||
Fail if:
|
||||
|
||||
- `voice_selected` is missing, because this blocks “哪个 TTS 音色比较多”的核心问题。
|
||||
- Official default voice is indistinguishable from custom configured voice.
|
||||
|
||||
### 4. PostHog: voice input friction
|
||||
|
||||
Action:
|
||||
|
||||
1. Start voice input.
|
||||
2. Test one failure path: deny microphone permission, use a browser/device with no microphone, or cancel input.
|
||||
|
||||
Expected PostHog events:
|
||||
|
||||
```text
|
||||
voice_input_started
|
||||
microphone_permission_requested
|
||||
microphone_permission_denied
|
||||
audio_device_unavailable
|
||||
voice_input_cancelled
|
||||
stt_failed
|
||||
```
|
||||
|
||||
Only the events that match the exercised path need to appear.
|
||||
|
||||
Fail if:
|
||||
|
||||
- Permission denied or device unavailable is only visible as a generic `stt_failed`.
|
||||
- `error_code` contains raw browser error text.
|
||||
|
||||
### 5. Postgres: server-side TTS metadata
|
||||
|
||||
Action:
|
||||
|
||||
1. Trigger one REST TTS request.
|
||||
2. Trigger one chat/WS TTS request if the deployed environment supports it.
|
||||
|
||||
Query:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
created_at,
|
||||
user_id,
|
||||
source,
|
||||
provider,
|
||||
model,
|
||||
action,
|
||||
status,
|
||||
metadata->>'voice_id' AS voice_id,
|
||||
metadata->>'voice_type' AS voice_type,
|
||||
metadata->>'voice_pack_id' AS voice_pack_id
|
||||
FROM product_events
|
||||
WHERE feature = 'tts'
|
||||
AND created_at >= now() - interval '1 hour'
|
||||
ORDER BY created_at DESC
|
||||
LIMIT 50;
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- `speech_requested` and `speech_succeeded` rows exist for successful TTS.
|
||||
- `voice_id` is present when the request provided a selected voice.
|
||||
- `voice_type` distinguishes official default / selected / custom / voice pack where available.
|
||||
|
||||
Fail if:
|
||||
|
||||
- TTS succeeds but no `product_events` row is written.
|
||||
- Metadata contains raw prompts, message text, API keys, or request bodies.
|
||||
|
||||
### 7. Grafana: Product Analytics row
|
||||
|
||||
Action:
|
||||
|
||||
1. Open `https://projairi.grafana.net/d/ad8qbp5/airi-server-overview`.
|
||||
2. Open the `Product Analytics` row.
|
||||
3. Use a 1h time range after running the smoke actions above.
|
||||
|
||||
Expected panels:
|
||||
|
||||
```text
|
||||
Product Events (range)
|
||||
Product Failure %
|
||||
TTS Success %
|
||||
TTS Failed / Blocked (range)
|
||||
Top Product Actions (range)
|
||||
Product Event Rate
|
||||
TTS Event Rate by Source
|
||||
```
|
||||
|
||||
PromQL sanity:
|
||||
|
||||
```promql
|
||||
sum(increase(airi_product_events_total{feature="tts"}[1h]))
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- Query returns a non-zero value after TTS smoke actions.
|
||||
- Legends only use bounded labels: `feature`, `action`, `status`, `source`.
|
||||
|
||||
Fail if:
|
||||
|
||||
- Prometheus labels contain `voice_id`, `voice_pack_id`, `user_id`, `session_id`, or `request_id`.
|
||||
- Grafana shows product panels but Postgres has no matching TTS rows.
|
||||
|
||||
## Quick Analysis Queries
|
||||
|
||||
Top selected voices should come from PostHog `voice_selected` for frontend intent. Server-side playback can be cross-checked from Postgres:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
metadata->>'voice_id' AS voice_id,
|
||||
metadata->>'voice_type' AS voice_type,
|
||||
provider,
|
||||
model,
|
||||
COUNT(*) AS play_count,
|
||||
COUNT(DISTINCT user_id) AS distinct_users
|
||||
FROM product_events
|
||||
WHERE feature = 'tts'
|
||||
AND action = 'speech_succeeded'
|
||||
AND created_at >= now() - interval '7 days'
|
||||
GROUP BY 1, 2, 3, 4
|
||||
ORDER BY play_count DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
Server-side TTS blocked / failed ranking:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
action,
|
||||
status,
|
||||
source,
|
||||
reason,
|
||||
COUNT(*) AS event_count,
|
||||
COUNT(DISTINCT user_id) AS distinct_users
|
||||
FROM product_events
|
||||
WHERE feature = 'tts'
|
||||
AND action IN ('speech_failed', 'speech_blocked')
|
||||
AND created_at >= now() - interval '24 hours'
|
||||
GROUP BY 1, 2, 3, 4
|
||||
ORDER BY event_count DESC;
|
||||
```
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
| Item | Pass condition |
|
||||
|---|---|
|
||||
| Activation | PostHog funnel shows `chat_activation_started -> chat_activation_succeeded` by `provider_mode` |
|
||||
| Provider config | Failed custom config emits `provider_config_failed` with bounded `error_code` |
|
||||
| TTS voice | PostHog can rank `voice_selected` by `voice_id`; Postgres can rank actual `speech_succeeded` by metadata voice |
|
||||
| Voice input | Permission / device / cancel paths are distinguishable |
|
||||
| Feedback | Feedback event API exists with bounded fields; product feedback UI/server submission is split into a separate PR |
|
||||
| Grafana | Product Analytics row renders and uses only bounded Prometheus labels |
|
||||
|
||||
## Known Pending Work
|
||||
|
||||
- PostHog dashboard and alerts still need to be created inside the PostHog account.
|
||||
- Updated Grafana JSON still needs to be imported or deployed to the production Grafana workspace.
|
||||
- Discord / QQ ingestion and daily / weekly automation scripts are intentionally excluded from this pass.
|
||||
@@ -1274,6 +1274,294 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"panel-99": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"data": {
|
||||
"kind": "QueryGroup",
|
||||
"spec": {
|
||||
"queries": [
|
||||
{
|
||||
"kind": "PanelQuery",
|
||||
"spec": {
|
||||
"hidden": false,
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "grafanacloud-projairi-prom"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"expr": "100 * sum(increase(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\", feature=\"tts\", action=\"speech_succeeded\", status=\"succeeded\"}[$__range])) / clamp_min(sum(increase(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\", feature=\"tts\", action=\"speech_requested\", status=\"started\"}[$__range])), 1)",
|
||||
"legendFormat": "success %"
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refId": "A"
|
||||
}
|
||||
}
|
||||
],
|
||||
"queryOptions": {},
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "Server-side TTS successes divided by TTS requests over the dashboard range. Includes REST and WS TTS product events. Drops here mean users are asking for speech but not receiving audio; inspect failed/blocked panels next.",
|
||||
"id": 99,
|
||||
"links": [],
|
||||
"title": "TTS Success %",
|
||||
"vizConfig": {
|
||||
"group": "gauge",
|
||||
"kind": "VizConfig",
|
||||
"spec": {
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "red",
|
||||
"value": 0
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 90
|
||||
},
|
||||
{
|
||||
"color": "green",
|
||||
"value": 98
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "percent",
|
||||
"decimals": 2,
|
||||
"noValue": "0",
|
||||
"min": 0,
|
||||
"max": 100
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"minVizHeight": 75,
|
||||
"minVizWidth": 75,
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showThresholdLabels": false,
|
||||
"showThresholdMarkers": true,
|
||||
"sizing": "auto"
|
||||
}
|
||||
},
|
||||
"version": "13.0.0-23630096546"
|
||||
}
|
||||
}
|
||||
},
|
||||
"panel-100": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"data": {
|
||||
"kind": "QueryGroup",
|
||||
"spec": {
|
||||
"queries": [
|
||||
{
|
||||
"kind": "PanelQuery",
|
||||
"spec": {
|
||||
"hidden": false,
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "grafanacloud-projairi-prom"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"expr": "topk(12, sum by (action, status, source) (increase(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\", feature=\"tts\", action=~\"speech_failed|speech_blocked\"}[$__range])))",
|
||||
"legendFormat": "{{action}} · {{status}} · {{source}}",
|
||||
"instant": true,
|
||||
"range": false
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refId": "A"
|
||||
}
|
||||
}
|
||||
],
|
||||
"queryOptions": {},
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "TTS user-impacting failures over the dashboard range, split by action/status/source. `speech_failed` usually means upstream/runtime failure; `speech_blocked` usually means balance/preflight blocked. Keep voice/model drilldown in Postgres metadata, not Prometheus labels.",
|
||||
"id": 100,
|
||||
"links": [],
|
||||
"title": "TTS Failed / Blocked (range)",
|
||||
"vizConfig": {
|
||||
"group": "bargauge",
|
||||
"kind": "VizConfig",
|
||||
"spec": {
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": 0
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short",
|
||||
"noValue": "0"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"displayMode": "gradient",
|
||||
"maxVizHeight": 300,
|
||||
"minVizHeight": 12,
|
||||
"minVizWidth": 8,
|
||||
"namePlacement": "auto",
|
||||
"orientation": "horizontal",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showUnfilled": true,
|
||||
"sizing": "auto",
|
||||
"valueMode": "color"
|
||||
}
|
||||
},
|
||||
"version": "13.0.0-23630096546"
|
||||
}
|
||||
}
|
||||
},
|
||||
"panel-101": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"data": {
|
||||
"kind": "QueryGroup",
|
||||
"spec": {
|
||||
"queries": [
|
||||
{
|
||||
"kind": "PanelQuery",
|
||||
"spec": {
|
||||
"hidden": false,
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "grafanacloud-projairi-prom"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"expr": "sum by (source, action, status) (rate(airi_product_events_total{service_name=~\"$service\", deployment_environment=~\"$env\", feature!=\"\", action!=\"\", feature=\"tts\"}[$__rate_interval]))",
|
||||
"legendFormat": "{{source}} · {{action}} · {{status}}"
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refId": "A"
|
||||
}
|
||||
}
|
||||
],
|
||||
"queryOptions": {},
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "TTS product event rate by source and action. Use this to distinguish chat auto-TTS, manual previews/settings tests, and API audio.speech traffic when speech health changes.",
|
||||
"id": 101,
|
||||
"links": [],
|
||||
"title": "TTS Event Rate by Source",
|
||||
"vizConfig": {
|
||||
"group": "timeseries",
|
||||
"kind": "VizConfig",
|
||||
"spec": {
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"barWidthFactor": 0.6,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 15,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 1,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "auto",
|
||||
"showValues": false,
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": 0
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "eps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"annotations": {
|
||||
"clustering": -1,
|
||||
"multiLane": false
|
||||
},
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"lastNotNull",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "right",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"hideZeros": false,
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
}
|
||||
},
|
||||
"version": "13.0.0-23630096546"
|
||||
}
|
||||
}
|
||||
},
|
||||
"panel-16": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
@@ -4217,14 +4505,40 @@
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-97"
|
||||
"name": "panel-99"
|
||||
},
|
||||
"height": 9,
|
||||
"width": 12,
|
||||
"height": 5,
|
||||
"width": 6,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "GridLayoutItem",
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-100"
|
||||
},
|
||||
"height": 5,
|
||||
"width": 6,
|
||||
"x": 18,
|
||||
"y": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "GridLayoutItem",
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-97"
|
||||
},
|
||||
"height": 8,
|
||||
"width": 12,
|
||||
"x": 0,
|
||||
"y": 5
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "GridLayoutItem",
|
||||
"spec": {
|
||||
@@ -4234,9 +4548,22 @@
|
||||
},
|
||||
"height": 4,
|
||||
"width": 12,
|
||||
"x": 0,
|
||||
"x": 12,
|
||||
"y": 5
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "GridLayoutItem",
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-101"
|
||||
},
|
||||
"height": 4,
|
||||
"width": 12,
|
||||
"x": 12,
|
||||
"y": 9
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { checkDashboardLayoutReferences, dashboard } from './build'
|
||||
|
||||
/**
|
||||
* Narrows unknown dashboard nodes into indexable records for assertions.
|
||||
*/
|
||||
function asRecord(value: unknown, label: string): Record<string, unknown> {
|
||||
if (!value || typeof value !== 'object')
|
||||
throw new TypeError(`${label} is not an object`)
|
||||
|
||||
return value as Record<string, unknown>
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads a generated panel title from the dashboard object.
|
||||
*/
|
||||
function panelTitle(panelName: string): string {
|
||||
const panel = asRecord(dashboard.elements[panelName], panelName)
|
||||
const spec = asRecord(panel.spec, `${panelName}.spec`)
|
||||
if (typeof spec.title !== 'string')
|
||||
throw new TypeError(`${panelName}.spec.title is not a string`)
|
||||
|
||||
return spec.title
|
||||
}
|
||||
|
||||
/**
|
||||
* Collects PromQL expression strings from nested Grafana panel objects.
|
||||
*/
|
||||
function collectQueryExpressions(value: unknown, expressions: string[] = []): string[] {
|
||||
if (!value || typeof value !== 'object')
|
||||
return expressions
|
||||
|
||||
const record = value as Record<string, unknown>
|
||||
if (typeof record.expr === 'string')
|
||||
expressions.push(record.expr)
|
||||
|
||||
for (const nestedValue of Object.values(record))
|
||||
collectQueryExpressions(nestedValue, expressions)
|
||||
|
||||
return expressions
|
||||
}
|
||||
|
||||
describe('grafana dashboard builder', () => {
|
||||
/**
|
||||
* @example
|
||||
* const result = checkDashboardLayoutReferences(dashboard)
|
||||
* expect(result.orphanRefs).toEqual([])
|
||||
*/
|
||||
it('keeps every generated panel connected to the row layout', () => {
|
||||
const result = checkDashboardLayoutReferences(dashboard)
|
||||
|
||||
expect(result.orphanRefs).toEqual([])
|
||||
expect(result.unusedElems).toEqual([])
|
||||
})
|
||||
|
||||
/**
|
||||
* @example
|
||||
* expect(panelTitle('panel-99')).toBe('TTS Success %')
|
||||
*/
|
||||
it('keeps the product analytics row focused on server-side TTS health', () => {
|
||||
expect(panelTitle('panel-95')).toBe('Product Events (range)')
|
||||
expect(panelTitle('panel-96')).toBe('Product Failure %')
|
||||
expect(panelTitle('panel-99')).toBe('TTS Success %')
|
||||
expect(panelTitle('panel-100')).toBe('TTS Failed / Blocked (range)')
|
||||
expect(panelTitle('panel-101')).toBe('TTS Event Rate by Source')
|
||||
})
|
||||
|
||||
/**
|
||||
* @example
|
||||
* const rendered = JSON.stringify(dashboard.elements['panel-101'])
|
||||
* expect(rendered).not.toContain('voice_id')
|
||||
*/
|
||||
it('keeps high-cardinality voice fields out of Prometheus queries', () => {
|
||||
const productPanelExpressions = collectQueryExpressions([
|
||||
dashboard.elements['panel-95'],
|
||||
dashboard.elements['panel-96'],
|
||||
dashboard.elements['panel-97'],
|
||||
dashboard.elements['panel-98'],
|
||||
dashboard.elements['panel-99'],
|
||||
dashboard.elements['panel-100'],
|
||||
dashboard.elements['panel-101'],
|
||||
]).join('\n')
|
||||
|
||||
expect(productPanelExpressions).not.toContain('voice_id')
|
||||
expect(productPanelExpressions).not.toContain('voice_pack_id')
|
||||
expect(productPanelExpressions).not.toContain('user_id')
|
||||
expect(productPanelExpressions).not.toContain('session_id')
|
||||
expect(productPanelExpressions).not.toContain('request_id')
|
||||
})
|
||||
})
|
||||
@@ -29,8 +29,8 @@
|
||||
|
||||
import { writeFileSync } from 'node:fs'
|
||||
import { dirname, join } from 'node:path'
|
||||
import { exit } from 'node:process'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { argv, exit } from 'node:process'
|
||||
import { fileURLToPath, pathToFileURL } from 'node:url'
|
||||
|
||||
const PROM = { name: 'grafanacloud-projairi-prom' }
|
||||
const LOKI = { name: 'grafanacloud-projairi-logs' }
|
||||
@@ -527,7 +527,7 @@ elements['panel-92'] = timeseriesPanel(
|
||||
{ unit: 'short', fillOpacity: 30 },
|
||||
)
|
||||
|
||||
// --- Product Analytics — event-volume view only ---------------------------
|
||||
// --- Product Analytics — event volume + server-side TTS health -------------
|
||||
// Prometheus deliberately does not carry user_id. These panels answer
|
||||
// "which product actions are happening and failing"; DB-side product_events
|
||||
// queries answer "how many distinct users used each feature".
|
||||
@@ -577,6 +577,42 @@ elements['panel-98'] = timeseriesPanel(
|
||||
{ unit: 'eps', fillOpacity: 15 },
|
||||
)
|
||||
|
||||
elements['panel-99'] = gaugePanel(
|
||||
99,
|
||||
'TTS Success %',
|
||||
'Server-side TTS successes divided by TTS requests over the dashboard range. Includes REST and WS TTS product events. Drops here mean users are asking for speech but not receiving audio; inspect failed/blocked panels next.',
|
||||
[query(
|
||||
`100 * sum(increase(airi_product_events_total{${PRODUCT_EVENT_FILTER}, feature="tts", action="speech_succeeded", status="succeeded"}[$__range])) / clamp_min(sum(increase(airi_product_events_total{${PRODUCT_EVENT_FILTER}, feature="tts", action="speech_requested", status="started"}[$__range])), 1)`,
|
||||
'success %',
|
||||
)],
|
||||
{ steps: [{ color: 'red', value: 0 }, { color: 'yellow', value: 90 }, { color: 'green', value: 98 }], max: 100, decimals: 2, noValue: '0' },
|
||||
)
|
||||
|
||||
elements['panel-100'] = barGaugePanel(
|
||||
100,
|
||||
'TTS Failed / Blocked (range)',
|
||||
'TTS user-impacting failures over the dashboard range, split by action/status/source. `speech_failed` usually means upstream/runtime failure; `speech_blocked` usually means balance/preflight blocked. Keep voice/model drilldown in Postgres metadata, not Prometheus labels.',
|
||||
[query(
|
||||
`topk(12, sum by (action, status, source) (increase(airi_product_events_total{${PRODUCT_EVENT_FILTER}, feature="tts", action=~"speech_failed|speech_blocked"}[$__range])))`,
|
||||
'{{action}} · {{status}} · {{source}}',
|
||||
'A',
|
||||
PROM,
|
||||
{ instant: true },
|
||||
)],
|
||||
{ unit: 'short', noValue: '0' },
|
||||
)
|
||||
|
||||
elements['panel-101'] = timeseriesPanel(
|
||||
101,
|
||||
'TTS Event Rate by Source',
|
||||
'TTS product event rate by source and action. Use this to distinguish chat auto-TTS, manual previews/settings tests, and API audio.speech traffic when speech health changes.',
|
||||
[query(
|
||||
`sum by (source, action, status) (rate(airi_product_events_total{${PRODUCT_EVENT_FILTER}, feature="tts"}[$__rate_interval]))`,
|
||||
'{{source}} · {{action}} · {{status}}',
|
||||
)],
|
||||
{ unit: 'eps', fillOpacity: 15 },
|
||||
)
|
||||
|
||||
// --- Row 2: HTTP — traffic ranking, error trend, latency trend -------------
|
||||
elements['panel-16'] = barGaugePanel(
|
||||
16,
|
||||
@@ -908,14 +944,17 @@ const rows = [
|
||||
item('panel-81', 8, 0, 8, 4),
|
||||
item('panel-82', 16, 0, 8, 4),
|
||||
]),
|
||||
// Row 3: Product Analytics — Prom-safe event volume and failure trend.
|
||||
// Row 3: Product Analytics — Prom-safe event volume and server TTS health.
|
||||
// Distinct-user analytics stay in Postgres `product_events`; this row
|
||||
// intentionally never uses user_id/session/request labels.
|
||||
row('Product Analytics', [
|
||||
item('panel-95', 0, 0, 6, 5),
|
||||
item('panel-96', 6, 0, 6, 5),
|
||||
item('panel-97', 12, 0, 12, 9),
|
||||
item('panel-98', 0, 5, 12, 4),
|
||||
item('panel-99', 12, 0, 6, 5),
|
||||
item('panel-100', 18, 0, 6, 5),
|
||||
item('panel-97', 0, 5, 12, 8),
|
||||
item('panel-98', 12, 5, 12, 4),
|
||||
item('panel-101', 12, 9, 12, 4),
|
||||
]),
|
||||
// Row 3: HTTP — full-width error breakdown on top, then traffic ranking +
|
||||
// latency trend side by side.
|
||||
@@ -1039,7 +1078,7 @@ const variables = [
|
||||
* 1. Service Health — signup/sessions/WS counts, req-rate, 5xx, status-code
|
||||
* heatmap, live WS trend: "is anything broken right now?"
|
||||
* 2. User Engagement — rolling DAU/WAU/MAU from user.last_seen_at
|
||||
* 3. Product Analytics — Prom-safe product event volume + failure trend
|
||||
* 3. Product Analytics — Prom-safe product event volume + server TTS health
|
||||
* 4. HTTP — error breakdown by route, request ranking, latency by route
|
||||
* 5. LLM Gateway — per-model request rate + latency (TTFB + end-to-end)
|
||||
* 6. Provider Upstreams — per-provider rate/latency/failure + TTS chars
|
||||
@@ -1056,7 +1095,7 @@ const variables = [
|
||||
* Variables source from `target_info` (always present, no business-metric
|
||||
* dependency) so the dashboard never goes blank when an app metric is renamed.
|
||||
*/
|
||||
const dashboard = {
|
||||
export const dashboard = {
|
||||
annotations: [
|
||||
{
|
||||
kind: 'AnnotationQuery',
|
||||
@@ -1097,28 +1136,65 @@ const dashboard = {
|
||||
variables,
|
||||
}
|
||||
|
||||
const here = dirname(fileURLToPath(import.meta.url))
|
||||
const outPath = join(here, 'airi-server-overview-cloud.json')
|
||||
writeFileSync(outPath, `${JSON.stringify(dashboard, null, 2)}\n`)
|
||||
console.info(`wrote ${outPath}`)
|
||||
export interface DashboardLayoutCheckResult {
|
||||
orphanRefs: string[]
|
||||
unusedElems: string[]
|
||||
}
|
||||
|
||||
// Cross-check elements ↔ layout references
|
||||
const elementNames = new Set(Object.keys(dashboard.elements))
|
||||
const refs = new Set<string>()
|
||||
function walk(o: unknown): void {
|
||||
if (!o || typeof o !== 'object')
|
||||
/**
|
||||
* Validates that every dashboard layout reference points to a defined element.
|
||||
*
|
||||
* Use when:
|
||||
* - Regenerating the Grafana JSON from this dashboard builder.
|
||||
* - Testing that row changes did not orphan panels or leave panels unused.
|
||||
*
|
||||
* Expects:
|
||||
* - A Grafana dashboard object shaped like {@link dashboard}.
|
||||
*
|
||||
* Returns:
|
||||
* - Orphan layout references and unused element names.
|
||||
*/
|
||||
export function checkDashboardLayoutReferences(targetDashboard: typeof dashboard): DashboardLayoutCheckResult {
|
||||
const elementNames = new Set(Object.keys(targetDashboard.elements))
|
||||
const refs = new Set<string>()
|
||||
collectElementReferences(targetDashboard.layout, refs)
|
||||
return {
|
||||
orphanRefs: [...refs].filter(r => !elementNames.has(r)),
|
||||
unusedElems: [...elementNames].filter(e => !refs.has(e)),
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Recursively collects Grafana row element references from the layout tree.
|
||||
*/
|
||||
function collectElementReferences(node: unknown, refs: Set<string>): void {
|
||||
if (!node || typeof node !== 'object')
|
||||
return
|
||||
const node = o as { kind?: unknown, name?: unknown }
|
||||
if (node.kind === 'ElementReference' && typeof node.name === 'string')
|
||||
refs.add(node.name)
|
||||
for (const v of Object.values(o)) walk(v)
|
||||
const layoutNode = node as { kind?: unknown, name?: unknown }
|
||||
if (layoutNode.kind === 'ElementReference' && typeof layoutNode.name === 'string')
|
||||
refs.add(layoutNode.name)
|
||||
for (const value of Object.values(node)) collectElementReferences(value, refs)
|
||||
}
|
||||
walk(dashboard.layout)
|
||||
const orphanRefs = [...refs].filter(r => !elementNames.has(r))
|
||||
const unusedElems = [...elementNames].filter(e => !refs.has(e))
|
||||
console.info(`panels defined: ${elementNames.size}, referenced: ${refs.size}, orphans: ${orphanRefs.length}, unused: ${unusedElems.length}`)
|
||||
if (orphanRefs.length || unusedElems.length) {
|
||||
console.error('orphans:', orphanRefs)
|
||||
console.error('unused:', unusedElems)
|
||||
exit(1)
|
||||
|
||||
/**
|
||||
* Writes the generated dashboard JSON and fails the CLI on layout drift.
|
||||
*/
|
||||
function writeDashboard(): void {
|
||||
const here = dirname(fileURLToPath(import.meta.url))
|
||||
const outPath = join(here, 'airi-server-overview-cloud.json')
|
||||
writeFileSync(outPath, `${JSON.stringify(dashboard, null, 2)}\n`)
|
||||
console.info(`wrote ${outPath}`)
|
||||
|
||||
const { orphanRefs, unusedElems } = checkDashboardLayoutReferences(dashboard)
|
||||
const definedCount = Object.keys(dashboard.elements).length
|
||||
const referencedCount = definedCount - unusedElems.length + orphanRefs.length
|
||||
console.info(`panels defined: ${definedCount}, referenced: ${referencedCount}, orphans: ${orphanRefs.length}, unused: ${unusedElems.length}`)
|
||||
if (orphanRefs.length || unusedElems.length) {
|
||||
console.error('orphans:', orphanRefs)
|
||||
console.error('unused:', unusedElems)
|
||||
exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(argv[1] ?? '').href)
|
||||
writeDashboard()
|
||||
|
||||
@@ -4,6 +4,7 @@ import type { AuthInstance } from './libs/auth'
|
||||
import type { Database } from './libs/db'
|
||||
import type { Env } from './libs/env'
|
||||
import type { OtelInstance } from './otel'
|
||||
import type { StreamingTtsVoiceType } from './routes/audio-speech-ws/session'
|
||||
import type { ConfigKVService } from './services/adapters/config-kv'
|
||||
import type { AdminFluxGrantsService } from './services/domain/admin/flux-grants'
|
||||
import type { AdminRouterConfigService } from './services/domain/admin/router-config'
|
||||
@@ -213,6 +214,7 @@ export async function buildApp(deps: AppDeps) {
|
||||
return audioSpeechWsSetup(session.user.id, {
|
||||
trigger: c.req.query('tts_trigger') === 'auto' ? 'auto' : 'manual',
|
||||
source: parseTtsSource(c.req.query('tts_source'), 'audio.speech.ws'),
|
||||
voiceType: parseTtsVoiceType(c.req.query('tts_voice_type')),
|
||||
})
|
||||
}))
|
||||
|
||||
@@ -458,6 +460,23 @@ function parseTtsSource(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalizes the client-provided streaming TTS voice bucket for product events.
|
||||
*/
|
||||
function parseTtsVoiceType(
|
||||
value: string | undefined,
|
||||
): StreamingTtsVoiceType {
|
||||
switch (value) {
|
||||
case 'official_default':
|
||||
case 'official_selected':
|
||||
case 'custom_configured':
|
||||
case 'voice_pack':
|
||||
return value
|
||||
default:
|
||||
return 'unknown'
|
||||
}
|
||||
}
|
||||
|
||||
export type AppType = Awaited<ReturnType<typeof buildApp>>['app']
|
||||
|
||||
export async function createApp() {
|
||||
|
||||
@@ -230,7 +230,7 @@ describe('audio-speech-ws route', () => {
|
||||
|
||||
const deps = makeFakeDeps({ upstreamURL: upstream.url, fluxBalance: 100 })
|
||||
const handlers = createAudioSpeechWsHandlers(deps as any)
|
||||
const events = handlers('user-123')
|
||||
const events = handlers('user-123', { voiceType: 'official_selected' })
|
||||
const client = makeMockClientWs()
|
||||
|
||||
await driveClientSession(events, client, [
|
||||
@@ -279,6 +279,17 @@ describe('audio-speech-ws route', () => {
|
||||
status: 200,
|
||||
fluxConsumed: 1,
|
||||
})
|
||||
expect(deps.productEventService.track).toHaveBeenCalledWith(expect.objectContaining({
|
||||
userId: 'user-123',
|
||||
feature: 'tts',
|
||||
action: 'speech_succeeded',
|
||||
status: 'succeeded',
|
||||
model: 'volcengine/seed-tts-2.0',
|
||||
metadata: expect.objectContaining({
|
||||
voice_id: 'mock',
|
||||
voice_type: 'official_selected',
|
||||
}),
|
||||
}))
|
||||
})
|
||||
|
||||
it('refuses the session with insufficient_flux when the user is broke', async () => {
|
||||
|
||||
@@ -52,10 +52,12 @@ export interface AudioSpeechSessionState {
|
||||
|
||||
export type StreamingTtsTrigger = 'auto' | 'manual'
|
||||
export type StreamingTtsSource = 'audio.speech.ws' | 'chat_auto_tts' | 'manual_preview' | 'settings_test'
|
||||
export type StreamingTtsVoiceType = 'official_default' | 'official_selected' | 'custom_configured' | 'voice_pack' | 'unknown'
|
||||
|
||||
export interface AudioSpeechSessionAnalytics {
|
||||
trigger?: StreamingTtsTrigger
|
||||
source?: StreamingTtsSource
|
||||
voiceType?: StreamingTtsVoiceType
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -93,6 +95,7 @@ export function createSessionState(
|
||||
let billed = false
|
||||
let totalInputChars = 0
|
||||
let modelLabel = STREAM_MODEL_LABEL_FALLBACK
|
||||
let voiceLabel: string | undefined
|
||||
/**
|
||||
* Frames the client sent before the upstream finished dialing. Buffered to
|
||||
* avoid silently dropping the `start` frame; flushed in arrival order once
|
||||
@@ -114,6 +117,7 @@ export function createSessionState(
|
||||
model: modelLabel,
|
||||
metadata: {
|
||||
trigger: analytics.trigger,
|
||||
...streamingVoiceMetadata(voiceLabel, analytics.voiceType),
|
||||
},
|
||||
})
|
||||
|
||||
@@ -227,6 +231,7 @@ export function createSessionState(
|
||||
metadata: {
|
||||
duration_ms: Date.now() - startedAt,
|
||||
trigger: analytics.trigger,
|
||||
...streamingVoiceMetadata(voiceLabel, analytics.voiceType),
|
||||
},
|
||||
})
|
||||
try {
|
||||
@@ -361,6 +366,9 @@ export function createSessionState(
|
||||
const model = (parsed as Record<string, unknown>).model
|
||||
if (typeof model === 'string' && model.length > 0)
|
||||
modelLabel = model
|
||||
const voice = (parsed as Record<string, unknown>).voice
|
||||
if (typeof voice === 'string' && voice.length > 0)
|
||||
voiceLabel = voice
|
||||
}
|
||||
}
|
||||
catch {
|
||||
@@ -433,6 +441,7 @@ export function createSessionState(
|
||||
duration_ms: durationMs,
|
||||
flux_consumed: fluxConsumed,
|
||||
trigger: analytics.trigger,
|
||||
...streamingVoiceMetadata(voiceLabel, analytics.voiceType),
|
||||
},
|
||||
})
|
||||
|
||||
@@ -470,6 +479,7 @@ export function createSessionState(
|
||||
close_code: code,
|
||||
duration_ms: Date.now() - startedAt,
|
||||
trigger: analytics.trigger,
|
||||
...streamingVoiceMetadata(voiceLabel, analytics.voiceType),
|
||||
},
|
||||
})
|
||||
if (clientWs) {
|
||||
@@ -503,6 +513,7 @@ export function createSessionState(
|
||||
close_code: code,
|
||||
duration_ms: Date.now() - startedAt,
|
||||
trigger: analytics.trigger,
|
||||
...streamingVoiceMetadata(voiceLabel, analytics.voiceType),
|
||||
},
|
||||
})
|
||||
if (clientWs) {
|
||||
@@ -531,6 +542,7 @@ function normalizeAnalytics(input: AudioSpeechSessionAnalytics): Required<AudioS
|
||||
return {
|
||||
trigger: normalizeTrigger(input.trigger),
|
||||
source: normalizeSource(input.source),
|
||||
voiceType: normalizeVoiceType(input.voiceType),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -550,6 +562,31 @@ function normalizeSource(source: AudioSpeechSessionAnalytics['source']): Streami
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalizes streaming TTS voice type into bounded analytics values.
|
||||
*/
|
||||
function normalizeVoiceType(voiceType: AudioSpeechSessionAnalytics['voiceType']): StreamingTtsVoiceType {
|
||||
switch (voiceType) {
|
||||
case 'official_default':
|
||||
case 'official_selected':
|
||||
case 'custom_configured':
|
||||
case 'voice_pack':
|
||||
return voiceType
|
||||
default:
|
||||
return 'unknown'
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds reusable streaming TTS voice metadata after the start frame is known.
|
||||
*/
|
||||
function streamingVoiceMetadata(voiceId: string | undefined, voiceType: StreamingTtsVoiceType): Record<string, unknown> {
|
||||
return {
|
||||
...(voiceId ? { voice_id: voiceId } : {}),
|
||||
voice_type: voiceType,
|
||||
}
|
||||
}
|
||||
|
||||
function isPaymentRequiredError(err: unknown): boolean {
|
||||
if (err instanceof ApiError)
|
||||
return err.statusCode === 402
|
||||
|
||||
@@ -849,6 +849,87 @@ describe('v1CompletionsRoutes', () => {
|
||||
}))
|
||||
})
|
||||
|
||||
/**
|
||||
* @example
|
||||
* POST /api/v1/audio/speech { "voice": "alloy", "extra_body": { "voice_pack": { "pack_id": "vp-premium" } } }
|
||||
*/
|
||||
it('records TTS voice and Voice Pack metadata in product events', async () => {
|
||||
globalThis.fetch = vi.fn(async () => new Response(new Uint8Array([1]), {
|
||||
status: 200,
|
||||
headers: { 'Content-Type': 'audio/mpeg' },
|
||||
}))
|
||||
|
||||
const productEventService = createMockProductEventService()
|
||||
const voicePackService = createMockVoicePackService({
|
||||
findById: vi.fn(async () => ({
|
||||
id: 'vp-premium',
|
||||
name: 'Premium',
|
||||
description: null,
|
||||
provider: 'azure',
|
||||
model: 'microsoft/v1',
|
||||
voiceId: 'alloy',
|
||||
ttsModelId: 'tts-1',
|
||||
params: {},
|
||||
costMultiplier: 2,
|
||||
enabled: true,
|
||||
createdAt: new Date(),
|
||||
updatedAt: new Date(),
|
||||
})),
|
||||
})
|
||||
const app = createTestApp(
|
||||
createMockFluxService(),
|
||||
createMockConfigKV(),
|
||||
undefined,
|
||||
undefined,
|
||||
undefined,
|
||||
undefined,
|
||||
createMockLlmTracing(),
|
||||
productEventService,
|
||||
voicePackService,
|
||||
)
|
||||
|
||||
await app.fetch(
|
||||
new Request('http://localhost/api/v1/audio/speech', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
model: 'auto',
|
||||
input: 'hello',
|
||||
voice: 'alloy',
|
||||
extra_body: {
|
||||
voice_pack: {
|
||||
pack_id: 'vp-premium',
|
||||
cost_multiplier: 2,
|
||||
},
|
||||
airi_analytics: {
|
||||
source: 'manual_preview',
|
||||
voice_type: 'voice_pack',
|
||||
},
|
||||
},
|
||||
}),
|
||||
}),
|
||||
{ user: testUser } as any,
|
||||
)
|
||||
|
||||
expect(productEventService.track).toHaveBeenCalledWith(expect.objectContaining({
|
||||
action: 'speech_succeeded',
|
||||
source: 'manual_preview',
|
||||
metadata: expect.objectContaining({
|
||||
voice_id: 'alloy',
|
||||
voice_type: 'voice_pack',
|
||||
voice_pack_id: 'vp-premium',
|
||||
}),
|
||||
}))
|
||||
expect(productEventService.track).toHaveBeenCalledWith(expect.objectContaining({
|
||||
action: 'speech_requested',
|
||||
metadata: expect.objectContaining({
|
||||
voice_id: 'alloy',
|
||||
voice_type: 'voice_pack',
|
||||
voice_pack_id: 'vp-premium',
|
||||
}),
|
||||
}))
|
||||
})
|
||||
|
||||
it('should not charge when routeTts upstream returns error', async () => {
|
||||
const llmRouter = createMockLlmRouter({
|
||||
routeTts: vi.fn(async () => new Response('{"error":"service down"}', {
|
||||
|
||||
@@ -63,10 +63,12 @@ export interface OpenAiSpeechRequest {
|
||||
}
|
||||
|
||||
type TtsTrigger = 'auto' | 'manual'
|
||||
type TtsVoiceType = 'official_default' | 'official_selected' | 'custom_configured' | 'voice_pack' | 'unknown'
|
||||
|
||||
interface TtsAnalyticsContext {
|
||||
trigger: TtsTrigger
|
||||
source: 'audio.speech' | 'chat_auto_tts' | 'manual_preview' | 'settings_test'
|
||||
voiceType: TtsVoiceType
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -100,6 +102,11 @@ export function createOpenAiSpeechService(deps: OpenAiSpeechServiceDeps) {
|
||||
voice: typeof input.body.voice === 'string' ? input.body.voice : undefined,
|
||||
voicePackService: deps.voicePackService,
|
||||
})
|
||||
const voiceMetadata = ttsVoiceMetadata({
|
||||
voice: typeof input.body.voice === 'string' ? input.body.voice : undefined,
|
||||
voicePackId: voicePackRequest.voicePackId,
|
||||
voiceType: analytics.voiceType,
|
||||
})
|
||||
const billingUnits = Math.ceil(inputText.length * voicePackRequest.costMultiplier)
|
||||
|
||||
logger.withFields({
|
||||
@@ -120,6 +127,7 @@ export function createOpenAiSpeechService(deps: OpenAiSpeechServiceDeps) {
|
||||
metadata: {
|
||||
input_chars: inputText.length,
|
||||
trigger: analytics.trigger,
|
||||
...voiceMetadata,
|
||||
},
|
||||
})
|
||||
|
||||
@@ -144,6 +152,7 @@ export function createOpenAiSpeechService(deps: OpenAiSpeechServiceDeps) {
|
||||
billing_units: billingUnits,
|
||||
balance_state: 'insufficient',
|
||||
trigger: analytics.trigger,
|
||||
...voiceMetadata,
|
||||
},
|
||||
})
|
||||
logger.withError(err).withFields({
|
||||
@@ -219,6 +228,7 @@ export function createOpenAiSpeechService(deps: OpenAiSpeechServiceDeps) {
|
||||
http_status: failure.status,
|
||||
duration_ms: Date.now() - startedAt,
|
||||
trigger: analytics.trigger,
|
||||
...voiceMetadata,
|
||||
},
|
||||
})
|
||||
throw err
|
||||
@@ -245,6 +255,7 @@ export function createOpenAiSpeechService(deps: OpenAiSpeechServiceDeps) {
|
||||
http_status: response.status,
|
||||
duration_ms: durationMs,
|
||||
trigger: analytics.trigger,
|
||||
...voiceMetadata,
|
||||
},
|
||||
})
|
||||
logger.withFields({ requestId, userId: input.userId, model: requestModel, status: response.status, durationMs })
|
||||
@@ -297,6 +308,7 @@ export function createOpenAiSpeechService(deps: OpenAiSpeechServiceDeps) {
|
||||
duration_ms: durationMs,
|
||||
flux_consumed: fluxConsumed,
|
||||
trigger: analytics.trigger,
|
||||
...voiceMetadata,
|
||||
},
|
||||
})
|
||||
deps.requestLogService.logRequest({
|
||||
@@ -354,8 +366,40 @@ function ttsAnalyticsContext(body: Record<string, unknown>): TtsAnalyticsContext
|
||||
|| rawSource === 'settings_test'
|
||||
? rawSource
|
||||
: 'audio.speech'
|
||||
const voiceType = normalizeVoiceType(analytics?.voice_type)
|
||||
|
||||
return { trigger, source }
|
||||
return { trigger, source, voiceType }
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalizes client-provided TTS voice type into bounded analytics values.
|
||||
*/
|
||||
function normalizeVoiceType(value: unknown): TtsVoiceType {
|
||||
switch (value) {
|
||||
case 'official_default':
|
||||
case 'official_selected':
|
||||
case 'custom_configured':
|
||||
case 'voice_pack':
|
||||
return value
|
||||
default:
|
||||
return 'unknown'
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds reusable low-cardinality voice metadata for every TTS product event.
|
||||
*/
|
||||
function ttsVoiceMetadata(input: {
|
||||
voice?: string
|
||||
voicePackId?: string
|
||||
voiceType: TtsVoiceType
|
||||
}): Record<string, unknown> {
|
||||
const voiceType = input.voicePackId ? 'voice_pack' : input.voiceType
|
||||
return {
|
||||
...(input.voice ? { voice_id: input.voice } : {}),
|
||||
voice_type: voiceType,
|
||||
...(input.voicePackId ? { voice_pack_id: input.voicePackId } : {}),
|
||||
}
|
||||
}
|
||||
|
||||
async function voicePackRequestOptions(
|
||||
@@ -365,12 +409,15 @@ async function voicePackRequestOptions(
|
||||
voice?: string
|
||||
voicePackService: VoicePackService
|
||||
},
|
||||
): Promise<{ extraOptions: Record<string, unknown> | undefined, costMultiplier: number }> {
|
||||
): Promise<{ extraOptions: Record<string, unknown> | undefined, costMultiplier: number, voicePackId?: string }> {
|
||||
const extraBody = asRecord(body.extra_body)
|
||||
const voicePackOptions = asRecord(extraBody?.voice_pack)
|
||||
const pitch = readOptionalNumber(voicePackOptions, 'pitch')
|
||||
const volume = readOptionalNumber(voicePackOptions, 'volume')
|
||||
const costMultiplier = await resolveVoicePackCostMultiplier(voicePackOptions, context)
|
||||
const voicePackId = typeof voicePackOptions?.pack_id === 'string' && voicePackOptions.pack_id.trim()
|
||||
? voicePackOptions.pack_id
|
||||
: undefined
|
||||
const extraOptions: Record<string, unknown> = {}
|
||||
if (pitch != null)
|
||||
extraOptions.pitch = pitch
|
||||
@@ -380,6 +427,7 @@ async function voicePackRequestOptions(
|
||||
return {
|
||||
extraOptions: Object.keys(extraOptions).length > 0 ? extraOptions : undefined,
|
||||
costMultiplier,
|
||||
voicePackId,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -35,6 +35,9 @@ function createHarness() {
|
||||
const assistantTurns: unknown[] = []
|
||||
const stateChanges: unknown[] = []
|
||||
const telemetry = {
|
||||
chatActivationStarted: [] as unknown[],
|
||||
chatActivationSucceeded: [] as unknown[],
|
||||
chatActivationFailed: [] as unknown[],
|
||||
messageSendStarted: [] as unknown[],
|
||||
llmRequestStarted: [] as unknown[],
|
||||
llmFirstToken: [] as unknown[],
|
||||
@@ -89,6 +92,9 @@ function createHarness() {
|
||||
onUserTurnReady: event => userTurns.push(event),
|
||||
onAssistantTurnReady: event => assistantTurns.push(event),
|
||||
onStateChange: state => stateChanges.push(state),
|
||||
onChatActivationStarted: event => telemetry.chatActivationStarted.push(event),
|
||||
onChatActivationSucceeded: event => telemetry.chatActivationSucceeded.push(event),
|
||||
onChatActivationFailed: event => telemetry.chatActivationFailed.push(event),
|
||||
onMessageSendStarted: event => telemetry.messageSendStarted.push(event),
|
||||
onLlmRequestStarted: event => telemetry.llmRequestStarted.push(event),
|
||||
onLlmFirstToken: event => telemetry.llmFirstToken.push(event),
|
||||
@@ -323,6 +329,46 @@ describe('createChatOrchestratorRuntime', () => {
|
||||
hasVoice: true,
|
||||
model: 'gpt-test',
|
||||
}])
|
||||
expect(harness.telemetry.chatActivationStarted).toEqual([{
|
||||
model: 'gpt-test',
|
||||
provider: 'mock-provider',
|
||||
source: 'voice',
|
||||
}])
|
||||
expect(harness.telemetry.chatActivationSucceeded).toEqual([{
|
||||
durationMs: 360,
|
||||
model: 'gpt-test',
|
||||
provider: 'mock-provider',
|
||||
source: 'voice',
|
||||
}])
|
||||
expect(harness.telemetry.chatActivationFailed).toEqual([])
|
||||
})
|
||||
|
||||
/**
|
||||
* @example
|
||||
* await expect(runtime.ingest('hello', { model, chatProvider })).rejects.toThrow('provider rejected')
|
||||
*/
|
||||
it('emits chat activation failure telemetry without raw provider messages', async () => {
|
||||
const harness = createHarness()
|
||||
harness.stream.mockRejectedValueOnce(new Error('provider rejected with sensitive details'))
|
||||
|
||||
await expect(harness.runtime.ingest('hello', {
|
||||
model: 'gpt-test',
|
||||
chatProvider: provider,
|
||||
})).rejects.toThrow('provider rejected')
|
||||
|
||||
expect(harness.telemetry.chatActivationStarted).toEqual([{
|
||||
model: 'gpt-test',
|
||||
provider: 'mock-provider',
|
||||
source: 'text',
|
||||
}])
|
||||
expect(harness.telemetry.chatActivationSucceeded).toEqual([])
|
||||
expect(harness.telemetry.chatActivationFailed).toEqual([{
|
||||
errorCode: 'llm_response_failed',
|
||||
failureStage: 'llm_response',
|
||||
model: 'gpt-test',
|
||||
provider: 'mock-provider',
|
||||
source: 'text',
|
||||
}])
|
||||
})
|
||||
|
||||
/**
|
||||
|
||||
@@ -190,6 +190,27 @@ export interface ChatOrchestratorRuntimeDeps {
|
||||
onSendSettled?: (event: { sessionId: string }) => void
|
||||
/** Called when a send starts and the first assistant placeholder is created. */
|
||||
onTrackFirstMessage?: () => void
|
||||
/** Called when a user starts a chat activation attempt. */
|
||||
onChatActivationStarted?: (event: {
|
||||
source: 'text' | 'voice'
|
||||
model: string
|
||||
provider: string
|
||||
}) => void
|
||||
/** Called after one user-to-assistant message round completes successfully. */
|
||||
onChatActivationSucceeded?: (event: {
|
||||
source: 'text' | 'voice'
|
||||
model: string
|
||||
provider: string
|
||||
durationMs: number
|
||||
}) => void
|
||||
/** Called after a chat activation attempt fails before assistant completion. */
|
||||
onChatActivationFailed?: (event: {
|
||||
source: 'text' | 'voice'
|
||||
model: string
|
||||
provider: string
|
||||
failureStage: 'llm_response'
|
||||
errorCode: 'llm_response_failed'
|
||||
}) => void
|
||||
/** Called when a user message send begins. */
|
||||
onMessageSendStarted?: (event: {
|
||||
source: 'text' | 'voice'
|
||||
@@ -401,9 +422,16 @@ export function createChatOrchestratorRuntime(deps: ChatOrchestratorRuntimeDeps)
|
||||
id: createId(),
|
||||
}
|
||||
patchForegroundStream(sessionId, buildingMessage)
|
||||
const sendSource = options.input ? 'voice' : 'text'
|
||||
const activeProvider = deps.getActiveProvider?.() ?? ''
|
||||
deps.onTrackFirstMessage?.()
|
||||
deps.onChatActivationStarted?.({
|
||||
source: sendSource,
|
||||
model: options.model,
|
||||
provider: activeProvider,
|
||||
})
|
||||
deps.onMessageSendStarted?.({
|
||||
source: options.input ? 'voice' : 'text',
|
||||
source: sendSource,
|
||||
model: options.model,
|
||||
})
|
||||
const roundStartedAt = monotonicNow()
|
||||
@@ -710,14 +738,28 @@ export function createChatOrchestratorRuntime(deps: ChatOrchestratorRuntimeDeps)
|
||||
})
|
||||
|
||||
resetForegroundStream(sessionId)
|
||||
const durationMs = Math.round(monotonicNow() - roundStartedAt)
|
||||
deps.onMessageRound?.({
|
||||
durationMs: Math.round(monotonicNow() - roundStartedAt),
|
||||
durationMs,
|
||||
hasVoice: !!options.input,
|
||||
model: options.model,
|
||||
})
|
||||
deps.onChatActivationSucceeded?.({
|
||||
durationMs,
|
||||
source: sendSource,
|
||||
model: options.model,
|
||||
provider: activeProvider,
|
||||
})
|
||||
}
|
||||
catch (error) {
|
||||
console.error('Error sending message:', error)
|
||||
deps.onChatActivationFailed?.({
|
||||
source: sendSource,
|
||||
model: options.model,
|
||||
provider: activeProvider,
|
||||
failureStage: 'llm_response',
|
||||
errorCode: 'llm_response_failed',
|
||||
})
|
||||
throw error
|
||||
}
|
||||
finally {
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
<script setup lang="ts">
|
||||
import type { VoiceType } from '@proj-airi/stage-ui/composables'
|
||||
import type { VoicePackSnapshot } from '@proj-airi/stage-ui/stores/modules/airi-card'
|
||||
import type { VoiceInfo } from '@proj-airi/stage-ui/stores/providers'
|
||||
import type { SpeechProviderWithExtraOptions } from '@xsai-ext/providers/utils'
|
||||
@@ -13,7 +14,7 @@ import {
|
||||
VoiceCardManySelect,
|
||||
} from '@proj-airi/stage-ui/components'
|
||||
import { useAnalytics } from '@proj-airi/stage-ui/composables'
|
||||
import { OFFICIAL_SPEECH_PROVIDER_ID } from '@proj-airi/stage-ui/libs/providers/providers/official'
|
||||
import { OFFICIAL_SPEECH_PROVIDER_ID, OFFICIAL_SPEECH_STREAMING_PROVIDER_ID } from '@proj-airi/stage-ui/libs/providers/providers/official'
|
||||
import { useAiriCardStore, useVoicePacksStore } from '@proj-airi/stage-ui/stores'
|
||||
import { useSpeechStore, voicePackForSpeechProvider } from '@proj-airi/stage-ui/stores/modules/speech'
|
||||
import { useProvidersStore } from '@proj-airi/stage-ui/stores/providers'
|
||||
@@ -55,7 +56,13 @@ const {
|
||||
availableVoices,
|
||||
} = storeToRefs(speechStore)
|
||||
|
||||
const { trackProviderClick } = useAnalytics()
|
||||
const {
|
||||
trackProviderClick,
|
||||
trackTtsProviderSelected,
|
||||
trackVoicePackBound,
|
||||
trackVoicePreviewPlayed,
|
||||
trackVoiceSelected,
|
||||
} = useAnalytics()
|
||||
|
||||
const voiceSearchQuery = ref('')
|
||||
const useSSML = ref(false)
|
||||
@@ -71,6 +78,9 @@ const shouldShowVoicePackSection = computed(() =>
|
||||
supportsVoicePackSelection.value
|
||||
&& (isLoadingVoicePacks.value || voicePacksError.value != null || voicePacks.value.length > 0),
|
||||
)
|
||||
const boundVoicePack = computed(() =>
|
||||
voicePackForSpeechProvider(activeSpeechProvider.value, activeCard.value?.extensions.airi.modules.speech.voicePack),
|
||||
)
|
||||
|
||||
const selectableSpeechProvidersMetadata = computed(() => {
|
||||
return [
|
||||
@@ -95,6 +105,77 @@ function formatCostMultiplier(multiplier: number) {
|
||||
return `${Number.isInteger(multiplier) ? multiplier : multiplier.toFixed(2).replace(/\.?0+$/, '')}x`
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the current TTS model id for low-cardinality analytics payloads.
|
||||
*/
|
||||
function currentTtsModelId() {
|
||||
return activeSpeechModel.value || 'unknown'
|
||||
}
|
||||
|
||||
/**
|
||||
* Classifies the selected voice without sending free-form provider config as a dimension.
|
||||
*/
|
||||
function currentVoiceType(voiceId: string, providerId = activeSpeechProvider.value, voicePack = boundVoicePack.value): VoiceType {
|
||||
if (voicePack?.voiceId === voiceId)
|
||||
return 'voice_pack'
|
||||
|
||||
const catalogVoice = availableVoices.value[providerId]?.some(voice => voice.id === voiceId)
|
||||
if (catalogVoice)
|
||||
return providerId === OFFICIAL_SPEECH_PROVIDER_ID || providerId === OFFICIAL_SPEECH_STREAMING_PROVIDER_ID ? 'official_selected' : 'custom_configured'
|
||||
|
||||
return 'custom_configured'
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds bounded voice analytics fields for catalog, voice pack, and manual voices.
|
||||
*/
|
||||
function voiceAnalyticsPayload(
|
||||
voiceId: string,
|
||||
voicePack: VoicePackSnapshot | undefined = boundVoicePack.value,
|
||||
providerId = activeSpeechProvider.value,
|
||||
): {
|
||||
voice_id: string
|
||||
voice_type: VoiceType
|
||||
voice_pack_id?: string
|
||||
} {
|
||||
const voiceType = voicePack?.voiceId === voiceId ? 'voice_pack' : currentVoiceType(voiceId, providerId, voicePack)
|
||||
const isCatalogVoice = availableVoices.value[providerId]?.some(voice => voice.id === voiceId) ?? false
|
||||
const shouldBucketVoiceId = voiceType === 'custom_configured' && !isCatalogVoice
|
||||
|
||||
return {
|
||||
voice_id: shouldBucketVoiceId ? 'custom' : voiceId,
|
||||
voice_type: voiceType,
|
||||
...(voiceType === 'voice_pack' && voicePack ? { voice_pack_id: voicePack.packId } : {}),
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Tracks the active TTS provider while preserving the legacy provider-card event.
|
||||
*/
|
||||
function selectSpeechProvider(providerId: string) {
|
||||
trackProviderClick(providerId, 'speech')
|
||||
trackTtsProviderSelected({
|
||||
tts_provider_id: providerId,
|
||||
tts_model_id: currentTtsModelId(),
|
||||
source: 'settings',
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Tracks explicit voice selection from catalog or custom input controls.
|
||||
*/
|
||||
function selectSpeechVoice(voiceId: string | undefined) {
|
||||
if (!voiceId)
|
||||
return
|
||||
|
||||
trackVoiceSelected({
|
||||
tts_provider_id: activeSpeechProvider.value || 'unknown',
|
||||
tts_model_id: currentTtsModelId(),
|
||||
...voiceAnalyticsPayload(voiceId),
|
||||
source: 'settings',
|
||||
})
|
||||
}
|
||||
|
||||
// Sync OpenAI Compatible model and voice from provider config
|
||||
function syncOpenAICompatibleSettings() {
|
||||
if (activeSpeechProvider.value !== 'openai-compatible-audio-speech')
|
||||
@@ -135,6 +216,19 @@ async function bindVoicePack(pack: (typeof voicePacks.value)[number]) {
|
||||
if (!bound)
|
||||
return
|
||||
await speechStore.loadVoicesForProvider(activeSpeechProvider.value, activeSpeechModel.value || undefined)
|
||||
trackVoicePackBound({
|
||||
tts_provider_id: activeSpeechProvider.value || 'unknown',
|
||||
tts_model_id: pack.ttsModelId,
|
||||
voice_id: pack.voiceId,
|
||||
voice_pack_id: pack.id,
|
||||
source: 'settings',
|
||||
})
|
||||
trackVoiceSelected({
|
||||
tts_provider_id: activeSpeechProvider.value || 'unknown',
|
||||
tts_model_id: pack.ttsModelId,
|
||||
...voiceAnalyticsPayload(pack.voiceId, boundVoicePack.value),
|
||||
source: 'settings',
|
||||
})
|
||||
}
|
||||
|
||||
watch(activeSpeechProvider, async (newProvider, oldProvider) => {
|
||||
@@ -203,7 +297,7 @@ async function generateTestSpeech() {
|
||||
}
|
||||
}
|
||||
|
||||
const voicePack = voicePackForSpeechProvider(activeSpeechProvider.value, activeCard.value?.extensions.airi.modules.speech.voicePack)
|
||||
const voicePack = boundVoicePack.value
|
||||
if (voicePack) {
|
||||
model = voicePack.ttsModelId
|
||||
if (!voice || voice.id !== voicePack.voiceId)
|
||||
@@ -220,6 +314,11 @@ async function generateTestSpeech() {
|
||||
return
|
||||
}
|
||||
|
||||
const previewVoicePack = voicePack
|
||||
const previewVoice = voice
|
||||
const previewModel = model
|
||||
const previewProvider = activeSpeechProvider.value || 'unknown'
|
||||
|
||||
isGenerating.value = true
|
||||
errorMessage.value = ''
|
||||
|
||||
@@ -260,7 +359,16 @@ async function generateTestSpeech() {
|
||||
// Play the audio
|
||||
setTimeout(() => {
|
||||
if (audioPlayer.value) {
|
||||
audioPlayer.value.play()
|
||||
void audioPlayer.value.play()
|
||||
.then(() => {
|
||||
trackVoicePreviewPlayed({
|
||||
tts_provider_id: previewProvider,
|
||||
tts_model_id: previewModel,
|
||||
...voiceAnalyticsPayload(previewVoice.id, previewVoicePack, previewProvider),
|
||||
source: 'manual_preview',
|
||||
})
|
||||
})
|
||||
.catch(() => {})
|
||||
}
|
||||
}, 100)
|
||||
}
|
||||
@@ -309,6 +417,7 @@ function updateCustomVoiceName(value: string | undefined) {
|
||||
provider: activeSpeechProvider.value,
|
||||
gender: 'male',
|
||||
}
|
||||
selectSpeechVoice(value)
|
||||
}
|
||||
|
||||
function updateCustomModelName(value: string | undefined) {
|
||||
@@ -408,7 +517,7 @@ function handleDeleteProvider(providerId: string) {
|
||||
:value="metadata.id"
|
||||
:title="metadata.localizedName || 'Unknown'"
|
||||
:description="metadata.localizedDescription"
|
||||
@click="trackProviderClick(metadata.id, 'speech')"
|
||||
@click="selectSpeechProvider(metadata.id)"
|
||||
>
|
||||
<template #topRight>
|
||||
<button
|
||||
@@ -618,6 +727,7 @@ function handleDeleteProvider(providerId: string) {
|
||||
:play-button-text="t('settings.pages.modules.speech.sections.section.provider-voice-selection.play_sample')"
|
||||
:pause-button-text="t('settings.pages.modules.speech.sections.section.provider-voice-selection.pause')"
|
||||
@update:custom-value="updateCustomVoiceName"
|
||||
@update:voice-id="selectSpeechVoice"
|
||||
/>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -35,7 +35,7 @@ import { initIOTracer } from '../../composables/use-io-tracer'
|
||||
import { useSpeechPipelineAnalytics } from '../../composables/use-speech-pipeline-analytics'
|
||||
import { Emotion, EMOTION_EmotionMotionName_value, EMOTION_VRMExpressionName_value, EmotionThinkMotionName } from '../../constants/emotions'
|
||||
import { getDefaultStreamingModel, getDefinedProvider } from '../../libs/providers/providers'
|
||||
import { OFFICIAL_SPEECH_PROVIDER_ID } from '../../libs/providers/providers/official'
|
||||
import { OFFICIAL_SPEECH_PROVIDER_ID, OFFICIAL_SPEECH_STREAMING_PROVIDER_ID } from '../../libs/providers/providers/official'
|
||||
import { bindSpeakingStateToPlaybackManager } from '../../libs/speech/playback-speaking-state'
|
||||
import { createStageTtsSession } from '../../libs/speech/tts-session'
|
||||
import { useAudioContext, useSpeakingStore } from '../../stores/audio'
|
||||
@@ -344,6 +344,15 @@ function createVoicePackVoice(voicePack: VoicePackSnapshot): VoiceInfo {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Classifies chat auto-TTS voice usage before forwarding analytics to the server.
|
||||
*/
|
||||
function resolveStageVoiceType(voicePack: VoicePackSnapshot | undefined): 'official_selected' | 'custom_configured' | 'voice_pack' {
|
||||
if (voicePack)
|
||||
return 'voice_pack'
|
||||
return activeSpeechProvider.value === OFFICIAL_SPEECH_PROVIDER_ID || activeSpeechProvider.value === OFFICIAL_SPEECH_STREAMING_PROVIDER_ID ? 'official_selected' : 'custom_configured'
|
||||
}
|
||||
|
||||
const speechPipeline = createSpeechPipeline<AudioBuffer>({
|
||||
tts: async (request, signal) => {
|
||||
if (signal.aborted)
|
||||
@@ -461,6 +470,7 @@ const speechPipeline = createSpeechPipeline<AudioBuffer>({
|
||||
airi_analytics: {
|
||||
trigger: 'auto',
|
||||
source: 'chat_auto_tts',
|
||||
voice_type: resolveStageVoiceType(voicePack),
|
||||
},
|
||||
},
|
||||
}
|
||||
@@ -675,6 +685,7 @@ function buildStreamingSnapshot(): StreamingSessionSnapshot | null {
|
||||
return {
|
||||
model: sessionModel,
|
||||
voice: voiceId,
|
||||
voiceType: resolveStageVoiceType(undefined),
|
||||
bufferEntireSession,
|
||||
extraBody: {
|
||||
api_resource_id: apiResourceId,
|
||||
|
||||
@@ -1,110 +1,100 @@
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const vueUseMock = vi.hoisted(() => ({
|
||||
audioInputs: undefined as unknown as { value: MediaDeviceInfo[] },
|
||||
ensurePermissions: vi.fn(async () => {}),
|
||||
startUserMediaStream: vi.fn(),
|
||||
const audioDeviceMock = vi.hoisted(() => ({
|
||||
audioInputsRef: undefined as unknown as { value: MediaDeviceInfo[] },
|
||||
ensurePermissions: vi.fn(),
|
||||
startStream: vi.fn(),
|
||||
stopStream: vi.fn(),
|
||||
trackAudioDeviceUnavailable: vi.fn(),
|
||||
trackMicrophonePermissionDenied: vi.fn(),
|
||||
trackMicrophonePermissionRequested: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('@vueuse/core', async () => {
|
||||
const vue = await vi.importActual<typeof import('vue')>('vue')
|
||||
vueUseMock.audioInputs = vue.ref<MediaDeviceInfo[]>([])
|
||||
const { ref } = await import('vue')
|
||||
|
||||
audioDeviceMock.audioInputsRef = ref([])
|
||||
|
||||
return {
|
||||
useDevicesList: () => ({
|
||||
audioInputs: vueUseMock.audioInputs,
|
||||
permissionGranted: vue.ref(false),
|
||||
ensurePermissions: vueUseMock.ensurePermissions,
|
||||
audioInputs: audioDeviceMock.audioInputsRef,
|
||||
permissionGranted: ref(false),
|
||||
ensurePermissions: audioDeviceMock.ensurePermissions,
|
||||
}),
|
||||
useUserMedia: ({ constraints }: { constraints: { value: MediaStreamConstraints } }) => ({
|
||||
stream: vue.shallowRef<MediaStream>(),
|
||||
stop: vueUseMock.stopStream,
|
||||
start: () => vueUseMock.startUserMediaStream(constraints.value),
|
||||
useUserMedia: () => ({
|
||||
stream: ref(undefined),
|
||||
stop: audioDeviceMock.stopStream,
|
||||
start: audioDeviceMock.startStream,
|
||||
}),
|
||||
}
|
||||
})
|
||||
|
||||
function createAudioInput(deviceId: string): MediaDeviceInfo {
|
||||
return {
|
||||
deviceId,
|
||||
groupId: '',
|
||||
kind: 'audioinput',
|
||||
label: deviceId,
|
||||
toJSON: () => ({}),
|
||||
}
|
||||
}
|
||||
vi.mock('../use-analytics', () => ({
|
||||
useAnalytics: () => ({
|
||||
trackAudioDeviceUnavailable: audioDeviceMock.trackAudioDeviceUnavailable,
|
||||
trackMicrophonePermissionDenied: audioDeviceMock.trackMicrophonePermissionDenied,
|
||||
trackMicrophonePermissionRequested: audioDeviceMock.trackMicrophonePermissionRequested,
|
||||
}),
|
||||
}))
|
||||
|
||||
function createDeviceNotFoundError() {
|
||||
const error = new Error('Requested device not found')
|
||||
error.name = 'NotFoundError'
|
||||
return error
|
||||
}
|
||||
describe('useAudioDevice analytics lifecycle', () => {
|
||||
beforeEach(() => {
|
||||
if (audioDeviceMock.audioInputsRef)
|
||||
audioDeviceMock.audioInputsRef.value = []
|
||||
audioDeviceMock.ensurePermissions.mockReset()
|
||||
audioDeviceMock.trackAudioDeviceUnavailable.mockReset()
|
||||
audioDeviceMock.trackMicrophonePermissionDenied.mockReset()
|
||||
audioDeviceMock.trackMicrophonePermissionRequested.mockReset()
|
||||
vi.spyOn(console, 'error').mockImplementation(() => {})
|
||||
})
|
||||
|
||||
describe('useAudioDevice', () => {
|
||||
afterEach(() => {
|
||||
vueUseMock.audioInputs.value = []
|
||||
vi.clearAllMocks()
|
||||
vi.restoreAllMocks()
|
||||
})
|
||||
|
||||
it('recognizes browser device-not-found errors that are not Error instances', async () => {
|
||||
const { isMissingAudioInputDeviceError } = await import('./audio-device')
|
||||
|
||||
expect(isMissingAudioInputDeviceError({ name: 'NotFoundError' })).toBe(true)
|
||||
expect(isMissingAudioInputDeviceError({ message: 'Requested device not found' })).toBe(true)
|
||||
})
|
||||
|
||||
it('retries with the system default microphone when a persisted device id is stale', async () => {
|
||||
/**
|
||||
* @example
|
||||
* await expect(askPermission()).rejects.toThrow()
|
||||
*/
|
||||
it('tracks microphone permission denial without exposing browser error text', async () => {
|
||||
const { useAudioDevice } = await import('./audio-device')
|
||||
const { selectedAudioInput, startStream } = useAudioDevice()
|
||||
selectedAudioInput.value = 'stale-device-id'
|
||||
const permissionError = new DOMException('User denied microphone', 'NotAllowedError')
|
||||
audioDeviceMock.ensurePermissions.mockRejectedValue(permissionError)
|
||||
|
||||
vueUseMock.startUserMediaStream
|
||||
.mockRejectedValueOnce(createDeviceNotFoundError())
|
||||
.mockResolvedValueOnce(undefined)
|
||||
const { askPermission } = useAudioDevice()
|
||||
|
||||
await startStream()
|
||||
await expect(askPermission()).rejects.toThrow(permissionError)
|
||||
|
||||
expect(selectedAudioInput.value).toBe('')
|
||||
expect(vueUseMock.startUserMediaStream).toHaveBeenNthCalledWith(1, {
|
||||
audio: {
|
||||
autoGainControl: true,
|
||||
deviceId: { exact: 'stale-device-id' },
|
||||
echoCancellation: true,
|
||||
noiseSuppression: true,
|
||||
},
|
||||
expect(audioDeviceMock.trackMicrophonePermissionRequested).toHaveBeenCalledWith({
|
||||
stt_provider_id: 'unknown',
|
||||
})
|
||||
expect(vueUseMock.startUserMediaStream).toHaveBeenNthCalledWith(2, {
|
||||
audio: {
|
||||
autoGainControl: true,
|
||||
echoCancellation: true,
|
||||
noiseSuppression: true,
|
||||
},
|
||||
expect(audioDeviceMock.trackMicrophonePermissionDenied).toHaveBeenCalledWith({
|
||||
stt_provider_id: 'unknown',
|
||||
error_code: 'permission_denied',
|
||||
})
|
||||
expect(audioDeviceMock.trackAudioDeviceUnavailable).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('prefers an enumerated default input before falling back to unconstrained audio', async () => {
|
||||
vueUseMock.audioInputs.value = [
|
||||
createAudioInput('default'),
|
||||
createAudioInput('microphone-1'),
|
||||
]
|
||||
|
||||
/**
|
||||
* @example
|
||||
* await askPermission()
|
||||
* expect(trackAudioDeviceUnavailable).toHaveBeenCalledWith(expect.objectContaining({ error_code: 'device_unavailable' }))
|
||||
*/
|
||||
it('tracks successful permission requests that still expose no microphone devices', async () => {
|
||||
const { useAudioDevice } = await import('./audio-device')
|
||||
const { selectedAudioInput, startStream } = useAudioDevice()
|
||||
selectedAudioInput.value = 'stale-device-id'
|
||||
audioDeviceMock.ensurePermissions.mockResolvedValue(undefined)
|
||||
|
||||
vueUseMock.startUserMediaStream.mockResolvedValueOnce(undefined)
|
||||
const { askPermission } = useAudioDevice()
|
||||
|
||||
await startStream()
|
||||
await askPermission()
|
||||
|
||||
expect(selectedAudioInput.value).toBe('default')
|
||||
expect(vueUseMock.startUserMediaStream).toHaveBeenCalledWith({
|
||||
audio: {
|
||||
autoGainControl: true,
|
||||
deviceId: { exact: 'default' },
|
||||
echoCancellation: true,
|
||||
noiseSuppression: true,
|
||||
},
|
||||
expect(audioDeviceMock.trackMicrophonePermissionRequested).toHaveBeenCalledWith({
|
||||
stt_provider_id: 'unknown',
|
||||
})
|
||||
expect(audioDeviceMock.trackAudioDeviceUnavailable).toHaveBeenCalledWith({
|
||||
stt_provider_id: 'unknown',
|
||||
error_code: 'device_unavailable',
|
||||
})
|
||||
expect(audioDeviceMock.trackMicrophonePermissionDenied).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
|
||||
@@ -1,10 +1,20 @@
|
||||
import { useDevicesList, useUserMedia } from '@vueuse/core'
|
||||
import { computed, nextTick, ref, watch } from 'vue'
|
||||
|
||||
import { useAnalytics } from '../use-analytics'
|
||||
|
||||
const UNKNOWN_STT_PROVIDER_ID = 'unknown'
|
||||
|
||||
/**
|
||||
* Selects the default microphone when available, otherwise the first detected input.
|
||||
*/
|
||||
function resolvePreferredAudioInput(audioInputs: MediaDeviceInfo[]) {
|
||||
return audioInputs.find(device => device.deviceId === 'default')?.deviceId || audioInputs[0]?.deviceId || ''
|
||||
}
|
||||
|
||||
/**
|
||||
* Detects browser errors caused by a stale or unavailable microphone device.
|
||||
*/
|
||||
export function isMissingAudioInputDeviceError(error: unknown) {
|
||||
if (!error || typeof error !== 'object')
|
||||
return false
|
||||
@@ -16,9 +26,30 @@ export function isMissingAudioInputDeviceError(error: unknown) {
|
||||
|| (typeof message === 'string' && message.includes('Requested device not found'))
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalizes browser microphone failures into low-cardinality analytics codes.
|
||||
*/
|
||||
function audioDeviceErrorCode(error: unknown): 'permission_denied' | 'device_unavailable' {
|
||||
if (error instanceof DOMException && (error.name === 'NotAllowedError' || error.name === 'PermissionDeniedError'))
|
||||
return 'permission_denied'
|
||||
|
||||
return 'device_unavailable'
|
||||
}
|
||||
|
||||
/**
|
||||
* Provides microphone device selection, permission requests, and audio stream lifecycle state.
|
||||
*/
|
||||
export function useAudioDevice(requestPermission: boolean = false) {
|
||||
const {
|
||||
trackAudioDeviceUnavailable,
|
||||
trackMicrophonePermissionDenied,
|
||||
trackMicrophonePermissionRequested,
|
||||
} = useAnalytics()
|
||||
const { audioInputs, permissionGranted, ensurePermissions } = useDevicesList({ constraints: { audio: true }, requestPermissions: requestPermission })
|
||||
const selectedAudioInput = ref<string>(audioInputs.value.find(device => device.deviceId === 'default')?.deviceId || '')
|
||||
/**
|
||||
* Keeps the selected microphone aligned with the currently available device list.
|
||||
*/
|
||||
function selectAvailableAudioInput() {
|
||||
if (!audioInputs.value.length)
|
||||
return
|
||||
@@ -49,12 +80,33 @@ export function useAudioDevice(requestPermission: boolean = false) {
|
||||
})
|
||||
|
||||
function askPermission() {
|
||||
trackMicrophonePermissionRequested({ stt_provider_id: UNKNOWN_STT_PROVIDER_ID })
|
||||
|
||||
return ensurePermissions()
|
||||
.then(() => nextTick())
|
||||
.then(() => {
|
||||
selectAvailableAudioInput()
|
||||
if (audioInputs.value.length <= 0) {
|
||||
trackAudioDeviceUnavailable({
|
||||
stt_provider_id: UNKNOWN_STT_PROVIDER_ID,
|
||||
error_code: 'device_unavailable',
|
||||
})
|
||||
}
|
||||
})
|
||||
.catch((error) => {
|
||||
const errorCode = audioDeviceErrorCode(error)
|
||||
if (errorCode === 'permission_denied') {
|
||||
trackMicrophonePermissionDenied({
|
||||
stt_provider_id: UNKNOWN_STT_PROVIDER_ID,
|
||||
error_code: errorCode,
|
||||
})
|
||||
}
|
||||
else {
|
||||
trackAudioDeviceUnavailable({
|
||||
stt_provider_id: UNKNOWN_STT_PROVIDER_ID,
|
||||
error_code: errorCode,
|
||||
})
|
||||
}
|
||||
console.error('Error ensuring permissions:', error)
|
||||
throw error // Re-throw so callers can handle the error
|
||||
})
|
||||
|
||||
@@ -132,4 +132,313 @@ describe('useAnalytics conversation product events', () => {
|
||||
source: 'history',
|
||||
})
|
||||
})
|
||||
|
||||
/**
|
||||
* @example
|
||||
* analytics.trackChatActivationStarted({ provider_mode: 'official', provider_id: 'official-provider', model_id: 'gpt-test', source: 'text' })
|
||||
* expect(posthog.capture).toHaveBeenCalledWith('chat_activation_started', expect.objectContaining({ surface: 'web' }))
|
||||
*/
|
||||
it('emits chat activation milestones with inferred surface and normalized fields', () => {
|
||||
const analytics = useAnalytics()
|
||||
|
||||
analytics.trackChatActivationStarted({
|
||||
provider_mode: 'official',
|
||||
provider_id: 'official-provider',
|
||||
model_id: 'gpt-test',
|
||||
source: 'text',
|
||||
})
|
||||
analytics.trackChatActivationSucceeded({
|
||||
provider_mode: 'official',
|
||||
provider_id: 'official-provider',
|
||||
model_id: 'gpt-test',
|
||||
time_to_first_message_ms: 1200,
|
||||
source: 'voice',
|
||||
})
|
||||
analytics.trackChatActivationFailed({
|
||||
provider_mode: 'custom',
|
||||
provider_id: 'openai-compatible',
|
||||
model_id: 'custom',
|
||||
error_code: 'provider_error',
|
||||
failure_stage: 'llm_response',
|
||||
source: 'voice',
|
||||
})
|
||||
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(1, 'chat_activation_started', {
|
||||
surface: 'web',
|
||||
provider_mode: 'official',
|
||||
provider_id: 'official-provider',
|
||||
model_id: 'gpt-test',
|
||||
source: 'text',
|
||||
})
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(2, 'chat_activation_succeeded', {
|
||||
surface: 'web',
|
||||
provider_mode: 'official',
|
||||
provider_id: 'official-provider',
|
||||
model_id: 'gpt-test',
|
||||
time_to_first_message_ms: 1200,
|
||||
source: 'voice',
|
||||
})
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(3, 'chat_activation_failed', {
|
||||
surface: 'web',
|
||||
provider_mode: 'custom',
|
||||
provider_id: 'openai-compatible',
|
||||
model_id: 'custom',
|
||||
error_code: 'provider_error',
|
||||
failure_stage: 'llm_response',
|
||||
source: 'voice',
|
||||
})
|
||||
})
|
||||
|
||||
/**
|
||||
* @example
|
||||
* analytics.trackVoiceSelected({ tts_provider_id: 'official-provider', tts_model_id: 'stepfun/tts', voice_id: 'voice-1', voice_type: 'official_selected', source: 'settings' })
|
||||
* expect(posthog.capture).toHaveBeenCalledWith('voice_selected', expect.objectContaining({ voice_id: 'voice-1' }))
|
||||
*/
|
||||
it('emits TTS voice selection events without losing provider context', () => {
|
||||
const analytics = useAnalytics()
|
||||
|
||||
analytics.trackTtsProviderSelected({
|
||||
tts_provider_id: 'official-provider',
|
||||
tts_model_id: 'stepfun/tts',
|
||||
source: 'settings',
|
||||
})
|
||||
analytics.trackVoiceSelected({
|
||||
tts_provider_id: 'official-provider',
|
||||
tts_model_id: 'stepfun/tts',
|
||||
voice_id: 'longxiaochun_v2',
|
||||
voice_type: 'official_selected',
|
||||
source: 'settings',
|
||||
})
|
||||
analytics.trackVoicePreviewPlayed({
|
||||
tts_provider_id: 'official-provider',
|
||||
tts_model_id: 'stepfun/tts',
|
||||
voice_id: 'longxiaochun_v2',
|
||||
voice_type: 'official_selected',
|
||||
source: 'manual_preview',
|
||||
})
|
||||
analytics.trackVoicePackBound({
|
||||
tts_provider_id: 'official-provider',
|
||||
tts_model_id: 'stepfun/tts',
|
||||
voice_id: 'longxiaochun_v2',
|
||||
voice_pack_id: 'pack-1',
|
||||
source: 'settings',
|
||||
})
|
||||
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(1, 'tts_provider_selected', {
|
||||
surface: 'web',
|
||||
tts_provider_id: 'official-provider',
|
||||
tts_model_id: 'stepfun/tts',
|
||||
source: 'settings',
|
||||
})
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(2, 'voice_selected', {
|
||||
surface: 'web',
|
||||
tts_provider_id: 'official-provider',
|
||||
tts_model_id: 'stepfun/tts',
|
||||
voice_id: 'longxiaochun_v2',
|
||||
voice_type: 'official_selected',
|
||||
source: 'settings',
|
||||
})
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(3, 'voice_preview_played', {
|
||||
surface: 'web',
|
||||
tts_provider_id: 'official-provider',
|
||||
tts_model_id: 'stepfun/tts',
|
||||
voice_id: 'longxiaochun_v2',
|
||||
voice_type: 'official_selected',
|
||||
source: 'manual_preview',
|
||||
})
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(4, 'voice_pack_bound', {
|
||||
surface: 'web',
|
||||
tts_provider_id: 'official-provider',
|
||||
tts_model_id: 'stepfun/tts',
|
||||
voice_id: 'longxiaochun_v2',
|
||||
voice_pack_id: 'pack-1',
|
||||
source: 'settings',
|
||||
})
|
||||
})
|
||||
|
||||
/**
|
||||
* @example
|
||||
* analytics.trackMicrophonePermissionDenied({ stt_provider_id: 'browser-web-speech-api' })
|
||||
* expect(posthog.capture).toHaveBeenCalledWith('microphone_permission_denied', expect.objectContaining({ surface: 'web' }))
|
||||
*/
|
||||
it('emits voice input friction events with low-cardinality error fields', () => {
|
||||
const analytics = useAnalytics()
|
||||
|
||||
analytics.trackVoiceInputStarted({
|
||||
stt_provider_id: 'browser-web-speech-api',
|
||||
})
|
||||
analytics.trackMicrophonePermissionRequested({
|
||||
stt_provider_id: 'browser-web-speech-api',
|
||||
})
|
||||
analytics.trackMicrophonePermissionDenied({
|
||||
stt_provider_id: 'browser-web-speech-api',
|
||||
error_code: 'permission_denied',
|
||||
})
|
||||
analytics.trackAudioDeviceUnavailable({
|
||||
stt_provider_id: 'browser-web-speech-api',
|
||||
error_code: 'device_unavailable',
|
||||
})
|
||||
analytics.trackVoiceInputCancelled({
|
||||
stt_provider_id: 'browser-web-speech-api',
|
||||
duration_ms: 420,
|
||||
})
|
||||
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(1, 'voice_input_started', {
|
||||
surface: 'web',
|
||||
stt_provider_id: 'browser-web-speech-api',
|
||||
})
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(2, 'microphone_permission_requested', {
|
||||
surface: 'web',
|
||||
stt_provider_id: 'browser-web-speech-api',
|
||||
})
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(3, 'microphone_permission_denied', {
|
||||
surface: 'web',
|
||||
stt_provider_id: 'browser-web-speech-api',
|
||||
error_code: 'permission_denied',
|
||||
})
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(4, 'audio_device_unavailable', {
|
||||
surface: 'web',
|
||||
stt_provider_id: 'browser-web-speech-api',
|
||||
error_code: 'device_unavailable',
|
||||
})
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(5, 'voice_input_cancelled', {
|
||||
surface: 'web',
|
||||
stt_provider_id: 'browser-web-speech-api',
|
||||
duration_ms: 420,
|
||||
})
|
||||
})
|
||||
|
||||
/**
|
||||
* @example
|
||||
* analytics.trackModelListLoaded({ provider_id: 'official-provider', provider_mode: 'official', model_count: 3, duration_ms: 25 })
|
||||
* expect(posthog.capture).toHaveBeenCalledWith('model_list_loaded', expect.objectContaining({ provider_id: 'official-provider' }))
|
||||
*/
|
||||
it('emits provider model-list health events', () => {
|
||||
const analytics = useAnalytics()
|
||||
|
||||
analytics.trackModelListLoaded({
|
||||
provider_id: 'official-provider',
|
||||
provider_mode: 'official',
|
||||
model_count: 3,
|
||||
duration_ms: 25,
|
||||
})
|
||||
analytics.trackModelListFailed({
|
||||
provider_id: 'openai-compatible',
|
||||
provider_mode: 'custom',
|
||||
error_code: 'provider_error',
|
||||
duration_ms: 40,
|
||||
})
|
||||
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(1, 'model_list_loaded', {
|
||||
surface: 'web',
|
||||
provider_id: 'official-provider',
|
||||
provider_mode: 'official',
|
||||
model_count: 3,
|
||||
duration_ms: 25,
|
||||
})
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(2, 'model_list_failed', {
|
||||
surface: 'web',
|
||||
provider_id: 'openai-compatible',
|
||||
provider_mode: 'custom',
|
||||
error_code: 'provider_error',
|
||||
duration_ms: 40,
|
||||
})
|
||||
})
|
||||
|
||||
/**
|
||||
* @example
|
||||
* analytics.trackProviderConfigFailed({ provider_id: 'openai-compatible', provider_mode: 'custom', step: 'settings_auto_validate', error_code: 'validation_failed', duration_ms: 32 })
|
||||
* expect(posthog.capture).toHaveBeenCalledWith('provider_config_failed', expect.objectContaining({ error_code: 'validation_failed' }))
|
||||
*/
|
||||
it('emits provider configuration health events with bounded fields', () => {
|
||||
const analytics = useAnalytics()
|
||||
|
||||
analytics.trackProviderConfigStarted({
|
||||
provider_id: 'openai-compatible',
|
||||
provider_mode: 'custom',
|
||||
step: 'settings_auto_validate',
|
||||
})
|
||||
analytics.trackProviderConfigSucceeded({
|
||||
provider_id: 'official-provider',
|
||||
provider_mode: 'official',
|
||||
step: 'manual_chat_ping',
|
||||
duration_ms: 18,
|
||||
})
|
||||
analytics.trackProviderConfigFailed({
|
||||
provider_id: 'openai-compatible',
|
||||
provider_mode: 'custom',
|
||||
step: 'settings_auto_validate',
|
||||
error_code: 'validation_failed',
|
||||
duration_ms: 32,
|
||||
})
|
||||
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(1, 'provider_config_started', {
|
||||
surface: 'web',
|
||||
provider_id: 'openai-compatible',
|
||||
provider_mode: 'custom',
|
||||
step: 'settings_auto_validate',
|
||||
})
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(2, 'provider_config_succeeded', {
|
||||
surface: 'web',
|
||||
provider_id: 'official-provider',
|
||||
provider_mode: 'official',
|
||||
step: 'manual_chat_ping',
|
||||
duration_ms: 18,
|
||||
})
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(3, 'provider_config_failed', {
|
||||
surface: 'web',
|
||||
provider_id: 'openai-compatible',
|
||||
provider_mode: 'custom',
|
||||
step: 'settings_auto_validate',
|
||||
error_code: 'validation_failed',
|
||||
duration_ms: 32,
|
||||
})
|
||||
})
|
||||
|
||||
/**
|
||||
* @example
|
||||
* analytics.trackBugReportSubmitted({ source: 'app', category: 'update', severity: 'major', user_type: 'unknown', entrypoint: 'about_update_error', description_length_bucket: 'medium', include_triage_context: true, screenshot_attached: true })
|
||||
* expect(posthog.capture).toHaveBeenCalledWith('bug_report_submitted', expect.objectContaining({ category: 'update' }))
|
||||
*/
|
||||
it('emits feedback and bug report events with community triage tags', () => {
|
||||
const analytics = useAnalytics()
|
||||
|
||||
analytics.trackBugReportSubmitted({
|
||||
source: 'app',
|
||||
category: 'update',
|
||||
severity: 'major',
|
||||
user_type: 'unknown',
|
||||
entrypoint: 'about_update_error',
|
||||
description_length_bucket: 'medium',
|
||||
include_triage_context: true,
|
||||
screenshot_attached: true,
|
||||
})
|
||||
analytics.trackFeedbackSubmitted({
|
||||
source: 'discord',
|
||||
category: 'voice_input',
|
||||
severity: 'minor',
|
||||
user_type: 'new_user',
|
||||
entrypoint: 'community_manual_tag',
|
||||
})
|
||||
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(1, 'bug_report_submitted', {
|
||||
surface: 'web',
|
||||
source: 'app',
|
||||
category: 'update',
|
||||
severity: 'major',
|
||||
user_type: 'unknown',
|
||||
entrypoint: 'about_update_error',
|
||||
description_length_bucket: 'medium',
|
||||
include_triage_context: true,
|
||||
screenshot_attached: true,
|
||||
})
|
||||
expect(analyticsMocks.posthogCaptureMock).toHaveBeenNthCalledWith(2, 'feedback_submitted', {
|
||||
surface: 'web',
|
||||
source: 'discord',
|
||||
category: 'voice_input',
|
||||
severity: 'minor',
|
||||
user_type: 'new_user',
|
||||
entrypoint: 'community_manual_tag',
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
@@ -20,6 +20,49 @@ export type ConversationAnalyticsSurface = 'web' | 'mobile' | 'electron'
|
||||
*/
|
||||
export type ConversationAnalyticsSource = 'chat_controls' | 'history' | 'sessions_drawer'
|
||||
|
||||
export type ProviderMode = 'official' | 'custom' | 'unknown'
|
||||
export type ChatActivationFailureStage = 'provider_config' | 'model_list' | 'message_send' | 'llm_response' | 'tts'
|
||||
export type ProviderConfigStep = 'settings_auto_validate' | 'manual_chat_ping' | 'onboarding_validate'
|
||||
export type VoiceType = 'official_default' | 'official_selected' | 'custom_configured' | 'voice_pack' | 'unknown'
|
||||
export type VoiceAnalyticsSource = 'settings' | 'onboarding' | 'chat_auto_tts' | 'manual_preview'
|
||||
export type FeedbackSource = 'app' | 'discord' | 'qq' | 'github' | 'email' | 'other'
|
||||
export type FeedbackCategory = 'provider_config' | 'model_list' | 'chat_activation' | 'tts' | 'voice_input' | 'performance' | 'payment' | 'ui_ux' | 'crash' | 'update' | 'live2d' | 'desktop_window' | 'mobile' | 'unknown'
|
||||
export type FeedbackSeverity = 'blocker' | 'major' | 'minor' | 'suggestion'
|
||||
export type FeedbackUserType = 'new_user' | 'paid_user' | 'overseas_user' | 'developer_user' | 'role_chat_user' | 'unknown'
|
||||
export type FeedbackDescriptionLengthBucket = 'empty' | 'short' | 'medium' | 'long'
|
||||
|
||||
interface ChatActivationBaseProperties {
|
||||
provider_mode: ProviderMode
|
||||
provider_id: string
|
||||
model_id: string
|
||||
source: 'text' | 'voice'
|
||||
}
|
||||
|
||||
interface TtsVoiceBaseProperties {
|
||||
tts_provider_id: string
|
||||
tts_model_id: string
|
||||
source: VoiceAnalyticsSource
|
||||
}
|
||||
|
||||
interface VoiceInputBaseProperties {
|
||||
stt_provider_id: string
|
||||
duration_ms?: number
|
||||
}
|
||||
|
||||
interface ProviderConfigBaseProperties {
|
||||
provider_id: string
|
||||
provider_mode: ProviderMode
|
||||
step: ProviderConfigStep
|
||||
}
|
||||
|
||||
interface FeedbackBaseProperties {
|
||||
source: FeedbackSource
|
||||
category: FeedbackCategory
|
||||
severity: FeedbackSeverity
|
||||
user_type: FeedbackUserType
|
||||
entrypoint: string
|
||||
}
|
||||
|
||||
function getConversationAnalyticsSurface(): ConversationAnalyticsSurface {
|
||||
if (isStageTamagotchi())
|
||||
return 'electron'
|
||||
@@ -228,6 +271,96 @@ export function useAnalytics() {
|
||||
posthog.capture('message_round', properties)
|
||||
}
|
||||
|
||||
// ─── Chat activation events ──────────────────────────────────────────
|
||||
|
||||
function trackChatActivationStarted(properties: ChatActivationBaseProperties) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('chat_activation_started', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
function trackChatActivationSucceeded(properties: ChatActivationBaseProperties & { time_to_first_message_ms?: number }) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('chat_activation_succeeded', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
function trackChatActivationFailed(properties: ChatActivationBaseProperties & {
|
||||
error_code: string
|
||||
failure_stage: ChatActivationFailureStage
|
||||
}) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('chat_activation_failed', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
function trackModelListLoaded(properties: {
|
||||
provider_id: string
|
||||
provider_mode: ProviderMode
|
||||
model_count: number
|
||||
duration_ms: number
|
||||
}) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('model_list_loaded', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
function trackModelListFailed(properties: {
|
||||
provider_id: string
|
||||
provider_mode: ProviderMode
|
||||
error_code: string
|
||||
duration_ms: number
|
||||
}) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('model_list_failed', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
function trackProviderConfigStarted(properties: ProviderConfigBaseProperties) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('provider_config_started', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
function trackProviderConfigSucceeded(properties: ProviderConfigBaseProperties & { duration_ms: number }) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('provider_config_succeeded', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
function trackProviderConfigFailed(properties: ProviderConfigBaseProperties & {
|
||||
error_code: string
|
||||
duration_ms: number
|
||||
}) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('provider_config_failed', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
// ─── Conversation action events ─────────────────────────────────────
|
||||
|
||||
function trackTtsStopClicked(properties: { reason: 'manual-chat' }) {
|
||||
@@ -295,6 +428,75 @@ export function useAnalytics() {
|
||||
posthog.capture('stt_failed', properties)
|
||||
}
|
||||
|
||||
function trackVoiceInputStarted(properties: VoiceInputBaseProperties) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('voice_input_started', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
function trackMicrophonePermissionRequested(properties: VoiceInputBaseProperties) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('microphone_permission_requested', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
function trackMicrophonePermissionDenied(properties: VoiceInputBaseProperties & { error_code?: 'permission_denied' | string }) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('microphone_permission_denied', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
function trackAudioDeviceUnavailable(properties: VoiceInputBaseProperties & { error_code?: 'device_unavailable' | string }) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('audio_device_unavailable', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
function trackVoiceInputCancelled(properties: VoiceInputBaseProperties) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('voice_input_cancelled', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
// ─── Feedback and community triage events ────────────────────────────
|
||||
|
||||
function trackBugReportSubmitted(properties: FeedbackBaseProperties & {
|
||||
description_length_bucket: FeedbackDescriptionLengthBucket
|
||||
include_triage_context: boolean
|
||||
screenshot_attached: boolean
|
||||
}) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('bug_report_submitted', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
function trackFeedbackSubmitted(properties: FeedbackBaseProperties) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('feedback_submitted', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
// ─── PTT events ──────────────────────────────────────────────────────
|
||||
|
||||
function trackPttPressed() {
|
||||
@@ -310,10 +512,9 @@ export function useAnalytics() {
|
||||
}
|
||||
|
||||
// ─── TTS events (forwarded from speech bus by use-speech-pipeline-analytics) ─
|
||||
// voice_id is `voice_type: 'catalog' | 'custom'` to keep cardinality
|
||||
// bounded — MiMo voice clone allows arbitrary user-supplied voice ids
|
||||
// (see codex F6). Actual voice_id is in properties for debug, NOT for
|
||||
// PostHog group-by.
|
||||
// Selection events use catalog `voice_id` values for adoption analysis.
|
||||
// Custom voices must pass `voice_id = custom` from the callsite when the
|
||||
// raw provider value is user supplied.
|
||||
|
||||
function trackTtsIntentStarted(properties: { intent_id: string, turn_id?: string }) {
|
||||
if (!canCapture())
|
||||
@@ -333,6 +534,53 @@ export function useAnalytics() {
|
||||
posthog.capture('tts_intent_cancelled', properties)
|
||||
}
|
||||
|
||||
function trackTtsProviderSelected(properties: TtsVoiceBaseProperties) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('tts_provider_selected', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
function trackVoiceSelected(properties: TtsVoiceBaseProperties & {
|
||||
voice_id: string
|
||||
voice_type: VoiceType
|
||||
voice_pack_id?: string
|
||||
}) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('voice_selected', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
function trackVoicePreviewPlayed(properties: TtsVoiceBaseProperties & {
|
||||
voice_id: string
|
||||
voice_type: VoiceType
|
||||
voice_pack_id?: string
|
||||
}) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('voice_preview_played', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
function trackVoicePackBound(properties: TtsVoiceBaseProperties & {
|
||||
voice_id: string
|
||||
voice_pack_id: string
|
||||
}) {
|
||||
if (!canCapture())
|
||||
return
|
||||
posthog.capture('voice_pack_bound', {
|
||||
...properties,
|
||||
surface: getConversationAnalyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
// ─── Autonomous LLM path (artistry-autonomous bypasses chat orchestrator) ─
|
||||
|
||||
function trackAutonomousGenerateText(properties: { model: string, reason?: string }) {
|
||||
@@ -430,6 +678,14 @@ export function useAnalytics() {
|
||||
trackLlmFirstToken,
|
||||
trackAssistantResponseRendered,
|
||||
trackMessageRound,
|
||||
trackChatActivationStarted,
|
||||
trackChatActivationSucceeded,
|
||||
trackChatActivationFailed,
|
||||
trackModelListLoaded,
|
||||
trackModelListFailed,
|
||||
trackProviderConfigStarted,
|
||||
trackProviderConfigSucceeded,
|
||||
trackProviderConfigFailed,
|
||||
trackTtsStopClicked,
|
||||
trackChatSessionSelected,
|
||||
trackChatMessageDeleted,
|
||||
@@ -439,6 +695,13 @@ export function useAnalytics() {
|
||||
trackSttStarted,
|
||||
trackSttSucceeded,
|
||||
trackSttFailed,
|
||||
trackVoiceInputStarted,
|
||||
trackMicrophonePermissionRequested,
|
||||
trackMicrophonePermissionDenied,
|
||||
trackAudioDeviceUnavailable,
|
||||
trackVoiceInputCancelled,
|
||||
trackBugReportSubmitted,
|
||||
trackFeedbackSubmitted,
|
||||
|
||||
trackPttPressed,
|
||||
trackPttReleased,
|
||||
@@ -446,6 +709,10 @@ export function useAnalytics() {
|
||||
trackTtsIntentStarted,
|
||||
trackTtsIntentEnded,
|
||||
trackTtsIntentCancelled,
|
||||
trackTtsProviderSelected,
|
||||
trackVoiceSelected,
|
||||
trackVoicePreviewPlayed,
|
||||
trackVoicePackBound,
|
||||
|
||||
trackAutonomousGenerateText,
|
||||
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
import type { RemovableRef } from '@vueuse/core'
|
||||
|
||||
import type { ProviderConfigStep, ProviderMode } from './use-analytics'
|
||||
|
||||
import { errorMessageFrom } from '@moeru/std'
|
||||
import { useDebounceFn } from '@vueuse/core'
|
||||
import { storeToRefs } from 'pinia'
|
||||
@@ -8,11 +10,29 @@ import { useI18n } from 'vue-i18n'
|
||||
import { useRouter } from 'vue-router'
|
||||
|
||||
import { useProvidersStore } from '../stores/providers'
|
||||
import { useAnalytics } from './use-analytics'
|
||||
|
||||
/**
|
||||
* Classifies provider ids into bounded analytics buckets.
|
||||
*/
|
||||
function providerModeForAnalytics(providerId: string): ProviderMode {
|
||||
if (!providerId)
|
||||
return 'unknown'
|
||||
|
||||
return providerId.startsWith('official-provider') || providerId.startsWith('vision-official-provider')
|
||||
? 'official'
|
||||
: 'custom'
|
||||
}
|
||||
|
||||
export function useProviderValidation(providerId: string) {
|
||||
const { t } = useI18n()
|
||||
const router = useRouter()
|
||||
const providersStore = useProvidersStore()
|
||||
const {
|
||||
trackProviderConfigFailed,
|
||||
trackProviderConfigStarted,
|
||||
trackProviderConfigSucceeded,
|
||||
} = useAnalytics()
|
||||
const { providers } = storeToRefs(providersStore) as { providers: RemovableRef<Record<string, any>> }
|
||||
|
||||
const providerMetadata = computed(() => providersStore.getProviderMetadata(providerId))
|
||||
@@ -59,6 +79,17 @@ export function useProviderValidation(providerId: string) {
|
||||
const manualTestPassed = ref(false)
|
||||
const manualTestMessage = ref('')
|
||||
|
||||
/**
|
||||
* Builds the stable provider analytics fields shared by validation events.
|
||||
*/
|
||||
function providerConfigAnalyticsBase(step: ProviderConfigStep) {
|
||||
return {
|
||||
provider_id: providerId,
|
||||
provider_mode: providerModeForAnalytics(providerId),
|
||||
step,
|
||||
}
|
||||
}
|
||||
|
||||
async function validateConfiguration() {
|
||||
if (!providerMetadata.value)
|
||||
return
|
||||
@@ -66,6 +97,7 @@ export function useProviderValidation(providerId: string) {
|
||||
isValidating.value++
|
||||
validationMessage.value = ''
|
||||
const startValidationTimestamp = performance.now()
|
||||
trackProviderConfigStarted(providerConfigAnalyticsBase('settings_auto_validate'))
|
||||
let finalValidationMessage = ''
|
||||
|
||||
try {
|
||||
@@ -82,8 +114,14 @@ export function useProviderValidation(providerId: string) {
|
||||
})
|
||||
isValid.value = validationResult.valid
|
||||
|
||||
if (!isValid.value)
|
||||
if (!isValid.value) {
|
||||
finalValidationMessage = validationResult.reason
|
||||
trackProviderConfigFailed({
|
||||
...providerConfigAnalyticsBase('settings_auto_validate'),
|
||||
error_code: 'validation_failed',
|
||||
duration_ms: Math.round(performance.now() - startValidationTimestamp),
|
||||
})
|
||||
}
|
||||
|
||||
// When a provider validates successfully on its settings page,
|
||||
// mark it as added so it appears in the model selector (e.g. Consciousness module).
|
||||
@@ -91,6 +129,10 @@ export function useProviderValidation(providerId: string) {
|
||||
// need an API key, yet should be selectable after successful validation.
|
||||
if (isValid.value) {
|
||||
providersStore.markProviderAdded(providerId)
|
||||
trackProviderConfigSucceeded({
|
||||
...providerConfigAnalyticsBase('settings_auto_validate'),
|
||||
duration_ms: Math.round(performance.now() - startValidationTimestamp),
|
||||
})
|
||||
}
|
||||
}
|
||||
catch (error) {
|
||||
@@ -98,6 +140,11 @@ export function useProviderValidation(providerId: string) {
|
||||
finalValidationMessage = t('settings.dialogs.onboarding.validationError', {
|
||||
error: errorMessageFrom(error) ?? 'Generic error (993b5ad7)',
|
||||
})
|
||||
trackProviderConfigFailed({
|
||||
...providerConfigAnalyticsBase('settings_auto_validate'),
|
||||
error_code: 'provider_error',
|
||||
duration_ms: Math.round(performance.now() - startValidationTimestamp),
|
||||
})
|
||||
}
|
||||
finally {
|
||||
setTimeout(() => {
|
||||
@@ -113,6 +160,8 @@ export function useProviderValidation(providerId: string) {
|
||||
|
||||
isManualTesting.value = true
|
||||
manualTestMessage.value = ''
|
||||
const startedAt = performance.now()
|
||||
trackProviderConfigStarted(providerConfigAnalyticsBase('manual_chat_ping'))
|
||||
|
||||
try {
|
||||
const config = { ...credentials.value }
|
||||
@@ -125,12 +174,29 @@ export function useProviderValidation(providerId: string) {
|
||||
onlyChatPingCheck: true,
|
||||
})
|
||||
manualTestPassed.value = result.valid
|
||||
if (!result.valid)
|
||||
if (result.valid) {
|
||||
trackProviderConfigSucceeded({
|
||||
...providerConfigAnalyticsBase('manual_chat_ping'),
|
||||
duration_ms: Math.round(performance.now() - startedAt),
|
||||
})
|
||||
}
|
||||
else {
|
||||
manualTestMessage.value = result.reason
|
||||
trackProviderConfigFailed({
|
||||
...providerConfigAnalyticsBase('manual_chat_ping'),
|
||||
error_code: 'validation_failed',
|
||||
duration_ms: Math.round(performance.now() - startedAt),
|
||||
})
|
||||
}
|
||||
}
|
||||
catch (error) {
|
||||
manualTestPassed.value = false
|
||||
manualTestMessage.value = errorMessageFrom(error) ?? 'Generic error (e56ae24f)'
|
||||
trackProviderConfigFailed({
|
||||
...providerConfigAnalyticsBase('manual_chat_ping'),
|
||||
error_code: 'provider_error',
|
||||
duration_ms: Math.round(performance.now() - startedAt),
|
||||
})
|
||||
}
|
||||
finally {
|
||||
isManualTesting.value = false
|
||||
|
||||
@@ -18,6 +18,7 @@ vi.mock('../server', () => ({
|
||||
interface MockServer {
|
||||
url: string
|
||||
receivedFrames: Array<{ kind: 'text' | 'binary', data: string | Buffer }>
|
||||
observedVoiceTypes: string[]
|
||||
/** Resolves when the server has observed a `start` frame from the client. */
|
||||
startObserved: Promise<void>
|
||||
stop: () => Promise<void>
|
||||
@@ -25,6 +26,7 @@ interface MockServer {
|
||||
|
||||
async function startMockServer(handler: (ws: import('ws').WebSocket) => void): Promise<MockServer> {
|
||||
const receivedFrames: MockServer['receivedFrames'] = []
|
||||
const observedVoiceTypes: string[] = []
|
||||
const httpServer = createServer()
|
||||
const wss = new WebSocketServer({ server: httpServer })
|
||||
|
||||
@@ -33,7 +35,12 @@ async function startMockServer(handler: (ws: import('ws').WebSocket) => void): P
|
||||
resolveStartObserved = res
|
||||
})
|
||||
|
||||
wss.on('connection', (ws) => {
|
||||
wss.on('connection', (ws, req) => {
|
||||
const u = new URL(req.url!, 'http://localhost')
|
||||
const voiceType = u.searchParams.get('tts_voice_type')
|
||||
if (voiceType != null)
|
||||
observedVoiceTypes.push(voiceType)
|
||||
|
||||
ws.on('message', (data, isBinary) => {
|
||||
const buf = Buffer.isBuffer(data) ? data : Buffer.from(data as ArrayBuffer)
|
||||
const decoded = isBinary ? buf : buf.toString('utf8')
|
||||
@@ -56,6 +63,7 @@ async function startMockServer(handler: (ws: import('ws').WebSocket) => void): P
|
||||
return {
|
||||
url: `http://127.0.0.1:${port}`,
|
||||
receivedFrames,
|
||||
observedVoiceTypes,
|
||||
startObserved,
|
||||
async stop() {
|
||||
wss.close()
|
||||
@@ -127,6 +135,7 @@ describe('createStreamingTtsPipeline', () => {
|
||||
serverUrl: server.url,
|
||||
model: 'volcengine/seed-tts-1.0',
|
||||
voice: 'mock',
|
||||
ttsVoiceType: 'official_selected',
|
||||
audioContext: makeStubAudioContext(),
|
||||
onSentence,
|
||||
onError,
|
||||
@@ -146,6 +155,7 @@ describe('createStreamingTtsPipeline', () => {
|
||||
await server.startObserved
|
||||
const textFrames = server.receivedFrames.filter(f => f.kind === 'text').map(f => JSON.parse(f.data as string))
|
||||
expect(textFrames.map(f => f.event)).toEqual(['start', 'text', 'text', 'finish'])
|
||||
expect(server.observedVoiceTypes).toEqual(['official_selected'])
|
||||
expect(textFrames[1]).toMatchObject({ event: 'text', text: 'hi ' })
|
||||
expect(textFrames[2]).toMatchObject({ event: 'text', text: 'there' })
|
||||
|
||||
|
||||
@@ -52,6 +52,8 @@ export interface StreamingTtsPipelineOptions extends StreamingTtsPipelineEvents
|
||||
ttsTrigger?: 'auto' | 'manual'
|
||||
/** Low-cardinality source hint sent to server-side product analytics. */
|
||||
ttsSource?: 'chat_auto_tts' | 'manual_preview' | 'settings_test'
|
||||
/** Low-cardinality voice bucket sent to server-side product analytics. */
|
||||
ttsVoiceType?: 'official_default' | 'official_selected' | 'custom_configured' | 'voice_pack' | 'unknown'
|
||||
/**
|
||||
* Decoder context. The pipeline calls `decodeAudioData` on it for each
|
||||
* sentence (or once at session end in buffered mode). Reusing the page's
|
||||
@@ -122,6 +124,7 @@ export function createStreamingTtsPipeline(options: StreamingTtsPipelineOptions)
|
||||
const wsUrl = toWebSocketUrl(options.serverUrl ?? SERVER_URL, '/api/v1/audio/speech/ws', token, {
|
||||
ttsTrigger: options.ttsTrigger ?? 'auto',
|
||||
ttsSource: options.ttsSource ?? 'chat_auto_tts',
|
||||
ttsVoiceType: options.ttsVoiceType ?? 'unknown',
|
||||
})
|
||||
const ws = new WebSocket(wsUrl)
|
||||
ws.binaryType = 'arraybuffer'
|
||||
@@ -411,13 +414,18 @@ function toWebSocketUrl(
|
||||
httpBase: string,
|
||||
path: string,
|
||||
token: string,
|
||||
analytics: { ttsTrigger: 'auto' | 'manual', ttsSource: 'chat_auto_tts' | 'manual_preview' | 'settings_test' },
|
||||
analytics: {
|
||||
ttsTrigger: 'auto' | 'manual'
|
||||
ttsSource: 'chat_auto_tts' | 'manual_preview' | 'settings_test'
|
||||
ttsVoiceType: 'official_default' | 'official_selected' | 'custom_configured' | 'voice_pack' | 'unknown'
|
||||
},
|
||||
): string {
|
||||
const u = new URL(path, httpBase)
|
||||
u.protocol = u.protocol === 'https:' ? 'wss:' : 'ws:'
|
||||
u.searchParams.set('token', token)
|
||||
u.searchParams.set('tts_trigger', analytics.ttsTrigger)
|
||||
u.searchParams.set('tts_source', analytics.ttsSource)
|
||||
u.searchParams.set('tts_voice_type', analytics.ttsVoiceType)
|
||||
return u.toString()
|
||||
}
|
||||
|
||||
|
||||
@@ -19,12 +19,14 @@ vi.mock('../server', () => ({
|
||||
interface MockServer {
|
||||
url: string
|
||||
observedTokens: string[]
|
||||
observedVoiceTypes: string[]
|
||||
closeUnexpectedly: () => void
|
||||
stop: () => Promise<void>
|
||||
}
|
||||
|
||||
async function startMockServer(handler: (ws: import('ws').WebSocket) => void): Promise<MockServer> {
|
||||
const observedTokens: string[] = []
|
||||
const observedVoiceTypes: string[] = []
|
||||
const httpServer = createServer()
|
||||
const wss = new WebSocketServer({ server: httpServer })
|
||||
|
||||
@@ -36,6 +38,9 @@ async function startMockServer(handler: (ws: import('ws').WebSocket) => void): P
|
||||
const token = u.searchParams.get('token')
|
||||
if (token != null)
|
||||
observedTokens.push(token)
|
||||
const voiceType = u.searchParams.get('tts_voice_type')
|
||||
if (voiceType != null)
|
||||
observedVoiceTypes.push(voiceType)
|
||||
|
||||
handler(ws)
|
||||
})
|
||||
@@ -46,6 +51,7 @@ async function startMockServer(handler: (ws: import('ws').WebSocket) => void): P
|
||||
return {
|
||||
url: `http://127.0.0.1:${port}`,
|
||||
observedTokens,
|
||||
observedVoiceTypes,
|
||||
closeUnexpectedly: () => {
|
||||
activeWs?.close(1011, 'simulated_truncation')
|
||||
},
|
||||
@@ -103,6 +109,7 @@ describe('streamingSynthesize', () => {
|
||||
serverUrl: server.url,
|
||||
model: 'volcengine/seed-tts-2.0',
|
||||
voice: 'mock',
|
||||
ttsVoiceType: 'official_selected',
|
||||
input: 'hello',
|
||||
})
|
||||
|
||||
@@ -114,6 +121,7 @@ describe('streamingSynthesize', () => {
|
||||
expect(result.sentences).toHaveLength(1)
|
||||
expect(result.sentences[0]).toMatchObject({ kind: 'end' })
|
||||
expect(server.observedTokens).toEqual(['test-jwt'])
|
||||
expect(server.observedVoiceTypes).toEqual(['official_selected'])
|
||||
})
|
||||
|
||||
it('rejects when the ws closes before session.finished (codex HIGH #2)', async () => {
|
||||
|
||||
@@ -58,6 +58,8 @@ export interface StreamingTtsSessionOptions {
|
||||
ttsTrigger?: 'auto' | 'manual'
|
||||
/** Low-cardinality source hint sent to server-side product analytics. */
|
||||
ttsSource?: 'chat_auto_tts' | 'manual_preview' | 'settings_test'
|
||||
/** Low-cardinality voice bucket sent to server-side product analytics. */
|
||||
ttsVoiceType?: 'official_default' | 'official_selected' | 'custom_configured' | 'voice_pack' | 'unknown'
|
||||
/** Caller-side abort signal. Closes the ws and rejects with `AbortError`. */
|
||||
signal?: AbortSignal
|
||||
}
|
||||
@@ -92,6 +94,7 @@ export async function streamingSynthesize(options: StreamingTtsSessionOptions):
|
||||
const wsUrl = toWebSocketUrl(baseUrl, '/api/v1/audio/speech/ws', token, {
|
||||
ttsTrigger: options.ttsTrigger ?? 'manual',
|
||||
ttsSource: options.ttsSource ?? 'manual_preview',
|
||||
ttsVoiceType: options.ttsVoiceType ?? 'unknown',
|
||||
})
|
||||
|
||||
const audioChunks: ArrayBuffer[] = []
|
||||
@@ -236,13 +239,18 @@ function toWebSocketUrl(
|
||||
httpBase: string,
|
||||
path: string,
|
||||
token: string,
|
||||
analytics: { ttsTrigger: 'auto' | 'manual', ttsSource: 'chat_auto_tts' | 'manual_preview' | 'settings_test' },
|
||||
analytics: {
|
||||
ttsTrigger: 'auto' | 'manual'
|
||||
ttsSource: 'chat_auto_tts' | 'manual_preview' | 'settings_test'
|
||||
ttsVoiceType: 'official_default' | 'official_selected' | 'custom_configured' | 'voice_pack' | 'unknown'
|
||||
},
|
||||
): string {
|
||||
const u = new URL(path, httpBase)
|
||||
u.protocol = u.protocol === 'https:' ? 'wss:' : 'ws:'
|
||||
u.searchParams.set('token', token)
|
||||
u.searchParams.set('tts_trigger', analytics.ttsTrigger)
|
||||
u.searchParams.set('tts_source', analytics.ttsSource)
|
||||
u.searchParams.set('tts_voice_type', analytics.ttsVoiceType)
|
||||
return u.toString()
|
||||
}
|
||||
|
||||
|
||||
@@ -48,6 +48,7 @@ function makeStreamingSnapshot(overrides: Partial<StreamingSessionSnapshot> = {}
|
||||
return {
|
||||
model: 'volcengine/seed-tts-2.0',
|
||||
voice: 'mock-voice',
|
||||
voiceType: 'official_selected',
|
||||
bufferEntireSession: false,
|
||||
extraBody: { api_resource_id: 'seed-tts-2.0' },
|
||||
ownerId: 'card-1',
|
||||
@@ -191,6 +192,8 @@ describe('createStreamingTtsSession (adapter)', () => {
|
||||
pipelineFactory: pipe.factory as any,
|
||||
})
|
||||
|
||||
expect(pipe.options.ttsVoiceType).toBe('official_selected')
|
||||
|
||||
// Simulate the pipeline emitting two sentences.
|
||||
const audio0 = { __id: 0 } as unknown as AudioBuffer
|
||||
const audio1 = { __id: 1 } as unknown as AudioBuffer
|
||||
|
||||
@@ -71,6 +71,7 @@ function fromIntent(intent: IntentHandleSubset): StageTtsSession {
|
||||
export interface StreamingSessionSnapshot {
|
||||
model: string
|
||||
voice: string
|
||||
voiceType: 'official_default' | 'official_selected' | 'custom_configured' | 'voice_pack' | 'unknown'
|
||||
bufferEntireSession: boolean
|
||||
extraBody: Record<string, unknown>
|
||||
/**
|
||||
@@ -158,6 +159,7 @@ export function createStreamingTtsSession<TAudio = AudioBuffer>(
|
||||
const handle = pipelineFactory({
|
||||
model: snapshot.model,
|
||||
voice: snapshot.voice,
|
||||
ttsVoiceType: snapshot.voiceType,
|
||||
audioContext,
|
||||
bufferEntireSession: snapshot.bufferEntireSession,
|
||||
extraBody: snapshot.extraBody,
|
||||
|
||||
@@ -67,6 +67,9 @@ vi.mock('../composables', () => ({
|
||||
trackLlmFirstToken: vi.fn(),
|
||||
trackAssistantResponseRendered: vi.fn(),
|
||||
trackMessageRound: vi.fn(),
|
||||
trackChatActivationStarted: vi.fn(),
|
||||
trackChatActivationSucceeded: vi.fn(),
|
||||
trackChatActivationFailed: vi.fn(),
|
||||
}),
|
||||
}))
|
||||
|
||||
|
||||
@@ -56,6 +56,9 @@ export const useChatOrchestratorStore = defineStore('chat-orchestrator', () => {
|
||||
trackLlmFirstToken,
|
||||
trackAssistantResponseRendered,
|
||||
trackMessageRound,
|
||||
trackChatActivationStarted,
|
||||
trackChatActivationSucceeded,
|
||||
trackChatActivationFailed,
|
||||
} = useAnalytics()
|
||||
|
||||
const chatSession = useChatSessionStore()
|
||||
@@ -132,6 +135,15 @@ export const useChatOrchestratorStore = defineStore('chat-orchestrator', () => {
|
||||
ownedActiveTurnSpan = undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Classifies configured chat providers into low-cardinality product analytics buckets.
|
||||
*/
|
||||
function providerMode(providerId: string | undefined): 'official' | 'custom' | 'unknown' {
|
||||
if (!providerId)
|
||||
return 'unknown'
|
||||
return providerId.startsWith('official-provider') ? 'official' : 'custom'
|
||||
}
|
||||
|
||||
const runtime = createChatOrchestratorRuntime({
|
||||
session: {
|
||||
ensureSession: sessionId => chatSession.ensureSession(sessionId),
|
||||
@@ -187,6 +199,27 @@ export const useChatOrchestratorStore = defineStore('chat-orchestrator', () => {
|
||||
has_voice: hasVoice,
|
||||
model,
|
||||
}),
|
||||
onChatActivationStarted: ({ model, provider, source }) => trackChatActivationStarted({
|
||||
provider_mode: providerMode(provider),
|
||||
provider_id: provider || 'unknown',
|
||||
model_id: model || 'unknown',
|
||||
source,
|
||||
}),
|
||||
onChatActivationSucceeded: ({ model, provider, durationMs, source }) => trackChatActivationSucceeded({
|
||||
provider_mode: providerMode(provider),
|
||||
provider_id: provider || 'unknown',
|
||||
model_id: model || 'unknown',
|
||||
time_to_first_message_ms: durationMs,
|
||||
source,
|
||||
}),
|
||||
onChatActivationFailed: ({ model, provider, errorCode, failureStage, source }) => trackChatActivationFailed({
|
||||
provider_mode: providerMode(provider),
|
||||
provider_id: provider || 'unknown',
|
||||
model_id: model || 'unknown',
|
||||
error_code: errorCode,
|
||||
failure_stage: failureStage,
|
||||
source,
|
||||
}),
|
||||
onLifecycle: record => contextObservability.recordLifecycle(record),
|
||||
onPromptProjection: payload => contextObservability.capturePromptProjection(payload),
|
||||
onUserMessageAppended: ({ sessionId, message, messageText }) => {
|
||||
|
||||
@@ -3,9 +3,17 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const analyticsMock = vi.hoisted(() => ({
|
||||
allowComposableCall: true,
|
||||
trackAudioDeviceUnavailable: vi.fn(),
|
||||
trackMicrophonePermissionDenied: vi.fn(),
|
||||
trackSttFailed: vi.fn(),
|
||||
trackSttStarted: vi.fn(),
|
||||
trackSttSucceeded: vi.fn(),
|
||||
trackVoiceInputCancelled: vi.fn(),
|
||||
trackVoiceInputStarted: vi.fn(),
|
||||
}))
|
||||
|
||||
const transcriptionMock = vi.hoisted(() => ({
|
||||
generateTranscription: vi.fn(async () => ({ text: 'hello' })),
|
||||
}))
|
||||
|
||||
vi.mock('../../composables/use-analytics', () => ({
|
||||
@@ -14,15 +22,19 @@ vi.mock('../../composables/use-analytics', () => ({
|
||||
throw new Error('Must be called at the top of a `setup` function')
|
||||
|
||||
return {
|
||||
trackAudioDeviceUnavailable: analyticsMock.trackAudioDeviceUnavailable,
|
||||
trackMicrophonePermissionDenied: analyticsMock.trackMicrophonePermissionDenied,
|
||||
trackSttFailed: analyticsMock.trackSttFailed,
|
||||
trackSttStarted: analyticsMock.trackSttStarted,
|
||||
trackSttSucceeded: analyticsMock.trackSttSucceeded,
|
||||
trackVoiceInputCancelled: analyticsMock.trackVoiceInputCancelled,
|
||||
trackVoiceInputStarted: analyticsMock.trackVoiceInputStarted,
|
||||
}
|
||||
},
|
||||
}))
|
||||
|
||||
vi.mock('@xsai/generate-transcription', () => ({
|
||||
generateTranscription: vi.fn(async () => ({ text: 'hello' })),
|
||||
generateTranscription: transcriptionMock.generateTranscription,
|
||||
}))
|
||||
|
||||
vi.mock('vue-i18n', () => ({
|
||||
@@ -36,9 +48,15 @@ describe('useHearingStore analytics lifecycle', () => {
|
||||
beforeEach(() => {
|
||||
setActivePinia(createPinia())
|
||||
analyticsMock.allowComposableCall = true
|
||||
analyticsMock.trackAudioDeviceUnavailable.mockReset()
|
||||
analyticsMock.trackMicrophonePermissionDenied.mockReset()
|
||||
analyticsMock.trackSttFailed.mockReset()
|
||||
analyticsMock.trackSttStarted.mockReset()
|
||||
analyticsMock.trackSttSucceeded.mockReset()
|
||||
analyticsMock.trackVoiceInputCancelled.mockReset()
|
||||
analyticsMock.trackVoiceInputStarted.mockReset()
|
||||
transcriptionMock.generateTranscription.mockReset()
|
||||
transcriptionMock.generateTranscription.mockResolvedValue({ text: 'hello' })
|
||||
})
|
||||
|
||||
/**
|
||||
@@ -60,6 +78,9 @@ describe('useHearingStore analytics lifecycle', () => {
|
||||
)
|
||||
|
||||
expect(result.text).toBe('hello')
|
||||
expect(analyticsMock.trackVoiceInputStarted).toHaveBeenCalledWith({
|
||||
stt_provider_id: 'openai-compatible-audio-transcription',
|
||||
})
|
||||
expect(analyticsMock.trackSttStarted).toHaveBeenCalledWith('openai-compatible-audio-transcription')
|
||||
expect(analyticsMock.trackSttSucceeded).toHaveBeenCalledWith({
|
||||
provider: 'openai-compatible-audio-transcription',
|
||||
@@ -68,4 +89,34 @@ describe('useHearingStore analytics lifecycle', () => {
|
||||
stream: false,
|
||||
})
|
||||
}, 10000)
|
||||
|
||||
/**
|
||||
* @example
|
||||
* await expect(hearingStore.transcription(providerId, provider, model, file)).rejects.toThrow()
|
||||
*/
|
||||
it('normalizes microphone permission failures for analytics', async () => {
|
||||
const { useHearingStore } = await import('./hearing')
|
||||
const hearingStore = useHearingStore()
|
||||
const permissionError = new DOMException('User denied microphone', 'NotAllowedError')
|
||||
transcriptionMock.generateTranscription.mockRejectedValueOnce(permissionError)
|
||||
|
||||
await expect(hearingStore.transcription(
|
||||
'openai-compatible-audio-transcription',
|
||||
{
|
||||
transcription: () => ({}),
|
||||
} as any,
|
||||
'FunAudioLLM/SenseVoiceSmall',
|
||||
new File(['hello'], 'recording.wav', { type: 'audio/wav' }),
|
||||
)).rejects.toBe(permissionError)
|
||||
|
||||
expect(analyticsMock.trackSttFailed).toHaveBeenCalledWith({
|
||||
provider: 'openai-compatible-audio-transcription',
|
||||
error_code: 'permission_denied',
|
||||
})
|
||||
expect(analyticsMock.trackMicrophonePermissionDenied).toHaveBeenCalledWith({
|
||||
stt_provider_id: 'openai-compatible-audio-transcription',
|
||||
error_code: 'permission_denied',
|
||||
})
|
||||
expect(analyticsMock.trackAudioDeviceUnavailable).not.toHaveBeenCalled()
|
||||
}, 10000)
|
||||
})
|
||||
|
||||
@@ -48,6 +48,33 @@ function isExpectedStreamStopError(err: unknown): boolean {
|
||||
&& (err.message === 'Stopped' || err.message === 'Aborted' || err.message === 'Closed' || err.message === 'Idle timeout')
|
||||
}
|
||||
|
||||
type TranscriptionAnalyticsErrorCode = 'permission_denied' | 'device_unavailable' | 'input_unavailable' | 'provider_error' | 'unknown'
|
||||
|
||||
/**
|
||||
* Normalizes transcription failures into bounded analytics error codes.
|
||||
*/
|
||||
function transcriptionAnalyticsErrorCode(err: unknown): TranscriptionAnalyticsErrorCode {
|
||||
if (err instanceof DOMException) {
|
||||
if (err.name === 'NotAllowedError' || err.name === 'PermissionDeniedError')
|
||||
return 'permission_denied'
|
||||
|
||||
if (err.name === 'NotFoundError' || err.name === 'NotReadableError')
|
||||
return 'device_unavailable'
|
||||
}
|
||||
|
||||
const message = (errorMessageFrom(err) ?? '').toLowerCase()
|
||||
if (message.includes('permission') || message.includes('notallowed'))
|
||||
return 'permission_denied'
|
||||
|
||||
if (message.includes('microphone') || message.includes('audio track') || message.includes('device'))
|
||||
return 'device_unavailable'
|
||||
|
||||
if (message.includes('file input') || message.includes('compatible input'))
|
||||
return 'input_unavailable'
|
||||
|
||||
return message ? 'provider_error' : 'unknown'
|
||||
}
|
||||
|
||||
function haveStreamingCallbacksChanged(
|
||||
previous: { onSentenceEnd?: (delta: string) => void, onSpeechEnd?: (text: string) => void } | undefined,
|
||||
next: { onSentenceEnd?: (delta: string) => void, onSpeechEnd?: (text: string) => void },
|
||||
@@ -281,7 +308,14 @@ export function resolveTranscriptionProviderOptions(providerConfig?: Record<stri
|
||||
export const useHearingStore = defineStore('hearing-store', () => {
|
||||
const providersStore = useProvidersStore()
|
||||
const { allAudioTranscriptionProvidersMetadata } = storeToRefs(providersStore)
|
||||
const { trackSttStarted, trackSttSucceeded, trackSttFailed } = useAnalytics()
|
||||
const {
|
||||
trackAudioDeviceUnavailable,
|
||||
trackMicrophonePermissionDenied,
|
||||
trackSttFailed,
|
||||
trackSttStarted,
|
||||
trackSttSucceeded,
|
||||
trackVoiceInputStarted,
|
||||
} = useAnalytics()
|
||||
|
||||
// State
|
||||
const activeTranscriptionProvider = useLocalStorageManualReset('settings/hearing/active-provider', '')
|
||||
@@ -378,6 +412,7 @@ export const useHearingStore = defineStore('hearing-store', () => {
|
||||
const streamExecutor = resolveStreamTranscriptionExecutor(providerId)
|
||||
|
||||
const sttStartedAt = performance.now()
|
||||
trackVoiceInputStarted({ stt_provider_id: providerId })
|
||||
trackSttStarted(providerId)
|
||||
|
||||
function emitSucceeded(charCount: number, stream: boolean) {
|
||||
@@ -389,7 +424,20 @@ export const useHearingStore = defineStore('hearing-store', () => {
|
||||
})
|
||||
}
|
||||
function emitFailed(err: unknown) {
|
||||
trackSttFailed({ provider: providerId, error_code: (errorMessageFrom(err) ?? 'unknown').slice(0, 64) })
|
||||
const errorCode = transcriptionAnalyticsErrorCode(err)
|
||||
trackSttFailed({ provider: providerId, error_code: errorCode })
|
||||
if (errorCode === 'permission_denied') {
|
||||
trackMicrophonePermissionDenied({
|
||||
stt_provider_id: providerId,
|
||||
error_code: errorCode,
|
||||
})
|
||||
}
|
||||
if (errorCode === 'device_unavailable') {
|
||||
trackAudioDeviceUnavailable({
|
||||
stt_provider_id: providerId,
|
||||
error_code: errorCode,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
@@ -515,6 +563,11 @@ export const useHearingSpeechInputPipeline = defineStore('modules:hearing:speech
|
||||
const hearingStore = useHearingStore()
|
||||
const { activeTranscriptionProvider, activeTranscriptionModel } = storeToRefs(hearingStore)
|
||||
const providersStore = useProvidersStore()
|
||||
const {
|
||||
trackAudioDeviceUnavailable,
|
||||
trackVoiceInputCancelled,
|
||||
trackVoiceInputStarted,
|
||||
} = useAnalytics()
|
||||
const streamingSession = shallowRef<{
|
||||
audioContext: AudioContext | Record<string, never>
|
||||
workletNode: AudioWorkletNode | Record<string, never>
|
||||
@@ -765,6 +818,8 @@ export const useHearingSpeechInputPipeline = defineStore('modules:hearing:speech
|
||||
|
||||
// Special handling for Web Speech API - it works directly with MediaStream
|
||||
if (providerId === 'browser-web-speech-api') {
|
||||
trackVoiceInputStarted({ stt_provider_id: providerId })
|
||||
|
||||
// Check if Web Speech API is available
|
||||
const isAvailable = typeof window !== 'undefined'
|
||||
&& ('webkitSpeechRecognition' in window || 'SpeechRecognition' in window)
|
||||
@@ -1063,11 +1118,16 @@ export const useHearingSpeechInputPipeline = defineStore('modules:hearing:speech
|
||||
|
||||
if (!recording) {
|
||||
error.value = 'No recording captured from microphone'
|
||||
trackVoiceInputCancelled({ stt_provider_id: activeTranscriptionProvider.value || 'unknown' })
|
||||
return
|
||||
}
|
||||
|
||||
if (recording.size <= 0) {
|
||||
error.value = 'Recording captured from microphone is empty'
|
||||
trackAudioDeviceUnavailable({
|
||||
stt_provider_id: activeTranscriptionProvider.value || 'unknown',
|
||||
error_code: 'device_unavailable',
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
|
||||
@@ -408,6 +408,7 @@ export const useSpeechStore = defineStore('speech', () => {
|
||||
? withAiriTtsAnalytics(providerConfig, {
|
||||
trigger: 'manual',
|
||||
source: 'manual_preview',
|
||||
voice_type: resolveVoiceType(voice),
|
||||
})
|
||||
: providerConfig
|
||||
const response = await generateSpeech({
|
||||
@@ -421,7 +422,11 @@ export const useSpeechStore = defineStore('speech', () => {
|
||||
|
||||
function withAiriTtsAnalytics(
|
||||
providerConfig: Record<string, any>,
|
||||
analytics: { trigger: 'auto' | 'manual', source: 'chat_auto_tts' | 'manual_preview' | 'settings_test' },
|
||||
analytics: {
|
||||
trigger: 'auto' | 'manual'
|
||||
source: 'chat_auto_tts' | 'manual_preview' | 'settings_test'
|
||||
voice_type?: 'official_default' | 'official_selected' | 'custom_configured' | 'voice_pack'
|
||||
},
|
||||
): Record<string, any> {
|
||||
return {
|
||||
...providerConfig,
|
||||
@@ -432,6 +437,14 @@ export const useSpeechStore = defineStore('speech', () => {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Classifies the active speech voice before forwarding analytics to the server.
|
||||
*/
|
||||
function resolveVoiceType(voiceId: string): 'official_selected' | 'custom_configured' {
|
||||
const catalogVoice = availableVoices.value[activeSpeechProvider.value]?.some(voice => voice.id === voiceId)
|
||||
return activeSpeechProvider.value === OFFICIAL_SPEECH_PROVIDER_ID && catalogVoice ? 'official_selected' : 'custom_configured'
|
||||
}
|
||||
|
||||
function generateSSML(
|
||||
text: string,
|
||||
voice: VoiceInfo,
|
||||
|
||||
@@ -24,7 +24,7 @@ import type { ProviderOnboardingField } from '../libs/providers/types'
|
||||
import type { AliyunRealtimeSpeechExtraOptions } from './providers/aliyun/stream-transcription'
|
||||
|
||||
import { errorMessageFrom } from '@moeru/std'
|
||||
import { isCustomProvidersDisabled, isStageTamagotchi, isUrl } from '@proj-airi/stage-shared'
|
||||
import { isCustomProvidersDisabled, isStageCapacitor, isStageTamagotchi, isUrl } from '@proj-airi/stage-shared'
|
||||
import { getCachedWebGPUCapabilities, isWebGPUSupported } from '@proj-airi/stage-shared/webgpu'
|
||||
import { computedAsync, useIntervalFn, useLocalStorage } from '@vueuse/core'
|
||||
import {
|
||||
@@ -55,6 +55,7 @@ import { getKokoroAdapter } from '../libs/inference/adapters/kokoro'
|
||||
import { getProviderValidationIntervalMs, listProviders as listDefinedProviders, ProviderValidationCheck } from '../libs/providers'
|
||||
import { resolveProviderSourceMetadata } from '../libs/providers/source-metadata'
|
||||
import { getDefaultKokoroModel, KOKORO_MODELS, kokoroModelsToModelInfo } from '../workers/kokoro/constants'
|
||||
import { capturePosthogEvent, ensurePosthogInitialized, isPosthogAvailableInBuild } from './analytics/posthog'
|
||||
import { useAuthStore } from './auth'
|
||||
import { createAliyunNLSProvider as createAliyunNlsStreamProvider } from './providers/aliyun/stream-transcription'
|
||||
import { convertProviderDefinitionsToMetadata } from './providers/converters'
|
||||
@@ -63,6 +64,7 @@ import { buildGoogleGeminiSpeechProvider } from './providers/google-gemini-speec
|
||||
import { buildOpenAICompatibleProvider } from './providers/openai-compatible-builder'
|
||||
import { buildOpenRouterAudioSpeechProvider } from './providers/openrouter/audio-speech'
|
||||
import { createWebSpeechAPIProvider } from './providers/web-speech-api'
|
||||
import { useSettingsAnalytics } from './settings/analytics'
|
||||
|
||||
const ALIYUN_NLS_REGIONS = [
|
||||
'cn-shanghai',
|
||||
@@ -80,6 +82,78 @@ function toListVoicesOptions<T>(provider: VoiceProviderWithExtraOptions<T>, opti
|
||||
return voiceOptions
|
||||
}
|
||||
|
||||
/**
|
||||
* Classifies provider ids into bounded analytics buckets.
|
||||
*/
|
||||
function analyticsProviderMode(providerId: string): 'official' | 'custom' | 'unknown' {
|
||||
if (!providerId)
|
||||
return 'unknown'
|
||||
return providerId.startsWith('official-provider') || providerId.startsWith('vision-official-provider') ? 'official' : 'custom'
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the current app surface without importing the analytics store.
|
||||
*/
|
||||
function analyticsSurface(): 'web' | 'mobile' | 'electron' {
|
||||
if (isStageTamagotchi())
|
||||
return 'electron'
|
||||
|
||||
if (isStageCapacitor())
|
||||
return 'mobile'
|
||||
|
||||
return 'web'
|
||||
}
|
||||
|
||||
/**
|
||||
* Checks analytics settings and initializes PostHog without loading build metadata.
|
||||
*/
|
||||
function canCaptureProviderAnalytics(): boolean {
|
||||
if (!isPosthogAvailableInBuild())
|
||||
return false
|
||||
|
||||
const settingsAnalytics = useSettingsAnalytics()
|
||||
if (!settingsAnalytics.analyticsEnabled)
|
||||
return false
|
||||
|
||||
return ensurePosthogInitialized(true)
|
||||
}
|
||||
|
||||
/**
|
||||
* Emits model-list analytics from the provider store without loading build metadata.
|
||||
*/
|
||||
function trackModelListLoaded(properties: {
|
||||
provider_id: string
|
||||
provider_mode: 'official' | 'custom' | 'unknown'
|
||||
model_count: number
|
||||
duration_ms: number
|
||||
}) {
|
||||
if (!canCaptureProviderAnalytics())
|
||||
return
|
||||
|
||||
capturePosthogEvent('model_list_loaded', {
|
||||
...properties,
|
||||
surface: analyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Emits model-list failure analytics from the provider store without loading build metadata.
|
||||
*/
|
||||
function trackModelListFailed(properties: {
|
||||
provider_id: string
|
||||
provider_mode: 'official' | 'custom' | 'unknown'
|
||||
error_code: string
|
||||
duration_ms: number
|
||||
}) {
|
||||
if (!canCaptureProviderAnalytics())
|
||||
return
|
||||
|
||||
capturePosthogEvent('model_list_failed', {
|
||||
...properties,
|
||||
surface: analyticsSurface(),
|
||||
})
|
||||
}
|
||||
|
||||
export interface ProviderMetadata {
|
||||
id: string
|
||||
to?: string
|
||||
@@ -2566,6 +2640,7 @@ export const useProvidersStore = defineStore('providers', () => {
|
||||
|
||||
// Function to fetch models for a specific provider
|
||||
async function fetchModelsForProvider(providerId: string) {
|
||||
const startedAt = Date.now()
|
||||
const metadata = providerMetadata[providerId]
|
||||
if (!metadata)
|
||||
return []
|
||||
@@ -2594,8 +2669,20 @@ export const useProvidersStore = defineStore('providers', () => {
|
||||
deprecated: model.deprecated,
|
||||
provider: providerId,
|
||||
}))
|
||||
trackModelListLoaded({
|
||||
provider_id: providerId,
|
||||
provider_mode: analyticsProviderMode(providerId),
|
||||
model_count: runtimeState.models.length,
|
||||
duration_ms: Date.now() - startedAt,
|
||||
})
|
||||
return runtimeState.models
|
||||
}
|
||||
trackModelListLoaded({
|
||||
provider_id: providerId,
|
||||
provider_mode: analyticsProviderMode(providerId),
|
||||
model_count: 0,
|
||||
duration_ms: Date.now() - startedAt,
|
||||
})
|
||||
return []
|
||||
}
|
||||
catch (error) {
|
||||
@@ -2603,6 +2690,12 @@ export const useProvidersStore = defineStore('providers', () => {
|
||||
if (runtimeState) {
|
||||
runtimeState.modelLoadError = errorMessageFrom(error) ?? 'Unknown error'
|
||||
}
|
||||
trackModelListFailed({
|
||||
provider_id: providerId,
|
||||
provider_mode: analyticsProviderMode(providerId),
|
||||
error_code: 'provider_error',
|
||||
duration_ms: Date.now() - startedAt,
|
||||
})
|
||||
return []
|
||||
}
|
||||
finally {
|
||||
|
||||
Reference in New Issue
Block a user