mirror of
https://github.com/moeru-ai/airi.git
synced 2026-08-14 08:52:42 +00:00
feat(server): new metrics for auth
- Added a new ObservableGauge for distinct active users to track real active user count, mitigating session row inflation issues. - Updated the Grafana dashboard to reflect changes, including the removal of redundant WS Connections panel and the addition of new metrics for active sessions and distinct users. - Improved documentation for verification automation processes, outlining a structured approach to automate verification steps and maintain evidence of tests.
This commit is contained in:
RainbowBird
@@ -50,6 +50,14 @@
|
||||
| LLM token / cost | **Grafana**（短期） | 后续若引入 Langfuse 则迁过去 |
|
||||
| 用户行为漏斗各步骤 | **PostHog**（必须） | 第一步通常是前端事件，Grafana 拿不到 |
|
||||
|
||||
### Better Auth session table 与活跃用户
|
||||
|
||||
`user_active_sessions`（COUNT(\*)）和 `user_distinct_active`（COUNT(DISTINCT user_id)）共享同一张 `session` 表：
|
||||
|
||||
- **Better Auth 每次 sign-in / 每次 OIDC access-token 颁发都新建一条 session row，从不主动 GC 过期 row**——因为 `oauth_access_token.session_id` FK 指向 session（`apps/server/src/libs/auth.ts:513` 注释）
|
||||
- 实战观察：~80K `user_active_sessions` 对应实际只有几百 distinct user。比例 5+ 就该考虑加 session GC cron 或缩短 Better Auth `expiresIn`
|
||||
- 永远展示 `user_distinct_active` 给非工程师看（PM、运营）；`user_active_sessions` 留给工程师 debug
|
||||
|
||||
### Dashboard 标注规则
|
||||
|
||||
两边都展示的指标，**必须**在 Grafana panel description 和 PostHog insight description 里：
|
||||
@@ -95,7 +103,7 @@
|
||||
| WS | `ws_connections_active` / `ws_messages_*_total` | Grafana | |
|
||||
| LLM | `gen_ai_client_operation_count_total` / `gen_ai_client_first_token_duration_seconds` | Grafana | |
|
||||
| Billing | `airi_billing_flux_unbilled_total` | Grafana | **告警必须**：`increase(airi_billing_flux_unbilled_total[5m]) > 0` |
|
||||
| Auth | `user_active_sessions` | Postgres → Grafana 派生 | 集群级 gauge，用 `avg()` 不要 `sum()` |
|
||||
| Auth | `user_active_sessions` / `user_distinct_active` | Postgres → Grafana 派生 | 集群级 gauge，用 `avg()` 不要 `sum()`。两个一起看：`user_active_sessions` = `COUNT(*)`（session row 数，会膨胀）, `user_distinct_active` = `COUNT(DISTINCT user_id)`（真实活跃用户数）|
|
||||
| Stripe | `airi_stripe_revenue_minor_unit_total` / `stripe_events_total` | Postgres → 两边展示 | Grafana 是系统侧 webhook 计数 |
|
||||
| Runtime | `v8js_memory_*` / `nodejs_eventloop_delay_*` | Grafana | per `service_instance_id` |
|
||||
| Rate-limit | `airi_rate_limit_blocked_total` | Grafana | in-memory per replica |
|
||||
@@ -217,6 +225,30 @@ PostHog UI 配 cohort：
|
||||
|
||||
不能只用 source connector：它是 data warehouse 层，**不生成 person event，做不了漏斗**。
|
||||
|
||||
## 5xx Triage 路径
|
||||
|
||||
Dashboard 上 follow 这条 panel 链可以从"出事了"一路 drill 到"哪个 trace 是真凶"：
|
||||
|
||||
1. **panel-4 `5xx Rate %`**（Row 1）— 数字 / gauge 颜色变红，说明出事
|
||||
2. **panel-9 `Top Routes by 5xx`**（Row 2 donut）— "现在哪些 route 在失败"
|
||||
3. **panel-44 `5xx Rate by Route`**（Row 5.5 timeseries）— "什么时候开始的、是单点还是普遍"
|
||||
4. **panel-91 `5xx Error Logs`**（Row 8 上半）— 实际错误消息，里面有 `trace_id` field 可点 → Tempo 看完整 trace 回放
|
||||
|
||||
### Tempo / Loki derived fields 配置（一次性）
|
||||
|
||||
panel-91 的 `trace_id` 字段必须配 Grafana Cloud Loki datasource 的 **Derived fields** 才能跳 Tempo。这不在 dashboard JSON 范围内，是 datasource 级配置：
|
||||
|
||||
- **Grafana Cloud** → Connections → Data sources → 选 `grafanacloud-projairi-logs`（Loki）→ Derived fields
|
||||
- 添加：
|
||||
- **Name**: `trace_id`
|
||||
- **Type**: Regex in label or value
|
||||
- **Regex**: `"trace_id":"([a-f0-9]+)"`（匹配我们 logger 的 JSON 输出）
|
||||
- **URL**: 留空
|
||||
- **Internal link**: ✓，datasource 选 `grafanacloud-projairi-traces`（Tempo）
|
||||
- 同样手法可加 `req` (request id) → 配 internal link 回 Loki 自身，按 requestId filter
|
||||
|
||||
配置完之后日志面板里 `trace_id` 会变成蓝色可点，直接跳 Tempo waterfall。这一步配置只做一次，新加 panel 自动享有。
|
||||
|
||||
## Grafana Alert SOP
|
||||
|
||||
Alert rules **不放在** `apps/server/otel/grafana/dashboards/build.ts` 里——Grafana Cloud 用 Unified Alerting，rule 在 Grafana UI 或 alerting API 管理，跟 dashboard JSON 解耦。这一节维护我们应该配的 alert rule，新加 rule 时同步更新这里。
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
# Verification: Flux Unbilled Exploit Fix
|
||||
|
||||
Status: **patched in commit `7267b0d6b`** (2026-05-15) for the chat-completion path. TTS flux-meter adaptation followed up in the same PR as this verification doc (see "Remaining gaps → Gap 1").
|
||||
Last attempted: 2026-05-15
|
||||
Owner: rbxin2003@gmail.com
|
||||
|
||||
## 用户路径
|
||||
|
||||
- **场景**：用户 `0 < balance < fallbackRate`，开 N 个并发 LLM completion 请求
|
||||
- **预期（修补后）**：第一个 request 之后 pre-flight 拒绝；最多触发一次 partial debit（`charged < requested`），所有剩余 request 收 402
|
||||
- **实际（修补前）**：N 个 request 全部走到 stream end，每个 catch `consumeFluxForLLM` 失败回滚 → balance 不变 → 用户拿到 N 次免费 LLM 响应
|
||||
|
||||
## Before / After 行为
|
||||
|
||||
### Before（修补前漏洞链）
|
||||
|
||||
1. Pre-flight 仅 `if (flux.flux <= 0)`（`apps/server/src/routes/openai/v1/index.ts`，旧版）
|
||||
2. 用户余额 1 flux，但 `FLUX_PER_REQUEST` (fallback) 通常远大于 1
|
||||
3. 并发 N 请求全部通过 pre-flight，上游 LLM 全部完成（响应已 stream 出去）
|
||||
4. `consumeFluxForLLM` 走 `debitFlux` (`apps/server/src/services/billing/billing-service.ts:107`)，旧逻辑：
|
||||
```ts
|
||||
if (balanceBefore < input.amount) {
|
||||
throw createPaymentRequiredError('Insufficient flux')
|
||||
}
|
||||
```
|
||||
整个 DB tx 回滚 → balance 维持 1 flux
|
||||
5. catch 路径上报 `airi_billing_flux_unbilled_total{reason='debit_failed'}` += full amount
|
||||
6. 用户重复同样的脚本，每次都拿免费响应
|
||||
|
||||
这是 Grafana panel-43 (Flux Unbilled) 累积到 **70.2K** 的根因。
|
||||
|
||||
### After（commit `7267b0d6b`）
|
||||
|
||||
**1. Pre-flight 阈值改为 fallbackRate**（`apps/server/src/routes/openai/v1/index.ts:138-142`）：
|
||||
|
||||
```ts
|
||||
const fallbackRate = await configKV.getOrThrow('FLUX_PER_REQUEST')
|
||||
const fluxPer1kTokens = await configKV.get('FLUX_PER_1K_TOKENS')
|
||||
|
||||
const flux = await fluxService.getFlux(user.id)
|
||||
if (flux.flux < fallbackRate) {
|
||||
throw createPaymentRequiredError('Insufficient flux')
|
||||
}
|
||||
```
|
||||
|
||||
并发 N 请求时，pre-flight 直接 402 拒绝，不会进入上游 LLM 调用。
|
||||
|
||||
**2. Partial-debit 语义**（`apps/server/src/services/billing/billing-service.ts:107-130`）：
|
||||
|
||||
```ts
|
||||
if (balanceBefore <= 0) {
|
||||
metrics?.fluxInsufficientBalance.add(1)
|
||||
throw createPaymentRequiredError('Insufficient flux')
|
||||
}
|
||||
|
||||
const chargedAmount = Math.min(input.amount, balanceBefore)
|
||||
const balanceAfter = balanceBefore - chargedAmount
|
||||
const isPartial = chargedAmount < input.amount
|
||||
```
|
||||
|
||||
- `balance > 0` 但不够 → drain 到 0，ledger 记 `amount = charged`，metadata 带 `requestedAmount + unbilled`
|
||||
- `balance <= 0` 才 throw（catch path 上报 `reason='debit_failed'`）
|
||||
- Partial drain 上报 `reason='partial_debit_drained'`，跟真实 DB 错误区分
|
||||
|
||||
**3. Idempotency 反映原始 charged**（`billing-service.ts:71-94`）：
|
||||
|
||||
替换 request 的同 requestId 重放复用历史 `existing.amount` 作为 `charged`，避免重试时双扣 unbilled counter。
|
||||
|
||||
## Evidence
|
||||
|
||||
| Item | Reference |
|
||||
|---|---|
|
||||
| Fix commit | `7267b0d6b feat(server/billing): partial-debit semantics to prevent unpaid usage exploit` |
|
||||
| Pre-flight gate | `apps/server/src/routes/openai/v1/index.ts:138-142` |
|
||||
| Partial-debit logic | `apps/server/src/services/billing/billing-service.ts:107-148` |
|
||||
| Streaming path unbilled metric | `apps/server/src/routes/openai/v1/index.ts:299-313` (label `reason='partial_debit_drained'`, `stage='streaming'`) |
|
||||
| Non-streaming path unbilled metric | `apps/server/src/routes/openai/v1/index.ts:374-388` (label `stage='non_streaming'`) |
|
||||
| Regression tests added | `apps/server/src/routes/openai/v1/route.test.ts` (+97 lines), `apps/server/src/services/billing/tests/billing-service.test.ts` (+89 lines) |
|
||||
|
||||
测试覆盖的两个核心 case：
|
||||
|
||||
- `'rejects pre-flight when balance is below FLUX_PER_REQUEST (Issue: unpaid-usage-exploit)'` — 验证 pre-flight 在 partial-balance 用户上 reject，upstream 未被调用
|
||||
- `'non-streaming completion drains partial balance and logs charged (Issue: unpaid-usage-exploit)'` — 验证 partial-debit drain 到 zero + metric 上报
|
||||
|
||||
## Remaining gaps
|
||||
|
||||
### Gap 1 — TTS flux-meter partial-debit 适配 ✅ 已修复
|
||||
|
||||
`apps/server/src/services/billing/flux-meter.ts:135-228` 的 `accumulate()` 现在解构 `{ charged, requested }`，partial drain 时：
|
||||
|
||||
- 上报 `airi_billing_flux_unbilled_total{source='tts_meter', reason='partial_debit_drained', meter, gen_ai.request.model?}`
|
||||
- `INCRBY` `(requested - charged) * unitsPerFlux` 回 Redis debt counter
|
||||
- `AccumulateResult` 增加 `unbilledFlux` 字段供调用方观测
|
||||
- 加 invariant 校验 `charged > requested` / 非整数 / 负数 → throw 而不是静默 under-restore
|
||||
|
||||
测试覆盖：`flux-meter.test.ts:217-260`，case 名带 `Issue: unpaid-usage-exploit follow-up`。
|
||||
|
||||
**残余风险（已知，留 follow-up）**：settlement (LUA `runScript`) 和 `INCRBY` restore 之间非原子，并发请求理论上可能在窗口内读到 mid-state。窗口很小（一个 DB tx），实际命中很难触发；长期修复需要 Redis lock 或 unbilled 单独 bucket。代码里 `flux-meter.ts` 有 `// REVIEW:` 标记。
|
||||
|
||||
### Gap 2 — 修补前的 70.2K 历史漏账未核销
|
||||
|
||||
panel-43 显示的 70.2K 是 counter 累积值（`increase($__range)`），代表修补前漏出去的总量。修补**不会让 panel 自动归零**，只会让新的增量趋近 0（除真实 DB 错误）。
|
||||
|
||||
**建议**：
|
||||
- 把 dashboard 时间窗调到 commit `7267b0d6b` 部署后（2026-05-15 之后）观察增量斜率
|
||||
- 加 Grafana alert：`increase(airi_billing_flux_unbilled_total[5m]) > 0` → PagerDuty
|
||||
- 如果业务需要核销历史 70.2K，从 `flux_transaction` ledger 反查 `metadata->>'reason' = 'debit_failed'` 的记录，配合 Loki 错误日志定位涉事 userId
|
||||
|
||||
## What's verified
|
||||
|
||||
- ✓ 代码层：pre-flight gate + partial-debit semantics 确实改了，逻辑正确
|
||||
- ✓ 测试层：两个核心 regression test 覆盖 exploit 场景，case 名带 `Issue: unpaid-usage-exploit` 标识
|
||||
|
||||
## What's pending live verification
|
||||
|
||||
- ⊘ 生产 Grafana 上 panel-43 在 `2026-05-15` commit 部署后的斜率趋近 0
|
||||
- ⊘ 没跑 `pnpm -F @proj-airi/server exec vitest run` 实际确认新测试 pass（建议在 push 之前跑一次）
|
||||
- ⊘ TTS partial-debit 适配的 live verification（代码已修补 + 单测 14/14 pass，需要在生产观察 `airi_billing_flux_unbilled_total{source='tts_meter'}` 出现合理流量后再 close）
|
||||
|
||||
## Recommended follow-ups
|
||||
|
||||
按优先级：
|
||||
|
||||
1. **P0 — 加 Grafana alert** `increase(airi_billing_flux_unbilled_total[5m]) > 0` → 通知
|
||||
2. **P1 — 历史 70.2K 漏账处理**：查 ledger + Loki 决定核销还是补账
|
||||
3. ~~**P1 — 修 TTS flux-meter 适配 partial-debit 新语义**~~ ✅ 已修复，见 Gap 1
|
||||
4. **P2 — Dashboard 改造**（panel-43 时间窗注解 + 区分 `reason` label 的 stack 图，把 `partial_debit_drained` 跟 `debit_failed` 分色展示）
|
||||
@@ -0,0 +1,195 @@
|
||||
# Verification: Flux Unbilled Historical Reconciliation
|
||||
|
||||
Status: **investigation framework — data gathering pending**
|
||||
Owner: rbxin2003@gmail.com
|
||||
Last updated: 2026-05-15
|
||||
Related: [`flux-unbilled-exploit-fix.md`](./flux-unbilled-exploit-fix.md), [Grafana panel-43](../../../otel/grafana/dashboards/airi-server-overview-cloud.json)
|
||||
|
||||
## 用户路径
|
||||
|
||||
- **场景**：commit `7267b0d6b` 之前累积了 ~70.2K Flux 的 unpaid usage（panel-43 `airi_billing_flux_unbilled_total` 显示值）。需要决定核销、补账、还是不处理
|
||||
- **预期**：跑下面的 SQL + Loki query → 区分 partial-drain（用户已部分付款）vs debit-failed（DB 错误，真零付款）→ 按 user 聚合 → 给出 reconciliation 决策
|
||||
- **当前状态**：没有 prod DB 访问权限的工程师跑下面的 query。下方 SQL/queries 是**待执行的模板**，不是已采集的数据
|
||||
|
||||
## 两类漏账的区分
|
||||
|
||||
修补前 `airi_billing_flux_unbilled_total` 是单一 counter，没区分 reason。修补后（`7267b0d6b`）按 `reason` 拆成两个 label：
|
||||
|
||||
| reason label | 触发条件 | Ledger 是否有记录 | 用户实际付款比例 |
|
||||
|---|---|---|---|
|
||||
| `partial_debit_drained` | `0 < balance < amount`，drain 到 0 | ✓ 有（`amount = charged`，metadata 带 `unbilled`） | 部分付款（drain 数额） |
|
||||
| `debit_failed` | `balance <= 0` 或 DB tx 抛错 | ✗ 无（tx 回滚） | 零付款 |
|
||||
|
||||
**70.2K 全部发生在 5/15 之前**，那时所有失败都走 catch path → 全部记为 `reason='debit_failed'`，**全部无 ledger row** → 用户实际付款为 0。
|
||||
|
||||
但实际不全是漏洞：少部分是真正的 DB 错误（DB outage / 唯一索引冲突）。绝大部分是 exploit。
|
||||
|
||||
## 取证 SQL（待执行）
|
||||
|
||||
> 在 Railway Postgres console 或本地 `psql $DATABASE_URL` 跑。如果 query 太重，先 `EXPLAIN ANALYZE` 看 cost；`flux_transaction` 有 `flux_tx_user_id_idx` 和 `flux_tx_created_at_idx` 索引可以走
|
||||
|
||||
### 1. 按 type 分类的 ledger 写入分布
|
||||
|
||||
`reason='debit_failed'` 没 ledger row，所以这条 query **拿不到**修补前的漏洞数据——它只能 sanity check 修补后的新 row：
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
type,
|
||||
COUNT(*) AS row_count,
|
||||
SUM(amount) AS total_amount,
|
||||
MIN(created_at) AS first_seen,
|
||||
MAX(created_at) AS last_seen
|
||||
FROM flux_transaction
|
||||
WHERE created_at >= '2026-04-15' -- 4 周窗口
|
||||
GROUP BY type
|
||||
ORDER BY total_amount DESC;
|
||||
```
|
||||
|
||||
### 2. Partial-drain ledger rows（修补后）
|
||||
|
||||
`commit 7267b0d6b` 之后才会有这种 row。修补前漏出去的 70K 在这里**看不到**：
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
user_id,
|
||||
COUNT(*) AS partial_debit_count,
|
||||
SUM(amount) AS total_charged,
|
||||
SUM((metadata->>'unbilled')::bigint) AS total_unbilled,
|
||||
SUM((metadata->>'requestedAmount')::bigint) AS total_requested,
|
||||
MIN(created_at) AS first_partial,
|
||||
MAX(created_at) AS last_partial
|
||||
FROM flux_transaction
|
||||
WHERE type = 'debit'
|
||||
AND metadata ? 'unbilled'
|
||||
AND (metadata->>'unbilled')::bigint > 0
|
||||
AND created_at >= '2026-05-15' -- 修补后窗口
|
||||
GROUP BY user_id
|
||||
ORDER BY total_unbilled DESC
|
||||
LIMIT 50;
|
||||
```
|
||||
|
||||
### 3. 流量最高的用户（用来定位 exploit 嫌疑）
|
||||
|
||||
修补前的漏账主要靠这条 + Loki 日志交叉定位涉事 user：
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
user_id,
|
||||
COUNT(*) AS debit_count,
|
||||
SUM(amount) AS total_debited,
|
||||
SUM(balance_after - balance_before) AS net_balance_change,
|
||||
MIN(created_at) AS first_debit,
|
||||
MAX(created_at) AS last_debit
|
||||
FROM flux_transaction
|
||||
WHERE type = 'debit'
|
||||
AND created_at BETWEEN '2026-05-01' AND '2026-05-15' -- 修补前 2 周
|
||||
GROUP BY user_id
|
||||
HAVING COUNT(*) > 100
|
||||
ORDER BY debit_count DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
异常用户特征：`debit_count` 极高 + `net_balance_change` 接近 0 即"balance 一直被推到底但没归零"。这种是 exploit 的核心 signature——攻击者维持 balance 卡在 `0 < x < fallbackRate` 区间反复触发免费请求。注意：`net_balance_change` 在 ledger 模型下应该等于 `-SUM(amount)`；如果两者接近 0 但 `SUM(amount)` 很大，说明 balance 被人工补回去过（信用 / 充值 / promo），需要进一步交叉检查。
|
||||
|
||||
### 4. 当前 user_flux 余额 vs ledger 一致性 sanity
|
||||
|
||||
```sql
|
||||
WITH ledger_balance AS (
|
||||
SELECT
|
||||
user_id,
|
||||
SUM(CASE WHEN type = 'credit' OR type = 'initial' OR type = 'promo' THEN amount
|
||||
WHEN type = 'debit' THEN -amount
|
||||
ELSE 0
|
||||
END) AS computed_balance
|
||||
FROM flux_transaction
|
||||
GROUP BY user_id
|
||||
)
|
||||
SELECT
|
||||
uf.user_id,
|
||||
uf.flux AS recorded_balance,
|
||||
lb.computed_balance AS ledger_sum,
|
||||
uf.flux - lb.computed_balance AS drift
|
||||
FROM user_flux uf
|
||||
LEFT JOIN ledger_balance lb USING (user_id)
|
||||
WHERE ABS(uf.flux - COALESCE(lb.computed_balance, 0)) > 0
|
||||
ORDER BY ABS(uf.flux - COALESCE(lb.computed_balance, 0)) DESC
|
||||
LIMIT 50;
|
||||
```
|
||||
|
||||
正常情况下 drift 应该是 0——任何 drift 都说明 ledger 和 user_flux 表脱钩了，是 P0 事件。
|
||||
|
||||
## 取证 Loki（待执行）
|
||||
|
||||
Grafana → Explore → Loki datasource。这是**修补前漏账数据的唯一来源**（无 ledger row）：
|
||||
|
||||
### 漏账 error log 全量
|
||||
|
||||
```logql
|
||||
{service_name="server"} |= "Failed to debit flux after streaming — unpaid usage"
|
||||
| json
|
||||
| line_format "{{.userId}} | req={{.requestId}} | flux={{.fluxConsumed}} | {{.error}}"
|
||||
```
|
||||
|
||||
### 按 userId 聚合 unbilled 数量
|
||||
|
||||
```logql
|
||||
sum by (userId) (
|
||||
count_over_time({service_name="server"} |= "Failed to debit flux after streaming" | json [30d])
|
||||
)
|
||||
```
|
||||
|
||||
### 时间分布（找爆发时段）
|
||||
|
||||
```logql
|
||||
sum (
|
||||
rate({service_name="server"} |= "Failed to debit flux after streaming" | json [5m])
|
||||
)
|
||||
```
|
||||
|
||||
修补前若有 sustained > 0 → exploit；若是窄峰 → 真实 DB outage。
|
||||
|
||||
## 处理决策框架
|
||||
|
||||
跑完上面 query + Loki 后，按下面决策树走：
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────┐
|
||||
│ 单用户漏账 fluxConsumed > 1000? │
|
||||
├──────────────────────────────────────────┤
|
||||
│ YES → exploit 嫌疑 │
|
||||
│ ├─ 多 IP / 短时间高频 → confirmed │
|
||||
│ │ │ 不补账（用户已知道是漏洞） │
|
||||
│ │ │ Ban user 或 require email │
|
||||
│ │ │ verification + 强制 reauth │
|
||||
│ │ │ 已修补 → 单纯历史损失 │
|
||||
│ │ └─ 不需要在 Postgres 写新 ledger │
|
||||
│ └─ 单 IP / 时间分散 → 可能正常 power user │
|
||||
│ │ 主动联系用户，问明情况 │
|
||||
│ └─ 视情况决定是否赠送 flux 补偿 │
|
||||
│ │
|
||||
│ NO (用户总漏账 < 1000 flux) → 真异常 │
|
||||
│ │ 多半是 DB outage / 单次错误 │
|
||||
│ │ 不值得逐个追账 │
|
||||
│ └─ 整体核销 + 跑 sanity SQL 4 验证 │
|
||||
│ user_flux ≡ ledger 仍然一致 │
|
||||
└──────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**关键判断**：修补前的漏账**不在 ledger 里**（debit_failed 不写 row），所以**不需要在 DB 做任何"核销"操作**——余额是干净的，损失只是"曾经免费送出去的 LLM token 成本"。
|
||||
|
||||
唯一需要写 DB 操作的场景：sanity SQL #4 跑出非零 drift。那是另一个 bug（ledger ↔ user_flux 脱钩），跟漏账无关。
|
||||
|
||||
## 修补后的监控建议（持续）
|
||||
|
||||
1. 加 Grafana alert：`increase(airi_billing_flux_unbilled_total{reason!="partial_debit_drained"}[5m]) > 0` → 立即 page（partial drain 是合理路径，不 page）
|
||||
2. 加每周自动 cron job 跑 sanity SQL #4，drift > 0 → 报警（注意：项目里**不允许**新加后台 worker / cron，所以这个 job 应该走外部 ops 工具，比如 Railway scheduled command 或 GitHub Action）
|
||||
|
||||
## What's verified / What's pending
|
||||
|
||||
| Item | Status |
|
||||
|---|---|
|
||||
| 漏洞已堵（commit `7267b0d6b`） | ✓ 已确认（见 `flux-unbilled-exploit-fix.md`） |
|
||||
| 70.2K 历史漏账的 user 分布 | ⊘ 待跑 Loki query |
|
||||
| user_flux ↔ ledger drift 是否存在 | ⊘ 待跑 SQL #4 |
|
||||
| Exploit 涉事 user 是否已 ban / re-auth | ⊘ 等数据出来后决定 |
|
||||
| `airi_billing_flux_unbilled_total{reason!="partial_debit_drained"}` Grafana alert | ⊘ 待配（见 `metrics-ownership.md` Alert SOP） |
|
||||
@@ -41,8 +41,8 @@
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"expr": "avg(user_active_sessions{service_name=~\"$service\", deployment_environment=~\"$env\"})",
|
||||
"legendFormat": "sessions"
|
||||
"expr": "avg(user_distinct_active{service_name=~\"$service\", deployment_environment=~\"$env\"})",
|
||||
"legendFormat": "users"
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
@@ -54,7 +54,7 @@
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "Currently active sessions in Postgres (Better Auth `session.expires_at > now()`). Cluster-wide gauge — every replica polls the same DB on a 10s cache. We aggregate with `avg()` (not `sum()`, which would multiply by replica count; not `max()`, which biases high when one replica's cache is fresher than another's after a logout).",
|
||||
"description": "COUNT(DISTINCT user_id) over the Better Auth `session` table where `expires_at > now()`. This is the *real* active-user count. The historical \"Active Users\" panel queried `user.active_sessions` (COUNT(*) on the same table), which counts session **rows** not users — Better Auth creates a new row per sign-in and per OIDC access-token issuance and never GCs expired rows, so the row count drifts up and we have seen it report ~80K on a deployment with hundreds of actual users. Compare with `panel-15` (Active Sessions) to spot session-row inflation; ratio > ~5 means it's time for a session GC cron. Cluster-wide gauge — `avg()`, not `sum()`.",
|
||||
"id": 1,
|
||||
"links": [],
|
||||
"title": "Active Users",
|
||||
@@ -106,7 +106,7 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"panel-2": {
|
||||
"panel-15": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"data": {
|
||||
@@ -124,8 +124,8 @@
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"expr": "sum(ws_connections_active{service_name=~\"$service\", deployment_environment=~\"$env\"})",
|
||||
"legendFormat": "connections"
|
||||
"expr": "avg(user_active_sessions{service_name=~\"$service\", deployment_environment=~\"$env\"})",
|
||||
"legendFormat": "sessions"
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
@@ -137,10 +137,10 @@
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "Live registry size from chat-ws (ObservableGauge, scraped each export interval).",
|
||||
"id": 2,
|
||||
"description": "COUNT(*) over the Better Auth `session` table where `expires_at > now()`. Counts session **rows**, not users — see `panel-1` for the de-duplicated user count. Useful as a denominator to spot row inflation: divide by panel-1 to get rows-per-user, watch for sustained growth.",
|
||||
"id": 15,
|
||||
"links": [],
|
||||
"title": "WS Connections",
|
||||
"title": "Active Sessions",
|
||||
"vizConfig": {
|
||||
"group": "stat",
|
||||
"kind": "VizConfig",
|
||||
@@ -156,6 +156,10 @@
|
||||
{
|
||||
"color": "green",
|
||||
"value": 0
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 5000
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -216,7 +220,7 @@
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "5-minute average inbound HTTP request rate. /health (Railway probe) is excluded at the @hono/otel middleware level so this reflects real user traffic.",
|
||||
"description": "5-minute average inbound HTTP request rate. /health (Railway probe) is excluded at the @hono/otel middleware level so this reflects real user traffic. **Fixed 5m window — intentionally does not follow the dashboard time picker** (see row-level note). For trends, see panel-14 (HTTP Request Rate by Route).",
|
||||
"id": 3,
|
||||
"links": [],
|
||||
"title": "Req/s (5m)",
|
||||
@@ -304,7 +308,7 @@
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "5xx responses ÷ all responses over the last 5m. Spikes correlate with deploys, upstream outages, or DB problems. >1% warns, >5% pages.",
|
||||
"description": "5xx responses ÷ all responses over the last 5m. **Fixed 5m window — intentionally does not follow the dashboard time picker**: this is an on-call glance (\"is the service failing right now\"). For range-aware triage use panel-9 donut and panel-44 timeseries. Spikes correlate with deploys, upstream outages, or DB problems. >1% warns, >5% pages.",
|
||||
"id": 4,
|
||||
"links": [],
|
||||
"title": "5xx Rate %",
|
||||
@@ -392,7 +396,7 @@
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "5-minute average LLM gateway request rate (chat + tts).",
|
||||
"description": "5-minute average LLM gateway request rate (chat + tts). **Fixed 5m window — see row-level note.** For trends and per-model breakdown see panel-11.",
|
||||
"id": 5,
|
||||
"links": [],
|
||||
"title": "LLM Req/s (5m)",
|
||||
@@ -472,7 +476,7 @@
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "Email failures ÷ total attempts over the last 5m. >5% means Resend / DNS / suppression-list problems blocking auth flows.",
|
||||
"description": "Email failures ÷ total attempts over the last 5m. **Fixed 5m window — see row-level note.** >5% means Resend / DNS / suppression-list problems blocking auth flows.",
|
||||
"id": 6,
|
||||
"links": [],
|
||||
"title": "Email Failure %",
|
||||
@@ -548,7 +552,7 @@
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"expr": "topk(8, sum by (gen_ai_request_model) (increase(gen_ai_client_operation_count_total{service_name=~\"$service\", deployment_environment=~\"$env\", gen_ai_request_model!=\"\"}[5m])))",
|
||||
"expr": "topk(8, sum by (gen_ai_request_model) (increase(gen_ai_client_operation_count_total{service_name=~\"$service\", deployment_environment=~\"$env\", gen_ai_request_model!=\"\"}[$__range])))",
|
||||
"legendFormat": "{{gen_ai_request_model}}"
|
||||
},
|
||||
"version": "v0"
|
||||
@@ -561,10 +565,10 @@
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "Share of LLM gateway calls by model. Quickly shows which model is doing the heavy lifting.",
|
||||
"description": "Share of LLM gateway calls by model, summed over the dashboard time range. Follows the time picker — pick 1h to see the last hour's model mix, pick 7d to see this week's.",
|
||||
"id": 8,
|
||||
"links": [],
|
||||
"title": "LLM Models (last 5m)",
|
||||
"title": "LLM Models (range)",
|
||||
"vizConfig": {
|
||||
"group": "piechart",
|
||||
"kind": "VizConfig",
|
||||
@@ -639,7 +643,7 @@
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"expr": "topk(10, sum by (http_route) (increase(http_server_request_duration_seconds_count{service_name=~\"$service\", deployment_environment=~\"$env\", http_request_method!=\"OPTIONS\", http_route!=\"\"}[5m])))",
|
||||
"expr": "topk(10, sum by (http_route) (increase(http_server_request_duration_seconds_count{service_name=~\"$service\", deployment_environment=~\"$env\", http_request_method!=\"OPTIONS\", http_route!=\"\", http_response_status_code=~\"5..\"}[$__range])))",
|
||||
"legendFormat": "{{http_route}}"
|
||||
},
|
||||
"version": "v0"
|
||||
@@ -652,10 +656,10 @@
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "Top 10 Hono-matched routes by request count over the last 5 minutes. Answers \"which endpoint is being hit, and how much\" — replaces the previous HTTP status-code donut whose 2xx slice dominated everything else. Cardinality is bounded because `http_route` is the matched route pattern, not the concrete URL.",
|
||||
"description": "Top 10 Hono-matched routes by 5xx response count over the dashboard time range. Follows the time picker — pick 1h for \"what's failing right now\", pick 24h for \"what failed most today\". The overall 5xx% gauge (panel-4) is a fixed-5m snapshot for on-call glance; this donut respects the time picker for triage.",
|
||||
"id": 9,
|
||||
"links": [],
|
||||
"title": "Top Routes by Requests (last 5m)",
|
||||
"title": "Top Routes by 5xx (range)",
|
||||
"vizConfig": {
|
||||
"group": "piechart",
|
||||
"kind": "VizConfig",
|
||||
@@ -673,7 +677,7 @@
|
||||
}
|
||||
},
|
||||
"unit": "short",
|
||||
"noValue": "no traffic"
|
||||
"noValue": "no 5xx"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
@@ -1947,6 +1951,122 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"panel-44": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"data": {
|
||||
"kind": "QueryGroup",
|
||||
"spec": {
|
||||
"queries": [
|
||||
{
|
||||
"kind": "PanelQuery",
|
||||
"spec": {
|
||||
"hidden": false,
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "grafanacloud-projairi-prom"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"expr": "topk(10, sum by (http_route) (rate(http_server_request_duration_seconds_count{service_name=~\"$service\", deployment_environment=~\"$env\", http_request_method!=\"OPTIONS\", http_route!=\"\", http_response_status_code=~\"5..\"}[$__rate_interval])))",
|
||||
"legendFormat": "{{http_route}}"
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refId": "A"
|
||||
}
|
||||
}
|
||||
],
|
||||
"queryOptions": {},
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "5xx response rate split by route. Use this to confirm whether a 5xx spike in `panel-4` is concentrated on one endpoint (e.g. a broken deploy of /api/v1/openai/*) or scattered (e.g. DB outage taking down everything). Drill into the Logs row (`panel-91`) for the matching error bodies + trace ids.",
|
||||
"id": 44,
|
||||
"links": [],
|
||||
"title": "5xx Rate by Route (top 10)",
|
||||
"vizConfig": {
|
||||
"group": "timeseries",
|
||||
"kind": "VizConfig",
|
||||
"spec": {
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"barWidthFactor": 0.6,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 20,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 1,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "auto",
|
||||
"showValues": false,
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": 0
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "reqps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"annotations": {
|
||||
"clustering": -1,
|
||||
"multiLane": false
|
||||
},
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"lastNotNull",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "right",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"hideZeros": false,
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
}
|
||||
},
|
||||
"version": "13.0.0-23630096546"
|
||||
}
|
||||
}
|
||||
},
|
||||
"panel-30": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
@@ -2239,7 +2359,7 @@
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "PostgreSQL query duration P95 from PgInstrumentation. Spikes correlate with index misses, connection exhaustion, or backend lock contention.",
|
||||
"description": "PostgreSQL query duration P95 from PgInstrumentation. **Fixed 5m window — does not follow the dashboard time picker** (same posture as the Service Health row stats: this is a \"right now\" glance). Spikes correlate with index misses, connection exhaustion, or backend lock contention.",
|
||||
"id": 50,
|
||||
"links": [],
|
||||
"title": "DB Query P95 (5m)",
|
||||
@@ -2708,6 +2828,71 @@
|
||||
"version": "13.0.0-23630096546"
|
||||
}
|
||||
}
|
||||
},
|
||||
"panel-91": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"data": {
|
||||
"kind": "QueryGroup",
|
||||
"spec": {
|
||||
"queries": [
|
||||
{
|
||||
"kind": "PanelQuery",
|
||||
"spec": {
|
||||
"hidden": false,
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "grafanacloud-projairi-logs"
|
||||
},
|
||||
"group": "loki",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"expr": "{service_name=~\"$service\", deployment_environment=~\"$env\"} | json | level=~\"warn|error\"",
|
||||
"legendFormat": ""
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refId": "A"
|
||||
}
|
||||
}
|
||||
],
|
||||
"queryOptions": {},
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "Server-side error logs (level=warn|error) from Loki. Loki derived fields turn `trace_id` and `req` into clickable links — `trace_id` jumps to Tempo for full request playback (spans + child calls + DB queries), `req` filters this panel to a single request id. Use this together with panel-9 (which route) and panel-44 (when).",
|
||||
"id": 91,
|
||||
"links": [],
|
||||
"title": "5xx Error Logs",
|
||||
"vizConfig": {
|
||||
"group": "logs",
|
||||
"kind": "VizConfig",
|
||||
"spec": {
|
||||
"fieldConfig": {
|
||||
"defaults": {},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableInfiniteScrolling": false,
|
||||
"enableLogDetails": true,
|
||||
"prettifyLogMessage": false,
|
||||
"showCommonLabels": false,
|
||||
"showControls": false,
|
||||
"showFieldSelector": false,
|
||||
"showLabels": true,
|
||||
"showLevel": true,
|
||||
"showLogAttributes": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"timestampResolution": "ms",
|
||||
"unwrappedColumns": false,
|
||||
"wrapLogMessage": true
|
||||
}
|
||||
},
|
||||
"version": "13.0.0-23630096546"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"layout": {
|
||||
@@ -2740,7 +2925,7 @@
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-2"
|
||||
"name": "panel-15"
|
||||
},
|
||||
"height": 4,
|
||||
"width": 4,
|
||||
@@ -3045,6 +3230,33 @@
|
||||
"title": "Errors / Quality"
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "RowsLayoutRow",
|
||||
"spec": {
|
||||
"collapse": false,
|
||||
"layout": {
|
||||
"kind": "GridLayout",
|
||||
"spec": {
|
||||
"items": [
|
||||
{
|
||||
"kind": "GridLayoutItem",
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-44"
|
||||
},
|
||||
"height": 7,
|
||||
"width": 24,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"title": "5xx Triage"
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "RowsLayoutRow",
|
||||
"spec": {
|
||||
@@ -3177,13 +3389,26 @@
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-90"
|
||||
"name": "panel-91"
|
||||
},
|
||||
"height": 12,
|
||||
"height": 10,
|
||||
"width": 24,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "GridLayoutItem",
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-90"
|
||||
},
|
||||
"height": 10,
|
||||
"width": 24,
|
||||
"x": 0,
|
||||
"y": 10
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -355,28 +355,50 @@ function row(title: string, items: ReturnType<typeof item>[], { collapse = false
|
||||
// between defined panel ids and layout references.
|
||||
const elements: Record<string, unknown> = {}
|
||||
|
||||
// Row 1: Service Health — answers "is anything broken right now?"
|
||||
// Row 1: Service Health — answers "is anything broken **right now**?"
|
||||
//
|
||||
// Time-window policy for this row: all rate / ratio queries use a fixed
|
||||
// `[5m]` window and DO NOT follow the dashboard time picker. Reason:
|
||||
// these panels are designed for on-call glance ("is the service healthy
|
||||
// at this instant"), and we want the number to be stable across whatever
|
||||
// time range the viewer happened to pick. If we used `$__rate_interval`,
|
||||
// the same panel would show different numbers depending on whether the
|
||||
// time picker is set to "last 1 hour" vs "last 7 days", which is
|
||||
// confusing for an at-a-glance health board.
|
||||
//
|
||||
// To see trends over the time-picker range, use the Row 3 timeseries
|
||||
// (HTTP / LLM / WS by-* panels) which DO follow the time picker.
|
||||
// Panels titled "(range)" (Distribution donuts, Business stats) also
|
||||
// follow the time picker by design.
|
||||
//
|
||||
// Mix of stats (absolute counts) and gauges (bounded ratios with thresholds).
|
||||
elements['panel-1'] = statPanel(
|
||||
1,
|
||||
'Active Users',
|
||||
'Currently active sessions in Postgres (Better Auth `session.expires_at > now()`). Cluster-wide gauge — every replica polls the same DB on a 10s cache. We aggregate with `avg()` (not `sum()`, which would multiply by replica count; not `max()`, which biases high when one replica\'s cache is fresher than another\'s after a logout).',
|
||||
[query(`avg(user_active_sessions{${SERVICE_FILTER}})`, 'sessions')],
|
||||
'COUNT(DISTINCT user_id) over the Better Auth `session` table where `expires_at > now()`. This is the *real* active-user count. The historical "Active Users" panel queried `user.active_sessions` (COUNT(*) on the same table), which counts session **rows** not users — Better Auth creates a new row per sign-in and per OIDC access-token issuance and never GCs expired rows, so the row count drifts up and we have seen it report ~80K on a deployment with hundreds of actual users. Compare with `panel-15` (Active Sessions) to spot session-row inflation; ratio > ~5 means it\'s time for a session GC cron. Cluster-wide gauge — `avg()`, not `sum()`.',
|
||||
[query(`avg(user_distinct_active{${SERVICE_FILTER}})`, 'users')],
|
||||
{ unit: 'short', steps: [{ color: 'green', value: 0 }, { color: 'yellow', value: 1000 }] },
|
||||
)
|
||||
|
||||
elements['panel-2'] = statPanel(
|
||||
2,
|
||||
'WS Connections',
|
||||
'Live registry size from chat-ws (ObservableGauge, scraped each export interval).',
|
||||
[query(`sum(ws_connections_active{${SERVICE_FILTER}})`, 'connections')],
|
||||
{ unit: 'short' },
|
||||
elements['panel-15'] = statPanel(
|
||||
15,
|
||||
'Active Sessions',
|
||||
'COUNT(*) over the Better Auth `session` table where `expires_at > now()`. Counts session **rows**, not users — see `panel-1` for the de-duplicated user count. Useful as a denominator to spot row inflation: divide by panel-1 to get rows-per-user, watch for sustained growth.',
|
||||
[query(`avg(user_active_sessions{${SERVICE_FILTER}})`, 'sessions')],
|
||||
{ unit: 'short', steps: [{ color: 'green', value: 0 }, { color: 'yellow', value: 5000 }] },
|
||||
)
|
||||
|
||||
// WS Connections stat was removed — its sparkline duplicated the
|
||||
// timeseries in Row 3 (`panel-13`), which already shows the live
|
||||
// connection count over time with the same `sum(ws_connections_active)`
|
||||
// query. Keeping both meant the same number rendered twice on first
|
||||
// look. The Row 3 timeseries wins because it lets you actually read off
|
||||
// a value at a specific timestamp instead of squinting at the sparkline.
|
||||
|
||||
elements['panel-3'] = statPanel(
|
||||
3,
|
||||
'Req/s (5m)',
|
||||
'5-minute average inbound HTTP request rate. /health (Railway probe) is excluded at the @hono/otel middleware level so this reflects real user traffic.',
|
||||
'5-minute average inbound HTTP request rate. /health (Railway probe) is excluded at the @hono/otel middleware level so this reflects real user traffic. **Fixed 5m window — intentionally does not follow the dashboard time picker** (see row-level note). For trends, see panel-14 (HTTP Request Rate by Route).',
|
||||
[query(`sum(rate(http_server_request_duration_seconds_count{${SERVICE_FILTER}, http_request_method!="OPTIONS"}[5m]))`, 'req/s')],
|
||||
{ unit: 'reqps', steps: [{ color: 'green', value: 0 }, { color: 'yellow', value: 100 }, { color: 'red', value: 500 }], decimals: 2 },
|
||||
)
|
||||
@@ -384,7 +406,7 @@ elements['panel-3'] = statPanel(
|
||||
elements['panel-4'] = gaugePanel(
|
||||
4,
|
||||
'5xx Rate %',
|
||||
'5xx responses ÷ all responses over the last 5m. Spikes correlate with deploys, upstream outages, or DB problems. >1% warns, >5% pages.',
|
||||
'5xx responses ÷ all responses over the last 5m. **Fixed 5m window — intentionally does not follow the dashboard time picker**: this is an on-call glance ("is the service failing right now"). For range-aware triage use panel-9 donut and panel-44 timeseries. Spikes correlate with deploys, upstream outages, or DB problems. >1% warns, >5% pages.',
|
||||
[query(
|
||||
`100 * sum(rate(http_server_request_duration_seconds_count{${SERVICE_FILTER}, http_request_method!="OPTIONS", http_response_status_code=~"5.."}[5m])) / clamp_min(sum(rate(http_server_request_duration_seconds_count{${SERVICE_FILTER}, http_request_method!="OPTIONS"}[5m])), 1)`,
|
||||
'fail %',
|
||||
@@ -395,7 +417,7 @@ elements['panel-4'] = gaugePanel(
|
||||
elements['panel-5'] = statPanel(
|
||||
5,
|
||||
'LLM Req/s (5m)',
|
||||
'5-minute average LLM gateway request rate (chat + tts).',
|
||||
'5-minute average LLM gateway request rate (chat + tts). **Fixed 5m window — see row-level note.** For trends and per-model breakdown see panel-11.',
|
||||
[query(`sum(rate(gen_ai_client_operation_count_total{${SERVICE_FILTER}}[5m]))`, 'req/s')],
|
||||
{ unit: 'reqps', decimals: 2 },
|
||||
)
|
||||
@@ -403,7 +425,7 @@ elements['panel-5'] = statPanel(
|
||||
elements['panel-6'] = gaugePanel(
|
||||
6,
|
||||
'Email Failure %',
|
||||
'Email failures ÷ total attempts over the last 5m. >5% means Resend / DNS / suppression-list problems blocking auth flows.',
|
||||
'Email failures ÷ total attempts over the last 5m. **Fixed 5m window — see row-level note.** >5% means Resend / DNS / suppression-list problems blocking auth flows.',
|
||||
[query(
|
||||
`100 * sum(rate(airi_email_failures_total{${SERVICE_FILTER}}[5m])) / clamp_min(sum(rate(airi_email_send_total{${SERVICE_FILTER}}[5m])) + sum(rate(airi_email_failures_total{${SERVICE_FILTER}}[5m])), 1)`,
|
||||
'fail %',
|
||||
@@ -422,22 +444,30 @@ elements['panel-6'] = gaugePanel(
|
||||
// surfaces 4xx/5xx independently.
|
||||
elements['panel-8'] = piePanel(
|
||||
8,
|
||||
'LLM Models (last 5m)',
|
||||
'Share of LLM gateway calls by model. Quickly shows which model is doing the heavy lifting.',
|
||||
'LLM Models (range)',
|
||||
'Share of LLM gateway calls by model, summed over the dashboard time range. Follows the time picker — pick 1h to see the last hour\'s model mix, pick 7d to see this week\'s.',
|
||||
[query(
|
||||
`topk(8, sum by (gen_ai_request_model) (increase(gen_ai_client_operation_count_total{${SERVICE_FILTER}, gen_ai_request_model!=""}[5m])))`,
|
||||
`topk(8, sum by (gen_ai_request_model) (increase(gen_ai_client_operation_count_total{${SERVICE_FILTER}, gen_ai_request_model!=""}[$__range])))`,
|
||||
'{{gen_ai_request_model}}',
|
||||
)],
|
||||
)
|
||||
|
||||
// "Top Routes by Requests" lives as a timeseries in `panel-14` (Top
|
||||
// Endpoints row) — keeping a donut here too would just be a frozen
|
||||
// snapshot of the timeseries. Instead, this slot answers the higher-
|
||||
// value question "which routes are producing the 5xx right now?" so
|
||||
// the dashboard surfaces *failing* endpoints, not just busy ones.
|
||||
// Pair with panel-44 (5xx Rate by Route timeseries) for the same data
|
||||
// over time.
|
||||
elements['panel-9'] = piePanel(
|
||||
9,
|
||||
'Top Routes by Requests (last 5m)',
|
||||
'Top 10 Hono-matched routes by request count over the last 5 minutes. Answers "which endpoint is being hit, and how much" — replaces the previous HTTP status-code donut whose 2xx slice dominated everything else. Cardinality is bounded because `http_route` is the matched route pattern, not the concrete URL.',
|
||||
'Top Routes by 5xx (range)',
|
||||
'Top 10 Hono-matched routes by 5xx response count over the dashboard time range. Follows the time picker — pick 1h for "what\'s failing right now", pick 24h for "what failed most today". The overall 5xx% gauge (panel-4) is a fixed-5m snapshot for on-call glance; this donut respects the time picker for triage.',
|
||||
[query(
|
||||
`topk(10, sum by (http_route) (increase(http_server_request_duration_seconds_count{${SERVICE_FILTER}, http_request_method!="OPTIONS", http_route!=""}[5m])))`,
|
||||
`topk(10, sum by (http_route) (increase(http_server_request_duration_seconds_count{${SERVICE_FILTER}, http_request_method!="OPTIONS", http_route!="", http_response_status_code=~"5.."}[$__range])))`,
|
||||
'{{http_route}}',
|
||||
)],
|
||||
{ noValue: 'no 5xx' },
|
||||
)
|
||||
|
||||
// Row 3: Traffic Trends — same data as Row 2, but answering "how is it changing"
|
||||
@@ -564,6 +594,22 @@ elements['panel-42'] = timeseriesPanel(
|
||||
{ unit: 'ops' },
|
||||
)
|
||||
|
||||
// 5xx by route over time — complements panel-9 (donut: which routes
|
||||
// are failing right now) and panel-40 (4xx/5xx by status code: what
|
||||
// kind of error). This is the "when did /foo start blowing up" view.
|
||||
// topk(10) keeps the legend readable when one bad deploy lights up
|
||||
// the whole API surface.
|
||||
elements['panel-44'] = timeseriesPanel(
|
||||
44,
|
||||
'5xx Rate by Route (top 10)',
|
||||
'5xx response rate split by route. Use this to confirm whether a 5xx spike in `panel-4` is concentrated on one endpoint (e.g. a broken deploy of /api/v1/openai/*) or scattered (e.g. DB outage taking down everything). Drill into the Logs row (`panel-91`) for the matching error bodies + trace ids.',
|
||||
[query(
|
||||
`topk(10, sum by (http_route) (rate(http_server_request_duration_seconds_count{${SERVICE_FILTER}, http_request_method!="OPTIONS", http_route!="", http_response_status_code=~"5.."}[$__rate_interval])))`,
|
||||
'{{http_route}}',
|
||||
)],
|
||||
{ unit: 'reqps' },
|
||||
)
|
||||
|
||||
// Row 6: Business — money flow
|
||||
elements['panel-30'] = statPanel(
|
||||
30,
|
||||
@@ -602,7 +648,7 @@ elements['panel-32'] = piePanel(
|
||||
elements['panel-50'] = statPanel(
|
||||
50,
|
||||
'DB Query P95 (5m)',
|
||||
'PostgreSQL query duration P95 from PgInstrumentation. Spikes correlate with index misses, connection exhaustion, or backend lock contention.',
|
||||
'PostgreSQL query duration P95 from PgInstrumentation. **Fixed 5m window — does not follow the dashboard time picker** (same posture as the Service Health row stats: this is a "right now" glance). Spikes correlate with index misses, connection exhaustion, or backend lock contention.',
|
||||
[query(
|
||||
`histogram_quantile(0.95, sum by (le) (rate(db_client_operation_duration_seconds_bucket{${SERVICE_FILTER}}[5m])))`,
|
||||
'p95',
|
||||
@@ -651,25 +697,43 @@ elements['panel-90'] = logsPanel(
|
||||
`{${SERVICE_FILTER}} |= \`\``,
|
||||
)
|
||||
|
||||
// 5xx-only log stream — paired with the 5xx by-route timeseries and
|
||||
// donut so on-call goes panel-4 (something is wrong) → panel-9
|
||||
// (where) → panel-44 (when) → panel-91 (actual error message + trace
|
||||
// id, click trace_id → Tempo for full request playback). Filters at
|
||||
// the Loki query level so Grafana doesn't ship the entire log
|
||||
// firehose to the browser just to client-side filter.
|
||||
elements['panel-91'] = logsPanel(
|
||||
91,
|
||||
'5xx Error Logs',
|
||||
'Server-side error logs (level=warn|error) from Loki. Loki derived fields turn `trace_id` and `req` into clickable links — `trace_id` jumps to Tempo for full request playback (spans + child calls + DB queries), `req` filters this panel to a single request id. Use this together with panel-9 (which route) and panel-44 (when).',
|
||||
`{${SERVICE_FILTER}} | json | level=~"warn|error"`,
|
||||
)
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Layout
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const rows = [
|
||||
// Row 1: 6 stats/gauges × 4 wide × 4 high (full width)
|
||||
// Row 1: 6 stats/gauges, each 4 wide (4×6=24). Active Users (panel-1)
|
||||
// and Active Sessions (panel-15) sit side-by-side so on-call can spot
|
||||
// session-row inflation at a glance (panel-15 climbs while panel-1
|
||||
// stays flat → Better Auth row leak, not real user growth). WS
|
||||
// Connections stat is gone (duplicated by Row 3 timeseries panel-13).
|
||||
row('Service Health', [
|
||||
item('panel-1', 0, 0, 4, 4),
|
||||
item('panel-2', 4, 0, 4, 4),
|
||||
item('panel-15', 4, 0, 4, 4),
|
||||
item('panel-3', 8, 0, 4, 4),
|
||||
item('panel-4', 12, 0, 4, 4),
|
||||
item('panel-5', 16, 0, 4, 4),
|
||||
item('panel-6', 20, 0, 4, 4),
|
||||
]),
|
||||
// Row 2: 2 donuts × 12 wide × 7 high — current-state distribution
|
||||
// Dropped HTTP-methods donut (low-cardinality, redundant with Row 3 by-method
|
||||
// timeseries) and HTTP-status donut (2xx dominated, 4xx/5xx already broken
|
||||
// out in Row 5). Replaced status donut with Top Routes which answers a
|
||||
// higher-information question with the same visual budget.
|
||||
// Row 2: 2 donuts × 12 wide × 7 high — current-state distribution.
|
||||
// Left donut: LLM models (where load is going). Right donut: 5xx by
|
||||
// route (where failures are concentrated). The previous "Top Routes
|
||||
// by Requests" donut was replaced because its timeseries form in
|
||||
// Row 3.5 carries the same data with time context; 5xx-by-route is
|
||||
// the higher-value glance.
|
||||
row('Distribution (now)', [
|
||||
item('panel-8', 0, 0, 12, 7),
|
||||
item('panel-9', 12, 0, 12, 7),
|
||||
@@ -694,7 +758,7 @@ const rows = [
|
||||
item('panel-20', 0, 0, 12, 8),
|
||||
item('panel-21', 12, 0, 12, 8),
|
||||
]),
|
||||
// Row 5: 1 stacked area + 2 stats + 1 timeseries × 7 high
|
||||
// Row 5: 1 stacked area + 2 stats + 1 timeseries × 7 high.
|
||||
// Stream Interruptions and ⚠ Flux Unbilled sit next to the 4xx/5xx trend
|
||||
// so revenue-leak signal (which doesn't show up in 5xx) gets the same
|
||||
// glance-weight as transport-layer errors.
|
||||
@@ -704,6 +768,13 @@ const rows = [
|
||||
item('panel-43', 14, 0, 4, 7),
|
||||
item('panel-42', 18, 0, 6, 7),
|
||||
]),
|
||||
// Row 5.5: 5xx by-route trend full width. Triage path: panel-4
|
||||
// (something wrong) → panel-9 donut (which route now) → here (when
|
||||
// it started + per-route rates over time) → panel-91 (actual error
|
||||
// log lines + clickable trace_id for full request replay in Tempo).
|
||||
row('5xx Triage', [
|
||||
item('panel-44', 0, 0, 24, 7),
|
||||
]),
|
||||
// Row 6: 1 stat + 1 gauge + 1 donut × 8 wide × 7 high
|
||||
row('Business', [
|
||||
item('panel-30', 0, 0, 8, 7),
|
||||
@@ -719,9 +790,12 @@ const rows = [
|
||||
item('panel-52', 12, 0, 6, 6),
|
||||
item('panel-53', 18, 0, 6, 6),
|
||||
], { collapse: true }),
|
||||
// Row 8: full-width logs
|
||||
// Row 8: full-width logs. Two panels stacked: errors-only on top
|
||||
// (default focus for triage) and the full firehose below (manual
|
||||
// filter when you need broader context).
|
||||
row('Logs', [
|
||||
item('panel-90', 0, 0, 24, 12),
|
||||
item('panel-91', 0, 0, 24, 10),
|
||||
item('panel-90', 0, 10, 24, 10),
|
||||
]),
|
||||
]
|
||||
|
||||
|
||||
+13
-4
@@ -42,6 +42,7 @@ import { resolveRequestAuth } from './libs/request-auth'
|
||||
import { sessionMiddleware } from './middlewares/auth'
|
||||
import { emitOtelLog, initOtel } from './otel'
|
||||
import { registerActiveSessionsGauge } from './otel/gauges/active-sessions'
|
||||
import { registerDistinctActiveUsersGauge } from './otel/gauges/distinct-active-users'
|
||||
import { createAdminFluxGrantsRoutes } from './routes/admin/flux-grants'
|
||||
import { createAuthRoutes } from './routes/auth'
|
||||
import { createCharacterRoutes } from './routes/characters'
|
||||
@@ -527,11 +528,19 @@ export async function createApp() {
|
||||
userDeletionService,
|
||||
posthog,
|
||||
})
|
||||
// Register the cluster-wide ObservableGauge for active sessions. Each
|
||||
// replica polls the same DB (cached 10s, in-flight coalesced) and the
|
||||
// dashboard aggregates with avg(), not sum(). See observability-conventions.md.
|
||||
if (resolved.otel)
|
||||
// Register the cluster-wide ObservableGauges for sessions / users. Each
|
||||
// replica polls the same DB (cached 10s, in-flight coalesced); dashboards
|
||||
// aggregate with avg(), not sum(). See observability-conventions.md.
|
||||
//
|
||||
// Both gauges share the same `session` table: `user.active_sessions` is
|
||||
// `COUNT(*)` (row inflation prone), `user.distinct_active` is
|
||||
// `COUNT(DISTINCT user_id)` (real active-user count). Comparing the two
|
||||
// surfaces session-row leakage from missing GC + per-OIDC-token row
|
||||
// creation.
|
||||
if (resolved.otel) {
|
||||
registerActiveSessionsGauge(resolved.otel.auth.activeSessions, resolved.db, resolved.otel.observability.metricReadErrors)
|
||||
registerDistinctActiveUsersGauge(resolved.otel.auth.distinctActiveUsers, resolved.db, resolved.otel.observability.metricReadErrors)
|
||||
}
|
||||
|
||||
const { app, injectWebSocket } = await buildApp({
|
||||
auth: resolved.auth,
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
import type { AuthMetrics, ObservabilityMetrics } from '..'
|
||||
import type { Database } from '../../libs/db'
|
||||
|
||||
import { useLogger } from '@guiiai/logg'
|
||||
import { countDistinct, gt } from 'drizzle-orm'
|
||||
|
||||
import { session as sessionTable } from '../../schemas/accounts'
|
||||
|
||||
/**
|
||||
* Wire the `user.distinct_active` ObservableGauge to a Postgres
|
||||
* `COUNT(DISTINCT user_id)` over the Better Auth session table.
|
||||
*
|
||||
* Use when:
|
||||
* - Assembling DI in `createApp()`, exactly once per process.
|
||||
*
|
||||
* Why this exists alongside `registerActiveSessionsGauge`:
|
||||
* - `user.active_sessions` is `COUNT(*)` — counts session **rows**. Better
|
||||
* Auth creates a new row per sign-in and per OIDC access-token issuance
|
||||
* (the `oauth_access_token` table has a FK to `session.id`) and never GCs
|
||||
* expired rows, so the row-count drifts up over time independently of
|
||||
* the real user base. We've seen this metric show ~80K on a small
|
||||
* deployment where the real distinct-user count is ~hundreds.
|
||||
* - `user.distinct_active` is `COUNT(DISTINCT user_id)` — the actual
|
||||
* active-user gauge. Pair with `user.active_sessions` to spot session
|
||||
* inflation: if rows / users ratio climbs past ~5 it's probably time
|
||||
* to add a session-GC cron or shorten Better Auth's `expiresIn`.
|
||||
*
|
||||
* Multi-replica note:
|
||||
* - Cluster-wide gauge — every replica reads the same DB and reports the
|
||||
* same value. Dashboards MUST aggregate with `avg()`, NOT `sum()`. See
|
||||
* observability-conventions.md.
|
||||
*
|
||||
* Concurrency:
|
||||
* - Same in-flight promise lock pattern as `registerActiveSessionsGauge`,
|
||||
* so concurrent OTel collection cycles fold into one DB query.
|
||||
*
|
||||
* Failure mode:
|
||||
* - DB error → increment `airi.observability.read_errors{metric}` and skip
|
||||
* `result.observe(...)`. Prometheus staleness exposes the outage instead
|
||||
* of pinning a stale cached value forever.
|
||||
*/
|
||||
export function registerDistinctActiveUsersGauge(
|
||||
gauge: AuthMetrics['distinctActiveUsers'],
|
||||
db: Database,
|
||||
metricReadErrors: ObservabilityMetrics['metricReadErrors'],
|
||||
) {
|
||||
const log = useLogger('distinct-active-users-gauge').useGlobalConfig()
|
||||
const CACHE_TTL_MS = 10_000
|
||||
|
||||
let cachedAt = 0
|
||||
let cachedCount = 0
|
||||
let refreshInFlight: Promise<boolean> | null = null
|
||||
|
||||
async function refresh(): Promise<boolean> {
|
||||
try {
|
||||
// Use the app clock (`new Date()`) for the same reason as
|
||||
// `registerActiveSessionsGauge`: agree with Better Auth's own
|
||||
// session-validity check, which uses `new Date()` rather than
|
||||
// `NOW()`.
|
||||
const rows = await db
|
||||
.select({ count: countDistinct(sessionTable.userId) })
|
||||
.from(sessionTable)
|
||||
.where(gt(sessionTable.expiresAt, new Date()))
|
||||
cachedCount = Number(rows[0]?.count ?? 0)
|
||||
cachedAt = Date.now()
|
||||
return true
|
||||
}
|
||||
catch (err) {
|
||||
log.withError(err).warn('Failed to read distinct active users for gauge')
|
||||
metricReadErrors.add(1, { metric: 'user.distinct_active' })
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
gauge.addCallback(async (result) => {
|
||||
const now = Date.now()
|
||||
|
||||
if (cachedAt !== 0 && now - cachedAt < CACHE_TTL_MS) {
|
||||
result.observe(cachedCount)
|
||||
return
|
||||
}
|
||||
|
||||
if (!refreshInFlight) {
|
||||
refreshInFlight = refresh().finally(() => {
|
||||
refreshInFlight = null
|
||||
})
|
||||
}
|
||||
const ok = await refreshInFlight
|
||||
|
||||
if (ok) {
|
||||
result.observe(cachedCount)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -44,6 +44,7 @@ import {
|
||||
METRIC_STRIPE_PAYMENT_FAILED,
|
||||
METRIC_STRIPE_SUBSCRIPTION_EVENT,
|
||||
METRIC_USER_ACTIVE_SESSIONS,
|
||||
METRIC_USER_DISTINCT_ACTIVE,
|
||||
METRIC_USER_LOGIN,
|
||||
METRIC_USER_REGISTERED,
|
||||
METRIC_WS_CONNECTIONS_ACTIVE,
|
||||
@@ -77,6 +78,21 @@ export interface AuthMetrics {
|
||||
* "Multi-Replica Considerations".
|
||||
*/
|
||||
activeSessions: ObservableGauge
|
||||
/**
|
||||
* Pull-based gauge for distinct users with ≥1 non-expired session.
|
||||
*
|
||||
* Use when:
|
||||
* - Querying real "active users" — not session rows. Better Auth creates a
|
||||
* new `session` row per sign-in and per OIDC token refresh, and never
|
||||
* GCs expired rows, so {@link AuthMetrics.activeSessions} drifts up
|
||||
* over time even when the actual user base is small.
|
||||
*
|
||||
* Expects:
|
||||
* - Backed by `SELECT COUNT(DISTINCT user_id) FROM session WHERE expires_at > now()`.
|
||||
* Same cluster-wide truth as `activeSessions`; dashboards MUST aggregate
|
||||
* with `avg()`, not `sum()` — see observability-conventions.md.
|
||||
*/
|
||||
distinctActiveUsers: ObservableGauge
|
||||
}
|
||||
|
||||
export interface EngagementMetrics {
|
||||
@@ -225,7 +241,10 @@ export function initOtel(env: Env): OtelInstance | null {
|
||||
description: 'Number of user sign-ins',
|
||||
}),
|
||||
activeSessions: meter.createObservableGauge(METRIC_USER_ACTIVE_SESSIONS, {
|
||||
description: 'Active user sessions sourced from Postgres (cluster-wide; dashboard must use max(), not sum())',
|
||||
description: 'Active user sessions sourced from Postgres (cluster-wide; dashboard must use avg(), not sum())',
|
||||
}),
|
||||
distinctActiveUsers: meter.createObservableGauge(METRIC_USER_DISTINCT_ACTIVE, {
|
||||
description: 'Distinct users with ≥1 non-expired session — true active-user count, immune to per-row session inflation (cluster-wide; dashboard must use avg(), not sum())',
|
||||
}),
|
||||
}
|
||||
|
||||
|
||||
@@ -37,6 +37,11 @@ export const METRIC_AUTH_FAILURES = 'auth.failures'
|
||||
export const METRIC_USER_REGISTERED = 'user.registered'
|
||||
export const METRIC_USER_LOGIN = 'user.login'
|
||||
export const METRIC_USER_ACTIVE_SESSIONS = 'user.active_sessions'
|
||||
// Distinct users with at least one non-expired session row. Pair with
|
||||
// USER_ACTIVE_SESSIONS to detect "session row inflation" (Better Auth
|
||||
// creates a new row per sign-in / per OIDC token refresh and never GCs)
|
||||
// vs real user growth.
|
||||
export const METRIC_USER_DISTINCT_ACTIVE = 'user.distinct_active'
|
||||
|
||||
// Engagement (AIRI custom)
|
||||
export const METRIC_CHAT_MESSAGES = 'chat.messages'
|
||||
|
||||
@@ -0,0 +1,220 @@
|
||||
# Verification 自动化方案
|
||||
|
||||
设计稿，未实施。落到这里是为了让 verification 流程从「人工跑命令贴输出」走向「机器跑断言贴 evidence」，同时保留 AGENTS.md 里 Iron Law 的语义。
|
||||
|
||||
## TL;DR
|
||||
|
||||
1. **原因**：现有 5 份 verification 文档结构清晰，但执行步骤需要人工跑命令、人工贴输出、人工记录「最后验证」日期。一旦超过 30 天，AGENTS.md 规定默认 unverified，没有机制能识别这种过期。
|
||||
2. **猜想**：verification 文档继续作为 single source of truth，每份文档关联一份可执行 artifact，artifact 跑通就是 evidence，跑通时间就是「最后验证」。
|
||||
3. **决策**：分三层实施，集成测试覆盖 in-repo 可重现路径，live verifier 覆盖只能在已部署环境验证的路径，CI 守护过期时间。
|
||||
|
||||
## 背景
|
||||
|
||||
`apps/server/docs/ai-context/verifications/` 下 5 份文档，结构基本统一：
|
||||
|
||||
- `场景 / 用户路径`：写明用户敲 X，预期得到 Y
|
||||
- `命令 / 步骤`：手工敲的 curl、SQL、UI 操作
|
||||
- `预期 / 实际输出`：贴 response body、log 节选、screenshot 路径
|
||||
- `Evidence`：commit SHA、行号引用、测试文件路径
|
||||
- `Status` 与 `最后验证`：人工维护
|
||||
|
||||
其中 3 份文档（`flux-unbilled-exploit-fix`、`flux-unbilled-reconciliation`、`admin-flux-grants`）引用了已落库的 vitest 单测，剩下 2 份（`email-auth`、`account-deletion`）以手工 curl + 真实 Resend / 真实数据库为主。
|
||||
|
||||
## 拆解现状
|
||||
|
||||
把 5 份文档里的步骤按「证据来源」拆开，能看到三类：
|
||||
|
||||
1. **纯代码路径**，例如 partial-debit 的数值逻辑、ledger 行写入。这类已经被 vitest 单测覆盖，证据来源是 `expect()` 断言。
|
||||
2. **跨外部边界的用户路径**，例如「N 个并发 LLM completion 触发 pre-flight 拒绝 + ledger 写入 + metric 上报」。这类需要 pg、redis、Hono app、Prometheus `/metrics` 端点同时在场，目前没有自动化覆盖。
|
||||
3. **依赖部署环境的路径**，例如 Resend 真实投递、Stripe webhook 回调、Grafana panel 斜率、Better Auth 跨域 OIDC handoff。这类无论在 PR CI 还是本地都无法完整跑通，必须在 staging 或 prod 上验证。
|
||||
|
||||
第 1 类已经自动化，第 2、3 类是空缺。
|
||||
|
||||
## 提出猜想
|
||||
|
||||
verification 文档的「用户路径」描述天然适合作为测试用例标题。如果给每份文档加一份配套 artifact，artifact 类型按上面三类分发：
|
||||
|
||||
- 纯代码路径，归到 `*.test.ts`，已经这样做
|
||||
- 跨边界的用户路径，归到 `*.integration.test.ts`，testcontainers 起依赖
|
||||
- 依赖部署环境的路径，归到 `*.verifier.ts`，针对 staging URL 跑，post-deploy 触发
|
||||
|
||||
每份文档头部加一段 frontmatter，机器读取后能回答三个问题：
|
||||
|
||||
1. 这份文档对应的 feature 是什么
|
||||
2. 自动化 artifact 在哪里
|
||||
3. 上次自动化跑通是什么时候
|
||||
|
||||
## 分节解答
|
||||
|
||||
### 一、frontmatter schema
|
||||
|
||||
```yaml
|
||||
---
|
||||
feature: flux-unbilled-exploit-fix
|
||||
owner: rbxin2003@gmail.com
|
||||
automated_by:
|
||||
- kind: unit
|
||||
path: apps/server/src/services/billing/tests/billing-service.test.ts
|
||||
cases:
|
||||
- 'rejects pre-flight when balance is below FLUX_PER_REQUEST'
|
||||
- 'non-streaming completion drains partial balance and logs charged'
|
||||
- kind: integration
|
||||
path: apps/server/tests/verifications/flux-unbilled.integration.test.ts
|
||||
- kind: live
|
||||
path: apps/server/tests/verifications/flux-unbilled.verifier.ts
|
||||
schedule: post-deploy
|
||||
last_verified:
|
||||
unit: 2026-05-15
|
||||
integration: 2026-05-15
|
||||
live: 2026-05-14
|
||||
expires_after_days: 30
|
||||
---
|
||||
```
|
||||
|
||||
字段语义钉死：
|
||||
|
||||
- `feature`：文档 slug，与文件名同名
|
||||
- `automated_by[].kind`：`unit` / `integration` / `live`，三选一
|
||||
- `automated_by[].path`：可执行文件路径，CI 跑通后能写回 `last_verified`
|
||||
- `last_verified.<kind>`：YYYY-MM-DD，由 CI 自动写回，人不手动改
|
||||
- `expires_after_days`：默认 30，与 AGENTS.md 一致
|
||||
|
||||
### 二、集成测试 harness
|
||||
|
||||
放在每个 app 下的 `tests/verifications/` 目录，例如 `apps/server/tests/verifications/`。harness 提供：
|
||||
|
||||
1. testcontainers 起 Postgres 16 + Redis 7，注入与 `.env.example` 同 schema 的环境变量
|
||||
2. `createApp()` 直接 mount，不走真实端口，调用 `app.request(...)`
|
||||
3. 三种断言入口：
|
||||
- HTTP 响应，按现有 `app.test.ts` 范式
|
||||
- DB 状态，通过 drizzle 查 `flux_transaction` / `user_flux`
|
||||
- Metric 状态，scrape `/metrics` 文本，匹配 `airi_billing_flux_unbilled_total{...} <value>`
|
||||
|
||||
最小测试骨架：
|
||||
|
||||
```ts
|
||||
describe('verification: flux-unbilled-exploit-fix', () => {
|
||||
let ctx: VerificationContext
|
||||
|
||||
beforeAll(async () => {
|
||||
ctx = await startVerificationContext()
|
||||
})
|
||||
|
||||
afterAll(async () => {
|
||||
await ctx.stop()
|
||||
})
|
||||
|
||||
it('concurrent partial-balance requests yield one partial debit and N-1 pre-flight 402', async () => {
|
||||
await ctx.seedUser({ id: 'u1', balance: 5 })
|
||||
await ctx.setConfig({ FLUX_PER_REQUEST: 100 })
|
||||
|
||||
const responses = await Promise.all(
|
||||
Array.from({ length: 5 }, () => ctx.app.request('/api/v1/openai/...')),
|
||||
)
|
||||
|
||||
expect(responses.filter(r => r.status === 402)).toHaveLength(5)
|
||||
const ledger = await ctx.db.query.fluxTransaction.findMany({ where: ... })
|
||||
expect(ledger).toHaveLength(0)
|
||||
|
||||
const metrics = await ctx.scrapeMetrics()
|
||||
expect(metrics).toMatchMetric('airi_billing_flux_unbilled_total', {
|
||||
labels: { reason: 'partial_debit_drained' },
|
||||
delta: 0,
|
||||
})
|
||||
})
|
||||
})
|
||||
```
|
||||
|
||||
`MatchMetric` 与 `scrapeMetrics` 这两个 helper 放在 `packages/server-runtime` 或 `apps/server/src/testing/`，由集成测试和 live verifier 共用。
|
||||
|
||||
### 三、live verifier
|
||||
|
||||
针对 staging / prod。形态选 vitest 也可以，选独立 CLI 也可以，差别在「是否需要被 CI 用 `--include` pattern 隔离」。建议直接沿用 vitest，给文件后缀 `.verifier.ts`，配 `vitest.config.ts` 的 `include` / `exclude` 把它们与 unit / integration 隔离。
|
||||
|
||||
live verifier 的断言对象不再是「mount 的 Hono app」，是「真实 URL」：
|
||||
|
||||
```ts
|
||||
describe('live verifier: flux-unbilled-exploit-fix', () => {
|
||||
it('panel-43 slope is below alert threshold over the last 5 minutes', async () => {
|
||||
const slope = await prometheusQuery(
|
||||
'increase(airi_billing_flux_unbilled_total[5m])',
|
||||
{ url: process.env.PROM_URL! },
|
||||
)
|
||||
expect(slope).toBeLessThan(0.5)
|
||||
})
|
||||
})
|
||||
```
|
||||
|
||||
需要凭据的项目（Prometheus、Resend、Stripe）通过 env 注入，与 `secrets-management` 规则一致，不写进文件。
|
||||
|
||||
### 四、CI 编排
|
||||
|
||||
三条 GitHub Actions workflow：
|
||||
|
||||
1. **`verification-unit.yml`**：PR 触发，跑全部 `*.test.ts`。现状已有，作为 baseline。
|
||||
2. **`verification-integration.yml`**：PR 触发，跑全部 `*.integration.test.ts`。预计单跑 60 至 180 秒（testcontainers 启动），用 matrix 拆分到多个 worker。仅在改动触及 `apps/server/**` 或 `packages/server-*/**` 时跑，其他改动 skip。
|
||||
3. **`verification-live.yml`**：post-deploy 触发（Railway deploy hook → GitHub repository_dispatch），针对 staging URL 跑全部 `*.verifier.ts`。跑通后自动 PR 一份更新 `last_verified.live` 的提交，或者直接 commit 回 main（按团队偏好选）。
|
||||
|
||||
第 2 类必要的 secret：testcontainers 自身不需要 secret，只需要 docker daemon，GitHub Actions runner 默认带。第 3 类需要 `PROM_URL`、`PROM_TOKEN`、`STRIPE_TEST_KEY`、`RESEND_API_KEY` 等，放到 GitHub Actions secrets。
|
||||
|
||||
### 五、过期守护
|
||||
|
||||
新增 `scripts/verification-doctor.ts`，在 `verification-unit.yml` 末尾跑：
|
||||
|
||||
```ts
|
||||
// 遍历所有 verification 文档
|
||||
// 读 frontmatter.last_verified
|
||||
// 与 frontmatter.expires_after_days 比较
|
||||
// 超期 -> stderr 报告 + exit 1
|
||||
```
|
||||
|
||||
CI 失败时输出形如：
|
||||
|
||||
```
|
||||
✗ flux-unbilled-reconciliation: last_verified.integration = 2025-12-01 (expired 165 days)
|
||||
✗ email-auth: last_verified.live = (none)
|
||||
```
|
||||
|
||||
主分支跑过期检查也跑，跑失败不阻塞 main，只发到 Slack / Lark 通知频道，避免老文档过期把全员卡住。
|
||||
|
||||
## 回指前文
|
||||
|
||||
回到 TL;DR 的三条决策：
|
||||
|
||||
1. 「集成测试覆盖 in-repo 可重现路径」对应第二节，testcontainers + drizzle + metric scrape 是这一层的最小工具集。
|
||||
2. 「live verifier 覆盖只能在已部署环境验证的路径」对应第三节，针对真实 URL 跑 Prometheus query、Stripe test mode、Resend dashboard API。
|
||||
3. 「CI 守护过期时间」对应第五节，frontmatter 的 `last_verified` 由 CI 写回，doctor 脚本扫超期。
|
||||
|
||||
三层加起来，verification 文档从「人工 claim」变成「机器 claim + 人工 narrative」。
|
||||
|
||||
## 影响面
|
||||
|
||||
| 维度 | 影响 |
|
||||
|---|---|
|
||||
| 单测时间 | 不变 |
|
||||
| PR CI 时间 | 新增 60 至 180 秒（取决于 testcontainers 并发 + matrix 拆分） |
|
||||
| 本地开发 | 默认 `pnpm exec vitest run` 不跑 integration，要显式跑 `pnpm verify:integration` |
|
||||
| docker 依赖 | 本地跑 integration 需要 docker daemon，已有 `docker-compose.otel.yml` 范式 |
|
||||
| Secret 管理 | live verifier 需要 4 至 6 个 staging secret，放 GitHub Actions secrets |
|
||||
| 文档维护 | verification 文档新增 frontmatter，原有 markdown 正文不变 |
|
||||
| AGENTS.md | 加一段「如何写 verification artifact」，引用本文 |
|
||||
|
||||
## 可观测性 / eval
|
||||
|
||||
实施后用三个指标判断方案有效：
|
||||
|
||||
1. **集成测试覆盖率**：5 份文档里有几份对应有 `*.integration.test.ts`，目标 100%
|
||||
2. **live verifier 触发频率**：post-deploy 一次必跑，跑失败的次数与生产 incident 的相关性
|
||||
3. **doctor 报告超期数**：每周扫一次，超期数应当趋近 0
|
||||
|
||||
第 3 个指标如果长期不为 0，说明 verification 流程仍需要人工介入太多，要回头看 frontmatter 设计是否合适。
|
||||
|
||||
## 收束
|
||||
|
||||
这份方案保留 verification 文档的人工 narrative（root cause、why、tradeoff），把可执行部分挪到代码，把过期检测交给 CI。实施分三步：
|
||||
|
||||
1. 先做 frontmatter schema 与 doctor 脚本，零代码改动，立即能识别已有 5 份文档的过期状态。
|
||||
2. 再做 `flux-unbilled-exploit-fix` 的集成测试样板，跑通一个 case 形成模板。
|
||||
3. 最后逐份补齐 integration 与 live verifier。
|
||||
|
||||
如果某一份文档（例如 `email-auth`）的 live 验证依赖 Resend 真实投递，确认收件状态需要轮询 Resend `/emails` API，这部分实现成本较高，可以推到第三步的尾巴上单独立项。
|
||||
Reference in New Issue
Block a user