mirror of
https://github.com/Gen-Verse/OpenClaw-RL.git
synced 2026-08-14 00:47:53 +00:00
zhuchichi56
a111c686b0
Online Experiential Learning with On-Policy Context Distillation for personalized agents. - Experience-augmented teacher distillation (top-K reverse KL) - Single-session and cross-session experience modes - Post-hoc extraction + teacher replay (all turns benefit from full-session experience) - Qwen3-1.7B training script + GSM8K hard problem eval framework - Peak accuracy: 46.0 (+19.4 over baseline)
Binary Reward Summarized from Next State
Online RL for agentic tool-use, using binary process reward signals from environment feedback.
Method Overview
The policy model is deployed as an OpenAI-compatible chat proxy. External environments (e.g. OpenClaw) send multi-turn conversations through this proxy. For each main-line turn, the system:
- Forwards the request to the policy model (served by SGLang) and collects the response along with per-token log-probabilities.
- When the next turn arrives, its user/environment message serves as the "next state" for the previous turn.
- A Process Reward Model (PRM) judges the previous response quality given the next state (could be user or env feedback). It produces
mindependent evaluations via majority vote, scoring each turn as+1(good),-1(bad), or0(neutral). - The majority-voted score becomes the scalar reward for that turn.
- Turns that never receive a next state (i.e. the last turn in a session) are excluded from training (
loss_mask = 0), unless they are the only turn in the session (at-least-one guarantee).
Advantage Estimation (GRPO)
Advantages are computed using Group Relative Policy Optimization (GRPO). For each sample with scalar reward r, the advantage is broadcast uniformly to all response tokens:
A_t = r, \quad \forall t \in \text{response tokens}
No reward normalization is applied (--disable-rewards-normalization).
Policy Gradient Loss
Standard PPO-style clipped surrogate objective with asymmetric clipping:
\rho_t = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\text{old}}(a_t \mid s_t)}
\mathcal{L}_{\text{pg}} = -\mathbb{E}_t\Big[\min\!\big(\rho_t A_t,\ \text{clip}(\rho_t,\, 1-\varepsilon,\, 1+\varepsilon_{\text{high}}) \cdot A_t\big)\Big]
where \varepsilon = 0.2, \varepsilon_{\text{high}} = 0.28.
Total Loss
\mathcal{L} = \mathcal{L}_{\text{pg}} + \beta_{\text{KL}} \cdot \mathcal{L}_{\text{KL}}
where \beta_{\text{KL}} = 0.02. Entropy bonus is disabled (\beta_{\text{ent}} = 0).
How to Run
cd slime
# Qwen3
bash ../openclaw-rl/run_qwen3_4b_openclaw_rl.sh
File Structure
openclaw-rl/
├── README.md
├── run_qwen3_4b_openclaw_rl.sh # Launch script (Qwen3)
├── run_qwen35_4b_openclaw_rl.sh # Launch script (Qwen3.5)
├── openclaw_api_server.py # FastAPI proxy + PRM scoring + sample submission
├── openclaw_rollout.py # Async rollout worker (bridges API server ↔ SLIME trainer)
└── results/ # Runtime records (auto-created)