mirror of
https://github.com/Gen-Verse/OpenClaw-RL.git
synced 2026-08-14 08:52:10 +00:00
yinjjiew
177747e461
Core slime changes that unblock running a different-sized PRM teacher next
to the student (e.g. Qwen3-8B teacher with Qwen3-4B student) and add a
token-level top-K OPD distillation path. Also bundles several Qwen3.5
robustness fixes and an SGLang router fast-fail.
PRM teacher and top-K OPD
- actor.py: copy architectural fields (num_layers/hidden_size/ffn/heads/
query_groups/kv/RoPE/MoE/MLA/MTP/vocab) from the teacher's torch_dist
common.pt onto a deepcopied args namespace so the teacher can be a
different model size than the student's MODEL_ARGS without going through
--megatron-to-hf-mode bridge. Adds compute_student_topk and
gather_at_indices wired into ray/actor_group.py.
- loss.py: opt-in emit_topk_logprobs context manager;
_vocab_parallel_topk_log_probs and gather_log_probs_at_indices for
per-token top-K extraction. Default actor/old_actor/ref forwards are
byte-identical to before.
- arguments.py: --prm-teacher-rotary-base, --prm-teacher-megatron-to-hf-mode,
--prm-teacher-hf-checkpoint, --distill-subset-mode {student,teacher,overlap}.
- train_async.py: three-pass dispatch when distill_topk>0 with
subset_mode=student (student top-K, teacher gather, student loss).
- data.py / model.py: carry prm_teacher_topk_log_probs / topk_indices /
topk_log_probs through rollout logging and train_one_step packing.
Bridge / raw mode parity and OOM fix
- model_provider.py: forward CLI flags into bridge providers
(recompute, dropout, attention/dropout/SwiGLU/RoPE fusion, dtype, CP)
so bridge mode does not silently keep HF-config defaults
(e.g. attention_dropout=0.1) and cause train-vs-inference skew.
- model.py: pass fp32_output=False on the train forward to match the
forward-only path; without this, packing 2 samples into a 16K
microbatch produces a 7.58 GiB fp32 logits allocation that OOMs an
80GB GPU.
Qwen3.5 robustness
- backends/sglang_utils/qwen3_5.py: drop the .self_attn. prefix from
Megatron-emitted weight names and remove the erroneous model. re-prefix
so weights load correctly into SGLang's Qwen3.5 full-attention layer
(qkv_proj/o_proj/q_norm/k_norm registered as direct attributes of the
decoder layer, not under self_attn).
- slime_plugins/models/hf_attention.py: _load_hf_config JSON fallback for
Qwen3.5 / Qwen3.5-VL checkpoints whose model_type is not yet registered
in transformers; new _AllGatherForDuplicatedComputation autograd Function
whose backward returns the local slice (avoids the world_size gradient
inflation when wrapping duplicated computation); pass
tensor_parallel_output_grad=False on the SP gather of hidden_states for
hybrid linear-attention layers.
- slime_plugins/models/qwen3_5.py: re-export _load_hf_config.
SGLang router fail-fast
- utils/http_utils.py: stop drowning logs after N consecutive
no_available_workers / circuits-open 503s from the router; tunable via
SLIME_HTTP_PERMANENT_FAIL_FAST (default 3). Also throttles retry log
spam to attempts {1,5,10,25,50}.
Launch scripts (Qwen3.5)
- scripts/run-qwen35-4B.sh: Qwen3.5 4B training launcher.
- scripts/models/qwen3.5-9B-VL.sh, qwen3.5-27B-VL.sh: Qwen3.5 VL configs.