Keep the top-k OPD helper modules importable from openclaw-combine and avoid enabling incompatible full recompute for Qwen3-VL deepstack bridge providers.
Document and wire the top-k hint-selection objective so OpenClaw can train with hybrid GRPO and OPD signals while supporting OPD-only and GRPO-only ablations.
Adds the multi-candidate variant of the openclaw-combine pipeline that
uses the slime retool-hybrid-select infra (PR #105) to drive K teacher
forwards per Sample, with in-kernel k* selection over the 9-cell
(--distill-subset-mode x --hint-selection) matrix.
Files added (all under openclaw-combine/):
* openclaw_combine_select_api_server.py
OpenClawCombineSelectAPIServer subclass of OpenClawCombineAPIServer.
Three-case dispatch (OPD-only / OPD+RL / RL-only) is preserved; the
OPD evaluate step keeps ALL accepted hints (deduped, shortest-first,
capped at OPENCLAW_TOPK_MAX_CAND) and ships them as
sample.teacher_tokens_candidates instead of selecting a single hint.
RL-only turns ship a degenerate K_i=1 un-enhanced candidate so the
multi-cand pipeline always has something to gather.
* openclaw_combine_select_rollout.py
Drop-in alternative to openclaw_combine_rollout.generate_rollout_*;
same worker / queue / drain plumbing, swaps in
OpenClawCombineSelectAPIServer. Module-level singleton state is
intentionally separate so both rollout entry points can co-exist on
the same launcher PYTHONPATH.
* openclaw_topk_select_loss.py
Combined GRPO + top-K OPD loss with the 9-cell selection matrix.
Reuses _opd_one_sample / _gather_along_K /
_overlap_count_per_token / _select_k_star_per_token from
hint_opt_exp so the math stays identical to retool / hint_opd_exp.
Per-Sample sequence_optimal under openclaw-combine IS per-PRM-step
k* under the dynamic-history paradigm (one Sample per session-turn).
* run_qwen3_4b_openclaw_topk_select.sh
Launcher: 4 actor + 2 rollout + 1 SGLang PRM + 1 Megatron PRM
teacher GPUs. Forces OPENCLAW_COMBINE_OPD_TEACHER_SOURCE=megatron
(the topk-select kernel needs per-cand top-K log-probs / indices,
which the inference-side teacher path does not produce). Uses raw
torch_dist load (no bridge) for both student and teacher.
No changes to existing code paths; the legacy combine_loss /
openclaw_combine_api_server / openclaw_combine_rollout entries are
untouched. Activation is fully gated by the launcher's choice of
--rollout-function-path / --custom-loss-function-path /
--custom-generate-function-path.
Made-with: Cursor
Adds the multi-candidate teacher path used by
toolcall-rl/generate_with_retool_hybrid_select.py: each sample carries
K_i hint-enhanced token sequences (variable across samples), K teacher
forwards run in series, and per-position top-K selection picks the best
candidate per token / per sample.
Changes by file:
* slime/slime/utils/arguments.py
- --hint-m: number of PRM hint-generation rollouts per step (falls back
to --prm-m).
- --hint-selection {shortest,token_optimal,sequence_optimal}:
legacy single-forward 'shortest' vs new K-forward overlap-based
selectors. 'student' subset-mode is intentionally degenerate here.
* slime/slime/ray/rollout.py
- RolloutManager: transport per-sample teacher_tokens_candidates
(list[list[Tensor]]) and teacher_total_lengths_candidates through
the rollout payload alongside the existing teacher_tokens fields.
* slime/slime/backends/megatron_utils/actor.py
- compute_prm_teacher_log_probs dispatches to a new
_compute_prm_teacher_log_probs_multi_cand path when
teacher_tokens_candidates is present; runs K_max forwards (cyclic-
padded to K_max) and stacks per-candidate outputs along a leading
K axis, exposed under *_cand keys, with prm_teacher_K_per_sample /
prm_teacher_K_max for downstream masking.
- Single-candidate path is byte-identical when no _cand fields are set.
* slime/slime/backends/megatron_utils/loss.py
- emit_native_topk_indices(K) context manager: in the same
gather-at-indices forward, also compute the candidate's *own* native
top-K indices and return them under topk_native_indices, so the
K-loop can carry per-candidate selection signals to the loss with
no extra teacher pass. Default disabled (K=0) preserves existing
behavior bit-for-bit.
* slime/slime/backends/megatron_utils/data.py
- log_rollout_data: skip per-sample mean logging for the new
*_cand keys and the candidate token/length lists.
* slime/slime/backends/megatron_utils/model.py
- train_one_step: include the new prm_teacher_*_cand and
step_wise_step_token_spans keys in the keep-as-list set when
repacking the rollout batch.
Made-with: Cursor
Core slime changes that unblock running a different-sized PRM teacher next
to the student (e.g. Qwen3-8B teacher with Qwen3-4B student) and add a
token-level top-K OPD distillation path. Also bundles several Qwen3.5
robustness fixes and an SGLang router fast-fail.
PRM teacher and top-K OPD
- actor.py: copy architectural fields (num_layers/hidden_size/ffn/heads/
query_groups/kv/RoPE/MoE/MLA/MTP/vocab) from the teacher's torch_dist
common.pt onto a deepcopied args namespace so the teacher can be a
different model size than the student's MODEL_ARGS without going through
--megatron-to-hf-mode bridge. Adds compute_student_topk and
gather_at_indices wired into ray/actor_group.py.
- loss.py: opt-in emit_topk_logprobs context manager;
_vocab_parallel_topk_log_probs and gather_log_probs_at_indices for
per-token top-K extraction. Default actor/old_actor/ref forwards are
byte-identical to before.
- arguments.py: --prm-teacher-rotary-base, --prm-teacher-megatron-to-hf-mode,
--prm-teacher-hf-checkpoint, --distill-subset-mode {student,teacher,overlap}.
- train_async.py: three-pass dispatch when distill_topk>0 with
subset_mode=student (student top-K, teacher gather, student loss).
- data.py / model.py: carry prm_teacher_topk_log_probs / topk_indices /
topk_log_probs through rollout logging and train_one_step packing.
Bridge / raw mode parity and OOM fix
- model_provider.py: forward CLI flags into bridge providers
(recompute, dropout, attention/dropout/SwiGLU/RoPE fusion, dtype, CP)
so bridge mode does not silently keep HF-config defaults
(e.g. attention_dropout=0.1) and cause train-vs-inference skew.
- model.py: pass fp32_output=False on the train forward to match the
forward-only path; without this, packing 2 samples into a 16K
microbatch produces a 7.58 GiB fp32 logits allocation that OOMs an
80GB GPU.
Qwen3.5 robustness
- backends/sglang_utils/qwen3_5.py: drop the .self_attn. prefix from
Megatron-emitted weight names and remove the erroneous model. re-prefix
so weights load correctly into SGLang's Qwen3.5 full-attention layer
(qkv_proj/o_proj/q_norm/k_norm registered as direct attributes of the
decoder layer, not under self_attn).
- slime_plugins/models/hf_attention.py: _load_hf_config JSON fallback for
Qwen3.5 / Qwen3.5-VL checkpoints whose model_type is not yet registered
in transformers; new _AllGatherForDuplicatedComputation autograd Function
whose backward returns the local slice (avoids the world_size gradient
inflation when wrapping duplicated computation); pass
tensor_parallel_output_grad=False on the SP gather of hidden_states for
hybrid linear-attention layers.
- slime_plugins/models/qwen3_5.py: re-export _load_hf_config.
SGLang router fail-fast
- utils/http_utils.py: stop drowning logs after N consecutive
no_available_workers / circuits-open 503s from the router; tunable via
SLIME_HTTP_PERMANENT_FAIL_FAST (default 3). Also throttles retry log
spam to attempts {1,5,10,25,50}.
Launch scripts (Qwen3.5)
- scripts/run-qwen35-4B.sh: Qwen3.5 4B training launcher.
- scripts/models/qwen3.5-9B-VL.sh, qwen3.5-27B-VL.sh: Qwen3.5 VL configs.
Yinjie Wang