- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Zero-L, a tri-role framework for long-term self-evolution. It adds a trainable Thinker to the Questioner–Solver framework of R-Zero. The Thinker maps the reasoning traces for the same question into hidden representations and gives higher rewards to traces that are closer to the other traces. It is trained using the majority-vote Solver reward in early iterations. An iteration-dependent reward policy gradually reduces the weight of the majority-vote reward and increases the weight of the Thinker reward. Experiments on Qwen3-4B and Qwen3-8B show that Zero-L maintains better performance than R-Zero after 12 iterations.
Critical strengths
- The paper studies an important problem. Performance stagnation and degradation are practical problems in multi-round self-evolution.
- The framework is easy to understand. The Questioner generates problems, the Solver answers them, and the Thinker scores reasoning traces. The Thinker is fixed during each Solver update, which keeps the reward stable within that stage.
- The paper studies a longer training horizon. It reports results at iterations 3, 6, 9, and 12. This makes it possible to see the late-stage difference between Zero-L and R-Zero.
- The reward analysis is useful. The authors use an external model to label sampled responses after training and compare the rewards given to correct and incorrect responses. This provides some evidence for the IDR design.
Critical weaknesses
-
The novelty is limited. Zero-L mainly adds a hidden-state similarity reward and a fixed reward schedule to R-Zero. The Questioner and Solver training follows R-Zero closely. The Thinker reward is also a simple function of the average cosine distance between reasoning traces. The method is therefore an incremental extension of an existing framework.
-
The Thinker reward does not directly measure reasoning correctness. It rewards reasoning traces that are similar to other traces and gives lower rewards to isolated traces. A group of responses can follow the same wrong reasoning pattern, in which case the Thinker may still give them high rewards. The Thinker is also trained using majority-vote labels, so it may transfer early pseudo-label errors into its representation space instead of removing them. The paper does not provide step-level analysis or a study of such failure cases.
-
The experimental baselines are too limited. The main experiments only compare Zero-L with R-Zero. This is not enough to show that Zero-L is better than recent methods for self-evolution, stable co-training, or curriculum construction. Relevant comparisons include:
- [1] Self-Questioning Language Models
- [2] Guided Self-Evolving LLMs with Minimal Human Supervision.
- [3] R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training.
- [4] TTCS: Test-Time Curriculum Synthesis for Self-Evolving.
-
The ablation study is incomplete. The current ablations only freeze the Questioner or replace IDR with a fixed reward mixture. They do not show whether the gain comes from training the Thinker, using representation similarity, or simply changing the reward over time. Important controls include removing the Thinker, using a frozen or untrained Thinker, using only the Thinker reward, and testing other representation pooling methods.
-
Compute and statistical results are not fully reported. Zero-L trains a third Transformer model in addition to the Questioner and Solver. The paper does not report the added training time, memory use, or total GPU hours. Results are averaged over three seeds, but standard deviations and confidence intervals are missing. Since many gains are only 1–3 points, this information is important.
Questions for authors
-
How does Zero-L compare with R-Few, R-Diverse, and TTCS under the same backbone, number of iterations, data budget, and compute budget?
-
What happens if the Thinker is frozen at its initial checkpoint? This would show whether the gain comes from Thinker training or from hidden-state similarity alone.
-
What are the AUC, ranking accuracy, or correlation between the Thinker reward and answer correctness? The current analysis only reports average rewards for correct and incorrect groups.
-
How does the Thinker score cases where several responses share the same reasoning pattern but reach the same wrong answer?
-
How sensitive are the results to , , and ?
6 comments
Thank you for the detailed and transparent response. The added frozen-Thinker, early-switch, compute, and correctness-ranking results address some of my concerns and make the role of the Thinker clearer. However, the matched comparisons with other recent methods are still missing, and the shared-wrong-reasoning case remains an important open failure mode of the centrality-based reward. Since several of the added comparisons also lack variance results, I will keep my current score.
Thank you for reading my response! The comparison experiment is still going, the results will be brought out in 1-2 days. Thank you again for waiting it!
Thank you!
Q1. Baselines and matched compute. R-Zero shares the backbone, Q/S framework, question budget, and Q/S configuration. We lack matched R-Few, R-Diverse, or TTCS runs; our quantitative claim is therefore limited to R-Zero and component controls.
Supplementary profiling uses two RTX PRO 6000 GPUs, separate from the original four-GPU setup. Zero-L adds 0.6 GPU-hours per round (+12.77%). Continued R-Zero training under the 63.6-GPU-hour budget leaves a mean Zero-L advantage of 1.06/1.01 Math/General points. Average peak memory is 83.7/82.9 GB per card. The exact stopping checkpoint, stage-level timing, and control SDs are unavailable, so this is a mean comparison rather than a significance claim.
Q2. Frozen Thinker and Thinker-only reward. Frozen-init retains the pretrained checkpoint, reasoning-token mean pooling, distance mapping, and IDR, but disables Thinker updates. The early-switch control uses majority reward in iteration 1 and pure Thinker reward thereafter while retaining calibration and residual Thinker updates.
Full exceeds frozen-init by 1.86/1.51 points, supporting a learning contribution under the same mixture schedule. It exceeds the early switch by 1.46/1.31 and the fixed mixture by 4.04/2.97 points. These 12-iteration, three-seed controls test the listed configurations, not every transition policy; their SDs are unavailable.
Q3. Correctness ranking. On identical Zero-L response groups, learned/initial/pseudo-label AUROC is 0.70/0.60/0.58, and within-question ranking is 65%/58%/56%. Ranking macro-averages correct/incorrect pairs within each mixed group (ties = 0.5); pooled AUROC additionally reflects cross-question score differences.
The subsets may overlap, and plurality need not exceed 50%. Top-1 averages the correct fraction among tied maximizers. The pseudo-label reward gives 0%/5% Top-1 on the two subsets. A random-selection Top-1 is not reported. Audit labels are post-hoc; common-response scoring does not establish that the evaluated Thinker had not used these responses for calibration, so the evidence is limited to the audited groups.
Q4. Shared wrong reasoning. Eq. 13 rewards relative centrality. A compact shared-error cluster can therefore score highly; if all usable representations coincide, every response receives reward 1 regardless of correctness. Learned versus initial Thinker improves wrong-plurality Top-1 by 9 points. On 450 mixed groups with fewer than 50% correct candidates, ranking is 58%/52%. However, neither subset specifically identifies a majority sharing the same reasoning error, and no response-level shared-error case study is included. Thus, this failure-analysis request remains open.
Q5. Parameter roles and pooling. Defaults are t=0.5/0.4 for 4B/8B, τd=0.1, and α=0.7.
Statistics and scope. Available iteration-12 General Avg sample SDs are 0.83/0.75 (4B R-Zero/Zero-L) and 0.65/0.67 (8B). Math, other checkpoints, ablations, and compute controls lack SDs. The evidence supports the learned, scheduled extension within the tested R-Zero setting; broader cross-method superiority, dedicated shared-error correction, and hyperparameter robustness are not established.