- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a method called Residual Advantage, aiming to improve the post-training process of reinforcement learning with verifiable rewards. This method calculates the probability residuals between the teacher model and the student model on the vocabulary, treats them as bounded single-step rewards, and subtracts the state value under the student policy to construct the advantage function. Additionally, the authors introduce a collaborative residual advantage mechanism, using the feedback from the validator to synchronously update the low-rank adaptation module of the teacher model. The authors evaluated this method on multiple mathematical benchmark tests and claimed that it outperforms traditional online strategy distillation and basic reinforcement learning estimators. However, upon thorough review, this paper has serious flaws in terms of innovation, experimental design, and practical value, and is not recommended for publication.
Critical strengths
-
In the post-training stage of reinforcement learning, how to effectively utilize the fine-grained signals from the teacher model to guide the exploration of the student model is indeed a research direction worth exploring.
-
The author provided a formal proof of the boundaries of the local advantage function and the expected update direction, demonstrating the theoretical consistency of this method.
-
The author verified the effectiveness of the proposed method on two different reinforcement learning estimators, providing certain empirical support.
Critical weaknesses
-
The proposed residual advantage is essentially just a trivial variant of the baseline technique in the standard policy gradient. Using the probability difference as the reward and subtracting the expectation does not break out of the theoretical framework of the traditional advantage function and lacks substantive algorithmic breakthroughs.
-
The author compared a student model with a large teacher model with extremely small parameter quantities. This huge gap in parameter quantities makes it likely that the performance improvement is merely due to the compensation of model capacity rather than the inherent superiority of the proposed guidance mechanism.
-
All experiments were limited to specific mathematical reasoning tasks and were not tested in more challenging real scenarios such as general instruction following, code generation, or complex logical reasoning, resulting in doubts about the practical application value of the method.
-
The collaborative residual advantage mechanism requires simultaneously maintaining and updating the low-rank adaptation module of the teacher model, which significantly increases the memory usage and computing time during training. However, the experimental results show that the performance gain it brings is negligible and cannot prove the rationality of this additional expense.
Questions for authors
see weakness
3 comments
W3 (tasks beyond mathematics). We take this point and have added experiments on text-to-SQL, a code-generation task verified by execution. We train Qwen3-1.7B-Base on BIRD with the same Qwen3-8B teacher, keeping λ=10 and the training recipe of the paper, and report execution accuracy (Avg@8):
| Method | BIRD dev | Spider dev |
|---|---|---|
| Qwen3-1.7B-Base (untrained) | 23.4 | 54.9 |
| OPD (teacher only) | 40.6 | 74.2 |
| REINFORCE++ | 44.3 | 73.8 |
| RA + REINFORCE++ | 46.1 | 75.4 |
| Qwen3-8B teacher (reference) | 52.4 | 82.2 |
RA scores highest among the students on both benchmarks. General instruction following lies outside the scope of RA, which requires a verifiable reward; we will state the evaluation scope explicitly in the paper.
W4 (Co-RA). Co-RA asks whether letting the teacher adapt to the student's behavior yields better guidance, and the experiments show that it does. The gain is not negligible: Co-RA improves over RA in all four settings, and in each setting every Co-RA seed scores above every RA seed (Tables 1 and 18; macro Avg@8):
| Setting | RA | Co-RA |
|---|---|---|
| 1.7B, REINFORCE++ | 29.25 | 30.70 |
| 1.7B, GRPO | 29.86 | 31.09 |
| 4B, REINFORCE++ | 37.12 | 38.39 |
| 4B, GRPO | 37.57 | 38.60 |
The adapted teachers become weaker solvers yet better guides (App. D.9), which is the effect Co-RA is designed to test. We agree that this additional gain should be assessed together with its cost. For completeness, we report the measured cost (8×H20, 1.7B, REINFORCE++, three seeds):
| GPU-hours | Peak memory per GPU (GB) | Avg@8 | Pass@8 | |
|---|---|---|---|---|
| RA (step 270) | 175 | 25 | 29.3 | 45.2 |
| Co-RA (step 180) | 171 | 31 | 29.7 | 45.3 |
| Co-RA (step 270) | 265 | 31 | 30.7 | 46.4 |
Even with slightly less compute than RA, Co-RA scores higher. We present Co-RA as an optional extension; the main contribution of the paper is RA, which adds no generation passes and costs about 10% more per token than REINFORCE++.
We would be glad to answer any further questions.
Gentle reminder
Dear Reviewer vcwL,
Thank you again for your constructive comments on the evaluation beyond mathematics and the additional cost of Co-RA. We would like to follow up on our response and ask whether our clarifications and additional experiments have addressed your main concerns. If any questions remain, we would be happy to provide further details.
We would greatly appreciate it if you could take our response into consideration in your final assessment.
Best regards, The Authors
Response to Reviewer vcwL
We thank the reviewer for the comments. We respectfully note that W2 and W4 are contradicted by results already reported in the paper. Below we address each point with the relevant numbers and our new experiments.
W1 (novelty). We agree that RA keeps the standard form of an advantage, and we do not claim a new algorithmic paradigm. Our goal is to bring the teacher guidance of on-policy distillation into RLVR in a natural and principled way. The advantage framework, from REINFORCE with a baseline (Williams, 1992) to PPO, GRPO and REINFORCE++, has remained the foundation of this line of work because it is simple and well understood, which makes it the natural choice here: RA needs no critic, schedule or threshold, and plugs into GRPO and REINFORCE++ unchanged. RA's contribution is the label that enters this framework. It condenses the teacher–student difference over the whole vocabulary into one scalar per sampled token, centers it within each response, and adds it to the verifier advantage. These choices are not interchangeable. Under a fixed protocol (Tables 2–3), each single change lowers macro Avg@8, and every paired difference excludes zero:
W2 (teacher capacity). A larger teacher guiding a smaller student is the defining setting of knowledge distillation. On-policy distillation (Agarwal et al., 2024; Gu et al., 2024; Lu, 2025) has become a standard stage of LLM post-training precisely because it lets a small model improve quickly under a larger model's guidance. The teacher's capacity is therefore the resource that every method in this setting draws on, not a confound. What distinguishes methods is how well they use the same teacher. Every teacher-guided method below uses the same Qwen3-8B teacher, data and generation budget, and RA outperforms each of them:
Moreover, the student's gain does not track the teacher's own accuracy: at 1.7B, Co-RA's adapted teachers score 1.9–2.3 Avg@8 below the frozen teacher (App. D.9), yet they produce better students.