- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper combines verifiable reward in RLVR and teacher guidance in OPD, and proposes ELVER, a token-level credit assignment method that reassigns the token advantage based on teacher likelihood. ELVER ensures three properties:
- Endpoint consistency. The sum of token advantage equals the sequence advantage.
- Uniform credit under neutral evidence. If the teacher likelihood suggests neutral evidence (local hindsight evidence near 1), the assignment should be close to uniform credit.
- Probability bounds. The token value function should lie within [0,1] for binary reward setup.
The paper compares two variants of ELVER with Raw Value Recursion, GRPO and OPD on math benchmarks, covering both on-policy distillation and self-distillation scenarios. ELVER-Box outperforms all baselines on all the math benchmarks.
Critical strengths
None.
Critical weaknesses
- Missing baselines in experiments. The raw value recursion is an intermediate method proposed in the paper. GRPO is an RLVR approach without teacher guidance, OPD does not require verifiable reward signal. The paper does not compare with any published on-policy distillation method with verifiable reward, e.g., the paper mentioned in related work (Line 89-98). Specifically, OPDVR is closely related with the paper's approach and should be compared in experiments.
- The theoretical motivation does not support the method. According to my understanding, the paper's motivation is to estimate the advantage function (Proposition 1), and the teacher policy is used to estimate the hindsight probability. While this is an implicit assumption, the approximation is acceptable. However, this has no clear relation with the three design requirements (Line 210-215) that directly motivates ELVER. While the goals can be viewed as necessary for a valid advantage function, they are not sufficient. By constructing as in ELVER-Affine or ELVER-Box, we are no longer approximating the real advantage function in RL, but constructing a sequence that looks like a reasonable advantage.
Questions for authors
- Can the authors justify why they fail to compare ELVER with OPDVR, RLSD and CSCR in experiments, given that they are acknowledged already as prior work?
- Does it make sense to treat ELVER as a pure heuristic credit assignment approach? I believe the method itself is reasonable, but I cannot agree to motivate it as an approximation to the real advantage function.
4 comments
Response to Lbgd 2/2
We also test our methods on classical RL Games like mazes. Its action is state-dependent and advantages are computed by backward dynamic programming. The results is reproduced from Appendix D, Table 10 in the revised manuscript:
| Map | Horizon | Raw | ELVER-Affine | Box-unbounded | ELVER-Box | Uniform |
|---|---|---|---|---|---|---|
| Looped rooms | 64 | 0.00655 | 0.00641 | 0.00562 | 0.00562 | 0.98900 |
| Looped rooms | 128 | 0.02216 | 0.02195 | 0.01717 | 0.01658 | 1.05093 |
| Looped rooms | 256 | 0.15544 | 0.15471 | 0.13181 | 0.08644 | 0.94701 |
| Staggered gates | 64 | 0.00171 | 0.00169 | 0.00160 | 0.00160 | 1.04037 |
| Staggered gates | 128 | 0.02129 | 0.02097 | 0.01466 | 0.01274 | 0.88099 |
| Staggered gates | 256 | 0.05418 | 0.05386 | 0.04864 | 0.04712 | 0.92775 |
| Four rooms | 64 | 0.00543 | 0.00531 | 0.00435 | 0.00435 | 1.04434 |
| Four rooms | 128 | 0.03255 | 0.03218 | 0.02690 | 0.02561 | 0.91857 |
| Four rooms | 256 | 1.04128 | 1.03869 | 1.00794 | 0.62397 | 1.10421 |
The two experiments support that
- The Box method can provide good approximation of exact advantages under a Good teacher. It may be under-perfomed to the Uniform method with the guidance of a Bad teacher. So the approximation is determined by the teacher quality, which support our above discussion that the HCA theory support exact advantage reconstruction but biased teacher may miss it.
- The Goal III is important. We remove the box constraint (Box-unbounded), and the performance decay.
The more detailed information can be seen in the Appendix C and D of revised manuscript.
Question 1. Comparisons with OPDVR, RLSD, and CSCR
We have addressed the missing OPDVR comparison by adding the experiment described under Weakness 1, together with results from repeated training runs. We also added the Box-unbounded ablation and the two exact-advantage RL experiments to examine the reconstruction mechanism. The revised PDF includes the OPDVR and Box-unbounded results in Table 2 and the RL reconstruction experiments in Appendices C and D.
RLSD and CSCR are relevant additional baselines. Due to the available compute budget and the time constraints of the rebuttal period, we have not been able to complete their training and evaluation. We choose the new baseline comparison on OPDVR, which directly combines the teacher and verifier information used in our setting.
Question 2. Is ELVER a purely heuristic credit assignment method?
As we give a detailed discussion in Weakness 2 part, ELVER would give approximation to the real advantage function affected by teacher quality. Our added two classical RL experiments support this, which report the approximation may be nteed by the quality of teacher (Good or Bad). As we discussed, the approximation to exact advantage function is guaranteed by the HCA theory coming from the Bayesian recursion. Our method is to properly use the biased teacher evidence to reconstruct the approximated advantages under reasonable constraints.
Thanks for the authors' response.
- Regarding the experiment, the authors add OPDVR as a baseline, but RLSD and CSCR are not run due to time constraint. While the reason is acceptable, the baselines are necessary as they are closely related to the proposed method.
- This concern is resolved.
Given the missing baselines in experiments, I will keep the score.
Thank you for the review.
Baselines
RLSD: seeing the paper of RLCSD[1], we choose RLCSD other than RLSD since the performance comparison there, the performance of RLCSD is reported in the first draft. And in the first rebuttal round we think to add RLSD is low-ranked.
CSCR[2]: CSCR does not publish thier original code yet. So we do not add.
RLCSD[1]: RLCSD: Reinforcement learning with contrastive on-policy self-distillation. arXiv preprint arXiv:2606.11709, 2026.
CSCR[2]: Not all tokens deserve equal credit: Counterfactual sensitivity credit reallocation for long-CoT reasoning. arXiv preprint arXiv:2607.27888
Respone to Lbgd 1/2
Thank you for highlighting the need for a comparison of baselines with various seeds, and for asking us to clarify the relationship between our reconstruction criteria and the RL advantage. We have added an OPDVR comparison and two RL experiments with exact advantage references to the revised PDF.
Weakness 1. Missing baselines in experiments.
We have added OPDVR in the DeepSeek-R1-Distill-Qwen-1.5B setting. And to identify the contribution of Probability Bound we have added the Box-unbounded method, who keep the same with ELVER-Box but removing the box condition. The OPDVR and Box-unbounded experiment has been incorporated into Section 5 and Table 2 of the revised manuscript.
The experiment results show that
We additionally train with three seeds under the DeepSeek-R1-Distill-Qwen-1.5B setting. All results use the same hyperparameters in the Appendix B.
Weakness 2. How the three goals relate to approximating the exact advantage
We agree that our writing may get confused when reconstructing advantage functions. In our method, the theory of Hindsight Credit Assignment is the key to reconstruct the exact advantage functions (equation 8). Our main focus is on how to properly use the biased teacher evidence. So we propose three goals:
Endpoint consistency enforces the telescoping identity of the true advantage, ∑tAtπ=R−V1π. This supplies a global constraint which guarantee the reward signal can be used.
Uniform credit under neutral evidence specifies what the estimator does when the teacher supplies no local discrimination. This may explain the question if we do not have any prior intermediate information, what are the advantage like. Motivated by the success of group relative method and minimum energy intuition, we think that the uniform credit is the convergence when teacher evidence disappears.
Probability bounds use the fact that Vtπ=Prπ(R=1∣st)∈[0,1] under binary rewards. The bounds may provide the legal values during reconstruction. Our following experiments show the key role of it in reconstruction.
In general, to approximate the exact advantage is theoretically guaranteed by the HCA. The three goals are the proposed standard of designing an algorithm to properly use the biased teacher evidence.
Moreover for Box, Theorem 3 relates the reconstructed credits directly to Aπ. At the Box optimum, its bound reduces to
\le \sqrt{1+\frac{4b_1^2}{\lambda*}}|\widehat V_1^\pi-V_1^\pi| + \frac{2}{\sqrt{\lambda*}} ||\bar e(\mathbf V^\pi)||_2,$$ under the theorem's assumptions. It is that if we have good estimator of initial value and intermediate value, we can get the exact advantages.