- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper studies how to train a student LLM using fixed teacher responses and rewards without access to teacher probabilities. APRO first trains a proxy through SFT on both correct and incorrect responses. It then uses the proxy probabilities to compute importance ratios, replaces extreme ratios with current-to-old student policy ratios, and reweights advantages to strengthen positive updates and weaken negative updates based on student probabilities. The method is evaluated on mathematical reasoning and instruction-following tasks.
Critical strengths
-
Learning from external responses when behavior-policy probabilities are unavailable is interesting.
-
The paper includes extensive experiments with a broad set of baselines. It also provides ablation studies to examine the contributions of individual components.
Critical weaknesses
-
In PPO/GRPO, the importance ratio accounts for the mismatch between the rollout policy and the current policy, so its denominator should correspond to the actual rollout policy. When the data comes from an external teacher, using does not correct this mismatch. Using with clipping seems more reasonable as an approximate correction. Then the proposed fallback remains insufficiently justified.
-
The idea of reweighting advantages is natural, but the two mechanisms appear to overlap. In the off-policy setting, very small ratios attenuate gradients for positive advantages, while very large ratios can amplify gradients for negative advantages; these cases are not suppressed by PPO clipping. Your method first switches extreme proxy ratios to , thereby controlling their magnitude. It then reweights the advantage using a clipped scaling factor derived from the trajectory-averaged . However, the small proxy ratios have already been replaced by local ratios at this point. The stated motivation for scaling (gradient attenuation caused by small ratios) does not clearly justify its role after switching. A more direct analysis of the combined mechanism is needed to establish why the additional reweighting is necessary.
-
I have concerns about the proxy setup. I understand that the proxy is intended to approximate the teacher’s full sampling distribution. However, its necessity in the actual experimental setting is unclear, since the teachers provide probabilities. Moreover, with only eight responses per question, it is unclear whether the proxy can adequately approximate the teacher distribution, rather than overfit the observed correct and incorrect responses by increasing the likelihood of their tokens. More direct evidence of the proxy’s probability estimation quality and generalization is needed. Also, the switching rule depends on the proxy ratio, but this ratio conflates two possibilities: (1) a genuine discrepancy between the teacher and student policies, and (2) an inaccurate proxy estimate of the teacher policy. Even a perfectly accurate proxy can produce extreme ratios when the student differs substantially from the teacher. Conversely, a proxy close to the student, yielding a ratio near one, is not necessarily close to the teacher.
-
Comparing other algorithms in your off-policy setting is also strange. These algorithms were designed to address off-policy issues in online or mixed-policy RL, not in your setting. Overall, your method may be a heuristic for changing update magnitudes and the weights of positive and negative samples, but it has little to do with achieving more reliable off-policy distribution correction.
-
The theoretical stability guarantee relies on an assumption that the algorithm itself does not ensure. Appendix C.2 assumes that However, PPO clipping does not guarantee that the actual probability ratio remains within this interval. It only modifies certain branches of the objective rather than imposing a hard constraint on parameter updates. Therefore, the theorem is conditional and does not directly establish the stability of the actual algorithm.
Questions for authors
See Weaknesses.
9 comments
W2. The idea of reweighting advantages is natural, but the two mechanisms appear to overlap. In the off-policy setting, very small ratios attenuate gradients for positive advantages, while very large ratios can amplify gradients for negative advantages; these cases are not suppressed by PPO clipping. Your method first switches extreme proxy ratios to , thereby controlling their magnitude. It then reweights the advantage using a clipped scaling factor derived from the trajectory-averaged . However, the small proxy ratios have already been replaced by local ratios at this point. The stated motivation for scaling (gradient attenuation caused by small ratios) does not clearly justify its role after switching. A more direct analysis of the combined mechanism is needed to establish why the additional reweighting is necessary.
A2:
Why Switching and Scaling are complementary rather than overlapping. Adaptive Ratio Switching (ARS) and Discrepancy-Aware Advantage Scaling operate at different levels and solve distinct optimization problems:
- First, 67.95% of tokens are NOT switched to : Whenever both and assign small but similar probabilities to a hard token, remains inside ( of all tokens). These tokens never trigger fallback, yet the trajectory can still be far from the student's current mode (, yielding a large ).
- Second, on positive trajectories (), switching resets extreme token ratios to , which erases relative trajectory difficulty: When low-probability tokens in a high-reward external trajectory switch to , switching prevents the gradient from vanishing to , but it assigns the exact same unit weight () to an under-explored reasoning trajectory () as to an easy trajectory the student already masters (). Multiplying by amplifies updates on under-explored positive trajectories so the student prioritizes acquiring new reasoning behaviors.
- Third, on negative trajectories (), switching and scaling act in opposite, complementary directions: When an incorrect external trajectory has very low probability under the student (), switching its extreme ratios to prevents ratio explosion, but leaves a full-magnitude negative penalty () on actions the student already considers unlikely! Spending update budget to further push down low-probability negative tokens distorts the student's distribution (causing the entropy spike and accuracy collapse seen in
GRPO w/ Local Ratioin Figure 2b & Figure 3 Bottom). Scaling by attenuates penalties on low-probability negative trajectories, preserving update budget for informative errors.
Direct empirical and diagnostic proof of complementarity (Table 3, Table R2, and Non-Saturation Diagnostics).
- Orthogonal ablation: In Table 3 (Section 5.2),
ARS only(switching without scaling) achieves 41.1% ID / 22.9% OOD, whilew/o ARS (Local)(scaling with everywhere, without proxy switching) achieves 42.9% ID / 25.6% OOD. Combining both in APRO reaches 45.3% ID / 33.1% OOD—improving over switching alone by +4.2% ID / +10.2% OOD and over scaling alone by +2.4% ID / +7.5% OOD. - Non-saturation and adaptive trajectory reweighting (Table R2): Across all nonzero-advantage sequence evaluations in training, only 0.24% of positive multipliers hit the upper cap of (mean ), and only 0.36% of negative multipliers hit the lower cap of (mean , adapting from across early, middle, and late training as the student learns). Furthermore, in a 3-seed controlled experiment (Table R2), freezing the multipliers to their empirical means (,
APRO(GRPO) fixed weight) yields ID / OOD, whereas adaptive APRO(GRPO) achieves ID / OOD (+3.2% ID and +4.9% OOD). With switching enabled in both variants, trajectory-specific scaling gave higher mean ID and OOD scores than the tested fixed weights across these three runs.
Table R2: Orthogonal and fixed-weight ablations of Switching (ARS) and Discrepancy Scaling on DeepSeek-R1-Distill-Qwen-1.5B.
| Method Variant | Ratio Denominator | Advantage Scaling | ID Avg. (%) | OOD Avg. (%) |
|---|---|---|---|---|
GRPO w/ Local Ratio | only | None () | 41.8 | 13.0 |
GRPO w/ Proxy Ratio | only | None () | 39.1 | 23.5 |
ARS only | ARS () | None () | 41.1 | 22.9 |
w/o ARS (Local) | only | Adaptive () | 42.9 | 25.6 |
w/o ARS (Proxy) | only | Adaptive () | 36.8 | 25.8 |
APRO(GRPO) fixed weight (3 runs) | ARS () | Fixed () | ||
| APRO(GRPO) (Full, 3 runs) | ARS () | Adaptive () |
Response to Reviewer Coz2
We sincerely appreciate your careful review and constructive suggestions. Your feedback has helped us strengthen our experimental evaluation and improve the clarity of the manuscript. We provide point-by-point responses to your comments below.
A1:
Why πθ/πproj with standard PPO clipping is insufficient. We directly evaluated the exact baseline you suggested—using πθ/πproj with standard PPO clipping—throughout our paper as
PPO/GRPO w/ Proxy Ratio(see Figure 1, Figure 2, Figure 3 Top, Table 1, and Row 2 of Table 3). Both theoretical analysis (Appendix C.3.1, Eq. 16) and empirical dynamics show why standard PPO clipping min(rtA^t,clip(rt,1−ϵ,1+ϵ)A^t) fails to stabilize πθ/πproj on external teacher trajectories:Why falling back to rsafe,t=πθ/πold resolves both failure modes. We do not use πθ/πold to estimate the teacher distribution; rather, we use it as a local trust-region anchor when rproj,t leaves the reliable interval [1−ϵp,1+ϵp] (Section 3.2). Within [1−ϵp,1+ϵp] (67.95% of tokens in our training diagnostics), APRO retains the behavior-corrected proxy ratio rproj,t. When rproj,t is extreme (32.05% of tokens), switching to rsafe,t=πθ(ot∣ct)/πold(ot∣ct)≈1 simultaneously (i) restores a unit-scale trust-region update on low-probability positive tokens so the student can actually learn new teacher behaviors, and (ii) prevents inflated proxy ratios from destabilizing the policy.
Direct empirical verification (Table 3 & Figure 1). In Table 3 (Section 5.2),
GRPO w/ Proxy Ratio(πθ/πproj with PPO clipping) achieves only 39.1% ID / 23.5% OOD. Adding Adaptive Ratio Switching (ARS) alone raises ID performance to 41.1% while reducing the clip rate from >30% to <5% (Figure 1c). When combined with discrepancy scaling, using πθ/πproj with clipping (w/o ARS (Proxy)) yields only 36.8% ID / 25.8% OOD, whereas APRO (with ARS) achieves 45.3% ID / 33.1% OOD—an improvement of +8.5% ID and +7.3% OOD.