acceptodds
Under review as a conference paper at ICLR 2027

Beyond Self-Exploration: Stable Off-Policy Reinforcement Learning for LLMs

Abstract

Reinforcement learning (RL) has become a central paradigm for aligning LLMs, yet most existing approaches rely on on-policy self-exploration, which can lead to optimization stagnation when the policy is weak due to low-quality rollouts. Leveraging high-quality external trajectories offers a natural remedy, but this off-policy setting is notoriously unstable. Through an empirical analysis of PPO and GRPO, we identify unregulated importance ratios induced by off-policy distributions as a primary cause of this instability. Beyond instability, off-policy RL also suffers from attenuated gradients, as informative off-policy tokens typically receive low probability under the target policy. To address these challenges, we introduce Adaptive Projected Robust Optimization (APRO) to enable stable off-policy RL in LLM optimization. APRO constructs a tractable proxy for the unknown off-policy distribution and adaptively switches to a local policy anchor when proxy importance-ratio estimates become unreliable. In addition, APRO introduces discrepancy-aware advantage scaling to amplify informative learning signals that would otherwise be suppressed. We evaluate APRO on mathematical reasoning and instruction-following tasks in fully off-policy settings. Experimental results show that APRO achieves stable optimization and improves over the evaluated offline distillation and ratio-based off-policy RL baselines. Further experiments show that APRO can be used in mixed-policy training and can provide stronger initializations for subsequent on-policy RL, highlighting stable off-policy RL as a useful capability-transfer stage for weak target policies.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language models, Off-Policy Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Coz2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies how to train a student LLM using fixed teacher responses and rewards without access to teacher probabilities. APRO first trains a proxy through SFT on both correct and incorrect responses. It then uses the proxy probabilities to compute importance ratios, replaces extreme ratios with current-to-old student policy ratios, and reweights advantages to strengthen positive updates and weaken negative updates based on student probabilities. The method is evaluated on mathematical reasoning and instruction-following tasks.

Critical strengths
  1. Learning from external responses when behavior-policy probabilities are unavailable is interesting.

  2. The paper includes extensive experiments with a broad set of baselines. It also provides ablation studies to examine the contributions of individual components.

Critical weaknesses
  1. In PPO/GRPO, the importance ratio accounts for the mismatch between the rollout policy and the current policy, so its denominator should correspond to the actual rollout policy. When the data comes from an external teacher, using πθ/πold\pi_\theta/\pi_{\mathrm{old}} does not correct this mismatch. Using πθ/πproj\pi_\theta/\pi_{\mathrm{proj}} with clipping seems more reasonable as an approximate correction. Then the proposed fallback remains insufficiently justified.

  2. The idea of reweighting advantages is natural, but the two mechanisms appear to overlap. In the off-policy setting, very small ratios attenuate gradients for positive advantages, while very large ratios can amplify gradients for negative advantages; these cases are not suppressed by PPO clipping. Your method first switches extreme proxy ratios to rlocalr_{\mathrm{local}}, thereby controlling their magnitude. It then reweights the advantage using a clipped scaling factor derived from the trajectory-averaged δt\delta_t. However, the small proxy ratios have already been replaced by local ratios at this point. The stated motivation for scaling (gradient attenuation caused by small ratios) does not clearly justify its role after switching. A more direct analysis of the combined mechanism is needed to establish why the additional reweighting is necessary.

  3. I have concerns about the proxy setup. I understand that the proxy is intended to approximate the teacher’s full sampling distribution. However, its necessity in the actual experimental setting is unclear, since the teachers provide probabilities. Moreover, with only eight responses per question, it is unclear whether the proxy can adequately approximate the teacher distribution, rather than overfit the observed correct and incorrect responses by increasing the likelihood of their tokens. More direct evidence of the proxy’s probability estimation quality and generalization is needed. Also, the switching rule depends on the proxy ratio, but this ratio conflates two possibilities: (1) a genuine discrepancy between the teacher and student policies, and (2) an inaccurate proxy estimate of the teacher policy. Even a perfectly accurate proxy can produce extreme ratios when the student differs substantially from the teacher. Conversely, a proxy close to the student, yielding a ratio near one, is not necessarily close to the teacher.

  4. Comparing other algorithms in your off-policy setting is also strange. These algorithms were designed to address off-policy issues in online or mixed-policy RL, not in your setting. Overall, your method may be a heuristic for changing update magnitudes and the weights of positive and negative samples, but it has little to do with achieving more reliable off-policy distribution correction.

  5. The theoretical stability guarantee relies on an assumption that the algorithm itself does not ensure. Appendix C.2 assumes that rlocal∈[1−ϵ,1+ϵ].r_{\mathrm{local}}\in[1-\epsilon,1+\epsilon]. However, PPO clipping does not guarantee that the actual probability ratio remains within this interval. It only modifies certain branches of the objective rather than imposing a hard constraint on parameter updates. Therefore, the theorem is conditional and does not directly establish the stability of the actual algorithm.

Questions for authors

See Weaknesses.

9 comments
AuthorsOpenReview ↗

Response to Reviewer Coz2

We sincerely appreciate your careful review and constructive suggestions. Your feedback has helped us strengthen our experimental evaluation and improve the clarity of the manuscript. We provide point-by-point responses to your comments below.

W1. In PPO/GRPO, the importance ratio accounts for the mismatch between the rollout policy and the current policy, so its denominator should correspond to the actual rollout policy. When the data comes from an external teacher, using πθ/πold\pi_\theta/\pi_{\text{old}} does not correct this mismatch. Using πθ/πproj\pi_\theta/\pi_{\text{proj}} with clipping seems more reasonable as an approximate correction. Then the proposed fallback remains insufficiently justified.

A1:

Why πθ/πproj\pi_\theta/\pi_{\text{proj}} with standard PPO clipping is insufficient. We directly evaluated the exact baseline you suggested—using πθ/πproj\pi_\theta/\pi_{\text{proj}} with standard PPO clipping—throughout our paper as PPO/GRPO w/ Proxy Ratio (see Figure 1, Figure 2, Figure 3 Top, Table 1, and Row 2 of Table 3). Both theoretical analysis (Appendix C.3.1, Eq. 16) and empirical dynamics show why standard PPO clipping min⁡(rtA^t,clip(rt,1−ϵ,1+ϵ)A^t)\min(r_t \hat{A}_t, \text{clip}(r_t, 1-\epsilon, 1+\epsilon)\hat{A}_t) fails to stabilize πθ/πproj\pi_\theta/\pi_{\text{proj}} on external teacher trajectories:

  • Unclipped signal attenuation on positive tokens (A^t>0\hat{A}_t > 0): For A^t>0\hat{A}_t > 0, PPO clipping only caps ratios above 1+ϵ1+\epsilon. When an external trajectory contains crucial reasoning tokens that the weak student currently assigns very low probability (πθ(ot∣ct)≪πproj(ot∣ct)\pi_\theta(o_t|c_t) \ll \pi_{\text{proj}}(o_t|c_t)), the proxy ratio approaches zero (rproj,t→0r_{\text{proj}, t} \to 0). The min⁡\min operator leaves rproj,t→0r_{\text{proj}, t} \to 0 completely unclipped, multiplying the policy gradient A^trproj,t∇θlog⁡πθ(ot∣ct)\hat{A}_t r_{\text{proj}, t} \nabla_\theta \log \pi_\theta(o_t|c_t) by a near-zero weight and suppressing learning on the most informative tokens (shown by the red "Signal Attenuation" curve in Figure 1a and Figure 3 Top).
  • Unclipped variance explosion on negative tokens (A^t<0\hat{A}_t < 0): For A^t<0\hat{A}_t < 0, standard PPO clipping only clips ratios below 1−ϵ1-\epsilon. Whenever πproj\pi_{\text{proj}} underestimates support on a token (πθ(ot∣ct)>πproj(ot∣ct)\pi_\theta(o_t|c_t) > \pi_{\text{proj}}(o_t|c_t) so rproj,t≫1+ϵr_{\text{proj}, t} \gg 1+\epsilon), the min⁡\min operator selects the unclipped large negative term rproj,tA^tr_{\text{proj}, t} \hat{A}_t, causing severe trust-region violations (Figure 1c).

Why falling back to rsafe,t=πθ/πoldr_{\text{safe}, t} = \pi_\theta/\pi_{\text{old}} resolves both failure modes. We do not use πθ/πold\pi_\theta/\pi_{\text{old}} to estimate the teacher distribution; rather, we use it as a local trust-region anchor when rproj,tr_{\text{proj}, t} leaves the reliable interval [1−ϵp,1+ϵp][1-\epsilon_p, 1+\epsilon_p] (Section 3.2). Within [1−ϵp,1+ϵp][1-\epsilon_p, 1+\epsilon_p] (67.95% of tokens in our training diagnostics), APRO retains the behavior-corrected proxy ratio rproj,tr_{\text{proj}, t}. When rproj,tr_{\text{proj}, t} is extreme (32.05% of tokens), switching to rsafe,t=πθ(ot∣ct)/πold(ot∣ct)≈1r_{\text{safe}, t} = \pi_\theta(o_t|c_t)/\pi_{\text{old}}(o_t|c_t) \approx 1 simultaneously (i) restores a unit-scale trust-region update on low-probability positive tokens so the student can actually learn new teacher behaviors, and (ii) prevents inflated proxy ratios from destabilizing the policy.

Direct empirical verification (Table 3 & Figure 1). In Table 3 (Section 5.2), GRPO w/ Proxy Ratio (πθ/πproj\pi_\theta/\pi_{\text{proj}} with PPO clipping) achieves only 39.1% ID / 23.5% OOD. Adding Adaptive Ratio Switching (ARS) alone raises ID performance to 41.1% while reducing the clip rate from >30%>30\% to <5%<5\% (Figure 1c). When combined with discrepancy scaling, using πθ/πproj\pi_\theta/\pi_{\text{proj}} with clipping (w/o ARS (Proxy)) yields only 36.8% ID / 25.8% OOD, whereas APRO (with ARS) achieves 45.3% ID / 33.1% OOD—an improvement of +8.5% ID and +7.3% OOD.

W2. The idea of reweighting advantages is natural, but the two mechanisms appear to overlap. In the off-policy setting, very small ratios attenuate gradients for positive advantages, while very large ratios can amplify gradients for negative advantages; these cases are not suppressed by PPO clipping. Your method first switches extreme proxy ratios to rlocalr_{\text{local}}, thereby controlling their magnitude. It then reweights the advantage using a clipped scaling factor derived from the trajectory-averaged δt\delta_t. However, the small proxy ratios have already been replaced by local ratios at this point. The stated motivation for scaling (gradient attenuation caused by small ratios) does not clearly justify its role after switching. A more direct analysis of the combined mechanism is needed to establish why the additional reweighting is necessary.

A2:

Why Switching and Scaling are complementary rather than overlapping. Adaptive Ratio Switching (ARS) and Discrepancy-Aware Advantage Scaling operate at different levels and solve distinct optimization problems:

  • First, 67.95% of tokens are NOT switched to rlocalr_{\text{local}}: Whenever both πθ(ot∣ct)\pi_\theta(o_t|c_t) and πproj(ot∣ct)\pi_{\text{proj}}(o_t|c_t) assign small but similar probabilities to a hard token, rproj,tr_{\text{proj}, t} remains inside [1−ϵp,1+ϵp][1-\epsilon_p, 1+\epsilon_p] (67.95%67.95\% of all tokens). These tokens never trigger fallback, yet the trajectory can still be far from the student's current mode (max⁡vπθ(v∣ct)≫πθ(ot∣ct)\max_v \pi_\theta(v|c_t) \gg \pi_\theta(o_t|c_t), yielding a large Δi\Delta_i).
  • Second, on positive trajectories (A^i>0\hat{A}_i > 0), switching resets extreme token ratios to ≈1\approx 1, which erases relative trajectory difficulty: When low-probability tokens in a high-reward external trajectory switch to rsafe,t=πθ/πold≈1r_{\text{safe}, t} = \pi_\theta/\pi_{\text{old}} \approx 1, switching prevents the gradient from vanishing to 00, but it assigns the exact same unit weight (≈1\approx 1) to an under-explored reasoning trajectory (Δi≫1\Delta_i \gg 1) as to an easy trajectory the student already masters (Δi≈1\Delta_i \approx 1). Multiplying A^i\hat{A}_i by Δi>1\Delta_i > 1 amplifies updates on under-explored positive trajectories so the student prioritizes acquiring new reasoning behaviors.
  • Third, on negative trajectories (A^i<0\hat{A}_i < 0), switching and scaling act in opposite, complementary directions: When an incorrect external trajectory has very low probability under the student (Δi≫1\Delta_i \gg 1), switching its extreme ratios to rsafe,t≈1r_{\text{safe}, t} \approx 1 prevents ratio explosion, but leaves a full-magnitude negative penalty (1.0⋅A^i1.0 \cdot \hat{A}_i) on actions the student already considers unlikely! Spending update budget to further push down low-probability negative tokens distorts the student's distribution (causing the entropy spike and accuracy collapse seen in GRPO w/ Local Ratio in Figure 2b & Figure 3 Bottom). Scaling by Δi−1<1\Delta_i^{-1} < 1 attenuates penalties on low-probability negative trajectories, preserving update budget for informative errors.

Direct empirical and diagnostic proof of complementarity (Table 3, Table R2, and Non-Saturation Diagnostics).

  • Orthogonal ablation: In Table 3 (Section 5.2), ARS only (switching without scaling) achieves 41.1% ID / 22.9% OOD, while w/o ARS (Local) (scaling with rsafer_{\text{safe}} everywhere, without proxy switching) achieves 42.9% ID / 25.6% OOD. Combining both in APRO reaches 45.3% ID / 33.1% OOD—improving over switching alone by +4.2% ID / +10.2% OOD and over scaling alone by +2.4% ID / +7.5% OOD.
  • Non-saturation and adaptive trajectory reweighting (Table R2): Across all 263,504263,504 nonzero-advantage sequence evaluations in training, only 0.24% of positive multipliers hit the upper cap of 3.03.0 (mean w+=1.647w_+ = 1.647), and only 0.36% of negative multipliers hit the lower cap of 0.10.1 (mean w−=0.513w_- = 0.513, adapting from 0.580→0.505→0.4550.580 \to 0.505 \to 0.455 across early, middle, and late training as the student learns). Furthermore, in a 3-seed controlled experiment (Table R2), freezing the multipliers to their empirical means (w+≡1.647,w−≡0.513w_+ \equiv 1.647, w_- \equiv 0.513, APRO(GRPO) fixed weight) yields 42.2±0.9%42.2 \pm 0.9\% ID / 27.5±1.2%27.5 \pm 1.2\% OOD, whereas adaptive APRO(GRPO) achieves 45.4±0.1%45.4 \pm 0.1\% ID / 32.4±1.1%32.4 \pm 1.1\% OOD (+3.2% ID and +4.9% OOD). With switching enabled in both variants, trajectory-specific scaling gave higher mean ID and OOD scores than the tested fixed weights across these three runs.

Table R2: Orthogonal and fixed-weight ablations of Switching (ARS) and Discrepancy Scaling on DeepSeek-R1-Distill-Qwen-1.5B.

Method VariantRatio DenominatorAdvantage ScalingID Avg. (%)OOD Avg. (%)
GRPO w/ Local Ratioπold\pi_{\text{old}} onlyNone (1.01.0)41.813.0
GRPO w/ Proxy Ratioπproj\pi_{\text{proj}} onlyNone (1.01.0)39.123.5
ARS onlyARS (πproj↔πold\pi_{\text{proj}} \leftrightarrow \pi_{\text{old}})None (1.01.0)41.122.9
w/o ARS (Local)πold\pi_{\text{old}} onlyAdaptive (Δisgn(A^i)\Delta_i^{\text{sgn}(\hat{A}_i)})42.925.6
w/o ARS (Proxy)πproj\pi_{\text{proj}} onlyAdaptive (Δisgn(A^i)\Delta_i^{\text{sgn}(\hat{A}_i)})36.825.8
APRO(GRPO) fixed weight (3 runs)ARS (πproj↔πold\pi_{\text{proj}} \leftrightarrow \pi_{\text{old}})Fixed (w+=1.647,w−=0.513w_+ = 1.647, w_- = 0.513)42.2±0.942.2 \pm 0.927.5±1.227.5 \pm 1.2
APRO(GRPO) (Full, 3 runs)ARS (πproj↔πold\pi_{\text{proj}} \leftrightarrow \pi_{\text{old}})Adaptive (Δisgn(A^i)\Delta_i^{\text{sgn}(\hat{A}_i)})45.4±0.145.4 \pm 0.132.4±1.132.4 \pm 1.1
Reviewer_Do9eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

On-policy RL requires fresh rollouts and can struggle with exploration. This paper proposes Adaptive Projected Robust Optimization (APRO) for offline LLM post-training. It uses static, reward-labeled trajectories from external policies whose sampling probabilities are unavailable. APRO builds on a PPO-style cliped objective It combines a learned behavior-policy proxy with adaptive switching between proxy-based and previous-policy ratios. Discrepancy-aware advantage scaling strengthens positive-advantage updates and attenuates negative-advantage updates. Experiments cover mathematical reasoning and instruction following tasks showing the effectiveness of APRO.

Critical strengths
  1. The motivation of the paper is clear.
  2. Learning from these trajectories without access to behavior-policy probabilities is an important research problem.
  3. Its main contribution is an offline policy-gradient method that only uses the training instances from a static dataset. The resulting policy can initialize subsequent on-policy RL.
  4. Overall, the paper is proporlly written and easy to follow.
Critical weaknesses
  1. The SFT-trained behavior proxy may introduce substantial errors into importance weights. Even unbiased probability estimates need not yield unbiased reciprocal weights. Appendix C.1 identifies proxy-induced bias, but the sensitivity to the ratio bias in the implemented update remains unclear, particularly under strong behavior mismatch.
  2. Beyond the empirical study, it's important to understand what's the optimal policy coming out of the new objective for the proposed constructed surrogate in equation (6). Because of shifting the importance rate may deviate the goal from improving the expected cummulative rewards J(π)J(\pi). I found discussion in the paper missing the policy improvements guarantee from the new algorithm.
  3. A-LoL [1] already uses a fitted reference policy for importance weighting, yet is neither discussed nor evaluated. The novelty of APRO’s particular combination needs clearer justification.
  4. The stabilization role of ratio switching is clear, but its contribution to OOD generalization is less well understood. Table 4 shows substantial sensitivity to the switching threshold. Table 3 suggests that the OOD gains depend on the interaction between switching and advantage scaling. Clarifying this interaction and the threshold-selection procedure would strengthen the paper.

[1] Baheti, Ashutosh, et al. “Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models.” International Conference on Learning Representations (2024): 13793–13823.

Questions for authors
  1. What are the teacher models’ scores on the same evaluation benchmarks under a matched evaluation protocol? Reporting these alongside APRO would clarify the teacher–student performance gap.
  2. I'm confused what does PPO/GRPO + SFT means in the experiment? Are they under the offline setting or online setting?
  3. Under what coverage and relative-error conditions can the proxy-induced bias be controlled? Can the oracle-behavior setting assess these conditions, including robustness to model capacity or behavior mismatch?
  4. How does APRO differ from A-LoL [1]? Could the authors compare the two methods using the same backbone and offline dataset?
  5. How was the switching threshold selected, and was selection independent of the reported test sets? Does proximity of the proxy ratio to one correlate with smaller behavior-estimation error in the oracle setting?
7 comments
AuthorsOpenReview ↗

Response to Reviewer Do9e

We sincerely appreciate your careful review and constructive suggestions. Your feedback has helped us strengthen our experimental evaluation and improve the clarity of the manuscript. We provide point-by-point responses to your comments below.

W1. The SFT-trained behavior proxy may introduce substantial errors into importance weights. Even unbiased probability estimates need not yield unbiased reciprocal weights. Appendix C.1 identifies proxy-induced bias, but the sensitivity to the ratio bias in the implemented update remains unclear, particularly under strong behavior mismatch.

Q3. Under what coverage and relative-error conditions can the proxy-induced bias be controlled? Can the oracle-behavior setting assess these conditions, including robustness to model capacity or behavior mismatch?

A1:

Explicit coverage and relative-error conditions for controlling reciprocal weight bias. We agree that bounded errors in πproj\pi_{\text{proj}} do not automatically bound reciprocal importance weights unless support coverage and relative-error conditions hold. In Appendix C.1 (Theorem C.1), let μ(x)\mu(x) denote the behavior distribution and πproj(x)\pi_{\text{proj}}(x) the projected proxy policy. Assume:

  1. Support coverage: supp(πθ)⊆supp(μ)⊆supp(πproj)\text{supp}(\pi_\theta) \subseteq \text{supp}(\mu) \subseteq \text{supp}(\pi_{\text{proj}}); and
  2. Bounded relative probability error: ∣πproj(x)/μ(x)−1∣≤η<1|\pi_{\text{proj}}(x)/\mu(x) - 1| \le \eta < 1 on supp(πθ)\text{supp}(\pi_\theta).

Then the reciprocal ratio error in Theorem C.1 (Eq. 9) satisfies ∣μ(x)/πproj(x)−1∣=∣μ(x)−πproj(x)∣πproj(x)≤η1−η|\mu(x)/\pi_{\text{proj}}(x) - 1| = \frac{|\mu(x) - \pi_{\text{proj}}(x)|}{\pi_{\text{proj}}(x)} \le \frac{\eta}{1-\eta}. Consequently, for any response-level advantage A(x)A(x) with finite score-gradient norm, the bias between the proxy surrogate gradient gproxyg_{\text{proxy}} and the true off-policy surrogate gradient gμg_\mu is strictly bounded by:

∥gproxy−gμ∥=∥Ex∼πθ[(μ(x)πproj(x)−1)A(x)∇θlog⁡πθ(x)]∥≤η1−η,Ex∼πθ[∣A(x)∣,∥∇θlog⁡πθ(x)∥].\lVert g_{\mathrm{proxy}}-g_{\mu}\rVert=\lVert\mathbb{E}_{x\sim\pi_{\theta}}[(\frac{\mu(x)}{\pi_{\mathrm{proj}}(x)}-1)A(x)\nabla_{\theta}\log\pi_{\theta}(x)]\rVert\leq\frac{\eta}{1-\eta}\\,\mathbb{E}_{x\sim\pi_{\theta}}[|A(x)|\\,\lVert\nabla_{\theta}\log\pi_{\theta}(x)\rVert].

How our implemented token-level update controls sensitivity to reciprocal bias under strong mismatch. In practice, when the teacher μ\mu is much stronger than the student family Θ\Theta, the relative error η\eta can become large in low-support tail tokens where πproj(ot∣ct)→0\pi_{\text{proj}}(o_t|c_t) \to 0, inflating the reciprocal weight 1/πproj(ot∣ct)1/\pi_{\text{proj}}(o_t|c_t). APRO directly controls sensitivity to these tail errors through two mechanisms:

  • Adaptive Ratio Switching (Section 3.2): We only trust the proxy ratio rproj,t=πθ(ot∣ct)/πproj(ot∣ct)r_{\text{proj}, t} = \pi_\theta(o_t|c_t)/\pi_{\text{proj}}(o_t|c_t) when ∣rproj,t−1∣≤ϵp=0.1|r_{\text{proj}, t} - 1| \le \epsilon_p = 0.1 (67.95% of tokens in our training diagnostics), where πproj\pi_{\text{proj}} has sufficient support and reciprocal amplification is bounded inside [0.9,1.1][0.9, 1.1]. Whenever rproj,tr_{\text{proj}, t} falls outside [0.9,1.1][0.9, 1.1] (32.05% of tokens), APRO switches to the local trust-region anchor rsafe,t=πθ(ot∣ct)/πold(ot∣ct)r_{\text{safe}, t} = \pi_\theta(o_t|c_t)/\pi_{\text{old}}(o_t|c_t), preventing reciprocal proxy errors from entering the gradient.
  • Projector sensitivity verification (Appendix I.2, Table 9): To empirically test sensitivity to proxy estimation quality on DeepSeek-R1-Distill-Qwen-1.5B, we varied both the SFT training duration of πproj\pi_{\text{proj}} (1, 3, and 5 epochs) and its training data coverage (All trajectories vs. Positive-only or Negative-only subsets). Across 1, 3, and 5 epochs on all trajectories, APRO maintains stable ID scores (45.8% / 44.7% / 45.3%) while OOD generalization improves steadily (26.5% →\to 29.7% →\to 33.1%). By contrast, violating support coverage by training πproj\pi_{\text{proj}} on positive-only (43.4% ID / 23.9% OOD) or negative-only (43.7% ID / 24.1% OOD) trajectories degrades OOD accuracy by $\sim$9 points, directly confirming the importance of full-support projection.

Comparison under a known behavior denominator (Appendix G, Table 6). In Appendix G (Table 6), both updates use the conditional sampling probabilities μ(ot∣ct)\mu(o_t|c_t) of the same 32B teacher (DeepSeek-R1-Distill-Qwen-32B) to train the same 1.5B student. Standard IS (Oracle) reaches 38.6% ID / 9.9% OOD. Adding switching and discrepancy-aware scaling together raises performance to 43.6% ID / 32.3% OOD, a gain of 5.0 ID and 22.4 OOD percentage points. The proxy-based result of 45.3% ID / 33.1% OOD in Table 1 comes from the separate multi-teacher setting.

W2. Beyond the empirical study, it's important to understand what's the optimal policy coming out of the new objective for the proposed constructed surrogate in equation (6). Because of shifting the importance rate may deviate the goal from improving the expected cumulative rewards J(π)J(\pi). I found discussion in the paper missing the policy improvements guarantee from the new algorithm.

A2:

Token-level reweighting in the Eq. (6) surrogate. Eq. (6) optimizes a KL-regularized, reward-weighted surrogate on recorded external prefixes. For a fixed selected denominator, Theorem C.4 (Appendix C.3.2) describes how a detached positive multiplier changes an individual active token contribution.

  • Both the switched importance ratio r^t>0\hat{r}_t > 0 and the detached discrepancy multiplier wi,t=clip(Δisgn(A^i,t),wmin⁡,wmax⁡)∈[0.1,3.0]>0w_{i,t} = \text{clip}(\Delta_i^{\text{sgn}(\hat{A}_{i,t})}, w_{\min}, w_{\max}) \in [0.1, 3.0] > 0 are strictly positive.
  • On an active, differentiable branch, g~i,t=wi,tgi,t\tilde g_{i,t}=w_{i,t}g_{i,t}. The multiplier preserves the direction of this contribution and bounds its change in magnitude. Theorem C.3 (Appendix C.2) bounds the selected ratio under Assumption C.2 and gives a finite second moment for the per-token surrogate gradient under its stated moment condition. The full update combines contributions through shared parameters and includes KL regularization. These results give no general closed-form optimum for Eq. (6) or monotonic improvement guarantee for the on-policy return J(π)J(\pi).

Empirical performance of the trained student. We evaluate the resulting policy on held-out ID and OOD benchmarks.

  • Under exact teacher probabilities (Appendix G, Table 6), the combined APRO update improves the benchmark averages from 38.6% ID / 9.9% OOD (Standard IS) to 43.6% ID / 32.3% OOD.
  • Across all mathematical reasoning benchmarks (Table 1), APRO(GRPO) improves the ID average over the base model from 16.0% →\to 33.2% ID (Qwen-2.5-Math-1.5B) and 37.2% →\to 45.3% ID (DeepSeek-R1-Distill-Qwen-1.5B), and further enables subsequent on-policy GRPO (Appendix H, Table 7) to reach 46.6% ID / 35.9% OOD (surpassing SFT+RL at 45.5% ID / 23.4% OOD by +12.5% OOD).

W3. A-LoL [1] already uses a fitted reference policy for importance weighting, yet is neither discussed nor evaluated. The novelty of APRO’s particular combination needs clearer justification.

Q4. How does APRO differ from A-LoL [1]? Could the authors compare the two methods using the same backbone and offline dataset?

[1] Baheti, Ashutosh, et al. “Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models.” ICLR 2024.

A3:

Conceptual and algorithmic differences between APRO and A-LoL. Thank you for pointing out A-LoL, which we will cite and discuss in detail in Section 1 and Related Work. While both A-LoL and APRO fit a proxy/reference policy on the offline dataset to obtain a tractable denominator, they differ fundamentally in how the proxy is constructed, how importance ratios are regulated, and how positive and negative trajectories are utilized:

Design DimensionA-LoLAPRO (Ours)
Target Initialization vs. ProxyFits πref\pi_{\text{ref}} via SFT on Doff\mathcal{D}_{\text{off}} and initializes πθ←πref\pi_\theta \leftarrow \pi_{\text{ref}} (SFT init).Trains πproj\pi_{\text{proj}} on Doff\mathcal{D}_{\text{off}} separately; initializes πθ\pi_\theta from the base model to preserve exploration entropy.
Denominator & Ratio ControlUses the static reference πref\pi_{\text{ref}} and clips each token’s importance weight to [1−ϵ,1+ϵ][1-\epsilon,1+\epsilon] in the sequence variant.Adaptive Ratio Switching (ARS): Dynamically switches between πθ/πproj\pi_\theta / \pi_{\text{proj}} (when ∥rproj,t−1∥≤ϵp\|r_{\text{proj},t}-1\|\le \epsilon_p) and the local trust-region anchor πθ/πold\pi_\theta / \pi_{\text{old}}.
Handling of Negative SamplesHard indicator filtering: Discards all negative-advantage trajectories (A^<0\hat{A} < 0), losing contrastive error signals.Retains both signs: Uses group-relative advantages over both positive and negative responses, guarded by negative dual-clipping.
Signal Preservation / ScalingStandard unscaled positive advantages; suffers from signal attenuation when πθ(ot∥ct)≪πref(ot∥ct)\pi_\theta(o_t\|c_t) \ll \pi_{\text{ref}}(o_t\|c_t).Discrepancy-Aware Advantage Scaling: Amplifies under-explored positive trajectories (×Δi\times \Delta_i) and down-weights low-probability negative trajectories (×Δi−1\times \Delta_i^{-1}).

Direct multi-seed experimental comparison with A-LoL on the same backbone and offline dataset (Table R3). Following your suggestion, we implemented the sequence-level A-LoL algorithm on the exact same backbone (DeepSeek-R1-Distill-Qwen-1.5B), offline dataset Doff\mathcal{D}_{\text{off}}, reward function, and training budget, and evaluated it across 3 independent runs alongside APRO(GRPO) and the strongest ratio-based baselines (Table R3).

Table R3: Multi-seed comparison (Mean ±\pm SD over 3 independent runs) between APRO and A-LoL across all 6 ID and 2 OOD benchmarks on DeepSeek-R1-Distill-Qwen-1.5B.

MethodAIME 24AIME 25AMCMATH-500MinervaOlympiadID Avg.ARC-cMMLU-ProOOD Avg.
APRO(GRPO) (Ours)23.6±0.623.6 \pm 0.621.9±0.321.9 \pm 0.360.2±0.360.2 \pm 0.383.3±0.783.3 \pm 0.732.5±1.732.5 \pm 1.750.8±1.350.8 \pm 1.345.4±0.145.4 \pm 0.136.6±0.636.6 \pm 0.628.2±1.728.2 \pm 1.732.4±1.132.4 \pm 1.1
A-LoL24.0±0.824.0 \pm 0.819.6±1.519.6 \pm 1.559.1±0.359.1 \pm 0.382.1±2.182.1 \pm 2.127.9±3.027.9 \pm 3.048.9±0.348.9 \pm 0.343.6±0.543.6 \pm 0.524.3±1.524.3 \pm 1.525.7±0.925.7 \pm 0.925.0±0.625.0 \pm 0.6
TOPR(proxy)23.0±1.223.0 \pm 1.220.7±1.020.7 \pm 1.058.1±1.258.1 \pm 1.281.3±2.181.3 \pm 2.131.7±1.331.7 \pm 1.347.6±0.247.6 \pm 0.243.8±0.643.8 \pm 0.625.7±0.925.7 \pm 0.921.1±1.921.1 \pm 1.923.4±0.823.4 \pm 0.8
BAPO(local)21.4±0.721.4 \pm 0.718.7±0.818.7 \pm 0.858.7±1.258.7 \pm 1.281.7±1.381.7 \pm 1.333.1±2.133.1 \pm 2.148.7±0.548.7 \pm 0.543.7±0.843.7 \pm 0.815.3±1.315.3 \pm 1.323.0±0.623.0 \pm 0.619.2±0.719.2 \pm 0.7
ASPO(proxy)20.9±0.420.9 \pm 0.418.8±0.918.8 \pm 0.951.2±1.651.2 \pm 1.676.9±1.076.9 \pm 1.029.0±1.629.0 \pm 1.641.8±0.941.8 \pm 0.939.8±0.739.8 \pm 0.739.6±3.439.6 \pm 3.420.0±1.220.0 \pm 1.229.8±2.229.8 \pm 2.2
APRO(GRPO) fixed weight21.2±1.921.2 \pm 1.919.0±0.819.0 \pm 0.856.5±2.356.5 \pm 2.379.7±0.879.7 \pm 0.830.1±1.130.1 \pm 1.146.6±1.946.6 \pm 1.942.2±0.942.2 \pm 0.931.6±2.131.6 \pm 2.123.3±1.323.3 \pm 1.327.5±1.227.5 \pm 1.2

Across three runs, A-LoL averaged 43.6 ± 0.5% ID / 25.0 ± 0.6% OOD, and APRO(GRPO) averaged 45.4 ± 0.1% ID / 32.4 ± 1.1% OOD (Table R3, mean ± sample SD). The observed mean gains over A-LoL were 1.8 ID and 7.4 OOD percentage points. This confirms that simply fitting a reference policy and filtering positive advantages (as in A-LoL) is insufficient to prevent OOD degradation, and that APRO's adaptive ratio switching and discrepancy-aware scaling over both positive and negative trajectories provide substantial, statistically significant gains.


W4. The stabilization role of ratio switching is clear, but its contribution to OOD generalization is less well understood. Table 4 shows substantial sensitivity to the switching threshold. Table 3 suggests that the OOD gains depend on the interaction between switching and advantage scaling. Clarifying this interaction and the threshold-selection procedure would strengthen the paper.

Q5. How was the switching threshold selected, and was selection independent of the reported test sets? Does proximity of the proxy ratio to one correlate with smaller behavior-estimation error in the oracle setting?

A4:

Threshold sensitivity at a fixed PPO clipping width of 0.1 (Table 4).

Switching width (Table 4)ID avg.OOD avg.
0.0145.122.1
0.0545.434.8
0.10 (default)45.333.1
0.2045.224.6
0.5041.815.9

Threshold selection protocol (ϵp\epsilon_p) and strict independence from test sets. We set the default switching threshold ϵp=0.1\epsilon_p = 0.1 a priori to match the standard PPO clipping width ϵ=0.1\epsilon = 0.1 (Appendix E.2), requiring zero additional hyperparameter tuning and remaining completely independent of all reported ID and OOD test sets. Table 4 was conducted afterward as a post-hoc sensitivity study to examine how ϵp\epsilon_p affects performance across two orders of magnitude ([0.01,0.50][0.01, 0.50]). Notably, even though ϵp=0.05\epsilon_p = 0.05 yielded a higher OOD average (34.8% vs. 33.1%) and ID average (45.4% vs. 45.3%), we retained our pre-specified default ϵp=ϵ=0.1\epsilon_p = \epsilon = 0.1 throughout all main tables to ensure strict evaluation hygiene.

Reviewer_2pEHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies static, black-box off-policy post-training of weak LLMs. APRO combines an SFT-fitted behavior proxy, token-wise switching between proxy and previous-policy denominators, and discrepancy-dependent advantage scaling. Experiments cover mathematical reasoning and instruction following, with additional oracle-denominator, projector, mixed-policy, and continued-RL analyses. The method improves several offline baselines and appears useful as an initialization stage for subsequent on-policy RL.

Critical strengths
  1. The problem setting is practically valuable and clearly specified. Static external trajectories without stored behavior probabilities are a realistic resource. Evaluating this regime separately from mixed-policy training is useful.
  2. The comparison set is substantial. Table 1 includes SFT, RWR, AWR, unlikelihood training, TBRM, multiple adapted policy-gradient baselines, and joint RL+SFT objectives. Appendix H also tests continued RL; thus the paper does not rely only on a weak SFT comparison.
  3. There is evidence of complementary components. Table 3 reports 45.3/33.1 ID/OOD for the complete GRPO method, versus 42.9/25.6 for scaling with a local denominator and 41.1/22.9 for switching without scaling. Projector sensitivity and oracle-probability experiments provide additional useful diagnostics.
  4. The authors acknowledge a sensible role for the method. Fully off-policy training is presented as capability transfer before further self-exploration. Table 1 explicitly shows that on-policy PPO can outperform APRO on OOD averages. Appendix E.3 includes projector training in its reported training-stage costs.
Critical weaknesses

W1. The surrogate analysis does not establish current-policy improvement for the implemented update. The paper already calls its objective a surrogate, and Appendix C.1 explicitly distinguishes its complete-trajectory identity from the token-conditional clipped surrogate analyzed next. Those qualifications are correct and should be credited. The remaining interpretation limit is that Sections 2.4 and 3 use token-conditional ratios on externally sampled prefixes: even an exact conditional denominator corrects the action distribution at a prefix, not the prefix distribution. External group-relative advantages, switching, and clipping introduce further differences from an on-policy gradient. This is not a demonstrated theorem error or a claim that the authors promise unbiasedness. It motivates asking what empirical benefit the behavior proxy provides beyond a stable local-anchor update, and what policy-improvement interpretation, if any, is intended beyond the reported results.

W2. The conditional stability result needs an empirical applicability diagnostic. Theorem C.3 explicitly labels its guarantee conditional and assumes local-anchor ratios in [1-epsilon, 1+epsilon] through Assumption C.2. This is a valid conditional result. PPO's clipped objective does not itself ensure the assumption: a negative-advantage token with a very large ratio remains on an active branch. The unresolved question is how often the actual fallback satisfies the condition during training. Local-ratio violation frequencies and upper quantiles of active gradient coefficients would connect the theorem to the observed stability. I do not count the explicitly stated assumption as an undisclosed theoretical flaw.

W3. The discrepancy-specific benefit is not isolated from simpler sign-dependent weighting. The trajectory discrepancy averages ratios of the model's highest-probability token to the observed token. A few low-probability tokens can dominate this statistic. With the [0.1, 3] cap, the mechanism may frequently reduce to a nearly constant positive-versus-negative weighting. Table 3 isolates the presence of scaling, but not its adaptive content. A matched constant sign-weight control, with comparable gradient scale, and the fraction of multipliers hitting each cap would directly test the claimed mechanism. Likewise, extreme proxy-ratio magnitude measures disagreement, not necessarily proxy error: accurately modeled behavior can also be far from the student.

W4. Some empirical conclusions exceed the resolution of the comparisons. Several main math margins over the strongest alternatives are small, and no repeated-training uncertainty is reported. Appendix E.2 gives APRO a tenfold larger instruction-following learning rate than the RL baselines; this needs a comparable tuning account. The instruction-following table also omits the richer offline competitors from Table 1. Arena-Hard hard-prompt win rates of 0.8-1.5% provide limited evidence for the claim that the method “excels” on complex queries. These concerns do not erase the gains, but weaken broad superiority claims.

Questions for authors
  1. What precisely is the expected update represented by Eq. (6), and which sources of bias remain even with an exact behavior denominator? How should Appendix C.1 be related to it?

  2. How frequently does the fallback violate Assumption C.2, and what are the upper quantiles of the active, advantage-weighted ratios?

  3. What fraction of discrepancy weights saturate at 3 or 0.1? Does a constant sign-weight control reproduce the gains?

  4. Can you provide repeated runs for APRO and the strongest matched competitor on the main math backbone, plus a shared learning-rate comparison for instruction following?

  5. In Appendix G, are “oracle” probabilities those of the raw teacher sampler or the distribution after trajectory/prompt retention? Please state exactly what distribution is being corrected.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2pEH

We sincerely appreciate your careful review and constructive suggestions. Your feedback has helped us strengthen our experimental evaluation and improve the clarity of the manuscript. We provide point-by-point responses to your comments below.

W1. The surrogate analysis does not establish current-policy improvement for the implemented update. The paper already calls its objective a surrogate, and Appendix C.1 explicitly distinguishes its complete-trajectory identity from the token-conditional clipped surrogate analyzed next. Those qualifications are correct and should be credited. The remaining interpretation limit is that Sections 2.4 and 3 use token-conditional ratios on externally sampled prefixes: even an exact conditional denominator corrects the action distribution at a prefix, not the prefix distribution. External group-relative advantages, switching, and clipping introduce further differences from an on-policy gradient. This is not a demonstrated theorem error or a claim that the authors promise unbiasedness. It motivates asking what empirical benefit the behavior proxy provides beyond a stable local-anchor update, and what policy-improvement interpretation, if any, is intended beyond the reported results.

Q1. What precisely is the expected update represented by Eq. (6), and which sources of bias remain even with an exact behavior denominator? How should Appendix C.1 be related to it?

A1:

Table 3 (Section 5.2) isolates this comparison by keeping discrepancy-aware scaling fixed. Using the previous student as the denominator throughout gives 42.9 ID / 25.6 OOD. Allowing APRO to retain the projected behavior denominator inside the switching interval raises these scores to 45.3 / 33.1. The proxy adds 2.4 ID and 7.5 OOD points beyond local anchoring in this ablation. In Appendix F, replacing the fitted proxy with a constant or random denominator substantially degrades validation accuracy.

Eq. (6) averages reward-weighted updates on recorded responses. At a demonstrated prefix, an active term contributes toward increasing the token's log probability for positive advantage and decreasing it for negative advantage. The selected ratio and detached trajectory weight scale that contribution, and KL regularizes the aggregate update. Eq. (6) averages within each response. Training on recorded responses lets the student learn from expert actions at prefixes ct∼dμc_t \sim d_\mu before it can reliably reach those prefixes through self-exploration [1, 2]. Appendix H, Table 7 shows how the resulting checkpoint performs under subsequent on-policy training.

Appendix C.1 isolates the effect of substituting the denominator for a complete-response surrogate. With the exact behavior conditional, the token ratio reweights the action at a fixed prefix before switching and clipping. The prefixes, the continuations used to obtain rewards, and the groups used to normalize GRPO advantages still come from external data. Switching, clipping and scaling further reshape the update. We interpret Eq. (6) as a reward-guided update on demonstrated prefixes and evaluate the resulting student directly.

W2. The conditional stability result needs an empirical applicability diagnostic. ...

Q2. How frequently does the fallback violate Assumption C.2, and what are the upper quantiles of the active, advantage-weighted ratios?

A2:

Following reviewer's suggestion, we recorded fallback decisions throughout one APRO(GRPO) run on DeepSeek-R1-Distill-Qwen-1.5B. Of 1,716,845,256 valid response-token evaluations, 550,325,609 (32.05%) used fallback. 82.73% of fallback observations had a local ratio in [0.9, 1.1]. The local ratio fell outside this interval for 9.86% of positive advantage and 19.65% of negative advantage fallback observations (17.27% overall).

Table R2a. Local ratios on fallback tokens (ϵ=0.1\epsilon = 0.1).

Advantage GroupSatisfying Assumption C.2 ([0.9,1.1][0.9, 1.1])Below 0.90.9 (%)Above 1.11.1 (%)Total Outside [0.9,1.1][0.9, 1.1] (%)
Positive (A^>0\hat{A} > 0)90.14%4.53%5.33%9.86%
Negative (A^<0\hat{A} < 0)80.35%13.48%6.18%19.65%
Zero (A^=0\hat{A} = 0)81.30%12.85%5.85%18.70%
All Fallback Tokens82.73%11.34%5.93%17.27%

For the negative-advantage case in your question, ordinary PPO clipping leaves large ratios active. In verl, we use dual clipping with a threshold of 3. Above this ratio the negative-advantage term is flat. Away from branch boundaries, the absolute active coefficient multiplying the token score gradient is at most 3.3∣A∣3.3|A| for positive advantages and 3∣A∣3|A| for negative advantages. These bounds apply to the scalar policy-loss coefficient before gradient aggregation.

We measured that coefficient by differentiating the unreduced token policy loss with respect to its log probability. Table R2b reports active observations only for the quantiles and the maximum. The positive and negative pools each cover both denominator branches.

Table R2b: Upper quantiles of active advantage-weighted coefficients ∣Ct∣=∣r^tA~t∣|C_t| = |\hat{r}_t \tilde{A}_t| across training stages.

Training StageToken PoolActive Branch Rate (%)P95 | C_t |P99 | C_t |P99.9 | C_t |Max | C_t |
EarlyAll (A^≠0\hat{A} \neq 0)97.98%2.6864.0177.4248.164
Positive (A^>0\hat{A} > 0)96.74%3.5415.4247.4258.164
Negative (A^<0\hat{A} < 0)98.80%0.9551.7241.8634.694
Fallback (A^≠0\hat{A} \neq 0)92.50%2.7904.8717.4257.425
MiddleAll (A^≠0\hat{A} \neq 0)95.57%2.7804.0026.0566.789
Positive (A^>0\hat{A} > 0)98.93%3.6355.0246.1896.789
Negative (A^<0\hat{A} < 0)93.42%0.6801.1351.6822.742
Fallback (A^≠0\hat{A} \neq 0)86.08%2.6823.9856.1726.788
LateAll (A^≠0\hat{A} \neq 0)96.21%2.4263.7095.4415.984
Positive (A^>0\hat{A} > 0)98.69%3.0803.9905.4895.984
Negative (A^<0\hat{A} < 0)94.47%0.7091.1541.7552.732
Fallback (A^≠0\hat{A} \neq 0)89.09%2.1573.5415.4415.983

These statistics confirm that Assumption C.2 holds for the vast majority (82.73%82.73\%) of fallback tokens, and when local ratios do drift outside [0.9,1.1][0.9, 1.1], clipping and dual-clipping strictly cap the active gradient coefficients while preserving over 95%95\% of reward-directed updates.

W3. The discrepancy-specific benefit is not isolated from simpler sign-dependent weighting. ...

Q3. What fraction of discrepancy weights saturate at 3 or 0.1? Does a constant sign-weight control reproduce the gains?

A3:

We measured how often the final multiplier reached either cap during APRO(GRPO) training.

  • 0.24% of positive-advantage sequence evaluations hit the upper cap of 3.03.0, and 0.36% of negative-advantage sequence evaluations hit the lower cap of 0.10.1.
  • In each sign group, over 99.6% of evaluations stayed inside the caps. The mean multipliers were 1.647 (A^>0\hat A>0) and 0.513 (A^<0\hat A<0).
  • The mean negative multiplier fell from 0.580 in the first third of training to 0.505 in the middle third and 0.455 in the last, consistent with the trend in Appendix F, Figure 4(c).

Table R3a. Trajectory-level discrepancy multiplier statistics and cap saturation rates.

Training StageAdvantage SignEligible SequencesCap HitsSaturation Rate (%)Mean Multiplier wiw_i
Full TrainingA^>0\hat{A} > 0 (Cap =3.0= 3.0)121,0962920.24%1.647
Full TrainingA^<0\hat{A} < 0 (Cap =0.1= 0.1)142,4085190.36%0.513
Early Stage (First 1/3)A^>0\hat{A} > 040,3981720.43%1.679
Early Stage (First 1/3)A^<0\hat{A} < 047,418410.09%0.580
Middle Stage (Middle 1/3)A^>0\hat{A} > 040,332740.18%1.638
Middle Stage (Middle 1/3)A^<0\hat{A} < 047,5401970.41%0.505
Late Stage (Final 1/3)A^>0\hat{A} > 040,366460.11%1.624
Late Stage (Final 1/3)A^<0\hat{A} < 047,4502810.59%0.455

Following your suggestion, we evaluated a Constant Sign-Weight Control (APRO(GRPO) fixed weight) across 3 independent runs on DeepSeek-R1-Distill-Qwen-1.5B by freezing the positive and negative multipliers to the exact empirical means measured above (w+≡1.647w_+ \equiv 1.647 for A^>0\hat{A} > 0 and w−≡0.513w_- \equiv 0.513 for A^<0\hat{A} < 0), while keeping πproj\pi_{\text{proj}}, adaptive ratio switching, clipping, and all other hyperparameters identical (Table R3b).

Table R3b. Adaptive discrepancy scaling vs. matched constant sign-weight control on DeepSeek-R1-Distill-Qwen-1.5B (Mean ±\pm SD across 3 runs).

Scaling MechanismPositive Multiplier w+w_+Negative Multiplier w−w_-ID Avg. (%)OOD Avg. (%)
No Scaling (ARS only)1.0001.0001.0001.00041.122.9
Constant Sign-Weight Control (fixed weight)1.6471.647 (fixed)0.5130.513 (fixed)42.2±0.942.2 \pm 0.927.5±1.227.5 \pm 1.2
APRO (Adaptive Discrepancy Scaling)Adaptive (Δi\Delta_i, mean 1.6471.647)Adaptive (Δi−1\Delta_i^{-1}, mean 0.5130.513)45.4±0.145.4 \pm 0.132.4±1.132.4 \pm 1.1

As shown in Table R3b, while constant asymmetric sign-weighting improves over no scaling (42.2%42.2\% vs. 41.1%41.1\% ID; 27.5%27.5\% vs. 22.9%22.9\% OOD), our adaptive trajectory-specific discrepancy scaling Δi\Delta_i substantially outperforms the matched constant control across every single benchmark, yielding gains of +3.2 ID and +4.9 OOD percentage points with an observed ID sample SD of 0.1 points for adaptive scaling and 0.9 for fixed weights. Adaptive scaling gave a higher mean than the tested fixed-weight control on both metrics.

A large ∣rproj,t−1∣|r_{\text{proj}, t} - 1| can arise from either (i) proxy estimation error in low-support regions or (ii) genuine distributional disagreement between πθ\pi_\theta and an accurately modeled behavior policy μ\mu. Crucially, both sources of extreme ratios cause high-variance gradient spikes if used directly as importance weights. Adaptive Ratio Switching is designed to regulate the denominator in both cases by anchoring the step size to the local trust region rsafe,tr_{\text{safe}, t} whenever rproj,tr_{\text{proj}, t} leaves [1−ϵp,1+ϵp][1-\epsilon_p, 1+\epsilon_p], while Discrepancy-Aware Advantage Scaling Δi\Delta_i operates on the numerator signal to ensure that genuinely distant, high-reward trajectories still receive strong positive updates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.