acceptodds
Under review as a conference paper at ICLR 2027

ISAR: Calibrating Historical Advantages for Efficient Trajectory Reuse

Abstract

Reinforcement learning improves language models through repeated generation and policy optimization, but autoregressive rollouts make training expensive. Reusing historical trajectories reduces generation demand, yet aggressive reuse can impair learning when their reward-relative signals become poorly matched to the evolving policy. We introduce ISAR, which recalibrates advantages by estimating the current policy's prompt-specific expected reward from fresh and historical trajectories. It combines full-trajectory, self-normalized importance weighting, candidate exclusion, and a circular replay buffer with a token-level clipped actor objective. Our analysis characterizes historical advantage evolution under policy improvement and shows how mixed-group averaging attenuates the response to that improvement. In search-agent training with Qwen3.5-9B, ISAR obtains 46.0 average exact match versus 44.5 for GRPO, using 60.5% fewer new tokens and 50.2% less total training time at the same update horizon. It further reduces these costs by 18.2% and 12.3% relative to the most token-efficient and fastest strong baselines, respectively. Full-parameter DeepSeek-V4-Flash training on operations research tasks reduces generation by 54.9% and time by 38.7%, with similar aggregate quality. Component controls show that calibration recovers quality lost under uncalibrated replay, supporting current-reference estimation as a practical component of efficient reuse.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, Reinforcement Learning, Deep Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1ATEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ISAR, a method for reducing RL training costs by reusing historical trajectories and recalibrating their advantages. It estimates the current policy’s expected reward using self-normalized importance weights over complete trajectories, excluding each training response from its own reward baseline. The analysis explains how mixing old and fresh responses can distort this baseline as the policy changes. Experiments on search-agent and operations research tasks show reduced generation and training time with competitive final quality. Component ablations demonstrate that recalibration recovers performance lost under uncalibrated replay.

Critical strengths
  1. The replay ablation shows that recalibration recovers substantial quality at the same new-token budget, with the remaining training settings fixed. This isolates calibration’s contribution and supports its role in making trajectory reuse effective.

  2. The efficiency evaluation includes rescoring and replay overhead in total training time under matched hardware allocations. The resulting savings, alongside competitive final quality against strong reuse baselines, provide credible evidence of practical training efficiency.

Critical weaknesses
  1. The ablation experiments show that importance-weighted calibration outperforms an unweighted mixed-group baseline, but do not establish whether historical trajectories are needed to estimate the current policy’s reward baseline. Please keep the replay data used for policy updates unchanged and compare ISAR with a method that estimates the reward baseline using only fresh responses to the same prompt. This would clarify whether importance-weighted historical data further improve baseline estimation and training performance beyond updating the baseline with fresh samples alone.

  2. Although ISAR recalibrates the reward baseline, its token-level actor ratios do not fully correct the distribution shift in historical prefixes. As trajectories age, this residual mismatch may reduce the benefits of calibration. Please evaluate performance across different policy lags and clarify whether limiting the maximum number of policy updates since trajectory generation is necessary for reliable trajectory reuse.

  3. Does ISAR sometimes assign such large weights to a few historical responses that their rewards largely determine the reward baseline? If so, how accurately does this baseline estimate the current policy’s expected reward compared with an estimate based only on fresh responses to the same prompt?

Questions for authors

Please see weaknesses.

4 comments

We are grateful for your close reading and specific suggestions on reference construction and off-policy reuse. We have added the fresh-reference comparison and direct diagnostics in Appendix E (Tables 8 and 9), clarifying both the benefits and limitations of historical calibration.

1. Fresh-only reference while retaining replay. Thank you for proposing this control, which separates the source of the reference from the actor's training data. We use Qwen3-8B on deduplicated, shuffled DAPO-Math, with thinking enabled, an 8k context, and three runs per configuration. Each run has 32 actor calls, batch 128, FIFO capacity 1024, warmup 512, then 256 new actor responses and 8 calls per cycle.

Fresh-ref retains the same actor/replay configuration and estimates the reference from independent current-policy responses to the same prompts. These additional responses never enter actor training or the replay buffer, and their generation is counted in cost. Only the reference rule changes; the actor objective and replay schedule remain fixed.

For reference diagnostics, both estimators share the final frozen ISAR states, actual-buffer prompts, historical candidates, and an independent evaluation batch of 128 current-policy responses per prompt (MC128). Candidate-excluded historical references and disjoint Fresh-ref samples have matched sizes N=G-1=15/31. MC128 has sampling error and is not an exact value.

Table. Fresh-reference comparison. Each entry shows ISAR vs Fresh-ref. Accuracy, tokens, and MSE are descriptive means over three runs. Accuracy and tokens come from the respective training arms; reference diagnostics use shared frozen states.

MetricG=16: ISAR vs Fresh-refG=32: ISAR vs Fresh-ref
Final accuracy (%)47.7 vs 48.048.4 vs 48.8
Reference MSE vs MC1280.01802 vs 0.011920.01071 vs 0.00641
Generated training tokens (M)7.69 vs 30.987.62 vs 31.38
ESS/N, P100.100 vs 10.055 vs 1
Max. normalized weight, P950.960 vs 0.06670.970 vs 0.0323

P10/P95 denote percentiles; Fresh-ref's uniform-weight concentration entries are analytical.

Fresh-ref achieves lower reference MSE and slightly higher final accuracy. ISAR obtains the reported accuracies with 75.2%/75.7% fewer generated tokens, while accuracy is lower by 0.3/0.4 percentage points. This supports using historical references to reduce additional generation; it does not imply that historical data are necessary for estimating the current reward mean or more accurate than fresh sampling. The separate Unit control also shows a 2.0-point accuracy improvement from calibration at both group sizes. We interpret these means as a quality-cost tradeoff, not proof of performance equivalence.

2. Weight concentration. We appreciate you drawing attention to a limitation that aggregate averages can obscure. The lower-tail effective sample size is approximately 1.50/1.71, and the P95 maximum normalized weight reaches 0.96/0.97. A single response can dominate the historical reference. Log-space normalization improves numerical stability but does not remove statistical concentration.

ISAR's aggregate MSE is 29.8%/41.1% below Unit's, but remains above Fresh-ref's. The MSE comparison describes average reference error; strong concentration remains a practical limitation even when this average improves. We acknowledge this explicitly in Sec. 5 and Appendix E.

3. Historical prefixes and policy lag. We thank you for highlighting the distinction between reference calibration and correction of historical prefixes. Full-trajectory weights estimate the reward reference, whereas actor token-level ratios operate conditional on recorded prefixes. Reference calibration does not fully correct their distribution, so the actor remains a practical clipped surrogate.

The results characterize the stated FIFO schedule. Reference calibration and a bounded buffer do not provide a universal guarantee for arbitrary policy lag. Appendix E makes this scope explicit and relates it to the actor objective in Sec. 3.2.

Thank you for helping us sharpen these distinctions. Please feel free to raise any further points; we remain available for follow-up questions and would welcome a more detailed exchange.

Thank you for the additional experiments and clarifications, which substantially address my concerns. Regarding W2, I appreciate the clarification that reference calibration does not fully correct the historical-prefix distribution. I suggest evaluating performance across different policy lags to clarify how trajectory staleness affects the benefits of calibration and whether limiting replay age would improve reliability. I maintain my rating of 3.

Thank you for your constructive follow-up and for acknowledging that our earlier experiments and clarifications substantially address your concerns. We appreciate your suggestion to examine policy lag directly. We have evaluated ISAR under three replay-age policies to address W2.

1. Direct comparison of replay-age policies.

We use Qwen3.5-9B on DAPO-Math and compare the same ISAR method under Default, Age4, and Age8. Age is the number of completed AdamW updates between trajectory generation and the start of its current training update. Age4 and Age8 admit ages of at most 4 and 8, respectively. Default has no explicit age cap beyond FIFO retention. The training records show that used groups reached age 31 under Default and age 8 under Age8.

Groups are sampled uniformly from the FIFO buffer. Over-age groups are masked from the update, and the resulting vacancies are not refilled; the groups remain in the buffer until FIFO eviction. Exceeding the age limit does not itself trigger new generation.

Each setting uses a budget of 8,192 newly generated training responses, including refresh responses. The budget fixes new-response counts; the realized number of optimizer updates can vary with age screening. Each post-training row below reports one training run; the shared starting checkpoint provides the pre-RL reference. DAPO Test uses one response for each of 512 questions; AIME 2024 reports average per-response correctness over 30 questions with 32 responses per question. Truncation is measured on DAPO Test.

ISAR replay settingDAPO Test (%)AIME 2024 (%)Test truncation (%)
Shared starting checkpoint60.7462.92—
Default84.3877.500.98
Age478.7172.4016.02
Age886.3379.486.64

At the same new-response budget, Age8 improves observed accuracy over Default by 1.95 percentage points on DAPO Test and 1.98 points on AIME 2024. Tightening the cap to 4 reduces accuracy on both evaluations. All three settings improve over the shared starting checkpoint. Default has the lowest truncation rate, highlighting an additional tradeoff with accuracy.

2. Implications for reliable reuse.

Thank you for prompting us to make this practical tradeoff explicit. The results directly characterize performance under different replay-age limits: Age8 improves observed accuracy over Default, while the stricter Age4 cap yields lower accuracy. Because filtered positions are not refilled, stricter screening can reduce usable training data and optimization opportunities. We interpret the comparison as the practical effect of age control, including its effect on training exposure, without attributing the accuracy differences solely to historical-prefix mismatch.

Our conclusion is that replay age is an important practical control whose effect should be assessed jointly with data utilization. This complements our earlier clarification that reward-reference calibration does not fully correct historical-prefix distributions. The results support considering an explicit age cap in this setting, while showing that tighter caps are not automatically better. We interpret them as descriptive single-run evidence, without claiming statistical significance or a universally safe cutoff.

We hope this targeted analysis addresses your remaining concern about policy lag and provides a clearer empirical basis for assessing ISAR. If these additional results further address your concerns, we would be very pleased and grateful if you could reflect this in your final evaluation score. Thank you again for your careful review; we remain available for any further questions and would welcome continued discussion.

Thank you for the further update and the additional experiments. These address my remaining concerns, and I have no further questions.

Reviewer_B7r4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ISAR to reduce the expensive rollout cost of RL for LLM through historical trajectory reuse. The key observation is that while historical rewards remain fixed, the relative advantages change as the policy improves, making mixed-group baselines increasingly inaccurate. ISAR addresses this by using full-trajectory importance sampling to estimate the current policy's expected reward and recalibrate historical advantages, combined with circular experience replay and PPO-style optimization. Experiments on Qwen3.5-9B search-agent benchmarks show a modest 1.5-point improvement over GRPO, while reducing newly generated tokens by 60.5% and training time by 50.2%. Additional full-parameter DeepSeek-V4-Flash experiments demonstrate similar efficiency gains with comparable performance. Ablations show that recalibration substantially recovers the performance degradation caused by uncalibrated replay.

Critical strengths
  1. Clear motivation and practical significance. The paper addresses the challenge in RL for LLM: reducing expensive rollout generation through historical trajectory reuse without compromising training effectiveness.
  2. Clear theoretical analysis. The paper provides a well-motivated analysis of how historical advantages evolve as the policy improves and why mixed historical-fresh reward baselines may become increasingly inaccurate.
  3. Quite comprehensive empirical evaluation. The paper includes comparisons with several strong baselines, evaluations across different tasks and model scales, and controlled ablations demonstrating the importance of advantage recalibration.
Critical weaknesses

1.More analysis of the proposed advantage recalibration. The method estimates the current reward reference through self-normalized importance sampling, but it remains unclear how accurately this estimator approximates the current policy's expected reward in practice. More direct comparisons with fresh-rollout reward estimates would help validate the central mechanism, rather than relying primarily on downstream performance improvements. 2. Limited analysis of the performance–efficiency tradeoff. Although ISAR achieves substantial training savings while maintaining competitive performance, the evaluation primarily focuses on a fixed optimizer-update horizon. Time-to-target-quality comparisons and multi-seed evaluations would help establish whether the reported efficiency gains are robust across different training conditions.

Questions for authors
  1. The ablation in Table 3 (Qwen3.5-9B) shows a substantial improvement from replay-only (36.8 EM) to full ISAR (46.0 EM). Could the authors provide further empirical analysis explaining this large gap, particularly whether it can be attributed to improved reward-reference estimation accuracy rather than other effects of importance weighting?
3 comments

We appreciate your positive assessment and thoughtful suggestions for strengthening the connection between calibration and its empirical benefits. The supplementary comparisons and protocol are now included in Appendix E (Tables 8 and 9).

1. Calibration benefit and reference accuracy. Thank you for asking us to connect the training gain with direct estimation-error evidence. The original Table 3 reports 46.0 versus 36.8 mean EM for calibrated and unit-weight replay under matched settings. This supports the calibration component's contribution, although the endpoint alone cannot establish that the entire gain is mediated by lower reference error.

Our new experiments use Qwen3-8B on deduplicated, shuffled DAPO-Math, with thinking enabled, an 8k context, and three runs per configuration. All arms retain the same actor/replay configuration: 32 actor calls, batch 128, FIFO capacity 1024, warmup 512, then 256 new actor responses and 8 calls per cycle. Only the reward-reference rule changes; the actor loss and importance-ratio rule remain fixed.

Fresh-ref generates independent current-policy responses to the same prompts solely for reference estimation. These do not enter actor training or the buffer, and their generation is included in cost. Unit coincides with the latest stored-group mean here.

For the diagnostic, all estimators share the final frozen ISAR states, actual-buffer prompts, historical candidates, and an independent evaluation batch of 128 current-policy responses per prompt (MC128). Candidate-excluded historical pools and disjoint Fresh-ref samples have matched sizes N=G-1=15/31. MC128 is a noisy reference, not an exact value.

Table. Calibration accuracy, training benefit, and time to quality. Entries are descriptive means over three runs, listed as G=16; G=32. Accuracy and time come from the respective training arms; MSE uses shared frozen states. Time-to-45% is the cumulative training time at the first evaluated checkpoint reaching 45% accuracy.

ReferenceFinal accuracy (%)MSE vs MC128Time-to-45% (h)
Unit/recent45.7; 46.40.02565; 0.018191.036; 0.844
Fresh-ref48.0; 48.80.01192; 0.006412.031; 2.089
ISAR47.7; 48.40.01802; 0.010710.706; 0.738

Relative to Unit, ISAR lowers reference MSE by 29.8%/41.1% and improves mean final accuracy by 2.0 percentage points at both group sizes. This jointly supports reference calibration's practical value.

For a fixed historical response, Eq. (13) gives

A~i,t−Ai,t⋆=Vt(xi)−V^t,−i(xi).\widetilde A_{i,t}-A^\star_{i,t}=V_t(x_i)-\widehat V_{t,-i}(x_i).

The diagnostic therefore measures the reference contribution to advantage error. These results do not prove that lower MSE alone causes better optimization or quantitatively explain the full 9.2-point improvement on the different search-agent task. Eq. (14) remains a surrogate-gradient sensitivity bound.

2. Quality-cost tradeoff. We value your suggestion to assess efficiency at a shared quality target. ISAR reaches the 45% target 31.8%/12.5% sooner than Unit. Calibration adds overhead: its complete training time is 18.9%/21.7% higher than Unit's. We distinguish target-reaching time from fixed-horizon cost and interpret this result for the stated target.

Fresh-ref has lower MSE and slightly higher accuracy. ISAR uses 75.2%/75.7% fewer generated tokens, with accuracy lower by 0.3/0.4 points. This supports a measured quality-cost tradeoff, without establishing universal reference superiority or performance equivalence.

Your feedback has helped us make the evidence and its interpretation clearer. Thank you for your consideration; we would be glad to answer further questions and discuss these results in more depth.

We would welcome any further questions or comments you may have and are happy to provide clarification at any time. Many thanks for your time and consideration.

Thank you for addressing my concerns directly and concisely, with evidence from running additional experiments.

The new reference estimation and controlled comparisons provide useful evidence of the effectiveness of SNIS-based calibration. In particular, the reduction in reference MSE relative to the unit-weight baseline, together with improvements in downstream accuracy, helps validate the proposed recalibration mechanism.

The additional quality cost comparisons and multi-run experiments also help clarify the performance–efficiency tradeoff beyond the original fixed-update evaluation.

Overall, I consider my main concerns adequately addressed and would maintain my positive rating.

Reviewer_EKpeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ISAR, a trajectory-reuse framework for improving the efficiency of reinforcement learning for large language models. The central motivation is that historical trajectories become increasingly mismatched to the current policy as training progresses, potentially producing misleading group-relative advantages. The theoretical analysis characterizes the evolution of historical reward-relative advantages under policy improvement, derives the bias–variance behavior of mixed historical–fresh references, and provides a sensitivity bound relating reference estimation error to clipped-gradient error. Experiments on Qwen3.5-9B search-agent training report a mean exact-match score of 46.0 versus 44.5 for GRPO, with 60.5% fewer newly generated tokens and 50.2% less elapsed training time.

Critical strengths
  1. Improving the utility of previously generated trajectories is an important direction for scalable language-model reinforcement learning. The distinction between reducing fresh generation and preserving meaningful learning signals is well motivated.
  2. The Qwen3.5-9B experiments report substantial reductions in both fresh-token consumption and measured wall-clock time. Table 3 separates replay-only training from replay with calibration. The difference between 36.8 and 46.0 mean EM at an identical reported fresh-token budget is potentially important.
  3. The paper clearly separates the role of full-trajectory importance weights in estimating a reward reference from the token-level ratios used in the actor objective. It also acknowledges key issues involving support mismatch, self-normalization bias, adaptive data dependence, and the limitations of candidate exclusion. These distinctions make the formulation more careful than an approach that treats historical trajectories as interchangeable with on-policy data.
Critical weaknesses
  1. The theoretical results largely follow from standard properties of reward baselines and importance sampling. Equation (3) is a direct consequence of the advantage definition, Proposition 1 provides a basic bias–variance decomposition, and Equation (14) bounds gradient sensitivity without establishing improved optimization or convergence. Moreover, the method combines existing techniques from RLOO, SPEAR, and off-policy replay. The paper does not sufficiently demonstrate why the proposed SNIS-based calibration is theoretically or empirically preferable to simpler alternatives.
  2. The central claim is that SNIS provides a more accurate current-policy reward reference, yet the paper does not directly evaluate reference estimation error against an independent on-policy estimate. This is particularly concerning because full-trajectory importance weights can suffer from severe variance and weight degeneracy for long or stale trajectories. Furthermore, the ablation only compares SNIS calibration against unit-weight replay, without considering simpler alternatives such as fresh-only or recent-history baselines.
  3. ISAR does not outperform the strongest baselines in final accuracy, and the fixed-update comparisons do not establish better time-to-quality efficiency. Important implementation details, including replay ratios, buffer configurations, sampling settings, and hardware specifications, are also insufficiently reported.
Questions for authors
  1. The SNIS analysis assumes fixed target distributions and independent samples from known behavior policies. However, in the proposed replay framework, historical trajectories may have already influenced the current target policy through previous updates. Candidate exclusion does not eliminate this dependence. Can the authors theoretically justify the validity of the proposed estimator under such adaptive reuse, or characterize the resulting estimation bias?
  2. The paper motivates calibration by reducing the estimation error of the current-policy reward reference. However, a candidate-independent baseline cancels in the ideal policy gradient, and the expected reward is not necessarily the variance-optimal baseline. Equation (14) only bounds the difference between surrogate gradients without establishing improved optimization. Under what conditions does a more accurate current-policy reward reference actually lead to better gradient estimation or policy improvement?
  3. Can the authors directly evaluate the accuracy of the SNIS-estimated reward reference against an independent on-policy estimate? Additionally, can the authors provide multi-seed results with confidence intervals and quality-versus-wall-clock-time curves?
3 comments

Thank you for carefully examining the theoretical and empirical foundations of our work. We agree that reference accuracy, training benefit, and efficiency require separate evidence. We have added matched comparisons and direct diagnostics in Appendix E (Tables 8 and 9).

1. Scope of the analysis. We appreciate your request to clarify what the analysis establishes. Our contribution is the calibration design for trajectory reuse and its evaluation, building on established baseline and importance-sampling identities. Candidate exclusion removes direct self-inclusion but not a trajectory's earlier influence on the policy. Fixed-target, independent-sampling SNIS results do not establish adaptive unbiasedness; self-normalization also introduces finite-sample bias.

We also thank you for raising the role of the baseline. A candidate-independent baseline cancels in the ideal, exactly corrected, unclipped expected gradient, and expected reward need not be variance-optimal. Eq. (14) bounds a fixed clipped surrogate's sensitivity to weighted reference error; it does not guarantee smaller realized gradient error or policy improvement. Token-level actor ratios leave residual historical-prefix mismatch. Sec. 3.2 and App. B.5 specify the theoretical scope; Appendix E connects it to the new evidence.

2. Does calibration improve training under replay? Thank you for emphasizing the need to isolate calibration's contribution. We use Qwen3-8B on deduplicated, shuffled DAPO-Math, with thinking enabled and an 8k context. Each configuration has three runs; accuracy, cost, and MSE are reported as descriptive means. These short-horizon tests complement the search-agent experiments.

All arms retain replay: 32 actor calls, batch 128, FIFO capacity 1024, warmup 512, then 256 new actor responses and 8 calls per cycle. Only the reference rule varies. Fresh-ref generates independent current-policy reference responses for the same prompts; these do not enter actor training or the buffer, and their generation is included in cost. Unit equals the latest stored-group mean here.

Table 1. Training quality and cost. Cells list G=16; G=32. Time-to-45% is the cumulative training time at the first evaluated checkpoint reaching 45% accuracy.

ReferenceAccuracy (%)Tokens (M) / training hoursTime-to-45% (h)
Unit/recent45.7; 46.47.81/1.04; 7.68/1.071.036; 0.844
Fresh-ref48.0; 48.830.98/3.88; 31.38/4.002.031; 2.089
ISAR47.7; 48.47.69/1.23; 7.62/1.300.706; 0.738

ISAR improves mean accuracy over Unit by 2.0 percentage points at both group sizes, with similar generated-token costs. Relative to Fresh-ref, it uses 75.2%/75.7% fewer tokens, with accuracy lower by 0.3/0.4 points. Calibration adds overhead: total training time is 18.9%/21.7% higher than Unit's, while time to the shared target is 31.8%/12.5% shorter. This supports a useful quality-cost tradeoff, without claiming equivalence or faster attainment of every quality target.

3. Reference accuracy and weight concentration. We are grateful for your call for a direct accuracy check. At the final frozen ISAR checkpoints, all estimators share actual-buffer prompts and an independent current-policy evaluation batch of 128 responses per prompt (MC128). Historical references exclude the candidate and its copies; Fresh-ref uses a disjoint current-policy sample with matched size N=G-1=15/31. MC128 is a noisy evaluation reference, not an exact value.

Table 2. Reference diagnostics. Cells list G=16; G=32. P10/P95 denote percentiles; uniform-weight concentration entries follow analytically.

ReferenceMSE vs MC128ESS/N (P10)Max. normalized weight (P95)
Unit/recent0.02565; 0.018191; 10.0667; 0.0323
Fresh-ref0.01192; 0.006411; 10.0667; 0.0323
ISAR0.01802; 0.010710.100; 0.0550.960; 0.970

ISAR's MSE is 29.8%/41.1% lower than Unit's. Fresh-ref remains more accurate, at the additional generation cost in Table 1. These observations support calibration in the tested reuse setting, without establishing universal superiority or a causal guarantee linking MSE to optimization.

Your caution about concentration is well taken: lower-tail ESS is about 1.50/1.71, and the P95 maximum weight is 0.96/0.97. Strong concentration persists despite the aggregate improvements. We acknowledge this explicitly in Sec. 5 and Appendix E; these measurements do not establish robustness for arbitrarily long or stale trajectories.

Thank you again for the time and care devoted to our paper. We are available to answer any further questions and would welcome a deeper discussion of these findings.

Thank you again for your thoughtful review. If any questions or concerns remain, we would be grateful to hear them and would be happy to discuss them further.

Thank you again for your time and consideration. We would be grateful to hear whether any questions or concerns remain. If targeted additional validation would be helpful, we are ready to promptly allocate computational resources and prioritize the relevant experiments. We would welcome further discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the review discussion for ISAR

Dear Area Chair,

Thank you for considering our revised paper. The initial ratings from B7r4, 1ATE, and EKpe were 3, 3, and 2, respectively. The reviewers’ questions led us to test how accurately ISAR estimates the reward reference, how quickly it reaches a common accuracy target, and how replay age affects performance.

Reviewer B7r4 asked how improved reward-reference estimation contributes to the gain from 36.8 to 46.0 mean exact match (EM) on the search task at the same fresh-token budget (Sec. 4.4, Table 3). They also asked about time to a common quality target. In a separate Qwen3-8B study on DAPO-Math, we held the actor and replay settings fixed and varied only the reward-reference rule. Across three runs per setting, ISAR improved mean final accuracy over the uniform-weight replay baseline (Unit/recent) by 2.0 percentage points at both group sizes (16 and 32) and reached 45% accuracy sooner (Appendix E, Table 8). On 19 November, B7r4 stated that their main concerns had been adequately addressed and that they would maintain their positive rating.

Reviewer 1ATE asked about references estimated from fresh responses, concentration of importance weights, and replay age. Appendix E reports the fresh-reference comparison and weight diagnostics (Tables 8–9). In our follow-up, we compared three replay-age settings on Qwen3.5-9B with 8,192 newly generated training responses and one run per setting. An age cap of eight AdamW updates yielded the highest observed accuracy on both DAPO Test and AIME 2024. The stricter four-update cap scored below the uncapped FIFO control on both evaluations. On 17 November, 1ATE wrote, “These address my remaining concerns, and I have no further questions.”

Reviewer EKpe raised three related issues.

  • W1 / Q1–Q2. Contribution and theoretical scope. ISAR recalibrates historical advantages against an estimate of the current policy’s expected reward. Candidate exclusion removes direct self-inclusion, while adaptive dependence and finite-sample bias can persist (Sec. 3.1 and Appendix B.5). We explain when a candidate-independent baseline cancels in the ideal, exactly corrected, unclipped expected policy gradient. Equation (14) bounds the sensitivity of a fixed clipped surrogate gradient to reference error (Sec. 3.2 and Appendix B.6). Section 4.4 and Appendix E evaluate the training gains empirically.
  • W2 / Q3. Reference accuracy and weight concentration. At the same frozen ISAR checkpoints, we evaluated all estimators against an independent Monte Carlo estimate based on 128 current-policy responses per prompt. ISAR reduced reference MSE relative to Unit/recent by 29.8% and 41.1% at group sizes 16 and 32, respectively. Fresh-ref achieved lower MSE, while ISAR used about 75% fewer generated training tokens than Fresh-ref. We also quantified weight concentration. The 95th percentile of the largest normalized weight was 0.96 and 0.97 at the two group sizes (Appendix E, Tables 8–9).
  • W3 / Q3. Quality, training time, and experimental protocol. At the same update horizon on the search task, ISAR achieved 46.0 mean EM versus 44.5 for GRPO, with 60.5% fewer new tokens and 50.2% less total training time. SPEAR had the highest mean EM (46.6), and ISAR used the fewest new tokens and the least training time among the methods compared (Sec. 4.2, Table 1). Appendix E documents the replay and sampling protocol for the supplementary DAPO-Math study. Relative to Unit/recent, mean training time to the first evaluated checkpoint reaching 45% accuracy was 31.8% and 12.5% shorter at group sizes 16 and 32, respectively. Over the full 32 actor calls, mean training time was 18.9% and 21.7% longer than for Unit/recent (Table 8).

We thank the reviewers for the suggestions that led to these experiments and clarifications. We appreciate your consideration of the revised paper and the evidence presented during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.