- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ISAR, a method for reducing RL training costs by reusing historical trajectories and recalibrating their advantages. It estimates the current policy’s expected reward using self-normalized importance weights over complete trajectories, excluding each training response from its own reward baseline. The analysis explains how mixing old and fresh responses can distort this baseline as the policy changes. Experiments on search-agent and operations research tasks show reduced generation and training time with competitive final quality. Component ablations demonstrate that recalibration recovers performance lost under uncalibrated replay.
Critical strengths
-
The replay ablation shows that recalibration recovers substantial quality at the same new-token budget, with the remaining training settings fixed. This isolates calibration’s contribution and supports its role in making trajectory reuse effective.
-
The efficiency evaluation includes rescoring and replay overhead in total training time under matched hardware allocations. The resulting savings, alongside competitive final quality against strong reuse baselines, provide credible evidence of practical training efficiency.
Critical weaknesses
-
The ablation experiments show that importance-weighted calibration outperforms an unweighted mixed-group baseline, but do not establish whether historical trajectories are needed to estimate the current policy’s reward baseline. Please keep the replay data used for policy updates unchanged and compare ISAR with a method that estimates the reward baseline using only fresh responses to the same prompt. This would clarify whether importance-weighted historical data further improve baseline estimation and training performance beyond updating the baseline with fresh samples alone.
-
Although ISAR recalibrates the reward baseline, its token-level actor ratios do not fully correct the distribution shift in historical prefixes. As trajectories age, this residual mismatch may reduce the benefits of calibration. Please evaluate performance across different policy lags and clarify whether limiting the maximum number of policy updates since trajectory generation is necessary for reliable trajectory reuse.
-
Does ISAR sometimes assign such large weights to a few historical responses that their rewards largely determine the reward baseline? If so, how accurately does this baseline estimate the current policy’s expected reward compared with an estimate based only on fresh responses to the same prompt?
Questions for authors
Please see weaknesses.
4 comments
Thank you for the additional experiments and clarifications, which substantially address my concerns. Regarding W2, I appreciate the clarification that reference calibration does not fully correct the historical-prefix distribution. I suggest evaluating performance across different policy lags to clarify how trajectory staleness affects the benefits of calibration and whether limiting replay age would improve reliability. I maintain my rating of 3.
Thank you for your constructive follow-up and for acknowledging that our earlier experiments and clarifications substantially address your concerns. We appreciate your suggestion to examine policy lag directly. We have evaluated ISAR under three replay-age policies to address W2.
1. Direct comparison of replay-age policies.
We use Qwen3.5-9B on DAPO-Math and compare the same ISAR method under Default, Age4, and Age8. Age is the number of completed AdamW updates between trajectory generation and the start of its current training update. Age4 and Age8 admit ages of at most 4 and 8, respectively. Default has no explicit age cap beyond FIFO retention. The training records show that used groups reached age 31 under Default and age 8 under Age8.
Groups are sampled uniformly from the FIFO buffer. Over-age groups are masked from the update, and the resulting vacancies are not refilled; the groups remain in the buffer until FIFO eviction. Exceeding the age limit does not itself trigger new generation.
Each setting uses a budget of 8,192 newly generated training responses, including refresh responses. The budget fixes new-response counts; the realized number of optimizer updates can vary with age screening. Each post-training row below reports one training run; the shared starting checkpoint provides the pre-RL reference. DAPO Test uses one response for each of 512 questions; AIME 2024 reports average per-response correctness over 30 questions with 32 responses per question. Truncation is measured on DAPO Test.
| ISAR replay setting | DAPO Test (%) | AIME 2024 (%) | Test truncation (%) |
|---|---|---|---|
| Shared starting checkpoint | 60.74 | 62.92 | — |
| Default | 84.38 | 77.50 | 0.98 |
| Age4 | 78.71 | 72.40 | 16.02 |
| Age8 | 86.33 | 79.48 | 6.64 |
At the same new-response budget, Age8 improves observed accuracy over Default by 1.95 percentage points on DAPO Test and 1.98 points on AIME 2024. Tightening the cap to 4 reduces accuracy on both evaluations. All three settings improve over the shared starting checkpoint. Default has the lowest truncation rate, highlighting an additional tradeoff with accuracy.
2. Implications for reliable reuse.
Thank you for prompting us to make this practical tradeoff explicit. The results directly characterize performance under different replay-age limits: Age8 improves observed accuracy over Default, while the stricter Age4 cap yields lower accuracy. Because filtered positions are not refilled, stricter screening can reduce usable training data and optimization opportunities. We interpret the comparison as the practical effect of age control, including its effect on training exposure, without attributing the accuracy differences solely to historical-prefix mismatch.
Our conclusion is that replay age is an important practical control whose effect should be assessed jointly with data utilization. This complements our earlier clarification that reward-reference calibration does not fully correct historical-prefix distributions. The results support considering an explicit age cap in this setting, while showing that tighter caps are not automatically better. We interpret them as descriptive single-run evidence, without claiming statistical significance or a universally safe cutoff.
We hope this targeted analysis addresses your remaining concern about policy lag and provides a clearer empirical basis for assessing ISAR. If these additional results further address your concerns, we would be very pleased and grateful if you could reflect this in your final evaluation score. Thank you again for your careful review; we remain available for any further questions and would welcome continued discussion.
Thank you for the further update and the additional experiments. These address my remaining concerns, and I have no further questions.
We are grateful for your close reading and specific suggestions on reference construction and off-policy reuse. We have added the fresh-reference comparison and direct diagnostics in Appendix E (Tables 8 and 9), clarifying both the benefits and limitations of historical calibration.
1. Fresh-only reference while retaining replay. Thank you for proposing this control, which separates the source of the reference from the actor's training data. We use Qwen3-8B on deduplicated, shuffled DAPO-Math, with thinking enabled, an 8k context, and three runs per configuration. Each run has 32 actor calls, batch 128, FIFO capacity 1024, warmup 512, then 256 new actor responses and 8 calls per cycle.
Fresh-ref retains the same actor/replay configuration and estimates the reference from independent current-policy responses to the same prompts. These additional responses never enter actor training or the replay buffer, and their generation is counted in cost. Only the reference rule changes; the actor objective and replay schedule remain fixed.
For reference diagnostics, both estimators share the final frozen ISAR states, actual-buffer prompts, historical candidates, and an independent evaluation batch of 128 current-policy responses per prompt (MC128). Candidate-excluded historical references and disjoint Fresh-ref samples have matched sizes N=G-1=15/31. MC128 has sampling error and is not an exact value.
Table. Fresh-reference comparison. Each entry shows ISAR vs Fresh-ref. Accuracy, tokens, and MSE are descriptive means over three runs. Accuracy and tokens come from the respective training arms; reference diagnostics use shared frozen states.
P10/P95 denote percentiles; Fresh-ref's uniform-weight concentration entries are analytical.
Fresh-ref achieves lower reference MSE and slightly higher final accuracy. ISAR obtains the reported accuracies with 75.2%/75.7% fewer generated tokens, while accuracy is lower by 0.3/0.4 percentage points. This supports using historical references to reduce additional generation; it does not imply that historical data are necessary for estimating the current reward mean or more accurate than fresh sampling. The separate Unit control also shows a 2.0-point accuracy improvement from calibration at both group sizes. We interpret these means as a quality-cost tradeoff, not proof of performance equivalence.
2. Weight concentration. We appreciate you drawing attention to a limitation that aggregate averages can obscure. The lower-tail effective sample size is approximately 1.50/1.71, and the P95 maximum normalized weight reaches 0.96/0.97. A single response can dominate the historical reference. Log-space normalization improves numerical stability but does not remove statistical concentration.
ISAR's aggregate MSE is 29.8%/41.1% below Unit's, but remains above Fresh-ref's. The MSE comparison describes average reference error; strong concentration remains a practical limitation even when this average improves. We acknowledge this explicitly in Sec. 5 and Appendix E.
3. Historical prefixes and policy lag. We thank you for highlighting the distinction between reference calibration and correction of historical prefixes. Full-trajectory weights estimate the reward reference, whereas actor token-level ratios operate conditional on recorded prefixes. Reference calibration does not fully correct their distribution, so the actor remains a practical clipped surrogate.
The results characterize the stated FIFO schedule. Reference calibration and a bounded buffer do not provide a universal guarantee for arbitrary policy lag. Appendix E makes this scope explicit and relates it to the actor objective in Sec. 3.2.
Thank you for helping us sharpen these distinctions. Please feel free to raise any further points; we remain available for follow-up questions and would welcome a more detailed exchange.