- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper targets the critic in asynchronous PPO-style RL for LLMs: because the behaviour policy µ lags the target π, GAE regression converges to V^µ rather than V^π, and the paper argues that V-trace cannot provide both terminal-reward coverage and controlled weights. BRACE caps the importance-ratio product at k tokens and gives the remaining residuals a constant weight so they telescope into the behaviour-policy Monte-Carlo return. On four tasks against five actor-side baselines, BRACE is best on 13 of 16 metrics, with BrowseComp-Plus mean@1 rising from 0.245 (AReaL) to 0.269 (+9.8% relative).
Critical strengths
A simple, well-specified critic-side target with the right diagnostics. The target (Eq. 26) is one line on top of PPO and is evaluated in a production-style async pipeline across four tasks and five actor-side baselines. The paper also proposes measurable diagnostics and shows BRACE moving them in the predicted direction.
Critical weaknesses
The headline gains are not statistically supported, and the BrowseComp-Plus protocol is undisclosed. No seeds or variance are reported, so every Table 1 entry appears to be a single run. The BrowseComp-Plus number is mean@1 of one greedy sample.
The "does not transfer" claim is argued, not tested, and the closest prior constructions are neither cited nor ablated. The natural control, bootstrapped k-step V-trace (Eq. 8 with n = k), is absent, so the first stated contribution and the necessity of the Monte-Carlo tail are unsupported.
Questions for authors
- What is the BrowseComp-Plus training prompt set, does it overlap the evaluation questions, and is Fig. 3b's "BrowseComp-Plus accuracy" a held-out evaluation or the per-step training-batch score?
- How many seeds per Table 1 entry, and can you report run-to-run variance or evaluation CIs where margins over the best baseline are 0.001–0.024?
- Add the bootstrapped k-step V-trace control (Eq. 8 with n=k) to Fig. 4c, and PPO and the strongest actor-side baseline at S=15–50 to Fig. 4d.
2 comments
We provide additional multi-seed baseline comparisons on BrowseComp-Plus.
Our previous supplementary statistics covered only BRACE and did not fully characterize run-to-run variation in the baselines. To address this concern more directly, we have now added multi-seed statistics for PPO, PPO-EWMA, AReaL, KPop, and IcePop under the asynchronous training setup of Table 1. Each method is evaluated across four independent training seeds, and BRACE uses the four-seed statistics reported previously.
Table R3. Multi-seed comparisons on BrowseComp-Plus, measured by mean@1. "Original" refers to the single-run scores in Table 1. Sample variances use the denominator . The two-sided 95% Student-t CIs for the means assume approximately normal scores from independent training runs, with 3 degrees of freedom. CIs are approximate, computed from the rounded means and sample variances.
| Method | Original | Training seeds | Mean | Sample SD | Sample variance | 95% CI for mean |
|---|---|---|---|---|---|---|
| PPO | 0.206 | 4 | 0.207 | 0.023 | 0.0005 | [0.171, 0.243] |
| PPO-EWMA | 0.199 | 4 | 0.200 | 0.019 | 0.0003 | [0.172, 0.228] |
| AReaL | 0.245 | 4 | 0.237 | 0.029 | 0.0008 | [0.192, 0.282] |
| KPop | 0.183 | 4 | 0.185 | 0.025 | 0.0006 | [0.146, 0.224] |
| IcePop | 0.231 | 4 | 0.228 | 0.022 | 0.0005 | [0.192, 0.264] |
| BRACE | 0.269 | 4 | 0.267 | 0.022 | 0.0005 | [0.231, 0.303] |
In this four-seed comparison, BRACE achieves the highest mean@1 among all methods, at 0.267. AReaL remains the strongest baseline by mean score, at 0.237. Based on the four-seed means, BRACE's relative improvement is approximately 12.7%, numerically higher than the 9.8% in the original single-seed comparison. The mean-score advantage therefore remains when the comparison is expanded to multiple seeds.
BRACE also has a lower sample standard deviation than AReaL, at 0.022 versus 0.029. In this set of independent training runs, BRACE thus achieves both a higher mean score and lower observed run-to-run variability. Thank you for encouraging us to complete the baseline experiments and present a fuller comparison of the methods.
We would be grateful for your feedback on the added experimental controls.
Thank you again for the time you have devoted to carefully reviewing our paper. Your suggestion to include bootstrapped k-step V-trace was particularly helpful: it made us recognize that the original ablation did not fully answer the question about the tail design. Your comments gave us a concrete direction for the additional experiments, and we have added the following controls in our response:
The updated figures and first-80-step view are available in our anonymous supplement. We also clarify the non-overlapping BrowseComp-Plus split of 747 training and 83 evaluation questions, confirm that Fig. 3b reports held-out accuracy, and provide four-seed BRACE statistics on BrowseComp-Plus and Search-R1 in Table R1.
If you have time to revisit our response, we would greatly appreciate your perspective on whether these additions address your concerns about the missing direct comparisons. If we have misunderstood any part of your comments or left an important point unanswered, we would welcome the opportunity to clarify it and improve the revision accordingly. Thank you again for helping us identify these gaps and for the time you have devoted to the discussion.