- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose RATE, a framework for training critics to use at inference time. The critic is trained to decide which prefixes to allocate compute to in a particle search inference setup using a pointwise potential loss combined with a weighted cross-entropy loss over prefixes. Experiments on a set of RL environments and Qwen3 models show that RATE improves over baselines, even taking into account extra compute budget (through fixed-compute evaluations). Using RATE for SFT data generation also appears to improve performance.
Critical strengths
- The problem is interesting (how to train a critic well for allocation), and the method is well-justified. Experiments are described in thorough detail, with confidence intervals provided in most cases.
- Experimental design is strong, with fixed-compute and time-based comparisons.
Critical weaknesses
- Noisy evaluation: Based on Figure 3, RATE’s performance is within noise compared to most baselines, especially Norm-CE. Similarly, the reward curves in Figure 5 look especially noisy, with RATE oscillating quite wildly across training steps.
- It would be good to compare to baselines like Agentic Monte Carlo and contrastive twist learning, both of which are identified in the related work. As it is, the online results are a bit thin and seem fairly noisy as well (large error bars in Figure 4).
- The choice of a small lambda in all experiments might mean the proposed setup boils down to a return regression plus a relative error penalty, as explained by the authors themselves in Appendix A.7. This may make the method quite close to the Return-MSE + RankNet baseline.
Nit: BoN feels like a bit of a misleading name for the baseline: it’s uniform allocation over prefixes, not selecting the best single prefix to continue based on a critic.
edit: the authors addressed my concerns in the rebuttal phase and so I accordingly raised my score.
Questions for authors
- Do you have any results from hyperparameter sweeps, especially over lambda and gamma? How were the existing hyperparameters chosen?
3 comments
Hi, thank you for the response! I think you addressed all my concerns and so I am raising my score accordingly. I do agree with reviewer sjju that writing could be improved.
Thank you very much for your response and for taking the time to carefully consider our rebuttal. We sincerely appreciate your thoughtful feedback, constructive suggestions, and willingness to revisit your assessment. Your careful consideration and encouragement mean a great deal to us. We are truly grateful for the time and effort you have devoted to reviewing our work. Thank you again for your kindness and support throughout the review process.
Thank you for carefully reviewing our paper. We truly appreciate your time and effort, as well as your constructive, insightful, and friendly feedback. Your suggestions helped us further examine the reliability of our results. We have added online search comparisons and hyperparameter sweeps, with detailed responses below.
1. Evaluation Noise and Training Curves
Thank you for suggesting that we further examine the reliability of our results. In the matched-training fixed-frontier evaluation on ALFWorld, after selecting checkpoints by validation reward, RATE achieves a mean candidate reward of 75.70 compared to 72.64 for Return-MSE, a gain of 3.06 with a 95% bootstrap confidence interval of [1.07, 5.29].
We also checked whether this gain holds at the fixed end of training. At that point, RATE improves over Return-MSE by 2.24 [0.35, 4.34] and over Pointwise CGF by 1.96 [0.21, 3.92], with both intervals strictly above zero. This supports the benefit of adding frontier supervision in this experiment.
Regarding the training dynamics plot (Figure 6 in the paper), the curves show unsmoothed validation rewards. Because allocation involves discrete choices, small shifts in predicted scores can cause jumps in reward.
In addition, Norm-CE also includes relative frontier supervision, only replacing the sample-dependent weight with a fixed weight. Comparing it with RATE therefore tests the added contribution of the weighting mechanism, whereas comparing with Pointwise CGF directly tests the effect of frontier supervision. We will distinguish these two takeaways more clearly in the revision.
2. Comparisons with AMC and Contrastive Twist Learning
We added adapted comparisons for AMC and CTL in our environments, sharing the same actor, critic architecture, and training data. AMC-MSE directly uses the Return-MSE critic, with the core difference being that it switches to AMC incremental-weight resampling. The weighted MSE variant uses exp(return)-weighted regression. CTL trains on the continuation bank to approximate the forward marginal KL objective, pairing the same trained critic with either our allocation rule or incremental twist resampling. Thus, RATE versus Return-MSE compares training objectives, while the AMC variants compare full search configurations.
The table below reports online search mean candidate reward on TextCraft. All methods share the same search width, horizon, and compute budget cap, which includes all scoring overhead.
3. Small λ and the Difference from Return-MSE + RankNet
Your observation regarding the small-λ approximation is spot on. In this regime, RATE approximates return regression plus a relative error penalty. For frontiers consisting entirely of nonterminal prefixes, the coefficient on the relative-error variance term in the scaled loss (Appendix A.7) is 1/γ, which remains 4 at our default γ=0.25 and does not vanish as λ decreases.
The key difference is that RATE calibrates numerical return differences, reaching minimal relative penalty when the predicted differences match the target differences. RankNet instead uses pairwise ordering labels, so even when the ranking is already correct, it still tends to push the score margins further apart.
4. Hyperparameter Sweeps and Selection
We swept λ∈{0.01,0.05,0.2,1} at fixed γ=0.25, and swept γ∈{0.0625,0.25,1,4} at fixed λ=0.05. The table below lists all swept points for RATE using checkpoints selected by validation reward, reporting mean candidate reward on the fixed continuation bank (0–100 scale).
5. Naming of BoN
Thank you for pointing out the ambiguity in this naming. After consideration, we have decided to adopt your suggestion and rename this baseline Uniform allocation, and we will incorporate this change throughout the revised manuscript.
Thank you again for your time and effort!