- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies epistemic uncertainty in PRM-guided test-time search, arguing that PRMs score reasoning traces outside their training distribution unreliably. The authors present empirical evidence that MC Dropout score variance correlates with PRM miscalibration, and give a stylized analysis contrasting linear degradation under uncertainty-agnostic selection with sublinear degradation under UCB-style selection. Building on this, they propose UATS, which estimates PRM uncertainty via MC Dropout. A heuristic variant (H-UATS) uses a variance threshold and an optimism margin to selectively re-evaluate uncertain candidates before score-proportional expansion. An adaptive variant (A-UATS) replaces the fixed hyperparameters with an MLP controller trained via behavior cloning and REINFORCE. Experiments on MATH-500 and AIME24 across 15 policy–PRM pairs, plus a HumanEval extension, show consistent improvements over Best-of-N, Beam Search, DVTS, REBASE, and DoRA.
Critical strengths
-
The evaluation covers 15 policy–PRM pairs, seven candidate budgets, and two math benchmarks, with five runs each, plus a code-generation extension. The improvements over the baselines are consistent in direction across all reported pairs, and the per-pair results are fully reported in the appendix.
-
H-UATS is training-free, requires no change to the PRM, and can be added to existing score-proportional search procedures. The appendix includes sensitivity analyses for the main hyperparameters (, , dropout rate, , controller size).
Critical weaknesses
-
Proposition 4.1 follows almost directly from its assumptions. Given (ii) a loss of at least per selection error, and (iii) an error probability of at least under OOD and zero otherwise, the linear bound is obtained by summing over steps.
-
In Proposition 4.2, the confidence term depends only on and , so it is identical for all candidates in . Adding it therefore does not change the ranking, and the selection rule is equivalent to taking the argmax of the posterior mean. The sublinear bound thus reflects concentration from averaging unbiased samples, not the use of uncertainty in decision making.
-
The analysis does not cover the variance threshold , selective re-evaluation, the optimism margin , or the RL controller. App. D.5 acknowledges this for the controller, and App. D.7 notes that systematic PRM bias is likewise outside its scope. In addition, the link between method and analysis (Sec. 3.3, Prop. 4.2) relies on treating MC Dropout passes as samples from . Here dropout is enabled at inference and its rate is tuned per verifier (0.07 vs. 0.10, App. G.6). Under these conditions, this interpretation needs stronger justification. For instance: were these PRMs trained with dropout, and does the dropout mean remain a faithful estimate of the original PRM score?
-
Figure 1 and App. H (case 2) describe UATS as down-weighting high-score, high-uncertainty steps and favoring reliable low-variance nodes. Sec. 5.2, however, contains no explicit uncertainty penalty. Uncertain candidates are retained via an optimistic score, given additional evaluations, and then expanded according to their mean score. A high-score, high-uncertainty step is therefore corrected only if additional sampling lowers its mean, which is not guaranteed.
Questions for authors
1. Compute accounting.
Sec. 6.1 states that methods are compared under the same candidate budget . App. E.1 instead converts generation into PRM passes using a fixed ratio (571/32 ≈ 17.8) and states that baselines spend the saved budget on additional candidates. App. F.3, in turn, describes FLOPs-based matching.
- Which accounting underlies Tables 1, 4 and 5?
- Is the 17.8 ratio applied uniformly across policies (0.5B–8B) and PRMs (1.5B–7B)? Was PRM latency measured with batched or sequential passes?
- In the "w/o Uncert." ablation, how is the budget freed by removing MC Dropout reallocated?
2. Hyperparameter selection protocol.
App. E.1 states that MATH-500 is used only for evaluation. Yet App. G sweeps , , the dropout rate and on MATH, and Table 6 selects the controller architecture on MATH-500. The best entry in Table 6 (86.54) coincides with the corresponding entry in Table 4.
- Was a held-out split used for these choices, and how were the baselines' hyperparameters chosen?
- Eq. 7 defines , but Eq. 8 compares with , whereas the values in Fig. 1 and App. G.2–G.3 are on the variance scale. Does threshold the standard deviation or the variance?
3. Controller details.
- Is a single controller, trained on four policy–PRM pairs from the MATH training split, used for all 15 pairs, for AIME24, and for HumanEval?
- Are the state features normalized across PRMs with different score and variance scales?
- How are the action ranges constrained? In particular, can and lead to spending beyond the matched budget?
- Please specify the action distribution, the REINFORCE baseline/advantage, and the BC objective.
- The cost term changes the reward by at most . Is there evidence, e.g., from existing training logs, that it measurably affects the learned policy?