-
Conditions for the accuracy-loss predictor.
The measured accuracy loss between a reference evaluator and a target matcher can be written as:
Measured Loss=P(Reference Correct, Target Incorrect)−P(Reference Incorrect, Target Correct)
The paper predicts this loss using:
Predicted Loss=P(Reference Correct, Affected Style)
These quantities become approximately equal under two conditions. First, the reference evaluator accepts nearly all answers the target matcher accepts, making the second term in the measured loss negligible. Second, the chosen style indicator closely identifies the answers accepted by the reference evaluator but rejected by the target matcher. Under these conditions, the predicted loss should be close to the measured loss, making a fitted slope near 1 largely expected.
Figure 1 appears close to this special case. SQuAD normalization covers the answers accepted by plain exact matching, while the punctuation-related indicator closely corresponds to the main source of false negatives. In EVOUNA, the authors use human correctness labels and a verbosity-related indicator for another source of matching errors. Although human judgments and SQuAD matching are not necessarily nested, the selected indicator also produces a near-unit slope.
However, the slope need not be 1 when the style indicator does not accurately capture the actual matching errors. If it misses some false negatives, the predicted loss can underestimate the measured loss, potentially producing a slope greater than 1. Conversely, if it identifies answers that the target matcher actually accepts, the predicted loss can overestimate the measured loss, potentially producing a slope below 1. False positives from the target matcher can also reduce the measured loss. These effects may vary across datasets and matching rules.
The authors also report a slope of 0.80 on NQ-Open in the prompt-style experiment. I would be interested in a more detailed analysis of the differences in matching behavior between the evaluator pairs. For example, how frequently does the reference evaluator accept an answer that the target matcher rejects, and how often does the reverse occur? Among these disagreements, which types of matching errors are responsible? It would also be useful to examine how closely the chosen style indicators correspond to these disagreements, including how many actual false negatives they capture, how many they miss, and how often they flag answers that the target matcher would still accept. Such an analysis could help explain both the near-unit fits and the deviations observed in the prompt-style experiments.
More generally, the style indicators in Figure 1 and EVOUNA are chosen to capture the particular mismatch patterns examined in those experiments. It would be helpful to discuss how researchers could assess a style indicator's suitability when the main sources of matching errors are not readily identifiable.
-
PRM score aggregation and its relationship with trajectory length.
The PRM experiments show that PRM-weighted voting does not statistically improve over the simple inverse-length weighting baseline. I wonder whether this result could be affected by how the step-level PRM scores are aggregated into a trajectory-level score.
If the trajectory score is calculated as the product of the step-level correctness probabilities, it takes the form:
Trajectory Score=∏i=1TP(Step i Correct)
Since these probabilities are between 0 and 1, the product generally decreases as the number of steps increases, even when the individual steps have comparable correctness probabilities. This introduces a dependence on trajectory length and may favor shorter trajectories. As a result, the similarity between PRM-weighted voting and the length-based baseline could partly reflect the aggregation method rather than the information provided by the PRM itself.
Appendix M.5 discusses several alternatives for aggregating step-level PRM scores, including minimum, mean, last-step, and product-based aggregation. These methods have different relationships with trajectory length and may produce different rankings of candidate solutions. It would be helpful to clarify which aggregation method was used for the reported results and provide a more complete comparison of these alternatives.
In particular, a table showing the results of different PRM score aggregation methods under both Best-of-N and weighted majority voting, using the same candidate sets, would help determine whether the conclusions depend on the aggregation and selection methods. If feasible, including results from an outcome reward model (ORM) would also provide a useful comparison with the process-level verifier.
-
Sensitivity of observer results to answer matching.
The observer experiments construct their prediction targets using the robust matching procedure. However, the paper shows that different matching rules can change how generated answers are classified and evaluated, potentially affecting the disagreement and majority-vote error labels used in these experiments.
Appendix E reports that SQuAD-style normalization has low formatting-error canary rates across the evaluated pools. It would be helpful to clarify why the robust matcher was chosen over SQuAD-official normalization for the observer experiments. If feasible, could the authors also examine whether the predictive performance and relative rankings of prefix NLL and the learned observers remain consistent when the labels are recomputed using SQuAD-official normalization?
-
Evaluation scale and generalization to stronger solvers.
The observer experiments use 256 questions per setting, and the authors acknowledge that the limited training data may affect the learned observers. Therefore, the current results establish the advantage of prefix NLL mainly under the evaluated low-data setting. If feasible, it would be useful to examine whether this advantage persists with more training questions.
The main sampling and adaptive stopping experiments use two 8B solvers. Stronger solvers may produce answers with different formatting, diversity, and correctness distributions, which could affect matcher bias and the benefits of additional sampling or adaptive stopping. Although Figure 1 includes multiple model families and EVOUNA provides results from other systems, these analyses do not directly establish whether the main TTS findings generalize to stronger solvers. Results from at least one larger solver model would help assess this generalization.
-
Consistency of the single-sample results in Table 1.
In Table 1, the reported single-sample accuracies are 0.185 and 0.645 for the exposed and robust matchers, while the corresponding R1 values are 0.781 and 0.336. Since R1 represents the error rate at a sampling budget of 1, I would expect the single-sample accuracy to equal 1−R1 if both are computed using the same samples and estimation procedure. However, the reported values do not match.
Could the authors clarify how they calculated these quantities? For example, is single-sample accuracy averaged over all generated samples, while R1 is estimated using only the first sample from each question's pool?