acceptodds
Under review as a conference paper at ICLR 2027

Answer Matchers Determine What Test-Time Sampling Appears to Do

Abstract

Test-time scaling is often framed as a policy problem. The system chooses how many samples to draw, when to stop, and whether to rerank, assuming that the measured outcome reflects the policy. On open-ended factual tasks, however, the measured outcome often reflects the answer matcher instead. We use the exact decomposition to split plurality error into non-coverage and aggregation failure. A matcher false negative enters both terms with opposite -dependence, creating an apparent conflict between candidate acquisition and selection. On Llama-3.1-8B/TriviaQA, one normalization step affects of all generations. Correcting it moves single-sample accuracy , turns total error from an increase () into a decrease (), and reverses the measured gain of adaptive stopping from to points. The distortion is predictable rather than idiosyncratic. For the exposed class of matching rules, it equals the fraction of a model's correct answers in the affected style times its accuracy. This relation has unit slope over 11 cells, replicates on EVOUNA's held-out human labels, and separates contemporaneous 8B open-weight models by an order of magnitude. Plain exact match is exposed in of cells and SQuAD-family normalization in none. We release the canary that detects the failure, which standard parsing metrics miss, together with two further cases in which a cheap scalar accounts for most of an effect attributed to an expensive mechanism. We compare process reward models with trace length and learned trajectory observers with prefix likelihood.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llm, explainability, probability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_oTDtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper is mainly about how answer matching rules affects the evaluation of test timing sampling in large language models. The authors show that normalization errors can distort both non-converge and aggregation failure and reverse conclusions about the benefits of additional sampling and adaptive stopping. Thery also compare adaptive stopping, process reward models and learned error predictors against inexpensive baselines.

Critical strengths

1.Clear and controlled demonstration of how evaluation errors affext sampling conclusions.

Critical weaknesses

1.The practical impact of this paprt's findings is insufficiently demonstrated. The paper does not provide convincing evidence that the specific matching errors studied occur in widely used evaluation pipelines or materially affect conclusions in existing work. Consequently, the relevance of the findings beyond the particular manually designed matcher remains unclear.

2.The core findings offer limited new insight. Answer-matching errors and their effects on evaluation have already been well studied. The additional observation that such errors can reverse the measured benefits of adaptive sampling is a reasonably straight forward consequence of systematically misclassifying correct answers.

Questions for authors

1.Clarification of practical relevance: is there any evidence supporting the prevalence of the specific matching error (or other matching errors) studied here in evaluation used by existing test-time sampling methods?

2.About the novelty: What makes the reversal of measured sampling benefits a nontrivial finding beuond the expected consequences of misclassifying correct answers? I'll appreciate it if authors can clarify which aspect of the mechanism or empirical result provides new insights beyond existing study of answer mismatching.

Reviewer_TCPQOpenReview ↗
Rating
1
Confidence
3(−1)
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how the rules used to check whether an LLM's answer is correct can affect conclusions about test-time sampling. The authors show that, on Llama-3.1-8B with TriviaQA, an evaluation script marks many correct answers as wrong largely because they end with a period. This makes taking more samples appear to hurt accuracy, while fixing the matching rule shows that more samples reduce errors. The paper also finds that a process reward model does not performs significantly better than a simple heuristic based on reasoning-trace length when selecting among answers.

Critical strengths

The paper highlights a practical evaluation problem. Standard parsing statistics, such as unparsed and empty-extraction rates, do not detect answers that are extracted correctly but incorrectly marked as wrong. The error decomposition provides a useful way to understand how incorrect labels can distort conclusions about repeated sampling.

Critical weaknesses

The proposed proportionality law is not well supported. The initial slope drops from ~0.97 to ~0.35 after removing one influential cell (Figure 1). The slopes from controlled prompts also vary across datasets: 1.02 on TriviaQA, 0.80 on NQ-Open, and 0.69 when the results are combined (Figure 6). The EVOUNA replication also covers a narrow range of predictor values (Appendix L). These results do not support the claim of a predictable law with a coefficient close to one.

The robust matcher is not tested against human labels on the authors' own generations (Section 6), but it is used as the reference for the main comparisons (Table 1, Figure 7). The canary only checks punctuation and casing errors that the corrected matcher is designed to remove. Therefore, getting a zero error after correction does not prove that the matcher is reliable overall. The matcher also changes punctuation handling, article removal, and suffix matching at the same time, without testing their individual effects (Appendix E).

In EXP 1, the claim that a single prefix-NLL value performs better than learned observers can be due to overfitting. The learned model is trained on a 4096-dimensional representation using only 256 questions. The paper itself notes that this setup leads to near-random performance, with ~0.50 AUROC on Qwen3-8B/ARC (Figure 3). In EXP 3, the verifier experiments use only 143 to 161 prompts per setting, so a one-point difference represents roughly 1.5 questions (Appendix E).

Some results are also inconsistent. For example, the Abstract reports the corrected error as 0.354→0.2250.354 \rightarrow 0.225, while Table 1 reports 0.336→0.2250.336 \rightarrow 0.225. Sections 5 and 6 state that all robustness settings use 256 questions, but Appendix E reports only 143 to 161.


Suggestion:

The paper does not include a Results section, making it difficult to identify the main claims and the results that support them. I would also recommend improving the overall presentation of the paper.

Questions for authors
  • If the style predicate matches the strict matcher's rejection rule, what new insight does the proposed law provide beyond this basic relationship?
Reviewer_EwdDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how answer-matching rules affect the evaluation of test-time sampling. It shows that the choice of answer matcher can change whether additional sampling and adaptive stopping appear beneficial, even when evaluated on the same generated responses. The authors separate plurality error into cases where no correct answer was sampled and cases where a correct answer was sampled but not selected. They use this distinction to show how a matcher error can change the observed effect of increasing the sampling budget.

Beyond answer matching, the paper examines whether partial generations can help predict the outcomes of independently sampled answers. The authors compare learned observers with a simple prefix-NLL measure and find that the latter often performs better at predicting answer disagreement and majority error. They also evaluate adaptive stopping and PRM-weighted voting. Adaptive stopping provides modest improvements over fixed budget sampling in some settings, while PRM-weighted voting shows no clear advantage over a simple length-based voting rule.

The clearest finding is on Llama-3.1-8B/TriviaQA. Using the same saved generations, the measured gain from adaptive stopping changes from −5.50 percentage points with the exposed matcher to +1.53 percentage points with the robust matcher. The authors also find that prefix NLL often performs better than the learned observers they evaluate, while the tested PRM does not show a statistically established advantage over a length-based voting heuristic.

Critical strengths
  1. The paper examines three aspects of test-time sampling: how sampled answers are evaluated, whether partial generations can predict final outcomes, and whether adaptive computation improves performance. These analyses cover several key components of the TTS pipeline.

  2. The authors compare different answer matchers on the same saved trajectories. Holding the generations fixed helps isolate changes caused by the matching rules. Comparing different kinds of matchers also makes the analysis more informative.

  3. The experiments include useful simple baselines. Prefix NLL provides a strong point of comparison for the learned observers. The stopping experiments compare fixed-budget sampling and ESC, and PRM-weighted voting against a simple length-based rule. These controls help assess whether the more complex methods offer meaningful improvements.

Critical weaknesses
  1. Matcher errors in current TTS evaluations. The paper shows that matcher choice can reverse a TTS result, but it is unclear how often this happens in practice. In Appendix E, the formatting-error canary exceeds 1% in 15 of 33 pools under plain exact matching, but in 0 of 33 under SQuAD-style normalization. The authors reimplemented these rules rather than testing them in existing evaluation pipelines. It would be useful to know how often matcher errors change TTS method comparisons under commonly used protocols. This limits how strongly the demonstrated failure can be generalized to current TTS evaluation practices.

  2. Coverage of the matcher canary. The proposed canary detects matching errors caused by punctuation or casing differences, but it does not cover other mismatch types. The authors also examine verbosity-related errors in EVOUNA using a different predicate. However, in practical evaluations, the main sources of matching errors may not be readily apparent. The paper provides limited guidance on how researchers should assess whether the proposed canary is sufficient or determine what additional diagnostics are needed.

  3. Generalization across solver models and selection methods. The main sampling and stopping experiments use two 8B solvers. Stronger solver models may produce answers in different formats and make different types of errors, which could change how much answer matching affects the results. The verifier experiments use only one PRM and weighted voting. It would be useful to check whether these results also hold with other verifiers, such as an outcome reward model, or other selection methods, such as Best-of-N.

The writing could be polished to improve readability. The experiment numbering does not follow the order in which the experiments are presented. It would be helpful to include more complete results in the appendix, especially comparisons of different PRM score aggregation methods.

Questions for authors
  1. Conditions for the accuracy-loss predictor.

    The measured accuracy loss between a reference evaluator and a target matcher can be written as:

    Measured Loss=P(Reference Correct, Target Incorrect)−P(Reference Incorrect, Target Correct)\text{Measured Loss} = P(\text{Reference Correct, Target Incorrect}) - P(\text{Reference Incorrect, Target Correct})

    The paper predicts this loss using:

    Predicted Loss=P(Reference Correct, Affected Style)\text{Predicted Loss} = P(\text{Reference Correct, Affected Style})

    These quantities become approximately equal under two conditions. First, the reference evaluator accepts nearly all answers the target matcher accepts, making the second term in the measured loss negligible. Second, the chosen style indicator closely identifies the answers accepted by the reference evaluator but rejected by the target matcher. Under these conditions, the predicted loss should be close to the measured loss, making a fitted slope near 1 largely expected.

    Figure 1 appears close to this special case. SQuAD normalization covers the answers accepted by plain exact matching, while the punctuation-related indicator closely corresponds to the main source of false negatives. In EVOUNA, the authors use human correctness labels and a verbosity-related indicator for another source of matching errors. Although human judgments and SQuAD matching are not necessarily nested, the selected indicator also produces a near-unit slope.

    However, the slope need not be 1 when the style indicator does not accurately capture the actual matching errors. If it misses some false negatives, the predicted loss can underestimate the measured loss, potentially producing a slope greater than 1. Conversely, if it identifies answers that the target matcher actually accepts, the predicted loss can overestimate the measured loss, potentially producing a slope below 1. False positives from the target matcher can also reduce the measured loss. These effects may vary across datasets and matching rules.

    The authors also report a slope of 0.80 on NQ-Open in the prompt-style experiment. I would be interested in a more detailed analysis of the differences in matching behavior between the evaluator pairs. For example, how frequently does the reference evaluator accept an answer that the target matcher rejects, and how often does the reverse occur? Among these disagreements, which types of matching errors are responsible? It would also be useful to examine how closely the chosen style indicators correspond to these disagreements, including how many actual false negatives they capture, how many they miss, and how often they flag answers that the target matcher would still accept. Such an analysis could help explain both the near-unit fits and the deviations observed in the prompt-style experiments.

    More generally, the style indicators in Figure 1 and EVOUNA are chosen to capture the particular mismatch patterns examined in those experiments. It would be helpful to discuss how researchers could assess a style indicator's suitability when the main sources of matching errors are not readily identifiable.

  2. PRM score aggregation and its relationship with trajectory length.

    The PRM experiments show that PRM-weighted voting does not statistically improve over the simple inverse-length weighting baseline. I wonder whether this result could be affected by how the step-level PRM scores are aggregated into a trajectory-level score.

    If the trajectory score is calculated as the product of the step-level correctness probabilities, it takes the form:

    Trajectory Score=∏i=1TP(Step i Correct)\text{Trajectory Score} = \prod_{i=1}^{T} P(\text{Step }i\text{ Correct})

    Since these probabilities are between 0 and 1, the product generally decreases as the number of steps increases, even when the individual steps have comparable correctness probabilities. This introduces a dependence on trajectory length and may favor shorter trajectories. As a result, the similarity between PRM-weighted voting and the length-based baseline could partly reflect the aggregation method rather than the information provided by the PRM itself.

    Appendix M.5 discusses several alternatives for aggregating step-level PRM scores, including minimum, mean, last-step, and product-based aggregation. These methods have different relationships with trajectory length and may produce different rankings of candidate solutions. It would be helpful to clarify which aggregation method was used for the reported results and provide a more complete comparison of these alternatives.

    In particular, a table showing the results of different PRM score aggregation methods under both Best-of-N and weighted majority voting, using the same candidate sets, would help determine whether the conclusions depend on the aggregation and selection methods. If feasible, including results from an outcome reward model (ORM) would also provide a useful comparison with the process-level verifier.

  3. Sensitivity of observer results to answer matching.

    The observer experiments construct their prediction targets using the robust matching procedure. However, the paper shows that different matching rules can change how generated answers are classified and evaluated, potentially affecting the disagreement and majority-vote error labels used in these experiments.

    Appendix E reports that SQuAD-style normalization has low formatting-error canary rates across the evaluated pools. It would be helpful to clarify why the robust matcher was chosen over SQuAD-official normalization for the observer experiments. If feasible, could the authors also examine whether the predictive performance and relative rankings of prefix NLL and the learned observers remain consistent when the labels are recomputed using SQuAD-official normalization?

  4. Evaluation scale and generalization to stronger solvers.

    The observer experiments use 256 questions per setting, and the authors acknowledge that the limited training data may affect the learned observers. Therefore, the current results establish the advantage of prefix NLL mainly under the evaluated low-data setting. If feasible, it would be useful to examine whether this advantage persists with more training questions.

    The main sampling and adaptive stopping experiments use two 8B solvers. Stronger solvers may produce answers with different formatting, diversity, and correctness distributions, which could affect matcher bias and the benefits of additional sampling or adaptive stopping. Although Figure 1 includes multiple model families and EVOUNA provides results from other systems, these analyses do not directly establish whether the main TTS findings generalize to stronger solvers. Results from at least one larger solver model would help assess this generalization.

  5. Consistency of the single-sample results in Table 1.

    In Table 1, the reported single-sample accuracies are 0.185 and 0.645 for the exposed and robust matchers, while the corresponding R1R_1 values are 0.781 and 0.336. Since R1R_1 represents the error rate at a sampling budget of 1, I would expect the single-sample accuracy to equal 1−R11-R_1 if both are computed using the same samples and estimation procedure. However, the reported values do not match.

    Could the authors clarify how they calculated these quantities? For example, is single-sample accuracy averaged over all generated samples, while R1R_1 is estimated using only the first sample from each question's pool?

Reviewer_KL2QOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Exact ground truth answer matching is problematic as we need to include full stops etc. such answer matching leads to flawed scaling curves and other such adverse effects related to false negatives, example: "obama" vs "obama."

As the paper itself says, "Stripping punctuation removes this defect class entirely."

Critical strengths
  1. Table 1 cleanly shows how a buggy string normalizer can flip majority-voting and early-stopping trends on identical saved generations.
  2. Null results are reported candidly, so thanks to the authora for that
Critical weaknesses

w1: My main concern is that the central phenomenon is an avoidable regex bug already solved by standard normalization or LLM judges.

w2: The scaling law is close to an accounting identity, and the formal theory adds little. Because the defect removes credit only from correct answers carrying edge punctuation (Sec. 5, p. 5), loss equals accuracy times that punctuation rate almost by definition.

w3: novelty is highly suspect. this is not a 2026 problem for llms i think.

Questions for authors

Q1. Which widely used evaluation harnesses or published test-time scaling papers grade open-ended QA without punctuation normalization or an LLM judge?

Q2. How do LLM-as-a-judge or learned matchers behave across the pools, especially where SQuAD regex misses verbose answers (Sec. 5, p. 6)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.