acceptodds
Under review as a conference paper at ICLR 2027

To Aggregate or Not to Aggregate? Test-Time Aggregation Beyond Verifier-Friendly Benchmarks

Abstract

Aggregation-based test-time scaling has produced strong gains on competition mathematics and code generation, but it remains unclear whether those gains transfer beyond verifier-friendly benchmarks, under which task conditions aggregation helps, and when one-step aggregation is sufficient relative to recursive aggregation. We study these questions across structured reasoning, knowledge-intensive reasoning, and medical reasoning, spanning proof-style mathematics, expert-level STEM reasoning, social-science knowledge tasks, BrowseComp-style information seeking, and both tool-free and tool-integrated regimes. Aggregation is effective when sampled trajectories contain recoverably complementary information: complementary reasoning progress in structured reasoning, or complementary retrieved evidence in tool-integrated knowledge/evidence-seeking. Structured reasoning and tool-integrated knowledge/evidence-seeking recover 48% and 57% of available headroom on average, whereas medical reasoning without tools recovers only 21% despite a comparable diversity window. Tool use improves medical base performance, but aggregation remains weak because trajectories more often reflect competing clinical interpretations than composable intermediate progress. Within favorable regimes, aggregation type also matters: RSA is most useful for open-ended proof generation, whereas SSA captures most of the gain in tool-integrated knowledge/evidence-seeking at lower cost. Aggregation value therefore depends jointly on task structure, tool access, and aggregation type rather than following a uniform test-time scaling law across domains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time scaling, aggregation, reasoning, tool use, open-ended math, medical reasoning, information-seeking, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6Rn2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the transfer of gains beyond verifier-friendly benchmarks in aggregation-based test-time scaling, and the scenarios where single-step aggregation (SSA) is sufficient rather than recursive aggregation (RSA). The authors investigate three key questions: whether aggregation improves performance across domains beyond competition math and coding, and under what conditions it helps; and whether RSA provides advantages over SSA. The study considers three task categories, including structured reasoning, medical reasoning, and knowledge/information seeking. Results show that the value of aggregation does not follow a uniform test-time scaling law across domains, but depends jointly on task structure, tool access, and aggregation type. Specifically, aggregation is useful for structured reasoning and tool-integrated knowledge/evidence-seeking tasks and less beneficial for medical reasoning tasks. SSA is preferred for tool-integrated knowledge- and evidence-seeking tasks, whereas RSA is preferred for open-ended proof generation.

Critical strengths
  • The paper provides a thorough empirical study of aggregation in reasoning, which is a meaningful research question and an important gap in the literature. It shows that aggregation is effective in some scenarios but fails in tasks like medical reasoning. It also shows that SSA is more efficient in tool-integrated knowledge/evidence-seeking tasks, and RSA is more useful in proof-style structured reasoning.
  • The experimental studies span different tasks with multiple open-weight and proprietary LLMs.
  • The important findings include both positive and negative results. The heterogeneous behavior reported in the paper provides a meaningful foundation for future research.
Critical weaknesses
  • Some terminology may need further clarification, especially for readers with less knowledge in the area (e.g., SSA and RSA).
  • According to Appendix E.1, SSA is based on a selector prompt that picks the best trajectory judged by the LLM. Comparatively, Appendix E.2 shows that RSA combines multiple trajectories and produces results based on multiple reasoning paths. Their comparison does not clearly show that the gain of RSA is contributed by the recursion rather than combination.
  • The distinction between MedCaseReasoning and MedXpertQA results needs further clarification to isolate the effect caused by the domain/task and the benchmark's heterogeneity.
  • The effect of k on the efficiency (trade-off between utility and cost) is not discussed,
  • Formatting issues (e.g., font size in the figures makes them hard to read). Key findings in the appendix can be summarized in the main text.
Questions for authors
  • Why is the k value set to 8 in 3.2? How would the k value affect the results?
  • How would the effects of trajectory selection and aggregation be compared separately?
  • CEA is used in all main tool-integrated experiments. How does it contribute to the results?
  • Can the authors explain the reason why there is a larger improvement in MedCaseReasoning but a lower gain in MedXpertQA?
  • How sensitive are the results to aggregation hyperparameters in RSA?
  • Can the authors provide a decision rule for when SSA or RSA is recommended?
Reviewer_zxbQOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates aggregation methods of LLM parallel sampling in various domains. They claim that the benefit of aggregation methods depends on the domain and identify the important components that makes this difference between domains.

The authors evaluate majority voting (MV), single-step aggregation (SSA), where an LLM looks at all parallel results and synthesizes the final answer, and recursive self-aggregation (RSA), where SSA recursively aggregates parallel subsets. They distinguish between 3 types of domains: structured reasoning (STEM), medical reasoning, and information seeking (social sciences). They also distinguish between reasoning-only and tool-augmented reasoning (whose aggregation uses a preliminary compression of the parallel traces).

They define the recoverable headroom as the diversity window DV = pass@8 - pass@1 and find that:

  • the DV value does not say much about how much is recoverable; instead, it's the signal in the traces that depends on the domain and the tool outputs, which determines how much is recoverable. (Although knowledge has higher headroom than stem, without tools stem recovers twice as much as knowledge, but with tools this is flipped, and knowledge recovers more than stem)
  • Tool results, which provide new, diverse information to the model depending on the domain (more for info-seeking, less for stem), are responsible for this difference.
Critical strengths

The systematic comparison of aggregation methods across structured-reasoning, knowledge-intensive, and medical tasks, both with and without tool access, is a very useful empirical contribution that bridges a gap in several previous works. In particular, examining how access to external evidence changes the benefits of aggregation adds a valuable dimension beyond differences across benchmarks alone.

Critical weaknesses
  • The paper is easy to follow up to the analysis section, which gives the main claims with enough evidence in 4.2 and 4.3, but then starts repeating itself with narrow results that are most of the time statistically insignificant. The main claims and observations are already clear by 4.2/4.3, and the deep dive into each domain feels unstructured and doesn't give a clear message. It feels like the paper could fit in 5 pages and still convey the same message with the same accuracy. This reveals that the contributions are fairly limited. For a venue such as ICLR I would expect a deeper study: richer evaluation metrics than performance to validate and explain the qualitative interpretations made about the differences between domains, and a more careful study of test-time compute use (RSA is not a single point; it can scale with test-time compute, and total tokens is not a good metric to track the scaling, as cached input tokens can cost up to 100x less than output tokens, which are otherwise mixed).
  • For example, the distinction between SSA and RSA on HLE vs. IMO is not clear. The authors suggest that SSA helps more on HLE and that RSA helps more on IMO, where proofs are longer, but in Table 2, RSA and SSA always match according to the error bars. In general, the bolding in the tables is confusing, as most aggregation results intersect when accounting for error bars. Also, SSA uses less budget than RSA, and RSA can scale with more budget, making the pointwise comparison answer a ceiling question rather than an efficiency question, which is only discussed in the very last section.
  • also, the positioning relative to prior work should be clearer. RSA [1] already reports different aggregation gains across mathematical and knowledge benchmarks (Table 1), and TUMIX [2] studies refinement across benchmarks and tool configurations (Figures 3 and 6). The interesting contribution here is the systematic cross-domain comparison with and without tools; the paper should distinguish this contribution more explicitly from the already-observed dependence of aggregation gains on the benchmark.

[1] Recursive Self-Aggregation (RSA). https://arxiv.org/abs/2509.26626 [2] TUMIX: Multi-Agent Test-Time Scaling with Tool-Use Mixture. https://arxiv.org/abs/2510.01279

Questions for authors

None

Reviewer_oDy5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether aggregation based test time scaling, which samples multiple reasoning trajectories and then selects or synthesizes a final answer, works beyond competition mathematics and code generation, where correctness is easy to verify. The paper studies three aggregation methods, namely majority voting, single step LLM aggregation called SSA, and recursive self aggregation across IMO ProofBench, HLE STEM, HLE Social Science, BrowseComp, MedXpertQA, and MedCaseReasoning. The paper concludes that aggregation helps only when sampled trajectories are complementary, so structured reasoning and tool integrated evidence seeking recover about half their headroom while medical reasoning recovers far less because its trajectories encode competing interpretations rather than composable progress.

Critical strengths

The paper tests something practitioners need answered but that academic work has largely avoided because it is hard to evaluate. Aggregation research has stayed on competition math and code precisely because those domains have clean verifiers, and the question of whether the same gains hold on proofs, open ended knowledge questions, and clinical diagnosis was left open even though those are the settings where deployed systems actually run.

The second strength is the Aggregation Yield framing. Normalizing gains by available headroom is simple, but it is the right normalization, and it lets the paper make a claim that raw accuracy tables cannot, namely that two tasks with the same Pass@8 ceiling can differ enormously in how much of it is recoverable.

Critical weaknesses
  1. While the paper explains that the main reason aggregation fails in medicine because trajectories encode competing interpretations rather than complementary pieces, this "complementarity" is merely measured from three hand picked cases not quantiatively measured.

  2. The paper in general lacks novelty. While this majority voting on domains else than competition math or coding is something relatively less studies the paper also relies on well-formulated benchmarks. HLE is either MCQA or short-form answer which is almost equally easy to verify. Majority voting on proofs is not a new idea either.

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.