- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the transfer of gains beyond verifier-friendly benchmarks in aggregation-based test-time scaling, and the scenarios where single-step aggregation (SSA) is sufficient rather than recursive aggregation (RSA). The authors investigate three key questions: whether aggregation improves performance across domains beyond competition math and coding, and under what conditions it helps; and whether RSA provides advantages over SSA. The study considers three task categories, including structured reasoning, medical reasoning, and knowledge/information seeking. Results show that the value of aggregation does not follow a uniform test-time scaling law across domains, but depends jointly on task structure, tool access, and aggregation type. Specifically, aggregation is useful for structured reasoning and tool-integrated knowledge/evidence-seeking tasks and less beneficial for medical reasoning tasks. SSA is preferred for tool-integrated knowledge- and evidence-seeking tasks, whereas RSA is preferred for open-ended proof generation.
Critical strengths
- The paper provides a thorough empirical study of aggregation in reasoning, which is a meaningful research question and an important gap in the literature. It shows that aggregation is effective in some scenarios but fails in tasks like medical reasoning. It also shows that SSA is more efficient in tool-integrated knowledge/evidence-seeking tasks, and RSA is more useful in proof-style structured reasoning.
- The experimental studies span different tasks with multiple open-weight and proprietary LLMs.
- The important findings include both positive and negative results. The heterogeneous behavior reported in the paper provides a meaningful foundation for future research.
Critical weaknesses
- Some terminology may need further clarification, especially for readers with less knowledge in the area (e.g., SSA and RSA).
- According to Appendix E.1, SSA is based on a selector prompt that picks the best trajectory judged by the LLM. Comparatively, Appendix E.2 shows that RSA combines multiple trajectories and produces results based on multiple reasoning paths. Their comparison does not clearly show that the gain of RSA is contributed by the recursion rather than combination.
- The distinction between MedCaseReasoning and MedXpertQA results needs further clarification to isolate the effect caused by the domain/task and the benchmark's heterogeneity.
- The effect of k on the efficiency (trade-off between utility and cost) is not discussed,
- Formatting issues (e.g., font size in the figures makes them hard to read). Key findings in the appendix can be summarized in the main text.
Questions for authors
- Why is the k value set to 8 in 3.2? How would the k value affect the results?
- How would the effects of trajectory selection and aggregation be compared separately?
- CEA is used in all main tool-integrated experiments. How does it contribute to the results?
- Can the authors explain the reason why there is a larger improvement in MedCaseReasoning but a lower gain in MedXpertQA?
- How sensitive are the results to aggregation hyperparameters in RSA?
- Can the authors provide a decision rule for when SSA or RSA is recommended?