- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper compares two test-time scaling strategies for visual reasoning: increasing the maximum response length and aggregating multiple independently sampled responses through voting. Experiments are conducted on 300 multiple-choice questions from MathVista using Qwen2.5-VL-3B and 7B. The authors compare a single 512-token response, a single 2,048-token response, and voting over four 512-token responses.
The results show that repeated sampling improves accuracy substantially, while increasing the response length ceiling provides limited gains. The paper further analyzes actual token consumption, answer parsing, sampling frequency, and prediction transitions, supplemented by archived results from larger models.
Critical strengths
None.
Critical weaknesses
1. Insufficient novelty, research depth, and scientific contribution.
- Limited novelty. The paper primarily compares two well-established strategies: Self-Consistency voting and increasing the maximum output token limit. It introduces no new algorithm, theoretical framework, or substantial mechanistic insight.
- Insufficient significance. Given the extensive literature on test-time scaling, the findings offer limited additional understanding. The research question and experimental depth are insufficient to justify a standalone paper at a top-tier AI conference.
- Inadequate academic presentation. The Introduction consists of only two brief paragraphs and fails to adequately establish the research motivation, limitations of prior work, or distinctive contributions. This overly abbreviated presentation further weakens the paper's academic rigor and completeness.
2. Inadequate experimental comparison and limited empirical validation.
- Unfair compute comparison. Although the two strategies share the same maximum output token budget, the longer-response condition generates only approximately 12–14 additional tokens, whereas voting consumes roughly four times as many output tokens. Thus, the experiment does not establish which strategy is more effective under equal actual compute. The absence of a meaningful long-reasoning baseline substantially limits the value of the central comparison.
- Unclear source of performance gains. Approximately 70% of newly correct voting outcomes originate from previously unparseable answers rather than corrections of explicitly incorrect predictions. This raises concerns about whether the reported gains reflect improved visual reasoning or simply greater answer availability under the custom parser.
- Limited generalizability. The primary evaluation covers only two closely related models and 300 multiple-choice questions from a single benchmark. The supplementary larger-model analysis relies on incomplete records and inconsistent evaluation protocols, providing little additional evidence for broader conclusions.
Questions for authors
-
Novelty and significance. Both Self-Consistency and increasing output length limits are well-established strategies. What substantial new insight does this paper provide beyond existing test-time scaling research? Why should this relatively straightforward comparison constitute a standalone research contribution?
-
Validity of the central comparison. The longer-response baseline generates only approximately 12–14 additional tokens, while voting consumes roughly four times the output tokens. Why is simply increasing
max_new_tokensconsidered a meaningful long-reasoning baseline? Without a comparison under equal actual compute using methods that genuinely extend reasoning, how can the paper support a meaningful conclusion about the relative effectiveness of these strategies? -
Interpretation of accuracy improvements. Approximately 70% of newly correct voting outcomes come from previously unparseable responses rather than corrected wrong answers. Can the authors demonstrate, using the official MathVista evaluation pipeline or carefully controlled answer-format experiments, that the gains reflect improvements in visual reasoning rather than primarily improved answer extraction?
2 comments
The additional experiments do not address my primary concern. Even as a diagnostic study, the research question is too narrow and the contributions are too limited to justify a standalone paper. The rebuttal does not change my overall assessment, and I will maintain my original score.
Thank you for stating the central concerns directly. The paper does not propose a new inference algorithm. Its contribution is a diagnostic separation of nominal budget, realized generation, answer availability, and aggregation. Our new Qwen3-VL-8B-Thinking experiment shows that this separation changes the scientific conclusion.
We evaluated a fixed 64-question confirmation set using the same image and explicit reasoning prompt for every condition. Four independently sampled paths were capped at 2048 tokens each, while one long path was capped at 8192 tokens. The long path uses materially more realized generation than one short path: its mean and 90th-percentile output lengths are 2123 and 5414 tokens, versus 1257 and 2048. It nevertheless uses far fewer output tokens than four-path voting, which averages 5028.
Rather than relying on a model-assisted judge, we evaluated every arm with the same two deterministic, reference-blind answer grammars. Under the broader grammar, which reads only explicit answers after
</think>, the long path reaches 84.4% and voting reaches 70.3%. The paired long-minus-vote difference is 14.1 percentage points [6.3, 23.4], while the long path uses 42% of voting's output tokens. Under the strict requested Final: grammar, voting is 62.5% and the long path is 56.3%, with a paired interval spanning zero. The broader grammar changes only strict-missing responses; it does not overwrite any already parsed prediction. This sensitivity demonstrates that the answer contract must be controlled before endpoint accuracy is interpreted as reasoning quality.We also separated missing-answer recovery from correction. With the broader grammar, voting versus the first path contains five missing-to-correct, one wrong-to-correct, and one correct-to-wrong transition. Sequential regeneration that stops at the first parseable answer reaches the same 70.3% accuracy as voting, at 2815 rather than 5028 output tokens. Thus answer availability accounts for the net improvement over one path in this experiment. In the original 300-question Qwen2.5-VL data, voting retains a smaller residual advantage over first-valid regeneration: 4.0 points for 3B and 1.7 points for 7B.
This is the new insight we will foreground: under a common broad answer contract, the apparent winner reverses when a reasoning-oriented model actually uses the long allowance, and the benefit of multiple calls separates into answer recovery and aggregation under measured cost. We will expand the Introduction to motivate this audit, distinguish it clearly from proposing a new method, and present the Qwen3 result alongside the original paired analyses.