acceptodds
Under review as a conference paper at ICLR 2027

Sampling Versus Longer Responses for Visual Reasoning: A Matched-Prompt MathVista Study

Abstract

Visual reasoning can receive additional inference budget through longer responses or repeated sampling, but these interventions need not have the same effect. We compare a 512-token single response, a 2,048-token single response, and voting over four 512-token responses on the same 300 multiple-choice MathVista questions, using Qwen2.5-VL-3B and 7B with matched prompts and image preprocessing. Under the fixed explicit-answer scorer, four-response voting improves accuracy by 15.3 and 6.7 percentage points, respectively; paired question-bootstrap 95% intervals are [11.3, 19.7] and [3.7, 10.0]. Increasing the single-response ceiling yields gains of 2.7 and 2.0 points with intervals spanning zero. Actual generated-token counts explain why equal ceilings are not equal compute: four-response voting uses roughly four times as many output tokens, whereas most single responses finish well below either ceiling. A broader, label-blind answer-parser sensitivity and an average-single-path baseline preserve the voting gains. Most newly correct voting outcomes arise on questions where the first response lacks a parsed final answer. A separate analysis of larger checkpoints uses saved predictions; incomplete response records prevent uniform regrading. The findings support repeated sampling for these two checkpoints and this subset; they establish neither a general model-size scaling law nor a compute-efficiency advantage at equal realized cost.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
test-time compute, visual reasoning, vision-language models, chain-of-thought, self-consistency, MathVista

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_aRADOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares two test-time scaling strategies for visual reasoning: increasing the maximum response length and aggregating multiple independently sampled responses through voting. Experiments are conducted on 300 multiple-choice questions from MathVista using Qwen2.5-VL-3B and 7B. The authors compare a single 512-token response, a single 2,048-token response, and voting over four 512-token responses.

The results show that repeated sampling improves accuracy substantially, while increasing the response length ceiling provides limited gains. The paper further analyzes actual token consumption, answer parsing, sampling frequency, and prediction transitions, supplemented by archived results from larger models.

Critical strengths

None.

Critical weaknesses

1. Insufficient novelty, research depth, and scientific contribution.

  • Limited novelty. The paper primarily compares two well-established strategies: Self-Consistency voting and increasing the maximum output token limit. It introduces no new algorithm, theoretical framework, or substantial mechanistic insight.
  • Insufficient significance. Given the extensive literature on test-time scaling, the findings offer limited additional understanding. The research question and experimental depth are insufficient to justify a standalone paper at a top-tier AI conference.
  • Inadequate academic presentation. The Introduction consists of only two brief paragraphs and fails to adequately establish the research motivation, limitations of prior work, or distinctive contributions. This overly abbreviated presentation further weakens the paper's academic rigor and completeness.

2. Inadequate experimental comparison and limited empirical validation.

  • Unfair compute comparison. Although the two strategies share the same maximum output token budget, the longer-response condition generates only approximately 12–14 additional tokens, whereas voting consumes roughly four times as many output tokens. Thus, the experiment does not establish which strategy is more effective under equal actual compute. The absence of a meaningful long-reasoning baseline substantially limits the value of the central comparison.
  • Unclear source of performance gains. Approximately 70% of newly correct voting outcomes originate from previously unparseable answers rather than corrections of explicitly incorrect predictions. This raises concerns about whether the reported gains reflect improved visual reasoning or simply greater answer availability under the custom parser.
  • Limited generalizability. The primary evaluation covers only two closely related models and 300 multiple-choice questions from a single benchmark. The supplementary larger-model analysis relies on incomplete records and inconsistent evaluation protocols, providing little additional evidence for broader conclusions.
Questions for authors
  1. Novelty and significance. Both Self-Consistency and increasing output length limits are well-established strategies. What substantial new insight does this paper provide beyond existing test-time scaling research? Why should this relatively straightforward comparison constitute a standalone research contribution?

  2. Validity of the central comparison. The longer-response baseline generates only approximately 12–14 additional tokens, while voting consumes roughly four times the output tokens. Why is simply increasing max_new_tokens considered a meaningful long-reasoning baseline? Without a comparison under equal actual compute using methods that genuinely extend reasoning, how can the paper support a meaningful conclusion about the relative effectiveness of these strategies?

  3. Interpretation of accuracy improvements. Approximately 70% of newly correct voting outcomes come from previously unparseable responses rather than corrected wrong answers. Can the authors demonstrate, using the official MathVista evaluation pipeline or carefully controlled answer-format experiments, that the gains reflect improvements in visual reasoning rather than primarily improved answer extraction?

2 comments

Thank you for stating the central concerns directly. The paper does not propose a new inference algorithm. Its contribution is a diagnostic separation of nominal budget, realized generation, answer availability, and aggregation. Our new Qwen3-VL-8B-Thinking experiment shows that this separation changes the scientific conclusion.

We evaluated a fixed 64-question confirmation set using the same image and explicit reasoning prompt for every condition. Four independently sampled paths were capped at 2048 tokens each, while one long path was capped at 8192 tokens. The long path uses materially more realized generation than one short path: its mean and 90th-percentile output lengths are 2123 and 5414 tokens, versus 1257 and 2048. It nevertheless uses far fewer output tokens than four-path voting, which averages 5028.

Rather than relying on a model-assisted judge, we evaluated every arm with the same two deterministic, reference-blind answer grammars. Under the broader grammar, which reads only explicit answers after </think>, the long path reaches 84.4% and voting reaches 70.3%. The paired long-minus-vote difference is 14.1 percentage points [6.3, 23.4], while the long path uses 42% of voting's output tokens. Under the strict requested Final: grammar, voting is 62.5% and the long path is 56.3%, with a paired interval spanning zero. The broader grammar changes only strict-missing responses; it does not overwrite any already parsed prediction. This sensitivity demonstrates that the answer contract must be controlled before endpoint accuracy is interpreted as reasoning quality.

We also separated missing-answer recovery from correction. With the broader grammar, voting versus the first path contains five missing-to-correct, one wrong-to-correct, and one correct-to-wrong transition. Sequential regeneration that stops at the first parseable answer reaches the same 70.3% accuracy as voting, at 2815 rather than 5028 output tokens. Thus answer availability accounts for the net improvement over one path in this experiment. In the original 300-question Qwen2.5-VL data, voting retains a smaller residual advantage over first-valid regeneration: 4.0 points for 3B and 1.7 points for 7B.

This is the new insight we will foreground: under a common broad answer contract, the apparent winner reverses when a reasoning-oriented model actually uses the long allowance, and the benefit of multiple calls separates into answer recovery and aggregation under measured cost. We will expand the Introduction to motivate this audit, distinguish it clearly from proposing a new method, and present the Qwen3 result alongside the original paired analyses.

The additional experiments do not address my primary concern. Even as a diagnostic study, the research question is too narrow and the contributions are too limited to justify a standalone paper. The rebuttal does not change my overall assessment, and I will maintain my original score.

Reviewer_YAE1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the effectiveness of different test-time compute strategies for multimodal visual mathematical reasoning. Specifically, the authors compare three settings: a single response with a maximum length of 512 tokens, a single response with a maximum length of 2,048 tokens, and voting over four independently sampled responses, each limited to 512 tokens. The experiments are conducted on a subset of 300 multiple-choice questions from MathVista using Qwen2.5-VL-3B and Qwen2.5-VL-7B.

The results show that four-response voting substantially improves the accuracy of both models, yielding gains of 15.3 and 6.7 percentage points, respectively, while simply increasing the maximum output length provides only marginal improvements. The authors further analyze actual token consumption, answer parsing failures, different sampling baselines, voting strategies, and performance across question categories. Additionally, the paper includes supplementary analyses based on previously saved predictions from larger models.

Overall, this paper presents a relatively detailed empirical study. However, its main contributions lie in the evaluation and diagnostic analysis of existing inference strategies rather than the development of new methods.

Critical strengths
  • The paper adopts consistent prompts and image preprocessing settings, and validates its main findings through paired statistical analyses and sensitivity experiments, making the results reasonably reliable.

  • Beyond comparing accuracy, the paper also analyzes actual token consumption and answer parsing failures, providing useful insights into the sources of performance gains from repeated sampling.

Critical weaknesses
  • The main experiments are limited to Qwen2.5-VL-3B/7B, while the supplementary experiments rely on relatively old models such as InternVL2.5 and QVQ. More recent models with stronger reasoning capabilities, such as Qwen3-VL [1] and InternVL3.5 [2], are already available. The lack of evaluation on these newer models significantly limits the relevance and generalizability of the findings.

  • MathVista is a relatively old benchmark and may not adequately reflect the challenges faced by state-of-the-art multimodal models. Moreover, the main experiments cover only 300 multiple-choice questions, resulting in limited evaluation scale and task diversity. I suggest including more challenging benchmarks, such as MathVision [3], MMMU-Pro [4], and VisuLogic [5], to provide a more comprehensive evaluation.

  • Increasing the maximum token limit for a single response does not substantially increase the actual generation length, whereas repeated sampling consumes approximately four times as many output tokens. Therefore, the current experiments do not convincingly demonstrate that repeated sampling is more effective than genuinely longer reasoning. I suggest conducting comparisons under matched actual computational budgets.

  • The paper finds that a substantial portion of the voting gains comes from recovering unparseable answers. However, it does not compare repeated sampling against simpler strategies, such as regeneration or answer repair. Furthermore, the paper lacks an analysis of prediction diversity and error correlations across different sampled responses, making it difficult to fully understand the sources of the observed voting improvements.

[1] Bai et al. Qwen3-VL Technical Report. arXiv, 2025.

[2] Wang et al. InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency. arXiv, 2025.

[3] Wang et al. Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset. arXiv, 2024.

[4] Yue et al. MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark. ACL, 2025.

[5] Xu et al. VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models. arXiv, 2025.

Questions for authors

same with weaknesses.

2 comments

Thank you for identifying the model-recency, compute, regeneration, and diversity gaps. We ran a new controlled experiment with Qwen3-VL-8B-Thinking that directly addresses these points.

On a fixed 64-question MathVista confirmation set, we compare four independently sampled 2048-token reasoning paths with one 8192-token reasoning path. Every condition uses the same image and prompt, ending with: “Reason through the problem and end with Final: followed by one option letter.” Unlike the submitted Qwen2.5-VL ceiling comparison, the reasoning checkpoint uses the additional allowance: the long path averages 2123 output tokens with a 5414-token 90th percentile, versus 1257 and 2048 for a short path.

Under a label-blind answer grammar applied uniformly after </think>, the long path reaches 84.4% accuracy and four-path voting reaches 70.3%. The paired long-minus-vote difference is 14.1 percentage points, with a question-bootstrap 95% interval of [6.3, 23.4]. The long path uses 2123 output tokens per question, compared with 5028 for voting; including prompt tokens, the totals are 2362 and 5985. Thus, under this common answer rule, the long path is both more accurate on the confirmation set and less costly by the recorded token measures. The strict requested Final: grammar gives 56.3% and 62.5%, respectively, with an interval spanning zero. We will report both: the broader grammar changes only strict-missing responses and never reads the reference answer.

We also added the requested simpler baseline. Sequential regeneration that stops at the first parseable answer reaches the same 70.3% as four-path voting while using 2815 output tokens on average. Between first-valid regeneration and voting, one incorrect answer is corrected and one correct answer is lost, giving zero net gain. In the original 300-question Qwen2.5-VL results, voting retains 4.0 points over first-valid regeneration for 3B and 1.7 points for 7B. Answer availability therefore explains most of the single-path-to-vote improvement, while the remaining aggregation effect differs by checkpoint.

The four Qwen3 short paths have pairwise error correlations of 0.52–0.71 and answer-disagreement rates of 15.6%–25.0% when missing answers are included. Their errors are substantially correlated, which explains the limited marginal value of additional paths.

We retained the same fixed MathVista questions so that the changed model and inference strategy, rather than a changed task, drive this confirmation comparison. This directly addresses the reasoning-model question but not cross-benchmark breadth; the revised claim will remain MathVista-specific. The main conclusion will be that response ceilings, genuine long reasoning, regeneration, and voting can rank differently and must be compared using realized cost and a common answer contract.

Thanks for the additional experiments. I still find the evaluation too narrow to draw a useful conclusion about test-time compute for visual reasoning. The new comparison covers only 64 MathVista questions, and the ranking changes with the answer parser. I’d like to see whether the result holds on a more challenging benchmark such as MathVision or VisuLogic.

Reviewer_7SUpOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates repeated sampling vs increasing the token limit for visual math reasoning. The main experiments evaluate Qwen2.5-VL-3B and 7B on 300 MathVista questions.

The results show that four-response voting improves accuracy whereas increasing the token ceiling yields only small gains.

The main contribution is an empirical study of repeated-sampling gains and their dependence on answer availability and evaluation protocols.

Critical strengths
  1. good empirical analysis The paper provides a controlled comparison. I appreciate the paired statistical analysis and the sensitivity checks involving answer parsing, single-response baselines, and tie-breaking rules.

  2. Useful investigation of where the improvements come from.

The analysis of realized token consumption and missing final answers is informative. In particular, the finding that a substantial fraction of voting improvements comes from recovering previously unparseable answers provides useful insight into how repeated sampling affects evaluation.

Critical weaknesses
  1. Insufficient evaluation

My main concern is that the experimental setup is somewhat outdated given the availability of newer reasoning models, such as Qwen3-VL-Thinking. The main experiments only evaluate Qwen2.5-VL-Instruct models, which may not adequately represent modern multimodal reasoning capabilities. Simply increasing the maximum output tokens from 512 to 2,048 does not necessarily encourage longer or more meaningful reasoning, especially without explicit COT prompting. Indeed, the actual output lengths barely increase in the experiments. As a result, the paper does not provide a convincing comparison between repeated sampling and genuinely extended reasoning, which is central to its research question.

  1. Limited evidence that improvements reflect better reasoning and generalize to other models.

A substantial portion of the voting improvements comes from recovering missing or unparseable answers. For example, in the 3B model, 33 of the 47 newly correct voting outcomes come from previously missing answers. This raises questions about whether the improvements primarily reflect better reasoning or simply more reliable answer extraction.

Questions for authors
  1. Clarification on the longer-response baseline.

Could the authors provide the exact prompts used for the short- and long-response conditions? In particular, were the models explicitly instructed to reason step by step in the longer-response condition? If the only intervention was increasing max_new_tokens, what motivates treating this as a meaningful comparison against extended reasoning?

  1. Evidence from reasoning-oriented models and comparable computation budgets.

Would the main conclusions hold for dedicated multimodal reasoning models, such as Qwen3-VL-8B-Thinking, that are designed to generate extended reasoning traces?

  1. Separating reasoning improvements from answer-format recovery.

Can the authors further quantify the voting gains after controlling for missing or unparseable final answers, for example by applying a consistent answer-recovery procedure to both single-response and voting conditions? I appreciate the existing parser sensitivity analysis, but would like to understand how much of the improvement remains attributable to correcting incorrect answers rather than recovering missing ones.

1 comment

Thank you for the careful reading and for recognizing the value of the paired analysis. Your questions led us to run a new confirmation experiment with Qwen3-VL-8B-Thinking and to separate answer availability from majority aggregation.

The exact suffix used for both Qwen2.5-VL conditions was: “Reason through the problem and end with Final: followed by one option letter.” The image, question, letter-coded choices, and prompt were identical; the only difference was max_new_tokens=512 versus 2048. Thus the model was asked to reason, but the long condition received no additional instruction that would make it use the larger allowance. We will call this a response-ceiling comparison rather than extended reasoning.

We then evaluated Qwen3-VL-8B-Thinking on a fixed 64-question confirmation set, using the same suffix and image processing in every arm. We compared four independently sampled paths capped at 2048 tokens with one path capped at 8192 tokens. Here the larger allowance changed realized generation: the long path averaged 2123 output tokens and had a 90th percentile of 5414, compared with 1257 and 2048 for a short path.

The new result changes the substantive conclusion. Under the strict requested Final: grammar, voting scores 62.5% and the long path 56.3%; the paired vote-minus-long interval is [-4.7, 17.2] points. Under a broader label-blind grammar applied uniformly after </think>, accepting explicit forms such as boxed answers and “Final Answer,” the long path scores 84.4% and voting 70.3%. The paired long-minus-vote difference is 14.1 points [6.3, 23.4]. This grammar never reads the reference label and changes only responses that were missing under the strict grammar; it does not replace an already parsed answer. The long path uses 2123 output tokens per question, versus 5028 for voting.

We also quantified answer recovery separately from correction. With the broader grammar, voting versus the first path contains five missing-to-correct, one wrong-to-correct, and one correct-to-wrong transition. Sequential regeneration that stops at the first parseable answer reaches the same 70.3% as voting, using 2815 rather than 5028 output tokens on average. Relative to this policy, voting makes one wrong-to-correct and one correct-to-wrong change, for zero net gain. On the original 300-question Qwen2.5-VL data, first-valid regeneration likewise captures most of the improvement; voting's remaining advantage is 4.0 points for 3B and 1.7 points for 7B.

We will revise the paper around this sharper finding: repeated sampling helps the two instruct checkpoints, but a reasoning-oriented checkpoint that actually uses the long allowance produces a different ordering. The relevant comparison is determined by realized generation, the answer contract, and whether multiple calls add answer availability or useful aggregation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.