acceptodds
Under review as a conference paper at ICLR 2027

ButtonsBench: A Controllable Visual Reasoning Benchmark with Co-Origin Verification

Abstract

Visual puzzles offer an ideal testbed for visual reasoning due to their minimal knowledge priors and essential reliance on visual information . However, current evaluation practice on such puzzles exhibits two fundamental flaws: endpoint-only scoring inflates performance by masking unfaithful shortcuts behind illusory competence, while intuitive, non-parametric difficulty grading obscures models' concrete collapse thresholds and combinatorial limits. We present four contributions toward reliable evaluation: (1) ButtonsBench-1, a diagnostic benchmark spanning 9 visual puzzle tasks parameterized by explicit difficulty knobs (“buttons”) with discrete levels, transforming difficulty from a subjective tier into a measurable, programmatically grounded property; (2) Co-Origin Probing Methodology, an evaluation protocol that pairs base queries with decoupled intermediate behavioral checkpoints under majority consensus, formalizing verified joint accuracy (), falsification rate (), and structural crash points (); (3) ButtonsBench Platform, a modular three-stage system for on-demand programmatic generation of tailored visual materials, base questions, probes and final benchmark with ground-truth metadata to eliminate benchmark saturation and contamination; and (4) an audit of four frontier MLLMs across instances. We find that endpoint accuracy severely inflates perceived reasoning: probe auditing reveals widespread falsification rates ( to ), overturning the apparent performance ranking by elevating qwen-3.8-max () over gpt-5.5 (). Furthermore, mapping multi-factor crash frontiers reveals a stark decoupling: while frontier models display high endurance along isolated 1D sweeps ( on up to 11 slices), concurrent multi-factor scaling triggers catastrophic compositional collapse (). The data of ButtonsBench-1 benchmark and the source code of ButtonsBench platform will be fully open-sourced upon publication.

Primary area:
datasets and benchmarks
Keywords:
Visual Reasoning, Benchmark, MLLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3X7jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses two problems in visual puzzle benchmarks for multimodal large language models, entangled variables with uncontrolled difficulty and illusory competence from endpoint-only scoring. The authors introduce the ButtonsBench platform and ButtonsBench-1, a benchmark of 691 basic questions and 903 paired co-origin probes across 9 visual puzzle tasks parameterized by task-specific buttons with discrete levels. They also propose a Co-Origin Probing Methodology that uses majority consensus over 5 trials to measure baseline endpoint accuracy, verified joint accuracy, falsification rate, and structural crash points. Across four MLLMs, macro falsification rate ranges from 26.1% to 72.4% and verified joint accuracy reverses the baseline endpoint accuracy ranking between gpt-5.5 and qwen-3.8-max.

Critical strengths
  • The three-stage ButtonsBench platform logs intermediate state metadata during material generation and generates paired co-origin probes on the same material instance to test different parts of reasoning.
  • Across four MLLMs and 1,594 question instances, macro falsification rate FR reaches 26.1% to 72.4%, and verified joint accuracy Acc_joint reverses the baseline endpoint accuracy Acc_base ranking as qwen-3.8-max drops from 57.4% to 46.1% while gpt-5.5 drops from 60.1% to 45.1%.
Critical weaknesses
  • Several probing questions seem harder than their base questions because they require a more comprehensive answer, and the paper gives no per-probe results. Probes in mirroring_patterns and pattern_completion require identifying cell color mismatches or conflicting hole IDs across every distractor, and they are scored with binary correct or wrong with no partial credit.
  • The Single-Button Crash Point metric CP_single does not really isolate individual factors because it anchors all other buttons at L1 and evaluates only one question instance per level. For two-level buttons such as rotation_enabled and overlap_level, L1 corresponds to the hardest setting, which forces rotation across every 1D sweep in pattern_completion where all four models got CP_single = 0 despite reaching up to 11.1% joint accuracy.
Questions for authors
  • What is the model accuracy for each probe category?
  • How does CP_single change in pattern_completion and count_multiple_patterns if two-level buttons like rotation_enabled and overlap_level are fixed at L0 instead of L1, and how much does CP_single vary across random seeds?
7 comments
AuthorsOpenReview ↗

Reply to Question 1 (model accuracy for each probe)

We greatly appreciate the reviewer's question regarding "What is the model accuracy for each probe category?" Here, we supplement both the detailed accuracy statistics of each model on the base question and all individual co-origin probes across every task, as well as the aggregated accuracy results of each model across each probe category (all of which will also be added to the Appendix of the revised manuscript). The results are presented below:

  1. Model Accuracy Across Each Probe |Task Name|Model Name|Acc_base (%)|Acc_joint (%)|FR (%)|probe1 Acc_maj (%)|probe2 Acc_maj (%)|probe3 Acc_maj (%)|Micro Acc_maj Instance-level (%)|Macro Acc_maj Probe-level(%)| |-|-|-:|-:|-:|-:|-:|-:|-:|-:| |rolling_dice|qwen-3.8-max|69.39|46.94|32.35|59.18|N/A|N/A|59.18|59.18| |rolling_dice|gpt-5.5|73.47|40.82|44.44|44.9|N/A|N/A|44.9|44.9| |rolling_dice|gemini-3.7-flash|67.35|24.49|63.64|34.69|N/A|N/A|34.69|34.69| |rolling_dice|MiniMax-M3|12.24|2.04|83.33|2.04|N/A|N/A|2.04|2.04| |connect_lines|qwen-3.8-max|62.86|62.86|0|65.71|N/A|N/A|65.71|65.71| |connect_lines|gpt-5.5|68.57|65.71|4.17|68.57|N/A|N/A|68.57|68.57| |connect_lines|gemini-3.7-flash|51.43|48.57|5.56|54.29|N/A|N/A|54.29|54.29| |connect_lines|MiniMax-M3|37.14|28.57|23.08|28.57|N/A|N/A|28.57|28.57| |count_multiple_patterns|qwen-3.8-max|98.75|96.25|2.53|96.88|N/A|N/A|96.88|96.88| |count_multiple_patterns|gpt-5.5|94.38|88.75|5.96|91.25|N/A|N/A|91.25|91.25| |count_multiple_patterns|gemini-3.7-flash|61.25|36.88|39.8|45.62|N/A|N/A|45.62|45.62| |count_multiple_patterns|MiniMax-M3|10|3.12|68.75|11.25|N/A|N/A|11.25|11.25| |cubes_count|qwen-3.8-max|48.41|30.95|36.07|38.1|31.75|N/A|34.92|34.92| |cubes_count|gpt-5.5|26.19|16.67|36.36|23.81|22.22|N/A|23.02|23.02| |cubes_count|gemini-3.7-flash|36.51|8.73|76.09|11.9|11.11|N/A|11.51|11.51| |cubes_count|MiniMax-M3|11.9|0.79|93.33|1.59|0.79|N/A|1.19|1.19| |find_different|qwen-3.8-max|90|88|2.22|94|92|N/A|93|93| |find_different|gpt-5.5|100|100|0|100|100|N/A|100|100| |find_different|gemini-3.7-flash|42|38|9.52|56|42|N/A|49|49| |find_different|MiniMax-M3|40|34|15|56|38|N/A|47|47| |gear_rotation|qwen-3.8-max|83.33|55.56|33.33|83.33|66.67|66.67|72.22|72.22| |gear_rotation|gpt-5.5|94.44|61.11|35.29|77.78|61.11|61.11|66.67|66.67| |gear_rotation|gemini-3.7-flash|66.67|16.67|75|66.67|33.33|33.33|44.44|44.44| |gear_rotation|MiniMax-M3|50|0|100|33.33|5.56|5.56|14.81|14.81| |mirroring_patterns|qwen-3.8-max|24.44|11.11|54.55|11.11|N/A|N/A|11.11|11.11| |mirroring_patterns|gpt-5.5|35.56|11.11|68.75|15.56|N/A|N/A|15.56|15.56| |mirroring_patterns|gemini-3.7-flash|24.44|2.22|90.91|2.22|N/A|N/A|2.22|2.22| |mirroring_patterns|MiniMax-M3|13.33|0|100|0|N/A|N/A|0|0| |pattern_completion|qwen-3.8-max|24.31|11.11|54.29|12.5|N/A|N/A|12.5|12.5| |pattern_completion|gpt-5.5|34.03|7.64|77.55|12.5|N/A|N/A|12.5|12.5| |pattern_completion|gemini-3.7-flash|17.36|5.56|68|7.64|N/A|N/A|7.64|7.64| |pattern_completion|MiniMax-M3|15.28|0.69|95.45|3.47|N/A|N/A|3.47|3.47| |trailer_cubes_count|qwen-3.8-max|15.62|12.5|20|14.06|N/A|N/A|14.06|14.06| |trailer_cubes_count|gpt-5.5|14.06|14.06|0|20.31|N/A|N/A|20.31|20.31| |trailer_cubes_count|gemini-3.7-flash|9.38|1.56|83.33|1.56|N/A|N/A|1.56|1.56| |trailer_cubes_count|MiniMax-M3|0|0|--|0|N/A|N/A|0|0|

  2. Model Accuracy Across Each Probe Category Following our probe taxonomy in Appendix A.4.5, the specific category of each probe is:

  • Format Variants: connect_lines (Probe 1)
  • Local Condition Perturbations: count_multiple_patterns (Probe 1), gear_rotation (Probe 1)
  • Intermediate State Probes: All remaining probes across the tasks
Probe Categoryqwen-3.8-maxgpt-5.5gemini-3.7-flashMiniMax-M3
Format Variants65.7168.5754.2928.57
Local Condition Perturbations90.1084.5156.1522.2
Intermediate State Probes48.6046.1523.3811.30
AuthorsOpenReview ↗

Reply to Questions 2 (Buttons Fixed at L0 instead of L1)

We thank the reviewer for this comment. Regarding the reviewer's question "How does CP_single change in pattern_completion and count_multiple_patterns if two-level buttons like rotation_enabled and overlap_level are fixed at L0 instead of L1", we supplement the following statistical results:

CP_single (Acc_joint) when two-level buttons are anchored at L1 vs. L0 (L1→L0)

Task / Buttonqwen-3.8-maxgpt-5.5gemini-3.7-flshMiniMax-M3
Count Patterns
Rotation (L0-1)>1→>1>1→>1>1→>10→0
Occlusion (L0-1)>1→>1>1→>10→>10→0
Quantity (L0-4)>4→>4>4→>40→>40→0
Categories (L0-7)>7→>73→>70→>70→0
Pattern Comp.
Rotation (L0-1)0→00→00→00→0
Grid Size (L0-5)0→10→00→00→0
Quantity (L0-3)0→10→00→10→0
Hole Size (L0-2)0→00→00→00→0

From the results above, we can see that:

  1. On the count_multiple_patterns task, fixing the two-level buttons (Rotation and Occlusion) at L0 (disabling rotation and occlusion) substantially increases CP_single on the remaining buttons for gemini-3.7-flash and gpt-5.5, while qwen-3.8-max and MiniMax-M3 remain unchanged: When fixed at L1 (rotation and occlusion enabled), gemini-3.7-flash has a CP_single of 0 across the Occlusion, Quantity, and Categories buttons, and gpt-5.5 has a CP_single of 3 on Categories. When fixed at L0 (no rotation or occlusion), gemini-3.7-flash passes the highest levels across all buttons (increasing to >1, >4, and >7), and gpt-5.5 also improves to >7 on Categories; meanwhile, qwen-3.8-max (consistently never reaching a crash point, >L_max) and MiniMax-M3 (consistently 0) remain unchanged. This indicates that in this task, enabling rotation and occlusion (L1) indeed introduces substantial extra visual interference for gemini-3.7-flash and gpt-5.5; fixing the two-level buttons at L0 eliminates this interference, thereby reflecting the single-dimension capacity of these two models regarding Quantity and Categories under non-rotated and unoccluded conditions.
  2. On pattern_completion, changing the two-level button (Rotation) from L1 to L0 (disabling rotation) results in almost no change in CP_single across the models: After fixing Rotation at L0, CP_single for gpt-5.5 and MiniMax-M3 remains 0 across all buttons; qwen-3.8-max (on Grid Size and Quantity) and gemini-3.7-flash (on Quantity) only slightly change from 0 to 1 (i.e., answering correctly only at the L0 level and failing consecutively starting from L1). This indicates that the widespread CP_single=0 across models on pattern_completion is not caused by fixing Rotation at its maximum level L1, but rather because the visual complexity introduced by the current starting level parameters of the remaining buttons in this task (such as the grid size, missing component count, and hole size corresponding to L0/L1) is still too high for these four models. Leveraging the parameterized generation capability of the ButtonsBench platform, in the next version of the benchmark we can easily map the lower levels of these buttons to parameter values with lower visual complexity (e.g., smaller grid sizes), thereby characterizing the capability boundaries of models in the lower-difficulty regime of this task at a finer granularity.
AuthorsOpenReview ↗

Reply to Questions 2 (CP_single Variance Across Random Seeds)

Regarding the reviewer's question on "how much does CP_single vary across random seeds," we generated three additional random seeds for both the count_multiple_patterns and pattern_completion tasks, and evaluated all four models (qwen-3.8-max, gpt-5.5, gemini-3.7-flash, and MiniMax-M3) across these new seeds. Table 1 and Table 2 below show the CP_single results across the original seed and the 3 new seeds under the default setting (remaining buttons fixed at L1) and when the two-level buttons are fixed at L0, respectively (formatted as Orig/S1/S2/S3 (Δ), where Δ denotes the maximum range across seeds):

Table 1: CP_single across original & 3 new seeds (Orig/S1/S2/S3, Δ=max-min), all L1

Task / Buttonqwen-3.8-maxgpt-5.5gemini-3.7-flashMiniMax-M3
count_multiple_patterns
Rotation (L0-1)>1/>1/>1/>1 (0)>1/>1/>1/>1 (0)>1/>1/>1/>1 (0)0/>1/>1/>1 (2)
Occlusion (L0-1)>1/>1/>1/>1 (0)>1/>1/>1/>1 (0)0/>1/>1/>1 (2)0/>1/>1/>1 (2)
Quantity (L0-4)>4/>4/>4/>4 (0)>4/>4/>4/>4 (0)0/3/3/3 (3)0/>4/>4/>4 (5)
Categories (L0-7)>7/>7/>7/>7 (0)3/2/2/2 (1)0/3/>7/3 (8)0/>7/>7/>7 (8)
pattern_completion
Rotation (L0-1)0/0/0/0 (0)0/0/0/0 (0)0/0/0/0 (0)0/0/0/0 (0)
Grid Size (L0-5)0/0/0/0 (0)0/0/0/0 (0)0/0/0/0 (0)0/0/0/0 (0)
Quantity (L0-3)0/0/0/0 (0)0/0/0/0 (0)0/0/0/0 (0)0/0/0/0 (0)
Hole Size (L0-2)0/0/0/0 (0)0/0/0/0 (0)0/0/0/0 (0)0/0/0/0 (0)

Table 2: CP_single across original & 3 new seeds (Orig/S1/S2/S3, Δ=max-min) two-level buttons pinned L0

Task / Buttonqwen-3.8-maxgpt-5.5gemini-3.7-flashMiniMax-M3
count_multiple_patterns
Rotation (L0-1)>1/>1/>1/>1 (0)>1/>1/>1/>1 (0)>1/>1/>1/>1 (0)0/>1/>1/>1 (2)
Occlusion (L0-1)>1/>1/>1/>1 (0)>1/>1/>1/>1 (0)0/>1/>1/>1 (2)0/>1/>1/>1 (2)
Quantity (L0-4)>4/>4/>4/>4 (0)>4/>4/>4/>4 (0)0/3/3/3 (3)0/>4/>4/>4 (5)
Categories (L0-7)>7/>7/>7/>7 (0)3/2/2/2 (1)0/3/>7/3 (8)0/>7/>7/>7 (8)
pattern_completion
Rotation (L0-1)0/0/0/0 (0)0/0/0/0 (0)0/0/0/0 (0)0/0/0/0 (0)
Grid Size (L0-5)0/0/0/0 (0)0/0/0/0 (0)0/0/0/0 (0)0/0/0/0 (0)
Quantity (L0-3)0/0/0/0 (0)0/0/0/0 (0)0/0/0/0 (0)0/0/0/0 (0)
Hole Size (L0-2)0/0/0/0 (0)0/0/0/0 (0)0/0/0/0 (0)0/0/0/0 (0)

From the results, we can see that:

  1. CPsingle remains highly stable across random seeds, and the results are almost completely consistent across the three newly generated random seeds (S1/S2/S3):
    • In Table 2 (two-level buttons fixed at L0): Out of all 32 (Model × Button) cells, 27 cells are completely identical across all 4 seeds (Δ=0).
    • In Table 1 (fixed at L1): Out of all 32 cells, 24 cells are completely identical across all 4 seeds (Δ=0).
  2. Larger cross-seed variations (Δ≥2) only appear in boundary models (gemini-3.7-flash and MiniMax-M3) under single-instance slices, and comparing Table 1 and Table 2 reveals the source of this variation:
    • On the count_multiple_patterns task, when fixed at L1 (Rotation and Occlusion enabled, Table 1), gemini-3.7-flash scores 0 on the original seed Orig (because it happened to fail on the two single-question instances at L0 and L1), whereas on the three new seeds S1/S2/S3 it stably achieves >1 (Occlusion) and 3/3/3 (Quantity).
    • However, once the two-level buttons are fixed at L0 (rotation and occlusion disabled, Table 2), gemini-3.7-flash's cross-seed variation completely disappears, achieving 100% consistency across all 4 seeds (>1/>1/>1/>1, >4/>4/>4/>4, and >7/>7/>7/>7, Δ=0). This indicates that gemini-3.7-flash's cross-seed variation in Table 1 is likely because enabling "Rotation + Occlusion (L1)" places the task difficulty right at gemini-3.7-flash's capability boundary (where success or failure is more sensitive to the specific occlusion positions of patterns in a single random seed); whereas under no occlusion/rotation (L0) or extremely high difficulty (pattern_completion), its performance is completely stable across seeds.

This brings us valuable new insights, and in future versions of the benchmark, we will generate question instances across multiple random seeds for each button-level combination.

Reviewer_jkMqOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper propose use code to generate some difficulty-controllabe visual puzzles, together with them mid-check questions. Some analysis show that lots of correct answer based on incorrect mid-inference.

Critical strengths

The puzzles are generated by code. When models improve, you turn up the difficulty and generate fresh puzzles. Each question comes with probe questions about the MID checkt.

Critical weaknesses
  • As a benchmark paper, the work only test four models, it will be better add more models, including both opensource and closesource.

  • The novelty need additional claim... for example some word puzzle can also check the mid-step correctness. The mid incorrectness with correct outcome may show the self-correction of a model?

  • need a small validate for gemini as judge. Also wondering whether a small model as judge is enough,.

Questions for authors

see weakness

5 comments
AuthorsOpenReview ↗

Reply to Critical Weakness 1 (Adding More Models)

We fully agree with the reviewer's constructive suggestion that "it will be better add more models, including both opensource and closesource." Accordingly, based on the existing evaluations of 2 closed-source models (gemini-3.7-flash, gpt-5.5) and 2 open-source models (qwen-3.8-max, MiniMax-M3), we conducted supplementary evaluations on 2 additional closed-source models (claude-fable-5, glm-5v-turbo) and 1 additional open-source model (MiMo-V2.6-Pro). The comprehensive results across all 7 models are presented below:

Metric Format: A_base↑ / A_joint↑ / FR↓

Task Namegemini-3.7-flashgpt-5.5glm-5v-turboclaude-fable-5qwen-3.8-maxMiniMax-M3MiMo-V2.6-Pro
Rolling Dice67.3/24.5/63.673.5/40.8/44.414.3/2.0/85.736.7/0.0/100.069.4/46.9/32.412.2/2.0/83.314.3/0.0/100.0
Connect Lines51.4/48.6/5.668.6/65.7/4.234.3/25.7/25.048.6/45.7/5.962.9/62.9/0.037.1/28.6/23.137.1/28.6/23.1
Count Patterns61.2/36.9/39.894.4/88.8/6.031.2/17.5/44.085.0/77.5/8.898.8/96.2/2.510.0/3.1/68.822.5/10.6/52.8
Cube Count36.5/8.7/76.126.2/16.7/36.419.8/5.6/72.030.2/15.9/47.448.4/30.9/36.111.9/0.8/93.36.3/0.0/100.0
Find Different42.0/38.0/9.5100.0/100.0/0.038.0/32.0/15.886.0/78.0/9.390.0/88.0/2.240.0/34.0/15.044.0/40.0/9.1
Gear Rotation66.7/16.7/75.094.4/61.1/35.338.9/0.0/100.061.1/44.4/27.383.3/55.6/33.350.0/0.0/100.061.1/0.0/100.0
Mirroring Patterns24.4/2.2/90.935.6/11.1/68.815.6/0.0/100.024.4/2.2/90.924.4/11.1/54.513.3/0.0/100.015.6/0.0/100.0
Pattern Completion17.4/5.6/68.034.0/7.6/77.520.8/2.1/90.027.8/4.9/82.524.3/11.1/54.315.3/0.7/95.54.2/0.0/100.0
Trailer Cubes9.4/1.6/83.314.1/14.1/0.03.1/0.0/100.012.5/7.8/37.515.6/12.5/20.00.0/0.0/---1.6/0.0/100.0
Overall Macro-Avg.41.7/20.3/56.960.1/45.1/30.324.0/9.4/70.345.8/30.7/45.557.4/46.1/26.121.1/7.7/72.423.0/8.8/76.1

From the evaluation results across the 4 closed-source models (gemini-3.7-flash, gpt-5.5, claude-fable-5, glm-5v-turbo) and 3 open-source models (qwen-3.8-max, MiniMax-M3, MiMo-V2.6-Pro), we can draw the following key conclusions, which further corroborate and reinforce the core findings of our paper:

  1. The phenomenon of "Illusory Endpoint Competence" is pervasive across diverse open-source and closed-source models: The three newly added models likewise exhibit remarkably high falsification rates (with FRs of 45.5% for claude-fable-5, 70.3% for glm-5v-turbo, and 76.1% for MiMo-V2.6-Pro). For example, on the binary-choice task Gear Rotation, MiMo-V2.6-Pro and glm-5v-turbo achieve baseline accuracies (Acc_base) of 61.1% and 38.9%, respectively; however, under co-origin probe verification, their joint accuracies (Acc_joint) both drop to 0.0% (FR = 100.0%). Similarly, on Rolling Dice, claude-fable-5 drops from Acc_base = 36.7% to Acc_joint = 0.0% (FR = 100.0%). This demonstrates that even when models arrive at the correct final answer, they often fail to completely and consistently master the indispensable intermediate steps along the reasoning chain; relying solely on final-answer evaluation significantly overestimates models' genuine visual reasoning capabilities.
  2. Co-origin probes enable sharper model capability stratification, and the top open-source model maintains its lead: The 7 models exhibit a clear tiered distribution. Even with the inclusion of additional closed-source models, the open-source flagship qwen-3.8-max continues to rank first among all 7 models in both probe-verified joint accuracy (Acc_joint = 46.1%) and the lowest falsification rate (FR = 26.1%), maintaining a slight edge over the closed-source flagship gpt-5.5 (45.1%). Meanwhile, probe verification effectively differentiates models with similar surface-level scores: for instance, while claude-fable-5 and gemini-3.7-flash differ by only 4.1% in Acc_base (45.8% vs. 41.7%), claude-fable-5—by virtue of its higher reasoning fidelity—establishes a substantial lead of over 10% in Acc_joint (30.7% vs. 20.3%).

We will include more comprehensive evaluations covering a wider range of open-source and closed-source models in future iterations of our benchmark.

AuthorsOpenReview ↗

Reply to Critical Weakness 3

We thank the reviewer for raising the questions regarding "need a small validate for gemini as judge. Also wondering whether a small model as judge is enough." Our responses are as follows:

  • A lighter model like Qwen3.5-35B-A3B as judge is also capable: In our early experiments, we locally deployed a relatively lightweight open-source model, Qwen3.5-35B-A3B, for answer grading, and manual spot-checks confirmed that its judging accuracy reached 100%. (When officially running the complete benchmark evaluation, because our local GPU compute resources were heavily constrained at the time, we switched to API calls to avoid occupying local compute and to accelerate high-concurrency evaluation—selecting gemini-3.5-flash, which offered the most stable API service and the most abundant quota resources at that time.)

  • Manual spot-check validation of gemini-3.5-flash's judging capability on our evaluation results:

    • Grading Pipeline Explanation: Our grading workflow during evaluation operates as follows: after extracting the final answer and performing format normalization (e.g., removing whitespace, standardizing punctuation, and unifying letter case), the answer is first graded directly by programmatic regex matching rules. Only if the programmatic grading marks the answer as Incorrect is it passed to the LLM-as-a-Judge for secondary review.

    • Spot-Check Methodology and Results: For the base question and each probe question of every task, and across each evaluated model, we sampled up to 30 grading instances judged as Correct and up to 30 instances judged as Incorrect by the LLM-as-a-Judge (gemini-3.5-flash) after secondary review for manual inspection. Across all sampled instances, gemini-3.5-flash achieved a 100% judging accuracy. We found that gemini-3.5-flash accurately identifies inconsequential formatting deviations not covered by our regex matching rules and provides correct judgment results for answers that were initially misjudged as incorrect by regex, for example:

      taskquestion idbox_answeranswerLLM-as-judge response
      connect_linesbase1{-}O,2{-}Q,3{-}G,4{-}C,5{-}J1-O,2-Q,3-G,4-C,5-JCorrect
      rolling_diceprobe 1(0, 1, 5, 3, 6, 2, 4), (1, 3, 5, 6, 4, 2, 1), (2, 6, 5, 4, 1, 2, 3), (3, 2, 6, 4, 5, 1, 3), (4, 3, 6, 2, 4, 1, 5), (5, 5, 6, 3, 2, 1, 4)(step0, 1, 5, 3, 6, 2, 4), (step1, 3, 5, 6, 4, 2, 1), (step2, 6, 5, 4, 1, 2, 3), (step3, 2, 6, 4, 5, 1, 3), (step4, 3, 6, 2, 4, 1, 5), (step5, 5, 6, 3, 2, 1, 4)Correct
      cubes_countbase*6*6Correct
AuthorsOpenReview ↗

Reply to Critical Weakness 2

We thank the reviewer for raising the interesting perspective that "the mid incorrectness with correct outcome may show the self-correction of a model." We would like to emphasize that, as described in our paper (Lines 371–375), the base question and the probe question are evaluated as completely independent single-turn API requests. Therefore, when answering the base question, the model has no awareness of the probe's existence; likewise, when answering the probe question, it is completely isolated from the base question and other probe questions. There is no contextual connection between the two.

Meanwhile, to investigate whether models possess self-correction capabilities, we conducted a supplementary "Erroneous Intermediate State Injection" experiment on the gear_rotation task:

1. Overall Experimental Design:

The gear_rotation task includes two intermediate-state probes:

  • Probe 2 (Connection Relation Probe): Identifying whether the connection relation between adjacent gears is mesh, parallel, or cross.
  • Probe 3 (Rotation Trajectory Probe): Inferring the clockwise or counterclockwise rotation direction of each gear from the first to the last gear.

We independently construct two supplementary test sets targeting Probe 2 and Probe 3 to evaluate how the model's accuracy on the base question changes after injecting its own erroneous intermediate states.

2. Construction of Supplementary Experimental Data:

For each model, we filter samples and construct the test data for Probe 2 and Probe 3 via the following two steps:

  • Step 1 (Filtering instances with "Base Correct, Probe Incorrect"): We select question instances where the model answered the base question correctly (majority-voting accuracy = 1) but answered the corresponding probe question incorrectly (majority-voting accuracy = 0).

  • Step 2 (Selecting erroneous probe responses that lead to the opposite final conclusion): Among the above question instances, we select an erroneous probe response generated by the model that logically entails the exact opposite final gear rotation direction to the ground-truth answer of the base question (i.e., for Probe 2, based on the erroneous sequence of connection relations identified by the model, the deduced rotation direction of the terminal gear is opposite to the ground truth; for Probe 3, the direction of the last gear in the model's output rotation trajectory sequence is opposite to the ground truth).

  • Based on the filtered samples, we construct the new test items as follows:

    • We concatenate the erroneous response with the original questions using the following 2 templates to obtain the question field value:

    • Template A:

      question = (
          f"问题1:{probe_q}\n\n"
          f"对于问题1,你之前的回答是:\\boxed{{box_answer}}。\n\n"
          f"请基于以上你对问题1的回答,继续回答问题2:{base_q}"
      )
      # English translation for new question template
      # question = (
      #     f"Question 1: {probe_q}\n\n"
      #     f"Your previous answer to Question 1 was: \\boxed{{{box_answer}}}}.\n\n"
      #     "Based on your answer to Question 1 above, please proceed to answer Question 2: {base_q}"
      # )
      
    • Template B:

      question = (
          f"问题1:{probe_q}\n\n"
          f"对于问题1,你之前的回答是:\\boxed{{box_answer}}。\n\n"
          f"现在请回答问题2:{base_q}"
      )
      # English translation for new question template
      # question = (
      #     f"Question 1: {probe_q}\n\n"
      #     f"Your previous answer to Question 1 was: \\boxed{{{box_answer}}}}.\n\n"
      #     "Now please answer Question 2: {base_q}"
      # )
      
    • The input image uses the image from the base question.

    • The answer field value uses the answer field value from the base question.

Reviewer_wofGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors proposed ButtonsBench, a framework for reliable evaluation of multimodal large language models on visual puzzles, targeting two flaws in current practice: endpoint-only scoring that hides unfaithful shortcuts behind illusory competence, and subjective difficulty grading that obscures models’ collapse thresholds. The contributions include ButtonsBench-1, a benchmark of 9 puzzle tasks parameterized by explicit difficulty “buttons” with discrete levels; Results show endpoint accuracy severely inflates perceived reasoning, with probe auditing revealing widespread falsification that overturns rankings, and that models robust on isolated 1D difficulty sweeps collapse catastrophically under concurrent multi-factor scaling.

Critical strengths
  • The paper setup is clear and easy to understand. The gear rotation direction is an easy example to see what kind of questions the model struggles with. The probe question generation pipeline provides an easy and scalable way to generate harder questions that are easily verifiable.

  • The randomly generated questions can have exponentially many different combinations, which is a good way to test how models may be overfitting to a few samples, while being not generalizable at all. The open source of platform and data is also useful for future iterations of models.

  • The author notes a few counterintuitive findings, like the Qwen-3.8-Max performing better than GPT-5.5, which is interesting.

Critical weaknesses
  • The authors provide insufficient justification on the use of FR as primary metrics. They noted "A high FR reveals that endpoint successes are deceptive-stemming from superficial heuristic shortcuts or random guessing rather than grounded visual reasoning." However, from the problem generation protocol and settings, it seems that probe questions are strictly harder than base questions. Therefore, a lower accuracy on probe questions may not indicate that the model does not perform "grounded visual reasoning." Instead, it may just be that the model's capability, or reasoning efforts, is simply not enough to solve harder questions. GPT-5.5, in particular, has a reasoning effort setting. However, such setting has not been reported under the current protocol, and the results are not compared across settings.

  • The core finding seems to be that the frontier models (like GPT-5.5) has a higher FR and lower acc then Qwen-3.6-max under more complex settings. However, the authors also noted all instructions are in Chinese across all evaluations. I wonder if the results or orderings may change if the author used another lanaguage (like English). In addition, all instructions in the paper figures are in English, and it's hard to evaluate the benchmark if we cannot see the original instructions passed.

  • It's not clear why the authors used gemini-3.5-flash as an LLM judge. As far as I can see, all outputs / answers are strictly structural, and LLM is essentially doing a string match. Can authors clarify why this is needed?

Questions for authors

Please see weaknesses.

13 comments
AuthorsOpenReview ↗

Reply to Critical Weakness 1 (Probe Necessity)

We thank the reviewer for this insightful comment. Regarding the concern that "it seems that probe questions are strictly harder than base questions. Therefore, a lower accuracy on probe questions may not indicate that the model does not perform 'grounded visual reasoning.' Instead, it may just be that the model's capability, or reasoning efforts, is simply not enough to solve harder questions," we address it from two core perspectives: (1) providing a task-by-task justification for the logical necessity of the probe questions, demonstrating that solving them is a necessary prerequisite for solving the base questions; and (2) conducting a controlled experiment with 7–11-year-old children under a "no-guessing" condition, empirically proving that when performing genuine visual reasoning, subjects with limited capabilities exhibit an FR of strictly 0.0%.

First, we do not deny that certain probe questions (especially those requiring the enumeration of complete intermediate states) are more demanding than base questions in terms of execution and output granularity. However, solving these probe questions is necessary for solving the base questions. The reason these probe questions appear harder is that base questions typically only evaluate a highly compressed final outcome with a very small answer space (e.g., a 50% random guessing probability for clockwise/counterclockwise in gear_rotation, a 25% guessing probability for four-option multiple-choice questions, or simply outputting a single numerical value). Consequently, models can easily bypass genuine deduction and hit the correct final answer via random guessing or local visual shortcuts. In contrast, probe questions require explicitly outputting the indispensable intermediate states needed to derive the base answer, thereby raising the anti-guessing threshold. We have supplemented specific justifications for the necessity of each probe question across all tasks, which will also be included in the revised appendix, as detailed below:

  • Task 1: pattern_completion
    • base:
      • question summary: Analyze the patterns in the options one by one to find the option that can exactly fill all the missing regions in the large hexagonal pattern to complete the large hexagon.
    • probe 1:
      • question summary: Analyze the patterns in the options one by one to determine whether each option can exactly fill all the missing regions in the large hexagonal pattern to complete the large hexagon.
    • Probe Necessity:
      • probe 1: Solving the base question requires spatially comparing each option pattern against the numbered missing regions (1–5) in the image. To select the single correct option and rule out the incorrect ones, the solver must confirm that the correct option has no conflicts with any missing region (i.e., none), while identifying the specific missing region(s) that each incorrect option fails to fill (i.e., conflicting_hole_id) as the basis for elimination. Therefore, solving this probe question is an indispensable intermediate step for solving the base question, making this probe necessary.
  • Task 2: find_different
    • base:
      • question summary: Find the 3 odd-one-out elements in the image that differ from all other elements, and list their position coordinates.
    • probe 1:
      • question summary: Identify the number of odd-one-out elements in each row.
    • probe 2:
      • question summary: Identify the number of odd-one-out elements in each column.
    • Probe Necessity:
      • probe 1 & probe 2: The base question requires locating all outlier elements in a 2D grid and determining their 2D coordinates (x,y), which spatially requires accurately identifying the presence and count of outlier elements along both the orthogonal 1D "row dimension (determining x)" and "column dimension (determining y)." Probe 1 and Probe 2 separately test the capability to identify and count the outlier elements along the row dimension alone (per-row counts) and along the column dimension alone (per-column counts). If a solver fails when independently answering Probe 1 or Probe 2, it proves that when facing this image, the solver cannot even accurately identify and localize the outliers along a single 1D axis (row-wise or column-wise)—making it impossible to genuinely determine both x and y to solve the 2D coordinates (x,y) in the base question. Therefore, being able to solve both probe questions is a necessary prerequisite for genuinely solving the base question, making both probes necessary.
  • Task 3: count_multiple_patterns
    • base:
      • question summary: Count each type of pattern and report the quantity of each pattern.
    • probe 1:
      • question summary: Count the quantity of each pattern, correct the erroneous quantity values in a given (pattern_id, quantity) sequence containing incorrect counts, and provide the correct quantity for each pattern.
    • Probe Necessity:
      • probe 1: Solving the base question requires accurately counting the quantity of each type of pattern among a scattered pile. Meanwhile, the probe requires verifying and correcting a quantity sequence that contains erroneous values to output the correct sequence for all patterns. In fact, when solving Probe 1, the solver can completely skip the step of cross-checking the erroneous numbers in the prompt; as long as they accurately count the true quantity of each pattern, they can directly output the answer to the probe question. This means that the core visual counting process required to solve Probe 1 on this image is completely identical to that required for the base question. If a solver fails to provide the correct quantities when independently answering Probe 1 (e.g., being misled by the erroneous numbers in the prompt due to an inability to clearly perceive the patterns), it indicates that when faced with this image, the solver is fundamentally unable to independently and accurately count the patterns of each type. Therefore, the ability to solve this probe question is a necessary condition for genuinely solving the base question, making this probe necessary.
  • Task 4: trailer_cubes_count
    • base:
      • question summary: Based on the three orthographic views in the image, deduce the maximum possible number of unit cubes.
    • probe 1:
      • question summary: Based on the three orthographic views in the image, deduce the maximum possible height of the cube column at each coordinate in the top view.
    • Probe Necessity:
      • probe 1: The base question asks for the maximum total number of unit cubes based on the three orthographic views. The rigorous reasoning chain to solve the base question is: first, combine the column heights from the front view and the row heights from the left view to deduce the maximum possible column height (height) at each (x, y) coordinate in the top view, and then sum the heights across all positions. The answer to the base question directly equals the cumulative sum of all height values in Probe 1; without first computing the maximum column height at each position in Probe 1, it is fundamentally impossible to compute the maximum total count in the base question. Therefore, solving this probe question is an indispensable intermediate step for solving the base question, making this probe necessary.
  • Task 5: cubes_count
    • base:
      • question summary: Based on the 3D isometric view, deduce the total number of unit cubes.
    • probe 1:
      • question summary: Based on the 3D view, deduce the height of the cube column at each coordinate position in the top view.
    • probe 2:
      • question summary: Based on the 3D view, deduce whether a unit cube exists at each 3D spatial coordinate position.
    • Probe Necessity:
      • probe 1 & probe 2: Due to occlusion in the 3D view, solving the base question to deduce the total number of unit cubes cannot rely on merely counting surface-visible cubes; instead, it requires a complete reasoning chain from "spatial occlusion completion" to "vertical counting aggregation." First, the solver must accurately determine whether a cube exists at each (x, y, z) spatial coordinate; without first identifying which (x, y, z) positions in the 3D space are occupied and which are empty (the exact objective of Probe 2), valid counting cannot even begin. Subsequently, once the spatial distribution of the cubes in 3D space is established, accurately computing the total count without omission or duplication requires aggregating the number of existing cubes along the vertical dimension at each base grid (x, y) into the column height h at that coordinate (the exact objective of Probe 1), and then summing the column heights across all (x, y) positions to obtain the total number of unit cubes required by the base question. Therefore, solving both probe questions is an indispensable intermediate step for solving the base question, making both probes necessary.
  • Task 6: gear_rotation
    • base:
      • question summary: Given the initial rotation direction of the first gear (clockwise), deduce the rotation direction of the last gear.
    • probe 1:
      • question summary: Given the initial rotation direction of the first gear (counterclockwise), deduce the rotation direction of the last gear.
    • probe 2:
      • question summary: Identify the connection type between each pair of adjacent gears.
    • probe 3:
      • question summary: Given the initial rotation direction of the first gear (clockwise), deduce the rotation direction of every gear.
    • Probe Necessity:
      • probe 1: Solving the base question requires determining the overall directional transmission relationship (same or opposite direction) from the start to the end of the entire gear train. Solving Probe 1 simply requires flipping the initial gear's rotation direction from clockwise (as set in the base question) to counterclockwise, and then applying the exact same end-to-end directional transmission relationship established during the base question to deduce the same target (the rotation direction of the last gear). Thus, Probe 1 serves as a necessary counterfactual test to verify whether the directional transmission relationship of the gear train has been genuinely established (rather than relying on a binary guess), making this probe necessary.
      • probe 2: Solving the base question requires determining the overall directional transmission relationship (same or opposite direction) across the entire gear train. Meanwhile, the directional transmission between adjacent gears depends on their connection type (direct meshing or crossed belt yields opposite directions, whereas parallel belt yields the same direction). Without first accurately identifying the connection type of each adjacent gear pair in the image, directional transmission reasoning cannot proceed. Therefore, solving this probe question is an indispensable intermediate step for solving the base question, making this probe necessary.
      • probe 3: In the base question, the rotation direction of each subsequent gear directly depends on that of the preceding gear. Therefore, in the process of solving the base question, once the directional transmission relationships across the gear train are determined, arriving at the rotation direction of the last gear requires recursively deducing the direction of every gear step-by-step from the 1st gear all the way to the last gear. Probe 3 directly corresponds to this indispensable recursive chain. Therefore, solving this probe question is an indispensable intermediate step for solving the base question, making this probe necessary.
  • Task 7: connect_lines
    • base:
      • question summary: Trace the green lines in the image, provide the connection mapping between each number on the left and its corresponding letter on the right, and output the result in the required format.
    • probe 1:
      • question summary: Trace the green lines in the image to find the corresponding left-side number connected to each right-side letter, provide the connection mapping between each number and its corresponding right-side letter, and output the result in the required format (which differs slightly from the output format of the base question).
    • Probe Necessity:
      • probe 1: Solving the base question requires tracing the intertwined green lines one by one to establish a one-to-one correspondence between the numbers on the left and the letters on the right. Probe 1 requires starting from the letters on the right (from top to bottom) to find the connected numbers on the left and outputting them as tuples; this involves the exact same visual tracing of all green lines and pairing establishment required by the base question (imposing no additional visual tracing burden). Its necessity lies in the fact that genuine topological connectivity of continuous curves inherently possesses bidirectional symmetry (tracing from a left-end number to a right-end letter is equivalent to tracing backward from a right-end letter to a left-end number). If a solver has genuinely established the correct line-tracing mapping, altering the tracing direction and output format will not affect its accuracy; conversely, if a model happens to get the base question right merely via unidirectional local spatial shortcuts or by guessing pairings by elimination, it is likely to collapse under reverse tracing. Therefore, the ability to solve this probe question is a necessary prerequisite for genuinely solving the base question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.