- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper addresses two problems in visual puzzle benchmarks for multimodal large language models, entangled variables with uncontrolled difficulty and illusory competence from endpoint-only scoring. The authors introduce the ButtonsBench platform and ButtonsBench-1, a benchmark of 691 basic questions and 903 paired co-origin probes across 9 visual puzzle tasks parameterized by task-specific buttons with discrete levels. They also propose a Co-Origin Probing Methodology that uses majority consensus over 5 trials to measure baseline endpoint accuracy, verified joint accuracy, falsification rate, and structural crash points. Across four MLLMs, macro falsification rate ranges from 26.1% to 72.4% and verified joint accuracy reverses the baseline endpoint accuracy ranking between gpt-5.5 and qwen-3.8-max.
Critical strengths
- The three-stage ButtonsBench platform logs intermediate state metadata during material generation and generates paired co-origin probes on the same material instance to test different parts of reasoning.
- Across four MLLMs and 1,594 question instances, macro falsification rate FR reaches 26.1% to 72.4%, and verified joint accuracy Acc_joint reverses the baseline endpoint accuracy Acc_base ranking as qwen-3.8-max drops from 57.4% to 46.1% while gpt-5.5 drops from 60.1% to 45.1%.
Critical weaknesses
- Several probing questions seem harder than their base questions because they require a more comprehensive answer, and the paper gives no per-probe results. Probes in mirroring_patterns and pattern_completion require identifying cell color mismatches or conflicting hole IDs across every distractor, and they are scored with binary correct or wrong with no partial credit.
- The Single-Button Crash Point metric CP_single does not really isolate individual factors because it anchors all other buttons at L1 and evaluates only one question instance per level. For two-level buttons such as rotation_enabled and overlap_level, L1 corresponds to the hardest setting, which forces rotation across every 1D sweep in pattern_completion where all four models got CP_single = 0 despite reaching up to 11.1% joint accuracy.
Questions for authors
- What is the model accuracy for each probe category?
- How does CP_single change in pattern_completion and count_multiple_patterns if two-level buttons like rotation_enabled and overlap_level are fixed at L0 instead of L1, and how much does CP_single vary across random seeds?
7 comments
Reply to Questions 2 (Buttons Fixed at L0 instead of L1)
We thank the reviewer for this comment. Regarding the reviewer's question "How does CP_single change in pattern_completion and count_multiple_patterns if two-level buttons like rotation_enabled and overlap_level are fixed at L0 instead of L1", we supplement the following statistical results:
CP_single (Acc_joint) when two-level buttons are anchored at L1 vs. L0 (L1→L0)
| Task / Button | qwen-3.8-max | gpt-5.5 | gemini-3.7-flsh | MiniMax-M3 |
|---|---|---|---|---|
| Count Patterns | ||||
| Rotation (L0-1) | >1→>1 | >1→>1 | >1→>1 | 0→0 |
| Occlusion (L0-1) | >1→>1 | >1→>1 | 0→>1 | 0→0 |
| Quantity (L0-4) | >4→>4 | >4→>4 | 0→>4 | 0→0 |
| Categories (L0-7) | >7→>7 | 3→>7 | 0→>7 | 0→0 |
| Pattern Comp. | ||||
| Rotation (L0-1) | 0→0 | 0→0 | 0→0 | 0→0 |
| Grid Size (L0-5) | 0→1 | 0→0 | 0→0 | 0→0 |
| Quantity (L0-3) | 0→1 | 0→0 | 0→1 | 0→0 |
| Hole Size (L0-2) | 0→0 | 0→0 | 0→0 | 0→0 |
From the results above, we can see that:
- On the
count_multiple_patternstask, fixing the two-level buttons (RotationandOcclusion) at L0 (disabling rotation and occlusion) substantially increases CP_single on the remaining buttons forgemini-3.7-flashandgpt-5.5, whileqwen-3.8-maxandMiniMax-M3remain unchanged: When fixed at L1 (rotation and occlusion enabled),gemini-3.7-flashhas a CP_single of 0 across theOcclusion,Quantity, andCategoriesbuttons, andgpt-5.5has a CP_single of3onCategories. When fixed at L0 (no rotation or occlusion),gemini-3.7-flashpasses the highest levels across all buttons (increasing to>1,>4, and>7), andgpt-5.5also improves to>7onCategories; meanwhile,qwen-3.8-max(consistently never reaching a crash point,>L_max) andMiniMax-M3(consistently 0) remain unchanged. This indicates that in this task, enabling rotation and occlusion (L1) indeed introduces substantial extra visual interference forgemini-3.7-flashandgpt-5.5; fixing the two-level buttons at L0 eliminates this interference, thereby reflecting the single-dimension capacity of these two models regardingQuantityandCategoriesunder non-rotated and unoccluded conditions. - On
pattern_completion, changing the two-level button (Rotation) from L1 to L0 (disabling rotation) results in almost no change in CP_single across the models: After fixingRotationat L0, CP_single forgpt-5.5andMiniMax-M3remains 0 across all buttons;qwen-3.8-max(onGrid SizeandQuantity) andgemini-3.7-flash(onQuantity) only slightly change from 0 to 1 (i.e., answering correctly only at the L0 level and failing consecutively starting from L1). This indicates that the widespread CP_single=0 across models onpattern_completionis not caused by fixingRotationat its maximum level L1, but rather because the visual complexity introduced by the current starting level parameters of the remaining buttons in this task (such as the grid size, missing component count, and hole size corresponding to L0/L1) is still too high for these four models. Leveraging the parameterized generation capability of the ButtonsBench platform, in the next version of the benchmark we can easily map the lower levels of these buttons to parameter values with lower visual complexity (e.g., smaller grid sizes), thereby characterizing the capability boundaries of models in the lower-difficulty regime of this task at a finer granularity.
Reply to Questions 2 (CP_single Variance Across Random Seeds)
Regarding the reviewer's question on "how much does CP_single vary across random seeds," we generated three additional random seeds for both the count_multiple_patterns and pattern_completion tasks, and evaluated all four models (qwen-3.8-max, gpt-5.5, gemini-3.7-flash, and MiniMax-M3) across these new seeds.
Table 1 and Table 2 below show the CP_single results across the original seed and the 3 new seeds under the default setting (remaining buttons fixed at L1) and when the two-level buttons are fixed at L0, respectively (formatted as Orig/S1/S2/S3 (Δ), where Δ denotes the maximum range across seeds):
Table 1: CP_single across original & 3 new seeds (Orig/S1/S2/S3, Δ=max-min), all L1
| Task / Button | qwen-3.8-max | gpt-5.5 | gemini-3.7-flash | MiniMax-M3 |
|---|---|---|---|---|
| count_multiple_patterns | ||||
| Rotation (L0-1) | >1/>1/>1/>1 (0) | >1/>1/>1/>1 (0) | >1/>1/>1/>1 (0) | 0/>1/>1/>1 (2) |
| Occlusion (L0-1) | >1/>1/>1/>1 (0) | >1/>1/>1/>1 (0) | 0/>1/>1/>1 (2) | 0/>1/>1/>1 (2) |
| Quantity (L0-4) | >4/>4/>4/>4 (0) | >4/>4/>4/>4 (0) | 0/3/3/3 (3) | 0/>4/>4/>4 (5) |
| Categories (L0-7) | >7/>7/>7/>7 (0) | 3/2/2/2 (1) | 0/3/>7/3 (8) | 0/>7/>7/>7 (8) |
| pattern_completion | ||||
| Rotation (L0-1) | 0/0/0/0 (0) | 0/0/0/0 (0) | 0/0/0/0 (0) | 0/0/0/0 (0) |
| Grid Size (L0-5) | 0/0/0/0 (0) | 0/0/0/0 (0) | 0/0/0/0 (0) | 0/0/0/0 (0) |
| Quantity (L0-3) | 0/0/0/0 (0) | 0/0/0/0 (0) | 0/0/0/0 (0) | 0/0/0/0 (0) |
| Hole Size (L0-2) | 0/0/0/0 (0) | 0/0/0/0 (0) | 0/0/0/0 (0) | 0/0/0/0 (0) |
Table 2: CP_single across original & 3 new seeds (Orig/S1/S2/S3, Δ=max-min) two-level buttons pinned L0
| Task / Button | qwen-3.8-max | gpt-5.5 | gemini-3.7-flash | MiniMax-M3 |
|---|---|---|---|---|
| count_multiple_patterns | ||||
| Rotation (L0-1) | >1/>1/>1/>1 (0) | >1/>1/>1/>1 (0) | >1/>1/>1/>1 (0) | 0/>1/>1/>1 (2) |
| Occlusion (L0-1) | >1/>1/>1/>1 (0) | >1/>1/>1/>1 (0) | 0/>1/>1/>1 (2) | 0/>1/>1/>1 (2) |
| Quantity (L0-4) | >4/>4/>4/>4 (0) | >4/>4/>4/>4 (0) | 0/3/3/3 (3) | 0/>4/>4/>4 (5) |
| Categories (L0-7) | >7/>7/>7/>7 (0) | 3/2/2/2 (1) | 0/3/>7/3 (8) | 0/>7/>7/>7 (8) |
| pattern_completion | ||||
| Rotation (L0-1) | 0/0/0/0 (0) | 0/0/0/0 (0) | 0/0/0/0 (0) | 0/0/0/0 (0) |
| Grid Size (L0-5) | 0/0/0/0 (0) | 0/0/0/0 (0) | 0/0/0/0 (0) | 0/0/0/0 (0) |
| Quantity (L0-3) | 0/0/0/0 (0) | 0/0/0/0 (0) | 0/0/0/0 (0) | 0/0/0/0 (0) |
| Hole Size (L0-2) | 0/0/0/0 (0) | 0/0/0/0 (0) | 0/0/0/0 (0) | 0/0/0/0 (0) |
From the results, we can see that:
- CPsingle remains highly stable across random seeds, and the results are almost completely consistent across the three newly generated random seeds (
S1/S2/S3):- In Table 2 (two-level buttons fixed at L0): Out of all 32 (Model × Button) cells, 27 cells are completely identical across all 4 seeds (Δ=0).
- In Table 1 (fixed at L1): Out of all 32 cells, 24 cells are completely identical across all 4 seeds (Δ=0).
- Larger cross-seed variations (Δ≥2) only appear in boundary models (gemini-3.7-flash and MiniMax-M3) under single-instance slices, and comparing Table 1 and Table 2 reveals the source of this variation:
- On the
count_multiple_patternstask, when fixed at L1 (Rotation and Occlusion enabled, Table 1), gemini-3.7-flash scores 0 on the original seedOrig(because it happened to fail on the two single-question instances at L0 and L1), whereas on the three new seedsS1/S2/S3it stably achieves>1(Occlusion) and3/3/3(Quantity). - However, once the two-level buttons are fixed at L0 (rotation and occlusion disabled, Table 2), gemini-3.7-flash's cross-seed variation completely disappears, achieving 100% consistency across all 4 seeds (
>1/>1/>1/>1,>4/>4/>4/>4, and>7/>7/>7/>7, Δ=0). This indicates that gemini-3.7-flash's cross-seed variation in Table 1 is likely because enabling "Rotation + Occlusion (L1)" places the task difficulty right at gemini-3.7-flash's capability boundary (where success or failure is more sensitive to the specific occlusion positions of patterns in a single random seed); whereas under no occlusion/rotation (L0) or extremely high difficulty (pattern_completion), its performance is completely stable across seeds.
- On the
This brings us valuable new insights, and in future versions of the benchmark, we will generate question instances across multiple random seeds for each button-level combination.
Reply to Question 1 (model accuracy for each probe)
We greatly appreciate the reviewer's question regarding "What is the model accuracy for each probe category?" Here, we supplement both the detailed accuracy statistics of each model on the base question and all individual co-origin probes across every task, as well as the aggregated accuracy results of each model across each probe category (all of which will also be added to the Appendix of the revised manuscript). The results are presented below:
Model Accuracy Across Each Probe |Task Name|Model Name|Acc_base (%)|Acc_joint (%)|FR (%)|probe1 Acc_maj (%)|probe2 Acc_maj (%)|probe3 Acc_maj (%)|Micro Acc_maj Instance-level (%)|Macro Acc_maj Probe-level(%)| |-|-|-:|-:|-:|-:|-:|-:|-:|-:| |rolling_dice|qwen-3.8-max|69.39|46.94|32.35|59.18|N/A|N/A|59.18|59.18| |rolling_dice|gpt-5.5|73.47|40.82|44.44|44.9|N/A|N/A|44.9|44.9| |rolling_dice|gemini-3.7-flash|67.35|24.49|63.64|34.69|N/A|N/A|34.69|34.69| |rolling_dice|MiniMax-M3|12.24|2.04|83.33|2.04|N/A|N/A|2.04|2.04| |connect_lines|qwen-3.8-max|62.86|62.86|0|65.71|N/A|N/A|65.71|65.71| |connect_lines|gpt-5.5|68.57|65.71|4.17|68.57|N/A|N/A|68.57|68.57| |connect_lines|gemini-3.7-flash|51.43|48.57|5.56|54.29|N/A|N/A|54.29|54.29| |connect_lines|MiniMax-M3|37.14|28.57|23.08|28.57|N/A|N/A|28.57|28.57| |count_multiple_patterns|qwen-3.8-max|98.75|96.25|2.53|96.88|N/A|N/A|96.88|96.88| |count_multiple_patterns|gpt-5.5|94.38|88.75|5.96|91.25|N/A|N/A|91.25|91.25| |count_multiple_patterns|gemini-3.7-flash|61.25|36.88|39.8|45.62|N/A|N/A|45.62|45.62| |count_multiple_patterns|MiniMax-M3|10|3.12|68.75|11.25|N/A|N/A|11.25|11.25| |cubes_count|qwen-3.8-max|48.41|30.95|36.07|38.1|31.75|N/A|34.92|34.92| |cubes_count|gpt-5.5|26.19|16.67|36.36|23.81|22.22|N/A|23.02|23.02| |cubes_count|gemini-3.7-flash|36.51|8.73|76.09|11.9|11.11|N/A|11.51|11.51| |cubes_count|MiniMax-M3|11.9|0.79|93.33|1.59|0.79|N/A|1.19|1.19| |find_different|qwen-3.8-max|90|88|2.22|94|92|N/A|93|93| |find_different|gpt-5.5|100|100|0|100|100|N/A|100|100| |find_different|gemini-3.7-flash|42|38|9.52|56|42|N/A|49|49| |find_different|MiniMax-M3|40|34|15|56|38|N/A|47|47| |gear_rotation|qwen-3.8-max|83.33|55.56|33.33|83.33|66.67|66.67|72.22|72.22| |gear_rotation|gpt-5.5|94.44|61.11|35.29|77.78|61.11|61.11|66.67|66.67| |gear_rotation|gemini-3.7-flash|66.67|16.67|75|66.67|33.33|33.33|44.44|44.44| |gear_rotation|MiniMax-M3|50|0|100|33.33|5.56|5.56|14.81|14.81| |mirroring_patterns|qwen-3.8-max|24.44|11.11|54.55|11.11|N/A|N/A|11.11|11.11| |mirroring_patterns|gpt-5.5|35.56|11.11|68.75|15.56|N/A|N/A|15.56|15.56| |mirroring_patterns|gemini-3.7-flash|24.44|2.22|90.91|2.22|N/A|N/A|2.22|2.22| |mirroring_patterns|MiniMax-M3|13.33|0|100|0|N/A|N/A|0|0| |pattern_completion|qwen-3.8-max|24.31|11.11|54.29|12.5|N/A|N/A|12.5|12.5| |pattern_completion|gpt-5.5|34.03|7.64|77.55|12.5|N/A|N/A|12.5|12.5| |pattern_completion|gemini-3.7-flash|17.36|5.56|68|7.64|N/A|N/A|7.64|7.64| |pattern_completion|MiniMax-M3|15.28|0.69|95.45|3.47|N/A|N/A|3.47|3.47| |trailer_cubes_count|qwen-3.8-max|15.62|12.5|20|14.06|N/A|N/A|14.06|14.06| |trailer_cubes_count|gpt-5.5|14.06|14.06|0|20.31|N/A|N/A|20.31|20.31| |trailer_cubes_count|gemini-3.7-flash|9.38|1.56|83.33|1.56|N/A|N/A|1.56|1.56| |trailer_cubes_count|MiniMax-M3|0|0|--|0|N/A|N/A|0|0|
Model Accuracy Across Each Probe Category Following our probe taxonomy in Appendix A.4.5, the specific category of each probe is:
connect_lines(Probe 1)count_multiple_patterns(Probe 1),gear_rotation(Probe 1)