- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines why VLMs can make accurate local decisions but fail to output the correct final structure. The authors propose local-first-then-solve, which uses a frozen VLM to score individual decisions and a task-specific solver to combine them into a final answer. Experiments cover different tasks and examine how the gap changes with model size, additional reasoning, and post-training.
Critical strengths
-
The experiments are comprehensive, covering multiple tasks, model sizes, and backbones. Stronger inference baselines and post-training experiments further strengthen the evaluation.
-
The task design is useful for examining the local–global gap. Comparing Global, Local, and Local+Solve separates the benefits of decomposition and structured inference.
Critical weaknesses
-
The novelty is somewhat limited. Combining local VLM scores with solvers is a natural application of decomposition and structured inference.
-
The benefit of stronger VLMs remains insufficiently tested. Increasing the Qwen3-VL thinking budget reduces the GUI task gap from 20.5% to 7.4%. More evaluation on recent open-source models like Qwen 3.8 and closed-source models is needed.
-
The tasks rely on predefined local decisions, constraints, and solvers, while the public benchmark evaluation emphasizes ordering and arrangement. Applicability beyond these predefined task formulations remains unclear.
Questions for authors
-
Does local+solve retain a substantial advantage over newer SoTA open-source and closed-source VLMs under comparable inference budgets?
-
Can the shared-context experiment be extended to other tasks?
-
What new insight distinguishes the proposed assembly deficit from previously studied compositionality gaps and the benefits of task decomposition?
5 comments
Q3. Difference from compositionality gaps and decomposition
Two behavioral insights distinguish our diagnosis. First, the compositionality-gap condition of Press et al. (2023), correct sub-answers, does not suffice in VLMs without thinking, whereas assembly outside the model uses those answers and recovers errors from imperfect local judgments. Second, local queries and the solver have separable effects, and a solver gain need not mean recovery. Decomposition and structured inference are established, and these findings are behavioral.
(1) Relation to the compositionality gap
Press et al. show that language models can answer all sub-questions correctly yet fail the whole question, and narrow this gap with self-ask prompting.
-
The same condition arises in VLMs. When Qwen2.5-VL-7B's own four RecipeQA step answers, kept in one conversation, are all correct, they uniquely determine the candidate, yet its final choice is wrong in 308/465 such cases (App. C.1).
-
Retained answers are not fully used. We put separately obtained local answers in the conversation, then ask for the final order. Qwen3.8 reaches 74.2% complete-order accuracy, versus 84.8% when exactly those answers are assembled outside the model, and Gemma 57.3% versus 74.6%. Wrong final choices despite all-correct retained answers occur in RecipeQA but not in the tested GUI and SPIQA tasks, where history still trails assembly but the solver sees full scores and the model only chosen letters. With sequentially obtained answers, assembly beats the final choice for Gemma (+5.3 pp), while Qwen3.8's own choice is slightly higher (74.6% versus 72.4%, not significant, App. J.12).
-
Inserting correct answers does not close it. For Qwen3.8 and Gemma 4, we replace their retained answers with a candidate's answers, with no new instruction (App. J.12).
| RecipeQA, 929 questions (%) | Qwen3.8-27B | Gemma 4-31B-it |
|---|---|---|
| Direct choice, no answers | 67.9 | 58.7 |
| Correct candidate's answers inserted | 76.7 | 61.7 |
| Same answers assembled outside the model | 100 | 100 |
| Wrong candidate's answers: that order chosen | 13.7 | 23.0 |
With wrong answers, most choices stay as with correct ones (86.3% and 79.5%), so the final choice depends only weakly on the sub-answers in context.
-
A self-ask-style control does not close it. A single-pass, self-ask-style decomposition with self-chosen or prescribed sub-questions and a 768-token cap leaves 7B 25 to 34 percentage points (pp) below local+solve on capacitated assignment (App. C.3).
-
Recovery from imperfect local judgments. With Qwen3.8 on all 1,375 figure–caption links of 166 SPIQA papers, Local links every pair correctly in only 41.0% of papers, yet local+solve does so in 75.9% on the same scores, versus 60.2% for Global (App. J.7).
(2) Relation to decomposition and structured inference
-
Separate queries and the solver are distinct. In six of seven Qwen2.5-VL-7B settings, the Global-to-Local gain exceeds the Local-to-local+solve gain (Sec. 4.3, Table 1). In Qwen3.8 recipe alignment, the solver adds 14.9 pp over Local on identical scores, yet local+solve remains below Global (Tables 31–32, p. 51).
-
Structural validity alone does not explain recovery. A label-free rule replaces only invalid or infeasible primary-prompt Global answers with local+solve. local+solve is more accurate than this rule in 41 of 52 settings, and the rule in 11. The rule's margin exceeds 1.1 pp only in six settings where Global already beats local+solve. Excluding Molmo2 hierarchy leaves 40 of 51 favoring local+solve (App. J.11). The rule and local+solve differ only on already-feasible Global outputs.
W3. Predefined formulations and applicability
SPIQA transfers an existing assignment formulation to document processing, while new structures still need manual formulation.
Five figures are matched to five captions per paper. Constraints improve all five tested backbones over Local. Complete-set accuracy also beats Global on all five, by 7.2 to 53.0 pp, although Molmo2's Global may confuse image and caption indices (App. J.7).
SPIQA inputs fit Phi-4-reasoning-vision-15B's 16,384-token context limit (longest input 12,447 tokens, App. J.9). Only 3 of its 166 outputs are unparseable. Even crediting those as correct gives Global 60.6% caption accuracy, below local+solve's 89.6% (App. J.7).
On 250 JigShape 4×4 puzzles, Qwen3.8 uses edge queries and a heuristic solver. The strongest tested no-thinking forced-answer Global control gives:
| Method | Piece accuracy (%) | Exact puzzles (%) |
|---|---|---|
| Global, forced answer | 7.5 | 0.0 |
| local+solve | 93.2 | 64.0 |
Forced answers parse for 245 puzzles yet solve none, so the recovery is not explained by completion failures alone. Shape ambiguity remains, and this new solver does not automate task formulation (App. J.6).
Dear Reviewer km3X,
Thank you once again for your careful and detailed review. The points you raised were precise, and responding to them has strengthened both our argument and the evidence we provide. We are also grateful for the references you pointed us to, which we have now discussed in relation to our own formulation.
We have posted our responses and hope that they resolve your concerns. If anything remains unclear or unconvincing, we would be very glad to discuss it further while the discussion period is open.
Revision marks. In the revised PDF, voNi: red text, KSyL: blue text, km3X: yellow highlight. Changes shared by reviewers combine marks (e.g., voNi and KSyL: purple text). Marks show added or revised explanations.
We thank the reviewer for recognizing the comprehensive experiments and the comparison of Global, Local, and local+solve.
Q1. Newer models and comparable inference budgets
Partly. With thinking, Global never exceeds local+solve at any tested budget on GUI (Qwen3.8) or on Gemma 4's recipe and document tasks, and reaches it on same-input RecipeQA (Qwen3.8) only at about 48 to 63 times the runtime. GPT-6 Sol closes the gap on JigShape 4×4 with high reasoning, at 1.7 times the API cost.
Thinking budgets. Global thinks up to each token budget and is then asked for its answer, except at RecipeQA's largest budget (original output). Times are relative to no-thinking local+solve and mostly estimated.
Without thinking, GUI Global reaches 87.8% in 0.43× the time. With thinking, the lowest budgets already take 1.6 to 4.2 times as long and are less accurate. It reaches local+solve only on same-input RecipeQA, from 2,048 tokens (85.6% at 47.9×, not significant, App. J.10). Gemma ratios include model loading (App. J.5), and JigShape uses one budget (App. J.6).
For Qwen3-VL-30B-A3B-Thinking, the review's 20.5 percentage point (pp) gap narrows in a post hoc 16,384-token run to 7.4 pp on all 222 screens and to 2.4 pp on the 207 screens where thinking finishes (App. D.2).
Other open backbones. Without thinking, local+solve is more accurate than the strongest tested Global control, chosen per setting to favor Global, in four, seven, and five of seven settings for Qwen3.8, Gemma, and GLM-4.6V, and Global in the rest. The solver improves Local in six of seven settings for each. Crediting invalid primary-prompt Global answers as correct still leaves positive gaps on Gemma's recipe, ordering and document tasks and GLM's assignment and ordering. Molmo2-8B's local+solve is more accurate in all seven settings, although its hierarchy Global has 27.4% invalid outputs (App. J.1).
Closed model: GPT-6 Sol. The API rejects token-probability requests with reasoning enabled, so our local+solve uses no thinking. Even then, the API returns only the top candidate's probability for 97.8% of JigShape 4×4 edge queries.
On JigShape 4×4, a pre-specified rule fills missing probabilities before heuristic placement:
Adapted local+solve leads low-reasoning Global by 10.6 pp. High-reasoning Global reaches similar piece accuracy (94.2% versus 94.3%) and solves more puzzles exactly (187 versus 170 of 250, not significant).
In a cross-model comparison without reasoning, Sol Global is at least as accurate as Qwen3.8's local+solve in all seven settings. It reaches 98.7% on 100-page documents, where local+solve was not evaluated. Sol's own recovery with local+solve is thus shown only on JigShape 4×4 and depends on its score-completion rule (App. J.13).
Q2. Shared context beyond RecipeQA
We extended shared-context queries to GUI partition and document alignment. Against the strongest controls, a pair-accuracy gain remains only for Qwen2.5-VL-7B's partition.
Each query sees all screens or all pages and section descriptions. On partition, the strongest tested controls give:
Against primary prompts, both models gain (7B: +10.8 pp, Qwen3.8: +4.9 pp), while Qwen3.8 matches its stronger control above (95.7% versus 96.1%). The controls are the most accurate tested Global variants, here including prompted reasoning for 7B and a format example for Qwen3.8 (App. J.3). Complete-partition accuracy is lower with local+solve for both models, and document-alignment gaps versus primary Global are +2.2 pp for 7B and -4.2 pp for Qwen3.8, neither establishing an advantage (Table 36, App. J.3). Thus giving each local query the full context changes how much is recovered (7B partition local+solve: 94.4% with original queries, 75.9% shared), and higher pair accuracy need not mean better complete partitions.