acceptodds
Under review as a conference paper at ICLR 2027

Can VLMs Assemble Local Decisions into Global Structures?

Abstract

Many vision-language tasks require combining multiple local decisions into a single structured output. We ask whether vision-language models (VLMs) can reliably assemble such decisions into assignments, orderings, groupings, and hierarchies. We identify an assembly deficit: a model can answer the underlying local questions accurately yet fail to produce the corresponding global structure. On RecipeQA, we isolate this deficit while holding the model and task input fixed, finding that local step-image decisions are substantially stronger than whole-order prediction or scoring. We introduce local+solve, a training-free procedure that extracts local scores from a frozen VLM and assembles them with a task-specific solver under task-defined constraints. Across six controlled structural families, five tasks drawn from public benchmarks, and GUI-agent settings, local+solve improves over matched global prediction, including stronger inference variants where tested, and its gain operationally measures the recoverable part of the deficit. The deficit narrows from 7B to 72B but persists in most families. Post-training on local+solve's outputs nearly internalizes assembly on the trained task, while multi-family local+solve pseudo-labels improve global prediction on two unseen families, suggesting that assembly is a distinct, learnable capability.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Multimodal Reasoning, Structured Prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_km3XOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines why VLMs can make accurate local decisions but fail to output the correct final structure. The authors propose local-first-then-solve, which uses a frozen VLM to score individual decisions and a task-specific solver to combine them into a final answer. Experiments cover different tasks and examine how the gap changes with model size, additional reasoning, and post-training.

Critical strengths
  1. The experiments are comprehensive, covering multiple tasks, model sizes, and backbones. Stronger inference baselines and post-training experiments further strengthen the evaluation.

  2. The task design is useful for examining the local–global gap. Comparing Global, Local, and Local+Solve separates the benefits of decomposition and structured inference.

Critical weaknesses
  1. The novelty is somewhat limited. Combining local VLM scores with solvers is a natural application of decomposition and structured inference.

  2. The benefit of stronger VLMs remains insufficiently tested. Increasing the Qwen3-VL thinking budget reduces the GUI task gap from 20.5% to 7.4%. More evaluation on recent open-source models like Qwen 3.8 and closed-source models is needed.

  3. The tasks rely on predefined local decisions, constraints, and solvers, while the public benchmark evaluation emphasizes ordering and arrangement. Applicability beyond these predefined task formulations remains unclear.

Questions for authors
  1. Does local+solve retain a substantial advantage over newer SoTA open-source and closed-source VLMs under comparable inference budgets?

  2. Can the shared-context experiment be extended to other tasks?

  3. What new insight distinguishes the proposed assembly deficit from previously studied compositionality gaps and the benefits of task decomposition?

5 comments

Revision marks. In the revised PDF, voNi: red text, KSyL: blue text, km3X: yellow highlight. Changes shared by reviewers combine marks (e.g., voNi and KSyL: purple text). Marks show added or revised explanations.

We thank the reviewer for recognizing the comprehensive experiments and the comparison of Global, Local, and local+solve.

Q1. Newer models and comparable inference budgets

Partly. With thinking, Global never exceeds local+solve at any tested budget on GUI (Qwen3.8) or on Gemma 4's recipe and document tasks, and reaches it on same-input RecipeQA (Qwen3.8) only at about 48 to 63 times the runtime. GPT-6 Sol closes the gap on JigShape 4×4 with high reasoning, at 1.7 times the API cost.

Thinking budgets. Global thinks up to each token budget and is then asked for its answer, except at RecipeQA's largest budget (original output). Times are relative to no-thinking local+solve and mostly estimated.

Task, model (local+solve)Lowest budgetBest budget
GUI, 222 screens, Qwen3.8-27B (91.7%)65.7% at 1.6×90.9% at 28×
Same-input RecipeQA, 929 questions, Qwen3.8 (84.8%)24.4% at 4.1×87.6% at 62.5×
Capacitated recipes, 50 windows, Gemma 4-31B-it (83.3%)51.7% at 4.2×65.3% at 7.0×
Documents, 41 windows, Gemma 4 (92.7%)86.2% at 3.0×91.3% at 4.8×
JigShape 4×4, 100 puzzles, Qwen3.8 (93.3%)not run51.9% at 8,192 tokens

Without thinking, GUI Global reaches 87.8% in 0.43× the time. With thinking, the lowest budgets already take 1.6 to 4.2 times as long and are less accurate. It reaches local+solve only on same-input RecipeQA, from 2,048 tokens (85.6% at 47.9×, not significant, App. J.10). Gemma ratios include model loading (App. J.5), and JigShape uses one budget (App. J.6).

For Qwen3-VL-30B-A3B-Thinking, the review's 20.5 percentage point (pp) gap narrows in a post hoc 16,384-token run to 7.4 pp on all 222 screens and to 2.4 pp on the 207 screens where thinking finishes (App. D.2).

Other open backbones. Without thinking, local+solve is more accurate than the strongest tested Global control, chosen per setting to favor Global, in four, seven, and five of seven settings for Qwen3.8, Gemma, and GLM-4.6V, and Global in the rest. The solver improves Local in six of seven settings for each. Crediting invalid primary-prompt Global answers as correct still leaves positive gaps on Gemma's recipe, ordering and document tasks and GLM's assignment and ordering. Molmo2-8B's local+solve is more accurate in all seven settings, although its hierarchy Global has 27.4% invalid outputs (App. J.1).

Closed model: GPT-6 Sol. The API rejects token-probability requests with reasoning enabled, so our local+solve uses no thinking. Even then, the API returns only the top candidate's probability for 97.8% of JigShape 4×4 edge queries.

On JigShape 4×4, a pre-specified rule fills missing probabilities before heuristic placement:

Sol, 250 puzzlesPiece accuracy (%)List-cost ratio
local+solve, no reasoning94.31.00
Global, no reasoning14.60.09
Global, low reasoning83.70.81
Global, high reasoning94.21.70

Adapted local+solve leads low-reasoning Global by 10.6 pp. High-reasoning Global reaches similar piece accuracy (94.2% versus 94.3%) and solves more puzzles exactly (187 versus 170 of 250, not significant).

In a cross-model comparison without reasoning, Sol Global is at least as accurate as Qwen3.8's local+solve in all seven settings. It reaches 98.7% on 100-page documents, where local+solve was not evaluated. Sol's own recovery with local+solve is thus shown only on JigShape 4×4 and depends on its score-completion rule (App. J.13).

Q2. Shared context beyond RecipeQA

We extended shared-context queries to GUI partition and document alignment. Against the strongest controls, a pair-accuracy gain remains only for Qwen2.5-VL-7B's partition.

Each query sees all screens or all pages and section descriptions. On partition, the strongest tested controls give:

BackboneGlobal (%)local+solve (%)
Qwen2.5-VL-7B69.175.9
Qwen3.896.195.7

Against primary prompts, both models gain (7B: +10.8 pp, Qwen3.8: +4.9 pp), while Qwen3.8 matches its stronger control above (95.7% versus 96.1%). The controls are the most accurate tested Global variants, here including prompted reasoning for 7B and a format example for Qwen3.8 (App. J.3). Complete-partition accuracy is lower with local+solve for both models, and document-alignment gaps versus primary Global are +2.2 pp for 7B and -4.2 pp for Qwen3.8, neither establishing an advantage (Table 36, App. J.3). Thus giving each local query the full context changes how much is recovered (7B partition local+solve: 94.4% with original queries, 75.9% shared), and higher pair accuracy need not mean better complete partitions.

Q3. Difference from compositionality gaps and decomposition

Two behavioral insights distinguish our diagnosis. First, the compositionality-gap condition of Press et al. (2023), correct sub-answers, does not suffice in VLMs without thinking, whereas assembly outside the model uses those answers and recovers errors from imperfect local judgments. Second, local queries and the solver have separable effects, and a solver gain need not mean recovery. Decomposition and structured inference are established, and these findings are behavioral.

(1) Relation to the compositionality gap

Press et al. show that language models can answer all sub-questions correctly yet fail the whole question, and narrow this gap with self-ask prompting.

  • The same condition arises in VLMs. When Qwen2.5-VL-7B's own four RecipeQA step answers, kept in one conversation, are all correct, they uniquely determine the candidate, yet its final choice is wrong in 308/465 such cases (App. C.1).

  • Retained answers are not fully used. We put separately obtained local answers in the conversation, then ask for the final order. Qwen3.8 reaches 74.2% complete-order accuracy, versus 84.8% when exactly those answers are assembled outside the model, and Gemma 57.3% versus 74.6%. Wrong final choices despite all-correct retained answers occur in RecipeQA but not in the tested GUI and SPIQA tasks, where history still trails assembly but the solver sees full scores and the model only chosen letters. With sequentially obtained answers, assembly beats the final choice for Gemma (+5.3 pp), while Qwen3.8's own choice is slightly higher (74.6% versus 72.4%, not significant, App. J.12).

  • Inserting correct answers does not close it. For Qwen3.8 and Gemma 4, we replace their retained answers with a candidate's answers, with no new instruction (App. J.12).

RecipeQA, 929 questions (%)Qwen3.8-27BGemma 4-31B-it
Direct choice, no answers67.958.7
Correct candidate's answers inserted76.761.7
Same answers assembled outside the model100100
Wrong candidate's answers: that order chosen13.723.0

With wrong answers, most choices stay as with correct ones (86.3% and 79.5%), so the final choice depends only weakly on the sub-answers in context.

  • A self-ask-style control does not close it. A single-pass, self-ask-style decomposition with self-chosen or prescribed sub-questions and a 768-token cap leaves 7B 25 to 34 percentage points (pp) below local+solve on capacitated assignment (App. C.3).

  • Recovery from imperfect local judgments. With Qwen3.8 on all 1,375 figure–caption links of 166 SPIQA papers, Local links every pair correctly in only 41.0% of papers, yet local+solve does so in 75.9% on the same scores, versus 60.2% for Global (App. J.7).

(2) Relation to decomposition and structured inference

  • Separate queries and the solver are distinct. In six of seven Qwen2.5-VL-7B settings, the Global-to-Local gain exceeds the Local-to-local+solve gain (Sec. 4.3, Table 1). In Qwen3.8 recipe alignment, the solver adds 14.9 pp over Local on identical scores, yet local+solve remains below Global (Tables 31–32, p. 51).

  • Structural validity alone does not explain recovery. A label-free rule replaces only invalid or infeasible primary-prompt Global answers with local+solve. local+solve is more accurate than this rule in 41 of 52 settings, and the rule in 11. The rule's margin exceeds 1.1 pp only in six settings where Global already beats local+solve. Excluding Molmo2 hierarchy leaves 40 of 51 favoring local+solve (App. J.11). The rule and local+solve differ only on already-feasible Global outputs.

W3. Predefined formulations and applicability

SPIQA transfers an existing assignment formulation to document processing, while new structures still need manual formulation.

Five figures are matched to five captions per paper. Constraints improve all five tested backbones over Local. Complete-set accuracy also beats Global on all five, by 7.2 to 53.0 pp, although Molmo2's Global may confuse image and caption indices (App. J.7).

SPIQA inputs fit Phi-4-reasoning-vision-15B's 16,384-token context limit (longest input 12,447 tokens, App. J.9). Only 3 of its 166 outputs are unparseable. Even crediting those as correct gives Global 60.6% caption accuracy, below local+solve's 89.6% (App. J.7).

On 250 JigShape 4×4 puzzles, Qwen3.8 uses edge queries and a heuristic solver. The strongest tested no-thinking forced-answer Global control gives:

MethodPiece accuracy (%)Exact puzzles (%)
Global, forced answer7.50.0
local+solve93.264.0

Forced answers parse for 245 puzzles yet solve none, so the recovery is not explained by completion failures alone. Shape ambiguity remains, and this new solver does not automate task formulation (App. J.6).

Dear Reviewer km3X,

Thank you once again for your careful and detailed review. The points you raised were precise, and responding to them has strengthened both our argument and the evidence we provide. We are also grateful for the references you pointed us to, which we have now discussed in relation to our own formulation.

We have posted our responses and hope that they resolve your concerns. If anything remains unclear or unconvincing, we would be very glad to discuss it further while the discussion period is open.

Reviewer_KSyLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the assembly deficit across six structural families of vision-language tasks: models can provide useful answers to local sub-problems yet fail to produce the correct global structure. The authors introduce local+solve, a training-free approach that extracts answer-token distribution from a frozen VLM and combines them using task-specific solvers. Experiments across multiple benchmarks demonstrate substantial improvements over direct global prediction. The authors further show that local+solve outputs can serve as high-quality pseudo-labels for self-distillation, substantially improving direct global prediction on the trained task and providing evidence of limited transfer to unseen structural families.

Critical strengths
  1. Substantial performance improvements with practical potential. The proposed approach achieves meaningful accuracy gains across several tasks without requiring additional model training. The analysis of inference costs and shared-prefix KV caching provides a useful initial assessment of deployment trade-offs. Furthermore, self-distillation offers a promising way to retain much of the improvement while reducing reliance on repeated local queries and external solving.

  2. Broad evaluation with extensive supporting details. The experimental design includes component ablations, comparisons across multiple benchmarks, and evaluations using different model backbones and scales. Controls for stronger global inference, prompt sensitivity, and alternative training labels help contextualize the results. The detailed appendices document task formulations, solver choices, prompts, and evaluation protocols, supporting reproducibility and a clearer understanding of the method’s limitations.

Critical weaknesses
  1. Limited conceptual and methodological novelty. The central observation closely resembles the compositionality gap studied by Press et al. (2023), which the authors acknowledge. This work extends that perspective to vision-language tasks and a broader range of structured outputs. However, the local-scoring-plus-solver pipeline largely builds on established decomposition and structured-inference techniques. Its principal contribution therefore lies in the breadth and rigor of the empirical investigation, rather than a substantially new reasoning or optimization method.
Questions for authors

Please refer to weaknesses.

2 comments

Revision marks. In the revised PDF, voNi: red text, KSyL: blue text, km3X: yellow highlight. Changes shared by reviewers combine marks (e.g., voNi and KSyL: purple text). Marks show added or revised explanations.

W1. Conceptual and methodological contribution

We thank the reviewer for recognizing the practical gains, the breadth of the evaluation, and the supporting controls. We agree that our observation relates to the compositionality gap of Press et al. (2023) and that decomposition and structured inference are established, and we do not claim a new solver. What is new is a controlled behavioral diagnosis of recoverable global errors in frozen VLMs, which differs from both:

  1. Relation to the compositionality gap. Press et al. show that language models can answer all sub-questions correctly yet fail the whole question, and narrow this gap with self-ask prompting. In VLMs, we test that condition, correct sub-answers, and a self-ask-style control, and measure recovery, the gain of local+solve over direct Global prediction, even when local judgments are not all correct.

  2. Relation to decomposition plus a solver. We separate the effect of asking each decision separately (Local, without the solver) from the solver's effect on identical scores, and test whether valid outputs alone explain recovery.

(1) Relation to the compositionality gap

  • The gap appears in VLMs. Press et al.'s condition, all sub-answers correct, arises in RecipeQA: when Qwen2.5-VL-7B's own four step answers, kept in one conversation, are all correct, they uniquely determine the candidate, yet its final choice is wrong in 308/465 such cases (App. C.1).

  • Correct sub-answers do not close it. For Qwen3.8 and Gemma 4, we replace their retained answers with a candidate's answers, with no new instruction (App. J.12).

RecipeQA, 929 questions (%)Qwen3.8-27BGemma 4-31B-it
Direct choice, no answers67.958.7
Correct candidate's answers inserted76.761.7
Same answers assembled outside the model100100
Wrong candidate's answers: that order chosen13.723.0

With wrong answers, most choices stay as with correct ones (86.3% and 79.5%). The final choice thus depends only weakly on the sub-answers in context, which assembly outside the model uses fully.

  • A self-ask-style control does not close it. A single-pass, self-ask-style decomposition (Press et al.) with self-chosen or prescribed sub-questions and a 768-token cap leaves 7B 25 to 34 pp below local+solve on capacitated assignment (App. C.3).

  • Recovery from imperfect local judgments. With the same images, step texts, and candidates and no thinking, step answers plus assembly beat the most accurate tested direct method, which favors direct prediction, by 16.9, 5.2, and 1.3 pp in complete-order accuracy for Qwen3.8, Gemma 4, and GLM-4.6V, significantly for the first two (App. J.2). With Qwen3.8 on all 1,375 figure–caption links of 166 SPIQA papers, Local links every pair correctly in only 41.0% of papers, yet local+solve does so in 75.9% on the same scores, versus 60.2% for Global (App. J.7).

(2) Relation to decomposition plus a solver

ComparisonWhat it measures
Global vs LocalDifference associated with separate local queries
Local vs local+solveAdditional solver effect on identical scores
Global vs local+solveRecovery over direct global prediction
  • Separate queries matter most for 7B. In six of seven Qwen2.5-VL-7B settings, the Global-to-Local gain exceeds the Local-to-local+solve gain (Sec. 4.3, Table 1), which a solver-only view misses.

  • The solver adds on newer backbones. On Qwen3.8, Gemma 4, and GLM-4.6V, it improves Local in six of seven settings for each model (App. J.1).

  • A solver gain is not recovery. In Qwen3.8 recipe alignment, the solver raises Local from 69.7% to 84.5%, but Global reaches 88.5% (Tables 31–32).

  • The solver helps most when choices conflict. Phi-4-reasoning-vision-15B's parent choices contain cycles in 26 of 84 outlines, and enforcing a forest raises parent accuracy over Local by 2.6 pp (App. J.9).

  • Valid outputs alone are not enough. On ScreenSpot, valid one-to-one decoding gives 7B Global 68.2% (unconstrained: 68.8%), versus 95.3% for local+solve (Table 22). Replacing only invalid or infeasible Global answers with local+solve still trails local+solve in 41 of 52 backbone–setting pairs (App. J.11).

Training direct prediction

Training Qwen2.5-VL-7B on assembled labels raises held-out RecipeQA Global accuracy from 38.4% to 93.0%, near frozen local+solve's 93.3% (Sec. 4.4).

Thus, without thinking, correct sub-answers and self-ask-style decomposition leave the gap open, while assembly closes it and recovers errors from imperfect local judgments, with separable contributions from local queries and the solver.

Dear Reviewer KSyL,

Thank you once again for your careful and detailed review. The points you raised were precise, and responding to them has strengthened both our argument and the evidence we provide. We are also grateful for the references you pointed us to, which we have now discussed in relation to our own formulation.

We have posted our responses and hope that they resolve your concerns. If anything remains unclear or unconvincing, we would be very glad to discuss it further while the discussion period is open.

Reviewer_voNiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper investigates an assembly deficit that VLMs can make accurate local visual decisions yet struggle to produce the corresponding global structure, mainly on image assigning and sorting tasks. It proposes local+solve, which obtains local scores from a frozen VLM and combines them using a task-specific solver. The authors evaluate six structural task families, with additional controls and experiments across model sizes and backbones. Local+solve generally outperforms direct global prediction, and training on its outputs nearly closes the gap on the trained RecipeQA task and produces some gains on unseen structures.

Critical strengths

The paper separates local visual judgments from global structure prediction, exposes the assembly deficit in the studied tasks, and suggests a simple but effective direction for improvement.

The authors conduct extensive experiments. The work test multiple benchmarks, model sizes and backbones. The consistent gains from local+solve make the identified gap both measurable and practically actionable.

Critical weaknesses

Limited applicability and scalability. Local+solve requires predefined local queries, feasible structures, and task-specific algorithm solvers, which may limit its practical use beyond the studied problems. Experiments involves screenshots, photo or page sequences contain at most 16 images (and usually < 10). without an accuracy–cost analysis as sequence length grows, it is unclear whether the trade-off remains worthwhile in more complex settings.

Limited evidence of generality for stronger models. The systematic results rely mainly on Qwen2.5-VL-7B and short, explicitly constrained visual sequences. Although the paper includes larger and newer models, the gap narrows with scale, and testing on frontier visual reasoning models remains limited. It is therefore unclear whether the observed assembly gap is a general bottleneck or is amplified by the chosen model and task settings.

Insufficient differentiation from prior work. TempVS[1] already examines local visual grounding alongside global temporal reasoning, while Sort Story[2] derives a global ordering from pairwise judgments. The paper should clarify what its diagnostic and method add beyond a systematic evaluation across more task types.

Unclear mechanism behind the pseudo-label transfer result. In the cross-structure training experiment, local+solve pseudo-labels yield better transfer than ground-truth labels, but their accuracy on that training set and their disagreement with the ground truth are not reported. An analysis of those disagreements and of why they improve transfer is needed to support the interpretation of this result.

[1] Song, Y., Du, Y., Paperno, D., & Gatt, A. (2025, July). Burn after reading: Do multimodal large language models truly capture order of events in image sequences?. In Findings of the Association for Computational Linguistics: ACL 2025 (pp. 24316-24342).

[2] Agrawal, H., Chandrasekaran, A., Batra, D., Parikh, D., & Bansal, M. (2016, November). Sort story: Sorting jumbled images and captions into stories. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (pp. 925-931).

Questions for authors

Most of the tasks studied seem easy for humans to complete with little effort. Could you identify application scenarios where local+solve would be more useful and evaluate its effectiveness in those settings?

How do accuracy and inference cost change as the sequence length increases (e.g, 25, 50, 100)?

Does the assembly gap persist across the six task families when evaluated systematically on newer, stronger vision-language reasoning models?

How does this work differ from TempVS and Sort Story beyond covering more task types?

In the cross-structure training experiment, how often do the local+solve pseudo-labels differ from the ground truth, and why might they transfer better?

7 comments

Revision marks. In the revised PDF, voNi: red text, KSyL: blue text, km3X: yellow highlight. Changes shared by reviewers combine marks (e.g., voNi and KSyL: purple text). Marks show added or revised explanations.

We appreciate the reviewer's careful feedback. It motivated additional tests and helped us clarify the scope of our findings.

Q1. Application scenarios

Figure–caption linking and GUI interaction are useful applications: each decision may be easy, but many must stay mutually consistent, so one direct error breaks the whole structure. Our diagnostic measures recovery, the accuracy local+solve gains over Global on the same task instances, even when individual local choices are wrong or conflict. Scientific-document indexing. We link figures to their captions in 166 curated SPIQA (Pramanick et al., 2024) papers, with 5–26 included pairs per paper and 1,375 links in total. With Qwen3.8-27B, local+solve correctly links every included pair in 75.9% of papers (Global: 60.2%, Local: 41.0%, App. J.7).

GUI interaction. On MiniWoB++ click-checkboxes-soft, Qwen2.5-VL-7B reaches 68.5% task success with local+solve (Global: 18.5%) over 200 independently drawn episodes per method. The model sees the screenshot and the page's HTML, its assignments are executed as browser clicks, and the environment grades task success (App. D.1). Long-document indexing is covered in Q2.

Q2. Length, accuracy and cost

From 25 to 100 pages, Global page accuracy drops by 12.2 percentage points (pp), versus 2.6 pp for local+solve, which also uses less runtime and memory at all three lengths. We evaluate Qwen3.8 on assigning each page to its document section, using 25-, 50-, and 100-page windows that share the same starting page in each of 46 documents. Benchmark details are in App. J.8.

Per window (Global → local+solve)25 pages50 pages100 pages
Page accuracy (%)94.5 → 94.487.0 → 94.382.3 → 91.8
Time (s)15.8 → 12.533.9 → 24.176.0 → 48.3
Peak GPU memory (GiB)63.9 → 51.776.6 → 51.7101.9 → 51.7

Time includes the solver. Local calls read slightly more input tokens in total but generate none, scoring answer options instead, whereas Global generates 394 tokens on average over one long 100-page input. Longer windows also contain more sections to choose from (3.3 on average at 25 pages and 5.8 at 100 pages), so the changes reflect both more pages and more sections. This saving holds for page-to-section alignment, which needs one local query per page. Families that compare item pairs, such as ordering, need quadratically many queries, and their logged runtime is 3 to 13 times that of Global (App. A.5).

W1. Predefined components

Our approach applies to tasks where local decisions and structural constraints can be explicitly specified. Existing formulations can be reused, but tasks with different structures may require new query designs and solvers. SPIQA reuses one-to-one assignment for figure–caption linking. For the JigShape 4×4 jigsaw-puzzle benchmark (Li et al., 2026), we design edge-type queries and a heuristic placement search. On 250 puzzles, Qwen3.8 reaches 93.2% piece accuracy and 64.0% exact accuracy with local+solve (greedy Local: 66.5% piece accuracy). The strongest tested no-thinking Global control, with a forced final answer, reaches 7.5% piece accuracy and solves none (App. J.6).

Selecting a prepared solver. Qwen2.5-VL-32B and 72B select the intended solver from a fixed six-solver library with 100.0% and 99.3% accuracy using paraphrased task descriptions (App. A.4). This suggests that, for tasks whose components can be specified in advance, choosing the solver can already be automated.

Q3. Stronger models across six families

Partly. Stronger models narrow the gap on the seven controlled settings, but it persists where direct assembly is hard, as in JigShape 4×4 puzzles, where even GPT-6 Sol needs high reasoning to reach its own local+solve (below). We compare all six families in seven settings with thinking disabled, and test thinking on selected GUI, recipe, document, and puzzle tasks.

Seven controlled settings. With thinking disabled, local+solve is more accurate than the strongest tested direct Global control, chosen per setting to favor Global, in 4, 7, and 5 of seven settings for Qwen3.8-27B, Gemma 4-31B-it, and GLM-4.6V, and Global in the rest (App. J.1). The solver improves Local in six of seven settings for each model. GPT-6 Sol's no-thinking Global matches or exceeds Qwen3.8's local+solve in all seven settings and reaches 98.7% page accuracy on the 100-page documents of Q2 (App. J.13).

Global is not limited by greedy decoding: for Qwen2.5-VL-7B, majority votes over five sampled answers, or over as many samples as local queries, stay well below local+solve (Table 7).

GUI and RecipeQA. On 222 GUI screens, Qwen3.8 Global stays below no-thinking local+solve (91.7%) in point estimate at every tested thinking budget: 86.0% at about 4.2 times the runtime, and at best 90.9% at about 28 times (App. J.5). On same-input RecipeQA it reaches 85.6% versus 84.8%, at about 47.9 times the runtime (App. J.10). Gemma 4 stays below on recipes at every budget (best 65.3% versus 83.3%) and nearly closes document alignment (best 91.3% versus 92.7%, at 4.8 times the runtime, App. J.5).

JigShape 4×4. We use it as a stress test because direct assembly is hard even for frontier models: in the benchmark paper, only GPT-5.5 of five frontier models beats random placement. On 100 puzzles, thinking Qwen3.8 Global reaches 51.9%, whereas no-thinking local+solve reaches 93.3% from 64 edge-type queries per puzzle (App. J.6).

GPT-6 Sol on JigShape. Sol's local+solve runs with less information than on open models: its API returns no token probabilities with reasoning, and without it only the top probability for 97.8% of edge queries, so it uses no reasoning and a pre-specified rule fills in the missing probabilities. Even so, on all 250 puzzles it reaches 94.3% piece accuracy, versus 14.6%, 83.7%, and 94.2% for Sol's Global without, with low, and with high reasoning. High-reasoning Global solves more puzzles exactly (187 versus 170, not significant) at 1.70 times the list-price API cost of local+solve (low reasoning: 0.81 times, App. J.13).

Q4. Beyond TempVS and Sort Story

TempVS analyzes temporal reasoning conditioned on successful grounding, and Sort Story learns pairwise order scores and optimizes complete orderings. We instead measure, on the same task instances for a frozen VLM, how much global error its own local judgments recover, whether correct intermediate answers are used, and what the solver adds on the same scores.

Same-input comparison. Fixing RecipeQA images, step texts and candidate orders, Qwen3.8 reaches 84.8% complete-order accuracy by assembling step answers (direct selection: 67.9%, whole-order scoring: 62.5%, Table 3).

Correct answers versus final use. We put local questions and answers in the conversation, then ask for the final order. Four correct answers uniquely determine the correct candidate order, yet Qwen2.5-VL-7B fails in 308/465 such cases when questions are asked sequentially (App. C.1). Even with correct answers inserted, Qwen3.8 and Gemma are right in only 76.7% and 61.7% of questions (assembly outside the model: 100%), and with a wrong candidate's answers they select that order in just 13.7% and 23.0%. Such failures did not occur when all retained choices were correct on GUI (both models) or SPIQA (Qwen3.8), but there the solver sees full scores while the model keeps only chosen letters (App. J.12).

Sort Story's protocol. Under Sort Story's VIST protocol, Qwen2.5-VL-7B gains 3.5 percentage points (pp) over Global: 3.3 pp from separate local queries and only 0.2 pp from exact optimization on identical scores (pair accuracy, App. E.3). On VIST, recovery thus comes mainly from local queries, not from Sort Story-style optimization.

Q5. Pseudo-label disagreement and transfer

About half of the assembled labels differ from ground truth as complete structures, but most individual decisions agree with it.

Label quality. Across the five original training seeds, 154–170 of 336 assembled labels exactly match ground truth, versus 59–66 for Global self-training labels (three seeds). Individual decisions agree with ground truth in 79–84% of assignments, 93–95% of partition pairs, and 75–76% of ordering pairs (App. J.4). In the original training pools, 73 of 99 wrong assignments differ by one swap, 98 of 106 wrong partitions only split ground-truth groups, and wrong orders disagree with ground truth on a median of three of ten pairs. Where assembled labels, the local+solve outputs, disagree with ground truth, the frozen model's local scores favor the label in 86% of partition pairs and 90% of ordering pairs, with smaller score margins than on agreeing pairs (App. J.4). Disagreements thus fall where the model's own evidence is weak. Higher label accuracy may help versus Global self-training.

Transfer controls. The three original seeds that also have a ground-truth control favor assembled labels in accuracy on the unseen RecipeQA alignment task, with the third nearly tied (Table 27). To check that transfer does not depend on the GroundUI-derived training data, we rebuild it from Multimodal-Mind2Web and AndroidControl. Across four runs, unseen-alignment accuracy averages 70.7% with assembled labels, versus 67.0% with ground truth and 59.4% with Global self-training, and assembled labels have higher point accuracy in every run (Table 38). Assembled-label and ground-truth adapters use matched training items and seeds. About half of the 3.7 pp mean gain over ground truth comes from fewer unparseable answers, and the gain is 1.7 pp on recipes both adapters parse (App. J.4). All adapters share the same answer format, parser, and scoring rule. We revised Sec. 4.4 to report this as useful transfer, not as better supervision than ground truth.

Why they may transfer. In the four reconstruction runs, assembled-label adapters agree more with their own local+solve solutions on unseen alignment (+4.6 pp over ground-truth labels). Their gain comes from photos where the frozen model's local+solve is correct (+4.8 pp), with no gain where it is wrong (-0.4 pp), and it is not concentrated in recipes with ambiguous local scores (App. J.4). Assembled labels thus seem to teach consistency with the model's own local evidence, which helps when that evidence is correct. This is an observed association, not an established mechanism.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.