acceptodds
Withdrawn from ICLR 2027

Seeing and Solving Are Not Enough for Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) answer visual questions by combining visual information extraction with downstream problem solving. We investigate a fundamental question: Does an incorrect answer necessarily reflect a failure in visual extraction or problem solving? A model may succeed at both abilities when tested separately yet still fail on the original multimodal question, a distinction that overall answer accuracy cannot reveal. To study this, we perform a question-level empirical analysis across multiple VLMs and visual domains. We define an exactly scorable *task state* (i.e., the visual information sufficient to solve a question) and use it to test whether the same model can extract the required state, solve the question from the ground-truth state, and answer the original multimodal question. We find that *composition failures*, where extraction and solving both succeed but direct answering fails, account for 17.7–75.6% of direct-answering errors across multiple VLMs and datasets. To address this failure mode, we introduce a simple yet effective method, termed *State Realization Tuning* (SRT). SRT fine-tunes LoRA adapters attached to the language-model layers while keeping the pretrained VLM weights frozen. It trains the model to output the ground-truth task state before the final answer in a single autoregressive response. SRT improves over standard supervised fine-tuning by 1.7–14.1 percentage points and repairs 92.5–98.1% of diagnosed composition failures. A single LoRA adapter trained with SRT also improves performance across substantially different task-state structures. Our work shows that having both visual extraction and problem-solving capabilities does not guarantee correct multimodal answering. Requiring the model to first output the visual information needed to solve the question can help bridge this gap.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, multimodal reasoning, visual question answering, compositionality, error diagnosis, task states, composition failures, State Realization Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZTvROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a behavioral failure of vision-language models: the same model can correctly extract the visual information required for a question and separately solve the question given that information, yet fail to answer the original image-based question. The authors construct exactly scorable task states from dataset metadata and identify these composition failures through three interfaces—Extract, Solve, and Direct—evaluated on the same question. They introduce two metrics, Composition Failure Share (CFS) and Composition Failure Rate (CFR). They then propose State Realization Tuning (SRT), which uses LoRA adapters in the language-model layers to supervise task-state generation followed by the final answer within a single autoregressive response. Experiments cover questions constructed from ChartQA and CLEVR, together with four additional task structures.

Critical strengths
  1. The question-level, exactly verifiable diagnostic design is the paper’s strongest contribution. Jointly evaluating the three interfaces on the same question avoids inferring a composition deficit directly from differences in aggregate accuracy. Task states have independent reference labels, while CFS and CFR distinguish the composition of errors from conditional failure probability. Appendix B examines answer formatting, prompt paraphrases, state ordering, and Solve with the image retained, increasing confidence in the observed phenomenon.

  2. SRT delivers consistent improvements in the evaluated settings and includes a meaningful ordering control. Table 2 reports gains across all six model–domain combinations, with three training seeds for ChartQA. In particular, Answer-to-State uses the same state labels but reverses the output order and still underperforms SRT, supporting the value of making the state available before answering. Content-based rescoring also shows that the main improvements cannot simply be attributed to better output-format compliance.

Critical weaknesses
  1. The key comparisons do not sufficiently establish SRT’s additional value over actual explicit step-by-step reasoning.
  • The “Zero-shot State-first” baseline in Appendix D asks the model to identify the relevant values but explicitly prohibits reporting intermediate values. It therefore does not provide the visible state prefix on which SRT relies. The authors mention preliminary experiments with two-field outputs but do not report complete results because of formatting and generation-length issues. InternVL’s 78.8% invalid-output rate in Table 16 further limits the interpretability of this comparison. Table 4 does include a genuine Extract→Solve baseline, but it compares an untuned two-stage model against SRT trained with additional state supervision. The existing controls support the importance of output ordering, but they do not answer how much of the same improvement could be obtained by directly prompting the model to output the state before answering, or by conventional supervision on visual facts or scratchpads, under adequate generation budgets and consistent content-based scoring. Given the established precedent for explicit intermediate generation, this gap directly affects the assessment of the methodological contribution.
  • Moreover, a high CF Repair rate does not establish that the overall improvement primarily comes from repairing a specific composition deficit. On CLEVR, Standard SFT already repairs 95.3% and 93.9% of the fixed CF examples, compared with 96.8% and 95.9% for SRT, respectively (Table 3). The paper needs to distinguish general task learning, improved extraction, and improved composition more clearly.
  1. The experimental coverage and additional insights remain insufficient to establish a strong general contribution.
  • The main experiments use reconstructed questions. The ChartQA test set contains 359 images, with three questions—comparison, difference, and sum—generated from each selected value pair (Appendix G.2). This design is appropriate for controlled diagnosis, but the decomposition and state structures are highly regular.
  • Although Section 4.6 introduces four additional state structures, all four are included in training, with 1,000 examples per structure, and the corresponding schema is supplied at test time. This demonstrates that multiple predefined structures can be learned jointly; it does not establish transfer to unseen structures, unseen compositions, or naturally occurring questions.
  • This limitation matters particularly for the contribution’s significance because Compose and Fuse [1] already studies the recognition–reasoning composition bottleneck and two-stage mitigation, while Scratchpads [2] already studies explicit intermediate-state supervision before answer generation. I acknowledge the incremental value of exact question-level diagnosis, but the submission currently lacks either a sufficiently distinctive new finding or broader validation demonstrating the scientific value of this refinement.

[1] Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning

[2] Show Your Work: Scratchpads for Intermediate Computation with Language Models

Questions for authors
  1. Request for a key control experiment: Can you report results for an untuned model that actually generates TASK_STATE → FINAL_ANSWER in a single response, using a sufficient generation budget and consistent answer-content scoring? For the preliminary two-field experiments mentioned in Appendix D, what were the answer accuracy, invalid-output rate, and truncation rate?

  2. Clarification and evidence concerning attribution: Can you report improvements and regressions for Standard SFT and SRT separately on the fixed pre-training extraction-failure, solving-failure, composition-failure, and originally correct subsets? Together with remeasured Extract/Solve performance, what evidence indicates that the gains arise primarily from improved composition rather than improved state extraction?

  3. Clarification of the contribution’s scope: Are there results on operations, question templates, state structures, or naturally occurring questions not included in training? If not, how should “generalization” in Section 4.6 be interpreted, and what substantive new conclusion does exact question-level diagnosis enable beyond existing work on composition bottlenecks?

Reviewer_PqsEOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents State Realization Tuning (SRT) which uses LoRA-style adapters to finetune pretrained multimodal LLMs to mitigate composition failures. The papers shows composition failures in a way that if questions can be decomposed into a perception component and a problem solving component, models may successfully solve each of them separately but not in a single-pass question answering when asked directly. This kind of failures is roughly 17.7–75.6% of direct-answering errors across multiple evaluation datasets according to the analysis in this paper. The intuition of SRT is to train the model to output task state(visual information to solve the question) before the final answer. Experiments show that SRT improves over standard supervised fine-tuning by 1.7–14.1 percentage points and repairs 92.5–98.1% of diagnosed composition failures.

Critical strengths
  1. One critical strength of this paper is the experiment design, which is reasonably systematic in my opinion. While I have some concerns regarding missing baselines (see weakness), the experiment provides detailed metrics in terms of accuracy, failure and error breakdown related to the main problem the paper is solving, i.e, composition error. Additional method-related baselines such as format control and answer-to-state also make the evaluation more compact.
Critical weaknesses
  1. The novelty of this paper is weak in my opinion. The main problem to solve in this paper, composition error, is not new to the community and the paper already cites rich literature of this direction. While solving an existing problem is definitely acceptable, the problem of interest itself does not add much novelty to this paper. Meanwhile, the main intuition behind the approach SRT is also not new. It appears to me as a chain-of-thought SFT with a specific template. The approach is evaluated in a task-specific finetuning setup, which makes the capability of generalization also questionable.

  2. Given the existing work such as multimodal-cot and llava-cot, which follows the same intuition of intermediate supervision, it is not clear to me what extra insight SRT brings to the community. This paper does not compare with any of the existing work, or at minimum, a controlled CoT-SFT baseline in addition to the standard SFT under the same training setup. I have a strong concern on this especially given that existing methods using intermediate supervision typically explore broader data curation and holistic evaluations across multiple benchmarks.

Questions for authors
  1. How does SRT compare with existing intermedia-supervision approaches, such as LLaVA-CoT and Multimodal-CoT? Would it be possible to provide CoT SFT baseline results in addition to the standard SFT baseline?

  2. What advantage does SRT offer over existing approaches that explore broader multimodal reasoning data curation and training? The current approach and evaluation appears to be task-specific to me and I have a large concern regarding the generalization of this approach.

  3. Does SRT really solve the composition failure by improving model's capability of solving composed tasks? Could the improvement mostly come from better visual state extraction? Since the experiments are conducted in a task-specific training setup, it is possible that the improvement basically comes from the model learns better to respond to specific visual inputs instead of improving composition problem solving. Could authors provide further evidence to address this concern?

Reviewer_pe3JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies the issue of composition failures, where vision-language models (VLMs) can perform well on individual component abilities (specifically, Extract and Solve) but perform poorly when answering the original question directly. To address this issue, the authors propose State Realization Tuning (SRT). Built upon standard supervised fine-tuning (SFT), SRT introduces supervision on the task state before the final answer. The experiments cover a broad range of settings and demonstrate that the proposed method consistently outperforms several baselines.

Critical strengths

The main contribution of this paper is the SRT mechanism implemented with LoRA. SRT supervises the model to generate the task state followed by the final answer. During inference, the model predicts the task state itself and then continues generating the final answer. In terms of novelty, this is a simple yet effective approach that outperforms standard SFT.

Critical weaknesses

The paper partitions Direct errors into three disjoint sets, where set S contains only cases in which Solve fails while Extract succeeds. Table 1 reports the results under this formulation, showing substantially higher Solve accuracy than would be expected in practice. Tables 7 and 8 provide some additional statistical results, but the information appears insufficient to reproduce and verify the results reported in Table 1.

The main limitation of this paper is its generalization to novel tasks. It remains unclear how well the proposed method performs when tasks require more intermediate steps or involve intermediate steps that are not explicitly defined. The need to construct training data with ground-truth intermediate-state annotations is also an important consideration.

Questions for authors
  1. Please explain the rationale for partitioning Direct errors into three disjoint sets. Why is set S defined to include only cases where Extract succeeds but Solve fails?
  2. Please provide sufficient statistics to enable the reproduction and verification of the results in Table 1.
  3. In Table 16, under zero-shot state-first prompting, all models perform worse than their respective Direct baselines, and InternVL exhibits a particularly high number of invalid outputs. Have the authors tried revising or refining the prompt to improve instruction compliance and answer accuracy?
  4. The paper evaluates multiple task-state structures but does not directly investigate settings involving multiple sequential, interdependent intermediate states. It would be valuable to evaluate whether SRT can handle such settings, where each intermediate state depends on previous states and guides subsequent reasoning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.