- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies a behavioral failure of vision-language models: the same model can correctly extract the visual information required for a question and separately solve the question given that information, yet fail to answer the original image-based question. The authors construct exactly scorable task states from dataset metadata and identify these composition failures through three interfaces—Extract, Solve, and Direct—evaluated on the same question. They introduce two metrics, Composition Failure Share (CFS) and Composition Failure Rate (CFR). They then propose State Realization Tuning (SRT), which uses LoRA adapters in the language-model layers to supervise task-state generation followed by the final answer within a single autoregressive response. Experiments cover questions constructed from ChartQA and CLEVR, together with four additional task structures.
Critical strengths
-
The question-level, exactly verifiable diagnostic design is the paper’s strongest contribution. Jointly evaluating the three interfaces on the same question avoids inferring a composition deficit directly from differences in aggregate accuracy. Task states have independent reference labels, while CFS and CFR distinguish the composition of errors from conditional failure probability. Appendix B examines answer formatting, prompt paraphrases, state ordering, and Solve with the image retained, increasing confidence in the observed phenomenon.
-
SRT delivers consistent improvements in the evaluated settings and includes a meaningful ordering control. Table 2 reports gains across all six model–domain combinations, with three training seeds for ChartQA. In particular, Answer-to-State uses the same state labels but reverses the output order and still underperforms SRT, supporting the value of making the state available before answering. Content-based rescoring also shows that the main improvements cannot simply be attributed to better output-format compliance.
Critical weaknesses
- The key comparisons do not sufficiently establish SRT’s additional value over actual explicit step-by-step reasoning.
- The “Zero-shot State-first” baseline in Appendix D asks the model to identify the relevant values but explicitly prohibits reporting intermediate values. It therefore does not provide the visible state prefix on which SRT relies. The authors mention preliminary experiments with two-field outputs but do not report complete results because of formatting and generation-length issues. InternVL’s 78.8% invalid-output rate in Table 16 further limits the interpretability of this comparison. Table 4 does include a genuine Extract→Solve baseline, but it compares an untuned two-stage model against SRT trained with additional state supervision. The existing controls support the importance of output ordering, but they do not answer how much of the same improvement could be obtained by directly prompting the model to output the state before answering, or by conventional supervision on visual facts or scratchpads, under adequate generation budgets and consistent content-based scoring. Given the established precedent for explicit intermediate generation, this gap directly affects the assessment of the methodological contribution.
- Moreover, a high CF Repair rate does not establish that the overall improvement primarily comes from repairing a specific composition deficit. On CLEVR, Standard SFT already repairs 95.3% and 93.9% of the fixed CF examples, compared with 96.8% and 95.9% for SRT, respectively (Table 3). The paper needs to distinguish general task learning, improved extraction, and improved composition more clearly.
- The experimental coverage and additional insights remain insufficient to establish a strong general contribution.
- The main experiments use reconstructed questions. The ChartQA test set contains 359 images, with three questions—comparison, difference, and sum—generated from each selected value pair (Appendix G.2). This design is appropriate for controlled diagnosis, but the decomposition and state structures are highly regular.
- Although Section 4.6 introduces four additional state structures, all four are included in training, with 1,000 examples per structure, and the corresponding schema is supplied at test time. This demonstrates that multiple predefined structures can be learned jointly; it does not establish transfer to unseen structures, unseen compositions, or naturally occurring questions.
- This limitation matters particularly for the contribution’s significance because Compose and Fuse [1] already studies the recognition–reasoning composition bottleneck and two-stage mitigation, while Scratchpads [2] already studies explicit intermediate-state supervision before answer generation. I acknowledge the incremental value of exact question-level diagnosis, but the submission currently lacks either a sufficiently distinctive new finding or broader validation demonstrating the scientific value of this refinement.
[1] Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning
[2] Show Your Work: Scratchpads for Intermediate Computation with Language Models
Questions for authors
-
Request for a key control experiment: Can you report results for an untuned model that actually generates TASK_STATE → FINAL_ANSWER in a single response, using a sufficient generation budget and consistent answer-content scoring? For the preliminary two-field experiments mentioned in Appendix D, what were the answer accuracy, invalid-output rate, and truncation rate?
-
Clarification and evidence concerning attribution: Can you report improvements and regressions for Standard SFT and SRT separately on the fixed pre-training extraction-failure, solving-failure, composition-failure, and originally correct subsets? Together with remeasured Extract/Solve performance, what evidence indicates that the gains arise primarily from improved composition rather than improved state extraction?
-
Clarification of the contribution’s scope: Are there results on operations, question templates, state structures, or naturally occurring questions not included in training? If not, how should “generalization” in Section 4.6 be interpreted, and what substantive new conclusion does exact question-level diagnosis enable beyond existing work on composition bottlenecks?