acceptodds
Withdrawn from ICLR 2027

CRAFT: Teaching Vision-Language Reasoners What Makes Reasoning Worth Keeping

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal reasoning models leverage Chain-of-Thought (CoT) reasoning to solve complex visual tasks, yet incorrect deductions, visually ungrounded descriptions, and irrelevant visual attention can degrade reasoning quality and incur unnecessary inference cost. We introduce CRAFT (Correctness, gRounding, And Focus Training), a discrimination-like auxiliary verification framework for multimodal CoT compression. Unlike existing approaches that directly construct, imitate, or optimize shorter reasoning chains, CRAFT shifts the training objective toward reasoning quality assessment. Specifically, it introduces three simple yet complementary auxiliary verification tasks targeting reasoning correctness, visual faithfulness, and visual focus. These tasks supervise the model to identify incorrect deductions, visually unsupported descriptions, and task-irrelevant visual information, thereby strengthening its internal ability to distinguish high-quality reasoning from flawed or redundant content. Through joint training, the learned verification capability transfers to open-ended multimodal reasoning, enabling the model to reduce redundant reasoning while preserving critical visual evidence, without requiring carefully constructed concise CoTs, reward models, or reinforcement learning. Experiments on four public multimodal question-answering benchmarks across multiple model scales show that CRAFT substantially reduces output token usage while maintaining answer accuracy, and further improves accuracy in most settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Reasoning, Efficient Reasoning, Chain-of-Thought Compression, Auxiliary Verification Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JDKXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces CRAFT (Correctness, gRounding, And Focus Training), an auxiliary verification‑based multimodal Chain‑of‑Thought (CoT) compression framework submitted to ICLR 2027. Different from prior multimodal CoT compression techniques that construct concise reasoning samples or adopt reinforcement‑learning pipelines, CRAFT formulates three discrimination‑oriented auxiliary training tasks targeting reasoning correctness, visual faithfulness, and visual focus. During training, the multimodal model learns to spot flawed logical deductions, image‑unsupported visual claims, and task‑irrelevant visual content; these verification capabilities transfer to regular open‑ended multimodal reasoning. No extra verification, re‑ranking or multi‑round computation is required at inference time. Evaluations across four multimodal QA benchmarks and four backbone models of varying sizes show CRAFT cuts generated token counts substantially, while maintaining or even boosting answer accuracy. Ablation studies validate the necessity of each of the three auxiliary tasks, and the method keeps its advantages under diverse maximum‑token generation budgets。

Critical strengths
  1. Novel training paradigm for multimodal CoT compression: Instead of directly imitating or generating shortened reasoning chains, CRAFT builds discriminative verification tasks to teach the model to evaluate reasoning quality itself, removing the need for manually curated high‑quality concise CoT datasets, separate reward models, or reinforcement learning pipelines。
  2. Three‑dimensional fine‑grained supervision: It explicitly addresses three core failure modes of multimodal reasoning: wrong logical inferences, visually hallucinated descriptions ungrounded in input images, and redundant attention toward question‑irrelevant visual information, covering both logical and cross‑modal grounding requirements for high‑quality reasoning。
  3. Zero additional inference overhead: All verification supervision is only applied in the fine‑tuning phase. At test time, the model directly outputs reasoning and answers with standard generation, without extra verification passes or reranking steps, making it practical for real‑world deployment。
  4. Consistent empirical improvements across diverse settings: Experiments over four benchmarks and multiple model scales demonstrate that CRAFT reduces output token usage from 10.7% to 70.8% while preserving or improving answer accuracy in most scenarios; ablation results confirm each component contributes positively, and performance gains hold across different token generation budgets。
Critical weaknesses
  1. This paper doesn't have a related work section, and the authors don't clearly discuss the related works and the difference between the proposed method and other concurrent works. This makes the reviewers hard to judge the novelty of the work.
  2. No state-of-the-art methods are included for comparison in the experiments. Only simple baselines like SFT-CoT or prompt-based compressing are included. The authors should include state-of-the-art methods for comparison.
  3. More benchmarks are needed. The authors only use 4 benchmarks in the experiments, which can't fully reflect the effects of the proposed method.
  4. Key implementation details are needed. As the authors should train the proposed method based on the proposed losses, they should give the training dataset, training strategy and other necessary details. Besides, as the proposed method relies on additional datasets for tuning, it's maybe unfair to directly compare the proposed method against other untrained methods.
  5. The technical part of the paper may have some questions. The authors propose three different losses, targeting reasoning correctness, visual faithfulness, and visual focus to regulate the model. However, as the main goal of the paper is to reduce inference length, ensuring reasoning correctness, visual faithfulness, and visual focus doesn't guarantee the reasoning chain can be squeezed. Thus this is a main concern for the method design.
  6. In eq.5, the equation is unclear for me. Do the authors try to directly let the model predict the target textual feedback determined by the rules above in an auto-regressive manner like normal LLMs? However, i don't quite understand why this manner can be translated to short reasoning chains, as they are not directly related to the previous CoT chains. Besides, as the authors have defined numerical values as the feedback, why not use RL for regulation?
Questions for authors

See above

Reviewer_zvxLOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CRAFT, a framework for efficient multimodal reasoning that combines auxiliary verification tasks for reasoning correctness, visual faithfulness, and visual focus with concise-solution replay during supervised fine-tuning. Verification is used only during training, without additional verification at inference. Experiments across four backbones and four benchmarks show consistent reductions in output tokens and accuracy improvements in most settings.

Critical strengths
  1. The paper aims to solve a problem of reducing redundant multimodal reasoning while retaining necessary visual evidence.

  2. The method is easy to follow through supervised fine-tuning and requires no additional verifier or verification rounds at inference.

  3. The results demonstrate consistent token savings across all evaluated settings. On MathVista, all four backbones improve both accuracy and efficiency.

Critical weaknesses

1.Evaluated benchmarks are limited. The authors conduct evaluation most on science-based and math-based benchmarks except for MMBench. However, both Qwen3-VL model sizes show accuracy drops on MMBench. Additional results on some common general visual benchmark such as CountBench, CV-Bench, and BLINK would help assess whether CRAFT preserves broader multimodal capabilities.

  1. Insufficient error analysis. Token counts consistently decrease, but accuracy gains vary across models and benchmarks, with some settings showing degradation. The paper should investigate whether compression removes necessary evidence or reasoning steps and explain when it helps or hurts performance.

  2. Unclear contribution of verification training. CRAFT also uses substantial short-CoT solver-replay data. A solver-only baseline using identical solver data and a matched training budget is needed to distinguish gains from verification learning versus concise-solution imitation.

  3. Inconsistent generation settings. Appendix A.2 specifies a maximum generation length of 4096 tokens, but Table 1 reports average outputs exceeding 6,000 tokens. Please clarify the decoding settings and token-counting procedure.

  4. Missing annotation and evaluation details. Please identify the models used for faithfulness annotation, anchor extraction, and focus annotation, and explain how the three reasoning-quality metrics are computed. Evidence of annotation reliability is also needed.

Questions for authors

see the above weakness.

Reviewer_uK4XOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether auxiliary verification training can improve the efficiency of multimodal reasoning. It introduces CRAFT, which combines verification tasks for reasoning correctness, visual faithfulness, and visual focus with SFT on concise solution trajectories. Experiments across four visual reasoning benchmarks and multiple VLM backbones report reduced output token counts while maintaining or improving answer accuracy in most settings.

Critical strengths
  1. The paper studies an interesting question: whether learning to verify multimodal reasoning can improve the model’s ability to generate accurate and efficient solutions. In particular, understanding whether verification ability transfers to reducing redundant reasoning while preserving necessary visual evidence is a worthwhile research direction.
Critical weaknesses
  1. The three verification tasks are intuitively reasonable, but their selection is not sufficiently justified in the context of reasoning efficiency. The paper motivates reasoning correctness, visual faithfulness, and visual focus as aspects of reasoning quality. However, it is less clear why verifying these particular aspects should help reduce unnecessary reasoning. I would like to see a stronger justification for this choice.

  2. I have concerns about the connection drawn between verification ability and reasoning efficiency. In lines 268 and 269, the paper states that verification training helps the model distinguish good reasoning from flawed or redundant content, “thereby promoting more efficient reasoning generation.” The paper does not adequately explain why this ability should transfer to generating shorter solutions. The empirical improvements are encouraging, but they do not by themselves establish the proposed explanation.

  3. The overall training objective presented in the main text does not fully reflect the actual training procedure. Appendix A.1.1 on page 12 describes generating reasoning trajectories with prompts that explicitly encourage concise reasoning. Appendix A.1.4 on page 13 then introduces a solver replay objective using these trajectories. However, Equation (7) only includes the three verification losses. This additional short CoT SFT component should be explained in the main method section and included in the overall objective.

  4. The source of the efficiency gains remains unclear. Since training also includes SFT on short CoTs, some of the gains could come from directly learning to generate shorter solutions. The paper includes a short CoT SFT baseline, but does not make clear whether it uses the same solver data and a comparable training budget as CRAFT. Without this information and a controlled comparison, it is difficult to determine how much of the improvement comes from verification training.

  5. The verification labels are not sufficiently validated. For visual faithfulness and visual focus, the appendix describes using model judgments but does not identify the evaluator models or report how the resulting labels were checked. LLM judgments can be unreliable, so the paper should specify the models used and provide evidence that their labels are accurate. Providing annotation prompts alone is not enough to establish the quality of the supervision.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.