- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces CRAFT (Correctness, gRounding, And Focus Training), an auxiliary verification‑based multimodal Chain‑of‑Thought (CoT) compression framework submitted to ICLR 2027. Different from prior multimodal CoT compression techniques that construct concise reasoning samples or adopt reinforcement‑learning pipelines, CRAFT formulates three discrimination‑oriented auxiliary training tasks targeting reasoning correctness, visual faithfulness, and visual focus. During training, the multimodal model learns to spot flawed logical deductions, image‑unsupported visual claims, and task‑irrelevant visual content; these verification capabilities transfer to regular open‑ended multimodal reasoning. No extra verification, re‑ranking or multi‑round computation is required at inference time. Evaluations across four multimodal QA benchmarks and four backbone models of varying sizes show CRAFT cuts generated token counts substantially, while maintaining or even boosting answer accuracy. Ablation studies validate the necessity of each of the three auxiliary tasks, and the method keeps its advantages under diverse maximum‑token generation budgets。
Critical strengths
- Novel training paradigm for multimodal CoT compression: Instead of directly imitating or generating shortened reasoning chains, CRAFT builds discriminative verification tasks to teach the model to evaluate reasoning quality itself, removing the need for manually curated high‑quality concise CoT datasets, separate reward models, or reinforcement learning pipelines。
- Three‑dimensional fine‑grained supervision: It explicitly addresses three core failure modes of multimodal reasoning: wrong logical inferences, visually hallucinated descriptions ungrounded in input images, and redundant attention toward question‑irrelevant visual information, covering both logical and cross‑modal grounding requirements for high‑quality reasoning。
- Zero additional inference overhead: All verification supervision is only applied in the fine‑tuning phase. At test time, the model directly outputs reasoning and answers with standard generation, without extra verification passes or reranking steps, making it practical for real‑world deployment。
- Consistent empirical improvements across diverse settings: Experiments over four benchmarks and multiple model scales demonstrate that CRAFT reduces output token usage from 10.7% to 70.8% while preserving or improving answer accuracy in most scenarios; ablation results confirm each component contributes positively, and performance gains hold across different token generation budgets。
Critical weaknesses
- This paper doesn't have a related work section, and the authors don't clearly discuss the related works and the difference between the proposed method and other concurrent works. This makes the reviewers hard to judge the novelty of the work.
- No state-of-the-art methods are included for comparison in the experiments. Only simple baselines like SFT-CoT or prompt-based compressing are included. The authors should include state-of-the-art methods for comparison.
- More benchmarks are needed. The authors only use 4 benchmarks in the experiments, which can't fully reflect the effects of the proposed method.
- Key implementation details are needed. As the authors should train the proposed method based on the proposed losses, they should give the training dataset, training strategy and other necessary details. Besides, as the proposed method relies on additional datasets for tuning, it's maybe unfair to directly compare the proposed method against other untrained methods.
- The technical part of the paper may have some questions. The authors propose three different losses, targeting reasoning correctness, visual faithfulness, and visual focus to regulate the model. However, as the main goal of the paper is to reduce inference length, ensuring reasoning correctness, visual faithfulness, and visual focus doesn't guarantee the reasoning chain can be squeezed. Thus this is a main concern for the method design.
- In eq.5, the equation is unclear for me. Do the authors try to directly let the model predict the target textual feedback determined by the rules above in an auto-regressive manner like normal LLMs? However, i don't quite understand why this manner can be translated to short reasoning chains, as they are not directly related to the previous CoT chains. Besides, as the authors have defined numerical values as the feedback, why not use RL for regulation?
Questions for authors
See above