acceptodds
Under review as a conference paper at ICLR 2027

Distilling Visual Reasoning into Text Space

Abstract

Large Vision–Language Models (LVLMs) have shown strong promise for multimodal reasoning, yet often struggle with tasks requiring concepts beyond what is directly observable in the input image. Existing methods generate intermediate images or latent visual tokens to guide reasoning, but these representations can introduce errors and increasingly interfere with textual reasoning as reasoning progresses. We propose Visual-to-Text Chain-of-Thought Distillation (V2T), a framework that enables LVLMs to internalize visual reasoning without generating intermediate visual representations at inference time. V2T first trains a teacher LVLM using interleaved visual and textual chains of thought, and then uses knowledge distillation to train a student LVLM using the teacher’s logits and cross-entropy supervision from ground-truth textual reasoning. When reasoning images can be mapped to the original image, V2T can additionally distill the teacher’s attention to corresponding regions, while ground-truth bounding boxes can further guide a subsequent reinforcement learning stage. Experiments across multiple multimodal reasoning benchmarks show that V2T consistently outperforms the teacher and existing baselines, improving average accuracy by 14.3% on a held-out set and 2.7% on the broader visual evaluation suite. Moreover, lightweight SFT and substantially reduced RL make V2T up to 42× faster to train than state-of-the-art baselines.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
large vision language models, reasoning, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_26prOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose V2T, a 2-stage method to improve visual reasoning. First, it trains a teacher LVLM on filtered interleaved data. Second, it distills a student that reasons only in text. The student is trained with ground-truth text cross-entropy plus a KL loss to the teacher's logits When a reasoning image is a crop of the original, the teacher's attention is also distilled. An optional GRPO stage adds an attention-IoU grounding reward. Experiments reports clear gain of V2T over teacher and other baselines.

Critical strengths
  1. A simple, practical idea: Distilling visual evidence into text is a clean approach to retain useful multimodal information while reducing inference cost. The attention distillation idea is simple yet shown effective in experiments.
  2. Clear experiment design and gains: Gains are observed in comparison with multiple representative visual reasoning baselines. The training and inference cost reported is substantially lower than other methods.
Critical weaknesses
  1. Fair and complete baselines:

The authors compare against latent reasoning and text-only CoT methods on the Zebra-CoT held-out test set, but the paper does not describe how these baselines were trained. Given most results are lower than the Qwen2.5-VL base model, I suspect the compared methods were not trained on the Zebra-CoT data. A comparison under the same training data budget would make the results considerably more convincing. In addition, the paper would benefit from two simple baselines trained on the same data: (1) SFT on answers only, without reasoning traces, and (2) SFT on text-only CoT, both on same training set. These would help show that the gains come from the proposed method rather than from the additional training data.

In addition, some reported baseline results appear to differ from those in the original papers (e.g., MONET reports 83.25 on V* and LVR reports 81.7 on V*). If this is due to differences in model versions or evaluation settings, it would be helpful for the authors to clarify this in the paper. Addressing these points would strengthen the consistency of the results in Table 2 and the overall empirical claims.

  1. Unfaithful cot of student and generalization ability:

In the demo presented (fig.7,8), the student still outputs text chain of thought without the actual image. While I understand this is inherited from teacher, but the pattern suggest the student model may overfit to answer distribution and lose generalization. It would be helpful if the authors could discuss this behavior and evaluate the student on additional datasets, including out-of-distribution datasets for the same task, to better assess its generalization ability.

Questions for authors

See weakness above.

I have one additional, more minor question. The proposed method relies heavily on attention distillation, which can only operate over the original input image. Visual reasoning problems of this kind, which mainly require grounding in and operating on the input image, are now largely addressed by tool-calling approaches. It is unclear whether the method can generalize to settings where the visual evidence comes from outside the original image, for example generating new image as evidence. I would argue these are the settings where this kind of visual reasoning matters most. Could the authors comment on how the method might extend to such problems and thoughts on visual reasoning pattern?

4 comments

We thank the reviewer for their valuable feedback, recognizing V2T as a simple and practical approach, the effectiveness of attention distillation, and the clear experimental design showing gains over representative visual reasoning baselines at substantially lower training and inference cost.

  1. W1: Fair and complete baselines.

    The reviewer is correct that LVR and MONET are not trained on Zebra-CoT; we evaluate their publicly released checkpoints. These methods are defined by their training pipelines as much as their architectures — MONET trains on 125K interleaved samples plus RL, LVR on 438K Visual-CoT samples — so retraining either on our 23K set would produce a data-starved variant rather than a methodologically equivalent comparison. The released checkpoints represent each method as its authors intended, trained on substantially more data than V2T. This follows standard practice in this area: LVR compares against PixelReasoner and Vision-R1, and MONET against DeepEyes and LVR, in each case without retraining on matched data. Bagel-Zebra is the informative case, since it is trained on Zebra-CoT with 182K samples, roughly 8× ours, and reaches 57.8 against V2T’s 62.8.

    Table: Training budgets for all compared methods. SFT and RL columns give the number of training samples; — indicates the method does not include that stage. V2T trains on the least data and compute of any method compared; even with GRPO it uses fewer GPU-hours than every baseline.

    MethodR1-VLVLAAVL-RethinkerVision-R1LVRMONETBagel-ZebraThymeDeepEyesV2TV2T GRPO
    SFT260K––200K438K125K182K500K–23K23K
    RL10K25.2K39K10K–3.2K–10K47K–37K
    GPU-hours1671671283,3921603301101,4241,5362080

    That these checkpoints fall below Qwen2.5-VL-7B on the Zebra-CoT held-out set reflects the training-data mismatch the reviewer identifies together with the protocol differences discussed below, not a failure to reproduce them.

    Controlled baselines on matched data. We report both baselines the reviewer suggests — Answer-Only SFT, trained on final answers alone, and Text-Only SFT, trained on textual CoT without intermediate images or distillation — from the same base model on the same Zebra-CoT set.

    ModelGraphRPMTetrisVisual SearchAvg.
    Text-Only SFT48.0074.3766.0071.5064.97
    Answer-Only SFT33.5085.4336.5071.5056.73
    V2T61.5081.9081.5077.0075.48

    Reasoning traces matter in-domain: answer-only supervision loses 8.3 points to text-only CoT on the held-out set, though out of domain the two are indistinguishable (58.5 vs. 58.9) and both sit near the base model. V2T exceeds both on both evaluations — by 18.8 and 10.5 points on the held-out set, and 4.3 and 3.9 on the broader suite — so the gains come from the distillation signal rather than from exposure to the training set.

    The exception is RPM, where answer-only supervision scores highest (85.4). RPM patterns are recoverable by direct visual pattern matching, so answer-only training can exploit this shortcut rather than reasoning through the transformation.

    On differing baseline numbers. Reported scores for the same model vary across papers because protocols differ. On V*Bench, MONET reports 83.25 for MONET and LVR reports 81.7 for LVR, while Qwen2.5-VL-7B is reported at 78.5 by LVR, 76.4 by MONET, and 75.9 by us. LVR uses exact matching; MONET uses exact matching primarily, invoking DeepSeek-V3.1 or Gemini-2.5-Pro only when ambiguous. These diverge in both directions — exact matching rejects semantically correct answers that differ in formatting, common for long chains of thought, and accepts superficial string matches regardless of whether the reasoning supports them. Published numbers from different papers are therefore not directly comparable.

    What matters for our claims is internal consistency. Every model in Tables 1, 2, and 4 is evaluated with the same framework, the same Qwen3-235B-A22B-Instruct-2507-FP8 judge, and the same settings, under a shared 32,768-token budget chosen to accommodate longer tool-calling contexts. For the latent-reasoning baselines we use MONET’s released vLLM implementation, adapted for LVR, with both set to 10 latent steps per MONET’s public implementation. Appendix D verifies judge stability: GPT-4o and our judge agree within 0.5 points across HR-Bench 4K/8K and V*Bench.

  1. W2: Unfaithful CoT of student and generalization ability.

    Thank you for raising this. We read the concern as two questions: whether the inherited phrasing indicates the student absorbed surface form rather than capability, and whether it generalizes.

    On the phrasing: it is inherited vocabulary, not an empty template. No image is generated, but the operation of the phrasing names — concentrating on a sub-region before reporting what is there — is what the student does. Attention IoU rises from 0.191 for the base model to 0.269 for V2T (Table 5), so the regions the trace refers to are the regions the student attends to, and in Figs. 7 and 8 it resolves cases the teacher gets wrong. Inheriting a format while exceeding the teacher’s accuracy is not the signature of imitation.

    On generalization: this is already tested. V2T is trained on Zebra-CoT and evaluated on HR-Bench 4K/8K, V*Bench, BLINK, and POPE, none of which share Zebra-CoT’s distribution or task format, and it improves over the base model on every one (Table 2). A model overfit to the teacher’s answer distribution would not transfer across four unrelated benchmarks.

    We additionally test whether the framework itself transfers beyond visual reasoning. This is a separate training run rather than the same checkpoint: we apply V2T to ViRC and evaluate on mathematical reasoning benchmarks (Table 3 in the revision).

    ModelPost Training RecipePost Training GPU hoursMathVista GeometryMathVista AlgebraicMathVista Math WordMathVista Geometry ProblemLogicVistaAvg.
    Qwen2.5-VL-7B base--70.369.474.270.243.665.5
    Text-only SFTSFT4074.974.073.776.043.268.4
    Interleaved SFTSFT5073.673.771.075.046.367.9
    V2TSFT5074.175.172.076.449.069.3
    V2T + GRPOSFT + RL13078.777.283.981.246.173.4
    Vision-R1-7BSFT + RL339276.274.776.378.445.470.2
    VL-Rethinker-7BRL12876.274.480.678.445.971.1
    VLAA-Thinker-7BRL16768.267.683.968.846.367.0
    MONET-7BSFT + RL33058.258.777.457.239.658.2
    LVR-7BSFT16066.564.175.367.835.361.8

    In this setting V2T is trained on ViRC [1]; V2T + GRPO is initialized from that checkpoint and trained further with GRPO on ViRL [2]. These test whether the V2T framework transfers to another domain, and are not evaluations of the Zebra-CoT checkpoint used in the main experiments.

    The framework transfers. Among SFT-only models, V2T improves over the base model (65.5 → 69.3) and over Text-Only SFT trained on the same data (68.4 → 69.3), with its clearest gain on LogicVista (49.0, the highest of any model listed). Vision-R1 and VL-Rethinker both use RL, so the matched comparison for them is V2T + GRPO, which reaches 73.4 against their 70.2 and 71.1 — at 130 GPU-hours against Vision-R1’s 3,392. These baselines are likewise trained on mathematical reasoning data. Neither MathVista nor LogicVista appears in ViRC or ViRL training, so these are out-of-distribution evaluations within the mathematics setting.

    [1] Wang et al., ViRC: Enhancing Visual Interleaved Mathematical CoT with Reason Chunking, CVPR 2026, arXiv:2512.14654.

    [2] Wang et al., From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoning, 2025, arXiv:2511.23031.

  1. Q1: Generalizing beyond the original input image.

    Thank you for raising this. We would first clarify that V2T neither relies primarily on attention distillation nor is limited to reasoning images that are crops of the input. Nearly 30% of our filtered Zebra-CoT set consists of newly constructed visual states — generated molecular structures in Chemistry (12.5%) and future board positions in Chess (15.9%, Table 9). These cannot be mapped back to the input, and their reasoning signal transfers through logit distillation alone.

    Attention distillation is optional, applied only where a spatial mapping exists. Logit distillation alone raises the average from 58.9 to 61.5; attention alone reaches 59.4; together 62.8 (full ablation in our response to Reviewer 3VAQ’s W2). The teacher’s reasoning behavior is the primary signal. For settings involving newly constructed evidence, V2T simply omits the attention term — no change to the framework. Logit distillation is agnostic to how the reasoning image was produced, so the binding constraint on extending to such settings is the availability of interleaved training trajectories, not the method.

    On tool-calling: we would push back on the premise that it largely solves this class of problem. The tool-based baselines are the direct evidence — Thyme and DeepEyes reach 58.0 and 59.1 against V2T’s 62.8 on the broader suite, at 45× inference cost, and BAGEL-Zebra-7B, which generates new images via a diffusion head rather than cropping, reaches 57.8 at 30×. Fig. 2 suggests why: each visual state inserted into the reasoning interferes with the text that follows, and the cost compounds as states accumulate, so producing more visual evidence at inference does not straightforwardly help.

    V2T can build on these methods rather than compete with them. Tool-generated images are intermediate visual states of exactly the kind V2T distills, so a teacher trained on tool-call trajectories would let a student inherit the reasoning those tools enable without invoking them at inference. The Visual Search portion of Zebra-CoT is already this case, since its reasoning images are crops of the input — the same artifacts a cropping tool produces.

    On visual reasoning patterns more broadly: the fact that distillation works at all suggests much of what a reasoning image contributes is recoverable in text, with the image serving during training to fix which properties matter rather than supplying something text cannot carry. The open question, in our view, is when carrying a visual state beats describing it. Existing methods assume the answer is nearly always; our results suggest otherwise.

Dear Reviewer,

Thank you again for your feedback. We address your concerns about baseline fairness and generalization by running the baselines you suggested, reporting the student’s performance on out-of-distribution visual tasks, and testing V2T separately in mathematics.

Please let us know if any questions remain before the discussion deadline. If our response addresses your concerns, we would appreciate it if you would reconsider your score.

Best regards,
The Authors

Reviewer_9YrFOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Visual-to-Text Chain-of-Thought Distillation (V2T), a new framework that enables VLMs to visually reason without explicit visual representations during inference. The proposed V2T first trains a teacher model with interleaved visual and textual CoT and then distill the visual reasoning ability into a student model with only textual reasoning. Extensive experiments and ablation studies demonstrate the state-of-the-art performance of V2T.

Critical strengths
  1. To solve the interference caused by existing visual reasoning method with explicit images or latent visual tokens, the proposed V2T internalizes visual reasoning into pure textual CoT without intermediate visual representations. To achieve this, V2T distills a teacher model trained with interleaved multimodal data to a student model using auxiliary attention KL loss and visual grounding RL reward. Experiments demonstrate V2T achieves SOTA performance with less training and inference costs.
  2. This paper is clear and easy to follow.
Critical weaknesses

None

Questions for authors

How does V2T perform on other base models, e.g., Qwen3-VL, Qwen3.5, and InternVL3.5? Will the performance of V2T degrade as the model parameter size increases?

2 comments

We thank the reviewer for the positive assessment, and for recognizing that V2T avoids interference from intermediate visual representations while achieving state-of-the-art performance at lower cost, as well as the clarity of the paper.

  1. Q1: Other base models and model scale.

    We thank the reviewer for their valuable and encouraging feedback.

    To evaluate V2T on a different model family and at a larger scale, we are training V2T on InternVL3.5-14B and Qwen3-VL, and will report results if they complete within the discussion period. We do not expect the mechanism to be scale-specific: the interference arises from how the context is composed rather than from model capacity, and the inference-cost advantage follows from generating no intermediate visual states, which holds at any size.

We trained V2T on InternVL3.5-14B, a much bigger model from a different family, during the discussion period, using the same recipe without modification. V2T improves over the base model (58.1 vs. 57.4) and over the interleaved teacher (54.1), reproducing at 14B the student-exceeds-teacher pattern from our 7B results, and matching or exceeding base on nine of twelve categories. We see two likely reasons the margin over base is modest, neither specific to V2T. First, compute limits required reduced image resolution for training and evaluation. All three models share this setting, so the comparison is controlled, but it places the base model well below its reported numbers (HR-Bench 4K/8K at 58.9/53.6 vs. 66.6/61.5 with full resolution) and removes much of the high-resolution perception advantage the larger model brings. These results therefore likely understate V2T's gains at full resolution. Second, Zebra-CoT's coverage is narrow for a stronger base model: half the filtered set is Visual Search and Maze, both spatial-search tasks. Unlike at 7B, interleaved training on it hurts the 14B teacher (54.1 vs. 57.4 base), yet V2T recovers the teacher's loss and surpasses the base. Recovery is incomplete on three categories. Two, Forensic Detection and Visual Correspondence, are fine-grained comparison tasks the mixture does not cover. The third, Jigsaw, is in the mixture, but the base model is already strong there (77.3), so Zebra-CoT's Jigsaw data likely pulls it down rather than up: interleaved training drops it to 52.7, and V2T recovers to 62.7. The framework transferred to a larger model from a different family without modification, and we expect the margin to widen at full resolution and with training data matched to the base model's level.

ModelHR-4KHR-8KV*ForensicFunc. Corr.IQ TestJigsawMulti-viewObj. Loc.Vis. Corr.Vis. Sim.POPEAvg
InternVL3.5-14B (base)58.953.664.440.928.522.777.345.153.373.886.783.157.4
Interleaved teacher59.953.163.924.227.724.752.751.150.265.189.686.854.1
V2T (ours)60.853.667.536.430.832.062.751.154.970.391.185.858.1
Reviewer_3VAQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes V2T, a framework that distills a teacher trained on interleaved image–text reasoning into a student that generates textual reasoning without intermediate images. Training combines textual supervision, logit distillation, optional attention distillation, and subsequent GRPO refinement. Experiments report improved visual reasoning performance while avoiding intermediate image generation at inference.

Critical strengths
  1. The paper explores whether intermediate visual information can benefit learning without remaining part of the inference pipeline. This is a useful direction for reducing the overhead and potential error propagation associated with generating intermediate images in VLM reasoning.

  2. V2T provides a coherent teacher–student framework that makes the information asymmetry explicit through privileged-information distillation. The reported experiments demonstrate performance gains without requiring intermediate images at inference.

Critical weaknesses
  1. Limited novelty. Distilling a visually-privileged teacher into an information-restricted student with standard KL and cross-entropy losses is conventional privileged-information distillation. The paper does not show what challenge specific to visual reasoning requires a new method, so the contribution reads as a training configuration rather than a new learning mechanism.

  2. Unclear source of gains. In Table 3, V2T without attention distillation scores 57.1, versus 56.2 for Text-Only SFT and 61.8 for the full method, suggesting attention supervision drives most of the improvement. Without a matched cross-entropy-plus-attention baseline lacking logit distillation, the necessity of the interleaved teacher remains unclear.

  3. Weak evidence for internalized visual reasoning. The teacher is trained with intermediate images but evaluated without them, so student gains may reflect train-inference alignment. Higher attention IoU shows better localization, not transfer of visual reasoning operations. Context length, token positions, and effective image resolution remain confounds in the interference analyses.

Questions for authors
  1. What distinguishes V2T from a general privileged-information distillation framework? Please identify the technical contribution beyond supplying intermediate images to the teacher. Would a teacher conditioned on relevant crops, without stepwise interleaved visual reasoning, provide similar benefits?

  2. Can the authors isolate the contributions of logit and attention distillation? A matched comparison of CE, CE + KD, CE + Attention, and CE + KD + Attention would clarify whether teacher prediction matching provides a substantial benefit beyond direct grounding supervision.

  3. How should the student–teacher performance gap be interpreted? Could the authors separately report diagnostic teacher performance with and without ground-truth intermediate images? This would help distinguish knowledge transfer from the benefit of removing the teacher’s training–inference mismatch.

  4. Does the intermediate-image interference persist under controlled context budgets? Comparisons controlling visual-token count, token positions, and effective content resolution would help determine whether the observed degradation is specific to intermediate visual reasoning or reflects broader context-management limitations.

5 comments

We thank the reviewer for their valuable feedback and acknowledging our promising research direction which reduces overhead and error propagation for VLM reasoning.

  1. W1, Q1: Limited novelty.

    We respectfully disagree that our work has limited novelty and answer the points raised by the reviewer one by one below, followed by two new experiments further confirming our claims.

    Visual-reasoning-specific challenges. One of our main contributions is to show that the intermediate visual states interfere with the textual reasoning that follows, and the damage compounds with each state added, not only with generated but also with ground-truth reasoning images (see (i) below). The problem is therefore not only inaccurate generations, but that carrying them in context interferes with text reasoning (see (ii) below). This has gone uncharacterized, and every dominant paradigm is built on the opposite assumption, reproducing these states at inference via tools or latent tokens.

    V2T distillation. Unlike standard PI distillation for text reasoning, where the privileged signal is gone at test time and distillation is the only route, here it can be produced at inference with tool use or generating latent visual states. So choosing to distill instead needs a positive argument. Our interference result supplies it: remove visual conditioning at inference rather than improve it. The gain is both in accuracy and in cost — V2T outperforms all of these methods on the broader suite at 1× inference cost, against 2× for latent-token methods, 30× for unified interleaved generation, and 45× for tool-based ones (Table 2).

    V2T vs conventional PI distillation. The reviewer’s alternative — a teacher conditioned on relevant crops, supplied upfront rather than at their reasoning steps — is the standard PI setup, and it discards what makes this signal useful. Each reasoning image is produced by the preceding steps: an auxiliary line is drawn only once the reasoning calls for it, a board position exists only after a move is chosen. The image is consumed at that step and no other, so the signal is the correspondence between step t’s text and step t’s visual state. Supplying the same images at the beginning preserves the pixels and destroys the alignment (see (iii) below).

    Our novelty. We agree the KL divergence is standard and do not claim otherwise. What is new is the diagnosis and the design that follows from it: use intermediate visual states as step-aligned training supervision, and remove them from inference. A novel loss would address neither. We believe this is more significant than merely proposing a new loss or learning paradigm without addressing the root cause of a problem.

    Three experiments support this. For (i) and (ii), we expand the interference analysis from the paper, increasing the sample size tenfold to 1,500 samples to make the conclusions more general.

    (i) Interference persists with correct images. Even with ground-truth intermediate images, inserting them into the reasoning raises the loss relative to text-only reasoning, and the gap grows from +0.35 to +4.15 as the number of images increases from 3 to 7 (cf. Fig. 2).

    (ii) Interference is content-sensitive, not a context-budget artifact. To rule out that the growing gap reflects the added context length, we use two controls holding the total token count fixed: the reasoning images are removed and the same number of neutral text tokens is placed at the start of the user message, so every reasoning token has at least as many preceding tokens as in the interleaved sequence; and the intermediate images replaced with incorrect ones. The first row repeats (i) for reference; positive values indicate greater interference.

    Condition3 images5 images7 images
    Original images, interleaved − text-only+0.35+2.34+4.15
    Original images, interleaved − text padding+0.48+4.08+5.67
    Incorrect images, interleaved − text-only+3.72+4.49+6.84

    Against the text-padding control the gap stays positive and still grows with image count, so the interference comes from inserting visual states into the reasoning, not from longer context. Incorrect images interfere more at every count, showing the effect also depends on content — a fragility that methods reproducing these states at inference cannot avoid.

  1. W1, Q1: Limited novelty (continued).

    (iii) Static conditioning does not substitute for step alignment. We train a teacher with all reasoning images placed upfront in the prompt — the reviewer’s setup. Where reasoning images are spatially mappable, they are crops of the original image, so this teacher receives the crops the reviewer describes; where they are not, no crop equivalent exists. We distill it with CE and logit distillation on the same data and hyperparameters. Attention distillation cannot be applied here, since it requires the step positions that only interleaving creates, so the comparison is against V2T’s logit-only variant.

    ModelAvg. (10 benchmarks)
    Text-Only SFT (CE)58.9
    CE + Logit KD, static teacher (images upfront)59.2
    CE + Logit KD, interleaved teacher61.5
    V2T (+ attention distillation)62.8

    Between the two logit-distilled students, the static teacher adds 0.3 points over Text-Only SFT while the step-aligned teacher adds 2.6, and only the latter enables attention distillation for a further 1.3. Access to the same visual evidence is not enough; what transfers is its alignment with the reasoning steps.

    Attention distillation is therefore auxiliary: it applies only where a reasoning image maps spatially to the original image, and is itself defined by step alignment, since the teacher’s attention is read at the first text token following each reasoning image. We show next that logit distillation is the main driver, with attention contributing only on top of it.

  2. W2, Q2: Unclear source of gains.

    Correction to Tables 2 and 4 and Figure 6: We found that the V2T ablations were evaluated with a lower maximum image-pixel budget than the base model. After LoRA fine-tuning, we merged the adapters into the base weights and exported standalone checkpoints using LLaMA-Factory, whose generated preprocessing config defaulted to a lower pixel budget (1,003,520 vs. 12,845,056). The evaluation loader automatically picked up this config. We have re-evaluated all affected checkpoints using the correct pixel budget. This issue affects only our LoRA-trained checkpoints exported through LLaMA-Factory. External baselines and the Qwen2.5-VL-7B base model use official checkpoints with default preprocessing and are therefore unaffected. The correction improves performance by over 10 points on average on the first three high-resolution benchmarks, while differences on the remaining seven benchmarks average only 1–2 points. We report the corrected results below, with all other affected results (Tables 2 and 4, Figure 6) updated in blue in the revised paper.

    ModelHR4KHR8KV*Func.VisualObjLocForen.JigsawMultiPOPEAvg.
    Interleaved SFT (teacher)69.865.473.326.961.648.445.562.047.491.859.2
    Text-Only SFT (CE)68.262.472.826.258.749.250.864.044.491.858.9
    CE + Logit KD70.866.273.327.762.850.048.568.754.991.661.5
    CE + Attention68.165.073.829.247.157.454.565.341.491.659.4
    CE+Logit KD+attention (V2T)70.265.677.526.964.552.553.070.755.691.962.8

    Using the corrected results above, we can identify the source of V2T’s gains.

    Logit KD is the main driver. CE alone is 58.9; logit KD alone gives 61.5 (+2.6), while attention alone gives 59.4 (+0.5). Most of the standalone improvement comes from the interleaved teacher’s predictions.

    Attention pays off only on top of logit KD. Added to logit KD, it improves 61.5 → 62.8 (+1.3). On its own it exposes weak reasoning: the model over-focuses on a small region and cannot recover when the target is not there. This shows per task — attention alone is best on Object Localization (57.4) and Forensic Detection (54.5), but collapses on Visual Correspondence (47.1 vs. 58.7 for CE) and Multi-view Reasoning (41.4 vs. 44.4), both of which require comparing regions and integrating global structure, and it underperforms on V* (73.8 vs. 77.5), where search depends on recovering from an initially wrong focus. Logit KD does the opposite, driving Multi-view (44.4 → 54.9) and Visual Correspondence (58.7 → 62.8).

    Attention distillation transfers where to look; logit distillation transfers how to reason with what is found. V2T needs both, and combining them is best (62.8).

  1. W3, Q3: Weak evidence for internalized visual reasoning.

    Alignment does not explain the gain. Text-Only SFT shares V2T’s base model, training data, and text-only inference format, differing only in receiving no teacher signal. It has no train–inference mismatch and serves as a direct alignment control, yet averages 58.9 against V2T’s 62.8 across a suite spanning high-resolution perception, visual search, fine-grained reasoning, and hallucination. The static-teacher student (W1, experiment iii) is equally aligned and gains only 0.3 points over Text-Only SFT, so alignment by itself accounts for very little of the 3.9-point gap.

    On the suggested diagnostic: the teacher evaluated with ground-truth reasoning images is not an informative reference, because in many categories the final reasoning image is the answer or determines it directly — the board position after the chosen move in Chess, the final molecular structure in Chemistry (Figs. 9, 10). Conditioning on the full ground-truth trajectory therefore measures the leakage, not the teacher’s reasoning. The informative comparison is Table 5, where the teacher is given reasoning images and its attention to the problem image drops to IoU 0.168, below the base model’s 0.191: with the images present it stops grounding in the original input. We therefore compare among models that never see reasoning images at inference.

    Localization does not explain it either. We agree attention IoU measures localization and have tempered that claim in the revision. Attention supervision alone behaves exactly as a localization-only signal should: it raises Object Localization by 8.2 points (49.2 → 57.4) and IoU from 0.215 to 0.266, yet improves the average by only 0.5 (58.9 → 59.4) and costs 11.6 points on Visual Correspondence (58.7 → 47.1). Knowing where to look is not sufficient.

    Adding logit distillation to that setting raises the average by 3.4 points, Multi-view Reasoning by 14.2, and Jigsaw by 5.4 — tasks that require inferring spatial change across views and reconstructing structure from shape, color, and texture continuity, neither solved by localizing a region. Logit distillation alone also improves grounding (IoU 0.257) without any attention supervision, indicating the teacher’s output distributions carry spatial information a grounding loss does not supply. This matches Table 6, where Spatial and Global Context errors fall 16.1% and Task and Answer Consistency 14.4% while Perception and Grounding stay flat — the pattern expected if reasoning over visual evidence improves rather than raw perception.

    Attention distillation transfers where to look; logit distillation transfers how to reason with what is found.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.