- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper constructs CM-CoT, a compact multimodal chain-of-thought dataset containing 120K samples, and proposes VGG-CoT, which first probes the receptiveness of individual decoder layers in a student vision-language model to visual grounding knowledge, then distills visual grounding only into the most receptive contiguous decoder-layer interval, and finally distills multimodal reasoning into all decoder layers.
Critical strengths
The problem is important and clearly motivated. Small VLMs indeed face difficulty in simultaneously learning fine-grained visual grounding and long-chain reasoning, and decoupling these two capabilities is intuitively well justified.
The method is simple, modular, and relatively easy to reproduce. Layer-by-layer probing, contiguous-interval selection, and two-stage LoRA distillation are all built on commonly used training components and do not require teacher logits or intermediate representations.
Critical weaknesses
The evaluation may suffer from teacher self-consistency bias. Seven of the 12 source datasets do not provide task-aligned ground-truth bounding boxes, so their reference grounding boxes are generated by the same teacher model used for annotation. A student that more closely imitates the teacher will naturally achieve a higher IoU, but this does not necessarily mean that it better captures the true visual evidence. The paper does not report any human verification or assessment of bounding-box annotation quality.
The compact-CoT experiments conflate multiple factors. The compact version simultaneously changes sequence length, annotation structure, visual anchors, self-correction fields, grounding crops, and the supervision format. Therefore, the accuracy improvement cannot be attributed specifically to “compactness.” Length-matched, structure-matched, and field-removal controls are needed.
Questions for authors
-
How extensive was the human quality audit of the teacher-generated grounding boxes and compact chains of thought? Please report the audit sample size, annotation protocol, bounding-box IoU or inter-annotator agreement, and error rates for each task category.
-
Could the authors provide controlled comparisons using length-matched Native CoT, compressed CoT without additional structured fields, structured CoT without length reduction, compact CoT without visual anchors, and compact CoT without self-correction fields? These experiments would help disentangle the effects of sequence length and annotation structure.
-
Why does Native SeqKD perform approximately 40–55 ANLS points below the Original Model? Does the conclusion still hold when the generation length is controlled, the loss weight assigned to final-answer tokens is increased, or only the final answer is distilled?
-
Please report the complete computational cost of layer-by-layer probing, including the number of training runs for each student model, NPU hours, total training tokens, and the additional overhead relative to standard all-layer LoRA training. Can the selected VGR layers be reused across datasets after a single probing procedure?
5 comments
Q2: Could the authors provide controlled comparisons using length-matched Native CoT, compressed CoT without additional structured fields, structured CoT without length reduction, compact CoT without visual anchors, and compact CoT without self-correction fields? These experiments would help disentangle the effects of sequence length and annotation structure.
A2:
We conducted controlled comparisons on reasoning length, structural formatting, and individual field removal to examine how each component affects student performance. First, to separate the effects of reasoning length and structure, we retrained Native SeqKD at full scale alongside two controlled variants: compressing the reasoning without adding structured fields, and adding structured fields while preserving the original length. Both variants keep the teacher's native final answer and use the same single-stage all-layer training.
We also compare performance under different settings of the image token budget, which is defined as the maximum number of visual tokens per input image during inference. We evaluate each trained model with fixed weights at 256 and 100 tokens to check the supervision effects when more or less image detail is available.
We compare these variants with One-stage Compact in Table R23. One-stage Compact achieves higher ANLS and F1 than both controlled variants under both image token budgets.
Table R23: ANLS and F1 (%) for compressed CoT without additional structured fields, structured CoT without length reduction, and One-stage Compact on Qwen3-VL-2B.
| image token budget | Supervision & Method | ANLS | F1 |
|---|---|---|---|
| 256 | Native SeqKD | 12.31 | 15.44 |
| Compressed CoT without additional structured fields | 11.78 | 14.93 | |
| Structured CoT without length reduction | 12.76 | 15.68 | |
| One-stage Compact | 69.14 | 64.17 | |
| 100 | Native SeqKD | 11.19 | 12.73 |
| Compressed CoT without additional structured fields | 10.98 | 12.61 | |
| Structured CoT without length reduction | 10.67 | 12.12 | |
| One-stage Compact | 61.06 | 53.56 |
Second, we remove only the visual anchor or only the self-correction field at a time and report the MME, MMStar, and POPE results in Table R24.
Specifically, retaining the visual anchor improves position and scene recognition, helping the student answer from spatial layout and scene context. Retaining self-correction improves code reasoning and position recognition, supporting logical checking and visual judgment. Complete compact CoT also achieves higher MMStar coarse perception accuracy and POPE recall across all three sampling splits, identifying more present objects and reducing false negatives.
Overall, these results further verify that both visual anchors and self-correction contribute to the performance gains of compact CoT.
Table R24: Results for compact CoT without visual anchors and compact CoT without self-correction fields on Qwen3-VL-2B (image token budget 256). MME reports subtask scores, MMStar reports accuracy (%), and POPE reports recall (%).
| Supervision | MME Position | MME Scene | MME Code Reasoning | MMStar Coarse Perception (%) | POPE Random (%) | POPE Popular (%) | POPE Adversarial (%) | POPE Overall (%) |
|---|---|---|---|---|---|---|---|---|
| Complete compact CoT | 133.33 | 164.00 | 77.50 | 68.00 | 92.00 | 92.07 | 92.20 | 92.09 |
| Compact CoT without visual anchors | 118.33 | 162.25 | 77.50 | 66.80 | 89.53 | 89.60 | 89.53 | 89.56 |
| Compact CoT without self-correction fields | 128.33 | 164.00 | 70.00 | 66.80 | 89.87 | 89.93 | 89.93 | 89.91 |
Q3: Why does Native SeqKD perform approximately 40–55 ANLS points below the Original Model? Does the conclusion still hold when the generation length is controlled, the loss weight assigned to final-answer tokens is increased, or only the final answer is distilled?
A3:
The low performance of Native SeqKD can be partly attributed to the free-form answer patterns that the student learns by imitating the teacher's native CoT responses. Specifically, the student generates a long rationale followed by a final response that may include additional explanations or multiple candidate answers. Such responses may not conform to the concise, unambiguous answer format specified by these datasets. For example, the response in Table R25 gives both floor and table as possible locations. Extracting only floor would discard the alternative table and misrepresent the model's uncertain response as a single, definite prediction.
Table R25: A Native SeqKD response from Qwen3-VL-2B on V7W. The reference answer is Floor.
| Item | Content |
|---|---|
| Question | Where is the power strip? |
| Native final response | The power strip is on the wooden surface, likely the floor or a table, to the right of the hand holding the smartphone. |
Since Native SeqKD produces long responses, we checked whether hitting the generation token limit accounts for the low performance. As shown in Table R26, the scores remain low at both image token budgets even when we evaluate only the untruncated responses.
Table R26: Truncation statistics and EM, ANLS, and F1 (%) on untruncated examples for Qwen3-VL-2B Native SeqKD. EM denotes Exact Match of the answer.
| image token budget | Total | Truncated | Untruncated | EM (%) | ANLS | F1 |
|---|---|---|---|---|---|---|
| 256 | 30000 | 1329 | 28671 | 1.49 | 12.77 | 16.00 |
| 100 | 30000 | 1807 | 28193 | 1.11 | 11.97 | 13.57 |
We also evaluated three training variants of Native SeqKD while keeping the teacher's native final answer fixed. As shown in Table R23, shortening the reasoning without adding structured fields leaves both ANLS and F1 low. We see the same pattern in Table R27 when we increase the loss weight on final-answer tokens or distill only the final answer, keeping the other settings of the retrained Native SeqKD baseline fixed. By contrast, VGG-CoT trains the student to give concise answers grounded in visual evidence and achieves higher scores on both metrics.
Table R27: ANLS and F1 (%) when the loss weight assigned to final-answer tokens is increased or only the final answer is distilled. Qwen3-VL-2B is evaluated at an image token budget of 256. Scores are arithmetic means across datasets on the full test set.
| Supervision & Method | ANLS | F1 |
|---|---|---|
| Native SeqKD | 12.31 | 15.44 |
| Increased loss weight on final-answer tokens () | 11.55 | 14.51 |
| Only the final answer is distilled | 10.03 | 13.16 |
| VGG-CoT | 67.02 | 62.45 |
Thank you for your constructive suggestions regarding annotation quality and the composition of our supervision signal. We have added human audits, model-based verification, and controlled ablation experiments to address each point below.
A1:
We report the human quality audit covering all seven teacher-box source datasets in Table R20. We checked item by item whether the teacher box contains the visual evidence needed to answer the question and whether the compact explanation is grounded in the image. Across 350 audited questions, 93.71% of teacher boxes are usable and 96.29% of compact CoTs are image-grounded, with a mean box IoU of 0.590. Table R21 further breaks down these rates by source dataset across document, scene-text, and spatial-reasoning tasks.
Table R20: Human audit of teacher boxes and compact explanations.
Table R21: Annotation quality across source datasets in the human audit.
We further scaled up the verification to 7,000 examples using GPT and DeepSeek under the same criteria and report the results in Table R22. Both verifiers indicate that the teacher-generated boxes, compact CoTs, and answers are generally reliable.
Table R22: Annotation quality verification by GPT and DeepSeek.