- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates a prominent performance asymmetry in multimodal large language models: the same pretrained model performs substantially better on coding tasks when given source text than when given rendered images of the exact same task (e.g., Qwen3.5-4B achieves 75.1% pass@1 on text inputs vs. 55.2% on rendered image inputs across HumanEval and MBPP). Positing that this gap stems from uneven visual access rather than a lack of underlying coding knowledge, the authors propose Cross-Modal Self-OPD (On-Policy Distillation).
Critical strengths
The primary strength supporting acceptance is the clean problem formulation that isolates input representation privilege from confounding supervision sources (no external teacher capacity, no execution rewards, and no ground-truth targets), demonstrating that a multimodal model can bootstrap its weaker visual pathway purely by querying its own stronger text pathway on student-generated prefixes—and that doing so outperforms supervised fine-tuning on ground-truth solutions (60.9% vs. 56.5%).
Critical weaknesses
- Single-Model Evaluation and Small-Scale Synthetic Setup: The entire empirical validation is restricted to a single 4B model (Qwen3.5-4B) trained on a very small synthetic dataset (256 first-stage pairs and 360 second-stage tasks for 810 total updates) using clean, deterministically rendered monospaced text images. Without validation across different model families, parameter scales, or realistic visual coding environments, it remains unclear whether Cross-Modal Self-OPD scales as a general paradigm beyond repairing synthetic font-rendering bottlenecks in a single small VLM.
- Statistically Insignificant Headline Claims (Cross-View Transfer & Teacher Controls): Several central claims highlighted in the abstract and introduction are not supported by statistical significance under the paper's own exact McNemar tests. Specifically, the claim that image-trained checkpoints improve text-input coding beyond the frozen teacher (75.1% 76.4%, Table 7) relies on per-seed gains that are all statistically indistinguishable from noise ( in Table 8). Similarly, the matched teacher-view control (Image/text vs. start, in Table 2) and the teacher-solution ablation ( in Table 4) do not reach conventional statistical significance ().
Questions for authors
- Can you validate Cross-Modal Self-OPD on at least one additional VLM family or model scale beyond Qwen3.5-4B?
- Evaluate on realistic, noisy IDE screenshots (e.g., SWE-bench Multimodal) and explore unfreezing the vision-language projector to compare input-specific vs. shared-layer repair.
- Given that the per-seed McNemar -values for text-view transfer (Table 8, ), teacher-view control (Table 2, ), and solution context (Table 4, ) are all , can you provide pooled significance tests or temper these from headline claims to exploratory observations?
5 comments
Response to Reviewer fyZx
W2. Statistically Insignificant Headline Claims (Cross-View Transfer & Teacher Controls): Several central claims highlighted in the abstract and introduction are not supported by statistical significance under the paper's own exact McNemar tests. Specifically, the claim that image-trained checkpoints improve text-input coding beyond the frozen teacher (75.1% → 76.4%, Table 7) relies on per-seed gains that are all statistically indistinguishable from noise (p = 0.49,0.46, and 0.23 in Table 8). Similarly, the matched teacher-view control (Image/text vs. start, p = 0.13 in Table 2) and the teacher-solution ablation (p = 0.10 in Table 4) do not reach conventional statistical significance (p < 0.05).
Response. First, we would like to clarify that our primary finding is the consistent improvement in image-input coding. On Qwen3.5-4B, cross-modal self-OPD improves accuracy from 55.2% to 60.2–61.6% across three independent seeds, with statistically significant gains over both Pretrained and GT SFT under exact paired McNemar tests.
We have removed the claims about text-input transfer and the matched teacher-view comparison from the abstract, while retaining the statistically supported image-input improvements as our primary finding. The auxiliary code-stage teacher-view comparison, text-input transfer, and teacher solution context are presented as exploratory observations in Section 6.
Manuscript revision: We have removed the text-input transfer and matched teacher-view claims from the abstract (p. 1). Section 6 now explicitly qualifies the original code-stage teacher-view comparison in Observation 2 (pp. 7–8), text-input transfer in Observation 6 (p. 10), and teacher solution context in Observation 7 (p. 10) as auxiliary exploratory.
Response to Reviewer fyZx
Questions For Authors
Q1. Can you validate Cross-Modal Self-OPD on at least one additional VLM family or model scale beyond Qwen3.5-4B?
Response. To evaluate whether cross-modal self-OPD generalizes beyond Qwen3.5-4B, we conducted additional experiments on Molmo2-O-7B, a different VLM family with an OLMo3-7B-Instruct language backbone and a SigLIP 2 vision backbone. As shown in Table R1, cross-modal self-OPD consistently improves performance on both benchmarks across all three seeds. We have added the Molmo2-O-7B experiments, three-seed results, and paired tests to Section 6, Observation 4, and Table 3 (p. 9), marked in blue.
Q2. Evaluate on realistic, noisy IDE screenshots (e.g., SWE-bench Multimodal) and explore unfreezing the vision-language projector to compare input-specific vs. shared-layer repair.
Response. (a) Realistic IDE screenshots. As shown in Table R2, cross-modal self-OPD achieves mean improvements of +3.67 pp on IDE screenshots and +3.36 pp under image degradation across three seeds. These results extend our evaluation to realistic IDE screenshots without additional training. We have added the IDE screenshot and degradation experiments to Section 6, Observation 5, and Table 4 (p. 9).
(b) Projector-unfreezing ablation. We further investigated whether updating the vision-language projector provides additional benefits. On Qwen3.5-4B (seed 17), we compared the original language-layer LoRA adaptation with jointly training language-layer LoRA and the projector under the same training budget.
Table R3. Projector-unfreezing ablation on image-input coding.
| Trainable Components | HumanEval /164 | MBPP /427 | Combined /591 |
|---|---|---|---|
| Language LoRA (Projector Frozen) | 115 | 243 | 358 (60.58%) |
| Language LoRA + Projector | 119 | 235 | 354 (59.90%) |
As shown in Table R3, unfreezing the projector improves HumanEval by four tasks but reduces MBPP by eight, yielding no additional overall improvement. This suggests that language-layer adaptation alone can effectively improve image-input coding without requiring projector updates.
Manuscript revision: We have added this ablation to Appendix C.9 and Table 17 (p. 22), marked in blue, with a pointer in Section 6's Additional diagnostics paragraph (p. 10).
Q3. Given that the per-seed McNemar p-values for text-view transfer (Table 8, p ≥ 0.23), teacher-view control (Table 2, p = 0.13), and solution context (Table 4, p = 0.10) are all > 0.05, can you provide pooled significance tests or temper these from headline claims to exploratory observations?
Response. According to your suggestion, we have removed the claims about text-input transfer and the matched teacher-view comparison from the abstract, while retaining the statistically supported image-input improvements as our primary finding. The auxiliary code-stage teacher-view comparison, text-input transfer, and teacher solution context are presented as exploratory observations in Section 6.
Manuscript revision: We have removed the text-input transfer and matched teacher-view claims from the abstract (p. 1). Section 6 now explicitly qualifies the original code-stage teacher-view comparison in Observation 2 (pp. 7–8), text-input transfer in Observation 6 (p. 10), and teacher solution context in Observation 7 (p. 10) as exploratory.
Response to Reviewer fyZx
We sincerely thank the reviewer for the constructive feedback. We appreciate the recognition of our clean problem formulation, which isolates input representation privilege, and the gains over ground-truth supervised fine-tuning. We have addressed the concerns about cross-model generalization, realistic visual environments, and statistical evidence with additional experiments and clarifications, as detailed below.
Critical Weaknesses
Response.
Different model. To evaluate whether cross-modal self-OPD generalizes beyond Qwen3.5-4B, we conducted additional experiments with Molmo2-O-7B, a different VLM family with an OLMo3-7B-Instruct language backbone and a SigLIP 2 vision backbone. We retained the original training data, two-stage procedure, and update budget, and evaluated three independent training seeds.
Table R1. Image-input coding performance on Molmo2-O-7B.
As shown in Table R1, cross-modal self-OPD consistently improves performance on both benchmarks across all three seeds. Exact paired McNemar tests yield p = 1.37 × 10⁻⁸, 0.000987, and 0.00203, respectively. These results provide evidence that our method extends to a different VLM family without increasing the original training scale.
Manuscript revision: We have added the Molmo2-O-7B experiments, three-seed results, and paired tests to Section 6, Observation 4, and Table 3 (p. 9), marked in blue.
Realistic IDE screenshots. To evaluate performance beyond synthetic rendering, we tested the original Qwen3.5-4B checkpoints on all 427 MBPP tasks presented as screenshots from a running code-server IDE. The screenshots include light and dark editor themes. We also evaluated image degradation using 0.9× downsampling and JPEG compression (quality 85). No IDE-specific retraining was performed.
Table R2. Image-input coding performance on IDE screenshots.
As shown in Table R2, cross-modal self-OPD achieves mean improvements of +3.67 pp on IDE screenshots and +3.36 pp under image degradation across three seeds. These results extend our evaluation to realistic IDE screenshots without additional training.
Manuscript revision: We have added the IDE screenshot and degradation experiments to Section 6, Observation 5, and Table 4 (p. 9). These additions are marked in blue.