- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes Patch-Shuffle Counterfactual Distillation (PSCD), an on-policy distillation method for visual reasoning. A frozen teacher evaluates student-generated response prefixes using an intact evidence crop and a patch-shuffled version. Their prediction differences provide a contrastive signal, which is aligned with the intact teacher’s correction through a projection-based update budget and candidate-level agreement filtering. The resulting student-anchored target supervises a student that receives the full image, without additional cropping or search at inference. The main contribution is the integration of patch-shuffle visual interventions with this distillation target construction. The authors evaluate Qwen3.5-4B and 9B trained on 6,241 synthetic examples, reporting improved average performance across spatial, compositional, fine-grained, and broader visual benchmarks. Ablations examine alternative negative views, shuffle strength and granularity, divergence objectives, and target-construction components.
Critical strengths
-
Broad empirical evaluation with consistent aggregate improvements. PSCD achieves the highest reported group averages among the evaluated same-scale methods. In particular, its fine-grained benchmark average exceeds the strongest reported baseline. This breadth provides meaningful evidence of practical value beyond a single benchmark.
-
Direct empirical support for the negative-view choice. Table 4 compares patch shuffling with masking and downsampling while retaining the PSCD training recipe. Patch shuffling improves spatial and compositional over the strongest alternative. This comparison supports the usefulness of the proposed intervention within the evaluated setting, beyond the performance of the complete system alone.
Critical weaknesses
-
Limited novelty. Patch shuffling has already been explored in visual representation learning, including earlier jigsaw-based pretraining. The central conceptual contribution here is demonstrating the value of this established operation in visual OPD. While this adaptation has empirical merit, the paper does not clearly establish a substantial methodological advance beyond combining the intervention with its proposed distillation target.
-
Insufficient rigor and clarity in the motivation. The introduction’s motivating observations omit essential information, including sample size, sampling procedure, and annotation protocol, making their reliability and generality difficult to assess. Furthermore, weak teacher responses under masking or downsampling do not directly establish why patch shuffling should provide better supervision. The connection between the observed problem and the proposed solution remains insufficiently substantiated.
-
Incomplete coverage of closely related work. The discussion omits relevant studies on structural contrast, counterfactual visual distillation, and visual contrast for reasoning. In particular, the following papers should be discussed to clarify the contribution relative to existing approaches:
- SDCD: SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models
- CVPD: Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots
- VC-STaR: Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs, ICLR 2026
-
Insufficient analysis of why patch shuffling helps. The experiments demonstrate performance improvements, but provide limited insight into their underlying cause. Existing ablations and qualitative examples do not adequately distinguish structural disruption from perceptual degradation or isolate shuffling from the accompanying target-construction changes. Given the paper’s emphasis on “preserving content while disrupting structure,” a deeper analysis is needed to substantiate this central explanation and strengthen its contribution to ICLR.
Questions for authors
-
Why intervene at the image level rather than the token level? What motivates shuffling image patches instead of permuting initial visual patch embeddings before positional encodings are added? Please clarify how the shuffle grid aligns with the vision encoder’s native patch grid and whether resizing occurs after shuffling. Under your implementation, are these two interventions equivalent? If not, what justifies the image-level choice, particularly given possible boundary artifacts and interactions with the encoder’s patch partitioning?
-
What scope of “visual reasoning” do the experiments establish? The qualitative examples show mostly short answers, and evaluation disables thinking for some reference models. While short answers do not preclude reasoning, could you clarify whether the intended claim concerns improved spatial and compositional judgments under non-thinking decoding, or broader improvements in multi-step visual reasoning? Which results distinguish reasoning improvements from better perception or recognition, and how does the evaluation protocol support the scope claimed in the title?
7 comments
Author Response to Reviewer nCQ2 (Part2/6)
Insufficient rigor and clarity in the motivation. The introduction’s motivating observations omit essential information, including sample size, sampling procedure, and annotation protocol, making their reliability and generality difficult to assess. Furthermore, weak teacher responses under masking or downsampling do not directly establish why patch shuffling should provide better supervision. The connection between the observed problem and the proposed solution remains insufficiently substantiated.
- We revised the diagnostic protocol to evaluate patch shuffling, downsampling, and masking side by side using full-vocabulary distribution divergences (Figure 1 and Appendix B.1). Rather than measuring sensitivity only on a single sampled student token, we compute the teacher's full next-token distribution shift (JSD, Forward KL, and Reverse KL) and low-response rate (fraction of instances with divergence below nats) between the intact image and each intervened view on the exact same examples:
| Intervention | JSD | Forward KL | Reverse KL | Low-response rate (%) |
|---|---|---|---|---|
| Downsampling () | 0.0689 | 0.3628 | 0.4789 | 46.4% |
| Masking (50%) | 0.0569 | 0.2997 | 0.4104 | 51.6% |
| Patch shuffle () | 0.1088 | 0.5989 | 0.9439 | 36.6% |
| Patch shuffle (after training) | 0.1265 | 0.6697 | 1.1503 | 30.5% |
On identical examples, patch shuffling produces the largest teacher response across every metric while lowering the low-response rate from 46.4% (downsampling) and 51.6% (masking) to 36.6%, and further to 30.5% after training, directly demonstrating why it provides stronger supervision at relation tokens.
- Complete dataset sampling and lexical annotation details are now specified in Appendix B.1. The diagnostic set comprises 260 public examples randomly sampled from VSR zero-shot test (100), GQA balanced test-dev (100), and V$^\star$ Bench relative-position questions (60). From the student's sampled answers, we deterministically match all occurrences against a fixed list of 23 spatial relation words (e.g., above, below, left, right).
Author Response to Reviewer nCQ2 (Part3/6)
Incomplete coverage of closely related work. The discussion omits relevant studies on structural contrast, counterfactual visual distillation, and visual contrast for reasoning. In particular, the following papers should be discussed to clarify the contribution relative to existing approaches:
- SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models
- CVPD: Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots
- VC-STaR: Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs, ICLR 2026
- We have added a dedicated discussion comparing PSCD with SDCD, CVPD, and VC-STaR in Related Work (Section 2). While all three works leverage visual or structural contrast, PSCD differs fundamentally in its training paradigm, supervision granularity, and counterfactual mechanism:
- SDCD applies structure-disrupted contrastive decoding at inference time by subtracting logits of structurally disrupted views to suppress hallucinations, which doubles inference cost and leaves model weights unchanged. In contrast, PSCD distills structural sensitivity into the student's weights during on-policy training, adding zero inference overhead.
- CVPD performs self-contained distillation from counterfactual visual blind spots by erasing or masking image regions to identify visually grounded tokens. As shown in Figure 1 and Appendix B.7, erasing or masking destroys local object identity and yields weaker shifts on spatial relation words, whereas PSCD preserves local patch appearance while scrambling global spatial layout to isolate relation and binding errors.
- VC-STaR improves visual reasoning through sequence-level self-improvement and preference optimization over contrasting visual trajectories. By contrast, PSCD provides dense, token-level logit supervision anchored to a privileged evidence crop, targeting fine-grained spatial and compositional tokens directly within each rollout.
Author Response to Reviewer nCQ2 (Part1/6)
Thank you for your valuable feedback, which is crucial for improving our paper. We have addressed each point and revised the manuscript accordingly. In the revised PDF, all changes are shown in blue.
Weakness 1: Methodological novelty beyond jigsaw pretraining and combining patch shuffling with distillation
Response: