acceptodds
Withdrawn from ICLR 2027

Preserve Content, Disrupt Structure: Patch-Shuffle Counterfactual Distillation for Visual Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal on-policy distillation (OPD) provides dense, token-level teacher guidance on student-generated reasoning responses. To identify corrections that depend on visual evidence, previous methods contrast teacher predictions on original inputs and negative views created by masking or downsampling. However, these negatives mainly probe evidence availability and can miss signals for spatial relations and object–attribute bindings. We introduce Patch-Shuffle Counterfactual Distillation (PSCD), which constructs a negative visual view by shuffling patches within the teacher's evidence crop. This preserves pixels within each patch while disrupting spatial and compositional organization. At each student-generated prefix, PSCD compares the frozen teacher's predictions on the intact and shuffled crops, retaining only changes that support the intact teacher's correction of the student to construct a student-anchored distillation target. Experiments on two Qwen3.5 model scales show gains in spatial and compositional reasoning alongside improved fine-grained visual understanding. Among the evaluated Qwen3.5-4B-based methods, PSCD leads on all six spatial benchmarks and outperforms the strongest baseline on all three compositional understanding benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, multimodal reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nCQ2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Patch-Shuffle Counterfactual Distillation (PSCD), an on-policy distillation method for visual reasoning. A frozen teacher evaluates student-generated response prefixes using an intact evidence crop and a patch-shuffled version. Their prediction differences provide a contrastive signal, which is aligned with the intact teacher’s correction through a projection-based update budget and candidate-level agreement filtering. The resulting student-anchored target supervises a student that receives the full image, without additional cropping or search at inference. The main contribution is the integration of patch-shuffle visual interventions with this distillation target construction. The authors evaluate Qwen3.5-4B and 9B trained on 6,241 synthetic examples, reporting improved average performance across spatial, compositional, fine-grained, and broader visual benchmarks. Ablations examine alternative negative views, shuffle strength and granularity, divergence objectives, and target-construction components.

Critical strengths
  1. Broad empirical evaluation with consistent aggregate improvements. PSCD achieves the highest reported group averages among the evaluated same-scale methods. In particular, its fine-grained benchmark average exceeds the strongest reported baseline. This breadth provides meaningful evidence of practical value beyond a single benchmark.

  2. Direct empirical support for the negative-view choice. Table 4 compares patch shuffling with masking and downsampling while retaining the PSCD training recipe. Patch shuffling improves spatial and compositional over the strongest alternative. This comparison supports the usefulness of the proposed intervention within the evaluated setting, beyond the performance of the complete system alone.

Critical weaknesses
  1. Limited novelty. Patch shuffling has already been explored in visual representation learning, including earlier jigsaw-based pretraining. The central conceptual contribution here is demonstrating the value of this established operation in visual OPD. While this adaptation has empirical merit, the paper does not clearly establish a substantial methodological advance beyond combining the intervention with its proposed distillation target.

  2. Insufficient rigor and clarity in the motivation. The introduction’s motivating observations omit essential information, including sample size, sampling procedure, and annotation protocol, making their reliability and generality difficult to assess. Furthermore, weak teacher responses under masking or downsampling do not directly establish why patch shuffling should provide better supervision. The connection between the observed problem and the proposed solution remains insufficiently substantiated.

  3. Incomplete coverage of closely related work. The discussion omits relevant studies on structural contrast, counterfactual visual distillation, and visual contrast for reasoning. In particular, the following papers should be discussed to clarify the contribution relative to existing approaches:

    • SDCD: SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models
    • CVPD: Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots
    • VC-STaR: Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs, ICLR 2026
  4. Insufficient analysis of why patch shuffling helps. The experiments demonstrate performance improvements, but provide limited insight into their underlying cause. Existing ablations and qualitative examples do not adequately distinguish structural disruption from perceptual degradation or isolate shuffling from the accompanying target-construction changes. Given the paper’s emphasis on “preserving content while disrupting structure,” a deeper analysis is needed to substantiate this central explanation and strengthen its contribution to ICLR.

Questions for authors
  1. Why intervene at the image level rather than the token level? What motivates shuffling image patches instead of permuting initial visual patch embeddings before positional encodings are added? Please clarify how the shuffle grid aligns with the vision encoder’s native patch grid and whether resizing occurs after shuffling. Under your implementation, are these two interventions equivalent? If not, what justifies the image-level choice, particularly given possible boundary artifacts and interactions with the encoder’s patch partitioning?

  2. What scope of “visual reasoning” do the experiments establish? The qualitative examples show mostly short answers, and evaluation disables thinking for some reference models. While short answers do not preclude reasoning, could you clarify whether the intended claim concerns improved spatial and compositional judgments under non-thinking decoding, or broader improvements in multi-step visual reasoning? Which results distinguish reasoning improvements from better perception or recognition, and how does the evaluation protocol support the scope claimed in the title?

7 comments
AuthorsOpenReview ↗

Author Response to Reviewer nCQ2 (Part1/6)

Thank you for your valuable feedback, which is crucial for improving our paper. We have addressed each point and revised the manuscript accordingly. In the revised PDF, all changes are shown in blue.

Weakness 1: Methodological novelty beyond jigsaw pretraining and combining patch shuffling with distillation\color{#B45309}{\large\textbf{Weakness 1: Methodological novelty beyond jigsaw pretraining and combining patch shuffling with distillation}}

Limited novelty. Patch shuffling has already been explored in visual representation learning, including earlier jigsaw-based pretraining. The central conceptual contribution here is demonstrating the value of this established operation in visual OPD. While this adaptation has empirical merit, the paper does not clearly establish a substantial methodological advance beyond combining the intervention with its proposed distillation target.

Response:\color{#B45309}{\Large\textbf{Response:}}

  • Our primary contribution is diagnostic and insight-driven rather than claiming a complex standalone visual operator. We do not claim patch shuffling itself as a new image transformation. Instead, our core contribution lies in diagnosing why existing visual OPD methods struggle with spatial and compositional reasoning (Figure 1 and Appendix B.1): common counterfactual views such as masking and downsampling give the teacher VLM very weak corrective signals on spatial relation words (low-response rates of 51.6% and 46.4%), while heavily degrading local object recognition (31.3% and 13.1% in Appendix B.7). Motivated directly by this failure mode, we design a targeted distillation method that preserves local object integrity while disrupting global spatial layout, leading to consistent gains across both spatial and compositional benchmarks.
AuthorsOpenReview ↗

Author Response to Reviewer nCQ2 (Part2/6)

Weakness 2: Rigor and clarity of the motivating diagnostic in the Introduction\color{#B45309}{\Large\textbf{Weakness 2: Rigor and clarity of the motivating diagnostic in the Introduction}}

Insufficient rigor and clarity in the motivation. The introduction’s motivating observations omit essential information, including sample size, sampling procedure, and annotation protocol, making their reliability and generality difficult to assess. Furthermore, weak teacher responses under masking or downsampling do not directly establish why patch shuffling should provide better supervision. The connection between the observed problem and the proposed solution remains insufficiently substantiated.

Response:\color{#B45309}{\Large\textbf{Response:}}

  • We revised the diagnostic protocol to evaluate patch shuffling, downsampling, and masking side by side using full-vocabulary distribution divergences (Figure 1 and Appendix B.1). Rather than measuring sensitivity only on a single sampled student token, we compute the teacher's full next-token distribution shift (JSD, Forward KL, and Reverse KL) and low-response rate (fraction of instances with divergence below 0.010.01 nats) between the intact image and each intervened view on the exact same examples:
InterventionJSD ↑\uparrowForward KL ↑\uparrowReverse KL ↑\uparrowLow-response rate (%) ↓\downarrow
Downsampling (×0.1\times 0.1)0.06890.36280.478946.4%
Masking (50%)0.05690.29970.410451.6%
Patch shuffle (16×1616\times 16)0.10880.59890.943936.6%
Patch shuffle (after training)0.12650.66971.150330.5%

On identical examples, 16×1616\times16 patch shuffling produces the largest teacher response across every metric while lowering the low-response rate from 46.4% (downsampling) and 51.6% (masking) to 36.6%, and further to 30.5% after training, directly demonstrating why it provides stronger supervision at relation tokens.

  • Complete dataset sampling and lexical annotation details are now specified in Appendix B.1. The diagnostic set comprises 260 public examples randomly sampled from VSR zero-shot test (100), GQA balanced test-dev (100), and V$^\star$ Bench relative-position questions (60). From the student's sampled answers, we deterministically match all occurrences against a fixed list of 23 spatial relation words (e.g., above, below, left, right).
AuthorsOpenReview ↗

Author Response to Reviewer nCQ2 (Part3/6)

Weakness 3: Incomplete coverage of closely related work (SDCD, CVPD, and VC-STaR)\color{#B45309}{\Large\textbf{Weakness 3: Incomplete coverage of closely related work (SDCD, CVPD, and VC-STaR)}}

Incomplete coverage of closely related work. The discussion omits relevant studies on structural contrast, counterfactual visual distillation, and visual contrast for reasoning. In particular, the following papers should be discussed to clarify the contribution relative to existing approaches:

  • SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models
  • CVPD: Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots
  • VC-STaR: Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs, ICLR 2026

Response:\color{#B45309}{\Large\textbf{Response:}}

  • We have added a dedicated discussion comparing PSCD with SDCD, CVPD, and VC-STaR in Related Work (Section 2). While all three works leverage visual or structural contrast, PSCD differs fundamentally in its training paradigm, supervision granularity, and counterfactual mechanism:
    • SDCD applies structure-disrupted contrastive decoding at inference time by subtracting logits of structurally disrupted views to suppress hallucinations, which doubles inference cost and leaves model weights unchanged. In contrast, PSCD distills structural sensitivity into the student's weights during on-policy training, adding zero inference overhead.
    • CVPD performs self-contained distillation from counterfactual visual blind spots by erasing or masking image regions to identify visually grounded tokens. As shown in Figure 1 and Appendix B.7, erasing or masking destroys local object identity and yields weaker shifts on spatial relation words, whereas PSCD preserves local patch appearance while scrambling global spatial layout to isolate relation and binding errors.
    • VC-STaR improves visual reasoning through sequence-level self-improvement and preference optimization over contrasting visual trajectories. By contrast, PSCD provides dense, token-level logit supervision anchored to a privileged evidence crop, targeting fine-grained spatial and compositional tokens directly within each rollout.
Reviewer_FP1AOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Patch-Shuffle Counterfactual Distillation (PSCD) for multimodal on-policy distillation. At each student-generated prefix, a frozen teacher evaluates an intact evidence crop and a patch-permuted version of that crop. The resulting order-restoration effect is projected onto the intact-teacher correction, filtered for candidate-wise agreement, and used to construct a student-anchored distillation target. Experiments on Qwen3.5-4B and Qwen3.5-9B report improvements across spatial reasoning, compositional understanding, fine-grained perception, and broader visual benchmarks. The paper also studies counterfactual views, shuffle strength and granularity, divergence objectives, and target-construction components.

Critical strengths
  1. The intervention is simple, training-only, and clearly specified. Equations 1–5 and Appendix A.1 carefully connect the intact/shuffled teacher contrast to a correction the student actually needs, rather than indiscriminately distilling every teacher change.

  2. The empirical pattern is broad and largely consistent. In the matched Qwen3.5 blocks, PSCD leads the reported spatial and compositional results and improves broader-task averages at both scales (Tables 1–3). The counterfactual-view, shuffle, divergence, and component ablations (Tables 4–7) generally support the proposed design, while inference remains single-pass.

Critical weaknesses
  1. The experiments do not yet isolate the claimed spatial/binding mechanism. A full patch permutation preserves the pixel multiset, but not necessarily semantic “content”: it fragments objects and textures crossing patch boundaries and creates a strongly out-of-distribution image. The motivating diagnostic reports miss rates for downsampling and masking, but does not report the corresponding patch-shuffle miss rate, the number and annotation protocol of the relation/binding positions, or a corruption-severity-matched control. Consequently, the gains could partly reflect generic corruption-based regularization rather than specifically improved relation and binding supervision.

  2. Statistical and baseline coverage is insufficient for the strongest claims. No independent training runs or confidence intervals are reported, despite several margins being small and each example using one fixed permutation. Moreover, the primary spatial/compositional table omits several matched methods that appear in the fine-grained and broader-task tables, particularly VA-OPD and Decomposed OPD. It is therefore unclear whether the ranking is stable and whether PSCD remains best against the complete matched baseline set on its primary endpoints.

Questions for authors
  1. On exactly the annotated positions used in Figure 1 and Appendix B.1, what is the miss rate of patch shuffling under Equation 7? Please report the size and annotation procedure of the set, agreement if multiple annotators were used, variation across shuffle draws, and a control matched for corruption severity. A failure to recover substantially more relation/binding signal would weaken the paper’s central interpretation.

  2. How stable are the main gains across independent training seeds and patch permutations? Can the authors provide uncertainty estimates and evaluate the omitted matched OPD baselines on the primary spatial/compositional suite at both scales?

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer FP1A (Part 1/4)

Thank you for your valuable feedback, which is crucial for improving our paper. We have addressed each point and revised the manuscript accordingly. In the revised PDF, all changes are shown in blue.

Weakness 1: Isolating the spatial and binding mechanism from generic visual corruption\color{#B45309}{\Large\textbf{Weakness 1: Isolating the spatial and binding mechanism from generic visual corruption}}

The experiments do not yet isolate the claimed spatial/binding mechanism. A full patch permutation preserves the pixel multiset, but not necessarily semantic “content”: it fragments objects and textures crossing patch boundaries and creates a strongly out-of-distribution image. The motivating diagnostic reports miss rates for downsampling and masking, but does not report the corresponding patch-shuffle miss rate, the number and annotation protocol of the relation/binding positions, or a corruption-severity-matched control. Consequently, the gains could partly reflect generic corruption-based regularization rather than specifically improved relation and binding supervision.

Response:\color{#B45309}{\Large\textbf{Response:}}

  • Patch shuffling disrupts spatial arrangement while keeping objects recognizable. To verify that patch shuffling does not destroy semantic object content, we probe the frozen teacher VLM on individual transformed patches containing part of a SAM-segmented target object, asking "Is there a [target object] in this patch?" (Appendix B.7). Under 16×1616\times16 shuffling, the teacher still recognizes 89.0% of the objects, compared with 31.3% under 75% masking and 13.1% under ×0.05\times0.05 downsampling. Moreover, 16×1616\times16 shuffling achieves the highest geometric overlap with target object masks (47.7% Max IoU) among all grid sizes, explaining why the spatial and compositional gains in Table 5 follow the exact same ordering as patch-level object recognition (16×16>32×32>8×816\times16 > 32\times32 > 8\times8).

  • Full-vocabulary distribution divergences directly compare patch shuffling against downsampling and masking on the same diagnostic set. Following your suggestion, we replace the single-token miss rate with full-vocabulary next-token distribution divergences (JSD, Forward KL, and Reverse KL) and the low-response rate (fraction of relation words with divergence below 0.010.01 nats) across all interventions on identical examples (Figure 1 and Appendix B.1):

InterventionJSD ↑\uparrowForward KL ↑\uparrowReverse KL ↑\uparrowLow-response rate (%) ↓\downarrow
Downsampling (×0.1\times 0.1)0.06890.36280.478946.4%
Masking (50%)0.05690.29970.410451.6%
Patch shuffle (16×1616\times 16)0.10880.59890.943936.6%
Patch shuffle (after training)0.12650.66971.150330.5%

Although 16×1616\times16 shuffling preserves object identity far better than downsampling or masking, it elicits the strongest teacher shift on relation words and lowers the low-response rate to 36.6% (30.5% after training). This confirms that the teacher is responding to altered spatial layouts rather than generic visual corruption.

  • Complete dataset sampling and relation-word annotation details are now specified in Appendix B.1. The diagnostic set comprises 260 public examples sampled from VSR zero-shot test (100), GQA balanced test-dev (100), and V$^\star$ Bench relative-position questions (60). From the student's sampled answers, we extract all occurrences matching a fixed list of 23 spatial relation words (e.g., above, below, left, right), yielding M=1,744M = 1{,}744 annotated relation word instances. Full breakdown and extraction rules are documented in Appendix B.1.
AuthorsOpenReview ↗

Author Response to Reviewer FP1A (Part 2/4)

Weakness 2: Statistical stability and complete baseline coverage on primary endpoints\color{#B45309}{\Large\textbf{Weakness 2: Statistical stability and complete baseline coverage on primary endpoints}}

Statistical and baseline coverage is insufficient for the strongest claims. No independent training runs or confidence intervals are reported, despite several margins being small and each example using one fixed permutation. Moreover, the primary spatial/compositional table omits several matched methods that appear in the fine-grained and broader-task tables, particularly VA-OPD and Decomposed OPD. It is therefore unclear whether the ranking is stable and whether PSCD remains best against the complete matched baseline set on its primary endpoints.

Response:\color{#B45309}{\Large\textbf{Response:}}

  • We added VA-OPD (4B and 9B) and V-Zero (9B) to Table 1, and removed Decomposed OPD across all tables to keep comparisons focused on visual counterfactual baselines. Unlike Vision-OPD, V-Zero, and VA-OPD, Decomposed OPD does not construct visual counterfactuals, so we removed it from Tables 2 and 3 to maintain a consistent set of comparable baselines across all evaluations. For V-Zero-9B, we trained the model using its official recipe. For VA-OPD, we reimplemented the method following its paper using the exact same student, crop teacher, training data, and training budget as our runs, and added both 4B and 9B results to Table 1:
Model4B (Spatial / Comp.)9B (Spatial / Comp.)
Qwen3.5 (base)69.64 / 59.5574.21 / 64.14
Vision-OPD73.83 / 58.8976.39 / 64.45
V-Zero73.68 / 60.1375.50 / 61.43
VA-OPD75.94 / 59.85[TBD]
PSCD (Ours)76.18 / 63.8578.45 / 66.04

Across both model scales, PSCD achieves the highest Spatial and Compositional averages against the complete matched baseline set.

  • Statistical reliability across training seeds, patch permutations, and evaluation runs. To assess robustness to randomness in optimization, patch shuffling, and evaluation, we conducted three independent runs with distinct random seeds (each governing training, patch permutations, and evaluation together) and reported the resulting mean ± std Spatial and Compositional scores in Table 1. Variance across runs is small, and PSCD consistently outperforms all matched baselines at both scales, confirming that our gains and rankings are stable.
AuthorsOpenReview ↗

Author Response to Reviewer FP1A (Part 3/4)

Question 1: Patch-shuffle sensitivity, annotation protocol, shuffle variance, and severity-matched controls (Figure 1 / Appendix B.1)\color{#B45309}{\large\textbf{Question 1: Patch-shuffle sensitivity, annotation protocol, shuffle variance, and severity-matched controls (Figure 1 / Appendix B.1)}}

On exactly the annotated positions used in Figure 1 and Appendix B.1, what is the miss rate of patch shuffling under Equation 7? Please report the size and annotation procedure of the set, agreement if multiple annotators were used, variation across shuffle draws, and a control matched for corruption severity. A failure to recover substantially more relation/binding signal would weaken the paper’s central interpretation.

Response:\color{#B45309}{\Large\textbf{Response:}}

  • Full-distribution divergences and low-response rates across all interventions on the exact same diagnostic set (Figure 1 and Appendix B.1). To better address this question, we adopted a more rigorous diagnostic protocol. Rather than relying only on a single sampled student token, we evaluate the teacher's full next-token distribution shift at the first token of every relation word using JSD, Forward KL, Reverse KL, and the low-response rate (fraction of instances with divergence below 0.010.01 nats). Across the exact same M=1,744M = 1{,}744 annotated relation word instances (260 examples, 258 answers with at least one spatial word), 16×1616\times16 patch shuffling recovers substantially stronger relation sensitivity than both downsampling and masking:
InterventionJSD ↑\uparrowForward KL ↑\uparrowReverse KL ↑\uparrowLow-response rate (%) ↓\downarrow
Downsampling (×0.1\times 0.1)0.06890.36280.478946.4%
Masking (50%)0.05690.29970.410451.6%
Patch shuffle (16×1616\times 16)0.10880.59890.943936.6%
Patch shuffle (after training)0.12650.66971.150330.5%
  • Deterministic lexical annotation protocol (Appendix B.1). The diagnostic set consists of 260 public examples randomly sampled from VSR zero-shot test (100), GQA balanced test-dev (100), and V$^\star$ Bench relative-position questions (60). Instead of manual human annotation, relation positions are extracted deterministically using an exact match against a fixed lexicon of 23 spatial relation words (e.g., above, below, left, right, inside, beside), avoiding inter-annotator subjectivity.
Reviewer_AWseOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PSCD extends visual on-policy distillation by comparing a frozen teacher’s predictions on intact and patch-shuffled evidence crops. It retains changes aligned with the intact teacher’s correction of the student and uses them to construct a student-anchored target. The student learns from its own generated prefixes and requires only the full image at inference.

Critical strengths
  1. The paper asks whether negative visual views should probe spatial organization rather than only evidence availability. Patch shuffling is simple to implement and adds no inference-time operations.

  2. Instead of blindly applying contrastive logit differences, filtering the patch-shuffle signal so that it only updates tokens when agreeing with the intact teacher's correction effectively prevents noisy supervision from corrupted images.

  3. With the training recipe held fixed, patch shuffling outperforms masking and downsampling, with a larger gain in compositional understanding than spatial reasoning.

Critical weaknesses
  1. Patch shuffling does not isolate spatial organization. Preserving pixels within individual patches does not preserve object identity, shape, or recognizability across patches. The teacher’s prediction change may therefore reflect general visual corruption rather than sensitivity to relations or attribute bindings. Table 4 establishes that shuffling is useful, but does not resolve this interpretation. A control that preserves recognizable objects while changing their arrangement, or matches corruption severity across interventions, would strengthen the central claim.

  2. In Figure 1 and Appendix B.1, the authors report that downsampling and masking suffer from 32% and 37% miss rates on annotated relation and binding words. However, they never report the miss rate of their own patch-shuffling method on the exact same diagnostic set. Moreover, details on how many examples were annotated are missing, and measuring sensitivity solely on the single sampled student token (rather than the full next-token distribution) may underestimate the teacher's corrective signal.

  3. Building the distillation target involves a long chain of heuristics: inner-product projection, positive/negative branch splitting, a positive branch cap, token-wise sign gating, logit clipping, and an adaptive intact-teacher regularization term. While Appendix B.4 ablates two sub-steps, it remains unclear whether all this engineering is truly necessary compared to a much simpler contrastive update (e.g., directly adding a scaled difference between intact and shuffled teacher logits) or distilling solely from the intact crop under the exact same setup.

  4. Missing baselines in Table 1 and unclear crop source: VA-OPD and Decomposed OPD (as well as V-Zero at the 9B scale) are evaluated in Tables 2 and 3, but are omitted from Table 1, which tests the core spatial and compositional claims of the paper. In addition, the paper does not clearly explain how the privileged evidence crops are obtained for the 6,241 training samples (e.g., whether they require ground-truth bounding box annotations), nor does it report the extra training time caused by running two teacher forward passes per sequence.

Questions for authors

N/A

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer AWse (Part 1/4)

Thank you for your valuable feedback, which is crucial for improving our paper. We have addressed each point and revised the manuscript accordingly. In the revised PDF, all changes are shown in blue.

Weakness 1: Patch shuffling versus generic visual corruption\color{#B45309}{\Large\textbf{Weakness 1: Patch shuffling versus generic visual corruption}}

Patch shuffling does not isolate spatial organization. Preserving pixels within individual patches does not preserve object identity, shape, or recognizability across patches. The teacher's prediction change may therefore reflect general visual corruption rather than sensitivity to relations or attribute bindings. Table 4 establishes that shuffling is useful, but does not resolve this interpretation. A control that preserves recognizable objects while changing their arrangement, or matches corruption severity across interventions, would strengthen the central claim.

Response:\color{#B45309}{\Large\textbf{Response:}}

  • Patch shuffling disrupts spatial arrangement while keeping objects recognizable. A valid counterfactual requires breaking spatial relationships without destroying the objects themselves—otherwise, changes in the teacher's predictions could merely reflect a failure to recognize corrupted objects. To verify this, we probe the teacher VLM on individual transformed patches containing part of a SAM-segmented target object (asking "Is there a [target object] in this patch?"). Under 16×1616\times16 shuffling, the teacher still recognizes 89.0% of the objects, whereas 75% masking and ×0.05\times0.05 downsampling drop recognition to 31.3% and 13.1%, respectively (Appendix B.7). Crucially, despite preserving object identity far better, 16×1616\times16 shuffling induces the largest shift in the teacher's predictions on relation tokens among all three transformations (Appendix B.1). This confirms that the teacher is responding to altered spatial layouts rather than lost object evidence.

  • Preserving local object integrity directly explains the grid ablation in Table 5. Effective patch shuffle counterfactual training requires views that retain local object appearance while scrambling global layout. In our patch-level probe, the 16×1616\times16 grid best preserves object integrity, achieving both the highest geometric overlap (47.7% Max IoU) and the highest teacher recognition (89.0%), outperforming 32×3232\times32 (34.2% IoU / 76.5% recognition) and 8×88\times8 (37.4% IoU / 58.3% recognition) (Appendix B.7). Because 16×1616\times16 shuffling provides cleaner object-level visual cues during counterfactual distillation, it yields the strongest spatial and compositional gains in Table 5, strictly mirroring the recognition ranking (16×16>32×32>8×816\times16 > 32\times32 > 8\times8).

AuthorsOpenReview ↗

Author Response to Reviewer AWse (Part 2/4)

Weakness 2: Missing patch-shuffling baseline, annotation details, and full-distribution sensitivity in Figure 1 / Appendix B.1\color{#B45309}{\large\textbf{Weakness 2: Missing patch-shuffling baseline, annotation details, and full-distribution sensitivity in Figure 1 / Appendix B.1}}

In Figure 1 and Appendix B.1, the authors report that downsampling and masking suffer from 32% and 37% miss rates on annotated relation and binding words. However, they never report the miss rate of their own patch-shuffling method on the exact same diagnostic set. Moreover, details on how many examples were annotated are missing, and measuring sensitivity solely on the single sampled student token (rather than the full next-token distribution) may underestimate the teacher's corrective signal.

Response:\color{#B45309}{\Large\textbf{Response:}}

  • Full-vocabulary distribution divergences directly compare patch shuffling against downsampling and masking on the same diagnostic set. Measuring sensitivity solely on a single sampled token can overlook shifts across the broader vocabulary, and evaluating all interventions on identical examples is essential for a fair comparison. Following your suggestion, we replace the single-token miss rate with full-vocabulary next-token distribution divergences, including Jensen–Shannon Divergence (JSD⁡\operatorname{JSD}), Forward KL (KL⁡(pi ∥ pi(m))\operatorname{KL}(p_i \,\|\, p_i^{(m)})), and Reverse KL (KL⁡(pi(m) ∥ pi)\operatorname{KL}(p_i^{(m)} \,\|\, p_i)), computed between the teacher's distributions on the intact image (pip_i) and each intervened view (pi(m))p_i^{(m)}) at the first token of every relation word. We also report the low-response rate, defined as the fraction of relation words with divergence below 0.010.01 nats (Appendix B.1):
InterventionJSD ↑\uparrowForward KL ↑\uparrowReverse KL ↑\uparrowLow-response rate (%) ↓\downarrow
Downsampling (×0.1\times 0.1)0.06890.36280.478946.4%
Masking (50%)0.05690.29970.410451.6%
Patch shuffle (16×1616\times 16)0.10880.59890.943936.6%
Patch shuffle (after training)0.12650.66971.150330.5%

On the exact same diagnostic set, 16×1616\times16 patch shuffling achieves the strongest teacher response across every metric while lowering the low-response rate from 46.4% (downsampling) and 51.6% (masking) to 36.6%, and further to 30.5% after training. These results confirm that patch shuffling provides a substantially stronger corrective signal at relation tokens than generic visual corruptions.

  • Complete dataset sampling and relation-word annotation details are now provided in Appendix B.1. The diagnostic set comprises 260 examples sampled from VSR zero-shot test (100), GQA balanced test-dev (100), and V$^\star$ Bench relative-position questions (60). From the student's sampled answers, we extract all occurrences matching a fixed list of 23 spatial relation words (e.g., above, below, left, right). Full breakdown and extraction rules are detailed in Appendix B.1.
AuthorsOpenReview ↗

Author Response to Reviewer AWse (Part 3/4)

Weakness 3: Necessity of the distillation target components versus simpler baselines\color{#B45309}{\Large\textbf{Weakness 3: Necessity of the distillation target components versus simpler baselines}}

Building the distillation target involves a long chain of heuristics: inner-product projection, positive/negative branch splitting, a positive branch cap, token-wise sign gating, logit clipping, and an adaptive intact-teacher regularization term. While Appendix B.4 ablates two sub-steps, it remains unclear whether all this engineering is truly necessary compared to a much simpler contrastive update (e.g., directly adding a scaled difference between intact and shuffled teacher logits) or distilling solely from the intact crop under the exact same setup.

Response:\color{#B45309}{\Large\textbf{Response:}}

  • We added direct comparisons against both simpler targets under the exact same training setup in Appendix B.4. Following your suggestion, we trained two simpler baselines for 130 steps from the same initialization using identical data and random seeds (Appendix B.4): (1) intact-teacher distillation alone, using ℓt(ptord)\ell_t(p_t^{\mathrm{ord}}) as the target, and (2) a plain contrastive update, ℓt(ptord)+Δt\ell_t(p_t^{\mathrm{ord}}) + \Delta_t, which directly adds the logit difference between the intact and shuffled teacher views without projection budgeting, branch allocation, or candidatewise agreement:
Target constructionSpatial Avg⁡6\operatorname{Avg}_6Compositional Avg⁡3\operatorname{Avg}_3
Intact teacher only, ℓt(ptord)\ell_t(p_t^{\mathrm{ord}})73.3160.07
Plain contrast, ℓt(ptord)+Δt\ell_t(p_t^{\mathrm{ord}}) + \Delta_t76.0261.48
Full target (Ours)76.1863.85
  • Each component contributes measurably beyond plain contrastive distillation. While plain contrastive distillation improves over using the intact teacher alone (76.02 vs. 73.31 on Spatial and 61.48 vs. 60.07 on Compositional), our full target achieves the strongest results across all benchmarks (76.18 Spatial and 63.85 Compositional), gaining +2.37 points on compositional tasks over plain contrast. Removing either projection or candidatewise agreement consistently lowers accuracy across all three task groups. Full details are provided in Appendix B.4.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author Responses and Kind Request for Further Discussion

Dear Reviewers,

We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions. We provided detailed clarifications in response to your questions a few days ago. The author response period will close on November 18 AOE. If you have any additional feedback, concerns, or questions regarding our response, we would greatly appreciate hearing from you and welcome further discussion.


I. Acknowledgments

Before the discussion{\color{#B45309}\text{Before the discussion}}, all three reviewers recognized the value of PSCD: its simple, training-only design (“simple, training-only, and clearly specified” from Reviewer FP1A), its principled filtering of the counterfactual signal (“effectively prevents noisy supervision from corrupted images” from Reviewer AWse), and its broad empirical gains (“meaningful evidence of practical value beyond a single benchmark” from Reviewer nCQ2).

In our rebuttal{\color{#B45309}\text{In our rebuttal}}, we addressed every concern with new analyses and experiments, summarized in Section III.


II. Key Strengths

Reviewers highlighted strengths across five dimensions:

  • Well-Motivated Formulation
    • Asks whether negative views should probe spatial organization, not only evidence availability (AWse)
  • Principled Distillation Target
    • Connects the intact/shuffled contrast to the correction the student actually needs (FP1A)
    • Agreement filtering prevents noisy supervision from corrupted views (AWse)
  • Simplicity and Practicality
    • Simple, training-only intervention with single-pass inference on the full image (FP1A, AWse)
  • Broad and Consistent Gains
    • Highest group averages among same-scale methods (nCQ2)
    • Leads spatial and compositional results at both Qwen3.5 scales (FP1A)
  • Ablation Support
    • With the recipe fixed, shuffling beats masking and downsampling, especially on compositional tasks (nCQ2, AWse)
    • Ablations in Tables 4–7 support the design (FP1A)

III. Key Concerns and Our Responses

Key ConcernsReviewersOur Response
Shuffling vs. Generic Corruption. Gains may reflect generic corruption rather than spatial/binding signals.FP1A, AWse, nCQ2Object-integrity probe (App. B.7): under 16×16 shuffling the teacher still recognizes 89.0% of objects (vs. 31.3% masking, 13.1% downsampling), yet shuffling causes the largest shift on relation tokens. Table 5 grid ranking mirrors recognition.
Rigor of Motivating Diagnostic. Missing shuffle miss rate, sample size, protocol; single-token measure.FP1A, AWse, nCQ2Switched to full-vocabulary JSD/KL on the same set (Fig. 1, App. B.1): shuffling has the highest JSD (0.109 vs. 0.069/0.057) and lowest low-response rate (36.6% vs. 46.4%/51.6%). Set: 260 public examples, 1,744 relation words, deterministic 23-word lexicon.
Necessity of Target Construction. Are projection and agreement needed over simpler targets?AWse, nCQ2New baselines (App. B.4): plain contrast beats intact-only (76.02/61.48 vs. 73.31/60.07), isolating the gain from shuffling; the full target adds +2.37 compositional points.
Baselines & Stability. VA-OPD, V-Zero-9B missing from Table 1; no repeated runs.FP1A, AWseAdded VA-OPD (4B/9B, matched setup) and V-Zero-9B; PSCD stays best (76.18/63.85 at 4B, 78.45/66.04 at 9B). Reported mean ± std over three seeds; rankings are stable.
Novelty & Related Work. Shuffling exists in jigsaw pretraining; missing SDCD, CVPD, VC-STaR.nCQ2We do not claim shuffling as a new operator; we diagnose why masking/downsampling miss relation signals and build a student-anchored target around it. Section 2 now contrasts SDCD (inference-time), CVPD (erasing), and VC-STaR (sequence-level).
Image- vs. Token-Level Shuffling. Grid alignment and equivalence.nCQ2Token-level variant gives nearly identical JSD (0.110 vs. 0.109) and accuracy (App. B.8); image-level is simpler and encoder-agnostic.
Scope of "Visual Reasoning." Reasoning or perception?nCQ2Compositional benchmarks (Winoground, SugarCrepe++, CRPE) use identical objects across candidates, so recognition alone cannot solve them.
Crop Source & Cost. Box annotations? Extra teacher pass?AWseCrops come from Vision-OPD's released data; the extra pass adds 5.0% training time, zero at inference (App. B.2).

IV. Commitment to Revision

All changes are marked in blue{\color{blue}\text{blue}} in our revision, including the revised diagnostic (Fig. 1, App. B.1), runtime breakdown (App. B.2), simpler-target baselines (App. B.4), object-integrity probe (App. B.7), image- vs. token-level comparison (App. B.8), completed Table 1 with three-seed statistics, and expanded Related Work (Section 2).


We deeply appreciate the expertise and time of the AC and reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.