- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PReViD prevents error propagation in visual on-policy distillation by using contrastive, crop-based evidence prefixes to clean up the teacher's token targets. The authors claim to outperform standard visual OPD baselines across multiple high-res vision benchmarks with almost no training overhead.
Critical strengths
-
Low cost and solid controls. PReViD has zero inference footprint and negligible training overhead, with steady gains across all benchmarks and model scales. The random gate and shuffled prefix baselines are well-chosen and cleanly demonstrate that informative prefix content is what actually moves the needle.
-
Clean, transferable design. Inserting the prefix as an unfinished assistant turn preserves rollout alignment, while the placebo direction replaces arbitrary thresholding with a grounded baseline. The authors are also refreshingly honest that these are empirical input contrasts rather than causal claims, making the framework easily reusable for other privileged-context distillation problems.
Critical weaknesses
-
Underperforming baselines vs. published results: The re-implemented baselines (Vision-OPD, VAD) score noticeably below their officially published numbers, to the point where PReViD actually trails the original reported figures. Because the official baseline checkpoints are publicly available, the authors should simply run those released weights on their own harness to clarify whether this gap comes from evaluation differences or undertrained baselines.
-
The decomposition order artificially inflates the prefix path: Measuring the image step with a degraded text prefix already in context introduces a huge confound. Because models naturally defer to text over visual evidence under conflict, the visual contrast gets suppressed, handing the remaining 76–81% variance to the prefix by construction. Standard OPD runs without any prefix (), yet this baseline and the reversed path order are omitted entirely. The label flips in Appendix C confirm that this "pathway" is mostly just the familiar text-override effect rather than a newly isolated supervision channel. There is also zero visibility into what actually says (whether it's actively wrong or just vague).
-
Novelty framing & missing composite baseline: The core concept of feeding privileged text hints to the teacher is already established by π-Distill (ICLR 2026). The authors should reframe their contribution as merging OPSD's textual hints with Vision-OPD's visual crops, and run an "OPSD + crop" baseline to isolate the gain from their gated decomposition. Also, verify the Section 2.2 claim about Masking-KD, as it already intervenes on teacher-side prefixes.
-
Missing basic hyperparameters and reproducibility details: Section 4.1 points to Appendix A for training setup, but the appendix is missing nearly all of it. There are no learning rates, batch sizes, optimizers, rollout settings, or epoch counts for Table 1, nor any clarification on whether tuning is full-parameter or LoRA. Several core variables () and the exact implementation of the importance weight (clipped or standard ratio?) are also completely absent.
Questions for authors
- Is the scale-matched teacher simply a frozen base model (per Appendix A.2), or was it trained on privileged information (Sections 4.2)?
- Are the 70.24 MME-CN entries in Table 1 and Table 2 distinct runs?
4 comments
W2 on path order and the interpretation of degraded prefixes
Thank you for pointing out the effects of path order and prefix context on the decomposition. Section 3.2 of the original paper defines the two paths as controlled model input contrasts, rather than natural direct or indirect effects. Figure 2 and Appendix B report the 76%/81% results for the image first diagnostic. Appendix B also gives the score definition and subset selection rule.
We have extended the original diagnostic with the reverse decomposition and interaction term in Appendix B and Table 11 of the revised paper. This tests the effect of path order. The analysis reuses the cached scores for all four input combinations. All samples and the original fixed joint endpoint subset remain unchanged. Both image conditions use the same saved prefixes. The clear prefix is still generated from the clear image. It is not regenerated when the image is switched. The reverse path is
\qquad \delta'_e=\ell^{-+}-\ell^{--},\quad \delta'_x=\ell^{++}-\ell^{-+}.$$ We retain the log-odds score of the correct option relative to the set of valid incorrect options. We report mean scores and components. The prefix share is the mean prefix component divided by the mean endpoint contrast. |Sample group|Path order|Samples|Mean endpoint contrast|Mean image component|Mean prefix component|Prefix share| |---|---|---|---|---|---|---| |All samples|Image first|6241|3.444|0.820|2.623|76.179%| |All samples|Prefix first|6241|3.444|0.477|2.967|86.151%| |Original joint endpoint subset|Image first|2077|11.686|2.168|9.518|81.448%| |Original joint endpoint subset|Prefix first|2077|11.686|1.155|10.531|90.114%| The interaction term measures the difference between components under the two orders. Here, $\delta_e=\ell^{++}-\ell^{+-}$ and $\delta_x=\ell^{+-}-\ell^{--}$. $$I=\ell^{++}-\ell^{+-}-\ell^{-+}+\ell^{--} =\delta_e-\delta'_e =\delta'_x-\delta_x.$$ |Sample group|Mean interaction I| |---|---| |All samples|-0.343| |Original joint endpoint subset|-1.013| The reverse path gives prefix shares of 86.151% and 90.114%, higher than those under the original order. The mean prefix component is positive under both orders. **A large mean prefix component is therefore observed under both path orders.** In both sample groups, the mean image component is larger under degraded prefixes than under clear prefixes. Both mean interactions are negative. These results indicate that component sizes depend on prefix context. Section 3.1 and Appendix A.2 of the original paper define how clear, degraded, and placebo prefixes are generated. Figure 5 in Appendix B of the revised paper adds [evidence prefix examples](https://anonymous.4open.science/r/E02D2/supplement/replay.pdf) from three samples. These examples pair image inputs with the generated text. Each sample shows the clear evidence input, degraded evidence input, and uniform RGB placebo image, together with their evidence prefixes. The examples show that degraded inputs can produce incorrect observations.W3 on the method contribution and combined baselines
Thank you for pointing out the connections to prior work. Sections 2.1 and 2.2 of the original paper discuss π-Distill/OPSD, Vision-OPD, and VAD. Section 4.1 and Appendix A.1 specify that OPSD uses answer labels as textual privileged information. Section 4.4 explains that Vision-OPD follows a training procedure similar to OPSD, replaces textual privileged information with visual information, and supervises the full sequence.
If “OPSD + crop” means replacing answer label text with a clear evidence crop, it corresponds to Vision-OPD in our main table. Both use self-distillation with the current policy. They differ in the form of privileged information. Section 4.1 of the revised paper clarifies this correspondence. Section 4.2 gives a consistent description of the teacher based on the current policy.
Sections 3.2 to 3.4 of the original paper describe PReViD. It replays the same student response in four teacher contexts and selectively rebuilds the target. Student history is fixed. Evidence prefixes appear only in teacher inputs. Two replays pair the clear image with clear or degraded prefixes. The others pair the degraded image with a degraded prefix, and the clear image with a placebo prefix.
The path defines image and prefix corrections. We remove the image-aligned component from the prefix direction to avoid duplicate visual corrections. A deterministic gate weights the prefix correction using the placebo direction magnitude. One-sided projection forms from components aligned with the endpoint correction. The anchor separately regularizes the student toward and is not gated.
We also examined whether the gain can be explained by supplying the teacher with the additional context alone. The original Table 3 in Section 4.4 tests prefix selection through the no gate, random gate, and shuffled prefix controls. We added two objective controls in Section 4.4 of the revised paper, with both model sizes reported in Table 4. Direct endpoint OPD and PReViD (no anchor) were newly trained at 4B and 9B using their respective objectives. Student initialization and the training protocol are shared within each size. For each method, we used the same weights for three inference evaluations with different seeds. Vision-OPD and Ours retain the weights from their original training.
The two controls correspond to the two terms of the original Section 3.4 loss. We compare matching the constructed target with matching the teacher endpoint. The token loss is
.
The constructed target is matched in the first term. The anchor in the second term regularizes the student toward the teacher endpoint.
The endpoint distribution uses the clear image and clear evidence prefix. Four teacher replays, projection, and gating form the target . PReViD (no anchor) removes endpoint matching and retains only the first term. Direct endpoint OPD omits target construction and uses only the second term with unit weight. Full PReViD uses the two terms together. Results from the three inference evaluations for the target construction controls are provided in the supplement.
| Model | V* | Zoom | HR4K | HR8K | MME-EN | MME-CN | Avg |
|---|---|---|---|---|---|---|---|
| Qwen3.5-4B | |||||||
| Vision-OPD | 87.26±0.30 | 53.41±0.14 | 79.88±0.66 | 78.00±0.76 | 67.19±0.11 | 68.08±0.18 | 72.30±0.27 |
| Direct endpoint OPD | 88.83±0.30 | 56.73±0.18 | 80.79±0.19 | 79.00±0.33 | 70.73±0.06 | 68.71±0.19 | 74.13±0.08 |
| PReViD (no anchor) | 90.05±0.52 | 59.57±0.25 | 81.83±0.31 | 79.42±0.52 | 71.04±0.06 | 69.38±0.12 | 75.22±0.17 |
| Ours | 90.93±0.30 | 61.42±0.12 | 84.50±0.45 | 81.13±0.76 | 71.89±0.04 | 70.26±0.08 | 76.69±0.13 |
| Qwen3.5-9B | |||||||
| Vision-OPD | 90.75±0.30 | 58.70±0.31 | 84.67±0.26 | 82.54±0.75 | 71.30±0.04 | 68.20±0.11 | 76.03±0.15 |
| Direct endpoint OPD | 91.80±0.30 | 60.63±0.49 | 85.08±0.26 | 83.67±0.51 | 72.46±0.06 | 69.72±0.22 | 77.23±0.17 |
| PReViD (no anchor) | 92.50±0.30 | 62.09±0.38 | 85.63±0.38 | 84.50±0.33 | 73.74±0.07 | 70.12±0.10 | 78.09±0.11 |
| Ours | 93.54±0.30 | 64.34±0.18 | 86.13±0.13 | 84.84±0.19 | 74.38±0.05 | 70.42±0.04 | 78.94±0.06 |
PReViD improves over Direct endpoint OPD by 2.56 percentage points at 4B and 1.71 at 9B. The no anchor variant also outperforms Direct endpoint OPD. This supports an additional benefit from selective target construction with the same teacher context.
We retain the discussion of these methods and Masking-KD in Sections 2.1 and 2.2 of the original paper. Section 2.1 of the revised paper further explains that OPSD uses the same policy to compute distributions under different information contexts. Masking-KD changes the student prefix. PReViD adds evidence to the teacher context and replays the original response. The new target controls further test the benefit of our target construction.
W4 and Q1 on training parameters and teacher identity
Section 4.1 of the original paper gives the student sizes, initialization, training data, and hardware. Appendices A.1 and A.2 list the supervision inputs and generation procedure. We have added a consolidated hyperparameter table in Appendix A.1 of the revised paper. Table 5 includes the learning rates, optimizer, and update settings.
Section 4.3 of the original paper identifies the 9B PReViD checkpoint from Table 1 as the fixed teacher. However, the descriptions of the teacher parameter state in the main experiments were inconsistent. We apologize for the inconsistent descriptions in the original paper.
In the main PReViD experiments and ablations, the current student policy also provides teacher supervision. The two roles use shared parameters under different contexts. The student receives the full image and its own history; teacher inputs contain controlled images and evidence prefixes. Teacher distributions and reconstructed targets have their gradients stopped. Backpropagation acts only on the student distributions, so the teacher role follows the student's parameter updates. For Table 2, supervision comes from the frozen 9B PReViD checkpoint. The student is initialized from Qwen3.5-4B. Each setting uses its corresponding teacher to generate evidence prefixes. We have corrected the frozen teacher wording for the main experiments. The teacher and generator descriptions are now consistent across Sections 4.1 to 4.3 of the revised paper.
Section 3.4 of the original paper defines as the stored-rollout importance weight. Appendix A.1 of the revised paper makes its probability ratio explicit. It is the current student probability of the sampled token divided by its stored rollout probability. We use full fine-tuning and Adam. The original training seed is 42. The additional inference evaluations use different fixed seeds. The table below lists the settings.
| Parameter | Setting shared by 4B and 9B, except learning rate |
|---|---|
| Fine-tuning | Full fine-tuning |
| Optimizer | Adam |
| Adam coefficients | β₁=0.9,β₂=0.999 |
| Original training seed | 42 |
| Learning rate | 4B 5×10⁻⁷,9B 2×10⁻⁷ |
| Training epochs | 2 |
| Global question batch size | 64 |
| Microbatch size per GPU | 1 |
| Gradient accumulation | 4 steps |
| Weight decay | 0.01 |
| Warmup ratio | 0.05 |
| Maximum gradient norm | 1.0 |
| Student rollout | temperature=1,top-p=1 |
| Maximum prompt/response length | 8192/4096 tokens |
| Evidence prefix generation | Every training batch, greedy decoding, at most 64 tokens |
| Hardware/precision/sharding | 16 H20 GPUs (96 GB)/BF16/FSDP full shard |
| Distillation divergence/candidate support | JSD/Top-64 plus remaining probability mass |
| Anchor weight λ | 0.1 |
| Target shift clipping limit M | 20 |
| Projection stability term ε | 10⁻³ |
| Gate margin/logistic scale | 0/1 |
Q2 on the two MME-CN values and repeated inference evaluations
Section 4.3 and Table 2 of the original paper specify a fixed 9B PReViD teacher. This is a different training setup from the main experiments in Table 1, which use a teacher of the same model size. Section 4.2 of the revised paper clarifies that 4B PReViD in Table 1 uses the current student policy to construct teacher supervision with gradients stopped. Table 2 retains the fixed 9B PReViD teacher trained for Table 1. The table below reports three MME-CN inference evaluations for each setting, with the means and sample standard deviations. Section 4.1 of the revised paper describes the repeated inference protocol. Tables 1 and 2 have been updated separately.
| Experimental setting | Original inference evaluation | New seed inference evaluation 1 | New seed inference evaluation 2 | Mean and standard deviation |
|---|---|---|---|---|
| Table 1, 4B self-distillation with the current policy | 70.24 | 70.34 | 70.19 | 70.26±0.08 |
| Table 2, fixed 9B teacher distilling a 4B student | 70.24 | 69.85 | 70.31 | 70.13±0.25 |
Thank you for your detailed comments on baseline comparisons, path interpretation, and reproduction settings. We have added diagnostics and reproduction details to the paper, building on the original shared comparison and method definitions. The supplement also provides results from evaluating the official weights. Your suggestions helped us present the experimental evidence more fully and improve the clarity and reproducibility of the paper.
Thank you for your detailed review. We appreciate your comments “Low cost and solid controls” and “Clean, transferable design”. Below, we respond to your concerns and questions.
W1 on baseline results and reproduction settings
Section 4.1 of the original paper specifies the same training data, student initialization at each model size, and evaluation protocol for the trained methods. Appendix A.1 lists the supervision inputs for each method. Table 1 reports results under this shared setup. Each method retains its own supervision inputs and target construction.
Our shared training setup may differ from the full training procedures in the baseline papers. To check the sources of the reported results, we evaluated the official weights under our evaluation configuration. The table below compares results from the baseline papers, official weights, and models trained under our shared setup. “Retrained under our shared setup” refers to models trained for the original paper. For this revision, we retained those weights and added two inference evaluations with new seeds.
The means and standard deviations for official weights and our retrained models each come from three inference evaluations with fixed weights and different seeds. The supplement provides the three inference evaluations for Vision-OPD and VAD. Section 4.1 of the revised paper states this repeated inference protocol. Table 1 has been updated accordingly.
The official Vision-OPD weights give results closer to those reported in the VAD paper. For the 9B model, the original Vision-OPD paper and the VAD paper report average scores of 79.68 and 76.88, respectively. Under our evaluation configuration, the official weights score 77.89±0.18. Retraining under our shared setup gives 76.03±0.15. The original VAD paper reports an average score of 79.93. Its official weights score 78.03±0.06 under our evaluation configuration. Retraining under our shared setup gives 77.37±0.15. These differences may arise from training configurations, inference implementations, and hardware environments. Vision-OPD and VAD have clear motivations and simple supervision procedures that offer advantages in training cost. We use their evaluation benchmarks to compare the methods. PReViD achieves higher average performance in the comparison under shared training settings. It also adds evidence prefix generation and four teacher replays. These methods reflect a tradeoff between performance and training cost.