acceptodds
Withdrawn from ICLR 2027

OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Privileged on-policy distillation improves multimodal reasoning by allowing a teacher to evaluate student trajectories using rich, training-only visual evidence. Both models score these trajectories while conditioning on the same student-generated prefix. When a student misinterprets an image early in a response, this accumulating erroneous rationale eventually pulls the teacher away from its visual evidence. The teacher and student converge on the same hallucination, causing standard cross-model supervision to collapse precisely where correction is most needed. We find that the teacher's visual corrective preference is not lost under this misleading agreement. Comparing the predictions of the identical teacher given the real image and a visual null reveals that the privileged evidence still pushes the model toward the correct interpretation. We introduce OPD-Aha, which reconstructs the distillation target directly from this isolated visual preference rather than relying on the fragile teacher-student discrepancy. This reconstructed target aggressively suppresses continuations that contradict the image. Trained with this objective, students learn to naturally interrupt their own flawed reasoning with reflection tokens such as "wait" and "actually". After reflection, subsequent generation relies less on the accumulated erroneous text and more on the visual evidence. Correcting these trajectories mid-generation fundamentally alters the reasoning process, yielding broad and consistent improvements across diverse fine-grained perception and complex multimodal reasoning benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal learning, vision-language models, on-policy distillation, self-correction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aRhcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how erroneous student-generated prefixes can override privileged visual evidence during multimodal on-policy distillation. OPD-Aha contrasts the same frozen teacher under a localized evidence crop and a visual null, then reconstructs the distillation target by amplifying their log-probability difference. Distillation still uses unchanged student prefixes. Experiments report improved visual accuracy and reasoning transfer, accompanied by more frequent self-interruption and renewed visual support.

Critical strengths

The paper is clearly written and identifies an interesting failure mode. The prefix-length diagnostics, token-shuffling controls, alternative visual nulls, and qualitative examples provide informative analyses. The reported improvements over Vision-OPD are consistent across six visual benchmarks at both evaluated scales.

Critical weaknesses
  1. Target stability and reconstruction-strength selection are insufficiently characterized. Equation (7) can concentrate the reconstructed target qtq_t on tokens with very small null probabilities. Please specify any numerical safeguards. Figure 3(d) tests only β=1,2,3,4\beta=1,2,3,4, with no account of validation-based selection or behavior beyond this range. Table 2 fairly fixes both reconstruction signals at β=1\beta=1, but should also test whether its conclusion holds at the main setting β=4\beta=4 or across a shared tuning grid.
  2. The additional training cost is not reported. OPD-Aha requires two teacher evaluations rather than the single teacher branch of Vision-OPD. Please report training throughput, GPU-hours, and peak memory.
  3. The reflection analyses are suggestive but do not establish causality. Please specify the reflection-token set, diagnostic sample sizes, and matching procedure for non-reflection positions. Controlled suppression or insertion of reflection markers, with matched continuation budgets, would help distinguish causal visual recovery from correlated changes in response style or length.
  4. Table 3 shows substantial post-training transfer regressions for the 4B GRPO and Vision-OPD baselines, rather than demonstrating progressively worsening training. Please clarify their checkpoint selection and evaluation settings.
Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Author Response (W1): Target Stability and the Choice of β

We thank Reviewer aRhc for four concrete requests. We address target stability, training cost, reflection diagnostics, and transfer evaluation below.

W1: Where Eq. (7) Is Evaluated, What Bounds It, and How β Was Chosen

  • Support and normalization (Table 4). Eq. (7) is not evaluated over the full vocabulary. The two teacher passes return log-softmax outputs; for the student's top-100 tokens we form (1+β)log⁡pt+(v)−βlog⁡pt0(v)(1+\beta)\log p^+_t(v) - \beta\log p^0_t(v) directly in log space, so no probability is ever divided. The remaining vocabulary enters as one pooled tail whose mass is the complement of the retained mass (the retained log-mass is clamped below 0 so the tail stays positive). The target is then renormalized over these 101 entries. Two consequences: a token outside the student's top-100 cannot become a spike, because it has no individual entry; and the computation is bounded in log space for any β\beta.
  • Inside the retained set. A retained token with a very small null probability does receive a large weight: a ten-fold smaller pt0(v)p^0_t(v) at fixed pt+(v)p^+_t(v) multiplies its unnormalized weight by 10β10^{\beta}. We do not apply a per-token floor or a ratio cap, and we will say so explicitly in Appendix A. In practice training at β≤4\beta \le 4 is stable: token-level JSD and the actor gradient norm fall early and settle (Fig. 12), and we observed no divergence across the β\beta sweep. We will add targeted checks of small-null-probability cases and a log-ratio cap in the revision.

Comparison at the main strength. We trained the teacher–student target at β=4\beta=4 with the same privileged crop, data, and training recipe as OPD-Aha. Tab. R1a compares both targets at step 30 using the same scorer. The real–null target leads on all six benchmarks and both aggregate metrics. Thus, the advantage of the intra-teacher contrast also holds at the main reconstruction strength.

Tab. R1a: Teacher–Student versus Real–Null Targets at β=4 (Qwen3.5-4B, Step 30)

TargetV★ZoomHR-4KHR-8KMME-ENMME-CNMean₄Avg₆
Teacher–student90.658.285.381.575.571.678.977.1
Real–null94.862.287.486.078.173.182.680.3

Mean₄ averages the first four benchmarks, and Avg₆ averages all six.

Selection of β. We stopped at 4 because the accuracy increments had diminished, while the transient response-length increase grew with β\beta (Fig. 3c). We will clarify the data split and selection procedure in the revision.

Behavior beyond β=4. A further β=5\beta=5 run, evaluated at step 30 with the same scorer, shows no consistent accuracy improvement on the completed benchmarks (Tab. R1b). Responses also become longer, including late in training (Tab. R1c). These results support retaining β=4\beta=4.

Tab. R1b: Real–Null Reconstruction at β=4 and β=5 (Qwen3.5-4B, Step 30)

βV★HR-4KHR-8KMME-CN
494.887.486.073.1
594.287.685.371.2

Tab. R1c: Response Length at the Main and Higher Reconstruction Strengths

Length statistic (tokens)β=4β=5
Mean over training199229
Mean over the last ten steps171214

Takeaway: Log-space normalization addresses numerical computation, while concentration on small-null-probability tokens remains a separate stability question. The real–null target retains its advantage at the main strength, and stronger reconstruction brings longer responses without consistent accuracy gains.

AuthorsOpenReview ↗

Author Response (W2): Training Cost

W2: What the Second Teacher Pass Costs

  • Vision-OPD scores each student rollout once with the teacher; OPD-Aha scores the same rollout twice, with the crop and with the visual null. The student still generates eight rollouts per prompt (Table 4), so the extra work is one gradient-free teacher forward per rollout. Inference cost is unchanged: the student runs alone on the original image.
  • We measured both 4B arms of the full-image ablation on the same eight A100s over the same early-training measurement window:

Tab. R2: Training Cost of the Second Teacher Pass (4B, Measured Training Window)

4B, same hardwareRollouts / sMedian step (s)Null-teacher forward (median s / step)GPU-hours (measured window)Highest sampled memory / GPU
Standard target (β=0\beta=0, one teacher pass)1.45526010.5675.6 GiB
Real–null target (β=4\beta=4, two teacher passes)0.987756215.6479.2 GiB
  • Interpreting the cost difference. The null-teacher pass accounts for part of the measured step time. Longer responses also contribute to the total cost difference in this early window (Tab. R2b), consistent with Fig. 3c. Memory was sampled with nvidia-smi across all GPUs, and the table reports the largest observed sample during the measurement window. We will supplement these windowed measurements with full-run cost and peak-memory measurements for the main crop configuration in the revision.

Tab. R2b: Response Length and Timing Context for the Cost Measurements

MeasurementValue
Standard-target median response length145 tokens
Real–null median response length186 tokens
Null-teacher share of real–null step time8%
Device-memory sampling intervalApproximately 30 s

Takeaway: The extra teacher pass and longer student responses both contribute to the measured early-training cost. These windowed measurements will be supplemented with full-run measurements in the revision.

AuthorsOpenReview ↗

Author Response (W3): Reflection Diagnostics

W3: Lexicon, Matching, and the Intervention

Reflection lexicons. The reflection rate (Fig. 4b) counts wait, actually, however, look again, and looking again. The event analysis (Fig. 5) anchors on the first match of wait, hold on, looking again, look again, look more carefully, reexamine, doublecheck, now I see, made a mistake, misread, or closer inspection. Matching is case-insensitive and uses whole words. Reexamine and doublecheck also match with a space or hyphen.

Matched non-reflection positions. Each cue response is paired with a cue-free response from the same benchmark with the same final-answer correctness and enough remaining tokens. The control anchor has the same token offset as the end of the observed cue. Among eligible controls, we prefer least prior reuse, then matching category, then longer responses, with deterministic tie-breaking. Each curve subtracts the mean of the 16-token window preceding its anchor. We will include the diagnostic sample sizes in the revised figure captions.

Diagnostic quantities. For the emitted token yty_t, continuation support is log⁡pt0(yt)\log p^0_t(y_t) and visual-evidence support is ut(yt)=log⁡pt+(yt)−log⁡pt0(yt)u_t(y_t)=\log p^+_t(y_t)-\log p^0_t(y_t), evaluated at the same prefix. These diagnostics measure changes in textual and visual support around the cue.

  • Suppression (inference only, OPD-Aha 4B and 9B). On HR-Bench-4K and 8K we set the logits of wait, wrong and however (all capitalizations) to −∞ throughout generation. Both arms use our evaluation decoding (non-thinking, greedy, 8,192-token budget, seed 42) and the same LLM judge; the baseline runs the same code with a non-existent word suppressed. Tab. R3 reports the questions whose baseline answer used one of the three words, the only ones suppression can affect.

Tab. R3: Suppressing Three Reflection Words on HR-Bench-4K/8K

ModelBaseline acc. (%)Suppressed acc. (%)Net change, pp [95% CI]McNemar p
4B62.360.7−1.6 [−6.0, +2.8]0.60
9B64.363.8−0.5 [−5.7, +4.7]1.0
  • What suppression shows. Neither change is significant, and at 4B answers flip about as often as between two unsuppressed runs. Suppression also does not remove reflection: at 4B, alternative cues such as actually, check and look again still appear. The test therefore cannot show that these words are necessary for recovery.

  • Pilot. Inserting Wait versus Next at shared prefixes with a matched 2,048-token continuation budget yields similar accuracy with the real image and no advantage for Wait with a null image. The next-token real–null TV difference favors Wait, but its confidence interval includes zero at both scales (Tab. R3b). The pilot is small and its prefixes were not filtered for a verified perceptual error, so it remains inconclusive.

Tab. R3b: Next-Token Real–Null TV Difference in the Insertion Pilot (Wait − Next)

ModelTV difference95% interval includes zero
4B+0.064Yes
9B+0.045Yes
  • What we will claim. Neither intervention shows that the reflection words are necessary or sufficient for re-anchoring. We will revise Section 4.3 to describe reflection as a marker that accompanies renewed visual reliance rather than its driver, and keep the training-side result (Section 4.2), which does not depend on this question.

Takeaway: Both lexicons and the matching rule are now stated, and the matched-budget interventions do not establish causality, so we will present reflection as a marker of renewed visual reliance rather than its cause.

AuthorsOpenReview ↗

Author Response (W4): Checkpoints and Evaluation Settings behind Table 3

W4: Checkpoints and Evaluation Settings behind the Table 3 Regressions

  • Checkpoints. Every Table 3 model is the final checkpoint of its perception run (one epoch over Vision-OPD-6K, 65 steps at batch 96), with no selection on any math benchmark. The same checkpoints produce Table 1.

Evaluation mode and budget. Our initial MathVision evaluation uses non-thinking mode, following Vision-OPD and OPD-V, with a 4,096-token first-pass budget. Tab. R4a summarizes the related protocols.

Tab. R4a: Evaluation Settings Reported in Related Visual-Distillation Work

WorkReported settingRelevance to our evaluation
Vision-OPD [1], Section 4.1Qwen3.5 training and evaluation use non-thinking modeSupports the choice of non-thinking mode
OPD-V [2], Section 4.1 and Appendix ETraining and evaluation use non-thinking modeAlso uses the non-thinking convention
S²VOPD [3], Section 4.1Perception: greedy, 4,096 tokens. Mathematics: 24,576 tokens, temperature 0.3Uses a longer budget for math evaluation
Our initial MathVision passNon-thinking, 4,096 tokensA limited-budget transfer evaluation

Effect of a longer budget. Fresh base-model evaluations show that a longer non-thinking budget improves accuracy at both scales (Tab. R4b).

Tab. R4b: MathVision testmini Base-Model Accuracy under Non-Thinking Single-Pass Evaluation

Generation budgetQwen3.5-4BQwen3.5-9B
4,096 tokens43.847.0
32,768 tokens58.963.8
  • Evaluation settings, and an inconsistency we are correcting. The original first pass used non-thinking mode on MathVision testmini with a 4,096-token budget. Our logs show that the 4B base and Vision-OPD rows additionally had truncated answers regenerated at 8,192 tokens, while the GRPO row did not. Under the same first pass the picture is:

Tab. R4c: 4B MathVision testmini under Matched Evaluation Passes

4B MathVision testmini, accuracy %BaseGRPOVision-OPD
Non-thinking, 4,096 first pass (original logs)43.446.1 (+2.6)41.1 (−2.3)

Under the matched first-pass budget, Vision-OPD remains below the base model. The GRPO regression in Table 3 is a protocol artifact: GRPO is above base when both receive the same pass. We apologize for this inconsistency and will complete the uniform-protocol evaluation for all Table 3 methods in the revision.

Base and OPD-Aha under a shared thinking protocol. We evaluated the released base and OPD-Aha checkpoints on MathVision testmini with a shared thinking protocol, model-card style boxed-answer prompt, and 40,960-token budget (Tab. R5).

Tab. R5: MathVision testmini Accuracy under a Shared Thinking Protocol

ModelQwen3.5-4BQwen3.5-9B
Base64.575.3
OPD-Aha70.774.3

Scope of the transfer claim. Table 3 compares final checkpoints with the base models. It establishes post-training differences, rather than progressively worsening performance over training. We will revise the wording accordingly.

Independent evidence of transfer regression. S²VOPD [3] also reports in its Table 3 a MathVision decline for the Vision-OPD student alongside a MathVista improvement, using the same Vision-OPD-6K data and recipe (Tab. R6). This illustrates that perception gains need not translate to mathematical reasoning.

Tab. R6: Reported 4B Results from Table 3 of S²VOPD [3]

ModelMathVisionMathVista
Base65.375.8
Vision-OPD62.579.4
Other privileged methodsMathVision change, 4BMathVision change, 9B
OPSD−9.3−7.9
ZwZ−13.1−9.9

OPD-Aha improves MathVision at both scales under the original Table 3 protocol, and its gain under the shared thinking protocol is concentrated at the smaller scale (Tab. R5).

Takeaway: The reported transfer results compare final checkpoints under specific evaluation settings. Matching the evaluation passes removes the apparent GRPO regression, while Vision-OPD remains below the base model. These comparisons do not establish progressive deterioration during training.

References

[1] Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation, 2026.

[2] OPD-V: Visual On-Policy Self-Distillation with Modality Balance, 2026.

[3] Self-Supervised Visual On-Policy Distillation, 2026.

Reviewer_WtNLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a failure mode of privileged multimodal on-policy distillation: erroneous student prefixes can lead the teacher to follow the same incorrect interpretation, weakening corrective supervision. It proposes OPD-Aha, which reconstructs teacher targets by amplifying the prediction contrast between real visual evidence and a visual null. Experiments with Qwen3.5-4B and 9B show improvements over GRPO, V-Zero, and Vision-OPD on six visual evaluations, with positive transfer to multimodal reasoning tasks. Additional analyses associate the gains with stronger suppression of incorrect continuations and increased reflection followed by renewed visual reliance.

Critical strengths
  1. The paper identifies a concrete failure mode of privileged distillation. Its prefix-length analysis shows how erroneous student reasoning can override the teacher’s visual evidence. This provides useful insight into why clearer teacher inputs may still fail to produce corrective supervision.
  2. The method achieves clear gains across both model sizes. OPD-Aha outperforms Vision-OPD on all six visual evaluations, improving average accuracy by 4.0 and 3.2 percentage points at 4B and 9B, respectively (Table 1). Additional reasoning evaluations and reconstruction ablations further support its effectiveness.
Critical weaknesses

The methodological novelty over VCSD is unclear. VCSD already constructs distillation targets by amplifying the same teacher’s real-image versus content-erased prediction contrast, using the same core exponential-tilting form. The paper neither discusses nor compares against this closely related method, making it difficult to assess the contribution beyond the privileged-crop setting and reflection analysis.

The baseline results differ substantially from the official report. The Qwen3.5 model card reports MathVision scores of 74.6 and 78.9 for the 4B and 9B models, respectively, whereas Table 3 reports 54.9 and 45.5. Moreover, the reported 4B baseline outperforms the 9B baseline. This discrepancy raises concerns about the evaluation setup. The authors should verify these results and clearly explain any differences in benchmark splits, prompting, decoding settings, or scoring procedures.

The evidence does not establish that reflection causes renewed visual reliance. The analyses compare positions around naturally occurring reflection tokens, but do not control whether reflection occurs. Reflection may accompany successful correction rather than drive it. Without interventions, such as suppressing or replacing reflection tokens, the claimed mechanism remains insufficiently supported.

Questions for authors

What evidence shows that reflection drives renewed visual reliance rather than merely accompanying successful correction? Have the authors tested interventions that suppress or replace reflection tokens?

Could the authors verify the MathVision baselines in Table 3? The reported 4B/9B scores of 54.9/45.5 differ substantially from the official 74.6/78.9 and reverse the model-size trend. Please clarify the benchmark split, prompting, decoding, and scoring settings.

4 comments
AuthorsOpenReview ↗

Author Response (W1), Part 1: What OPD-Aha Contributes beyond VCSD

We thank Reviewer WtNL for three precise concerns. We address each concern below with the available experimental results.

W1: What OPD-Aha Contributes beyond VCSD, and Where the Gain Comes From

  • Same tilt, different problem. VCSD [1], concurrent with our submission, sharpens a teacher's prediction on the same image the student sees, by contrasting it with a content-erased image (its Appendix B, Table 7 also pairs the contrast with a crop teacher). OPD-Aha starts from a different observation: a teacher that holds privileged evidence (a localized crop) is still pulled onto the student's hallucination once the erroneous prefix grows. Its decision margin turns negative at about 50% prefix retention (Fig. 1a), so the standard privileged target collapses exactly where correction is needed. Our contribution is to show that the teacher's visual preference survives this collapse (Fig. 1b) and to turn it into a target that overturns the wrong continuation late in a rollout.
  • Mask-and-sharpen vs. overturn. VCSD applies its tilt inside a plausibility mask ({w:preal(w)≥0.1max⁡preal}\lbrace w : p_{\text{real}}(w) \ge 0.1 \max p_{\text{real}} \rbrace) at mild strength (α=1\alpha = 1): it redistributes mass among tokens the teacher already finds plausible. Under an erroneous prefix, the tokens that must win, the correct continuation or a reflection cue such as wait, are exactly the ones that linguistic momentum has pushed below such a cutoff, so a mask removes them before the tilt acts. OPD-Aha applies the tilt to the retained support without a plausibility cutoff and at β=4\beta = 4, because the strength needed to reverse a wrong teacher preference grows along the prefix (median threshold 2.24 in the first quartile to 5.55 in the last, Fig. 9b, Eq. 13). Section 4.2 shows the consequence: reflection cues win not because they are promoted but because the erroneous continuation is suppressed far more (Fig. 4a).
  • Why the teacher sees a crop at all. The crop is not a design choice of OPD-Aha. It is the privileged signal shipped with Vision-OPD-6K, the training set we share with Vision-OPD, GRPO and V-Zero in Table 1 (Appendix A.1). Every privileged method in that table receives the same crop; OPD-Aha only adds the visual null, which replaces that crop with its mean color. We kept the setting fixed so that Table 1 isolates the effect of the target: with the crop held constant, reconstruction adds +4.0 at 4B and +3.2 at 9B. Nothing in the method depends on the crop being a crop. The contrast is defined for any pair of teacher views (Eq. 2), and the full-image rows below are the same code with a different teacher view.

Takeaway: The shared contrastive rule does not by itself distinguish OPD-Aha from VCSD. Our contribution is to identify the collapse of privileged supervision under erroneous prefixes and use the surviving visual preference to reverse incorrect continuations. The crop supplies the shared privileged input.

References

[1] Visual Contrastive Self-Distillation, 2026.

AuthorsOpenReview ↗

Author Response (W1), Part 2: What OPD-Aha Contributes beyond VCSD

W1 (continued): What OPD-Aha Contributes beyond VCSD, and Where the Gain Comes From

  • Separating the privileged crop from the contrast (the "VCSD-equivalent" view). We trained the 4B student with the teacher given the full image instead of the crop, with and without real–null reconstruction, under the same data and recipe as the crop runs. The student's input is the full image in every row.

Tab. R1: Teacher View and Target Reconstruction (Qwen3.5-4B)

Teacher view during training (4B)TargetV★ZoomHR-4KHR-8KMME-ENMME-CNAvg₆
Full imageStandard (β=0\beta=0)83.849.186.980.963.163.571.2
Full imageReal–null reconstruction (β=4\beta=4)89.052.786.582.077.869.276.2
Privileged crop (Vision-OPD, Table 1)Standard (β=0\beta=0)89.059.583.480.574.671.676.4
Privileged crop (OPD-Aha, Table 1)Real–null reconstruction (β=4\beta=4)93.762.588.684.877.774.980.4

With the full-image teacher fixed, reconstruction improves performance and brings it close to standard privileged distillation. Adding the crop further improves fine-detail recognition. The full-image rows use step-30 checkpoints, while the Table 1 rows use final checkpoints. Tab. R1b uses matched checkpoints to quantify the reconstruction gain with the full-image teacher fixed and the additional crop gain with reconstruction fixed. The two ingredients are complementary. This is our contrast applied to the student's view, rather than a reproduction of VCSD's mask and schedule. We will add that comparison in the revision.

Tab. R1b: Separating Target Reconstruction and Teacher View at Matched Step 30

Teacher viewTargetV★ZoomHR-4KHR-8KMME-ENMME-CNAvg₆
Full imageStandard (β=0)83.849.186.980.963.163.571.21
Full imageReal–null (β=4)89.052.786.582.077.869.276.18
Evidence cropReal–null (β=4)94.862.287.486.078.173.180.27
Controlled comparisonFixed factorAvg₆ gain (pp)
Standard → real–null targetFull-image teacher+4.97
Full-image → crop teacherReal–null target, β=4+4.09

All rows use Qwen3.5-4B at step 30 and the same scoring protocol. Avg₆ is computed from unrounded benchmark accuracies.

  • Cost of a full-image teacher. The full image requires substantially more teacher input tokens than the crop (Tab. R1c). Because reconstruction evaluates both real and null views, teacher input size matters for training cost. The full-image ablation addresses attribution of the gain, while the crop remains the standard privileged setting.

Tab. R1c: Teacher Input Size and Measured Full-Image Timing

MeasurementValue
Full-image teacher, mean visual tokens3,289
Crop teacher, mean visual tokens430
Full-image / crop token ratio7.65×
Full-image null-teacher pass62 s
Full-image training step775 s

Takeaway: The crop supplies privileged evidence, while reconstruction improves how that evidence supervises the student. Their benefits are complementary. We will clarify the shared scoring rule and distinct mechanism relative to concurrent VCSD.

AuthorsOpenReview ↗

Author Response (W2, Q2): MathVision Baselines and the Corrected Protocol

W2 & Q2: Why Table 3's MathVision Baselines Differ from the Model Card, and the Corrected Protocol

  • Different mode and budget. The model card's 74.6 / 78.9 are thinking-mode scores with a long generation budget. All models in our paper, following Vision-OPD and other visual on-policy distillation work, run in non-thinking mode (enable_thinking=False). Table 3 evaluates MathVision testmini with a 4,096-token response budget and rule-based answer matching. The 1,024 tokens in Table 4 is the training rollout length, not the evaluation budget.
  • The 4B > 9B inversion is a protocol inconsistency on our side, not a property of the models. In our original logs, under the same 4,096-token first pass, 9B (46.1%) is already above 4B (43.4%). The 54.9 reported for 4B came from an additional pass that regenerated truncated answers with an 8,192-token budget; the 9B row did not go through that pass. The 45.5 printed for 9B matches neither this log nor any valid accuracy on this evaluation set; we could not trace it to an evaluation run. We apologize for the inconsistency and will replace every MathVision row of Table 3 with a single protocol (non-thinking, 32,768-token budget, no regeneration pass).
  • Longer budgets lift both sizes and restore 9B > 4B. Base and OPD-Aha (released weights) were re-evaluated on the same evaluation set with the same pipeline; the thinking rows use the model-card style boxed-answer prompt and a 40,960-token budget.

Tab. R2: MathVision testmini, Re-evaluated under Different Protocols

MathVision testmini, accuracy %Qwen3.5-4BQwen3.5-9B
Non-thinking, 4,096 first pass (base)43.847.0
Non-thinking, truncated answers regenerated at 8,192 (base)55.357.9
Non-thinking, 32,768 single pass (base)58.963.8
Thinking, 40,960 (base)64.575.3
Thinking, 40,960 (OPD-Aha)70.7 (+6.3)74.3 (−1.0)
Model card (thinking)74.678.9
  • What this changes about the claim. Under the uniform thinking protocol, OPD-Aha's gain at 4B is clear and the 9B difference is within noise. This matches Table 3, where the 9B gains on all math benchmarks are small. The remaining gap to the model card reflects prompt, image resolution and sampling settings in addition to budget.

GRPO and Vision-OPD under matched evaluation. The Table 3 models are final checkpoints from perception training, with no selection on math benchmarks. The original base and Vision-OPD rows received a regeneration pass for truncated answers, whereas GRPO did not. Under a matched first pass, GRPO exceeds the base model while Vision-OPD remains below it (Tab. R2b). Vision-OPD also remains below base after regeneration (Tab. R2c).

Tab. R2b: Historical 4B MathVision Accuracy under the Same Non-Thinking First Pass

BudgetBaseGRPOVision-OPD
4,096 tokens43.446.141.1

Tab. R2c: Historical 4B MathVision Accuracy after Regenerating Truncated Answers

Retry budgetBaseVision-OPD
8,192 tokens54.950.3

Independent evidence of transfer regression. S²VOPD [1] also reports in its Table 3 a MathVision decline for the Vision-OPD student alongside a MathVista improvement, using the same Vision-OPD-6K data and recipe (Tab. R2d). This supports treating perception gains and mathematical transfer as separate outcomes. Final-checkpoint comparisons establish post-training differences, rather than progressive deterioration during training.

Tab. R2d: Reported 4B Results from Table 3 of S²VOPD [1]

ModelMathVisionMathVista
Base65.375.8
Vision-OPD62.579.4

Takeaway: The original MathVision comparison mixed evaluation passes, and absolute scores also depend on decoding mode and budget. Matched evaluations restore the base-model size ordering and show that OPD-Aha’s reasoning-transfer benefit depends on model scale.

References

[1] Self-Supervised Visual On-Policy Distillation, 2026.

AuthorsOpenReview ↗

Author Response (W3, Q1): Interventions on Reflection

W3 & Q1: Testing Whether Interrupting the Continuation Causes Renewed Visual Reliance

  • We agree the observational evidence is not sufficient. Figs. 4–5 compare positions around naturally occurring cues. They show that visual support rises after a cue, but cannot separate "reflection drives recovery" from "recovery co-occurs with reflection".
  • Suppression (inference only, OPD-Aha 4B and 9B). On HR-Bench-4K and 8K we set the logits of wait, wrong and however (all capitalizations) to −∞ throughout generation. Both arms use our evaluation decoding (non-thinking, greedy, 8,192-token budget, seed 42) and the same LLM judge; the baseline runs the same code with a non-existent word suppressed. Tab. R3 reports the questions whose baseline answer used one of the three words, the only ones suppression can affect.

Tab. R3: Suppressing Three Reflection Words on HR-Bench-4K/8K

ModelBaseline acc. (%)Suppressed acc. (%)Net change, pp [95% CI]McNemar p
4B62.360.7−1.6 [−6.0, +2.8]0.60
9B64.363.8−0.5 [−5.7, +4.7]1.0
  • What suppression shows. Neither change is significant, and at 4B answers flip about as often as between two unsuppressed runs. Suppression also does not remove reflection: at 4B, alternative cues such as actually, check and look again still appear. The test therefore cannot show that these words are necessary for recovery.
  • Dynamics around the cues. Predictive entropy rises at wrong and then falls, with a smaller pattern at however and none at wait. Visual attention relative to matched controls becomes positive after wait and wrong, consistent with the delayed recovery of visual-evidence support in Fig. 5 (Tab. R3b).

Tab. R3b: Entropy and Visual-Attention Changes around Reflection Cues

DiagnosticCue / windowChangeStatistical evidence
Predictive entropyAt wrong, relative to preceding eight tokens+1.25p = 3e−9
Predictive entropyFollowing eight tokens after wrong−0.98p = 5e−7
Visual attention relative to matched controlswait, tokens 17–32+0.14395% CI [+0.098, +0.190]
Visual attention relative to matched controlswrong, tokens 33–64+0.12195% CI [+0.080, +0.167]
  • Pilot. Inserting Wait versus Next at shared prefixes with a matched 2,048-token continuation budget yields similar accuracy with the real image and no advantage for Wait with a null image. The next-token real–null TV difference favors Wait, but its confidence interval includes zero at both scales (Tab. R3c). The pilot is small and its prefixes were not filtered for a verified perceptual error, so it remains inconclusive.

Tab. R3c: Next-Token Real–Null TV Difference in the Insertion Pilot (Wait − Next)

ModelTV difference95% interval includes zero
4B+0.064Yes
9B+0.045Yes
  • What we will claim. Neither intervention shows that the reflection words are necessary or sufficient for re-anchoring. We will revise Section 4.3 to describe reflection as a marker that accompanies renewed visual reliance rather than its driver, and keep the training-side result (Section 4.2), which does not depend on this question.

Takeaway: The interventions do not establish that reflection markers cause visual recovery. Reflection accompanies renewed visual reliance, and we will revise the causal wording accordingly.

Reviewer_7RgvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses privileged multimodal on-policy distillation and its failure mode: when the student misreads the image early in a response, the error accumulates in the prefix, the crop-conditioned teacher is pulled toward the same wrong answer, and the teacher-student discrepancy that defines the standard distillation signal vanishes. The authors show that a correct-ward visual preference nonetheless survives when the same teacher is evaluated twice under the same prefix, once with the privileged crop and once with a mean-RGB visual null. OPD-Aha reconstructs the distillation target by exponentially tilting the teacher's real-image distribution along this contrast with strength β. The analysis sections show that the reconstructed target keeps the correct-answer probability, and that the trained student produces reflection tokens such as "wait" and "actually" more often, after which reliance on the accumulated text decreases and reliance on visual evidence recovers.

Critical strengths
  • The failure mode of privileged OPD is demonstrated with a clear experiment: the crop-conditioned teacher's decision margin flips negative once about 50% of the erroneous prefix is retained.
  • Ablations cover the divergence, probability-space versus log-space reconstruction, and the visual-null construction
Critical weaknesses
  • Related previous works are not fully addressed: VCSD (arXiv 2607.21556)/VA-OPD (arXiv 2605.21924). And the contribution of the privileged crop is not separated from the contribution of the contrast mechanism. For example, a variant that feeds the teacher the full image instead of the crop (the VCSD equivalent) is missing, so the source of the gain cannot be determined.
  • The diagnostic and analysis sections are understandable and the argument is persuasive, but how the reported values were obtained is not stated precisely, so confidence in those values is not yet sufficient. TV, margin gain, recovery, the shuffled visual signal, continuation support, visual-evidence support, the preceding baseline, matched positions, and the reflection-token list are all presented as prose without formulas or procedures.
Questions for authors

[question]

  • Is Recovery (%) in Figure 1a judged from the teacher's generated answer or from a likelihood comparison?
  • What is the exact list of reflection tokens?
  • In Figure 9a, as printed, the "Without reflection" curve lies above the "Before reflection" curve at every checkpoint, which reads as the opposite of the D.1 claim that the increase recedes at non-reflection states and remains elevated only immediately before reflection. Is the legend swapped, or should the text be revised?
  • β is tested only up to 4 and the increments shrink (0.8, 0.4, 0.4). What happens to accuracy, response length, and language quality at β of 5 or higher, and was there a reason for stopping at 4?
  • The abstract and contribution 2 rest on "real-null instead of the fragile teacher-student discrepancy", but the only evidence is the β=1 comparison in Table 2. Can the authors provide a β sweep (1 to 4) for the teacher-student variant so that the claim that the intra-teacher contrast is the key ingredient can be assessed on equal footing?

[minor]

The base model and Vision-OPD already produce reflection tokens in 13-15% of responses (Figure 4b). Training changes the timing and frequency of reflection, not its existence, so "Aha" in the title and "emergent" and "naturally interrupt" could overstate the finding.

4 comments
AuthorsOpenReview ↗

Author Response (W1): VCSD, VA-OPD, and Separating the Privileged Crop from the Contrast Mechanism

We thank Reviewer 7Rgv for a careful reading of the analysis sections. We answer the five questions and the minor point below, and give exact definitions for every diagnostic.

W1: VCSD, VA-OPD, and Separating the Privileged Crop from the Contrast Mechanism

We thank the reviewer for pointing out the overlap with VCSD and the connection to VA-OPD. VCSD is concurrent work, and both papers use the same anchored real–null exponential tilt as their core scoring rule. We will make this explicit in the revision. Our contribution is to identify how an erroneous student prefix can cause even a crop-privileged teacher to favor the wrong continuation (Fig. 1a), and to show how reconstruction can suppress that continuation, giving reflection tokens such as wait a relative advantage (Fig. 4a). VCSD also evaluates a crop-teacher variant in its Appendix B, Table 7. The matched-checkpoint comparison separates the gain from reconstruction with the full-image teacher fixed from the additional gain of the crop with reconstruction fixed (Tab. R1).

VA-OPD uses visual contrast at a different stage. It uses the visual advantage of the sampled token to weight the distillation loss. OPD-Aha uses the contrast across candidate next tokens to change the target distribution at the same prefix. This distinction matters when the teacher already favors an incorrect continuation: reconstruction can change which continuation the target favors.

Tab. R1: Separating Target Reconstruction and Teacher View at Matched Step 30

Teacher viewTargetV★ZoomHR-4KHR-8KMME-ENMME-CNAvg₆
Full imageStandard (β=0)83.849.186.980.963.163.571.21
Full imageReal–null (β=4)89.052.786.582.077.869.276.18
Evidence cropReal–null (β=4)94.862.287.486.078.173.180.27
Controlled comparisonFixed factorAvg₆ gain (pp)
Standard → real–null targetFull-image teacher+4.97
Full-image → crop teacherReal–null target, β=4+4.09

All rows use Qwen3.5-4B at step 30 and the same scoring protocol. Avg₆ is computed from unrounded benchmark accuracies.

Takeaway: OPD-Aha shares the contrastive scoring rule with VCSD and reconstructs the target distribution where VA-OPD weights sampled-token losses. The controlled comparisons show a benefit from reconstruction with the teacher view fixed and an additional benefit from privileged crops.

AuthorsOpenReview ↗

Author Response (W2, Q1–Q3), Part 1: Exact Definitions of the Diagnostics

W2 & Q1–Q3: Exact Definitions of the Diagnostics, and a Correction to Appendix D.1

  • Q1. Recovery is judged from the teacher's generated answer. For each failed student trajectory and each retained fraction, the teacher given the crop samples three continuations from the retained prefix; we parse the answer option from each and count a success when it matches the ground truth (an unparsed answer counts as a failure). Recovery (%) is the success rate over the 3N samples at that depth. The decision margin in Fig. 1a is a separate likelihood diagnostic, defined below.
  • Q2. Two fixed, case-insensitive, whole-word lexicons are used. The reflection rate (Fig. 4b) counts wait, actually, however, look again and looking again. The event alignment (Fig. 5) anchors on the first match of wait, hold on, looking again, look again, look more carefully, reexamine, doublecheck, now I see, made a mistake, misread or closer inspection (reexamine and doublecheck also match with a space or hyphen).
  • W2. We will add these definitions in a new Appendix F in the revision. The table below summarizes them.

Tab. R2: Definitions of the Diagnostics

QuantityDefinition
Recovery (Fig. 1a)Recovery(d)=1003N∑i=1N∑r=131[a^idr=ai]\mathrm{Recovery}(d) = \frac{100}{3N}\sum_{i=1}^{N}\sum_{r=1}^{3}\mathbf{1}[\hat a_{idr} = a_i], where a^idr\hat a_{idr} is the option parsed from the teacher's rr-th sampled continuation of the retained prefix hidh_{id} (unparsed counts as wrong).
Decision margin (Fig. 1a)mi(d)=ℓid(ai)−ℓid(ai−)m_i(d) = \ell_{id}(a_i) - \ell_{id}(a_i^-) with ℓid(a)=∣a∣−1log⁡pϕ(a∣Ii+,hid,answer cue)\ell_{id}(a) = \lvert a \rvert^{-1}\log p_\phi(a \mid I_i^+, h_{id}, \text{answer cue}); ai−a_i^- is the student's realized wrong option, scored after the same answer cue as letters alone and as letters with option text.
TV (Fig. 1b, top)12∑v∣p(v)−q(v)∣\tfrac{1}{2}\sum_v \lvert p(v) - q(v)\rvert between the two next-token distributions at the same prefix: teacher-crop vs. student, and teacher-crop vs. teacher-null.
Margin gain (Fig. 1b, bottom)M(pt+)−M(pt0)=ut(g)−ut(w)M(p^+_t) - M(p^0_t) = u_t(g) - u_t(w), where M(p)=log⁡p(g)−log⁡p(w)M(p) = \log p(g) - \log p(w) for the gold and realized wrong answer candidates g,wg, w at position tt: the correct-minus-wrong margin on the real view minus the same margin on the null view.
Shuffled visual signalThe entries of ut(⋅)=log⁡pt+(⋅)−log⁡pt0(⋅)u_t(\cdot) = \log p^+_t(\cdot) - \log p^0_t(\cdot) are permuted across candidate token identities before forming the target; magnitudes are preserved, token alignment is destroyed.
Continuation support (Fig. 5a)For the emitted token yty_t, log⁡pt0(yt)=log⁡pϕ(yt∣I0,ht)\log p^0_t(y_t) = \log p_\phi(y_t \mid I^0, h_t): how strongly the accumulated text alone supports the token.
Visual-evidence support (Fig. 5b)ut(yt)=log⁡pt+(yt)−log⁡pt0(yt)u_t(y_t) = \log p^+_t(y_t) - \log p^0_t(y_t) for the emitted token at the same prefix (Eq. 4 evaluated at the realized token).
Preceding baseline, event windowsResponses are aligned at the first cue. Each curve subtracts the mean of the 16-token window before the cue (the "change from preceding baseline" on the Fig. 5 axis); post-cue offsets are pooled into 1–8, 9–16, 17–32, 33–64.
Matched positionsEach cue response is paired with a cue-free response from the same benchmark with the same final-answer correctness and enough remaining tokens; the control anchor has the same token offset as the end of the observed cue. Among eligible controls we prefer least prior reuse, then matching category, then longer responses, with deterministic tie-breaking.
Reflection rate (Fig. 4b)% of rollouts containing at least one match from the Fig. 4b lexicon; each rollout counts once.

Takeaway: Recovery measures correctness of generated teacher answers, while the decision margin measures likelihood preference. Explicit formulas, fixed reflection lexicons, and a defined matching procedure specify how the diagnostics are computed.

AuthorsOpenReview ↗

Author Response (W2, Q1–Q3), Part 2: A Correction to Appendix D.1

W2 & Q1–Q3 (continued): Exact Definitions of the Diagnostics, and a Correction to Appendix D.1

Q3: Clarifying Fig. 9a and Correcting Appendix D.1

  • The curves show changes relative to the base model. Fig. 9a plots the change in reflection-token preference relative to the base model, with the axis labeled "Change from Base (nats)." Each position type is compared with its own base-model value. A higher curve therefore indicates a larger increase from that baseline, rather than a higher absolute preference for reflection. We will make the precise metric definition explicit in the revision.

  • Both position types remain above their base-model levels. The early increase at ordinary positions diminishes. Pre-reflection positions also decline after their intermediate peak, so describing them as having "no recession" would be inaccurate. Both retain a positive change relative to the base model (Tab. R2b).

Tab. R2b: Approximate Changes in Reflection Preference Read from Fig. 9a

Position typeTraining stepChange (nats)
Ordinary (no cue)10≈+3.3
Ordinary (no cue)40≈+1.8
Pre-reflection10≈+0.9
Pre-reflection30≈+1.3
Pre-reflection40≈+1.1
  • We will narrow the interpretation in Appendix D.1. The legend identifies the curves correctly, but our wording overstates what their comparison establishes. Fig. 9a does not show a larger increase at pre-reflection positions, and the changes alone do not establish which position type has higher absolute reflection preference. We will revise D.1 to state the trends above and add the absolute values for both position types. The separate evidence in Fig. 4a concerns the relative suppression of the erroneous continuation and the reflection cue. We thank the reviewer for highlighting this ambiguity.

Takeaway: Fig. 9a shows how reflection preference changes relative to the base model. Both position types remain above their baselines, and the figure alone does not establish a stronger absolute preference at pre-reflection positions. We will clarify the reference point and narrow the interpretation in Appendix D.1.

AuthorsOpenReview ↗

Author Response (Q4, Q5, Minor): The Teacher–Student Target at the Main Strength, β beyond 4, and Wording

Q4 & Q5: The Teacher–Student Target at the Main Strength, and β beyond 4

  • Q5. Matched strength. We trained the teacher–student target at β=4\beta = 4 with the privileged crop, same data and recipe as OPD-Aha, evaluated at step 30. The table includes the published real–null result and a separate step-30 evaluation with the same scorer as the new teacher–student run. We appreciate the request for a full sweep and will add the intermediate teacher–student strengths in the revision.

Tab. R3: Teacher–Student versus Real–Null Targets at Matched Strength (Qwen3.5-4B)

Qwen3.5-4B target and evaluationV★ZoomHR-4KHR-8KMME-ENMME-CNMean₄
Standard privileged target (β=0\beta=0, Table 1)89.059.583.480.574.671.678.1
Teacher–student, β=1\beta=1 (Table 2)91.659.284.579.374.971.878.7
Teacher–student, β=4\beta=4 (new)90.658.285.381.575.571.678.9
Real–null, β=1\beta=1 (Table 2)93.261.586.082.076.573.280.7
Real–null, β=4\beta=4 (OPD-Aha, Table 1)93.762.588.684.877.774.982.4
Real–null, β=4 (step 30, same scorer as teacher–student)94.862.287.486.078.173.182.6

Mean₄ averages the first four benchmark columns. The teacher–student target improves only slightly between the tested strengths, while the real–null target retains a clear advantage at the matched main strength (Tabs. R3–R4). This is consistent with Fig. 1b: late in an erroneous prefix, teacher and student distributions have nearly converged (TV 0.08), leaving little corrective direction for log⁡pt+−log⁡ptS\log p^+_t-\log p^S_t to amplify. The real–null difference retains a correct-answer direction at the same depth.

Tab. R4: Aggregate Target Comparisons

ComparisonTeacher–studentReal–null
Mean₄ gain from β=1 to β=4 (rows above)+0.2+1.7
Mean₄ at β=4, matched step 3078.982.6
Avg₆ at β=4, matched step 3077.180.3
  • Q4. β ≥ 5. We evaluated an additional β=5\beta=5 run at the same checkpoint and with the same scorer as the matched β=4\beta=4 run. Accuracy shows no consistent improvement, while responses become longer and remain longer late in training (Tabs. R5–R6). We stopped at the main strength because accuracy gains had diminished while response length and divergence from the anchor increased (Fig. 3c–d, Eq. 15). We will include these results and complete the language-quality assessment in the revision.

Tab. R5: Accuracy beyond the Main Reconstruction Strength (Qwen3.5-4B, Step 30)

βV★HR-4KHR-8KMME-CN
494.887.486.073.1
594.287.685.371.2

Tab. R6: Response Length under Different Reconstruction Strengths

βMean rollout length over training (tokens)
0 (standard target)123
1140
4199
5229
Length statistic (tokens)β=4β=5
Transient peak≈410 (Fig. 3c)542 (additional run)
Mean over the last ten steps171214

The β=4 peak is read approximately from the published Fig. 3c. The β=5 peak and the rollout summary statistics are supplementary measurements, not values tabulated in the paper.

Takeaway: The real–null target retains its advantage at the main strength. Increasing reconstruction strength further shows no consistent accuracy gain and produces longer responses.

Minor: Wording

We agree. The base model and Vision-OPD already emit cues in 13–15% of responses (Fig. 4b); training changes how often and, in particular, where they appear. We will replace "emergent" with "elicited", "naturally interrupt" with "learn to interrupt", and describe the change in timing with the event-aligned analysis (Fig. 5) separately from the change in frequency (Fig. 4b). We will also revise the naming and framing throughout the manuscript to reflect this distinction more accurately.

Takeaway: Training changes the frequency and timing of reflection already present in the base model. We will revise the terminology, naming, and framing accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank You for Your Feedback

Dear Reviewers,

Thank you for the time and care you have devoted to reviewing our work and for your thoughtful, constructive feedback. Your comments have helped us reconsider how we articulate our contribution and interpret the evidence supporting it.

In the revision, we will reflect more carefully on the contribution and positioning of our work, clarify its relationship to prior and concurrent methods, and refine our claims to better match the experimental evidence. We will also incorporate the additional analyses and clarify the evaluation protocols discussed in our responses.

We sincerely appreciate your guidance in helping us improve the paper.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.