acceptodds
Withdrawn from ICLR 2027

Partial-Observation-Aware Self-Distillation for Fine-Grained Visual Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

19 views

Abstract

Vision-language models often recognize fine-grained details more accurately in a relevant image crop than in the full image. Local-view on-policy self-distillation (OPSD) uses a crop-conditioned teacher to transfer this advantage to a full-image student. However, we uncover an unexpected localization–observation tradeoff: while local-view OPSD strengthens fine-grained observation, it can degrade the ability to localize relevant evidence under full-image inference. Through theoretical analysis and token-level empirical studies, we trace this failure to the partial-observation nature of local-view supervision: it assigns systematically negative optimization signals to tokens that depend on the omitted image context, disproportionately suppressing localization-related tokens. Based on this diagnosis, we propose Partial-Observation-Aware On-Policy Self-Distillation (PA-OPSD), which selectively attenuates unreliable strongly negative optimization signals and introduces a complementary box-guided teacher that restores localization guidance while preserving full-image context. Across fine-grained visual reasoning and localization-oriented benchmarks with 4B and 9B models, PA-OPSD consistently outperforms local-view OPSD and other on-policy baselines, recovering localization while preserving observation benefits. It requires no additional annotations beyond local-view OPSD and no inference-time visual operations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
On-policy self-distillation, Vision-language models, Fine-grained visual reasoning, Multimodal reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gZC2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a localization and observation tradeoff in visual on policy self distillation. A teacher conditioned on a relevant image crop can improve fine detail recognition while weakening localization in a student that receives the complete image. The authors connect teacher and student token probability ratios to coefficients in divergence gradients and analyze localization and observation spans in verified and model generated reasoning trajectories. Their proposed method, PA OPSD, attenuates strongly negative crop teacher log ratios through sigmoid gating and introduces a second teacher conditioned on the complete image with an annotated evidence box. The two signals define a fused target distribution for student distillation. Experiments use Qwen3.5 models with four and nine billion parameters on visual reasoning and grounding benchmarks. Component ablations, alternative teacher designs, annotation audits, and diagnostic analyses accompany the evaluation. The reported results suggest that the combined intervention preserves observation benefits while recovering localization performance without visual operations during inference.

Critical strengths
  1. The paper identifies a concrete failure mode of crop based distillation and evaluates it on dedicated grounding benchmarks, rather than relying solely on aggregate visual question answering accuracy.

  2. The experimental analysis includes component removal, alternative calibration rules, alternative localization teachers, and a human annotation audit. These provide useful evidence that both proposed components contribute within the tested setup.

Critical weaknesses
  1. Omitted context is confounded with other crop effects. Equation (5)'s mutual information interpretation lacks an established joint distribution linking the two input conditions. Cropping also changes resolution, tokenization, and framing. Appendix C checks parameter discrepancy under identical crops, not these effects. Without controlled observation changes, the analysis does not isolate omitted context as the cause of localization degradation.

  2. Negative gradient coefficients do not prove behavioral suppression. Appendix B derives a valid but nonunique scalar coefficient representation. Actual probability updates depend on the summed gradients, normalization, and interactions across tokens and contexts. Sections 4.3 and 6.4 infer suppression and correction from log ratios without measuring these updates. Their mechanistic interpretation therefore exceeds the derivation.

  3. Negative tail ratios do not identify unreliable supervision. Comparing tail probabilities for LOC and OBS omits category prevalence, OTHER tokens, and correctness. Verified references show some valid localization receives negative signals, but not that training tail tokens are mostly correct. Appendix D.1 also uses greedy decoding rather than policy sampling. Negative magnitude is therefore not established as a reliable reason to weaken correction.

  4. Calibration may preserve confident errors as well as valid localization. Equation (6) drives extremely negative corrections toward zero, and fusion normalization can alter individual signals. Section 6.4 usefully shows positive mean observation ratios on verified correct trajectories, but does not test adequate correction of genuinely incorrect tokens or preservation of individual valid tokens. Selective protection requires correctness conditioned evaluation within generated negative tails.

  5. A simpler box teacher alternative is missing. Tables 1 and 2 never evaluate the box teacher alone, so they cannot establish that retaining crop supervision is necessary. Table 3 supports calibration within the fused design, but not whether replacing the crop teacher avoids the need for repair. Comparisons with Section 2.1's selective distillation approaches are also absent. The full combination works, but its necessity relative to simpler privileged grounding remains unresolved.

Questions for authors

See Weaknesses.

Reviewer_2UxhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem from the existing method called OPSD, which utilizes cropped images and a related fine-tuned VLM as a teacher model to enhance the recognition/observation of the fine-grained object described in the question prompt. This paper observes a localization-recognition trade-off problem in this setting and proposes two calibration modules: Partial-observation-aware calibration and Localization guidance modules to mitigate such negative impact in this design.

Critical strengths
  1. The logic is clear and easy to follow the problem.
  2. The proposed method shows some strength compared to the existing OPSD method.
  3. The theory and some ablation studies prove the effectiveness of the proposed direction.
Critical weaknesses
  1. The observation is not exciting. It is easy to see that if the local view supervision is too strong for the student model to conduct alignment, the localization ability could decrease, since the localization information is mainly from the global view. So it is not surprising to have very low performance on localization without some balanced modules.
  2. The method is mainly based on the existing OPSD design, so the proposed method adds two complementary weight-balanced modules to ensure that training is not too biased toward fine-grained local details and does not lose the global ability. Based on this, this paper can be seen as an incremental calibration method to the existing structure, which is not sufficient to support a publication.
  3. The test models are mainly the Qwen series; since there are so many open-source VLMs, why not try different structures?
  4. The OPSD training is a little tricky. The original setting utilizes Qwen to localize the object first and then post-training the model with more local image data. Since the original model can localize the object, why not crop it and input it into Qwen again to recognize the object? Also, Qwen uses a pooling strategy on adjacent patches, which may lose small features. It could be discussed about it.

In a word, such OPSD training policy is more like engineering level post-training while lacking enough science. This paper is mostly based on such a setting and proposes two balanced weights to make sure the training is not biased, which is not novel for an ICLR paper.

Questions for authors

Please answer the weakness part.

Reviewer_kvWJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies local-view OPSD, represented by Vision-OPD, in which a teacher conditioned on a local image crop distills a student that receives the full image. The starting point is an empirical observation on MME-RealWorld: Vision-OPD outperforms GRPO on 37 of 39 subtasks, and the remaining two are both localization-oriented tasks. The paper calls this the localization–observation tradeoff. Based on the gradient form of the f-divergence, the sign of the teacher–student log-ratio determines whether a token is encouraged or suppressed. Under the approximation that the frozen teacher ≈ the current student, this log-ratio is approximately the negative of the conditional PMI between the next token and the cropped-out context. Token-level statistics on GPT-annotated LOC/OBS segments show that LOC tokens are concentrated in the tail of strongly negative signals.

Based on this, the paper proposes PA-OPSD, which has two components: a sigmoid gate that attenuates strongly negative log-ratios; an additional box-guided teacher operating on the full image with the box drawn on it, fused with the local-view teacher in log-ratio space.

Experiments on Qwen3.5-4B/9B show average gains of about 1.6–2.3 points across five fine-grained reasoning benchmarks. On GroundingME and Ref-Adv, PA-OPSD reverses Vision-OPD's degradation and brings performance back above the base model.

Critical strengths

The problem formulation is valuable. The paper shows that the partial observation of a privileged teacher can systematically harm behaviors that depend on the missing context. This perspective is relevant to the broader privileged-context OPSD line of work, including crop-based, high-resolution, and answer-conditioned variants, not just the setting studied here.

The method is practical. It requires no additional annotations and no visual operations at inference; it only modifies the distillation target. This makes it easy to adopt in practice.

Critical weaknesses
  1. The "theoretical analysis" is overstated.

  2. The evidence for the core phenomenon is weaker than the narrative suggests. The motivation for the entire paper rests on only two subtasks. According to Table 4, Vision-OPD scores 64.82 on the Localization category overall, versus 65.11 for the base model, which is essentially on par. On Vehicle Location, it even improves from 16.91 to 36.03. The so-called "degradation" exists mainly relative to GRPO. The only truly notable degradation is on GroundingME with the 9B model (49.75 → 34.08). This drop is unusually large, and the authors need to rule out non-capability causes such as output-format collapse or coordinate-format drift.

  3. Baselines and ablations are incomplete. Two simple alternatives are missing: Vision-OPD combined with GRPO/RL (GRPO happens to be stronger on localization), and using only a full-image teacher. There is no comparison with thinking-with-images methods (either training-based or training-free) or with region-to-image distillation methods, even though the paper cites these works.

Questions for authors

The 9B Vision-OPD model drops by about 15.7 points on GroundingME. What causes this drop: coordinate-format or output-parsing failures, or a genuine decline in localization ability? Please report the format error rate.

If GPT-generated trajectories are replaced with Qwen's own correct on-policy trajectories, does the conclusion in Fig. 2a that LOC tokens receive negative signals on average still hold?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.