acceptodds
Withdrawn from ICLR 2027

BAS-OPD: Budget-Aware Selective On-Policy Self-Distillation for Fine-Grained Multimodal Perception

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) often struggle with fine-grained visual perception when critical evidence appears in local regions. On-policy distillation (OPD) can transfer privileged regional knowledge into a single-pass full-image policy, but formulations typically supervise every eligible rollout. This implicitly treats teacher feedback as equally valuable, even though the student policy evolves during on-policy training and the supervision value of different rollouts need not remain uniform. We therefore formulate privileged OPD as an online rollout-selection problem to study which rollouts merit supervision and introduce Budget-Aware Selective On-Policy Self-Distillation (BAS-OPD). The student generates a full candidate pool, while BAS-OPD selectively assigns crop-teacher supervision using random, uncertainty-based, or learned utility-based selection. Because the target utility is observable only after teacher scoring, we adopt a lightweight predictor to estimate it beforehand from detached student-side rollout statistics. Only selected rollouts enter the crop-teacher forward pass during training, while inference remains single-pass and selector-free. Across seven benchmarks, learned selection outperforms random and entropy-based selection at the same ratio, showing that how teacher supervision is allocated matters in addition to how much is used. With Qwen3.5-4B/9B, our method achieves 81.60%/84.23% average accuracy, exceeding full querying by 1.84/2.04 percentage while using only 25% of eligible rollouts and scoring 15.78%/16.02% of teacher-scored tokens. Thus, supervising more rollouts does not necessarily improve downstream performance.

Primary area:
reinforcement learning
Keywords:
On-policy distillation, MLLMs, Budget-aware teacher querying

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6LFgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Budget-Aware Selective On-Policy Self-Distillation (BAS-OPD), a method for reducing the cost of privileged-crop supervision in multimodal on-policy distillation. Instead of querying the crop-conditioned teacher for every student rollout, BAS-OPD predicts which rollouts are most useful to supervise and queries the teacher only for the top-ranked subset.

The selector uses six statistics available before teacher evaluation, including token negative log-likelihood, entropy, response length, prompt length, and crop-area ratio. It is trained online to predict a handcrafted utility target combining student-teacher Jensen-Shannon divergence, top-k support overlap, and teacher confidence. The selector is discarded after training and therefore adds no inference overhead.

Experiments use Qwen3.5-4B and Qwen3.5-9B models trained for one epoch on approximately 3,120 image-crop pairs. The main configuration queries the teacher for 25% of the 224 candidate rollouts produced at each step. Evaluation covers seven multimodal benchmarks. The authors report that Learned-25 outperforms random, entropy-based, NLL-based, and linear selection strategies, while also exceeding the full-query baseline. The average improvements over full querying are 1.84 points for the 4B model and 2.04 points for the 9B model.

Critical strengths

Selecting rollouts before teacher evaluation is operationally meaningful. Unlike methods that compute all teacher outputs and subsequently reweight or mask losses, BAS-OPD genuinely avoids crop-teacher forward passes for unselected rollouts. The selector is lightweight and does not affect inference. If validated convincingly, this approach could reduce the teacher-side expense of multimodal on-policy distillation.

The empirical comparison includes random selection, entropy selection, several NLL heuristics, a linear model, budget sweeps, feature-removal experiments, and selector-objective ablations. The authors also analyze the utility concentration and gradients of selected examples. These experiments indicate that the selector learns a nontrivial ranking according to the proposed utility target rather than merely reproducing a simple uncertainty heuristic.

Critical weaknesses

This is the most consequential weakness. All training results are reported using seed 42, despite the relatively small margins separating the methods. Learned-25 exceeds the strongest alternative selectors by approximately 1.3 points and full querying by roughly 1.8 to 2.0 points. No standard deviations, confidence intervals, or repeated training runs are provided.

This is especially problematic because the method introduces several sources of randomness: sampled student rollouts, selector initialization, replay-buffer sampling, random exploration, and model optimization. Training is also extremely short, consisting of only one epoch and 111 optimization steps. Under these conditions, the reported differences could plausibly arise from run-to-run variation.

The ablations themselves raise further concerns about stability. Removing individual utility components or scalar features sometimes causes surprisingly large changes, including substantial drops on MMStar. Without repeated runs, it is impossible to determine whether these effects reflect genuine contributions or noise. The statistical test reported for utility rank bands does not address this issue because it evaluates examples produced by one trained selector rather than uncertainty across independent training runs.

Consequently, the paper has not established that Learned-25 reliably outperforms random selection, heuristic selection, or full querying. It only shows that this ordering occurred in one run.

The paper interprets its target as the “supervision value” of a rollout, but the target measures current student-teacher disagreement, support overlap, and teacher confidence. None of these quantities directly measures whether training on a rollout improves held-out performance.

The gradient analysis does not establish this connection. Rollouts with greater student-teacher disagreement will naturally generate larger distillation gradients. Showing that selected examples have larger or differently oriented gradients therefore largely restates the selection criterion. It does not show that these gradients are beneficial rather than noisy, redundant, or harmful.

Similarly, the ranking audit shows that the selector can predict its own handcrafted target. This validates predictor fitting, but not the target’s interpretation as causal supervision utility. A stronger validation would measure whether examples with high predicted utility produce larger reductions in held-out loss or larger improvements on an independently selected validation set.

The result that querying 25% of rollouts outperforms querying 100% is also insufficiently controlled. Reduced querying changes the total distillation strength, gradient variance, and number of supervised tokens. The full-query baseline apparently uses the same training schedule and optimizer configuration despite receiving substantially more supervision per step. Its poorer performance may therefore indicate overtraining or loss-weight mismatch rather than intelligent selection. A full-query baseline with a rescaled distillation loss, or a random-selection baseline matched for total teacher-scored tokens and optimization exposure, is necessary to isolate the effect of rollout identity.

Finally, the efficiency claim is narrower than the presentation suggests. BAS-OPD reduces crop-teacher calls but still generates all 224 student rollouts. The paper reports neither wall-clock time nor GPU-hours, so the end-to-end training savings remain unknown. The method should be described as improving teacher-scoring efficiency unless a meaningful total-compute reduction is demonstrated.

Questions for authors

What are the results across multiple independent seeds? Please report per-benchmark means and standard deviations for Full, Random-25, the strongest heuristic, and Learned-25. Given the small margins and short training runs, this evidence could materially change the assessment.

How was the full-query baseline tuned? Does Learned-25 still outperform full querying when the latter’s distillation-loss weight or number of updates is adjusted to match the effective supervision strength of the 25% condition?

What evidence shows that the proposed utility predicts causal training benefit rather than disagreement? For example, does predicted utility correlate with the reduction in held-out loss produced by an update on that rollout?

What are the end-to-end savings? Please report wall-clock time or GPU-hours for full querying and Learned-25, including student rollout generation, selector training, teacher evaluation, and backward passes.

Reviewer_cQg4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BAS-OPD adds budgeted rollout selection to regional-to-global on-policy self-distillation. A full-image student generates candidate responses, and a lightweight predictor selects which responses receive crop-conditioned teacher supervision. Its utility target combines student-teacher disagreement, teacher confidence, and token-support overlap. The selector is updated online from queried responses and removed at inference. Experiments use Qwen3.5-4B and 9B across seven visual benchmarks.

Critical strengths
  1. Table 3 compares learned selection with full querying, random selection, entropy/NLL heuristics, and a linear predictor. The reported advantages over full querying at both scales support a practical benefit beyond merely reducing teacher calls.
  2. Selection occurs before teacher scoring, so it reduces actual teacher work rather than simply masking an already-computed loss. Inference remains single-pass without crops or a selector.
Critical weaknesses
  1. Dependence on privileged regional supervision is insufficiently characterized. Section 3.1 uses 3,120 preprocessed full-image/crop pairs from an existing dataset. Appendix B.3 further shows full-scene training inputs with dataset-provided bounding-box overlays and paired evidence crops. Although this does not imply a segmentation model is needed at inference, the training pipeline presupposes useful localized evidence. The experiments do not establish how the selection advantage behaves with inaccurate, incomplete, automatically proposed, or missing crops. The crop-area feature ablation in Table 4 does not test these conditions. This limits confidence that the benefit extends beyond the particular regional-supervision pipeline.

  2. The contribution appears incremental and its broader significance is not yet demonstrated. The method retains regional-to-global OPD and adds a predictor of a handcrafted utility proxy. The policy controls and component ablations are valuable, so the concern is not an absence of experimental support. Rather, the evidence remains concentrated on one prepared dataset, two scales of Qwen3.5, and seed 42. Predicting the chosen utility and inducing different gradient directions do not by themselves demonstrate a generally better learning signal. Independent training runs and validation with another backbone or regional-supervision source are needed to distinguish a robust selection principle from a useful engineering improvement to this pipeline.

Questions for authors
  1. How were the evidence crops and bounding boxes obtained, and what supervision or preprocessing cost is needed for a new dataset? Does the advantage over random and full querying persist with automatically proposed or perturbed crops, and without the student-side bounding-box overlay?
  2. Could you repeat the main learned/random/full-query comparison across independent training seeds and another backbone or crop-supervision source, keeping teacher budgets matched?
  3. What is the end-to-end training saving after accounting for full-pool student generation, optimization, selector updates, and regional-data preparation? Teacher-call and scored-token reductions alone do not establish the total efficiency gain.
Reviewer_W2gNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies on-policy self-distillation for fine-tuning MLLMs to perform better on fine-grained visual perception, under a limited budget of teacher evaluations. The proposed methods estimate the utility of a student rollout to decide whether or not to run the teacher and produce a supervised loss target. Since this utility is only known once the teacher is evaluated, the paper proposes to learn a lightweight predictor to estimate the utility to decide at training time which rollouts to score. Among several baselines, the trained methods works best and improves over competing OPSD and alternatives methods.

Critical strengths

S1: The most interesting insight of this work concerns the results presented in Sec. 3.3 that unveil that a budget of 25% leads to higher improvements than using all student rollouts for training (100%). It suggests that more teacher supervision is harmful rather than universally beneficial, and a selective supervision strategy can obtain higher gains. S2: The experiments are extensive and analyize the proposed approach from different angles that expose interesting insights, such as S1.

Critical weaknesses

W1: The main weakness of the paper is its focus and narrative around the budget constraint.

  • The paper reads as if it is an efficiency consideration, especially because of the wording of "budget". Yet, there is no proper runtime of speedup analysis of inducing such a budget constraint. This is also acknowledged in the limitations section.
  • The notion that any tangible "speedup" can be obtained from this method is rather questionable as student rollouts are expensive (autoregressive generation), and teacher evaluations are, relatively speaking, cheap (parallel evaluation). Without a proper analysis of what actual speedup can be obtained, this motivation would not easily hold up.
  • Due to the nonetheless interesting insight of lower budget leading to better performance (S1), the paper could have benefited from a much stronger focus on this aspect. A better narrative would propose that training on all student rollouts is harmful, and better results can be achieve from being selective. This opens many more interesting directions, as now the goal is not to fix a predetermined "budget" but instead decide, in an online manner, which samples to supervise and train on, i.e., a variable "budget" optimized for utility, not an artificial constraint.

W2: Some of the experiments are not conclusive as comparisons are not on equal grounds or could be discussed better.

  • In Tab. 1, interesting comparisons with "Thinking with images" agentic models do you clearly state the base models of these methods. This is essential, as the base model used for this paper, Qwen3.5, already outperforms these specialized models. This makes these comparisons meaningless. It would have been better to provide some direct comparison, e.g., by applying BAS-OPD on the base models of some of these competitor methods, such as ZwZ (Qwen3-VL), Thyme (Qwen2.5-VL), DeepEyesV2 (Qwen2.5-VL), TreeVGR (Qwen2.5-VL) [A]. There is also Vision-OPD [B] that has the same base model, Qwen3.5. The table should be fully transparent on which base models each method uses.
  • A broader evaluation on more models (related to the direct comparison) is relevant to show that BAS-OPD generalizes well to different models. It is not clear whether the effect of 25% budget being best is an isolated observation for Qwen3.5, or if it generalizes.
  • There should be a much more extended discussion on why a budgeted OPSD outperforms 100% budget in the first place as it is counter-intuitive. If the utility is low, the update gradient is also weak. Hence, it is not obvious why 25% budget is better.

W3 (Minor): Some details about the on-policy self-distillation setup are not clear.

  • The student obtains the full image while the teacher has access to the crop. Is the student expected to predict the crop coordinates or only the final answer? Does the teacher provide any guidance to where the object is located in the image, or only what is the GT answer? If it's the latter, what is the difference between giving the teacher access to the crop vs. just telling it the correct answer?

[A] Wang et al., Traceable evidence enhanced visual grounded reasoning: Evaluation and method, ICLR 2026
[B] Yuan et al., Vision-opd: Learning to see fine details for multimodal llms via on-policy self-distillation. arXiv 2026

Questions for authors

The questions directly relate to the mentioned weaknesses. Please consider addressing them in detail.

  • Is the intention for this approach to obtain speedup gains with a fixed budget? Why not lean more into the performance improvement and analysis around it?
  • Can you explain why 25% budget is better than 100%?
  • How does BAS-OPD perform in fair comparisons with other fine-grained recognition methods as mentioned above?
  • Can the same observation of reduced budget leading to better results also be made on other models or even another model family?
  • Could you please clarify the teacher setup?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.