acceptodds
Withdrawn from ICLR 2027

Pixel-OPSD: Beyond Scalar Rewards for Pixel-Level MLLMs via On-Policy Self-Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

53 views

Abstract

Pixel-level multimodal large language models must preserve a region's meaning in language and its extent in pixels. Scalar outcome rewards assess the final prediction but do not specify how to correct its semantic and spatial errors. We introduce Pixel-OPSD, which shifts pixel-level learning from scalar-reward reinforcement learning to on-policy self-distillation (OPSD). Our central idea is to make evidence-conditioned self-teaching responsive to both the semantic state of a generated trajectory and the spatial effects of its mask codes. Reliability routing adapts semantic correction to the sampled description through verified replacement, on-policy prefix refinement, or retention. Evidence-Guided Credit Assignment (EGCA) grounds mask-code credit in changes between decoded partial masks, identifying contributions that a terminal score obscures. These two mechanisms make OPSD sensitive to how a trajectory should be corrected and where spatial credit should be assigned. Supervised mixing supports this self-teaching process by anchoring grounding and descriptive completeness to human annotations. Experiments demonstrate consistent improvements over the baseline across region understanding and localization. Pixel-OPSD thus advances pixel-level learning beyond scalar outcome feedback through trajectory-adaptive semantic correction and spatially localized credit, with all teaching mechanisms confined to training.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Referring Segmentation, On-Policy Self-Distillation, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CkVnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Pixel-OPSD, an on-policy self-distillation framework for pixel-level multimodal large language models. It addresses the limited granularity of scalar outcome rewards through reliability-routed caption correction, which selects replacement, refinement, or retention based on reconstruction reliability. Additional supervised anchors preserve grounding accuracy and semantic completeness. However, the joint training objective requires further clarification.

Critical strengths
  • The paper addresses an important limitation of scalar outcome rewards. Distinguishing semantic errors from spatial errors is particularly relevant when language and mask tokens are generated autoregressively.
  • Experiments cover referring segmentation, interactive segmentation, general grounding, and region understanding, providing reasonably broad evaluation.
  • Counterfactual evaluation offers an interesting approach to distinguishing coarse and fine mask-code contributions.
Critical weaknesses
  1. Insufficient analysis of computational overhead. Teacher-based caption generation, repeated localization verification, and counterfactual mask decoding introduce additional costs. GPU-hour reporting and compute-matched comparisons with CycleGRPO are needed.

  2. Limited hyperparameter sensitivity analysis. More systematic analysis is needed for τlow\tau_{\text{low}}, τhigh\tau_{\text{high}}, τver\tau_{\text{ver}}, α\alpha, and λe\lambda_e, particularly the routing thresholds and EGCA mixing coefficient.

  3. Unclear formulation and justification of the joint objective. Equation (15) combines advantage-weighted policy optimization, JSD-based distribution matching, and teacher-forcing supervision. The notation, loss weighting, and compatibility of these optimization signals are insufficiently explained.

Questions for authors
  1. How are ωs\omega_s, ωC\omega_C, and λalign\lambda_{\mathrm{align}} selected? How sensitive are results to these coefficients, given the potentially different loss scales and gradient magnitudes?

  2. Only the refinement branch appears to perform distribution-level on-policy distillation; replacement uses teacher-forcing NLL, while retention applies no auxiliary caption correction. How should the scope of ''on-policy self-distillation'' be understood across these branches?

Reviewer_fia3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Pixel-OPSD, an on-policy self-distillation (OPSD) framework for post-training MLLMs on pixel-level understanding and localization tasks. Following CycleGRPO, Pixel-OPSD adopts a caption-to-localization cycle for post-training. The key difference is that Pixel-OPSD augments scalar-reward policy optimization with fine-grained supervision through evidence-conditioned self-teaching. To achieve that, the authors develop reliability routing to adaptively select the caption correction strategy based on reconstruction reliability, and evidence-guided credit assignment (EGCA) to assign token-level spatial credit based on the contributions of individual mask codes. During training, they also adopt supervised mixing to balance semantic completeness and grounding accuracy. Evaluation shows that Pixel-OPSD generally outperforms CycleGRPO across the evaluated benchmarks.

Critical strengths
  • Well-motivated problem
  • Reliability routing is intuitive and empirically effective
Critical weaknesses
  • Reliability routing relies on an estimation for caption quality, which are determined by caption quality and locator capability. There could be cases where the cycle score is low but is due to locator capability, not caption quality.
  • The paper does not provide enough details on how the privileged evidence is constructed and how the self-teacher is implemented.
  • The counterfactual attribution depends on the reference codes, but the paper doesn't mention details of how reference codes are constructed.
  • From Table 3: (1) OPSD alone degrades localization performance compared to CycleGRPO; (2) EGCA provides limited improvements. Further analysis or ablations would help clarify their contributions.
Questions for authors

See critical weaknesses

Also, there're a few inconsistencies in the reference:

  • In the paper: Xin Lai, Zhuotao Tian, Yukang Chen, Lin Li, Shilong Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.

    • Google scholar: Lai, Xin, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. "Lisa: Reasoning segmentation via large language model." In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9579-9589. IEEE, 2024.
  • In the paper: Zhiliang Peng, Wenjun Wang, Li Dong, Yaru Hao, et al. Kosmos-2: Grounding multimodal large language models to the world. In arXiv preprint arXiv:2306.14824, 2023.

    • Google scholar: Peng, Zhiliang, Wenhui Wang, Li Dong, Yaru Hao, et al. "Kosmos-2: Grounding multimodal large language models to the world." arXiv preprint arXiv:2306.14824 (2023).
  • In the paper: Xiangtai Li, Haobo Yuan, Henghui Ding, Size Wu, Wenwei Zhang, Yining Li, Kai Chen, and Chen Change Loy. Omg-seg: Is one model good enough for all segmentation? In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.

    • Google scholar: Li, Xiangtai, Haobo Yuan, Wei Li, Henghui Ding, Size Wu, Wenwei Zhang, Yining Li, Kai Chen, and Chen Change Loy. "Omg-seg: Is one model good enough for all segmentation?." In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 27948-27959. IEEE, 2024.
  • In the paper: Nikhila Ravi, Valentin Gabeurion, Yuan-Ting Hu, Zhi Hu, et al. Sam 2: Segment anything in images and videos. In arXiv preprint arXiv:2408.00714, 2024.

    • Google scholar: Ravi, Nikhila, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr et al. "Sam 2: Segment anything in images and videos." In International Conference on Learning Representations, vol. 2025, pp. 28085-28128. 2025.
  • In the paper: Haoxuan You, Haotian Zhang, Zhe Gan, Xueqing Du, Bowen Zhang, Zhen Wang, Yu Cao, et al. Ferret: Refer and ground anything anywhere at any granularity. In International Conference on Learning Representations, 2024.

    • Google scholar: You, Haoxuan, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. "Ferret: Refer and ground anything anywhere at any granularity." In International Conference on Learning Representations, vol. 2024, pp. 57153-57180. 2024.
  • In the paper: Xueyan Zou, Jiahui Yang, et al. Seem: Segment everything everywhere all at once. In Advances in Neural Information Processing Systems, 2023.

    • Google scholar: Zou, Xueyan, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. "Segment everything everywhere all at once." Advances in neural information processing systems 36 (2023): 19769-19782.
Reviewer_GyYnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies joint region captioning and localization in pixel-level multimodal large language models. Building on SAMTok-4B and a region-to-caption-to-mask cycle, Pixel-OPSD introduces an evidence-conditioned self-teaching procedure with three reliability-dependent actions: replacing failed captions with verified teacher-generated alternatives, refining recoverable captions on student-generated prefixes, and retaining reliable captions. It also proposes Evidence-Guided Credit Assignment (EGCA), which uses counterfactual mask decoding and coarse/fine code contributions to modify token-level localization advantages. Additional grounding and caption-supervision streams anchor spatial alignment and semantic completeness. As formulated in the method section, the training objective combines these mechanisms with group-relative policy optimization. Evaluation covers referring segmentation, multi-round interactive segmentation, GroundingSuite, and DLC-Bench. The complete system improves the reported aggregate results over CycleGRPO, and the progressive ablation provides particularly encouraging evidence for reliability routing. However, the current comparisons leave uncertainty about the contributions of additional supervision and the more elaborate spatial-credit mechanism.

Critical strengths
  1. Reliability-dependent caption correction addresses a concrete failure mode and has meaningful empirical support. The replace/refine/retain design recognizes that a teacher conditioned on a fundamentally incorrect prefix may be unable to repair the trajectory effectively. Generating a fresh caption and verifying it through localization is a reasonable response to this failure mode, while prefix-level refinement preserves useful context in partially correct trajectories. Table 3 provides the strongest component-level evidence in the paper: adding reliability routing to “OPSD only” improves RefCOCO from 74.0 to 77.7, GroundingSuite from 64.4 to 69.0, and MR-PACO from 40.4 to 50.7. This makes routing a potentially useful contribution beyond simply applying a generic distillation loss.
  2. The complete training recipe improves complementary capabilities without changing the inference interface. The evaluation covers both language-side region understanding and spatial localization, including interactive segmentation. Relative to SAMTok-4B, the full system improves DLC-Bench average accuracy from 60.2 to 68.8, GroundingSuite overall gIoU from 56.9 to 68.8, and MR-PACO cIoU from 44.5 to 51.5. These results suggest practical value in the complete training recipe. Keeping the teacher and auxiliary credit-assignment machinery confined to training is also attractive, since the reported improvements do not require additional teaching procedures at inference.
Critical weaknesses
  1. The central methodological claim is inconsistent with the stated optimization objective, leaving the contribution insufficiently defined. The abstract and introduction characterize Pixel-OPSD as a shift from scalar-reward reinforcement learning to on-policy self-distillation. Section 1 further states that CycleGRPO is a separate baseline rather than the GRPO implementation underlying the method. However, Section 3.2 keeps caption advantages active for structurally valid captions, Eq. (12) adds EGCA credit to localization advantages, and Eq. (15) retains a shared group-relative policy functional across the training sources. Section 3.5 explicitly states that removing the auxiliary components recovers the shared GRPO objective. The equations therefore appear to describe GRPO-based policy optimization augmented with routed distillation, code-specific credit, and supervised losses. Such a hybrid can be valuable, but the distinction matters for assessing the claimed conceptual contribution and interpreting the “OPSD only” ablation. The paper needs a consistent account of the actual training algorithm and the exact components enabled in each experimental variant.
  2. The experiments do not adequately isolate the proposed mechanisms from additional supervision, and the evidence for EGCA is limited. Section 4.1 adds 10k supervised examples from RefCOCO and DAM to the 20k image-mask training set. The reported comparisons do not include CycleGRPO with the same supervised anchors and a comparable training budget. Consequently, the results establish the usefulness of the complete recipe more clearly than they establish the advantage of the proposed learning mechanisms over scalar-reward RL under matched conditions. This distinction is particularly relevant for referring segmentation: the full model’s gains over the original SAMTok backbone are only 0.3/0.3/1.2 points on RefCOCO/+/g, whereas the substantially larger gains over CycleGRPO partly recover that baseline’s degradation. The evidence for EGCA, a central claimed contribution, is also relatively weak. In Table 3, its incremental gains are 0.2 on RefCOCO, 0.2 on GroundingSuite, 0.1 on MR-RefCOCO, and 0.4 on MR-PACO. No run-to-run variability is reported, and there is no comparison against directly using the observed Shapley contributions without the learned evidence head. It is therefore unclear whether the full mechanism provides a reliable benefit that justifies its additional complexity.
Questions for authors
  1. Clarification of the actual objective and ablation definitions. Is (\mathcal{P}_{\theta}) in Eq. (15) the GRPO-style clipped policy objective, and does it remain active for captioning and localization in all reported variants? What exactly is enabled in the “OPSD only” row of Table 3? Please reconcile the implemented objective with the claim that GRPO does not underlie the proposed method.
  2. Evidence needed to attribute the gains under matched supervision and training resources. How does CycleGRPO perform when given the same 10k supervised examples, the same supervised anchor losses, and a comparable optimization and rollout budget? Please also clarify which baseline results were reproduced versus taken from prior reports, and whether the compared models use the same initialization and evaluation protocol, including the DLC-Bench evaluator and evaluation subset.
  3. Evidence for the necessity and reliability of EGCA. How does full EGCA compare with using the observed Shapley contributions directly, i.e., setting (\alpha=1) and omitting the learned evidence head? Are the incremental improvements in Table 3 stable across runs, and what additional training cost is incurred by counterfactual decoding and evidence-head training? These comparisons would help determine whether the proposed estimator contributes beyond direct decoded-mask credit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.