acceptodds
Withdrawn from ICLR 2027

AnchorPlay: Reliable Self-Play for Video Grounding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-play can expand training supervision for video grounding without additional human annotations, but it also risks reinforcing its own localization errors: agreement among predictions does not establish that an event's boundaries are correct. We introduce \ours, an end-to-end self-play framework that jointly optimizes a shared model as a questioner and a grounding solver. The central idea is to decouple question evolution from the construction of localization targets. A frozen reference model proposes evidence anchors, whose reliability is assessed by comparing predictions across transformed views after mapping them into a common coordinate system. Conditioned on accepted anchors, the questioner learns to generate queries matched to the solver's capabilities, while the solver learns to localize the corresponding objects or events. Both roles are updated in the same optimization step, allowing the curriculum to evolve around targets supplied by a reference model independent of the evolving policy. Reliability-weighted reinforcement learning emphasizes better-supported targets, while grounding-preservation regularization encourages retention of reliable localization behavior from the initial model. Experiments on temporal and spatio-temporal grounding benchmarks show improvements over initial checkpoints across both general-purpose and task-adapted backbones. Although self-play training uses only grounding tasks, \ours also improves performance on additional video and image understanding tasks, demonstrating transfer beyond its training objectives.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLM, video understanding, self-evolving

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zqxGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes anchorPlay, which uses one video model as both questioner and localizer for self-playing training. A frozen version of the reference model proposes intervals and boxes, which are checked across different views of the video. The accepted targets are used for reliability-weighted RL and a coordinate prediction loss. Questions change with the solver's ability. The paper reports temporal and spatiotemporal grounding results, plus transfer to other visual tasks.

Critical strengths
  1. This paper identifies the challenge of temporal grounding annotations. It proposes self-play training to avoid additional human-provided questions or localization annotations.
  2. It uses a frozen reference to keep the localization targets separate from the solver’s current predictions. The questions can still adapt to the solver’s ability, without using its own answers to define the targets.
Critical weaknesses
  1. The reference can be consistently wrong. The same frozen model proposes the target, checks it across changed sampling offsets, sizes and reflections, judges query validity, and supplies the coordinate targets. An early start, late end, or wrong subject may survive those checks and become supervision.
  2. Missing comparison with VideoSSR (CVPR 2026). It uses synthetic video tasks with known targets, including anomaly intervals, and reports temporal-grounding transfer with Qwen3-VL-8B. It is cited in Section 2 but not compared in the experiments.
  3. The anchor design seems only useful for single segment. What if the question needs to localize multiple segments? This method seems hard to generalize to multi-segment localizations.
  4. Some numbers are in consistent. Why in Table 8 the validity of Qwen3.5-9B on Charades is only 5.6%, but in Table 1 it can still achieve mIoU of 59.26.
  5. The training cost is unclear. The method adds anchor generation and checks across several reference views, then generates eight questions and eight answers for each valid question. The SFT + AnchorPlay model remains below the full OneThinker reference on temporal grounding. That could be a useful tradeoff if it saves enough annotation or compute, but the paper does not quantify this.
Questions for authors
  1. What will be the anchorplay result if initialized from OneThinker-8B.
  2. There is a dagger sign in Table 1 behind OneThinker-8B, what is the meaning of the dagger? There is no explanation for that.
  3. It is better to clearly write the name of the base model in the Table to avoid confusion.
  4. It seems that the anchorplay result is still far from the full annotated OneThinker-8B. One might question about the effectiveness. What if the method starts from a strong baseline like OneThinker-8B, would OneThinker-8B boost the performance?
  5. The anchor design seems only useful for single segment. What if the question needs to localize multiple segments? Would this method be applied to multi-segment localization? QVHighlights contains multi-segment localization questions, how would the method perform on this benchmark.
  6. It is quite interesting that the pure temporal grounding training can also enhance image grounding. But it is a little bit overclaimed, since the training data has SVTG task, which still contains spatial grounding.
  7. Table 5 needs checking. It says all ablations use 40 training steps, but the full model's scores (39.27 TG / 30.08 STVG) match step 60 in Table 7. The step-40 scores are about 37.64 / 27.07. I cannot tell whether the full model and the ablations were trained for the same number of steps.
Reviewer_riqxOpenReview ↗
Rating
1
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an end-to-end self-play framework for video grounding tasks. It first introduces a pseudo-label construction method in which a frozen reference model proposes anchors. An anchor reliability score is then computed to filter out anchors with low consistency. A shared model generates multiple questions conditioned on the anchors and answers them through localization. Questions of moderate difficulty are selected for RL training. Experiments on several models provide some evidence for the usefulness of the generated pseudo-labels.

Critical strengths
  1. The paper exploits unlabeled video data by pseudo-label construction and improves video grounding performance through reinforcement learning. The reported zero-shot gains on image and video segmentation tasks further suggest that the learned grounding capabilities can transfer beyond the training tasks.
Critical weaknesses
  1. The anchor construction pipeline is an important component of the method. However, the description of this part is unclear. For example, the paper does not provide the number of candidate anchors, the number retained after consistency screening or the prompts used to generate anchors. Moreover, key dataset statistics remain as placeholders, such as "x unique videos", lack the scale of training data. These missing limit the reproducibility of a central component of the method.
Questions for authors
  1. Could the authors report the total number of candidate anchors and the number retained after consistency screening?

  2. Could the authors provide the prompts used to generate questions from anchors? Could they also provide complete examples showing the anchor, generated questions, and validity decisions?

  3. In table1, the proposed ANCHORPLAY trained using RL is compared with OneThinker-SFT. It is unclear whether the gains from the template constraint of RL or the effectiveness of the generated pseudo-labels.

  4. In table 3, the caption states that both Qwen initializations improve across all eight evaluated splits. However, Qwen3-VL-8B shows performance decreases on five of the eight splits.

  5. The segmentation results on MeViS and ReasonVOS in Table 4 differ from those in Figure 4.

Reviewer_16uMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AnchorPlay, a self-play framework for video grounding without additional grounding annotations. A frozen reference model generates localization anchors, which are filtered using multi-view consistency. A shared model then learns to generate questions and localize their corresponding events through joint RL.

The authors also introduce reliability-weighted rewards and grounding-preservation regularization. Results show improvements on temporal and spatio-temporal grounding benchmarks across three model initializations, with some transfer to other visual tasks.

Critical strengths
  1. The motivation makes sense. Self-play can reinforce incorrect localization boundaries, and using a frozen reference with multi-view consistency is a reasonable way to reduce this problem. I like the separation between anchor reliability and question difficulty.

  2. The grounding results are quite promising, especially on ActivityNet, ANet-RTL and STVG. Table 5 also suggests that anchor screening and reliability weighting are important, although I have a concern about the ablation setup.

Critical weaknesses
  1. I'm not fully convinced that the improvement comes from better localization rather than better output formatting. Appendix Table 8 shows that Qwen3.5's valid output rate on Charades-STA increases from only 5.6% to 100%. This is a huge difference. The authors acknowledge this issue, but don't provide localization scores evaluated on the same valid-output subset. It is therefore difficult to tell how much actual grounding ability has improved.

  2. There are some inconsistencies in the experimental reporting. Table 5 claims a matched 40-step budget for the ablations, but the full model scores (39.27 TG / 30.08 STVG) correspond to the 60-step checkpoint in Table 7. Also, Table 3 says both Qwen initializations improve across all eight referring-expression splits, while Qwen3-VL actually drops on several of them. These issues make me less confident in the reported conclusions.

Questions for authors

Can the authors report localization performance on a shared valid-output subset, or apply the same coordinate-recovery parser to both models? This is especially important for Qwen3.5.

Could the authors clarify the training steps used in Table 5? Why do the full model numbers match the 60-step checkpoint if all ablations use 40 steps?

How well does multi-view consistency correlate with actual anchor accuracy? I noticed Appendix E mentions a labeled audit set, but I couldn't find quantitative results. A reliability calibration analysis would be helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.