acceptodds
Withdrawn from ICLR 2027

Beyond Fixed Supervision: Alternating Self-Play for Region-Grounded Visual Retrieval-Augmented Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Region-grounded visual retrieval-augmented generation models are typically trained on fixed datasets of human-annotated questions, answers, and evidence regions. Such supervision cannot adapt to the model's evolving weaknesses, particularly limiting continued improvement in region-level evidence localization. We present Self-Play Visual RAG, a framework comprising a proposer, a solver, and an environment. The proposer generates questions, reference answers, and evidence regions directly from visual documents; the environment filters malformed or unsupported tasks; and the solver learns from accepted tasks. Rather than selecting tasks according to a predefined ability boundary, our final strategy prioritizes valid tasks that the current solver fails, automatically constructing a hard-task curriculum without additional human annotations of question-answer-evidence-region tuples. We evaluate on a document-disjoint set of 317 manually reviewed questions constructed from SlideVQA and ViDoSeek source pages. The final solver achieves 80.44% Soft EM and 22.74% [email protected], exceeding the best evaluated general-purpose vision-language baseline for each metric by 2.52 and 15.53 percentage points, respectively. The proposer also improves through self-play, reaching an 81.70% joint environment acceptance rate, 11.30 percentage points higher than the strongest zero-shot proposer.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Retrieval-Augmented Generation, Alternating Self-Play, Evidence Localization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_d5TfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an alternating proposer–solver framework for region-grounded visual RAG. The proposer generates questions, answers, and evidence regions, while the solver trains on validated tasks it currently struggles to solve. Experiments on 317 custom questions compare this approach with continual RL and ability-boundary self-play. The reported gains are mainly in evidence localization.

Critical strengths

None.

Critical weaknesses

The experiments do not sufficiently isolate or validate the proposed mechanism. The hard-task and ability-boundary variants change both the proposer reward and the solver’s task-selection range (Section 3.5, p. 5). Table 3 provides useful ablations, but does not disentangle failure-driven proposer training from downstream task filtering. In addition, human review rejects 20.75% of an audited sample of automatically accepted tasks (Appendix C, pp. 14–15). The paper does not assess validity specifically within the selected hard-task curriculum, leaving open whether solver failure consistently identifies useful learning opportunities rather than unreliable supervision.

Questions for authors
  1. Could you clarify whether the frozen-proposer continual-RL baseline uses the same failure-based filtering as the proposed method?
  2. Could you provide any human-review results comparing the validity of selected hard tasks with uniformly sampled accepted tasks? It would also be helpful to understand how sensitive the results are to using eight samples for difficulty estimation.
  3. Could you specify the exact [email protected] aggregation and its relationship to Section 4.1? If available, VISA comparisons with continual RL and both self-play variants under a shared protocol would help assess generalization, particularly with retrieval failures distinguished from localization errors.
Reviewer_iiryOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Self-Play Visual RAG for document QA with region-level evidence. A proposer generates questions, answers, and evidence boxes from pages; an automated auditor filters them; and a solver trains on accepted tasks. Tasks the solver fails receive higher proposer rewards, and the two models are updated alternately. The method is evaluated on held-out documents from SlideVQA and ViDoSeek for answer quality and evidence localization.

Critical strengths
  1. The paper addresses both answer accuracy and evidence localization, a relevant document-RAG objective.
  2. Matched-initialization controls and a human audit provide useful diagnostic evidence.
Critical weaknesses
  1. The motivation is unsubstantiated: the paper does not show that fixed supervision has saturated or that alternating self-play is necessary. A compute-matched fixed-proposer hard-mining baseline is missing.
  2. The method is largely incremental relative to PasoDoble [1] and VisRAG 2.0 [2]. The paper does not isolate a new learning principle beyond combining failure-based task generation with evidence-box supervision.
  3. Solver failure is an unreliable proxy for useful training tasks: it can reflect label errors or retrieval misses. The auditor accepts invalid tasks, yet the selected hard curriculum is not audited or tested for sensitivity to label noise.
  4. The main comparison omits specialized methods LAT [3] and VisRAG 2.0 [2]. The latter evaluates on SlideVQA and ViDoSeek, making the absence of a common-protocol comparison particularly consequential.
  5. The evaluation relies on custom same-source questions and a permissive localization metric; out-of-domain transfer is weak. The boundary comparison also changes reward and task selection together, preventing attribution of the gains.

[1] Better LLM Reasoning via Dual-Play, Findings of ACL 2026.

[2] VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation, ACM MM 2026.

[3] Look as You Think: Unifying Reasoning and Visual Evidence Attribution for Verifiable Document RAG via Reinforcement Learning, AAAI 2026.

Questions for authors

Please refer to the Critical Weaknesses.

I think this paper is far below the standard of ICLR. The authors should redesign or rewrite it.

Reviewer_Tu1VOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an alternating self-play framework called Self-Play Visual RAG, designed to address the limitation where static supervised data fails to adapt to a model's evolving weaknesses in region-grounded Visual RAG tasks. The framework consists of three components: a Proposer, an Environment, and a Solver. The Proposer generates questions, reference answers, and evidence regions directly from visual documents; the Environment filters these outputs using a combination of programmatic rules and semantic auditing by the LLM; and the Solver learns from the accepted, challenging tasks. The core contribution is a Solver-failure-driven task selection strategy, which dynamically assigns rewards to tasks based on the Solver’s current success rate. This automatically constructs an adaptive curriculum, allowing the model to continuously uncover and rectify its own deficiencies in fine-grained region localization, all without the need for additional human annotation. Experimental results on a human-verified test set of 317 document-disjoint samples demonstrate that this approach achieves a Soft EM score of 80.44% and an [email protected] of 22.74%, outperforming existing general-purpose multimodal baselines and conventional reinforcement learning methods.

Critical strengths
  1. The article points out the core problem of region-grounded visual rag: relying on fixed manual annotation data for supervised learning or reinforcement learning cannot adapt to dynamic changes in model capabilities. And ideas on how to generate effective tasks that consistently expose weaknesses in current models are presented.

  2. An Alternating Self-Play framework consisting of Proposer, Environment and Solver is proposed to realize a closed loop of task generation and model optimization. Different from previous strategies for designing difficulty boundaries, this paper introduces Solver-failure-driven proposer update to generate appropriate questions, prioritize effective tasks that cannot be solved by solvers, and achieve continuous self-evolution without additional manual annotation.

Critical weaknesses
  1. The method exhibits limited generalizability and poor out-of-domain performance. While the model achieves a Soft EM of 80.44% and an [email protected] of 22.74% on test sets following the SlideVQA/ViDoSeek style, its performance drops on the VISA dataset (comprising only 100 questions) to 11.00% Soft EM and 1.00% [email protected] (Appendix B). This suggests that the "hard tasks" generated by the Proposer are highly likely to be artifacts of domain bias within the training set, essentially serving as shortcuts or instances of overfitting to a specific document style (SlideVQA/ViDoSeek) rather than reflecting genuine region-level reasoning capabilities.

  2. The paper lacks a clear description of the computational overhead. The proposed method entails the alternating training of two 8B models, 8 sampling iterations per task, and the use of an external large-scale model (Qwen-VL-Max) to perform semantic auditing for each task. Given the extremely high cost of this entire pipeline, the absence of reported training times and resource consumption metrics (such as GPU-hours or API call costs) hinders an accurate assessment of the method's practical feasibility.

  3. Although the paper emphasizes that the self-play framework eliminates the need for additional human annotation, Stage 1 still requires human review to construct 500 examples for the Proposer and 193 examples for the Solver. The quality of these seed data samples fundamentally determines the subsequent self-play process; however, the paper provides no details regarding their construction or the quality control measures employed.

Questions for authors

Please refer primarily to the "weaknesses" section; there are two additional questions:

  1. What are the selection criteria for the Auditor model, and to what extent does it influence the results?

  2. During the self-play process, might the Proposer gradually learn to "exploit" the Auditor's weaknesses? How can this be prevented?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.