acceptodds
Under review as a conference paper at ICLR 2027

What to Trust, What to Learn: A Grounded Learning Curriculum for Self-Evolving Large Multimodal Models

Abstract

In the self-evolving paradigm of large multimodal models, a Proposer and a Solver derived from a shared backbone cooperate in a closed training loop: the Proposer generates questions from visual input, and the Solver's most frequent answer is treated as a self-defined “golden label”. This avoids the need for human labels but raises a fundamental question: how is self-evolution steered, and can its self-rewarding signal be trusted? We identify two failure modes: Proposer bias, where question rewards favor invalid or unhelpful tasks, and Solver bias, where agreement-based supervision reinforces shared errors as self-generated ground truth. Across seven checkpoints on POPE and HallusionBench, unanimous errors account for 14.5–76.7% of wrong majority answers, while entropy-targeted question rewards can further increase invalid question generation. We propose a grounded learning curriculum that separates what to trust from what to learn: a frozen detector grounds candidate questions in visual evidence and enables the construction of supported answer sets in place of Solver consensus; the Solver learns from the full supported set, while the Proposer earns credit only when matched trial updates yield greater Solver progress on separate probe images. No task-specific question–answer labels, teacher model, or answer judge are used, preserving the self-evolving setting while grounding its training signals directly in visual evidence. On eleven standard benchmarks, our curriculum raises Qwen3.5-9B's average accuracy from 75.58% to 78.61%, with the largest gains on CV-Bench () and MMBench (). The same curriculum improves Qwen2.5-VL-7B and Qwen3-VL-8B, with average gains of and points, respectively, showing consistent benefits across model backbones. Evidence-grounded targets outperform majority-vote targets by 2.43 points, and a learned Proposer provides an additional 1.30-point gain over a frozen one.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-evolving, LMMs, multimodal models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_adbqOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies two important issues in self-evolving multimodal models: model consensus is not always reliable, and model uncertainty does not necessarily indicate that a question is useful for learning. The paper separates these two problems into “what to trust” and “what to learn,” and addresses them with external visual evidence and actual learning progress, respectively. I find this decomposition natural and conceptually useful.

Critical strengths

I like the overall idea of the paper. The experiments make a convincing case that relying only on the model’s own consensus can reinforce its mistakes, while an independent grounding signal can provide more reliable supervision. I also find the progress-based Proposer more conceptually aligned with the goal of curriculum learning than simply rewarding uncertainty or entropy. I do not view the use of a frozen detector as a major weakness. It serves a clear purpose, does not introduce inference-time overhead, and provides a source of supervision that is independent of the Solver’s own predictions. The paper is also reasonably transparent about the detector being imperfect and about the current method mainly covering spatial and object-relation skills.

Critical weaknesses

My main concern is about how much of the paper’s contribution should really be attributed to “self-evolution”. The most convincing part of the method appears to be the grounded target itself, while the additional gain from learning the Proposer is noticeably smaller. This does not make the method less useful, but I think the paper should be more careful in separating the contribution of reliable external supervision from the contribution of learning a better curriculum. A second concern is that the Proposer is repeatedly trained using the same fixed set of probe questions to measure learning progress. From the Proposer’s point of view, these probes are part of its training signal. It is therefore unclear whether the Proposer learns a general notion of “what is useful to teach”, or simply learns what works well for this particular probe set. I think this is the main point that currently limits the strength of the “what to learn” claim.

Questions for authors
  1. Does the advantage of the learned Proposer remain if its usefulness is evaluated on a fresh probe set that was never used to generate Proposer rewards? A rotating or separate held-out probe set would make the “learning what to teach” claim much more convincing.
  2. How do the authors interpret the relative importance of the two components? In my reading, the strongest result is that grounded supervision is substantially more reliable than self-consensus, while the learned curriculum gives a smaller but still meaningful additional gain. I think stating this distinction more explicitly would make the paper stronger rather than weaker.
  3. It would also be useful to report the rough training overhead of the paired-progress procedure. I do not consider the detector itself problematic, but the extra Solver trial updates and probe evaluations are part of the practical cost of the full method.
Reviewer_GQAZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the reliability of training signals in self-evolving multimodal models. The paper argues that loops between a Proposer and a Solver draw both their labels, obtained by majority vote, and their question rewards, given by answer entropy, from the Solver itself. It documents two resulting failure modes. First, unanimous wrong answers, which no vote-count score can distinguish from unanimous correct ones, make up 14.5 to 76.7% of wrong majority answers across seven checkpoints on POPE and HallusionBench. Second, an entropy-targeted reward raises the share of invalid questions from 20.8% to 36.5% on executable synthetic scenes. The proposed curriculum uses a frozen RT-DETRv2 detector to compile spatial-relation questions with detector-supported answer sets. The Solver maximizes the total probability of the set. The Proposer selects among compiled candidates and is credited by the probe-set progress difference between two matched trial updates. On Qwen3.5-9B, the eleven-benchmark average rises from 75.58 to 78.61. Evidence targets beat majority-vote targets by 2.43 points, and the learned Proposer adds 1.30 points over a frozen one.

Critical strengths

supported. The observation that any vote-count score assigns the same value to a unanimous error and a unanimous success is simple but important. The paper quantifies how common such errors are across four model families in Figure 2a and Table 5, and shows with Equation 1 why a running-baseline update keeps reinforcing a constant reward. Figure 8 then connects the diagnosis to the method. On the 44 unanimous-wrong questions where the detector supports a different answer, evidence training corrects 36% of answers and agreement training corrects none. I think this analysis is useful to the community independently of the proposed method. The experiments are carefully designed to separate the two components. Table 3 crosses the Solver target with the Proposer credit. Paired trials share weights, optimizer and random state, and a fair coin decides continuation. The authors also openly report exposed evaluation sets and failed Proposer designs, as listed in Table 11.

Critical weaknesses

The self-evolving component adds a modest gain, and the ablation reporting makes it hard to assess. The frozen-Proposer arm, which the paper itself relates to SpatialVLM-style supervision, already reaches a 77.31 average. The learned Proposer adds 1.30 points from three seeds with no reported interval. According to Appendix F.4, its effect on unseen compositions is +0.49 with a standard deviation of 0.63, and according to Appendix D.1 it selects among only 2 to 4 compiled questions per image. In addition, several ablation numbers do not line up, as detailed in questions 2 and 3, and the MMMU gains of over 3 points from 851 spatial questions are not yet explained. Clarifying these points would make the central "what to learn" claim much easier to evaluate.

Questions for authors
  1. Table 2 takes the Base scores from the official reports, while "+Ours" is measured under the protocol described in Appendix E.1. Could the authors also report Base scores for InternVL3.5-8B, LLaVA-OV-7B and Gemma3-12B-IT under the same protocol as "+Ours", so that the gains can be read directly? Please also state the thinking mode, parser, answer-length limits and MMBench variant used.
  2. In Table 9 on page 21, in the block headed "After 8,000 updates", which is labeled as the final method evaluated in-domain only, the row comparing progress credit with the frozen Proposer appears identical to the 500-update row. It includes benchmark entries that the heading says are not reported, and it differs from the +2.30, with a standard deviation of 0.90, reported at 8,000 updates in Table 8. Could the authors double-check this row and report the 8,000-update benchmark effects?
  3. Appendix D.5 states that in Section 5.3 the evidence versus agreement effect is +7.79. That value matches the 500-update contrast with a frozen Proposer in Table 9, whereas Section 5.3 and Table 3 report 84.6 versus 72.4 after 8,000 updates, a difference of 12.22 that the Conclusion also cites, and Table 9 gives +10.31 after 8,000 updates. D.5 also says the single-target ablation stops at 500 updates, while Table 9 lists an 8,000-update contrast. Could the authors add one table of all agreement arms, giving for each the starting checkpoint, Proposer version, updates and seeds?
  4. It would strengthen the "what to learn" claim to report a confidence interval for the 1.30-point gain over the frozen Proposer, include a uniform-selection baseline in Table 3, and describe what the learned Proposer comes to prefer, such as relation types and anchors.
  5. Appendix E.2 notes that part of the MathVista gain comes from fewer truncated answers. Could the authors report parse-failure and truncation rates before and after training for each benchmark, especially MMMU and MMStar?
Reviewer_CDQkOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper diagnoses two failure modes in self-evolving multimodal models: Solver bias (where consensus voting cannot flag unanimous errors, yielding a tie-aware AUC of 1/2) and Proposer bias (where finite-sample N=5 entropy rewards favor invalid or ambiguous questions). Building on this diagnosis, the authors propose a grounded curriculum over 256 unlabeled COCO images (851 spatial questions compiled from frozen RT-DETRv2 detections) and 64 probe images, where the Solver maximizes marginal likelihood over detector-supported answer sets W(I, q) and the Proposer receives paired-progress credit from two-branch trial updates committed via a fair coin, evaluating 6 VLM backbones across 11 benchmarks.

Critical strengths

(1) Rigorous diagnosis of self-evolution failure modes and clean causal trial design: Section 3 provides a clear theoretical and empirical analysis showing why vote margins cannot separate unanimous errors from unanimous successes (with unanimous errors comprising 14.5–76.7% of wrong majority answers across 7 checkpoints in Table 5) and why N=5 entropy rewards overpay invalid questions, while the fair-coin two-branch trial design cleanly separates Proposer credit from greedy branch selection.

(2) Broad multi-backbone validation and transparent negative development logs: The curriculum consistently improves 6 VLM backbones across 11 public benchmarks (raising Qwen3.5-9B average accuracy from 75.58% to 78.61% in Table 1), and Appendix F.7 candidly documents six negative Proposer designs explored during development.

Critical weaknesses

(1) Narrow spatial instantiation where most gains come from detector pseudo-label SFT and 91.2% of questions have singleton sets: The training environment is restricted to 851 pre-compiled 2D bounding-box relation questions ("left/right/above/below") on 256 COCO images from a supervised detector (RT-DETRv2), where 776 of the 851 questions (91.2%) have only a single detector-supported answer (|W(I, q)| = 1, Appendix F.2). Consequently, the multi-answer set objective adds almost nothing over single-answer supervision (Set - single is +0.00 +- 0.90 pp at 500 updates and +0.08 +- 0.84 pp at 8,000 updates in Table 9), and detector SFT with a frozen Proposer (Evidence + Frozen) already yields +12.47 pp in-domain and +1.73 pp on the 11-benchmark average, whereas the learned Proposer (Progress - Frozen) adds +2.30 +- 0.90 pp in-domain and +1.30 +- 0.20 pp on benchmarks at 2x two-branch trial compute (Tables 3 and 8).

(2) Short generation token caps on 9 of 11 benchmarks and Table 2 Base-row protocol questions: While MMMU and MMStar use a 2,048-token cap, Table 7 (Appendix E.1) caps generation at 128 tokens for MathVista, 64 for six suites, 32 for MMBench, and 16 for POPE and HallusionBench, scoring cut-off responses as they stand (Appendix E.2); because training on concise detector answers shortens outputs and reduces mid-sentence truncation on Qwen3.5-9B (from 335/1,000 to 238–268 on MathVista and from 131/4,329 to 32–42 on MMBench), part of the benchmark gain over Base reflects fewer truncated responses. Additionally, although Appendix E.1 states that all models share the same greedy no-thinking evaluation pipeline, the 1-decimal Base rows in Table 2 match published technical-report scores on 7/7 benchmarks for InternVL3.5-8B (whose official report uses Thinking mode) and Gemma3-12B-IT, and 6/7 for LLaVA-OneVision-7B.

(3) Reporting and comparison discrepancies in Tables 9 and 10: In Table 9 (Appendix F.2, Line 1124), the final Progress - frozen row under "After 8,000 updates (final method; in-domain only)" duplicates the 500-update row (+1.00+-0.60, +0.52+-1.20, +0.50+-0.60, +0.04+-0.05, +0.46+-0.25) instead of reporting the 8,000-update in-domain contrast (+2.30 +- 0.90 pp in Table 8). Furthermore, Table 10 (Appendix F.6) claims the method exceeds MM-Zero on every shared Qwen3-VL-8B benchmark except ChartQA by citing MM-Zero iteration 3 (+2.5 pp on MMMU), omitting that Table 2 of the same cited MM-Zero paper (arXiv:2603.09206v1) reports iteration 5 reaching 61.9 (+6.1 pp over 55.8 Base) on MMMU.

Questions for authors
  1. Since 91.2% of compiled questions have a single supported answer (so Set - single adds +0.08 +- 0.84 pp at 8,000 updates) and two-branch Proposer training doubles per-step Solver trial updates, how does Evidence + Progress compare against single-branch Evidence + Frozen under equal wall-clock training compute?

  2. How much of the benchmark gain on MathVista (cap=128), MMBench (cap=32), and the 64-token suites remains under a generous token budget (e.g., 512 or 2,048 tokens) with an explicit direct-answer prompt for Base, and were the Table 2 Base rows rerun under your greedy no-thinking pipeline or taken from external reports?

  3. Could you correct the duplicated 8,000-update Progress - frozen row in Table 9 and include MM-Zero iteration 5 (+6.1 pp on MMMU) in Table 10?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.