acceptodds
Under review as a conference paper at ICLR 2027

CAVE-RL: CONFIDENCE-ADAPTIVE VISUAL EVIDENCE FOR MULTIMODAL REINFORCEMENT LEARNING

Abstract

Reinforcement learning (RL) for multimodal models increasingly relies on learned visual verifiers, but those verifiers are imperfect precisely on the visually ambiguous inputs where policy optimization is most likely to exploit them. We propose CAVE-RL (Confidence-Adaptive Visual Evidence RL), a reward construction that combines sparse exact task feedback with visual proxy rewards whose influence is attenuated by predictive entropy and ensemble disagreement, then anchors policy updates to a frozen reference policy. The method is architecture-agnostic and can be inserted into PPO/GRPO-style post-training whenever an action can be mapped to a visual claim scored by one or more verifiers. We construct 30,555 multimodal image-instruction instances from 1,797 real handwritten-digit images, covering five language-conditioned transformation tasks, held-out prompt paraphrases, visual distribution shifts, 60% corruption in verifier training labels, and five independent random seeds. Only 15% of RL rollouts reveal the exact task outcome. Under this budget, CAVE-RL improves the five-condition robust average by 2.17 pp over SFT and 0.70 pp over sparse-reward PPO, while reducing expected calibration error from 12.49% to 8.34%. It recovers 74% of the robustness gain of an oracle PPO baseline that observes the exact outcome on every rollout.

Primary area:
reinforcement learning
Keywords:
Multimodal reinforcement learning, Visual verifiers, Uncertainty‑aware reward design, Reward hacking, Sparse rewards

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BLTCOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

the paper proposes a reward construction for multimodal reinforcement learning with exact outcome feedback and imperfect visual proxies. The method combines that reward with exact outcomes revealed on 15% of rollouts and uses a clipped policy objective with reference KL and entropy regularization.

Validation uses a small CNN-based policy on handwritten digits paired with five instruction digit transformations, corruptions, a held-out paraphrase, and five random seeds. CAVE-RL reports 51.94% average accuracy over five test conditions, compared with 51.24% for sparse-outcome PPO and 51.80% for an ungated ensemble reward.

Critical strengths
  • The experiments separate sparse exact rewards from dense visual proxy rewards, preserve source-image sets, and include five seeds multi runs, an ungated ensemble comparator, and dense exact-feedback PPO as a reference.
  • visual proxy information helps at the tested sparse-feedback budget. CAVE-RL improves Robust Avg. by 0.70 points relative to sparse-outcome PPO and improves occlusion accuracy by 1.07 points.
Critical weaknesses
  • The experiments provide limited evidence for the gate's contribution, compared to the un-gated Grounded-Ens baseline, the gate improves Robust Avg. by 0.14 points and reduces the gap by 0.49 points. ECE is worse. for multi runs, there are no entropy-only, disagreement-only, or other similar settings to distinguish reliability estimation from simply reducing visual reward strength.
  • All experiments use a small CNN policy, ten discrete actions, five digit transformations, and 1,797 source images; the expanded context count does not establish applicability to open-ended multimodal reasoning.
  • it's unclear that the data generation, algorithm 1 samples actions from the current policy πθ, whereas Section 3.3 defines the ratio as πθ/πref with πref frozen. The denominator is not identified as the policy that generated the training data.
Questions for authors
  • What policy actually generates the rollout data, and what exact log probability is stored in the ratio denominator? Is πref in Section 3.3 a frozen initialization or the rollout policy
  • what's the differences of multi-runs between CAVE-RL and Grounded-Ens in Robust Avg. and proxy-oracle gap?
  • What is the exact evaluation denominator for the task-wise results in Table 3? For CAVE-RL, the equal-weight mean of its five task entries is 76.552%, matching Table 1's clean accuracy, but these entries are not labeled as clean-only.
Reviewer_y31uOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This work proposes CAVE-RL, a reward framework for multimodal RL with sparse outcome feedback, available for only 15% of rollouts. It combines proxy rewards from three noisy visual verifiers with a reliability gate based on prediction entropy and verifier disagreement. The rewards are centered against a frozen reference policy, and training uses PPO with KL regularization. Experiments use 8×8 handwritten digits, a small CNN policy, and five instruction transformations, with 60% verifier label noise and five random seeds.

Critical strengths
  1. Careful experimental protocol. The paper reports five seeds, selects the gating parameter on validation data, and includes an oracle reward upper bound. The authors also acknowledge that the benefit of reliability gating is modest.
Critical weaknesses
  1. The experiments do not support the claimed multimodal LLM setting. The paper targets multimodal model post-training, but experiments use only 1,797 small digit images, a two-layer CNN, and a 10-class action space. No pretrained multimodal LLM, free-form generation, or standard multimodal benchmark is evaluated. The claimed applicability to MLLMs therefore remains unverified.

  2. The benefit of reliability gating is not demonstrated. CAVE-RL improves over Grounded-Ens by only 0.14 points, far below the reported standard deviation of 2.24. It also has worse calibration error (8.34 vs. 8.11) and slightly lower severe-shift accuracy than Grounded-PPO (36.09 vs. 36.32). In Fig. 5b, performance consistently decreases as the gating strength increases from γ=0.25\gamma=0.25 to 1, suggesting that weaker or no gating may work better. Most of the reported 74% recovery appears to come from adding proxy rewards, since Grounded-PPO already achieves about 70%.

Questions for authors

Clarifications That Could Change My Assessment:

  1. What are the validation and test results at γ=0\gamma=0, which removes the reliability gate?

  2. In Eq. (9), is the probability ratio computed against πref\pi_{\mathrm{ref}}, as written, or πold\pi_{\mathrm{old}}?

  3. Why does batch correctness suddenly increase from about 50% to 68% at the final update for every method in Fig. 4?

Suggestions:

  1. The paper should discuss related work on reward model ensembles, particularly "Reward Model Ensembles Help Mitigate Overoptimization"( ICLR 2024. arXiv: 2310.02743), and compare against their mean-minus-variance objective. Experiments with a real multimodal LLM and paired statistical tests across seeds would provide stronger evidence for the claimed benefits.
Reviewer_8BTxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposed a setting where real feedback is sparse, and uncertain visual feedback is avalible; The paper proposed CAVE-RL, which is a RL algorithm for better leveraging the uncertain visual judge feedback. The paper is evaluated mostly on a self-created digit related synthesis benchmark.

Critical strengths
  • the motivation of better leveraging uncertain feedbacks is solid
  • the text part of the paper is overall well-written and easy-to-follow
Critical weaknesses
  • the task setting is not realistic; the synthesis digit task reflects very few realistic meaning of why we care about this setting? while the high-level motivation is valid, it lacks a realistic setting where we have sparse/hard-to-obtain ground truth feedback; but we have a cheap visual judge. In the proposed setting; why the VLM itself cannot be the visual judge; or use an agentic framework like, we first as the model to describe the image; than answer the question
  • the experiment section is weak; only this synthesis toy dataset is evaluated; and results are quite marginal compared with baselines; although the paper still have a lot room, the authors did not choose to leverage
  • the figures legends are very small and hard to read; I suspect the entire paper is mostly AI-generated;
Questions for authors
  • why leaving almost 2 pages empty?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.