- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
the paper proposes a reward construction for multimodal reinforcement learning with exact outcome feedback and imperfect visual proxies. The method combines that reward with exact outcomes revealed on 15% of rollouts and uses a clipped policy objective with reference KL and entropy regularization.
Validation uses a small CNN-based policy on handwritten digits paired with five instruction digit transformations, corruptions, a held-out paraphrase, and five random seeds. CAVE-RL reports 51.94% average accuracy over five test conditions, compared with 51.24% for sparse-outcome PPO and 51.80% for an ungated ensemble reward.
Critical strengths
- The experiments separate sparse exact rewards from dense visual proxy rewards, preserve source-image sets, and include five seeds multi runs, an ungated ensemble comparator, and dense exact-feedback PPO as a reference.
- visual proxy information helps at the tested sparse-feedback budget. CAVE-RL improves Robust Avg. by 0.70 points relative to sparse-outcome PPO and improves occlusion accuracy by 1.07 points.
Critical weaknesses
- The experiments provide limited evidence for the gate's contribution, compared to the un-gated Grounded-Ens baseline, the gate improves Robust Avg. by 0.14 points and reduces the gap by 0.49 points. ECE is worse. for multi runs, there are no entropy-only, disagreement-only, or other similar settings to distinguish reliability estimation from simply reducing visual reward strength.
- All experiments use a small CNN policy, ten discrete actions, five digit transformations, and 1,797 source images; the expanded context count does not establish applicability to open-ended multimodal reasoning.
- it's unclear that the data generation, algorithm 1 samples actions from the current policy πθ, whereas Section 3.3 defines the ratio as πθ/πref with πref frozen. The denominator is not identified as the policy that generated the training data.
Questions for authors
- What policy actually generates the rollout data, and what exact log probability is stored in the ratio denominator? Is πref in Section 3.3 a frozen initialization or the rollout policy
- what's the differences of multi-runs between CAVE-RL and Grounded-Ens in Robust Avg. and proxy-oracle gap?
- What is the exact evaluation denominator for the task-wise results in Table 3? For CAVE-RL, the equal-weight mean of its five task entries is 76.552%, matching Table 1's clean accuracy, but these entries are not labeled as clean-only.