- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies two information bottlenecks in RLVR for multimodal reasoning: reward reliability, where false-positive rewards contaminate positive supervision, and sampling reachability, where all-negative groups provide no reward contrast. It proposes ContextRL, which combines Context-Augmented Reward modeling (CAR), using full reference solutions to verify intermediate reasoning and visual grounding; Context-Augmented Sampling (CAS), using mistake reports to guide resampling for all-negative groups; and context rollback, which trains accepted self-contained recoveries with original negatives under the original query. Experiments with Qwen3-VL-8B-Instruct across eleven benchmarks report gains over GRPO and DAPO, along with improved reward accuracy and recovered training coverage.
Critical strengths
-
The analysis of reward reliability and sampling reachability is accurate and valuable, and it clearly reveals how the two bottlenecks are coupled in RLVR. This provides a principled motivation for CAR and CAS.
-
The formalization of the information bottlenecks is clear and concise, and the group-contrast discussion offer an intuitive diagnostic framework for the proposed method.
Critical weaknesses
W1. Mistake reports and recovered positives lack auditing for correctness, hallucination, and answer leakage.
The mistake report is constrained only by prompt instructions, with potential risk of hallucination and answer leaking. Since CAS conditions generation on the reports, erroneous or leaking feedback can contaminate recovered positives and distort policy training.
W2. The evaluation lacks comparison with recent RL methods for multimodal reasoning.
The main results do not include recent approaches on RL improvement such as VL-Rethinker(NeurIPS 25), Shuffle-R1(ICLR 26). As a result, the paper does not establish whether ContextRL remains competitive within the broader recent RL landscape.
W3. The method heavily depends on the reward model, but robustness to systematic verifier bias is not discussed.
Both CAR and CAS rely on judgments from Qwen3-VL-32B-Instruct as the verifier. If the reward model itself has systematic biases on specific data distributions, training may amplify verifier biases, and the final policy could be affected.
Questions for authors
-
When CAS fails to correct the original group, the whole group is skipped (according to Algo. 1), which may shift the data distribution toward queries that are easier for the reward model to recover. This distribution shift might lead to systematic neglect of hard queries, limiting model performance on hard samples.
-
The performance improvement partly comes from the extended computation introduced by CAR and CAS. If the baselines (GRPO and DAPO) are scaled to a comparable compute level (e.g., by increasing the number of rollouts or training epochs), it is unclear whether ContextRL still achieves better performance.
-
The paper mainly discussed verifiable setting, in which hard 0-1 reward is used. I wonder if ContextRL can be generalized to broader RL settings such as continuous or soft reward scenario.
7 comments
Part II
Weakness III. Robustness to systematic verifier bias is not discussed
We agree a dedicated systematic-bias analysis is missing. Three pieces of existing evidence partially bear on the concern.
First, Table 2 reports detection rates for both Qwen3-VL-32B and Qwen3-VL-235B (81.98% vs. 82.12% with full solutions), which converge once given the same context, though both are in the Qwen3-VL family, so this does not rule out a shared architectural bias. We have since rebuilt that evaluation on a set constructed without any ContextRL-judge screening (500 responses), where the two judges again track each other closely, at 77.8% and 79.2%. Second, Table 3's false-positive and false-negative rates are measured against 4,000 human-annotated ground-truth labels, not against the judge's own self-consistency. Third, and most relevant to "training may amplify verifier bias": final evaluation on all eleven benchmarks is performed by GPT-4o via VLMEvalKit on the original query, without reference solutions or reports, a judge independent of the training-time verifier. If the gains were the policy learning idiosyncratic preferences of the 32B verifier, we would not expect them to appear under an independent judge on standard benchmarks.
That is evidence against simple preference-amplification rather than a robustness analysis, and we will say so in the revision. A further check bears directly on the form your concern takes. The verifier is Qwen3-VL-32B, and its own benchmark profile is already in Table 1:
| Model | Perception Avg | Reasoning Avg | MME-RW-L (reason) | SimpleVQA |
|---|---|---|---|---|
| Qwen3-VL-8B (backbone) | 59.31 | 57.12 | 41.73 | 44.93 |
| Qwen3-VL-32B (the verifier) | 64.10 | 65.03 | 45.73 | 53.07 |
| ContextRL (trained 8B policy) | 65.22 | 62.37 | 53.20 | 48.35 |
If training amplified the verifier's distributional biases, the policy would converge toward the verifier's profile. It does not. The 8B policy ends up above the 32B verifier on the perception average (65.22 vs. 64.10) while staying below it on reasoning (62.37 vs. 65.03), so it is neither bounded by the verifier's ability nor pulled to its optimum. The per-benchmark pattern runs against amplification more directly still: on MME-RW-L (reasoning), where the verifier is weakest at 45.73, the policy reaches 53.20, 7.47 points above it; on SimpleVQA, where the verifier is strongest at 53.07, the policy reaches only 48.35. A policy inheriting the verifier's biases would be expected to do the reverse. We will add this comparison to the revision.
Two further checks are available at low cost, and we propose them rather than claiming the above settles the matter: a domain-stratified breakdown of the judge's error rates on the existing 4,000 human-annotated labels (FPR/FNR by VQA versus math, or by answer type), which needs no new data collection and which we will provide during the discussion period; and re-judging with a model from outside the Qwen3-VL family, a larger job that will go into the revision.
Part III
Question 1. Skipped groups when CAS fails to recover, and distribution shift toward easier queries
It is true that this branch is missing from the pseudocode. When 𝒫(x) is empty, Algorithm 1 has no else-branch and the query is not used for that step; we will add the branch and state how such groups are handled.
How much data the concern applies to. In the first round of sampling, 11.3% of queries produce an all-negative group of eight. The paper's 8.56% all-query recovery coverage is the product of that 11.3% unconditional rate and a conditional recovery rate of ≈75.7% (8.56/11.3) given that a query started all-negative. CAS therefore recovers roughly three-quarters of the queries that would otherwise be discarded, and the residual that is never recovered is about 2.7% of all training queries (11.3% − 8.56%). These are exactly the queries GRPO and DAPO discard entirely (an all-negative group has zero reward-relative contrast, and DAPO's dynamic sampling drops zero-variance groups by construction) so the skipping behaviour is not unique to ContextRL, though that does not by itself make it harmless.
A targeted experiment on the residual. The natural remedy for the 2.7% is to stop skipping it and supply a positive from elsewhere. We tried exactly that: for queries CAS fails to recover, we substituted a correct answer produced by a stronger external model (Gemini-4-Carbon) and brought the query back into training through the same context-rollback procedure, under the same λ-scaling used for CAS's own recovered positives.
| Setup | Handling of the ≈2.7% residual | Perception (overall) | Reasoning (overall) | Policy entropy |
|---|---|---|---|---|
| ContextRL (as submitted) | skipped | 65.22 | 62.37 | stable |
| ContextRL + external positives | Gemini-4-Carbon answer, context rollback, same λ | 64.83 | 61.46 | rises by ≈1.5 |
The remedy makes both axes worse, and entropy rises despite the λ down-weighting, so the limiting factor is not insufficient down-weighting. These substitute positives are not policy-generated, and they carry a systematic distributional mismatch in style and token statistics that a scalar λ cannot correct, because λ shrinks gradient magnitude rather than distributional divergence. Independent work points the same way, finding that externally sourced positives belong in a separate stage rather than inside the RL update (Limozin et al., arXiv 2604.23747; Li et al., arXiv 2609.04108; Xia et al., RePO, arXiv 2602.10819).
We would like to make one clarification first: CAS does not discard hard queries, it recovers them. All-negative groups are precisely what GRPO and DAPO drop in full, since a zero-variance group carries no reward contrast, and they are 11.3% of all queries. CAS turns roughly three quarters of them back into usable training signal, leaving 2.7% unrecovered. Relative to a baseline without CAS, the neglect of hard queries is therefore reduced rather than introduced, and what remains is a minority of a minority. On that residual we observe no appreciable effect either: the hard evaluation slices show none of the degradation that systematic neglect of difficult queries would predict. Against DAPO, ContextRL gains 6.35 on DynaMath worst-case and 4.17 on MathVerse-mini, while removing CAS costs 2.00 on MathVerse-mini, the largest single-component loss we measure. The residual, the Algorithm 1 branch and the selection effect will all be stated explicitly in the revision. We are grateful for this concern: it prompted the external-positive experiment, which turned the skip from an unexamined implementation detail into a design choice with evidence behind it.
Official Review of Submission401 by Reviewer R29j
We thank the reviewer for recognizing the analysis of reward reliability and sampling reachability, and in particular the coupling between the two bottlenecks, since it is that coupling, rather than either bottleneck on its own, that motivates treating reward and sampling together. We are also grateful for the comments on auditing, the comparison with recent multimodal RL methods, robustness to verifier bias, the skipped groups, and generalization to continuous rewards. We ran a substantial set of experiments on these questions, and the findings have improved the work; we take each in turn below.
Weakness I. Mistake reports and recovered positives lack auditing for correctness, hallucination and answer leakage
We would first note that "constrained only by prompt instructions" understates what the paper already does, and then address the real gap, which is the absence of any quantitative audit.
Three existing mechanisms bear on this. (i) Both the context-augmented and the regular reward instructions (Appendix D) explicitly instruct the judge to list wrong reasoning content only, with no evidence, correct facts, or final-answer leakage. (ii) A recovered response is retained in 𝒫(x) only if it is independently judged correct and self-contained by CAR, which is process-aware: Table 3 shows FPR dropping from 18.71% to 3.44%, and acceptance of "correct answer with flawed process" dropping from 91.30% to 15.46%, so a response that leaked the answer but reasons poorly is precisely the failure mode CAR is best at rejecting. (iii) CAR explicitly rejects responses containing revision cues ("You are absolutely right", "I apologize for my mistake") so that retained responses remain self-contained after rollback.
We do acknowledge the circularity you are pointing at: the filter in (ii) uses the same model and the same reference that wrote the report, so self-consistency is not independent verification. We should also raise a second-order concern the review does not mention: the reference solution itself is model-selected by the 32B reward model (Appendix B.1) rather than human-verified, so an erroneous reference could propagate into both the report and the acceptance decision. We will add this to the Limitations section.
The strongest structural safeguard is that training uses context rollback (the report is not part of the training input) and final evaluation is done by GPT-4o via VLMEvalKit on the original query, with no reference solution or report. Contamination from the report, if present, cannot leak directly into the evaluation protocol.
What we are doing to close this. The quantitative figures are the part we do not yet have, and we are producing them now. We are annotating a sample of mistake reports by hand, checking each for explicit and inferable answer leakage and for whether the error it points to is real, and we are having a judge outside the training loop re-score a sample of recovered positives, which breaks the self-consistency circularity you identify. Whatever completes within the discussion period we will post in this thread, and the full audit goes into the revision.
Weakness II. No comparison with recent multimodal RL methods (VL-Rethinker, Shuffle-R1)
It is true that neither VL-Rethinker nor Shuffle-R1 is cited in the submission, and both should have been cited and discussed.
Both works target the same phenomenon as our Bottleneck II, and the distinction we would draw is where the positive comes from. VL-Rethinker's Selective Sample Replay reuses previously seen non-zero-advantage samples; Shuffle-R1's pairwise trajectory sampling and advantage-based batch shuffle reorder and filter within a larger per-query pool. Both operate on samples the policy's distribution can already produce, whereas CAS addresses the case where the pool holds nothing to replay: under a matched 16-rollout budget the unconditioned policy recovers 19.5% of all-negative queries against 66.5% with a mistake report (Table 6). We will position the two families as complementary rather than competing, and in principle stackable, noting that their published numbers are not directly comparable to ours: VL-Rethinker uses Qwen2.5-VL at 7B/72B, and Shuffle-R1 targets largely different mathematical-reasoning benchmarks.
We have since run a same-backbone comparison against Critique-GRPO, the closest feedback-guided method, under its original setup on the same Qwen3-VL-8B policy and the same 29K training set.
Editing a failed draft recovers much of what resampling cannot, while resampling under a mistake report reaches further and costs less.