acceptodds
Under review as a conference paper at ICLR 2027

ContextRL: Addressing Policy Information Bottlenecks with Context-Augmented RL

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become an important route for improving multimodal large language models (MLLMs), using reward signals provided by verifiers as training supervision. Its effectiveness depends on both reliable rewards and positive samples. In multimodal tasks, final-answer agreement can reward flawed reasoning, while difficult queries may yield no correct response within the sampling budget. Our analysis identifies two coupled information bottlenecks that restrict the quality and availability of policy supervision. To address these bottlenecks, we propose ContextRL, a framework that combines context-augmented reward modeling and context-augmented sampling. Context-augmented reward modeling uses full reference solutions to check visual grounding and reasoning, reducing false-positive supervision and generating mistake reports. Context-augmented sampling uses mistake reports to guide further attempts on all-negative groups. Training Qwen3-VL-8B with ContextRL, we achieve average gains of 2.58% and 1.09% over GRPO and DAPO, respectively, across 11 perception and reasoning benchmarks, and surpass Qwen3-VL-32B in perception. Our in-depth analysis shows that contextual information improves reward accuracy and the availability of positive samples, while revealing how falsepositive rewards undermine policy learning. These findings support context augmentation for improving RLVR supervision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RL, MLLM, Knowledge Discovery

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_R29jOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies two information bottlenecks in RLVR for multimodal reasoning: reward reliability, where false-positive rewards contaminate positive supervision, and sampling reachability, where all-negative groups provide no reward contrast. It proposes ContextRL, which combines Context-Augmented Reward modeling (CAR), using full reference solutions to verify intermediate reasoning and visual grounding; Context-Augmented Sampling (CAS), using mistake reports to guide resampling for all-negative groups; and context rollback, which trains accepted self-contained recoveries with original negatives under the original query. Experiments with Qwen3-VL-8B-Instruct across eleven benchmarks report gains over GRPO and DAPO, along with improved reward accuracy and recovered training coverage.

Critical strengths
  1. The analysis of reward reliability and sampling reachability is accurate and valuable, and it clearly reveals how the two bottlenecks are coupled in RLVR. This provides a principled motivation for CAR and CAS.

  2. The formalization of the information bottlenecks is clear and concise, and the group-contrast discussion offer an intuitive diagnostic framework for the proposed method.

Critical weaknesses

W1. Mistake reports and recovered positives lack auditing for correctness, hallucination, and answer leakage.

The mistake report is constrained only by prompt instructions, with potential risk of hallucination and answer leaking. Since CAS conditions generation on the reports, erroneous or leaking feedback can contaminate recovered positives and distort policy training.

W2. The evaluation lacks comparison with recent RL methods for multimodal reasoning.

The main results do not include recent approaches on RL improvement such as VL-Rethinker(NeurIPS 25), Shuffle-R1(ICLR 26). As a result, the paper does not establish whether ContextRL remains competitive within the broader recent RL landscape.

W3. The method heavily depends on the reward model, but robustness to systematic verifier bias is not discussed.

Both CAR and CAS rely on judgments from Qwen3-VL-32B-Instruct as the verifier. If the reward model itself has systematic biases on specific data distributions, training may amplify verifier biases, and the final policy could be affected.

Questions for authors
  1. When CAS fails to correct the original group, the whole group is skipped (according to Algo. 1), which may shift the data distribution toward queries that are easier for the reward model to recover. This distribution shift might lead to systematic neglect of hard queries, limiting model performance on hard samples.

  2. The performance improvement partly comes from the extended computation introduced by CAR and CAS. If the baselines (GRPO and DAPO) are scaled to a comparable compute level (e.g., by increasing the number of rollouts or training epochs), it is unclear whether ContextRL still achieves better performance.

  3. The paper mainly discussed verifiable setting, in which hard 0-1 reward is used. I wonder if ContextRL can be generalized to broader RL settings such as continuous or soft reward scenario.

7 comments
AuthorsOpenReview ↗

Official Review of Submission401 by Reviewer R29j

We thank the reviewer for recognizing the analysis of reward reliability and sampling reachability, and in particular the coupling between the two bottlenecks, since it is that coupling, rather than either bottleneck on its own, that motivates treating reward and sampling together. We are also grateful for the comments on auditing, the comparison with recent multimodal RL methods, robustness to verifier bias, the skipped groups, and generalization to continuous rewards. We ran a substantial set of experiments on these questions, and the findings have improved the work; we take each in turn below.

Weakness I. Mistake reports and recovered positives lack auditing for correctness, hallucination and answer leakage

We would first note that "constrained only by prompt instructions" understates what the paper already does, and then address the real gap, which is the absence of any quantitative audit.

Three existing mechanisms bear on this. (i) Both the context-augmented and the regular reward instructions (Appendix D) explicitly instruct the judge to list wrong reasoning content only, with no evidence, correct facts, or final-answer leakage. (ii) A recovered response is retained in 𝒫(x) only if it is independently judged correct and self-contained by CAR, which is process-aware: Table 3 shows FPR dropping from 18.71% to 3.44%, and acceptance of "correct answer with flawed process" dropping from 91.30% to 15.46%, so a response that leaked the answer but reasons poorly is precisely the failure mode CAR is best at rejecting. (iii) CAR explicitly rejects responses containing revision cues ("You are absolutely right", "I apologize for my mistake") so that retained responses remain self-contained after rollback.

We do acknowledge the circularity you are pointing at: the filter in (ii) uses the same model and the same reference that wrote the report, so self-consistency is not independent verification. We should also raise a second-order concern the review does not mention: the reference solution itself is model-selected by the 32B reward model (Appendix B.1) rather than human-verified, so an erroneous reference could propagate into both the report and the acceptance decision. We will add this to the Limitations section.

The strongest structural safeguard is that training uses context rollback (the report is not part of the training input) and final evaluation is done by GPT-4o via VLMEvalKit on the original query, with no reference solution or report. Contamination from the report, if present, cannot leak directly into the evaluation protocol.

What we are doing to close this. The quantitative figures are the part we do not yet have, and we are producing them now. We are annotating a sample of mistake reports by hand, checking each for explicit and inferable answer leakage and for whether the error it points to is real, and we are having a judge outside the training loop re-score a sample of recovered positives, which breaks the self-consistency circularity you identify. Whatever completes within the discussion period we will post in this thread, and the full audit goes into the revision.

Weakness II. No comparison with recent multimodal RL methods (VL-Rethinker, Shuffle-R1)

It is true that neither VL-Rethinker nor Shuffle-R1 is cited in the submission, and both should have been cited and discussed.

Both works target the same phenomenon as our Bottleneck II, and the distinction we would draw is where the positive comes from. VL-Rethinker's Selective Sample Replay reuses previously seen non-zero-advantage samples; Shuffle-R1's pairwise trajectory sampling and advantage-based batch shuffle reorder and filter within a larger per-query pool. Both operate on samples the policy's distribution can already produce, whereas CAS addresses the case where the pool holds nothing to replay: under a matched 16-rollout budget the unconditioned policy recovers 19.5% of all-negative queries against 66.5% with a mistake report (Table 6). We will position the two families as complementary rather than competing, and in principle stackable, noting that their published numbers are not directly comparable to ours: VL-Rethinker uses Qwen2.5-VL at 7B/72B, and Shuffle-R1 targets largely different mathematical-reasoning benchmarks.

We have since run a same-backbone comparison against Critique-GRPO, the closest feedback-guided method, under its original setup on the same Qwen3-VL-8B policy and the same 29K training set.

MethodRecovery on all-negative queries (16 rollouts)Perception AvgReasoning AvgTraining wall-clock (h)
Plain resampling (GRPO)19.5%64.2558.1747.9
Critique-GRPO56.9%63.6659.9667.0
ContextRL66.5%65.2262.3751.3

Editing a failed draft recovers much of what resampling cannot, while resampling under a mistake report reaches further and costs less.

Weakness III. Robustness to systematic verifier bias is not discussed

We agree a dedicated systematic-bias analysis is missing. Three pieces of existing evidence partially bear on the concern.

First, Table 2 reports detection rates for both Qwen3-VL-32B and Qwen3-VL-235B (81.98% vs. 82.12% with full solutions), which converge once given the same context, though both are in the Qwen3-VL family, so this does not rule out a shared architectural bias. We have since rebuilt that evaluation on a set constructed without any ContextRL-judge screening (500 responses), where the two judges again track each other closely, at 77.8% and 79.2%. Second, Table 3's false-positive and false-negative rates are measured against 4,000 human-annotated ground-truth labels, not against the judge's own self-consistency. Third, and most relevant to "training may amplify verifier bias": final evaluation on all eleven benchmarks is performed by GPT-4o via VLMEvalKit on the original query, without reference solutions or reports, a judge independent of the training-time verifier. If the gains were the policy learning idiosyncratic preferences of the 32B verifier, we would not expect them to appear under an independent judge on standard benchmarks.

That is evidence against simple preference-amplification rather than a robustness analysis, and we will say so in the revision. A further check bears directly on the form your concern takes. The verifier is Qwen3-VL-32B, and its own benchmark profile is already in Table 1:

ModelPerception AvgReasoning AvgMME-RW-L (reason)SimpleVQA
Qwen3-VL-8B (backbone)59.3157.1241.7344.93
Qwen3-VL-32B (the verifier)64.1065.0345.7353.07
ContextRL (trained 8B policy)65.2262.3753.2048.35

If training amplified the verifier's distributional biases, the policy would converge toward the verifier's profile. It does not. The 8B policy ends up above the 32B verifier on the perception average (65.22 vs. 64.10) while staying below it on reasoning (62.37 vs. 65.03), so it is neither bounded by the verifier's ability nor pulled to its optimum. The per-benchmark pattern runs against amplification more directly still: on MME-RW-L (reasoning), where the verifier is weakest at 45.73, the policy reaches 53.20, 7.47 points above it; on SimpleVQA, where the verifier is strongest at 53.07, the policy reaches only 48.35. A policy inheriting the verifier's biases would be expected to do the reverse. We will add this comparison to the revision.

Two further checks are available at low cost, and we propose them rather than claiming the above settles the matter: a domain-stratified breakdown of the judge's error rates on the existing 4,000 human-annotated labels (FPR/FNR by VQA versus math, or by answer type), which needs no new data collection and which we will provide during the discussion period; and re-judging with a model from outside the Qwen3-VL family, a larger job that will go into the revision.

Question 1. Skipped groups when CAS fails to recover, and distribution shift toward easier queries

It is true that this branch is missing from the pseudocode. When 𝒫(x) is empty, Algorithm 1 has no else-branch and the query is not used for that step; we will add the branch and state how such groups are handled.

How much data the concern applies to. In the first round of sampling, 11.3% of queries produce an all-negative group of eight. The paper's 8.56% all-query recovery coverage is the product of that 11.3% unconditional rate and a conditional recovery rate of ≈75.7% (8.56/11.3) given that a query started all-negative. CAS therefore recovers roughly three-quarters of the queries that would otherwise be discarded, and the residual that is never recovered is about 2.7% of all training queries (11.3% − 8.56%). These are exactly the queries GRPO and DAPO discard entirely (an all-negative group has zero reward-relative contrast, and DAPO's dynamic sampling drops zero-variance groups by construction) so the skipping behaviour is not unique to ContextRL, though that does not by itself make it harmless.

A targeted experiment on the residual. The natural remedy for the 2.7% is to stop skipping it and supply a positive from elsewhere. We tried exactly that: for queries CAS fails to recover, we substituted a correct answer produced by a stronger external model (Gemini-4-Carbon) and brought the query back into training through the same context-rollback procedure, under the same λ-scaling used for CAS's own recovered positives.

SetupHandling of the ≈2.7% residualPerception (overall)Reasoning (overall)Policy entropy
ContextRL (as submitted)skipped65.2262.37stable
ContextRL + external positivesGemini-4-Carbon answer, context rollback, same λ64.8361.46rises by ≈1.5

The remedy makes both axes worse, and entropy rises despite the λ down-weighting, so the limiting factor is not insufficient down-weighting. These substitute positives are not policy-generated, and they carry a systematic distributional mismatch in style and token statistics that a scalar λ cannot correct, because λ shrinks gradient magnitude rather than distributional divergence. Independent work points the same way, finding that externally sourced positives belong in a separate stage rather than inside the RL update (Limozin et al., arXiv 2604.23747; Li et al., arXiv 2609.04108; Xia et al., RePO, arXiv 2602.10819).

We would like to make one clarification first: CAS does not discard hard queries, it recovers them. All-negative groups are precisely what GRPO and DAPO drop in full, since a zero-variance group carries no reward contrast, and they are 11.3% of all queries. CAS turns roughly three quarters of them back into usable training signal, leaving 2.7% unrecovered. Relative to a baseline without CAS, the neglect of hard queries is therefore reduced rather than introduced, and what remains is a minority of a minority. On that residual we observe no appreciable effect either: the hard evaluation slices show none of the degradation that systematic neglect of difficult queries would predict. Against DAPO, ContextRL gains 6.35 on DynaMath worst-case and 4.17 on MathVerse-mini, while removing CAS costs 2.00 on MathVerse-mini, the largest single-component loss we measure. The residual, the Algorithm 1 branch and the selection effect will all be stated explicitly in the revision. We are grateful for this concern: it prompted the external-positive experiment, which turned the skip from an unexamined implementation detail into a design choice with evidence behind it.

Reviewer_XoaROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies two information bottlenecks in multimodal RLVR: false-positive rewards caused by correct final answers with flawed reasoning, and all-negative groups on difficult queries that provide no useful relative advantage. To address them, the authors propose ContextRL. CAR provides full reference solutions to a Qwen3-VL-32B verifier for process-aware evaluation and mistake-report generation. CAS then uses failed responses and their reports to resample all-negative groups, after which accepted responses are rolled back to the original query for training. Using Qwen3-VL-8B and approximately 29K FineVision examples, ContextRL improves over GRPO and DAPO by 2.58 and 1.09 points on average across 11 benchmarks.

Critical strengths

The diagnostic experiments are convincing. On 4,000 human-annotated responses, full reference solutions reduce false positives from 413 to 76. Under the same 16-rollout budget, mistake reports increase positive coverage on initially all-negative queries from 19.5% to 66.5%. These results provide fairly direct evidence that CAR improves reward reliability and CAS improves sampling reachability.

Critical weaknesses
  1. All policy results are from single runs. The compute-matched w/o report variant is only 0.32 points worse, while changing the advantage scaling causes fluctuations of about 2.6 points on individual benchmarks. It is therefore unclear whether the reported policy gain is larger than training noise.
  2. ContextRL requires substantial additional computation, including generating and selecting reference solutions with a 32B model, process-level verification, and extra rollouts for all-negative groups. The paper does not report policy tokens, verifier calls, or total compute, making the comparison with GRPO and DAPO difficult to assess.
  3. The paper omits closely related methods. Critique-GRPO also uses natural-language critiques to refine failed responses, while LUFFY uses off-policy traces to overcome the limitations of on-policy RLVR. Without discussing or comparing these methods, the novelty of CAS remains unclear.
Questions for authors
  1. With $\lambda=0.01$, recovered groups contribute very little gradient. Why does removing CAS reduce performance by roughly 1.2 points more than removing the mistake report? Please also report the total number of policy tokens and verifier calls for each method.

  2. Why is Critique-GRPO not included in the comparison, and how does ContextRL differ from Critique-GRPO and off-policy guidance methods such as LUFFY?

7 comments
AuthorsOpenReview ↗

Official Review of Submission401 by Reviewer XoaR

We thank the reviewer for recognizing that the two diagnostics establish the two mechanisms separately, CAR on reward reliability and CAS on sampling reachability, since keeping those two claims distinct is what the paper's structure is built on. We are also grateful for the comments on the reliability of single-run results, compute accounting, the distinction from prior hint-guided methods, and the gap between the two ablations. We ran a substantial set of experiments and compute accounting on these questions, and the findings have improved the work; they are reported below.

Weakness I. Single-run results; is the reported gain larger than training noise?

It is true that the submitted numbers show this: the compute-matched w/o report variant was 0.32 points below the full method (66.35 vs. 66.67), while removing advantage scaling alone (w/o scale) shifts individual benchmarks by up to about 2.6 points (HR-8K 78.25 vs. 75.63, a +2.62 swing). All policy-level results in the submission were single runs.

We have now trained three seeds for each of the four methods. Seed 1 is the run reported in the paper; seeds 2 and 3 are new. Accuracy on the main-table perception/reasoning overall metrics, mean ± SD over the three runs:

MethodPerception (overall)Reasoning (overall)
GRPO63.94 ± 0.2758.25 ± 0.09
DAPO64.66 ± 0.1860.53 ± 0.17
w/o report64.92 ± 0.1161.93 ± 0.10
ContextRL65.23 ± 0.1662.32 ± 0.06

Two observations, stated with the limits of a three-seed sample:

  1. Run-to-run SD on the overall averages is 0.06 to 0.27, an order of magnitude below the ±2.6 swing you cite. That swing is a per-benchmark effect and does not propagate to the overall averages, which is the level at which the paper's conclusions are drawn. The revision will separate the two scales.
  2. The one gap that remains close to noise is the one you identified: ContextRL over w/o report is +0.31 on perception and +0.39 on reasoning, consistent in sign but of the same order as the run-to-run spread. We therefore do not claim the mistake report is established as a final-policy driver on the strength of this gap; see Item 4.

We will add this table to the paper, report all policy results as mean ± SD over three seeds, and remove the single-run framing. Three seeds support a sign and a rough magnitude, not a significance claim, and we will not state one.

Weakness II and Question 1 (second half). Unmatched compute; policy tokens and verifier calls

The submission only stated qualitatively that policy tokens, verifier calls, and feedback costs need not be matched. The accounting, from training logs:

MethodRealized rollouts / queryAvg. output tokens / sequencePolicy sequences generated (= verifier calls)Training wall-clock (h, one 8-GPU node)
GRPO81617.8230K47.9
DAPO≈24 (3× oversampling)1769.0690K62.1
ContextRL8.9 average1696.5255K51.3
w/o report8.9 average1633.8255K50.8
w/o CAS81740.6230K48.4
w/o CAR81647.3248K51.7

Average output length is within 10% across all six methods (1618 to 1769 tokens), so the sequence counts also give the ratio of token volume. The second column answers both quantities you ask for at once, because the verifier is called exactly once per generated policy sequence. Section 3.1 defines the reward model this way: the reward R(x,y)=r_ψ(x,y;s) and the report M(x,y⁻)=h_ψ(x,y⁻;s) are outputs of the same model applied to the same candidate, so writing the report adds no call. The only genuinely extra calls come from CAS's second-stage resampling, and they are already in the table. The wall-clock column covers policy generation and all verifier inference, report generation included.

Two points follow. ContextRL is not the compute-heavy method: 3.4 hours above GRPO and 10.8 below DAPO. DAPO's dynamic sampling filters zero-variance groups and oversamples to replace them at roughly 3× GRPO's nominal budget; because those generations are batched, that costs about 2× GRPO's sampling time rather than 3×, giving the 62.1 hours above. Table 1 is therefore not ContextRL against two fixed-compute baselines: DAPO is already a compute-scaled GRPO, and ContextRL still exceeds it at lower total cost.

One cost the table does not include. Constructing the reference solutions with a 32B model is an inference cost: inference is run once over the training set and the sampled answers are cross-checked for correctness. It is incurred once offline before training (Appendix B.1) and reused across every ContextRL variant, so it does not appear in the per-run training time above. It takes 10 hours on the same 8-GPU node, which we will report as a separate one-off line in the revision rather than fold into per-run training time.

Weakness III and Question 2. Missing comparison with Critique-GRPO and LUFFY; novelty of CAS unclear

This is a real omission: neither Critique-GRPO nor LUFFY is cited in the submission, and both should have been discussed and ideally compared against. The differences that matter are where the new sample comes from and what triggers it. LUFFY injects a complete trajectory from a stronger external model, so its samples are not policy-generated at all. Critique-GRPO refines the policy's own failed draft in context, so the new response inherits the old one. CAS discards the failed response and has the policy re-answer the original question under a mistake report that is then stripped before training, so what enters the update is the policy's own output under the original input. On triggering, both prior methods act on every query, whereas CAS acts only on all-negative groups, which is 11.3% of queries.

Two further distinctions we would state plainly. Both prior methods apply a more principled per-sample correction to their non-on-policy samples than ContextRL's constant λ (Critique-GRPO: ratio shaping with clipping retained and KL dropped; LUFFY: importance sampling with π_φ=1, clipping dropped), which is a limitation on our side; CAS's negatives are genuinely on-policy with ratio 1, and only the recovered positives use λ. And both are text-only mathematical and STEM reasoning, whereas ContextRL covers multimodal image and video. The capability CAS adds is reaching positives the unconditioned policy cannot sample within the same budget (Table 6: 66.5% against 19.5% at a matched 16 rollouts), which is not what either prior method sets out to demonstrate.

We have now run Critique-GRPO as a same-backbone baseline, following the original setup (n_rollouts=7, n_refinements=1, γ=0.1, clipping and KL removed, Dr. GRPO advantage) on the same Qwen3-VL-8B-Instruct policy, the same 29K training set, and the same batch size and learning rate. Critiques are written by the same Qwen3-VL-32B-Instruct judge ContextRL uses, rather than by GPT-4o as in the original, both to hold the feedback model fixed across methods and because GPT-4o is our evaluation judge. One necessary adaptation: the original screens critiques against a rule-based checker (Math-Verify), which does not exist for multimodal tasks, so correctness comes from the same judge.

MethodGenerated / queryEntering trainingVerifier calls / queryTraining wall-clock (h)Perception AvgReasoning Avg
GRPO88847.964.2558.17
DAPO≈248≈2462.164.7460.68
Critique-GRPO1481467.063.6659.96
ContextRL8.98.38.951.365.2262.37

Three things follow:

  • It improves reasoning but not perception. Reasoning is 1.79 above GRPO while perception is 0.59 below it, with CharXiv-RQ back at GRPO's level and HalluBench, MME-RW-L and SimpleVQA all lower. Its critiques target faulty steps in a reasoning chain, which is what the method was designed for; CAR addresses reward reliability and CAS sampling reachability, neither confined to repairing a chain, so ContextRL gains on both axes.
  • It is the most expensive method we ran. It generates 14 sequences per query and trains on 8, against ContextRL's 8.9 and 8.3. Its two sampling rounds are sequentially dependent and cannot be folded into one batched pass the way DAPO's oversampling is, so its wall-clock generation time is the longest of all baselines and its 67.0 hours exceed DAPO's 62.1. ContextRL remains the second cheapest, 3.4 hours above GRPO.
  • On the mechanism, refinement helps but reaches less far. Under a matched 16-rollout budget on the all-negative queries of Table 6, in-context refinement recovers 56.9% of positives against plain resampling's 19.5% and CAS's 66.5%.

Question 1 (first half). Why does removing CAS cost about 1.2 points more than removing the report, if λ=0.01 leaves recovered groups with almost no gradient?

The gap is indeed 1.22 (66.35 − 65.13), and the submission does not contain a verified mechanism for a gap of that size. Below we separate what we can support from what we cannot.

What the three variants differ in. w/o CAS removes the second sampling round entirely, so a query whose first-stage group is all-negative contributes nothing. w/o report keeps the second round but without the mistake report in the context. Full ContextRL has both.

The recovery-coverage decomposition does not explain the gap. Coverage runs 0% to roughly 2.5% to 8.56% across the three, so the first increment buys 1.22 points and the much larger second increment buys 0.32. Recovery coverage is therefore not monotone with the observed gaps, and we will not claim the two ablations differ only in how many queries receive a recovered positive. We raised this explanation internally and are discarding it. We are grateful that Q1 was put this precisely: it is what made us re-examine the account we had been giving for this gap, and then drop it.

What we can support. First, the direction and rough size of the gap reproduce on benchmarks absent from the ablation table, which we have now evaluated: relative to w/o report, w/o CAS loses 1.71 on MathVerse-mini, 0.83 on We-Math (loose), 0.54 on CharXiv-RQ and 0.45 on DynaMath (worst-case), gaining only 0.49 on MME-RW-L (reasoning). The gap is not an artifact of the eight-metric subset. Second, from the three-seed results in Item 1, run-to-run SD on the overall averages is 0.06 to 0.27, so a 1.22-point gap sits well outside run variation while the 0.32-point gap does not, which is consistent with your intuition that the report's own final-policy contribution is the weaker of the two claims.

The two ablations are a clean single-factor pair, which we have confirmed. w/o report applies the same λ=0.01 scaling to its mixed groups as full ContextRL, and loss aggregation is identical to GRPO's throughout, so recovered-group samples cannot affect the effective step size for ordinary groups beyond the reward term. The only variable separating the two is whether the second sampling round happens at all, and the 1.22 can be read as the value of that round. On why the two increments are so unequal: the first buys the queries that one extra round of ordinary sampling can already solve, while the further queries the report unlocks are harder ones whose recovered positives are sampled under the augmented context and then down-weighted by λ. We will present the gap with this scoping in the revision.

Each of the three weaknesses and two questions above is answered, and the single-run framing is replaced throughout. This review led us to replace every single-run conclusion with three-seed repetitions, and to abandon an explanation that did not survive its own numbers.

We hope this evidence resolves the questions you raised. Anything still missing we will produce as quickly as we can before the period closes. We look forward to further discussion with you as a way to put the claims on firmer ground. If you consider the concerns met, we would be grateful if you would reconsider the score.

Reviewer_aM94OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper names two problems in RLVR for MLLMs. Answer-only verification rewards flawed reasoning (false positives), and hard queries produce groups where every sample fails, so there is no reward contrast. ContextRL addresses the first with CAR, a 32B judge that sees a full reference solution and writes mistake reports. It addresses the second with CAS: for groups where all samples fail, the policy is resampled conditioned on its failed attempt and the report. Accepted self-contained recoveries are then mixed back with the original negatives under the original query, with advantages scaled by λ. Qwen3-VL-8B is trained on 29K FineVision samples and evaluated on 11 benchmarks. It gains +1.09 average over DAPO, plus human-annotated reward accuracy (88.5 to 97.1) and recovery statistics.

Critical strengths
  • The reward reliability analysis is concrete and useful. On 4,000 human-labeled responses, full-solution context cuts FPR from 18.7% to 3.4% while FNR stays flat (Table 3). Process-flawed responses are accepted 15% of the time instead of 91%. This is the most convincing evidence in the paper, and practitioners can act on it directly.

  • At a matched 16-rollout budget, CAS recovers 66.5% of initially all-negative queries versus 19.5% for ordinary resampling (Table 6). So the report does change the sampling distribution, not just add attempts.

Critical weaknesses
  • CAS barely shows up in the final policy. The matched-budget control (w/o report, 66.35) is within 0.32 of the full method (66.67), from single runs with no variance. Recovered groups also have their advantages scaled by λ = 0.01 (B.4), so they carry almost no gradient. The "restores contrast for 8.56% of queries" story is therefore hard to square with the actual update. The recovered positives are also sampled from π(·|x, y⁻, M) but trained as if from π(·|x). The importance ratio and clipping under this context rollback are not discussed. As it stands, most of the gain seems to come from CAR, and CAS is closer to a sampling diagnostic than a training contribution.

  • The policy-level comparison is weak. Over DAPO the gains are +0.48 (perception) and +1.69 (reasoning), from one run each. Compute is not matched: ContextRL uses extra rollouts, extra 32B judge calls, and report generation. The paper does not say clearly which verifier and prompt GRPO/DAPO use. Table 2 is also biased by construction: the false-positive set was screened to be rejected by the ContextRL reward model itself, which inflates the "w. solution" column. Closely related critique-guided refinement methods for RL (e.g. Critique-GRPO, hint/guided-GRPO style approaches, LUFFY) are not discussed or compared.

Questions for authors
  • What reward model and prompt do the GRPO and DAPO baselines use? Is it the same 32B judge with answer-only reference?
  • With λ = 0.01, what fraction of the total advantage mass in a batch comes from recovered groups? Can you show results with larger λ, or with λ = 1 plus importance correction?
  • How do you handle the policy ratio for recovered samples generated under the augmented context?
  • Why does the ablation drop MathVerse and CharXiv, where the main gains are largest? Why does DynaMath improve in every ablation?
  • Table 2: what are the detection rates on a false-positive set not filtered by the ContextRL judge?
7 comments
AuthorsOpenReview ↗

Response of Submission401 for Reviewer aM94

We thank for recognizing Table 3 and the matched-budget comparison in Table 6, and in particular for taking the mistake report to change the sampling distribution rather than merely add attempts, which is precisely the claim CAS is meant to support. We are also grateful for the comments on the role of λ, the ratio and clipping treatment, compute matching, the construction of the false-positive set, and the benchmark coverage of the ablation. We ran a substantial set of measurements and experiments on these questions, and the findings have improved the work; they are reported below.

Weakness I (a) and Question 2. Gradient share of recovered groups under λ=0.01, and results at larger λ

It is true that recovered groups contribute very little of the gradient. Under GRPO's within-group normalization (Eq. 1), a recovered group's advantages are the same O(1) order as an ordinary group's before scaling; multiplying by λ=0.01 leaves recovered groups contributing 0.15% of the batch's total |advantage| mass, measured from the training logs.

What we want to separate is what that figure measures. Because the scaling is uniform across the group, its gradient contribution becomes λ·g, leaving the relative ordering, the relative magnitudes and the mean-zero property under Eq. (1) untouched, so λ·g and g point in exactly the same direction. The mass share measures ‖λ·g‖ and says nothing about that direction. It is equivalent to using a λ-times-smaller learning rate on this subset: the update says the same thing, it is simply taken more slowly.

The two quantities come apart in both directions here:

  • Increasing the magnitude degrades performance. At λ=1 the advantages are 100× larger and the mass share rises to roughly 8.6%, with the direction unchanged, yet perception drops 0.52 and reasoning 1.18 (λ sweep below).
  • Removing it entirely causes a substantial loss. The w/o CAS ablation deletes exactly these groups and costs 1.54 points on the ablation aggregate and 2.00 on MathVerse-mini, the largest single-component loss we measure (Q4 below).

If mass share were a good proxy for contribution, neither result would look like this. Our reading is that recovered groups are the hardest queries in the training set, failed on all eight attempts, so each recovered positive carries far more learning value than an ordinary sample, while the groups that dominate the mass sit on queries the policy already solves and largely re-confirm what it has learned. Their direction is also stable across steps and accumulates, whereas gradients on solved queries vary and cancel.

What λ is for. λ is not a judgement that the signal is worth 0.15% of the rest. It is the mechanism Section 3.3 describes, keeping a signal generated under (x, y⁻, M) but trained under x subordinate to the on-policy bulk; relaxing it costs the entropy growth and reasoning damage visible in the sweep. Where the mismatch is larger still, with positives supplied by an external model, λ-scaling does not rescue it at all.

On the 8.56% wording. Section 4.6 already distinguishes the two quantities, but the Introduction and Conclusion present 8.56% as a headline that reads as a claim about CAS's share of the final-policy gain. We will rescope it to "sampling-level recovery coverage" and add the 0.15% figure and the magnitude-versus-direction distinction around the scaling equation. We are grateful for this question: that distinction is one the paper never drew, and it becomes part of how the method is described.

On the second half of your question, we ran the intermediate point:

λPerception (overall)Reasoning (overall)Policy entropy over training
0.0165.2262.37stable, peaks around 0.7
0.165.0161.95peaks around 1.1
1 (= the w/o scale ablation)64.7061.19exceeds 1.5

The sweep is monotone in both directions: larger λ raises policy entropy and costs accuracy, and the damage is concentrated on reasoning (−0.42 then −0.76 further) rather than perception (−0.21 then −0.31). We read this as the same mechanism as W1b/Q3 below: recovered positives carry no per-sample importance correction, only a uniform group-level constant, so when the down-weighting is relaxed their gradient weight and variance are both too large, and multi-step reasoning is more sensitive to that than perception. We confine ourselves to these two observations (entropy rises, reasoning is hurt more than perception) and do not make any claim about whether training diverges at λ=1, which we have not checked against the training curves.

On "λ = 1 plus importance correction": we should be explicit that this option does not exist in our implementation. ContextRL makes one policy update per rollout batch, so training is strictly on-policy and there is no stale-policy ratio to correct with (see Item 2). The only available lever is λ itself, which is what the sweep varies.

Weakness I (b) and Question 3. Importance ratio and clipping under context rollback

It is true that recovered positives are sampled from π(·|x, y⁻, M) but, after context rollback, are paired with the original query x for training, and neither the main text nor Algorithm 1 states how the importance ratio or clipping is handled. We can now describe the actual implementation.

Training is on-policy by design (every sample used for a given update is freshly generated by the current policy for that same rollout batch, with no replay buffer and no multi-epoch reuse) so there is no PPO-style clipping anywhere in the implementation. For the retained negatives in a mixed group the policy ratio is exactly 1, and this is not an approximation: they were genuinely sampled from π(·|x), so training them under x involves no context mismatch.

For the recovered positives there is no ratio playing that role, and we apply no per-sample importance correction. Two mechanisms act on them, and they are worth separating because their scopes differ:

  • λ-scaling is group-level, not positive-specific. The advantage-scaling equation in Section 3.3 multiplies every response in the mixed group by the same λ, retained negatives included (Algorithm 1: "Scale all mixed-group advantages by λ"). It therefore does not act as a per-sample correction on the positives at all, it scales the whole group's influence while leaving the group's internal structure untouched.
  • KL masking is positive-specific. KL regularization excludes the recovered positives (Section 3.3; Algorithm 1: "mask KL on recovered positives"). The one term that treats them differently from the negatives is the KL term, not the advantage term.

So the accurate description is that the context mismatch on recovered positives is handled by subordinating the entire group rather than by correcting the individual samples, and λ substitutes for an importance correction only in that weaker, group-level sense. We recognize this is a biased surrogate relative to a true per-sample correction (uniform, rather than tailored to how far the two contexts diverge for each sample) and the λ sweep in Item 1 is consistent with that reading. We will state this plainly in the revision and make the scope of the scaling equation and of the KL masking explicit around Algorithm 1.

Weakness I (c). CAS's footprint on the final policy (w/o report at 66.35 versus 66.67)

It is true that a 0.32-point gap from single runs is not strong evidence. We would separate two claims the Weakness combines (the value of the mistake report specifically, and the value of the CAS two-stage recovery procedure as a whole) and we now have repeated runs for the first.

Three seeds per method (seed 1 is the submitted run), on the main-table overall metrics:

MethodPerception (overall)Reasoning (overall)
GRPO63.94 ± 0.2758.25 ± 0.09
DAPO64.66 ± 0.1860.53 ± 0.17
w/o report64.92 ± 0.1161.93 ± 0.10
ContextRL65.23 ± 0.1662.32 ± 0.06

Per-seed paired differences for the comparison you raise:

Paired differenceSeed 1Seed 2Seed 3Mean
ContextRL − w/o report (Perception)+0.29+0.27+0.37+0.31
ContextRL − w/o report (Reasoning)+0.55+0.23+0.38+0.39

The sign is stable across all three seeds, but the magnitude is of the same order as the run-to-run spread, so we will not claim the mistake report is established as a final-policy driver. We will reposition it as primarily a sampling-reachability mechanism, Table 6's 66.5% vs. 19.5% recovery at matched 16-rollout budgets, which is a sampling-distribution fact independent of gradient share, and state the final-policy gap as small and directionally consistent rather than demonstrated.

The procedure as a whole is a separate matter: w/o CAS drops to 65.13 on the eight-metric average, a 1.54-point loss, and the newly evaluated benchmarks (Item 7) show the same direction more sharply, w/o CAS loses 2.00 points on MathVerse-mini relative to full ContextRL, against 0.29 for w/o report. We are still working through what the relationship between these two gaps does and does not establish.

Weakness II (a). Weak gains over DAPO (+0.48 and +1.69), single run

These two numbers are correct as read from Table 1 (perception 65.22 vs. 64.74; reasoning 62.37 vs. 60.68). With three seeds:

Paired differenceSeed 1Seed 2Seed 3Mean
ContextRL − DAPO (Perception)+0.48+0.28+0.93+0.56
ContextRL − DAPO (Reasoning)+1.69+1.69+2.00+1.79
ContextRL − GRPO (Perception)+0.97+1.32+1.58+1.29
ContextRL − GRPO (Reasoning)+4.20+4.01+3.99+4.07

The perception-side margin over DAPO (+0.56 mean) is of the same order as the run-to-run SD and we will not emphasize it; the reasoning-side margins (+1.79 over DAPO, +4.07 over GRPO) are several times the observed spread and same-signed on every seed. We also note that ContextRL is built directly on GRPO (no clip-higher, no dynamic sampling), so the same-optimizer comparison is against GRPO; the comparison with DAPO is cross-optimizer and the two techniques are not mutually exclusive. Three seeds support a sign and a rough magnitude, not a significance claim, and we will not state one.

Weakness II (b). Unmatched compute (extra rollouts, extra 32B judge calls, report generation)

It is true that the submission only states qualitatively that "equal epochs need not match policy tokens, verifier calls, or feedback costs." From the training logs:

MethodRealized rollouts / queryAvg. output tokens / sequencePolicy sequences generated (= verifier calls)Training wall-clock (h, one 8-GPU node)
GRPO81617.8230K47.9
DAPO≈24 (3× oversampling)1769.0690K62.1
ContextRL8.9 average1696.5255K51.3
w/o report8.9 average1633.8255K50.8
w/o CAS81740.6230K48.4
w/o CAR81647.3248K51.7

Average output length is within 10% across all six methods (1618 to 1769 tokens), so the sequence counts also give the ratio of token volume. The verifier is called exactly once per generated policy sequence, as Section 3.1 defines it (the reward R(x,y)=r_ψ(x,y;s) and the report M(x,y⁻)=h_ψ(x,y⁻;s) come from the same model applied to the same candidate), so the sequence count and the verifier-call count are the same number, obtained from the logged run structure: 29K queries × 8 first-stage responses = 230K, plus CAS's stage-2 responses and DAPO's oversampled ones. The wall-clock column integrates policy generation and all verifier inference, including mistake-report generation, at the token level.

Two clarifications this makes concrete. First, ContextRL is not the compute-heavy method: 3.4 hours above GRPO and 10.8 below DAPO, as wall-clock on one 8-GPU node. DAPO's dynamic sampling filters zero-variance groups and oversamples to replace them, issuing roughly 3× GRPO's nominal rollout budget; because those generations are batched, the 3× oversampling costs roughly 2× GRPO's sampling wall-clock time rather than 3×, with 62.1 hours end to end. Second, writing the mistake report adds no extra verifier call: CAR already calls the verifier once per response in an all-negative group to judge correctness, and the report is additional output within that same call. The only genuinely extra calls come from CAS's stage-2 resampling, a sampling-count effect, not a reporting effect.

One cost the table does not include: building the reference solutions CAR consumes is an inference cost, run once over the training set with the sampled answers cross-checked for correctness (Appendix B.1), and reused across all ContextRL variants, so it is not part of the per-run training time. It takes 10 hours on the same 8-GPU node, which we will report as a separate one-off line.

Weakness II (c) and Question 1. Which verifier and prompt do the GRPO and DAPO baselines use?

GRPO, DAPO, and the w/o CAR ablation all use Qwen3-VL-32B-Instruct as the reward model with the Regular Reward Instruction given in Appendix D, which supplies the final reference answer only, not the full reference solution. One detail worth stating because it tightens rather than loosens the comparison: the Regular instruction is not pure answer-matching, it already requires the judge to check for hallucination or contradiction with the image and for logical correctness of the reasoning before accepting an answer match. CAR's improvement is therefore attributable to providing the full reference solution as additional context (plus the self-contained / revision-cue filtering rule), not to replacing a trivial judge with a non-trivial one. We will add this to Appendix B.3, together with confirmation that decoding settings are matched across baselines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.