acceptodds
Under review as a conference paper at ICLR 2027

Context-Aware RL for Agentic and Multimodal LLMs

Abstract

Large language models (LLMs) often fail when answering requires identifying a small but decisive piece of evidence within a long or complex context, such as a single line in a tool trace or a subtle detail in an image. We propose ContextRL, a context-aware reinforcement learning (RL) method that improves long-horizon reasoning and multimodal performance through an indirect auxiliary objective. Instead of supervising only the final answer, ContextRL presents the model with a query, an answer, and two highly similar contexts, and rewards it for selecting the context that supports the query–answer pair, thereby encouraging fine-grained grounding. We construct contrastive context data in two domains: for coding agents, trajectories serve as contexts, yielding 1k pairs built via condition filtering; for multimodal reasoning, images serve as contexts, yielding 7K pairs built via generative editing and similarity search. ContextRL achieves average gains of +2.0% over standard GRPO on 5 long-horizon benchmarks, and +1.8% across 12 diverse visual question answering benchmarks. To disentangle the effect of the proposed objective from that of additional data, we compare against data-augmentation baselines that repurpose the same contrastive contexts as standard query–context–answer examples. These baselines provide little to no improvement, showing that the gains arise from the proposed context-selection objective rather than from the contrastive data alone. Data and code will be publicly released.

Primary area:
reinforcement learning
Keywords:
Agentic LLM, Multimodal LLM, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hCEFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ContextRL, a reinforcement learning algorithm that improves multimodal large models and code agents through a context-selection objective. The authors observe that current open-source models struggle to identify evidence supporting a given answer. Using filtering, retrieval, and image editing, they construct 1,000 pairs of coding trajectories and 7,000 pairs of image data. The model chooses between two contexts, and the corresponding loss is jointly optimized with GRPO. Evaluations on coding and visual understanding benchmarks show consistent gains over baselines such as GRPO. The authors also demonstrate that the gains arise from the joint optimization objective.

Critical strengths
  1. The motivation is clear and important. Context selection and understanding remain important challenges in post-training research, particularly as coding agents attract increasing attention and their trajectories grow longer. Studying this problem will help advance the field.
  2. The authors demonstrate the method's effectiveness through a logical experimental design. They evaluate ContextRL's practical performance on a sufficient range of benchmarks and compare it with data augmentation strategies, demonstrating the advantages of the optimization method.
Critical weaknesses
  1. Context selection and context grounding are different capabilities. The training task is: (Q,A,C+,C−)⟶Select C+(Q,A,C^+,C^-)\longrightarrow \text{Select } C^+ Here, the model is given the correct answer AA and asked to select the context that supports it. In contrast, the actual inference task is: (Q,C)⟶A(Q,C)\longrightarrow A There is a gap between these two optimization objectives. The paper does not show the trained model's actual behavior, for example through case studies, to establish that ContextRL makes it rely more on contextual evidence during inference.
  2. The main experiments use Qwen2.5-VL-7B and Qwen3-VL-8B, with smaller gains on Qwen3-VL-8B than on the earlier Qwen2.5-VL-7B model (Table 2, 2.0 vs. 1.6). This raises concerns about whether the method will remain effective on more frontier models, such as Qwen3.5-9B.
Questions for authors
  1. Could the authors report results from RL training using only the constructed contrastive data? This would help substantiate the limitations of DA-RL discussed in Section 5.3.
  2. Could the authors provide the DA-SFT baseline's detailed settings, including the learning rate and number of training steps? Qwen3-8B's score drops to 0 after DA-SFT (Table 3). Could this result reflect an implementation issue?
5 comments
AuthorsOpenReview ↗

Response to Reviewer hCEF (Part 1/2)

Dear Reviewer hCEF,

Thank you for the careful and constructive review. We appreciate your recognition that context selection is an important challenge for post-training, particularly as coding agent trajectories grow longer. We are also grateful that you found our experimental design logical and our evaluation across a broad range of benchmarks convincing. We address your concerns below.

W1: Gap between context selection and grounding

We agree that context selection and grounding are not equivalent. A key contribution of our work is to formalize context grounding, which is difficult to label and scale, as a contrastive selection problem. This formulation makes data construction scalable and lets the signal be incorporated directly into RL training. To verify that better selection translates into better grounding, we provide three complementary lines of evidence:

  1. Grounding-focused benchmarks (Table 1). LongBench v2 and NIAH directly test context grounding, as they require the model to retrieve answer-relevant evidence from long inputs. On these benchmarks, ContextRL improves over GRPO by nearly 5 points with Klear-AgentForge as the base model.

  2. Gains concentrate where context grounding is required (new, Appendix G). On SWE-Bench Verified, ContextRL improves file-level localization by +3.6 points over GRPO, versus +1.0 for repair conditioned on correct localization. On LongBench v2, accuracy improves from 25.7% to 31.2% on hard examples (+5.5 points) and from 24.1% to 28.7% on long inputs (+4.6 points) (Appendix G.1). On MathVerse, Qwen2.5-VL improves from 43.4% to 48.6% on Vision Intensive (+5.2 points) and from 41.1% to 45.9% on Vision Only (+4.8 points) (Appendix G.2). We provide detailed subtask analyses in Appendix G, offering further evidence of ContextRL's effectiveness.

  3. Case studies (new, Appendix L). In the revised paper, we add six case studies (three agentic, three multimodal) in which the GRPO model answers incorrectly because it fails to ground its prediction in the relevant context, while ContextRL identifies the correct context and answers correctly.

W2: Effectiveness on more frontier models

We trained Qwen3.5-9B with the same training data and configuration as in our multimodal experiments. Both the GRPO baseline and ContextRL are trained for one epoch. As shown in Table A, ContextRL outperforms the GRPO baseline on most benchmarks and raises the average by +1.3 points.

Notably, on this stronger initial model, outcome-only RL brings only +1.2 points over the initial model, while ContextRL adds a further +1.3 on top of it, more than doubling the total gain from RL training (+2.5 vs. +1.2). This is consistent with Fig. 1, where Qwen3.5-9B remains close to random on the context-selection probe: context grounding is still a bottleneck for frontier open models, and outcome rewards alone do not address it.

Regarding the smaller absolute gain on Qwen3-VL-8B than on Qwen2.5-VL-7B (Table 2), this is expected because its RL baseline is already much stronger (64.1 vs. 51.4), leaving less room for improvement. If we instead measure what fraction of the RL baseline's remaining errors ContextRL fixes, the two models are comparable: about 4.1% on Qwen2.5-VL-7B (2.0 of 48.6 points) and 4.5% on Qwen3-VL-8B (1.6 of 35.9 points).

Table A. Results on Qwen3.5-9B.

ModelMathVistaMathVerseMathVisionMMMU-ProMMMUV*MMStarBLINKSciQAPhyXOlympiadBench-PhyMME-RealWorld-LiteAvg.
Qwen3.5-9B78.275.156.945.366.685.973.163.093.266.614.753.564.3
+ GRPO81.076.058.245.468.286.974.165.694.367.415.353.765.5
+ ContextRL80.876.857.947.269.288.075.466.996.469.116.258.066.8
AuthorsOpenReview ↗

Response to Reviewer hCEF (Part 2/2)

Q1: RL on contrastive data only

RL with contrastive data only. We trained Qwen3-VL-8B with outcome-based RL on the 7k contrastive pairs alone ("CD-only"). It uses the same binary reward as DA-RL. Results are shown below.

BenchmarkBaseRL baselineCD-onlyDA-RLContextRL
MathVista75.878.774.478.379.8
MathVerse56.165.057.365.266.4
MathVision46.249.248.449.552.0
MMMU-Pro41.355.941.356.257.5
MMMU66.469.165.169.470.1
V*82.284.885.384.885.9
MMStar70.573.570.973.674.8
BLINK64.465.167.665.666.6
SciQA94.495.694.495.896.6
PhyX45.572.146.072.173.4
OlympiadBench-Phy7.98.19.48.69.9
MME-RealWorld-Lite48.751.949.152.854.8
Average58.364.159.164.365.7

We have three observations.

  1. The contrastive data carry a real but narrow perceptual signal. CD-only improves over the base model by +0.8 on average, with the largest gains on perception-heavy benchmarks: BLINK (+3.2) and V* (+3.1). It even achieves the highest BLINK score among all configurations. This confirms that the contrastive pairs themselves are informative about fine-grained visual grounding.

  2. Under an outcome reward, this signal does not translate into consistent reasoning gains. CD-only declines on MathVista (−1.4) and MMMU (−1.3) and leaves MMMU-Pro unchanged (0.0). Its average (59.1) remains far below the RL baseline (64.1).

  3. This clarifies the limitation of DA-RL. Once the contrastive data are mixed with task data under a shared outcome reward (DA-RL), CD-only's perception gains largely disappear (BLINK 65.6 vs. 67.6), and the overall gain over the RL baseline is only +0.2. ContextRL consumes the same data through a dense, logit-level objective. It largely retains the perception gains while preserving the benefit of task RL (+1.6 over the RL baseline), which confirms that the limitation of DA-RL lies in how the contrastive signal is consumed, not in the data itself.

Q2: DA-SFT settings and collapse

DA-SFT settings. DA-SFT has two stages: (1) SFT on the contrastive pairs, and (2) standard GRPO on the task data, initialized from the SFT checkpoint, with the same hyperparameters as the RL baseline and no context-awareness loss. Stage 1 uses full-parameter fine-tuning: AdamW, peak learning rate 1e-5 (cosine decay, 3% warmup), weight decay 0, gradient clipping 1.0, and bf16, with the vision encoder frozen for the VL models.

Klear-AgentForge-8BQwen3-8BQwen2.5-VL-7BQwen3-VL-8B
SFT pairs1,0001,0007,0007,000
Batch size16163232
Epochs / steps2 / 1252 / 1252 / 4382 / 438
Max sequence length32,76832,7684,0964,096

Is the collapse an implementation issue? We believe not. First, the SFT stage itself works as intended: DA-SFT attains the highest context-selection accuracy among all methods (Figure 4), and the same pipeline yields models on par with the RL baseline in the multimodal setting (Table 4). Second, the failure is systematic. In most failed trajectories, mini-SWE-agent reports a format error (found 0 actions) after a few turns: the model emits long outputs without a THOUGHT section or bash command, so no action can be executed. Klear-AgentForge-8B shows a similar severe degradation (28.0 → 6.4); Qwen3-8B starts from a much weaker baseline (5.0), so it drops to 0. Once the format collapses, nearly all rollouts receive zero reward, so stage-2 GRPO has no signal to recover. We attribute this to a distribution shift. Context-selection targets are short, single-turn answers, whereas agentic coding requires many turns, each consisting of a THOUGHT section and one bash command. SFT on the former disrupts the multi-turn format that the agent scaffold requires.

We hope these clarifications and new experiment results address the concerns and make the paper suitable for a higher rating.

Thank you to the authors for their detailed response. Most of my concerns have been addressed. However, I noticed that the reported base-model scores for Qwen3.5-9B are substantially lower than those in the official Qwen3.5-9B model card [1]. Specifically, the scores are 22.0 points lower on MathVision, 24.8 points lower on MMMU-Pro, and 11.8 points lower on MMMU. These substantial discrepancies raise concerns about the reliability of the Qwen3.5-9B results. Could the authors check their evaluation logs to investigate the reasons for these discrepancies?

[1] https://huggingface.co/Qwen/Qwen3.5-9B

Reviewer_SnzSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ContextRL adds a supervised context-selection loss to GRPO. Given a question, its answer, and two similar contexts, the model learns to select the context supporting that answer. The authors construct contrastive coding trajectories and image pairs, then evaluate on five agentic/long-context benchmarks and twelve multimodal benchmarks. The method consistently improves over the reported GRPO baselines, with additional comparisons against sequential SFT and reward-based training on the contrastive examples.

Critical strengths

I like the simplicity of the question: can learning which context supports an answer improve ordinary task performance? Testing the same idea on both coding trajectories and images makes the paper more interesting. The consistent improvements over source-matched GRPO baselines are encouraging, particularly the transfer from coding training to long-context tasks. The data construction also makes a reasonable effort to remove obvious shortcuts.

Critical weaknesses
  1. The comparison with DA-SFT does not isolate the proposed objective. ContextRL applies its auxiliary loss jointly with GRPO, whereas DA-SFT first trains context selection separately and then runs GRPO. Without clipping, Eq. 1 is simply binary cross-entropy over the two option logits. A jointly trained, appropriately weighted cross-entropy baseline is therefore essential. The sequential baseline’s collapse could reflect the training schedule rather than an advantage of the proposed loss.
  2. The mechanism claims are stronger than the evidence. Figure 4 shows that selection accuracy and downstream performance need not move together; it does not establish that improved context selection is necessary for better grounding. The downstream gains are promising, but many are around one or two points, without reported run-to-run variation. It is also unclear whether the hyperparameters were selected on a separate validation set or directly from the reported benchmark sweeps.
Questions for authors
  1. How does ordinary two-option cross-entropy perform when trained jointly with GRPO, using the same data, auxiliary weight, and schedule? How much does margin clipping contribute?
  2. Were the loss coefficients and response-length settings chosen using held-out validation data? How stable are the main gains across training seeds?
  3. Eq. 1 clips negative as well as positive margins. This gives zero auxiliary gradient when the model confidently selects the wrong context, with margin below −c. Is this intended, and how often does it occur?
6 comments
AuthorsOpenReview ↗

Response to Reviewer SnzS (Part 1/3)

Dear Reviewer SnzS,

Thank you for the careful and constructive review. We appreciate your recognition that ContextRL poses a simple question and yields consistent gains over GRPO across both coding and multimodal settings, including transfer to long-context tasks. We are also grateful that you noted our efforts to remove shortcuts in data construction. We address your concerns below.

W1&Q1: Missing joint cross-entropy baseline

Rationale for Our Loss Design: Without clipping, Eq. 1 is mathematically equivalent to two-option cross-entropy. We use this formulation to supervise the relative preference between two contexts given a query–answer pair.

Why DA-SFT Uses Sequential Training: DA-SFT follows the established SFT-then-RL pipeline, separating offline supervision from online exploration. It tests whether learning context selection in a preliminary supervised stage is sufficient to improve subsequent task performance through GRPO.

Joint Training as a Key Contribution: ContextRL instead combines online exploration with offline supervision throughout training. Standard GRPO provides outcome-based rewards without explicitly supervising context grounding; our auxiliary supervision supplies this complementary signal while the policy continues learning to solve downstream tasks. The training schedule is therefore an integral part of our method design. This is also the central question examined in Section 5: how context-selection supervision should be integrated with task optimization to improve both grounding and reasoning.

New Baseline: Following your suggestion, we trained GRPO + two-option CE. It is identical to ContextRL except that the margin is not clipped. It uses the same contrastive data, the same λ, and the same batch sizes and number of steps. Its gap to ContextRL therefore directly measures the contribution of clipping.

MethodScheduleLoss on contrastive pairsSWE-VSWE-LiteMM avg (12)
GRPO baseline––28.021.764.1
DA-SFTsequentialCE6.41.363.9
DA-RLjointsampled 0/1 reward27.621.764.3
GRPO + CE (new)jointtwo-option CE, c = ∞29.422.065.1
ContextRLjointtwo-option CE, c = 530.224.065.7

Agentic: Klear-AgentForge-8B. Multimodal: Qwen3-VL-8B, with per-benchmark results below.

Qwen3-VL-8BMathVistaMathVerseMathVisionMMMU-ProMMMUV*MMStarBLINKSciQAPhyXOlyBench-PhyMME-RW LiteAvg
GRPO baseline78.765.049.255.969.184.873.565.195.672.18.151.964.1
GRPO + CE(new)78.966.252.355.770.485.372.767.096.271.99.055.565.1
ContextRL79.866.452.057.570.185.974.866.696.673.49.954.865.7

We have three observations.

  1. ContextRL's advantage over DA-SFT does not come from the specific form of Eq. 1. When consumed as a two-option loss inside the RL loop, the unclipped objective trains stably and improves over GRPO in both settings.

  2. Joint training alone is not sufficient. DA-RL uses the same data in the same joint loop, but as a sampled 0/1 reward, and yields essentially no gain. The dense, logit-level context-selection signal is what turns the contrastive data into improvements (Sec. 5.3(b)).

  3. Margin clipping adds a smaller further gain. Removing it costs 0.8 / 2.0 points on SWE-Bench Verified / Lite and 0.6 points on the multimodal average. ContextRL improves over GRPO on all 12 multimodal benchmarks, whereas GRPO + CE falls below GRPO on three (MMMU-Pro, MMStar, PhyX).

Our contribution is the context-selection supervision itself (selecting the supporting context for a fixed (Q, A)) and how the signal is integrated into RL, not the specific functional form of Eq. 1.

AuthorsOpenReview ↗

Response to Reviewer SnzS (Part 2/3)

W2: Necessity claim lacks causal evidence.

On Figure 4. Figure 4 is not intended to show that context selection is necessary for grounding. Rather, it shows that how context-selection supervision is integrated with task optimization determines whether it translates into downstream gains. ContextRL, which optimizes both objectives jointly, is the only method that improves both selection accuracy and end-task performance. We will revise §5.3(iii), which stated context selection as a necessity.

Context selection vs. grounding. We agree that context selection and grounding are not equivalent. A key contribution of our work is to formalize context grounding, which is difficult to label and scale, as a contrastive selection problem. This formulation makes data construction scalable and lets the signal be incorporated directly into RL training. To verify that better selection translates into better grounding, we provide three complementary lines of evidence:

  1. Gains concentrate where context grounding is required (Appendix G). On SWE-Bench Verified, ContextRL improves file-level localization by +3.6 over GRPO, versus +1.0 for repair conditioned on correct localization. On MathVerse, Qwen2.5-VL-7B gains +4.0 to +5.2 on vision-dependent subsets, versus +2.0 to +3.1 on text-heavy subsets.
  2. Case studies (new, Appendix L). We add six case studies (three agentic, three multimodal). In each, the GRPO model answers incorrectly because it fails to ground its prediction in the relevant context, while ContextRL identifies the correct context and answers correctly.
  3. Grounding-focused benchmarks (Table 1). LongBench v2 and NIAH directly test context grounding, as they require retrieving answer-relevant evidence from long inputs. On Klear-AgentForge-8B, ContextRL improves over GRPO by nearly 5%.

W2: Small gains lack variance reporting.

Gain magnitude in the context of related works. Average improvements of this magnitude are comparable to those reported by prior GRPO-based methods. For example, PAPO [1] reports average gains of 2.4 points over GRPO on its 3B/7B models, while NoisyRollout [2] achieves gains of around 1.1–2.0 points across five OOD benchmarks on its 7B model. ContextRL additionally demonstrates breadth: the same context-selection objective improves over GRPO on all 17 benchmarks across four base models.

Larger gains on context-dependent tasks. The overall averages mask stronger improvements where contextual evidence is especially important. On LongBench v2, accuracy improves from 25.7% to 31.2% on hard examples (+5.5 points) and from 24.1% to 28.7% on long inputs (+4.6 points) (Appendix G.1, Table 11). On MathVerse, Qwen2.5-VL improves from 43.4% to 48.6% on Vision Intensive (+5.2 points) and from 41.1% to 45.9% on Vision Only (+4.8 points) (Appendix G.2, Table 12). We provide detailed subtask analyses across agentic, long-context, and multimodal settings in Appendix G, offering further evidence of ContextRL's effectiveness.

Consistency across repeated runs. We conducted three runs with different seeds on Qwen3-VL-8B and reported mean ± std. ContextRL achieves an average score of 65.8 ± 0.16, compared with 64.0 ± 0.14 for GRPO. The average gain of 1.8 percentage points clearly exceeds the run-to-run variation, supporting the robustness of the improvement beyond a single favorable run.

ModelMathVistaMathVerseMathVisionMMMU-ProMMMUV*MMStarBLINKSciQAPhyXOlympiadBench-PhyMME-RealWorld-LiteAvg.
GRPO78.5 ± 0.3064.8 ± 0.4049.1 ± 0.3256.0 ± 0.4168.4 ± 0.8584.6 ± 0.8073.4 ± 0.3766.1 ± 0.6195.6 ± 0.0871.9 ± 0.188.0 ± 0.5551.6 ± 0.2964.0 ± 0.14
ContextRL79.7 ± 0.1566.6 ± 0.2052.0 ± 0.0257.6 ± 0.6270.2 ± 0.7286.6 ± 0.8074.7 ± 0.2367.0 ± 0.8396.7 ± 0.2673.6 ± 0.429.8 ± 0.8954.6 ± 0.3465.8 ± 0.16

[1] https://arxiv.org/abs/2507.06448
[2] https://arxiv.org/abs/2504.13055

AuthorsOpenReview ↗

Response to Reviewer SnzS (Part 3/3)

W2&Q2: Hyperparameter selection protocol.

Protocol. λ and the maximum response length were chosen from the benchmark sweeps reported in Appendix I. But benchmark data was never used for training, early stopping, or checkpoint selection: all runs are trained for a fixed number of epochs, and we report the final checkpoint. Moreover, in every setting we hold out a validation split drawn from the same sources as the training data and disjoint from it (100 tasks for agentic coding, 1,000 examples for multimodal), which we used to monitor overfitting during RL training. To check whether the benchmark-based choice affected our conclusions, we re-selected both settings using only this validation split. We will revise Appendix I to describe the validation-based protocol.

Context-selection coefficient λ. Table A reports validation performance for each candidate λ, with the RL baseline included for comparison. Selecting by validation performance recovers the λ values used in the paper for all four models. In addition, λ = 0.01 performs worst among the nonzero candidates and at or below the RL baseline for every model, supporting the use of a moderate auxiliary-loss weight.

λKlear-AgentForge-8BQwen3-8BQwen2.5-VL-7BQwen3-VL-8B
0 (RL baseline)0.5100.1000.6790.791
0.0010.5500.1500.7010.817
0.0050.5800.1400.7140.812
0.0100.5000.1000.6680.751
Selected λ0.0050.0010.0050.001

Table A. Validation performance for λ.

Max response length. This setting acts mainly as a reasoning (compute) budget rather than a method-specific hyperparameter. It is shared by the RL baseline and ContextRL, so it cannot favor our method. This follows common practice in RL post-training, where generation length is treated as a budget and methods are compared at matched budgets [1, 2, 3]. Re-selecting it on the validation split again recovers 4,096, the value used in the paper (Table B). Moreover, Table C shows that ContextRL outperforms the RL baseline at every budget we tested, so our conclusion does not depend on this choice.

Max response length (tokens)Validation score
2,0480.796
4,0960.817
8,1920.810
Selected4,096

Table B. Validation performance for max response length on Qwen3-VL-8B.

Max response length (tokens)RL baselineContextRLΔ
2,04862.764.8+2.1
4,09664.165.7+1.6
8,19263.965.4+1.5

Table C. Average accuracy over the 12 multimodal benchmarks on Qwen3-VL-8B.

[1] https://arxiv.org/abs/2503.04697
[2] https://arxiv.org/abs/2502.03373
[3] https://arxiv.org/abs/2503.14476

Q3:Negative-Margin Clipping

Clipping of negative margins. Yes, the symmetric clipping is intended. The upper clip (Δ > c) stops pushing the margin once C⁺ and C⁻ are well separated, so easy pairs do not keep inflating logits. The lower clip (Δ < −c) is a safeguard against label noise. For context selection, a correctly labeled pair rarely drives the model to a confidently wrong choice. Open-source models select near chance level on this task (Figure 1), and in our runs the margins at the start of training stay well below the threshold (mean |Δ| ≈ 0.9–1.4 vs. c = 5). When the model is highly confident that the labeled negative is the supporting context, the pair is therefore more likely to be mislabeled or ambiguous than learnable, and truncating its gradient keeps such examples from pulling the policy in the wrong direction. Our verification pipeline reduces such noise but cannot fully eliminate it, and residual noise is likely to grow once the contrastive data are scaled beyond what manual review can cover.

Frequency and manual verification. To verify this, we manually checked the training logs. The lower clip is active, but rarely, and in the targeted way this design intends. We tracked the per-example CA margin Δ during training. Only 0.3% of logged margins fell below −c (32 of ≈9.9k, all in the multimodal setting). These activations are concentrated on 4 distinct pairs that repeatedly fall below −c throughout training, rather than being spread across the data. On manual inspection, all 4 pairs are genuinely ambiguous: even a human annotator cannot reliably determine which image supports the answer. The lower clip therefore isolates a small set of persistently contradicted pairs rather than suppressing gradients broadly. Lower-clip activations also remain rare as training proceeds: margins shift upward, and the fraction of examples at the upper clip rises from roughly 0–5% to 60–85%, which makes the lower clip even less likely to activate. In the revised paper, we will explicitly explain the motivation for the symmetric clipping.

We hope these clarifications and new experiment results address the concerns and make the paper suitable for a higher rating.

Reviewer_ecJcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ContextRL, which augments GRPO with a supervised context-selection objective to improve grounding in coding agents and multimodal LLMs. Given a question, a reference answer, and two similar contexts, the model learns to identify the supporting context through a logit-level auxiliary loss.

Experiments on four base models span five agentic and long-context benchmarks and twelve multimodal benchmarks, reporting average gains over GRPO of approximately 2.0 and 1.8 percentage points, respectively. Comparisons with sequential SFT and reward-based context-selection baselines favor the joint training approach. Ablations and subtask analyses further examine the auxiliary objective and its effects on code localization, long-context understanding, and visual perception.

Critical strengths
  1. ContextRL integrates the same auxiliary context-selection objective into GRPO for coding agents and multimodal models, without changing inference. This makes it a useful, reusable post-training approach.

  2. The data pipelines identify similar contexts with different supporting evidence, while masking and verification procedures reduce obvious shortcut cues. This provides a useful approach to constructing supervision for learning finer distinctions in contextual evidence.

Critical weaknesses
  1. Training experiments cover only models below 10B parameters, mostly from the Qwen family (Appendix J). Effectiveness at larger scales and across other model families remains unverified.
  2. Average improvements over GRPO are approximately 2.0 percentage points on agentic and long-context benchmarks and 1.8 points on multimodal benchmarks (Tables 1–2), representing a relatively limited empirical advantage.
  3. Contrasting contexts while fixing the question–answer pair has already been explored in mDPO (Wang et al., 2024). Although ContextRL introduces explicit context selection within GRPO, its additional conceptual contribution and empirical advantage over closely related methods are not sufficiently established.
Questions for authors
  1. What new methodological contributions does ContextRL offer over related methods such as mDPO, and what evidence demonstrates a performance advantage? Why was mDPO not included as a baseline in the multimodal experiments under comparable training conditions?
  2. Why were coding agents and image question answering chosen as the main training settings? What task or data characteristics are required for ContextRL to work effectively, and can these conditions be met in other domains?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ecJc (Part 1/2)

Dear Reviewer ecJc,

Thank you for the careful and constructive review. We appreciate your recognition that ContextRL is a reusable post-training approach that applies to both coding agents and multimodal models. We are also grateful that you found our data pipelines useful for supervising finer distinctions in contextual evidence. We address your concerns below.

W1: Limited model scale and diversity

Model scale. We acknowledge this limitation in Appendix J. Our experiments were constrained to a single node with 4× H200 GPUs, and extending RL training to larger models under our settings exceeds our current computational budget. We consider validation at larger scales important future work.

Model family. We conducted an additional multimodal experiment on InternVL3.5-8B using the same training data, without any model-specific tuning. Both methods are trained for 1 epoch. ContextRL outperforms the GRPO baseline on most benchmarks and improves the average score by 1.6 points. The largest gains are on MMMU (+4.0), MMMU-Pro (+3.4), PhyX (+3.0), and MathVerse (+2.6), and the perception benchmarks improve consistently (MMStar +2.4, BLINK +2.0, V* +1.6). This gain is consistent with those on Qwen2.5-VL-7B (+2.0) and Qwen3-VL-8B (+1.6), suggesting that ContextRL generalizes beyond the Qwen-VL family.

ModelMathVistaMathVerseMathVisionMMMU-ProMMMUV*MMStarBLINKSciQAPhyXOlympiadBench-PhyMME-RealWorld-LiteAvg.
InternVL3.5-8B73.850.542.840.656.769.666.255.696.244.67.548.254.4
+ GRPO74.251.646.249.261.968.668.355.497.445.27.249.256.2
+ ContextRL75.354.245.652.665.970.270.757.497.148.28.148.757.8

W2: Limited empirical gains

Gain magnitude in the context of related work. Average improvements of this magnitude are comparable to those reported by prior GRPO-based methods. For example, PAPO [1] reports average gains of 2.4 points over GRPO on its 3B/7B models across a limited set of benchmarks, while NoisyRollout [2] achieves gains of around 1.1–2.0 points across five OOD benchmarks on its 7B model. ContextRL additionally demonstrates breadth: the same context-selection objective improves over GRPO on all 17 benchmarks across four base models.

Larger gains on context-dependent tasks. The overall averages mask stronger improvements where contextual evidence is especially important. On LongBench v2, accuracy improves from 25.7% to 31.2% on hard examples (+5.5 points) and from 24.1% to 28.7% on long inputs (+4.6 points) (Appendix G.1, Table 11). On MathVerse, Qwen2.5-VL improves from 43.4% to 48.6% on Vision Intensive (+5.2 points) and from 41.1% to 45.9% on Vision Only (+4.8 points) (Appendix G.2, Table 12). We provide detailed subtask analyses in Appendix G, offering further evidence of ContextRL's effectiveness.

Consistency across repeated runs. We conducted three runs with different seeds on Qwen3-VL-8B and reported mean ± std. ContextRL achieves an average score of 65.8 ± 0.16, compared with 64.0 ± 0.14 for GRPO. The average gain of 1.8 percentage points clearly exceeds the run-to-run variation, supporting the robustness of the improvement beyond a single favorable run.

ModelMathVistaMathVerseMathVisionMMMU-ProMMMUV*MMStarBLINKSciQAPhyXOlympiadBench-PhyMME-RealWorld-LiteAvg.
GRPO78.5 ± 0.3064.8 ± 0.4049.1 ± 0.3256.0 ± 0.4168.4 ± 0.8584.6 ± 0.8073.4 ± 0.3766.1 ± 0.6195.6 ± 0.0871.9 ± 0.188.0 ± 0.5551.6 ± 0.2964.0 ± 0.14
ContextRL79.7 ± 0.1566.6 ± 0.2052.0 ± 0.0257.6 ± 0.6270.2 ± 0.7286.6 ± 0.8074.7 ± 0.2367.0 ± 0.8396.7 ± 0.2673.6 ± 0.429.8 ± 0.8954.6 ± 0.3465.8 ± 0.16

[1] https://arxiv.org/abs/2507.06448
[2] https://arxiv.org/abs/2504.13055

AuthorsOpenReview ↗

Response to Reviewer ecJc (Part 2/2)

W3 & Q1: Novelty and advantage over mDPO

ContextRL differs from mDPO in three key respects:

Broader motivation and scope. mDPO mainly targets visual conditioning in multimodal alignment and emphasizes hallucination reduction. ContextRL targets context grounding across both images and agent trajectories, with a unified objective spanning coding, long-context understanding, and multimodal reasoning.

Joint online–offline training. The most fundamental difference between ContextRL and DPO-style methods such as mDPO lies in the training paradigm. mDPO is purely offline: it optimizes over fixed preference pairs, which may exploit preference shortcuts, and improved preference fitting does not guarantee stronger reasoning. ContextRL instead jointly optimizes online RL exploration and offline context-selection supervision in a single training process, so that explicit context grounding is learned together with task-reward optimization rather than in isolation. This joint design is the central contribution of ContextRL.

Data construction strategy. The different target tasks also motivate different data construction strategies. mDPO emphasizes hallucination reduction and constructs negative images through random cropping, a simple and uncontrolled corruption that removes objects or visual evidence and can yield pairs that differ in coarse, easily detectable ways. ContextRL targets context-grounded reasoning, which requires pairs that remain highly similar yet differ in answer-relevant details. Constructing such pairs is much more challenging: we combine filtering, retrieval, and targeted image editing to produce minimal, answer-relevant differences. As a result, distinguishing our pairs requires fine-grained reasoning over contextual evidence, not merely detecting whether an object is present.

Additional comparison with mDPO. Following your suggestion, we added an mDPO + GRPO baseline: we first train with mDPO on our contrastive data, then perform GRPO. This baseline achieves 63.4% average accuracy, compared with 65.7% for ContextRL (+2.3 percentage points). These results further support the benefit of our joint online exploration and offline supervision design over sequential preference optimization followed by RL.

ModelMathVistaMathVerseMathVisionMMMU-ProMMMUV*MMStarBLINKSciQAPhyXOlympiadBench-PhyMME-RealWorld-LiteAvg.
Qwen3-VL-8B75.856.146.241.366.482.270.564.494.445.57.948.758.3
mDPO + GRPO77.962.851.354.467.085.367.766.994.970.29.952.363.4
ContextRL79.866.452.057.570.185.974.866.696.673.49.954.865.7

Q2: Choice of settings and generalizability

Choice of settings. We chose agentic coding because it involves long-horizon reasoning over extensive code and tool-interaction histories, which makes context management and grounding essential and provides a natural testbed. We chose VQA to demonstrate that ContextRL generalizes naturally across modalities.

Required conditions and generality. ContextRL requires three conditions: context-dependent tasks, similar contexts that support different answers with clear labels, and task rewards for online RL. These are practical requirements that can be met in many domains. Our context-selection loss is modality-agnostic: text, code, images, and agent trajectories all fit the same framework, with potential extensions to audio and video. Our experiments already demonstrate effectiveness beyond the training settings: Table 1 reports gains on LongBench v2 and NIAH, showing transfer from agentic coding to general long-context understanding and retrieval.

We hope these clarifications and new experiment results address the concerns and make the paper suitable for a higher rating.

Thank you for the detailed response and additional experiments. The new results strengthen the empirical evidence, and the clarifications address several of my concerns regarding the method’s contribution and applicability. I will maintain my original rating of 3 (Weak Acceptance).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.