- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ContextRL, a reinforcement learning algorithm that improves multimodal large models and code agents through a context-selection objective. The authors observe that current open-source models struggle to identify evidence supporting a given answer. Using filtering, retrieval, and image editing, they construct 1,000 pairs of coding trajectories and 7,000 pairs of image data. The model chooses between two contexts, and the corresponding loss is jointly optimized with GRPO. Evaluations on coding and visual understanding benchmarks show consistent gains over baselines such as GRPO. The authors also demonstrate that the gains arise from the joint optimization objective.
Critical strengths
- The motivation is clear and important. Context selection and understanding remain important challenges in post-training research, particularly as coding agents attract increasing attention and their trajectories grow longer. Studying this problem will help advance the field.
- The authors demonstrate the method's effectiveness through a logical experimental design. They evaluate ContextRL's practical performance on a sufficient range of benchmarks and compare it with data augmentation strategies, demonstrating the advantages of the optimization method.
Critical weaknesses
- Context selection and context grounding are different capabilities. The training task is: Here, the model is given the correct answer and asked to select the context that supports it. In contrast, the actual inference task is: There is a gap between these two optimization objectives. The paper does not show the trained model's actual behavior, for example through case studies, to establish that ContextRL makes it rely more on contextual evidence during inference.
- The main experiments use Qwen2.5-VL-7B and Qwen3-VL-8B, with smaller gains on Qwen3-VL-8B than on the earlier Qwen2.5-VL-7B model (Table 2, 2.0 vs. 1.6). This raises concerns about whether the method will remain effective on more frontier models, such as Qwen3.5-9B.
Questions for authors
- Could the authors report results from RL training using only the constructed contrastive data? This would help substantiate the limitations of DA-RL discussed in Section 5.3.
- Could the authors provide the DA-SFT baseline's detailed settings, including the learning rate and number of training steps? Qwen3-8B's score drops to 0 after DA-SFT (Table 3). Could this result reflect an implementation issue?
5 comments
Response to Reviewer hCEF (Part 2/2)
Q1: RL on contrastive data only
RL with contrastive data only. We trained Qwen3-VL-8B with outcome-based RL on the 7k contrastive pairs alone ("CD-only"). It uses the same binary reward as DA-RL. Results are shown below.
| Benchmark | Base | RL baseline | CD-only | DA-RL | ContextRL |
|---|---|---|---|---|---|
| MathVista | 75.8 | 78.7 | 74.4 | 78.3 | 79.8 |
| MathVerse | 56.1 | 65.0 | 57.3 | 65.2 | 66.4 |
| MathVision | 46.2 | 49.2 | 48.4 | 49.5 | 52.0 |
| MMMU-Pro | 41.3 | 55.9 | 41.3 | 56.2 | 57.5 |
| MMMU | 66.4 | 69.1 | 65.1 | 69.4 | 70.1 |
| V* | 82.2 | 84.8 | 85.3 | 84.8 | 85.9 |
| MMStar | 70.5 | 73.5 | 70.9 | 73.6 | 74.8 |
| BLINK | 64.4 | 65.1 | 67.6 | 65.6 | 66.6 |
| SciQA | 94.4 | 95.6 | 94.4 | 95.8 | 96.6 |
| PhyX | 45.5 | 72.1 | 46.0 | 72.1 | 73.4 |
| OlympiadBench-Phy | 7.9 | 8.1 | 9.4 | 8.6 | 9.9 |
| MME-RealWorld-Lite | 48.7 | 51.9 | 49.1 | 52.8 | 54.8 |
| Average | 58.3 | 64.1 | 59.1 | 64.3 | 65.7 |
We have three observations.
-
The contrastive data carry a real but narrow perceptual signal. CD-only improves over the base model by +0.8 on average, with the largest gains on perception-heavy benchmarks: BLINK (+3.2) and V* (+3.1). It even achieves the highest BLINK score among all configurations. This confirms that the contrastive pairs themselves are informative about fine-grained visual grounding.
-
Under an outcome reward, this signal does not translate into consistent reasoning gains. CD-only declines on MathVista (−1.4) and MMMU (−1.3) and leaves MMMU-Pro unchanged (0.0). Its average (59.1) remains far below the RL baseline (64.1).
-
This clarifies the limitation of DA-RL. Once the contrastive data are mixed with task data under a shared outcome reward (DA-RL), CD-only's perception gains largely disappear (BLINK 65.6 vs. 67.6), and the overall gain over the RL baseline is only +0.2. ContextRL consumes the same data through a dense, logit-level objective. It largely retains the perception gains while preserving the benefit of task RL (+1.6 over the RL baseline), which confirms that the limitation of DA-RL lies in how the contrastive signal is consumed, not in the data itself.
Q2: DA-SFT settings and collapse
DA-SFT settings. DA-SFT has two stages: (1) SFT on the contrastive pairs, and (2) standard GRPO on the task data, initialized from the SFT checkpoint, with the same hyperparameters as the RL baseline and no context-awareness loss. Stage 1 uses full-parameter fine-tuning: AdamW, peak learning rate 1e-5 (cosine decay, 3% warmup), weight decay 0, gradient clipping 1.0, and bf16, with the vision encoder frozen for the VL models.
| Klear-AgentForge-8B | Qwen3-8B | Qwen2.5-VL-7B | Qwen3-VL-8B | |
|---|---|---|---|---|
| SFT pairs | 1,000 | 1,000 | 7,000 | 7,000 |
| Batch size | 16 | 16 | 32 | 32 |
| Epochs / steps | 2 / 125 | 2 / 125 | 2 / 438 | 2 / 438 |
| Max sequence length | 32,768 | 32,768 | 4,096 | 4,096 |
Is the collapse an implementation issue? We believe not. First, the SFT stage itself works as intended: DA-SFT attains the highest context-selection accuracy among all methods (Figure 4), and the same pipeline yields models on par with the RL baseline in the multimodal setting (Table 4). Second, the failure is systematic. In most failed trajectories, mini-SWE-agent reports a format error (found 0 actions) after a few turns: the model emits long outputs without a THOUGHT section or bash command, so no action can be executed. Klear-AgentForge-8B shows a similar severe degradation (28.0 → 6.4); Qwen3-8B starts from a much weaker baseline (5.0), so it drops to 0. Once the format collapses, nearly all rollouts receive zero reward, so stage-2 GRPO has no signal to recover. We attribute this to a distribution shift. Context-selection targets are short, single-turn answers, whereas agentic coding requires many turns, each consisting of a THOUGHT section and one bash command. SFT on the former disrupts the multi-turn format that the agent scaffold requires.
We hope these clarifications and new experiment results address the concerns and make the paper suitable for a higher rating.
Thank you to the authors for their detailed response. Most of my concerns have been addressed. However, I noticed that the reported base-model scores for Qwen3.5-9B are substantially lower than those in the official Qwen3.5-9B model card [1]. Specifically, the scores are 22.0 points lower on MathVision, 24.8 points lower on MMMU-Pro, and 11.8 points lower on MMMU. These substantial discrepancies raise concerns about the reliability of the Qwen3.5-9B results. Could the authors check their evaluation logs to investigate the reasons for these discrepancies?
Response to Reviewer hCEF (Part 1/2)
Dear Reviewer hCEF,
Thank you for the careful and constructive review. We appreciate your recognition that context selection is an important challenge for post-training, particularly as coding agent trajectories grow longer. We are also grateful that you found our experimental design logical and our evaluation across a broad range of benchmarks convincing. We address your concerns below.
W1: Gap between context selection and grounding
We agree that context selection and grounding are not equivalent. A key contribution of our work is to formalize context grounding, which is difficult to label and scale, as a contrastive selection problem. This formulation makes data construction scalable and lets the signal be incorporated directly into RL training. To verify that better selection translates into better grounding, we provide three complementary lines of evidence:
Grounding-focused benchmarks (Table 1). LongBench v2 and NIAH directly test context grounding, as they require the model to retrieve answer-relevant evidence from long inputs. On these benchmarks, ContextRL improves over GRPO by nearly 5 points with Klear-AgentForge as the base model.
Gains concentrate where context grounding is required (new, Appendix G). On SWE-Bench Verified, ContextRL improves file-level localization by +3.6 points over GRPO, versus +1.0 for repair conditioned on correct localization. On LongBench v2, accuracy improves from 25.7% to 31.2% on hard examples (+5.5 points) and from 24.1% to 28.7% on long inputs (+4.6 points) (Appendix G.1). On MathVerse, Qwen2.5-VL improves from 43.4% to 48.6% on Vision Intensive (+5.2 points) and from 41.1% to 45.9% on Vision Only (+4.8 points) (Appendix G.2). We provide detailed subtask analyses in Appendix G, offering further evidence of ContextRL's effectiveness.
Case studies (new, Appendix L). In the revised paper, we add six case studies (three agentic, three multimodal) in which the GRPO model answers incorrectly because it fails to ground its prediction in the relevant context, while ContextRL identifies the correct context and answers correctly.
W2: Effectiveness on more frontier models
We trained Qwen3.5-9B with the same training data and configuration as in our multimodal experiments. Both the GRPO baseline and ContextRL are trained for one epoch. As shown in Table A, ContextRL outperforms the GRPO baseline on most benchmarks and raises the average by +1.3 points.
Notably, on this stronger initial model, outcome-only RL brings only +1.2 points over the initial model, while ContextRL adds a further +1.3 on top of it, more than doubling the total gain from RL training (+2.5 vs. +1.2). This is consistent with Fig. 1, where Qwen3.5-9B remains close to random on the context-selection probe: context grounding is still a bottleneck for frontier open models, and outcome rewards alone do not address it.
Regarding the smaller absolute gain on Qwen3-VL-8B than on Qwen2.5-VL-7B (Table 2), this is expected because its RL baseline is already much stronger (64.1 vs. 51.4), leaving less room for improvement. If we instead measure what fraction of the RL baseline's remaining errors ContextRL fixes, the two models are comparable: about 4.1% on Qwen2.5-VL-7B (2.0 of 48.6 points) and 4.5% on Qwen3-VL-8B (1.6 of 35.9 points).
Table A. Results on Qwen3.5-9B.