acceptodds
Under review as a conference paper at ICLR 2027

Learning to Zoom at the Right Difficulty: Dynamic Curricula for Visual Tool-Use Reinforcement Learning

Abstract

Crop-and-zoom tools enable vision-language models to acquire fine-grained visual evidence and answer questions that require visual detail beyond their native input resolution. Prior work shows that reinforcement learning on tasks solvable without visual tools can induce overfitting rather than improve tool use capability, motivating training on hard tool-essential data. However, training on challenging datasets can be highly inefficient under Group Relative Policy Optimization (GRPO), as filtering out numerous uninformative samples with all-zero rewards requires substantial additional rollout generation. This overhead is especially high when visual tool use involves multiple rounds of generation, tool execution, and visual feedback. We propose Calibrated Dynamic Difficulty Shaping (CalDS), an adaptive sampling framework that makes efficient use of hard examples by matching training-batch difficulty to the model's evolving capabilities. CalDS organizes examples into dynamically updated buckets based on tool-augmented accuracy. It uses a bounded reward-feedback controller with difficulty shaping to adjust sampling probabilities based on bucket difficulty and reward history, emphasizing challenging but currently learnable examples. To account for stale difficulty estimates as the policy evolves, CalDS reuses recent rollouts to calibrate bucket difficulty estimation. On VisualProbe, a challenging high-resolution VQA benchmark, CalDS improves both answer accuracy and crop localization accuracy at a much lower training cost. These results highlight adaptive difficulty control as a means of learning effective visual tool use efficiently from hard training data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision Language Models, Visual Tool Use, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_a6JLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to train VLMs with RL to use visual tools such as crop-and-zoom. The authors observe that many training sets have low Visual Tool Necessity (VTN): the base model can answer most questions without tools. Models trained on such data learn the tool-call format but do not really rely on the cropped evidence. Hard, high-VTN data is needed, but GRPO on hard data is inefficient, since many rollout groups get all-zero rewards and give no learning signal. Dynamic sampling filters these groups after rollout and regenerates them, which is costly for multi-turn tool use. The paper proposes Calibrated Dynamic Difficulty Shaping (CalDS). It selects examples before rollout so that batch difficulty matches the model's current ability under a fixed rollout budget.

Contributions:

  1. It defines VTN and builds a hard, high-VTN training set. Images average 16.45M pixels, and the base Qwen3-VL model gets only 5.53% accuracy without tools.
  2. It proposes CalDS. Examples are grouped into difficulty buckets based on recent tool-augmented rollout accuracy. A bounded feedback controller targets a batch success rate of 0.5, which maximizes mixed-outcome groups in GRPO. Bucket estimates are calibrated online with an EMA over recent rollouts. Center-preserving shaping compresses the scores of middle buckets to avoid migration bottlenecks.
  3. It annotates ground-truth evidence boxes for VisualProbe. This enables crop localization accuracy and joint answer-and-localization accuracy.

Validation (Qwen3-VL-8B-Instruct):

  1. CalDS achieves state-of-the-art results among open-source visual tool-use models on V*Bench, HR-Bench, and VisualProbe. It outperforms Mini-o3 and SenseNova-MARS, which rely on costly multi-node SFT cold starts and larger datasets. CalDS applies RL directly to the base model on filtered hard examples. On VisualProbe Hard, it reaches 79.3% crop localization accuracy and 46.7% joint accuracy.
  2. Dynamic sampling discards 67.2% of rollouts on the hard set, while CalDS keeps all of them. Batch rollout time drops from 35.7 to 9.1 minutes (about 3.9x). Total GPU-hours drop to about 1/2.8, including the one-time bucket initialization. Ablations show that each component helps. The mean training reward stays near 0.5, bucket accuracy is tracked without extra evaluation cost, and samples stay balanced across middle buckets.
  3. When the returned crops are masked at inference, the gain over the baseline disappears. This shows the gains come from using visual evidence, not from fitting the tool-call format.
Critical strengths
  1. Difficulty control before rollout, with no extra rollout cost. With GRPO on hard data, many groups get all-zero or all-one rewards and give no gradient. Dynamic sampling (e.g., DAPO) drops these groups after generation and resamples, which is costly in multi-turn tool use. CalDS sets batch difficulty before rollout. It uses dynamic difficulty buckets, EMA-based online calibration, and a covariance-normalized feedback controller to keep the mean batch reward near 0.5. It keeps 100% of generated rollouts, compared with 32.8% for dynamic sampling. Batch rollout time drops from 35.7 to 9.1 minutes (3.9x faster), and accuracy on VisualProbe is higher.

  2. Principled derivation of the sampler. The sampling rule comes from a step-by-step derivation, not from heuristics (Sections 2.2 and 2.3). The goal of keeping the expected batch reward near 0.5 is cast as a KL-regularized problem. This gives an exponential sampling distribution with a single scalar parameter (Eq. 3 and 4). A center-preserving power transform compresses scores near 0.5 and keeps middle buckets balanced (Eq. 7). The gradient of the expected batch reward with respect to the control parameter is exactly the negative covariance between calibrated bucket accuracy and difficulty score (Eq. 8). Dividing the reward error by this covariance gives a first-order Newton-style update (Eq. 5, Appendix E.2). So every part of the update rule is justified.

Critical weaknesses
  1. Limited evaluation scope. All experiments use a single base model (Qwen3-VL-8B-Instruct) and a single deterministic tool (crop and zoom-in). It is unclear whether the difficulty controller generalizes to other model scales. It is also unclear whether it works in agentic settings with multiple heterogeneous tools, such as web image search, geometric transformations, or Python code execution (as in DeepEyes V2 and Thyme).
Questions for authors

Reliability of per-prompt difficulty estimates. Figure 4(a) shows that online calibration estimates the average reward of each bucket well. However, some prompts may not be revisited for many steps, especially early in training when the model improves quickly. Their true success rates may then differ a lot from their stored estimates, while they stay in their original buckets.

  1. How often is each prompt revisited during training? Please report the distribution of revisit counts (e.g., 0, 1, 2 to 5, and more than 5).
  2. You already evaluate the training set at step 100 in Figure 4(a). Can you report the distribution or variance of per-prompt accuracy within each bucket?
  3. Are prompts that were not sampled recently more likely to have inaccurate difficulty estimates? This would show whether accurate bucket-level estimates hide large per-prompt errors, which could affect the reliability of the curriculum.
Reviewer_8jjUOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CalDS, which combines dynamic difficulty bucketing with reward-based adaptive sampling to make GRPO training more effective under a fixed rollout budget. It improves the model's visual tool-use capabilities at a lower training cost.

Critical strengths
  • The paper identifies an important issue: while High-VTN data helps train visual tool use, it often produces many ineffective GRPO groups. The authors propose a good and reasonable solution to this problem.

  • The method is solid, and the paper provides convincing ablations to validate each component. Although the experiments only cover the crop-and-zoom tool, I understand this choice. It may reflect the authors' research focus, and making broader claims could also be harder to justify during paper submission.

Critical weaknesses
  • Qwen3-VL already has crop tool capabilities, but the paper only reports its tool-free performance, without tool-enabled performance and Acc_loc or Acc_joint. I think these results should be included to better show how much RL actually improves visual localization.

  • Currently, a trajectory is considered correct as long as any crop overlaps the GT region. This could favor models that make more tool calls. It would be helpful to report Acc_loc@1 and Acc_loc@2, or compare models under the same crop budget.

  • The authors target a batch mean reward of around 0.5, but this isn't exactly the same as maximizing the proportion of valid groups (those containing both correct and incorrect rollouts). Since CalDS can already adjust the sampling distribution before rollout, I'm curious why not sample directly based on each bucket's probability of producing valid groups. This seems like a more direct objective and wouldn't require post-hoc filtering like traditional Dynamic Sampling. I'd like to see some discussion or ablations on this.

  • I'm also curious whether the authors have tried the Tool Call Gain metric from the MED[1] paper. Does it show a clear improvement before and after training?

[1] What Does Vision Tool-Use Reinforcement Learning Really Learn? Disentangling Tool-Induced and Intrinsic Effects for Crop-and-Zoom, ICML 2026

Questions for authors

see weakness

Reviewer_zvVMOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CalDS, an adaptive curriculum framework for reinforcement learning with visual tools. It combines online difficulty buckets, reward-guided sampling, bucket-level accuracy calibration, and difficulty shaping to select training examples before rollout generation. Experiments on V*, HR-Bench, and VisualProbe demonstrate improved visual question answering performance, while comparisons with post-rollout dynamic sampling show reduced rollout cost.

Critical strengths
  1. Practical training efficiency. CalDS addresses the computational waste of generating tool-augmented rollouts that are subsequently discarded. Table 3 reports a reduction in batch rollout time from 35.7 to 9.1 minutes while maintaining or improving accuracy, demonstrating practical value for visual tool-use training.

  2. Evaluation of visual evidence acquisition. The paper supplements answer accuracy with manually annotated evidence regions and localization metrics on VisualProbe. This provides useful evidence that the trained model can locate relevant visual information.

Critical weaknesses

Limited methodological novelty and insufficient comparison with closely related methods. The methodological contribution appears to consist mainly of relatively minor refinements to existing sampling and curriculum strategies. Several central ideas have clear precedents.

  1. AdaRFT [1] adjusts curriculum difficulty using recent reward feedback, with a target success rate of 0.5, to maintain training on challenging but solvable problems.
  2. DOTS [2] predicts difficulty under the current policy and selects problems near a 50% success rate before rollout generation. It also incorporates difficulty calibration.
  3. AdaCuRL [3] combines difficulty estimation, difficulty buckets, and reward-driven curriculum scheduling, with experiments on both language and multimodal models.
  4. Prioritized Replay for RL Post-Training [4] describes online problem selection using EMA success estimates and the priority score p(1−p)p(1-p), favoring intermediate-difficulty problems before generating fresh rollouts.
  5. Dynamic Data Snoozing [5] uses historical outcomes to temporarily skip mastered examples before rollout generation, periodically revisiting them to account for changes in model performance.

These connections raise questions about the extent of CalDS’s methodological contribution and novelty, as related adaptive curriculum and sampling strategies were already publicly documented in 2025 and early 2026. Although CalDS differs in its specific calibration, control, and shaping mechanisms, Table 3 compares only against post-rollout dynamic sampling, without direct comparisons with closely related adaptive selection methods. A clearer explanation and evaluation of the advances over these approaches would help substantiate its contribution.

References:

[1] Efficient Reinforcement Finetuning via Adaptive Curriculum Learning.

[2] Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay.

[3] AdaCuRL: Adaptive Curriculum Reinforcement Learning with Invalid Sample Mitigation and Historical Revisiting.

[4] Prioritized Replay for RL Post-Training.

[5] When sleeping in saves you money: dynamic data snoozing for efficient online RL.

Questions for authors
  1. Novelty and comparison with prior work. How do CalDS’s calibration, feedback control, and difficulty shaping advance beyond the adaptive curriculum and sampling strategies mentioned in Critical Weaknesses? Since Table 3 compares only against post-rollout dynamic sampling, can the authors provide comparisons with relevant alternatives, particularly those that select examples before rollout generation, using the same backbone, training data, and compute budget, and reporting accuracy and total training cost? These comparisons would help establish the value of CalDS’s specific design choices.
  2. Scope and generality. CalDS’s sampling mechanism relies on success statistics rather than crop-specific information. What motivates evaluating it exclusively in the crop-and-zoom setting? If the paper’s main contribution is its difficulty-aware sampling method, which does not inherently depend on crop-and-zoom tools, should it also be evaluated on benchmarks beyond this setting, such as mathematical reasoning, coding, or other agentic tasks, to establish its broader effectiveness?
  3. Effect of calibration on hard examples. In Table 2, calibration yields only marginal gains on Hard set, both with and without difficulty shaping, whereas other components provide larger improvements. Can the authors explain why calibration benefits Easy and Medium more substantially but has limited impact on the hardest examples?

Comments on the paper

1 comment

Dear Reviewers,

The authors have submitted their rebuttal and tried to address the concerns raised in your reviews. If you haven't already, we would appreciate it if you could take the time to read their responses and engage in the discussion, particularly to assess whether your concerns have been adequately addressed.

Thank you for your time and valuable contributions to the review process!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.