- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to train VLMs with RL to use visual tools such as crop-and-zoom. The authors observe that many training sets have low Visual Tool Necessity (VTN): the base model can answer most questions without tools. Models trained on such data learn the tool-call format but do not really rely on the cropped evidence. Hard, high-VTN data is needed, but GRPO on hard data is inefficient, since many rollout groups get all-zero rewards and give no learning signal. Dynamic sampling filters these groups after rollout and regenerates them, which is costly for multi-turn tool use. The paper proposes Calibrated Dynamic Difficulty Shaping (CalDS). It selects examples before rollout so that batch difficulty matches the model's current ability under a fixed rollout budget.
Contributions:
- It defines VTN and builds a hard, high-VTN training set. Images average 16.45M pixels, and the base Qwen3-VL model gets only 5.53% accuracy without tools.
- It proposes CalDS. Examples are grouped into difficulty buckets based on recent tool-augmented rollout accuracy. A bounded feedback controller targets a batch success rate of 0.5, which maximizes mixed-outcome groups in GRPO. Bucket estimates are calibrated online with an EMA over recent rollouts. Center-preserving shaping compresses the scores of middle buckets to avoid migration bottlenecks.
- It annotates ground-truth evidence boxes for VisualProbe. This enables crop localization accuracy and joint answer-and-localization accuracy.
Validation (Qwen3-VL-8B-Instruct):
- CalDS achieves state-of-the-art results among open-source visual tool-use models on V*Bench, HR-Bench, and VisualProbe. It outperforms Mini-o3 and SenseNova-MARS, which rely on costly multi-node SFT cold starts and larger datasets. CalDS applies RL directly to the base model on filtered hard examples. On VisualProbe Hard, it reaches 79.3% crop localization accuracy and 46.7% joint accuracy.
- Dynamic sampling discards 67.2% of rollouts on the hard set, while CalDS keeps all of them. Batch rollout time drops from 35.7 to 9.1 minutes (about 3.9x). Total GPU-hours drop to about 1/2.8, including the one-time bucket initialization. Ablations show that each component helps. The mean training reward stays near 0.5, bucket accuracy is tracked without extra evaluation cost, and samples stay balanced across middle buckets.
- When the returned crops are masked at inference, the gain over the baseline disappears. This shows the gains come from using visual evidence, not from fitting the tool-call format.
Critical strengths
-
Difficulty control before rollout, with no extra rollout cost. With GRPO on hard data, many groups get all-zero or all-one rewards and give no gradient. Dynamic sampling (e.g., DAPO) drops these groups after generation and resamples, which is costly in multi-turn tool use. CalDS sets batch difficulty before rollout. It uses dynamic difficulty buckets, EMA-based online calibration, and a covariance-normalized feedback controller to keep the mean batch reward near 0.5. It keeps 100% of generated rollouts, compared with 32.8% for dynamic sampling. Batch rollout time drops from 35.7 to 9.1 minutes (3.9x faster), and accuracy on VisualProbe is higher.
-
Principled derivation of the sampler. The sampling rule comes from a step-by-step derivation, not from heuristics (Sections 2.2 and 2.3). The goal of keeping the expected batch reward near 0.5 is cast as a KL-regularized problem. This gives an exponential sampling distribution with a single scalar parameter (Eq. 3 and 4). A center-preserving power transform compresses scores near 0.5 and keeps middle buckets balanced (Eq. 7). The gradient of the expected batch reward with respect to the control parameter is exactly the negative covariance between calibrated bucket accuracy and difficulty score (Eq. 8). Dividing the reward error by this covariance gives a first-order Newton-style update (Eq. 5, Appendix E.2). So every part of the update rule is justified.
Critical weaknesses
- Limited evaluation scope. All experiments use a single base model (Qwen3-VL-8B-Instruct) and a single deterministic tool (crop and zoom-in). It is unclear whether the difficulty controller generalizes to other model scales. It is also unclear whether it works in agentic settings with multiple heterogeneous tools, such as web image search, geometric transformations, or Python code execution (as in DeepEyes V2 and Thyme).
Questions for authors
Reliability of per-prompt difficulty estimates. Figure 4(a) shows that online calibration estimates the average reward of each bucket well. However, some prompts may not be revisited for many steps, especially early in training when the model improves quickly. Their true success rates may then differ a lot from their stored estimates, while they stay in their original buckets.
- How often is each prompt revisited during training? Please report the distribution of revisit counts (e.g., 0, 1, 2 to 5, and more than 5).
- You already evaluate the training set at step 100 in Figure 4(a). Can you report the distribution or variance of per-prompt accuracy within each bucket?
- Are prompts that were not sampled recently more likely to have inaccurate difficulty estimates? This would show whether accurate bucket-level estimates hide large per-prompt errors, which could affect the reliability of the curriculum.
Dear Reviewers,
The authors have submitted their rebuttal and tried to address the concerns raised in your reviews. If you haven't already, we would appreciate it if you could take the time to read their responses and engage in the discussion, particularly to assess whether your concerns have been adequately addressed.
Thank you for your time and valuable contributions to the review process!
Best, AC