- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes CounterCredit, a training method that rewards visual requests only when they are both needed and used. Using gold-answer scores, it compares the actual crop with answering immediately and with matched random crops. Verified requests on correct trajectories receive cashback, while other requests incur charges. Outcome rewards and crop prices enter training through separate channels.
The main empirical evidence supports the complete training recipe. However, the external-scorer validation and the interpretation of the single-channel ablation require correction or qualification.
Critical strengths
- The matched training comparison supports the complete method.
Table 3 reports average accuracy of 76.1% for CounterCredit, compared with 68.9% for outcome-only training and 71.6% for the tested branch-reward control. This provides direct evidence that the proposed recipe improves performance under the evaluated conditions.
- The gains are not explained merely by making fewer requests.
On V*, CounterCredit and outcome-only training make similar numbers of requests: 1.78 and 1.84 per question, respectively. The performance difference therefore supports examining request quality rather than request count alone.
- Answer-level interventions provide additional behavioral evidence.
Table 9 reports that, at the first request, returning the actual crop instead of removing the request raises CounterCredit's V* accuracy from 78.5% to 85.3%. The corresponding change under outcome-only training is -1.1 percentage points. This supports the claim that CounterCredit learns to make requests whose returned information is useful to subsequent answers.
Critical weaknesses
- The external-scorer claim needs correction.
Section 4.2 describes the common and external scorers as preserving the spurious-call result and changing the baseline ranking by at most one adjacent swap. However, Appendix F.1 states that the external scorer checks only the actual crop against random patches. It omits the decision comparison and verification margin, so it does not reproduce the full spurious-call rate.
The reported correlation of 0.75 is against the common scorer, not the agents' own scorers. The adjacent-swap description instead belongs to the common-versus-own comparison. One adjacent swap among seven untied ranks gives a Spearman correlation of approximately 0.964, not 0.75.
The paper should report these diagnostic endpoints and ranking comparisons separately. The external check provides partial validation, but should not be described as reproducing the complete diagnostic.
- The single-channel ablation changes more than channel separation.
The single-channel arm achieves 64.5% average accuracy, compared with 76.1% for CounterCredit. This supports the complete implementation, but does not isolate the effect of channel separation because the ablation also lacks the variance floor.
Moreover, the single-channel arm can retain all-wrong groups with unequal crop charges, whereas CounterCredit masks the price advantage in those groups, as described in Section 3.5 and Appendix C, Algorithm 1. These differences can affect the training signal independently of channel separation.
Proposition 4's scale-cancellation argument remains valid. However, attributing the full empirical difference to channel separation requires either a comparison using the same variance floor and masking rule or a narrower interpretation of the existing ablation.
Questions for authors
-
Can the scorer-validation results be revised to separate the full spurious-call diagnostic from the external actual-versus-random-crop check?
-
Can the authors explicitly identify which scorer pairs produce the reported correlation and adjacent-swap ranking results?
-
Can the single-channel comparison use the same variance floor and all-wrong-group masking rule as CounterCredit? If not, can the attribution claim be narrowed to reflect the combined implementation differences?
-
Can the key matched comparisons be repeated across additional seeds to assess the stability of the reported gains?
4 comments
Thanks to Reviewer BvAt
We would like to sincerely thank you again for your dedicated reviewing effort and constructive feedback. We hope our response has successfully clarified the issues you raised. Please let us know if there are any lingering questions, as we warmly welcome the opportunity for further discussion.
Thanks to Reviewer BvAt
We would like to thank you again for reviewing our paper and providing valuable feedback, and particularly for recognizing the strengths and contributions of our work.
With only one day remaining in the discussion period, we would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns. We would be more than happy to answer any additional questions or provide further clarifications.
Thank you again for your time and consideration. Your feedback would be greatly appreciated.
A Gentle Reminder of the Final Feedback
Thank you very much again for your initial comments. They are extremely valuable for improving our work. We would be very grateful if you could take a look at our responses and revisions and let us know if there is anything else we could incorporate into the next version.
We thank the reviewer for the careful reading and for recognizing the gains under matched training conditions. We have added the requested controls to address the scorer validation and single-channel ablation, and repeated the central comparison across seeds. Changes are marked in blue in the revised PDF.
W1 / Q1 / Q2. Scorer validation
We agree that the original outside check validated only E and did not establish the full spurious-call result. We have now added the answer-now comparison and evaluated the complete diagnostic.
Two ranking statements also need correction. The original ρ=0.75 compared the outside scorer with the common scorer. The “one adjacent swap” result (ρ=0.964) compared the common scorer with the agents' own scorers on V*. Section 4.2 should have reported these separately.
Both the common SFT scorer and GPT-5.6 Terra now compute D, E, and the fraction of calls with min(E,D)≤ε, for all eight agents on V* and HR-Bench-4K. Each ρ below compares the seven baselines' ranking under the indicated scorer with their ranking under their own scorers (Table 2).
CounterCredit has the lowest full spurious-call rate in all four comparisons. Baseline ranks do change: 1–4 of the 21 pairs are reordered. Thus, the new evaluation supports the result for CounterCredit without requiring the entire baseline ordering to remain unchanged.
In the revision, Appendix F.1 replaces the E-only analysis with these results (Table 8). Section 4.2 identifies the scorer pair for each correlation and reports that one to four of the 21 baseline pairs change order. “One adjacent swap” is stated only for the common-versus-own comparison on V*, in Appendix F.1. Appendix G reports the lowest rate for CounterCredit and the observed changes in baseline ordering.
W2 / Q3. Single-channel ablation
CounterCredit still outperforms single-channel GRPO after matching the variance floor and all-wrong-group mask. We trained the requested control with the same floor (σmin=0.15), mask, cold start, prompt pool, budget, and seed.
Adding the floor and mask raises the single-channel average from 64.5 to 70.5, accounting for 6.0 points of the original 11.6-point gap. With these controls matched, CounterCredit retains a 5.6-point advantage and reduces the spurious-call rate from 59.3% to 31.4%. This new ablation uses one seed.
The remaining difference is how the two arms handle the price term. Single-channel GRPO divides it by max(std(O+P),σmin), so its scale still depends on group composition and can be amplified by up to 1/σmin≈6.7× in all-correct groups. The dual-channel price term is not normalized. This is the distinction analyzed in Proposition 4. The matched single-channel policy also makes fewer calls on V* (1.27 vs. 1.78 per question).
Table 3 now includes both single-channel rows, Section 4.4 reports the controlled comparison first, and Appendix F.6 gives the breakdown above.
Q4. Seeds
The improvement over outcome-only GRPO holds across all three evaluated seeds. We repeated outcome-only GRPO over three seeds. Its seven-benchmark average is 69.0±0.61, compared with 76.0±0.40 for CounterCredit (Tables 14 and 15, Appendix F.4). Each ± is the sample standard deviation of the three per-seed averages as listed in those tables. CounterCredit is higher on every seed, with a 7.0-point difference between the means.
The remaining ablation arms in Table 3 are single-seed.