- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines how evaluation rules affect comparisons of recovery methods for tool-using agents. It distinguishes two questions: whether additional recovery calls directly affect correctness scores, and how accuracy should be traded against token cost. The authors formalize conditions under which additional recovery calls reduce scores under call-sensitive evaluation rules. They also derive an upper bound on the cost weight at which the accuracy-maximizing method ceases to be uniquely optimal under an additive accuracy–cost objective. The experiments rescore identical BFCL recovery traces with recovery calls included or excluded, and examine how cost preferences affect method recommendations across two models, nine methods, and five benchmarks. The results show that both scoring rules and cost metrics can affect the conclusions of recovery-method comparisons.
Critical strengths
-
A useful distinction between scoring rules and cost preferences. Scoring rules determine whether recovery is counted as successful, while cost preferences determine which method is preferable. This distinction has practical and explanatory value: it helps separate effects of the scoring rule from effects of the accuracy–cost trade-off. It also supports a useful reporting recommendation: recovery comparisons should explicitly state their success criterion and cost preference.
-
Broad empirical coverage. The evaluation covers two models, nine methods, and five benchmarks, reporting both accuracy and token cost. The range of settings and evaluation perspectives strengthens the empirical basis and credibility of the findings.
Critical weaknesses
-
The cost metrics are not calibrated to the same cost preference, weakening the claim that the ratio metric has a blind spot. The weights in the additive and ratio objectives have different meanings, so sweeping the same numerical interval does not represent the same accuracy–cost trade-off. The paper explains why the ratio form is insensitive within its chosen parameter window, but does not establish that the two metrics are evaluated under equivalent cost preferences.
-
The BFCL experiment establishes a scoring penalty without independently verifying successful task recovery. It clearly shows that additional recovery calls lower call-matching scores, but does not establish that an injected fault causes an actual task failure that recovery subsequently resolves. The evidence therefore supports the existence of a scoring penalty, but does not substantiate the stronger interpretation that a successfully repaired task is incorrectly judged to have failed. The paper should clarify the scope of this stress test and whether its findings warrant criticism of BFCL’s original design.
Questions for authors
-
How should the claimed blind spot be interpreted after matching cost preferences? How should the conclusions of Section 4.3 be interpreted when the additive and ratio objectives are calibrated to represent the same accuracy–cost preference? Is the claim that the existing parameterization has limited sensitivity within the chosen numerical window, or that the ratio metric recommends inappropriate methods under a shared deployment objective? If the latter, please specify that objective and provide the parameter calibration or utility analysis needed to support the claim.
-
What evidence shows that the penalized recovery rescues tasks that would otherwise fail? Does the BFCL probe include an independent final-answer or environment-state checker to verify that recovery rescues tasks that would otherwise fail? If so, please identify the corresponding recovery-on and recovery-off results under the same faults. If not, please clarify which conclusions concern call-matching scores alone and which concern actual task completion, and identify the evidence supporting the latter.
Thank you for the additional experiments and analysis. They addressed some of my concerns. I’ll keep my original rating.