- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper audits BFCL Multi-Turn and shows that its official scorer skips the under-specified turns where agents are supposed to ask for clarification. The authors exploit BFCL’s existing base/missing perturbation structure to construct 223 matched “twin” pairs and introduce a judge-free decision accuracy based on whether the model attempts a world-changing action. Using this metric, they show that the official BFCL score can substantially misrepresent models’ ability to distinguish when they should act from when they should hold back. Prompt interventions further support this mismatch.
Critical strengths
- The paired evaluation provides a simple and well-structured way to isolate the act-versus-ask decision. The proposed metric avoids relying on an LLM judge and gives both “always act” and “always abstain” strategies a 50% baseline.
- The empirical analysis is well validated. The authors manually certify the relevant anchor turns and perform several robustness checks addressing alternative action definitions, compound turns, pre-anchor trajectory differences, and the semantic content of held-back responses.
Critical weaknesses
- The rank reversal is shown on 223 twin pairs, but it is unclear whether correcting this scoring issue would materially change the overall BFCL multi-turn ranking.
- The intervention study is only partially crossed: gpt-5.4 receives the pro-action prompt and gemma-4-31B-it receives the pro-caution prompt, but neither model is tested under both directions. This makes it unclear whether the intervention findings generalize across different models.
Questions for authors
- Do the authors have an estimate of how incorporating the proposed decision metric would affect the aggregate BFCL Multi-Turn ranking among the evaluated models?
- Were the crossed intervention conditions or additional models evaluated (e.g., pro-caution for gpt-5.4, pro-action for gemma-4-31B-it, or the same interventions on other models)?
2 comments
Follow-up: aggregate ranking and crossed interventions
Dear Reviewer n2Kg,
With about a day left in the discussion period, we would like to return to your two questions. Both are now in the revised PDF, at the places given below.
Does correcting the scorer matter for the overall ranking? (Appendix H) The order of the seven models changes little: no rank changes under one aggregation, and ranks 2 and 3 swap under the other, where gpt-5.4's gap to the top model shrinks from 26.8 to under 10 points. We read the small change in order as the finding itself. Two of the four multi-turn categories were built to test when an agent should ask, yet the aggregate is computed from the executed turns, so the decision those categories exist for cannot move it. A benchmark designed to test the act-or-ask decision reports a score from which that decision is absent. The paper argues this should change, and the paired column is how.
The crossed interventions. (Section 5.4, Table 2, Appendix G) All six cells on three models: both prompts on gpt-5.4, on gemma-4-31B-it, and on a third model.
If these additions answer your two questions, we would be grateful if you would reconsider your rating. If anything remains unclear, we will answer before the deadline.
Dear Reviewer n2Kg,
The discussion period closes on Nov 18 (AoE), so we are writing to check whether our response addressed your concerns.
In brief: on Q1, adding the decision column to the multi-turn aggregate leaves the order of the seven models the same under one aggregation and swaps ranks 2 and 3 under the other, while the gap from gpt-5.4 to the top model shrinks from 26.8 to under 10 points; the aggregate is dominated by the 48-point spread on the executed turns. On Q2, the full 2×2 on three models is in the general response.
If anything remains unclear or unresolved, please let us know and we will respond before the deadline.
Thank you again for your time.