acceptodds
Under review as a conference paper at ICLR 2027

Asking Earns Nothing: Scoring the Decision to Act in BFCL Multi-Turn

Abstract

An agent that lacks the information it needs should ask rather than act, and the task definitions of agent leaderboards say so. BFCL multi-turn builds two of its four categories around a turn on which the model is supposed to ask, and its scorer never looks at that turn: the gold trajectory there is empty, the checker skips it, and the scripted user cannot answer, so asking earns nothing, guessing costs nothing on that turn, and asking twice loses the item. The benchmark also contains the control experiment for that decision. A should-ask item is a base item with one piece of information removed from one turn, so the same request appears twice at the same turn index, once complete and once not: on the first the model should make the call that changes the world, on the second it should ask. We score one decision per pair, whether the model attempted a world-changing call on that turn, read off the stored trajectories with no LLM judge; acting always and asking always both score 50. On the 223 pairs that pose this decision, gpt-5.4 attempts the call on 83.4% of the complete turns and holds back on 78.0% of the incomplete ones, the best decision accuracy of seven models at 80.7%; on the same items the official score ranks it sixth and puts first a model that lands in the middle here. One added line telling gpt-5.4 not to ask pushes it toward acting on both sides of the pair, so its decision accuracy shows no detectable change, while its official score rises by 13.5 to 23.5 points on the two should-ask categories and on the base twins; the opposite line, telling gemma-4-31B-it to ask first, improves its decision by 4.5 points and gains no official score. The score moves with the push toward action, not with the decision. We release the pairs, a turn-level scorer that runs on any BFCL output directory without an API key, and 31 manually verified bad items.

Primary area:
datasets and benchmarks
Keywords:
tool calling, LLM agents, benchmark validity, BFCL, clarification, evaluation, agent calibration, multi-turn function calling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_n2KgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper audits BFCL Multi-Turn and shows that its official scorer skips the under-specified turns where agents are supposed to ask for clarification. The authors exploit BFCL’s existing base/missing perturbation structure to construct 223 matched “twin” pairs and introduce a judge-free decision accuracy based on whether the model attempts a world-changing action. Using this metric, they show that the official BFCL score can substantially misrepresent models’ ability to distinguish when they should act from when they should hold back. Prompt interventions further support this mismatch.

Critical strengths
  1. The paired evaluation provides a simple and well-structured way to isolate the act-versus-ask decision. The proposed metric avoids relying on an LLM judge and gives both “always act” and “always abstain” strategies a 50% baseline.
  2. The empirical analysis is well validated. The authors manually certify the relevant anchor turns and perform several robustness checks addressing alternative action definitions, compound turns, pre-anchor trajectory differences, and the semantic content of held-back responses.
Critical weaknesses
  1. The rank reversal is shown on 223 twin pairs, but it is unclear whether correcting this scoring issue would materially change the overall BFCL multi-turn ranking.
  2. The intervention study is only partially crossed: gpt-5.4 receives the pro-action prompt and gemma-4-31B-it receives the pro-caution prompt, but neither model is tested under both directions. This makes it unclear whether the intervention findings generalize across different models.
Questions for authors
  1. Do the authors have an estimate of how incorporating the proposed decision metric would affect the aggregate BFCL Multi-Turn ranking among the evaluated models?
  2. Were the crossed intervention conditions or additional models evaluated (e.g., pro-caution for gpt-5.4, pro-action for gemma-4-31B-it, or the same interventions on other models)?
2 comments

Dear Reviewer n2Kg,

The discussion period closes on Nov 18 (AoE), so we are writing to check whether our response addressed your concerns.

In brief: on Q1, adding the decision column to the multi-turn aggregate leaves the order of the seven models the same under one aggregation and swaps ranks 2 and 3 under the other, while the gap from gpt-5.4 to the top model shrinks from 26.8 to under 10 points; the aggregate is dominated by the 48-point spread on the executed turns. On Q2, the full 2×2 on three models is in the general response.

If anything remains unclear or unresolved, please let us know and we will respond before the deadline.

Thank you again for your time.

AuthorsOpenReview ↗

Follow-up: aggregate ranking and crossed interventions

Dear Reviewer n2Kg,

With about a day left in the discussion period, we would like to return to your two questions. Both are now in the revised PDF, at the places given below.

Does correcting the scorer matter for the overall ranking? (Appendix H) The order of the seven models changes little: no rank changes under one aggregation, and ranks 2 and 3 swap under the other, where gpt-5.4's gap to the top model shrinks from 26.8 to under 10 points. We read the small change in order as the finding itself. Two of the four multi-turn categories were built to test when an agent should ask, yet the aggregate is computed from the executed turns, so the decision those categories exist for cannot move it. A benchmark designed to test the act-or-ask decision reports a score from which that decision is absent. The paper argues this should change, and the paired column is how.

The crossed interventions. (Section 5.4, Table 2, Appendix G) All six cells on three models: both prompts on gpt-5.4, on gemma-4-31B-it, and on a third model.

If these additions answer your two questions, we would be grateful if you would reconsider your rating. If anything remains unclear, we will answer before the deadline.

Reviewer_XhiyOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The two should-ask categories of BFCL multi-turn are built around a turn where a function or parameter is missing, but the official scorer skips that turn. The authors pair each should-ask item with its complete base version and score whether the model makes a state-changing call on that turn in both versions, giving a balanced decision accuracy with no LLM judge.

Critical strengths
  1. The main finding is clear and well supported. The official scorer skips the turn that the should-ask categories are built for, and the paper shows this with concrete examples, the history of the scoring code, and officially passed items that still include guesses on that turn.

  2. Research question is valuable. The paper directly measures whether an agent stops to ask the user when information is missing, an important part of agent behavior that leaderboard scores like BFCL’s do not capture.

  3. The work is careful. All 231 pairs are checked by hand, 31 wrong reference answers are found, and the released scorer lets anyone rerun the results. The prompt experiment also shows that the official score misses changes in this decision in both directions.

Critical weaknesses
  1. On the complete side, any state-changing call counts as the right decision, even if it is not the function the turn asks for. This is looser than the paper's own definition ("commits to the action the turn requests"). It would help to report how often the function differs from the reference, and how the results change if it must match.

  2. On the missing side, holding back is always counted as correct. By the certification rules, a value still counts as missing even if the model could find it with a read-only call, and cheap, reversible actions are treated the same as orders or messages that cannot be undone. It would help to report how many withheld values can be found in the environment, and to separate reversible from irreversible actions.

Questions for authors
  1. For all seven models, how often do officially passed should-ask items include a state-changing call on the missing-info turn, and how would the official scores change if these were counted as failures?

  2. Each prompt is tested on only one model, and each model seems to have a clear weakness in the direction it is pushed. gpt-5.4 is the most cautious model and stops on later turns more often than the others, while gemma-4-31B-it is much more willing to act. The prompt effects may therefore come partly from fixing each model’s own weakness, rather than from a general property of the official score. Would the results still hold if both prompts were run on both models, or on a few more models?

Reviewer_uR4rOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary
  • The paper identifies a certain shortcoming in a popular function calling benchmarK: BFCL (specifically, the multi-turn category)
  • Specifically, they identify a flaw in the evaluation protocol of BFCL v4 multiturn category, where:
    • For certain samples in certain categories, the benchmark intentionally withholds information from the agent and the agent's goal is to recover it before proceeding.
    • However, benchmark does not evaluate this particular step where the agent attempts to recover the information. Furthermore, since the evaluation follows comparison with a scripted ground-truth trajectory, a subsequent question by the agent is penalized.
  • The paper proceeds to benchmark some popular LLMs (e.g., gpt-5.4, qwen) and analyze the discrepency.
  • Finally, the paper proposes a scoring method to mitigate this issue.
Critical strengths
  1. Identifying flaw in a popular benchmark: Indeed, BFCL is a popular benchmark used by many model developers to highlight advances in agentic categories. Highlighting a flaw as done in this paper is useful to the community.
Critical weaknesses

Major Concerns

1. Scope of Contributions; unclear whether ICLR is the right venue

  • Although the paper presents a flaw in a popular benchmark, I believe the findings are highly narrow in scope: the paper highlights a flaw in two categories of a very specific benchmark (which has 15+ categories).
  • As a result, the contributions (also listed by authors) appears to be:
    • Identifying the flaw (which appears to be already known by the community e.g., Zhao et al., EMNLP 2026)
    • Simple evaluation to quantify the flaw
    • Mitigating the flaw using a prompt modification
  • While I strongly believe that more robust evaluation protocols (after all, this is how we measure progress) make for great contributions, my concern is that paper exposes this in a very narrow setting and it is unclear whether the findings generally extend beyond this specific benchmark.
  • Given the narrow scope, I am concerned whether ICLR main conference makes for a reasonable venue for such a contribution.

2. Writing: Difficult to read

  • Another major concern is that the current state of writing.
  • On a more high-level, I find lacking a reasonable structure in the paper that helps the reader easily follow the core arguments and contributions.
    • There are many definitions that the paper invents (e.g., 'empty-turn rule', 'should-ask side') -- but is often used even before the definition.
    • Usage of these keywords also often appears before providing the reader preliminaries of the benchmark.
    • The definitions are not presented in a systematic manner. I find them scattered through the paper e.g., 'should-ask' defined multiple times identically, 'world-changing' defined much later in Section 4.1.
  • On a more low-level, I also believe many parts of the paper somewhat interfere with reading.
    • Multiple unusual or potentially incomplete sentences e.g., beginning introduction paragraph with "An anomalous ranking sent us there." (what ranking? sent where?)
    • The paper misses a clear flow in presenting its arguments e.g., multiple disconnected sentences appearing side by side "A tool call the model makes there is still executed and still changes the world, it is simply never compared with anything. Ask or guess, what the model did on the under-specified turn is never scored there. A guess enters the verdict only later, through the state it leaves behind, and a question does not enter it at all."

Minor Concerns

3. High scores: Artifact of Contamination?

  • The paper raises an interesting insight: some tasks require the model to gather information before proceeding (e.g., which stock to buy in a 'buy stock' task). Nonetheless, some models appear to hallucinate the correct arguments in this case.
  • While the paper primarily looks at the evaluation protocol issue, I wonder whether the hallucination here can be attributed to contamination. If the training data of models is contaminated, this indicates a larger problem than cannot be fixed be rewriting the evaluation protocol.
Questions for authors

Clarification requests:

  1. (wrt Concern 3) Can we attribute the high official scores to contamination?

Suggestions for future work:

  1. Improve writing
  2. Improving analysis so that readers who are not first-hand experienced with BFCL can gain relevant insights
2 comments

Dear Reviewer uR4r,

The discussion period closes on Nov 18 (AoE), so we are writing to check whether our response addressed your concerns.

In brief: the skipped turn is a documented scorer decision, not our reading of it; the 31 defective tasks are a separate deliverable and removing them changes no number by more than 0.9; of 183 guessing attempts on the missing values, only one hit a value that is specific, not derivable from the dialogue and not a common default; and the writing of the introduction and §3 is being reworked as described in our point 2.

If anything remains unclear or unresolved, please let us know and we will respond before the deadline.

Thank you again for your time.

AuthorsOpenReview ↗

Revised PDF uploaded: writing and scope

Dear Reviewer uR4r,

With about a day left in the discussion period, we would like to return to your two major concerns. We have uploaded a revised PDF. The findings and Table 1 are the same as in the submission; what changed is listed below by section.

Writing. The revision acts on every point you raised. A new Section 2 (Preliminaries) introduces the benchmark and defines each term once, before it is used: the four categories, the under-specified turn, the should-act and should-ask sides, the empty-turn rule, twin pairs, and world-changing calls. The introduction no longer opens with "An anomalous ranking sent us there" and states the problem directly. The three sentences you quoted are now one connected argument ("The skipped turn", end of Section 2).

Scope. By count, multi-turn is four of BFCL's categories. By weight, it is 30% of the overall leaderboard score, second only to the agentic track (40%). It is BFCL's test of multi-step tool use in a stateful environment: 800 tasks, each a scripted multi-turn dialogue inside a simulated API domain, the size of a standalone agent benchmark. Our audit covers that whole track. Two of its four categories were built to test whether an agent asks when information is missing, the official score does not register that decision in either, and the 31 defective tasks we release span all four. The finding therefore concerns how a widely reported agent score is computed, not a corner of the benchmark.

If the revision resolves these two concerns, we would be grateful if you would reconsider your rating. If either still stands, we would value hearing why, and we will answer before the deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.