- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces HiL-Bench, a benchmark evaluating whether agents can recognize when they need human clarification rather than proceeding with unsupported assumptions. It constructs 300 software-engineering and text-to-SQL tasks containing 1,131 human-validated blockers, including missing information, ambiguous requirements, and contradictory instructions.The authors further show that reinforcement learning with shaped Ask-F1 rewards improves clarification behavior and downstream task completion, with some cross-domain transfer.
Critical strengths
- Well-motivated and carefully controlled benchmark construction. The paper isolates help-seeking judgment by requiring tasks to be nearly unsolvable without clarification but solvable when all missing information is provided.
Critical weaknesses
W1. The benchmark relies on artificially guaranteed clarification necessity. Section 3.2 (p. 4) requires blocker resolutions to be unavailable from the environment and baseline pass rates without ask human() to be at most 5%. While this isolates help-seeking, it largely removes the realistic decision between independent investigation and human escalation, potentially making the benchmark measure blocker identification rather than broader information-gathering judgment.
W2. Ask-F1 does not capture the actual information value of questions. Section 3.5 (p. 5) scores questions based on whether they match registered blockers and penalizes unnecessary queries. However, it does not distinguish questions by information gain, resolution difficulty, or their effect on downstream decisions, even though these differences are central to effective human–agent collaboration.
W3. The human interaction model is overly idealized. Section 3.4 (p. 5) implements ask human() using a frozen Llama-3.3-70B semantic judge that returns an exact registered resolution when a question matches a blocker. This bypasses realistic human responses involving partial information, ambiguity, misunderstanding, or follow-up clarification, limiting the conclusions about practical human-in-the-loop performance.
Questions for authors
Q1. How would agent behavior change if blockers could be resolved through either additional autonomous investigation or human clarification, with different costs for each option?
Q2. How robust are the benchmark results and RL improvements when the human oracle provides incomplete or ambiguous responses rather than exact blocker resolutions?
3 comments
[UPDATED] If blockers could also be resolved through autonomous investigation
We ran an experiment on a small subset of the SWE and SQL tasks where half of the blockers could be resolved via searching throughout the environment. Below are the results.
SWE
| Model | Pass@3 | Precision | Search-and-ask recall | Searchable: asked | Searchable: searched instead | Searchable: missed |
|---|---|---|---|---|---|---|
| Fable 5 | 29.3% | 64.7% | 51.2% | 39.9% | 24.3% | 35.8% |
| GPT-5.6 Sol | 6.8% | 70.7% | 18.5% | 6.2% | 22.3% | 71.5% |
| Gemini 3.1 Pro | 14.9% | 63.3% | 35.7% | 28.9% | 13.6% | 57.5% |
| GLM 5.3 | 25.0% | 34.4% | 41.6% | 27.9% | 24.6% | 47.5% |
SQL
| Model | Pass@3 | Precision | Search-and-ask recall | Searchable: asked | Searchable: searched instead | Searchable: missed |
|---|---|---|---|---|---|---|
| Fable 5 | 69.4% | 75.7% | 82.6% | 75.5% | 5.3% | 19.2% |
| GPT-5.6 Sol | 53.5% | 66.0% | 55.3% | 46.1% | 9.5% | 44.4% |
| Gemini 3.1 Pro | 71.1% | 71.4% | 82.0% | 77.3% | 2.3% | 20.4% |
| GLM 5.3 | 73.0% | 53.3% | 82.6% | 75.2% | 7.4% | 17.4% |
Here, search-and-ask recall measures blockers resolved by either searching or asking, over total blockers. On SQL, agents mostly keep asking rather than searching: they ask about 46–77% of searchable blockers and find only 2–10% by search, so search-and-ask recall stays close to what asking alone gives. On SWE, agents ask far less (6–40% of searchable blockers), and searching recovers only 14–25%, leaving 36–72% unresolved, so search-and-ask recall is low (19–51%). In both cases, having the answer within reach doesn't make agents more self-sufficient. The choice to ask rather than search matters little for task success; the core limitation remains recognizing when there is an underspecification to resolve.
Thank you for the additional experiments on searchable blockers and imperfect oracle responses. These results meaningfully address my concerns about the evaluation setting. But the cost trade-off between searching and asking, and the robustness of RL improvements under imperfect responses are still unclear. I will maintain my rating.
Thank you for the feedback. We respond to your questions below.
Blocker identification vs. information-gathering judgment
We agree that the blocker design choices narrow the behavioral scope of the benchmark. However, our intention was to focus on the help-seeking behaviors. Namely, given critical blockers whose answers cannot be found anywhere but are absolutely necessary to task completion, can the agent detect this knowledge gap and escalate to a human when needed? We wanted to focus on this ability to exercise judgment in seeking assistance, and we show that even in this strict setting where the only way to gather information or be successful is to ask, agents still struggle. This necessity filter is a deliberate choice: settings that mix inferable and non-inferable blockers do exist (e.g. in LHAW), but in such settings, a failure cannot be attributed to only escalation judgment. HiL-Bench measures this judgment in isolation. Future work can then study how agents balance help-seeking vs. independent information-gathering.
Information value of questions
Thank you for the critique. Because we wanted to ensure reproducibility of
ask_human()answers and simplify the quantification of blocker recall, we chose to cleanly map relevant questions to the resulting blocker. The blocker criticality criterion means that every registered blocker is necessary for task success, so each one has high and comparable value by construction, and a question outside the registry has no task value because no other information is needed. We have noted your point in our limitations discussion.Regarding questions’ effect on downstream decisions, that is a per-agent characteristic; some agents struggle with asking at all but can integrate the resolution into their work, while others struggle with taking action after receiving the resolution, whether to incorporate it into their work or to ask follow-up questions (such as Claude Fable 5, as discussed in Appendix G). Finally, while we didn’t track resolution difficulty, we agree it will be an interesting analysis to include in the final version of how question quality changes based on how difficult the blocker is to uncover.
Human interaction model
We agree that our
ask_human()tool is straightforward. However, as discussed in section 5, this was a deliberate design choice. We wanted to have a reproducible simulator that responds perfectly helpful with the exact necessary information in order to prevent unhelpful human responses from confounding our agent evaluations. We wanted to focus on the agent’s help-seeking skills itself in a crisp, ideal setting. We definitely agree that an exciting direction of future work is to look at selective escalation with imperfect human responses.[UPDATED] If the human oracle provides vague responses
We ran an experiment on a small subset of the SWE and SQL tasks where half of the blockers received ambiguous resolutions upon the first match, with the opportunity to receive the true resolution upon subsequent matches. Below are the results.
SWE
SQL
Agents often accept incomplete answers. Many vague or partial answers are never followed up, and the rate is worse on SWE than SQL. When agents do notice an answer is incomplete, they usually follow up right away in the very next question (immediate follow-up rate), except GLM 5.3 on SWE at 58%. We also found models follow up in different styles; for example, GPT 5.6 Sol asks the least as we’ve seen before, and GLM 5.3 struggles with asking precise follow-up questions.