acceptodds
Under review as a conference paper at ICLR 2027

HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?

Abstract

Frontier coding agents solve complex tasks when given complete context but struggle when specifications are incomplete or ambiguous. The bottleneck is not raw capability, but _judgment_: knowing when to act autonomously and when to ask for help. Current benchmarks are blind to this failure mode: they supply unambiguous detailed instructions and reward solely execution correctness, so an agent that makes a lucky guess for a missing requirement will score identically to one that would have asked for certainty. We present HiL-Bench (Human-in-Loop Benchmark) to measure this _selective escalation_ skill. Each task contains human-validated blockers (missing information, ambiguous requests, contradictory information) that surface only through progressive exploration, not upfront inspection. Our core metric, Ask-F1, the harmonic mean of question precision and blocker recall, captures the tension between over-asking and silent guessing. Evaluation across SWE and text-to-SQL domains reveals a large universal judgment gap: no frontier model recovers more than a fraction of its full-information performance in settings where it needs to decide whether to ask. Failure analysis identifies key patterns: overconfident wrong beliefs with no information-gathering; inability to use human or tool feedback in problem-solving; and broad, imprecise escalation. These consistent patterns confirm poor help-seeking is a model-level, not task-specific, flaw. Fortunately, RL training on a shaped Ask-F1 reward shows judgment is trainable: a 32B model improves on both help-seeking quality and task pass rate, with gains that can transfer across SWE, SQL, and blocker types.

Primary area:
datasets and benchmarks
Keywords:
llm, agents, ambiguity, ask

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gcQ2OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HiL-Bench, a benchmark evaluating whether agents can recognize when they need human clarification rather than proceeding with unsupported assumptions. It constructs 300 software-engineering and text-to-SQL tasks containing 1,131 human-validated blockers, including missing information, ambiguous requirements, and contradictory instructions.The authors further show that reinforcement learning with shaped Ask-F1 rewards improves clarification behavior and downstream task completion, with some cross-domain transfer.

Critical strengths
  • Well-motivated and carefully controlled benchmark construction. The paper isolates help-seeking judgment by requiring tasks to be nearly unsolvable without clarification but solvable when all missing information is provided.
Critical weaknesses

W1. The benchmark relies on artificially guaranteed clarification necessity. Section 3.2 (p. 4) requires blocker resolutions to be unavailable from the environment and baseline pass rates without ask human() to be at most 5%. While this isolates help-seeking, it largely removes the realistic decision between independent investigation and human escalation, potentially making the benchmark measure blocker identification rather than broader information-gathering judgment.

W2. Ask-F1 does not capture the actual information value of questions. Section 3.5 (p. 5) scores questions based on whether they match registered blockers and penalizes unnecessary queries. However, it does not distinguish questions by information gain, resolution difficulty, or their effect on downstream decisions, even though these differences are central to effective human–agent collaboration.

W3. The human interaction model is overly idealized. Section 3.4 (p. 5) implements ask human() using a frozen Llama-3.3-70B semantic judge that returns an exact registered resolution when a question matches a blocker. This bypasses realistic human responses involving partial information, ambiguity, misunderstanding, or follow-up clarification, limiting the conclusions about practical human-in-the-loop performance.

Questions for authors

Q1. How would agent behavior change if blockers could be resolved through either additional autonomous investigation or human clarification, with different costs for each option?

Q2. How robust are the benchmark results and RL improvements when the human oracle provides incomplete or ambiguous responses rather than exact blocker resolutions?

3 comments

Thank you for the feedback. We respond to your questions below.

Blocker identification vs. information-gathering judgment

We agree that the blocker design choices narrow the behavioral scope of the benchmark. However, our intention was to focus on the help-seeking behaviors. Namely, given critical blockers whose answers cannot be found anywhere but are absolutely necessary to task completion, can the agent detect this knowledge gap and escalate to a human when needed? We wanted to focus on this ability to exercise judgment in seeking assistance, and we show that even in this strict setting where the only way to gather information or be successful is to ask, agents still struggle. This necessity filter is a deliberate choice: settings that mix inferable and non-inferable blockers do exist (e.g. in LHAW), but in such settings, a failure cannot be attributed to only escalation judgment. HiL-Bench measures this judgment in isolation. Future work can then study how agents balance help-seeking vs. independent information-gathering.

Information value of questions

Thank you for the critique. Because we wanted to ensure reproducibility of ask_human() answers and simplify the quantification of blocker recall, we chose to cleanly map relevant questions to the resulting blocker. The blocker criticality criterion means that every registered blocker is necessary for task success, so each one has high and comparable value by construction, and a question outside the registry has no task value because no other information is needed. We have noted your point in our limitations discussion.

Regarding questions’ effect on downstream decisions, that is a per-agent characteristic; some agents struggle with asking at all but can integrate the resolution into their work, while others struggle with taking action after receiving the resolution, whether to incorporate it into their work or to ask follow-up questions (such as Claude Fable 5, as discussed in Appendix G). Finally, while we didn’t track resolution difficulty, we agree it will be an interesting analysis to include in the final version of how question quality changes based on how difficult the blocker is to uncover.

Human interaction model

We agree that our ask_human() tool is straightforward. However, as discussed in section 5, this was a deliberate design choice. We wanted to have a reproducible simulator that responds perfectly helpful with the exact necessary information in order to prevent unhelpful human responses from confounding our agent evaluations. We wanted to focus on the agent’s help-seeking skills itself in a crisp, ideal setting. We definitely agree that an exciting direction of future work is to look at selective escalation with imperfect human responses.

[UPDATED] If the human oracle provides vague responses

We ran an experiment on a small subset of the SWE and SQL tasks where half of the blockers received ambiguous resolutions upon the first match, with the opportunity to receive the true resolution upon subsequent matches. Below are the results.

SWE

ModelPass@3PrecisionRecallAsk-F1Follow-Up RateImmediate Follow-Up Rate
Fable 537.0%36.4%62.8%46.1%61.7%77%
GPT-5.6 Sol21.3%48.0%41.3%44.4%59.9%84%
Gemini 3.1 Pro16.7%46.3%46.7%46.5%56.4%74%
GLM 5.330.9%18.9%57.2%28.4%74.1%58%

SQL

ModelPass@3PrecisionRecallAsk-F1Follow-Up RateImmediate Follow-Up Rate
Fable 577.0%42.6%89.5%57.7%76.9%84%
GPT-5.6 Sol60.0%49.2%82.5%61.6%68.9%88%
Gemini 3.1 Pro58.0%48.1%86.4%61.8%77.1%80%
GLM 5.371.4%31.7%89.7%46.8%78.5%75%

Agents often accept incomplete answers. Many vague or partial answers are never followed up, and the rate is worse on SWE than SQL. When agents do notice an answer is incomplete, they usually follow up right away in the very next question (immediate follow-up rate), except GLM 5.3 on SWE at 58%. We also found models follow up in different styles; for example, GPT 5.6 Sol asks the least as we’ve seen before, and GLM 5.3 struggles with asking precise follow-up questions.

[UPDATED] If blockers could also be resolved through autonomous investigation

We ran an experiment on a small subset of the SWE and SQL tasks where half of the blockers could be resolved via searching throughout the environment. Below are the results.

SWE

ModelPass@3PrecisionSearch-and-ask recallSearchable: askedSearchable: searched insteadSearchable: missed
Fable 529.3%64.7%51.2%39.9%24.3%35.8%
GPT-5.6 Sol6.8%70.7%18.5%6.2%22.3%71.5%
Gemini 3.1 Pro14.9%63.3%35.7%28.9%13.6%57.5%
GLM 5.325.0%34.4%41.6%27.9%24.6%47.5%

SQL

ModelPass@3PrecisionSearch-and-ask recallSearchable: askedSearchable: searched insteadSearchable: missed
Fable 569.4%75.7%82.6%75.5%5.3%19.2%
GPT-5.6 Sol53.5%66.0%55.3%46.1%9.5%44.4%
Gemini 3.1 Pro71.1%71.4%82.0%77.3%2.3%20.4%
GLM 5.373.0%53.3%82.6%75.2%7.4%17.4%

Here, search-and-ask recall measures blockers resolved by either searching or asking, over total blockers. On SQL, agents mostly keep asking rather than searching: they ask about 46–77% of searchable blockers and find only 2–10% by search, so search-and-ask recall stays close to what asking alone gives. On SWE, agents ask far less (6–40% of searchable blockers), and searching recovers only 14–25%, leaving 36–72% unresolved, so search-and-ask recall is low (19–51%). In both cases, having the answer within reach doesn't make agents more self-sufficient. The choice to ask rather than search matters little for task success; the core limitation remains recognizing when there is an underspecification to resolve.

Thank you for the additional experiments on searchable blockers and imperfect oracle responses. These results meaningfully address my concerns about the evaluation setting. But the cost trade-off between searching and asking, and the robustness of RL improvements under imperfect responses are still unclear. I will maintain my rating.

Reviewer_6Uk6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates human intervention in the agent work loops, testing the selective escalation skill of the agent, i.e. ask human for additional information needed for the task. For evaluating this skill, the paper proposes Human-in-Loop benchmark with blockers that are generated intentially to prevent the agent's work without human's help. Experiments are ran against popular agentic tools and metrics including ASK-F1 is calculated to measure agent's ability to accurately escalate to human. Additional, the paper shows the possibility of improving escalation ability via RLVR.

Critical strengths
  1. The paper focuses on an very important problem: the ability of agent to properly ask for human intervention. The benchmark provides a solid measurement of the ability and the experiments provide new insights for the community.
Critical weaknesses
  1. I have some concerns regarding the novelty of this paper: there seems to already be some papers working on constructing under-specified tasks to test agent's ability of escalation. (For example, [1]) The discussion in related works provides some discussion on the differences, but I do not fully understand why these differences are really important.

[1] LHAW: Controllable Underspecification for Long-Horizon Tasks, arxiv, 2602.10525

Questions for authors
  1. The RLVR example is shown only on Qwen3 32B model. If you can provide 1-2 more models (ideally from other families) the conclusion will be more convincing.
  2. Why do you think upfront visible ambiguity is a problem with prior benchmark? Does it make the underspecification tasks less challenging for modern agents?
  3. By embedding multiple blockers on each task, do you observe the metrics drop when increasing the number of blockers?
2 comments

Thank you for the feedback. We respond to your questions below.

Related work

There are several key differences between the LHAW paper and ours. We highlight some of the main ones here:

  • Construction: LHAW was created via a synthetic pipeline that removed information from the task prompt covering goals, constraints, inputs, and context; blockers only existed in the task prompt upfront. On the other hand, HiL-Bench blockers can appear in multiple different locations in the task environment and can surface as the agent progresses in its execution, not just at the beginning of the trajectory.
  • Blocker criticality: LHAW tasks mix blockers of different criticality levels, from those that must be resolved via asking to those the agent can infer from the environment. HiL-Bench tasks contain only critical blockers that cannot be resolved without asking and whose resolutions are absolutely necessary to the completion of the task.
  • Blocker types: LHAW blockers do not cover ambiguity from conflicting context or multiple valid interpretations due to their removal-based construction. HiL-Bench does cover these types of blockers.
  • Training: Post-training is listed as future work in LHAW, while the HiL-Bench paper shows how the judgment gap can be improved with RL.

We strongly believe in the importance of progressive discovery. Real-world tasks are very messy and can require extensive exploration, probing of the task, and even starting the execution or problem solving process, before obstacles and ambiguities can arise. Initial hypotheses and strategies can easily be undermined as new information is revealed or gaps are encountered. We wanted to capture the difficulty of such realistic settings in our benchmark. Additionally, modern agents are adept at understanding context and closing gaps that are encountered upfront; the real challenge is gaps that are hidden inside the task that can only be unlocked after trying to tackle it.

RLVR

We were able to additionally train Gemma 4 31B on part of the SQL tasks and evaluated it on the SWE tasks and the rest of the SQL tasks. The difference between its base and RL’ed performance is below:

Gemma 4 31BBaselineRLVR’d on SQL90% CI
SQL Pass@30.3040.398[+0.018, +0.181]
SQL Ask-F10.6180.682[+0.033, +0.096]
SWE Pass@30.010.033[+0.005, +0.036]
SWE Ask-F10.1600.202[−0.002, +0.089]

Performance by number of blockers

We measured the performance of the four models in our main paper based on how many blockers were present in the task. The data shows that pass rate changes but has no consistent relationship with the number of blockers in either task type. On the other hand, precision, recall, and ask-F1 stay relatively fixed given changing blocker counts; these metrics act as domain-specific model signatures, reflecting each model family’s asking tendencies and failure patterns.

SWE

ModelBlockersPass@3PrecisionRecallAsk-F1
Fable 5347.5%39.3%60.3%47.6%
452.0%43.6%58.3%49.9%
542.9%42.0%63.8%50.7%
GPT-5.6 Sol334.4%52.8%46.4%49.4%
428.0%56.6%44.3%49.7%
521.4%62.3%47.1%53.7%
Gemini 3.1 Pro311.5%53.6%40.4%46.1%
412.0%65.1%36.7%46.9%
57.1%49.7%37.6%42.8%
GLM 5.3344.3%26.7%57.4%36.4%
436.0%27.0%57.3%36.8%
535.7%24.9%57.6%34.8%

SQL

ModelBlockersPass@3PrecisionRecallAsk-F1
Fable 5368.6%67.9%71.7%69.8%
465.6%63.4%71.9%67.4%
578.8%64.5%79.2%71.1%
GPT-5.6 Sol337.1%58.7%33.3%42.5%
431.2%60.0%34.4%43.7%
539.4%59.9%33.7%43.2%
Gemini 3.1 Pro360.0%60.5%78.7%68.4%
465.6%56.6%71.6%63.2%
548.5%58.9%67.5%62.9%
GLM 5.3380.0%43.5%92.1%59.1%
465.6%43.9%88.0%58.6%
581.8%46.4%90.1%61.2%

Thank you for the rebuttal and new results. I think my questions are answered and I will maintain my score.

Reviewer_hoYzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper proposes a new benchmark that evaluates if frontier models ask clarifying questions during their workflows for solving code/SQL tasks. The paper argues it is the first work to benchmark the ability of these models to ask timely and necessary clarifying questions to the user to complete workflows.

Critical strengths
  1. The paper creates a challenging benchmark for the frontier models to highlight their failures on real world code/SQL tasks.
  2. It proposes a new way to curate data for SWE/SQL tasks that require human in the loop to help the models complete the tasks, failing which the models show degraded performance compared to their fully specified counterparts.
Critical weaknesses
  1. The paper is well-motivated but is not well-written to the standards of a conference style draft. The paper cites social media website posts/comments that cannot be considered at par with peer-reviewed paper citations as they may biased or personal views.

  2. The paper is very thin on technical details. There is not much elaboration on prompts, benchmark graders, qualitative analysis, or limitations.

  3. The paper references Appendices 6 times in the paper but there is no Appendix provided after the bibliography. This makes it very difficult to evaluate the completeness of the paper.

  4. The human auditing/annotation process is insufficiently explained that raises doubts whether it was throughly performed or the authors themselves checked the dataset, which may be a source of bias in the benchmark.

  5. Missing citations and version information for evaluated models (GPT 5.6 Sol, Claude Fable 5, Gemini 3.1 Pro, GLM 5.3) in setup. Authors are encouraged to provide these to avoid version mismatch for reproducibility.

Questions for authors
  1. The authors specify the tasks were annotated and verified by human auditors but they do not qualify what these auditors checked, what quantitative methods were used to judge the tasks, any agreement scores between these auditors or any rejections recommended by them. It appears this step was just done to decorate the paper details but any information about the process has been skipped.

  2. Section 3.3 PROGRESSIVE DISCOVERY - it is very difficult to understand why and how Claude model was used to check the progressive discovery validation. It is unlikely that a small spec test would tiefl any statistically significant results. Does this countrfactual eval form part of the full benchmark suit?

  3. THE ASK HUMAN() TOOL - what was the input to the LLM behind this tool beyond just question string? If the LLM already had access correct questions, is it possible the evaluation models can hack this toll to ask multiple hedged questions in same call? Did the authors takes measure to avoid the LLM leaking the questions bank to secure the universe of unknowns preemptively?

3 comments

Thank you for the feedback. We respond to your questions below. We also wanted to point out that our appendix was submitted as part of the supplementary material following ICLR’s guidelines. It covers blocker taxonomy (B), task generation pipeline (C), the ask_human() judge (D), dataset statistics (E), additional failure analysis (F, G), RLVR training details (H), and more. During this review period, for convenience, we updated our PDF to include both the main body and the appendix in one PDF.

Citations and version information

We have removed the social media post and comment citations in the introduction. We also added in the citations (section 4) and version information (appendix L) for the evaluated models.

Technical details

We’ve added the system prompt used for the agents and the prompt given to the ask_human() tool judge to the appendix. To be clear, the solving agent never gets access to the blocker registry or example correct questions, nor can it hack around the environment to gain access. The judge receives the agent’s question and only sees the specific task’s description and example trigger questions for each blocker. It never sees the resolution; it outputs a blocker ID or null and the tool server returns the stored resolution verbatim, so no judge-generated text ever reaches the agent. An agent’s message that touches more than one topic is rejected as a whole so that bundling hedged questions into one call earns nothing and counts against precision. Each blocker can be answered at most twice per task.

For SWE and SQL task verifiers: the SWE tasks run hidden tests on the agent’s solution and all tests must pass; the SQL tasks compare the agent’s query’s output with the golden query’s output and the two must match, accounting for ordering differences as appropriate. For ask-related metrics, they are derived from the output of the ask_human() tool and are described in section 3.5. The ask_human() judge itself was calibrated prior to deployment and achieved 96% precision, 77% recall on a set of 100 human-annotated question-blocker pairs when compared against the majority vote, and 100% precision and 86% recall when compared to the unanimous-agreement subset, as described in Appendix D.

We qualitatively discuss the results in sections 4.2, 4.3, 4.4, and appendix section G, and our limitations in section 5. We’re happy to expand on anything and add it to the final version of the paper if you could provide us with what you’d like to see more of.

Human auditing process

Like above, a more thorough description of this process can be found in appendix section C. Tasks were built and audited by a pipeline of more than a dozen trained annotators and auditors, none of whom were authors. Annotation and auditing were separate roles: annotators injected the tasks with blockers, and independent auditors checked each task against the quality criteria described in section 3.2. Every annotator and auditor went through thorough onboarding, training, and testing, and we also dedicated time to reviewing disputes and re-aligning the team whenever new requirements were set or disagreements arose. Every task went through at least 5-6 audit layers and any single failed criterion meant it was sent back for revision or removal. We also used automated checks to enforce the eligibility criteria: at most 5% of tasks pass without the ask_human() tool, and about 90% pass rate when the blockers are provided upfront. Appendix C details this pipeline.

Regarding quantitative methods to judge the tasks, these are described in appendix section C, section 3.1, and the eligibility criteria in section 3.2.

Please let us know if our response here and the appendix address your concerns or if there’s anything else we can clear up.

Progressive discovery experiment

We chose the Claude model for its high capability, and since the experiment was to test whether blockers were detectable from the spec alone, one strong model suffices. We ran it on 150 SQL samples which contained 598 blockers that theoretically could be detected in the experiment. The CI of performance given full environment access is 57% - 64.8%; the CI of performance without the environment is 8.7% - 13.8%; the p-value of McNemar’s test is effectively 0, showing that results are statistically significant. This experiment was merely to show the importance of progressive discovery so we do not provide this evaluation setup as part of the benchmark itself.

Thank you for the feedback. We respond to your questions below. We also wanted to point out that our appendix was submitted as part of the supplementary material following ICLR’s guidelines. It covers blocker taxonomy (B), task generation pipeline (C), the ask_human() judge (D), dataset statistics (E), additional failure analysis (F, G), RLVR training details (H), and more. During this review period, for convenience, we updated our PDF to include both the main body and the appendix in one PDF.

Citations and version information

We have removed the social media post and comment citations in the introduction. We also added in the citations (section 4) and version information (appendix L) for the evaluated models.

Technical details

We’ve added the system prompt used for the agents and the prompt given to the ask_human() tool judge to the appendix. To be clear, the solving agent never gets access to the blocker registry or example correct questions, nor can it hack around the environment to gain access. The judge receives the agent’s question and only sees the specific task’s description and example trigger questions for each blocker. It never sees the resolution; it outputs a blocker ID or null and the tool server returns the stored resolution verbatim, so no judge-generated text ever reaches the agent. An agent’s message that touches more than one topic is rejected as a whole so that bundling hedged questions into one call earns nothing and counts against precision. Each blocker can be answered at most twice per task.

For SWE and SQL task verifiers: the SWE tasks run hidden tests on the agent’s solution and all tests must pass; the SQL tasks compare the agent’s query’s output with the golden query’s output and the two must match, accounting for ordering differences as appropriate. For ask-related metrics, they are derived from the output of the ask_human() tool and are described in section 3.5. The ask_human() judge itself was calibrated prior to deployment and achieved 96% precision, 77% recall on a set of 100 human-annotated question-blocker pairs when compared against the majority vote, and 100% precision and 86% recall when compared to the unanimous-agreement subset, as described in Appendix D.

We qualitatively discuss the results in sections 4.2, 4.3, 4.4, and appendix section G, and our limitations in section 5. We’re happy to expand on anything and add it to the final version of the paper if you could provide us with what you’d like to see more of.

Human auditing process

Like above, a more thorough description of this process can be found in appendix section C. Tasks were built and audited by a pipeline of more than a dozen trained annotators and auditors, none of whom were authors. Annotation and auditing were separate roles: annotators injected the tasks with blockers, and independent auditors checked each task against the quality criteria described in section 3.2. Every annotator and auditor went through thorough onboarding, training, and testing, and we also dedicated time to reviewing disputes and re-aligning the team whenever new requirements were set or disagreements arose. Every task went through at least 5-6 audit layers and any single failed criterion meant it was sent back for revision or removal. We also used automated checks to enforce the eligibility criteria: at most 5% of tasks pass without the ask_human() tool, and about 90% pass rate when the blockers are provided upfront. Appendix C details this pipeline.

Regarding quantitative methods to judge the tasks, these are described in appendix section C, section 3.1, and the eligibility criteria in section 3.2.

Please let us know if our response here and the appendix address your concerns or if there’s anything else we can clear up.

Progressive discovery experiment

We chose the Claude model for its high capability, and since the experiment was to test whether blockers were detectable from the spec alone, one strong model suffices. We ran it on 150 SQL samples which contained 598 blockers that theoretically could be detected in the experiment. The CI of performance given full environment access is 57% - 64.8%; the CI of performance without the environment is 8.7% - 13.8%; the p-value of McNemar’s test is effectively 0, showing that results are statistically significant. This experiment was merely to show the importance of progressive discovery so we do not provide this evaluation setup as part of the benchmark itself.

Dear Reviewer,

Please read the authors’ rebuttal and join the discussion. Kindly clarify whether your main concerns have been addressed and raise any follow-up questions promptly so the authors have time to respond.

Thank you for your time and effort!

Comments on the paper

1 comment

We thank the reviewers for their feedback. They agree the problem is important (6Uk6, gcQ2) and the benchmark construction is carefully controlled (gcQ2). Below we summarize the revisions and new results; our updated PDF reflects all changes.

Revisions

  • Our appendix was submitted in the supplementary material as ICLR guidelines indicate.
  • We’ve added to our limitations in section 5 to cover the information value of questions.
  • We also added the agent and judge prompts (Appendix M) and model version details (Appendix L).

New Results

  • We conducted RLVR on a second model family (Gemma 4 31B), trained on SQL and evaluated on SQL and SWE. It was able to improve on SQL by 9.4pp in pass@3 and 6.4pp in Ask-F1; transferring to SWE, it improved by 2.3pp in pass@3 and 4.2pp in Ask-F1.
  • Performance by number of blockers, stratified by domain: while pass@3 varies with no consistent relationship to blocker count, precision, recall, and ask-F1 remain largely fixed, acting as domain-specific signatures of each model family’s asking behavior.
  • [UPDATED] We reported results on an experiment with an imperfect ask_human() judge on a subset of the data. We found that agents frequently do not follow up once they’ve received an answer, even if it’s vague, and some also struggle with targeted follow-up questions.
  • [UPDATED] We reported results on an experiment with blockers whose resolutions can be searched for. We found that this doesn’t automatically lead to them resolving blockers either. The agents still struggle with recognizing missing or confusing information, independent of the channel used to retrieve that information (ask vs. search).

Key Clarifications

  • The ask_human() judge never sees the blocker resolutions, so it cannot leak them.
  • Our differences from LHAW: human-authored blockers; blockers embedded in the environment; ambiguous and contradictory blockers, not just missing information ones; and difficulty necessity verified across models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.