acceptodds
Withdrawn from ICLR 2027

ELICITBench: Measuring Elicitation Capabilities of Interactive Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Language Model (LLM) agents deployed in interactive settings often receive user requests to make recommendations that satisfy certain requirements. An agent can satisfy every stated requirement and still return something the user is ineligible for, because the fact that decides eligibility was never elicited. However, which question is worth asking cannot always be specified in the system prompt. Such tasks require LLM agents to be good at _eliciting_ the right requirements from the user. We study this _hidden-state elicitation_ problem with ELICITBench, in which a user withholds part of their constraints and preferences, and the agent explores a catalog of services, asks questions grounded in what it finds, and submits a ranked shortlist. Existing benchmarks score interaction with missing information that is acquirable through a deterministic interaction where questions can be inferred from the system instructions and tool arguments; here both the missing information and the path to acquiring it are created by exploration and elicitation. This task allows us to diagnose agent failure modes in interactive settings, attributing failures to a) not acquiring _enough_ information through search, b) asking wrong or sub-optimal questions, or c) asking fewer questions than the agent should have. Given perfect information, frontier models are largely capable of providing satisfactory replies, but most models drop by 40% when asked to elicit the right requirements. Furthermore, when we investigate how different deployment conditions (e.g., code harness, test-time compute, self-reflection) affect elicitation capabilities, we find they do not substantially help elicitation.

Primary area:
datasets and benchmarks
Keywords:
interactive agents, elicitation in agents, LLM, tool-calling agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Xw3JOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ELICITBENCH to measure the interactive ability of agents.

Critical strengths
  • The problem is real. Prior benchmarks (ClarifyBench, PrefDisco, τ-bench) test missing info whose identity is fixed before the episode starts.
  • The evaluation is diagnostic, not just a leaderboard. Scoring every list against the oracle with a fixed denominator lets the paper decompose failures into never-found / never-asked / held-but-misused
Critical weaknesses
  • The “most valuable question” metric rewards luck, not judgment. Value is computed from the user’s true hidden answer, which the agent can’t see at ask time. A rational agent maximizes expected value, not realized value. So an agent that asks the sensible question but gets an answer that happens not to change the ranking is scored as “wrong.” The 25.9% vs 18.2% headline doesn’t tell me much about decision quality until this is addressed.
  • Your own numbers suggest asking is net-negative on this benchmark, which undercuts the premise. A never-ask policy scores 0.684, above most ASK cells, and random-order asking makes things worse (82.4% of preference reveals pull an infeasible item into the top 3). If not asking is a strong baseline, I can’t tell whether models are bad at elicitation or whether the task is built so that elicitation rarely pays. The paper needs to confront this directly.
  • The core novelty claim is asserted, not tested. You say the right question “cannot be specified in the system prompt,” but 2-1-1 eligibility fields are highly repetitive (ID, income docs, language, cost, walk-in). The obvious ablation—a system prompt with a domain checklist—is missing. The “ask more” intervention isn’t a substitute. If a checklist closes most of the gap, C1 doesn’t hold. You also note the task “does not need a lot of exploration,” which sits awkwardly with “only exploration enables elicitation.”
  • The gap decomposition (Eq. 1) is order-dependent. You relax three information boundaries sequentially, so the attribution depends on the order you relax them in (the usual Shapley-style issue). Since this is the basis for C2, please either justify the chosen order or show the attribution is stable across orderings.
Questions for authors

Please see above.

Reviewer_XtZFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies hidden-state elicitation in interactive LLM agents, where users initially withhold decision-critical constraints and preferences, requiring the agent to explore a service catalog, ask clarifying questions, and recommend a ranked shortlist. The authors introduce ELICITBench, a benchmark modeled after 2-1-1 referral workflows across four domains, along with some metrics. Evaluating seven models, the authors find that while models perform near the upper bound under complete information (FULL), performance drops by roughly 40% when elicitation is required (ASK), and interventions such as reasoning effort, a code harness, and self-reflection do not substantially improve elicitation.

Critical strengths
  1. Timely and important problem. Proactive information elicitation under hidden eligibility constraints is a critical yet often overlooked capability for interactive agents. Separating catalog exploration from user querying, as well as hard eligibility constraints from soft preferences, captures a realistic failure mode in recommendation workflows.
  2. Comprehensive analysis of deployment factors. The paper evaluates how practical levers—reasoning effort, a code-execution harness, self-reflection interventions, and user repair feedback—affect interactive behavior, and proposes a useful decomposition of the ranking gap into coverage, information, and policy errors.
Critical weaknesses
  1. Unverified data quality, contradictory claims on data grounding, and unexamined benchmark stability.

    • Contradiction on real-world data and lack of quality checks: Section 3.1 (lines 192–193) claims that the catalog is seeded with real-world data from Michigan 2-1-1. However, both the Ethics Statement and the provided code show that all catalogs and profiles are 100% synthetically generated from rule-based templates, with no real 2-1-1 records used. Furthermore, no experiments or evaluations are provided to verify the quality of the generated data.
    • Simulator stability: The paper states that information disclosure is governed by a decision tree, yet there is no validation of whether this decision tree can reliably handle diverse agent queries, nor any verification of the simulator's stability.
  2. Missing main benchmark table and multi-run variance.

    • As a benchmark paper, Section 4 presents only analytical charts (Figures 3–6) and lacks a main results table in the body of the paper. Readers cannot directly compare exact model scores across FULL, ASK, and REPAIR, or inspect side-by-side numbers for ONDCG@3, HVR, HIC, Critical-Fact Recall, and δC,δI,δΦ\delta_C, \delta_I, \delta_\Phi.
    • Despite using stochastic decoding (e.g., T=1.0T=1.0 for open-weight models and provider defaults for hosted models), results are reported from a single run without mean and standard deviation across multiple independent runs.
  3. Unclear writing, undefined notation, and formatting/citation issues.

    • Undefined mathematical notation: Several formulas are introduced without defining their symbols, such as u=(HK,HL,PK,PL)u = (H^K, H^L, P^K, P^L) and ∑pw(p)sp(r)\sum_p w(p)s_p(r) in lines 196–200, and gu(πi)g_u(\pi_i) in Table 2. Although line 241 refers to Appendix B for full definitions and tie handling, Appendix B omits both tie-handling rules and the formal definitions of δC,δI,δΦ\delta_C, \delta_I, \delta_\Phi in Equation (1).
    • Confusing overview figure: Figure 1 is a low-resolution raster image. Figure 1(a) has confusing arrow directions (e.g., it is unclear why Step 2 has two arrows pointing to Step 3), and Figure 1(b) is an uncalibrated conceptual sketch without numerical axes.
    • Broken links and editing errors: Because the appendix was submitted as a separate PDF, all internal hyperlinks to Appendices B–F in the main paper are broken. There are also frequent citation formatting issues (such as the cross-line citation at lines 161–162 and several improperly formatted citations nearby).
Questions for authors
  1. Real-world data vs. synthetic templates: Could you clarify how real Michigan 2-1-1 data was used, given that the provided code generates all catalogs and profiles entirely from synthetic templates? Please also provide evidence for the reliability of the dataset and the stability of the simulator (e.g., through human inspection or verification).
  2. Main results table and formal definitions: Could you provide a complete main table comparing all models across FULL, ASK, and REPAIR on all key metrics, and supply the missing formal definitions for HK,HL,PK,PL,w(p),sp(r),gu(r)H^K, H^L, P^K, P^L, w(p), s_p(r), g_u(r), tie handling, and δC,δI,δΦ\delta_C, \delta_I, \delta_\Phi?
Reviewer_7e7TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ELICITBench, a benchmark for evaluating whether LLM agents can uncover hidden user constraints and preferences through exploration and targeted questioning. It diagnoses failures in information acquisition, question selection, and deciding how many questions to ask. Experiments show that most models’ performance drops by around 40% when elicitation is required, while strategies such as additional test-time compute and self-reflection provide limited improvement.

Critical strengths

The paper addresses an important challenge in interactive LLM agents: eliciting hidden user constraints and preferences needed for suitable recommendations. ELICITBench provides a structured evaluation framework that distinguishes failures in information acquisition, question selection, and questioning effort. The experiments reveal a substantial gap between models’ ability to recommend with complete information and their ability to elicit the necessary requirements, while examining the limited effectiveness of strategies such as test-time compute and self-reflection.

Critical weaknesses
  1. The paper identifies hidden-state elicitation as an important capability of interactive agents. However, the distinction from existing work on active information acquisition, conversational search, and interactive decision-making remains insufficiently established. Can the authors demonstrate that ELICITBENCH measures a capability that existing benchmarks fail to capture, rather than primarily introducing a new task formulation? In particular, how robust are the findings to simple elicitation heuristics, such as prioritising hard eligibility constraints over soft preferences?

  2. The best model chooses the most useful question in 25.9% of cases, compared with 18.2% for random selection. However, different questions may be equally useful, and the value of a question may depend on what happens next. How do the authors show that this evaluation method is fair and reliable? How do they ensure that their analysis can correctly identify whether an agent fails because it does not find the right information, does not ask the right questions, or does not use the information correctly? Testing different ways to measure question usefulness and interaction costs would make the conclusions more convincing.

Questions for authors

See Weaknesses for details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.