- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents ELICITBENCH to measure the interactive ability of agents.
Critical strengths
- The problem is real. Prior benchmarks (ClarifyBench, PrefDisco, τ-bench) test missing info whose identity is fixed before the episode starts.
- The evaluation is diagnostic, not just a leaderboard. Scoring every list against the oracle with a fixed denominator lets the paper decompose failures into never-found / never-asked / held-but-misused
Critical weaknesses
- The “most valuable question” metric rewards luck, not judgment. Value is computed from the user’s true hidden answer, which the agent can’t see at ask time. A rational agent maximizes expected value, not realized value. So an agent that asks the sensible question but gets an answer that happens not to change the ranking is scored as “wrong.” The 25.9% vs 18.2% headline doesn’t tell me much about decision quality until this is addressed.
- Your own numbers suggest asking is net-negative on this benchmark, which undercuts the premise. A never-ask policy scores 0.684, above most ASK cells, and random-order asking makes things worse (82.4% of preference reveals pull an infeasible item into the top 3). If not asking is a strong baseline, I can’t tell whether models are bad at elicitation or whether the task is built so that elicitation rarely pays. The paper needs to confront this directly.
- The core novelty claim is asserted, not tested. You say the right question “cannot be specified in the system prompt,” but 2-1-1 eligibility fields are highly repetitive (ID, income docs, language, cost, walk-in). The obvious ablation—a system prompt with a domain checklist—is missing. The “ask more” intervention isn’t a substitute. If a checklist closes most of the gap, C1 doesn’t hold. You also note the task “does not need a lot of exploration,” which sits awkwardly with “only exploration enables elicitation.”
- The gap decomposition (Eq. 1) is order-dependent. You relax three information boundaries sequentially, so the attribution depends on the order you relax them in (the usual Shapley-style issue). Since this is the basis for C2, please either justify the chosen order or show the attribution is stable across orderings.
Questions for authors
Please see above.