- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies closed-set candidate disambiguation, where a task-oriented agent must ask one clarifying question to tell apart the records returned by a tool call. It contributes an information-gain-based regret metric that is computed exactly from the known candidate table, plus a benchmark built from real SGD, MultiWOZ, τ-bench and Olist data. Using this benchmark, column-editing counterfactuals and repair experiments on nine 7–32B open-weight instruction-tuned models, the authors show that the models often fall into an "attribute trap": they ask about a salient but less informative attribute, and this behavior is not reliably fixed by giving distinct-value counts or by stating the goal.
Critical strengths
The paper defines a well-founded, exactly computable regret metric based on expected information gain, carefully controls for confounds (information ties, near-identifier attributes, unresolvable candidate sets), and releases fully reproducible code, data and raw model outputs (we were able to rerun the pipeline with little effort), which makes the benchmark rigorous and easy to extend even though its conclusions are limited by the choice of models.
Critical weaknesses
-
The models in the main experiments (Qwen2.5, Llama-3.1, Gemma-2, Mistral-7B, GLM-4, Yi-1.5, DeepSeek-LLM-7B, and GPT-4o-2024-08-06 in the appendix) were all released in 2023–2024. They are clearly outdated and do not represent the capabilities of current LLMs, so it is unclear how well the conclusions generalize.
-
The paper does not evaluate any models with thinking/reasoning capabilities (e.g., OpenAI GPT6.1, DeepSeek-V4.1, or Qwen3.8-series models in thinking mode), which are exactly the models most likely to avoid the reported "attribute trap".
-
I reproduced the main experiment using the authors' released code and data, and additionally evaluated Qwen3.8-27B. Setup: I drew a stratified subset of 100 tasks from the 238-task final benchmark, spread evenly across data sources. The prompt, answer parser, and sampling settings were identical to the paper (k = 5 draws per task, temperature 0.3, single-turn zero-shot), with thinking mode enabled. Results:
Model Mean regret (bit) ↓ Regret-0 share ↑ Tasks with any suboptimal draw ↓ Qwen2.5-32B (best model in the paper) 0.028 95.3% 5.0% Qwen3.8-27B (thinking) 0.000 100.0% 0.0% With thinking enabled, the model no longer falls into the attribute trap: all 500 draws select the optimal attribute. This suggests the reported phenomenon is largely confined to older, non-reasoning models and does not reflect the capabilities of current LLMs.
Questions for authors
Can the authors show that this benchmark remains meaningful for current mainstream LLMs? For example, does the attribute trap still appear in recent reasoning models (e.g., GPT6.1, DeepSeek-v4.1, Qwen3.8 in thinking mode)? In my reproduction, Qwen3.8-27B with thinking enabled achieved zero regret on all 500 draws.
4 comments
Thank you for the detailed response and the additional experiments. I agree that cost and latency are real considerations when deploying such agents, and I appreciate the new results in Section 4.6.
My remaining concern is about what the benchmark measures. A benchmark should reflect the difficulty of the real task. In its current form (N ≤ 8 candidates), the benchmark is close to saturated for current models once reasoning is enabled (both my own run and yours show near-perfect choices), so I think its value as a measure of present-day capability is limited. Its discriminative power remains mainly for non-reasoning models, which is not what the main claims are about.
The larger-table results are more informative about real difficulty. I would encourage the authors to make them the core of the benchmark, with real retrieval results and confidence intervals, and to report results with thinking off for the newer models. With such a revision the contribution would be considerably stronger. However, I believe this is a substantial change to the benchmark and the paper's framing, and it is difficult to complete within the short rebuttal period.
Thanks so much for the quick and thoughtful follow-up! We really appreciate your feedback on the benchmark framing and cost/latency trade-offs.
To clarify what ATTRTRAP is fundamentally measuring: our core goal is to surface the pretraining prior biases and what models have really learnt that different models bring to table reading—for instance, as shown in Figure 2, models often favor specific attributes learned during pretraining rather than computing actual information gain from the current table.
The insights themselves would provide great guidelines on model selection in general for a customer service chatbot or at least acknowledge dev teams on the limitations of these models under this common scenario. These models, some just read the attribute names, some others may take a look at the record values and the rest actually do the math and thinking.
A couple of points on why the current benchmark and our scaling additions reflect real-world difficulty:
- Grounded in real task distributions: The candidate sets aren't artificially easy; they are drawn directly from real tool-call retrieval logs in established customer service and sales benchmarks (SGD, MultiWOZ, -bench, Olist), reflecting what agents actually encounter day-to-day.
- Degradation starts early & stress-testing confirms it: Even on these real distributions, Table 1 shows that as soon as the number of records exceeds 2 (), question quality already begins to degrade across non-reasoning setups. To stress-test how this scales when histories grow larger, we also put together the 8-to-64 candidate sets—where even strong reasoning models start to break down (e.g., Qwen3.8-27B dropping to 73.5% and Haiku to 35.5%).
- Evaluating thinking-off: We completely agree on reporting thinking-off baselines for the newer models on the larger tables—we've included those numbers in our updated Table 1 as well so teams can see the exact gap between raw prior bias and explicit reasoning.
While re-framing the entire paper around large tables isn't feasible in the short rebuttal window, we will make sure to highlight both the pretraining prior insights (Figure 2) and the large-table stress-test results much more prominently in the final revision(uploaded). Thanks again for helping us sharpen the paper!
Thank you for the clarification and the thinking-off numbers.
Since this is a Datasets and Benchmarks submission, my main question is whether the benchmark can tell models apart, and I do not yet see evidence that it does for current models. By your own Table 1, every configuration is almost always optimal on the two-record tables, the reasoning models are near ceiling on the rest, and a one-line rule is already optimal on every table. Evidence of discriminative power is needed, for example rank stability across samples and a harder subset on which the rule and the best models differ.
The larger-table results are more informative, but please separate real retrieval results from composed sets. As I read Appendix C, the 8–64 record sets are put together from different searches or customers and "are not single retrieval results", and the MultiWOZ searches are enumerated over the database. So "real tool-call retrieval logs" is not accurate. On the real MultiWOZ searches the reasoning models are close to the rule, and the largest losses appear on the composed sets, where the best attribute is often a price or address a caller may not know.
Without such evidence, I do not see the current version's value as a benchmark, and I keep my assessment.
Response to Reviewer 83Xw
Thanks so much for the close reading and for reproducing our experiment! I truly believe this is a very meaningful and practical finding for frontier labs and industry that build task-oriented agents!
In real-world customer support, agents routinely query databases that return multiple records, making a precise clarifying question essential. Because this happens before any task completion, a poor question leads to a silent, hard-to-debug failure where the conversation just moves on. Reasoning mode is a very good point but I also want to discuss the trade-off in real world settings where latest model may not be accessibile or researchers and engineers just don't realize reading tables itself could be a problem of their choice of any model.
To address your points regarding outdated models and reasoning models (Weaknesses 1 & 2), we’ve run extensive new evaluations (now in Section 4.6 and Table 1):
Many engineering teams don't even realize how suboptimal their clarification turns are. Knowing these trade-offs helps teams plan cost, latency, and decide when a simple code rule should make the choice instead.