acceptodds
Under review as a conference paper at ICLR 2027

The Attribute Trap: Systematically Suboptimal Clarifying Questions After Tool-Call Retrieval

Abstract

Do task-oriented dialogue agents ground their clarifying questions in the records they have just retrieved? We study closed-set candidate disambiguation: a task-oriented dialogue agent retrieves multiple candidate records (orders, bookings, search results returned by a database query or an application programming interface (API) call) matching a user's request, and must ask a single clarifying question to identify which one of the candidates is meant. Because the candidate set is known, the expected information gain of every attribute the agent could ask about can be computed exactly, so an agent's question can be scored by its regret against the most informative one. This lets us separate genuine judgment failures from three confounds that distort naive failure rates: candidate sets that no natural attribute can resolve, questions that were in expectation as good a bet as the best one, and near-identifier attributes that are informative whatever the table contains. Our main evidence is interventional. On real candidate sets from the Schema-Guided Dialogue corpus (SGD) and the Olist e-commerce data we edit a single column and leave every other cell unchanged. The edits split nine open-weight instruction-tuned models (7–32B parameters) in two. When the column a model asks about most is only partly duplicated, so that it is still informative but no longer the best, four models, among them the two largest, stop asking it (3–21% of draws); four keep asking on 73–85%; one lies between. When the column is flattened so that it carries no information, six stop, two fall to about the chance rate, and one keeps asking at twice it. So four models do not react to the values in the table; four do, in different degrees. We call asking about an attribute that is not the most informative one an attribute trap. On unedited tables from four dataset families (SGD, the Multi-Domain Wizard-of-Oz corpus (MultiWOZ), the -bench family, Olist) every model falls into it; for the strongest models the loss is small and sits almost entirely on attributes such as an exact price, which are the most informative but which a real user may not know. The effect persists across prompt wordings, a recency-weighted prior and scoring thresholds, and it is not explained by counting errors alone: giving the agent the exact number of distinct values of every attribute does not reliably repair it, and stating the goal makes most models worse.

Primary area:
datasets and benchmarks
Keywords:
Task-oriented dialogue, Closed-set candidate disambiguation, Dialogue agents, Information gain, Clarifying questions

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_83XwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies closed-set candidate disambiguation, where a task-oriented agent must ask one clarifying question to tell apart the records returned by a tool call. It contributes an information-gain-based regret metric that is computed exactly from the known candidate table, plus a benchmark built from real SGD, MultiWOZ, τ-bench and Olist data. Using this benchmark, column-editing counterfactuals and repair experiments on nine 7–32B open-weight instruction-tuned models, the authors show that the models often fall into an "attribute trap": they ask about a salient but less informative attribute, and this behavior is not reliably fixed by giving distinct-value counts or by stating the goal.

Critical strengths

The paper defines a well-founded, exactly computable regret metric based on expected information gain, carefully controls for confounds (information ties, near-identifier attributes, unresolvable candidate sets), and releases fully reproducible code, data and raw model outputs (we were able to rerun the pipeline with little effort), which makes the benchmark rigorous and easy to extend even though its conclusions are limited by the choice of models.

Critical weaknesses
  1. The models in the main experiments (Qwen2.5, Llama-3.1, Gemma-2, Mistral-7B, GLM-4, Yi-1.5, DeepSeek-LLM-7B, and GPT-4o-2024-08-06 in the appendix) were all released in 2023–2024. They are clearly outdated and do not represent the capabilities of current LLMs, so it is unclear how well the conclusions generalize.

  2. The paper does not evaluate any models with thinking/reasoning capabilities (e.g., OpenAI GPT6.1, DeepSeek-V4.1, or Qwen3.8-series models in thinking mode), which are exactly the models most likely to avoid the reported "attribute trap".

  3. I reproduced the main experiment using the authors' released code and data, and additionally evaluated Qwen3.8-27B. Setup: I drew a stratified subset of 100 tasks from the 238-task final benchmark, spread evenly across data sources. The prompt, answer parser, and sampling settings were identical to the paper (k = 5 draws per task, temperature 0.3, single-turn zero-shot), with thinking mode enabled. Results:

    ModelMean regret (bit) ↓Regret-0 share ↑Tasks with any suboptimal draw ↓
    Qwen2.5-32B (best model in the paper)0.02895.3%5.0%
    Qwen3.8-27B (thinking)0.000100.0%0.0%

    With thinking enabled, the model no longer falls into the attribute trap: all 500 draws select the optimal attribute. This suggests the reported phenomenon is largely confined to older, non-reasoning models and does not reflect the capabilities of current LLMs.

Questions for authors

Can the authors show that this benchmark remains meaningful for current mainstream LLMs? For example, does the attribute trap still appear in recent reasoning models (e.g., GPT6.1, DeepSeek-v4.1, Qwen3.8 in thinking mode)? In my reproduction, Qwen3.8-27B with thinking enabled achieved zero regret on all 500 draws.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 83Xw

Thanks so much for the close reading and for reproducing our experiment! I truly believe this is a very meaningful and practical finding for frontier labs and industry that build task-oriented agents!

In real-world customer support, agents routinely query databases that return multiple records, making a precise clarifying question essential. Because this happens before any task completion, a poor question leads to a silent, hard-to-debug failure where the conversation just moves on. Reasoning mode is a very good point but I also want to discuss the trade-off in real world settings where latest model may not be accessibile or researchers and engineers just don't realize reading tables itself could be a problem of their choice of any model.

To address your points regarding outdated models and reasoning models (Weaknesses 1 & 2), we’ve run extensive new evaluations (now in Section 4.6 and Table 1):

  • Evaluation of Newer & Reasoning Models: We tested Qwen3.8-27B and Claude Haiku 5.5 (both with thinking off and on), Claude Sonnet 5.5, and GPT-6 Luna (at high reasoning effort) across all 238 benchmark and 251 second-sample tables, including all edits and count hints.
  • Agreement with Your Reproduction: Our full run matches your findings. With thinking enabled, Qwen3.8-27B selects the highest-EIG attribute in 99.4% of benchmark questions and 98.4% on the second sample. However, no single LLM configuration achieves optimality across every table—whereas the deterministic rule in code does.
  • Cost & Latency Trade-offs: We've added output tokens and latency to Table 1. With thinking enabled, Qwen3.8-27B generates 266 output tokens instead of 35, taking 12.3 instead of 4.0 seconds. In practice, forcing massive reasoning models on this step is often overkill and an unnecessary waste of resources.
  • Scaling to Larger Tables: We evaluated 34 real MultiWOZ searches (9 to 69 records) and synthetic sets ranging from 8 to 64 records. At 64 records, Qwen3.8-27B with thinking asks a highest-EIG attribute in 73.5% of questions, while Haiku with thinking drops to 35.5%.

Many engineering teams don't even realize how suboptimal their clarification turns are. Knowing these trade-offs helps teams plan cost, latency, and decide when a simple code rule should make the choice instead.

Thank you for the detailed response and the additional experiments. I agree that cost and latency are real considerations when deploying such agents, and I appreciate the new results in Section 4.6.

My remaining concern is about what the benchmark measures. A benchmark should reflect the difficulty of the real task. In its current form (N ≤ 8 candidates), the benchmark is close to saturated for current models once reasoning is enabled (both my own run and yours show near-perfect choices), so I think its value as a measure of present-day capability is limited. Its discriminative power remains mainly for non-reasoning models, which is not what the main claims are about.

The larger-table results are more informative about real difficulty. I would encourage the authors to make them the core of the benchmark, with real retrieval results and confidence intervals, and to report results with thinking off for the newer models. With such a revision the contribution would be considerably stronger. However, I believe this is a substantial change to the benchmark and the paper's framing, and it is difficult to complete within the short rebuttal period.

Thanks so much for the quick and thoughtful follow-up! We really appreciate your feedback on the benchmark framing and cost/latency trade-offs.

To clarify what ATTRTRAP is fundamentally measuring: our core goal is to surface the pretraining prior biases and what models have really learnt that different models bring to table reading—for instance, as shown in Figure 2, models often favor specific attributes learned during pretraining rather than computing actual information gain from the current table.

The insights themselves would provide great guidelines on model selection in general for a customer service chatbot or at least acknowledge dev teams on the limitations of these models under this common scenario. These models, some just read the attribute names, some others may take a look at the record values and the rest actually do the math and thinking.

A couple of points on why the current benchmark and our scaling additions reflect real-world difficulty:

  • Grounded in real task distributions: The N≤8N \le 8 candidate sets aren't artificially easy; they are drawn directly from real tool-call retrieval logs in established customer service and sales benchmarks (SGD, MultiWOZ, τ\tau-bench, Olist), reflecting what agents actually encounter day-to-day.
  • Degradation starts early & stress-testing confirms it: Even on these real distributions, Table 1 shows that as soon as the number of records exceeds 2 (N>2N > 2), question quality already begins to degrade across non-reasoning setups. To stress-test how this scales when histories grow larger, we also put together the 8-to-64 candidate sets—where even strong reasoning models start to break down (e.g., Qwen3.8-27B dropping to 73.5% and Haiku to 35.5%).
  • Evaluating thinking-off: We completely agree on reporting thinking-off baselines for the newer models on the larger tables—we've included those numbers in our updated Table 1 as well so teams can see the exact gap between raw prior bias and explicit reasoning.

While re-framing the entire paper around large tables isn't feasible in the short rebuttal window, we will make sure to highlight both the pretraining prior insights (Figure 2) and the large-table stress-test results much more prominently in the final revision(uploaded). Thanks again for helping us sharpen the paper!

Thank you for the clarification and the thinking-off numbers.

Since this is a Datasets and Benchmarks submission, my main question is whether the benchmark can tell models apart, and I do not yet see evidence that it does for current models. By your own Table 1, every configuration is almost always optimal on the two-record tables, the reasoning models are near ceiling on the rest, and a one-line rule is already optimal on every table. Evidence of discriminative power is needed, for example rank stability across samples and a harder subset on which the rule and the best models differ.

The larger-table results are more informative, but please separate real retrieval results from composed sets. As I read Appendix C, the 8–64 record sets are put together from different searches or customers and "are not single retrieval results", and the MultiWOZ searches are enumerated over the database. So "real tool-call retrieval logs" is not accurate. On the real MultiWOZ searches the reasoning models are close to the rule, and the largest losses appear on the composed sets, where the best attribute is often a price or address a caller may not know.

Without such evidence, I do not see the current version's value as a benchmark, and I keep my assessment.

Reviewer_s9M3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies which attribute an LLM agent asks about when it must identify one record among 2 to 8 retrieved candidates. It introduces ATTRTRAP, comprising 238 tables drawn from four dataset families, and a second sample of 251 unmodified tables. The selected attribute is scored by its information-gain regret relative to the best available attribute, assuming a uniform prior and a user who can answer accurately.

The evaluation covers nine open-weight models and an additional GPT-4o comparison. Single-column edits reveal large differences between models' responses to completely uninformative and partly informative attributes. Prompt changes, supplied counts, and counting instructions have inconsistent effects. The strongest models' remaining regret is concentrated in settings where the most informative attribute, such as an exact price, may be difficult for a user to recall.

Critical strengths
  1. The measurement is precise and supported by useful controls. Computing partitions directly from table values avoids the model annotation used in earlier information-gain evaluations such as Mazzaccara et al. (2024). The authors also merge duplicate tables, report task-level uncertainty, and evaluate a second sample whose selection rules were fixed before its model runs (Section 3–4.1). These choices make the findings more credible than a collection of selected failures.

  2. The paired edits reveal behavior that aggregate scores miss. Changing one column while holding the other cells fixed allows a direct test of whether choices respond to that column’s usefulness (Section 4.3, Table 4). The contrast between extreme and mild edits shows that recognizing an obviously useless question and selecting the best informative question are different abilities.

Critical weaknesses
  1. The headline claim about reading tables is overstated. Mistral's selection of the edited column falls from 97.5% to 0.9% after flattening, although it remains at 75.9-82.5% after mild edits (Table 4). This demonstrates sensitivity to values and incomplete optimization, which conflicts with the abstract’s claim that four models do not react to table values and the introduction’s claim that weak models do not read the table.

  2. The repair experiments do not rule out the proposed explanations. Appendix B supplies counts without an instruction to select their maximum, permits a naturalness exception in the counting prompt, and tests the explicit objective separately without supplied counts. The repair conclusion also needs to account for GPT-4o improving from 0.049 to 0.007 bit with supplied counts and to approximately zero with counting instructions (Table 17).

  3. The practical significance of the strongest-model errors is limited. Table 25 attributes 91% of Qwen2.5-32B's second-sample regret to Movies, where price is usually optimal. Section 5 correctly acknowledges that users may not remember this value, so these results establish information-gain loss but do not establish worse clarification decisions for real users.

Questions for authors
  1. Does the failure persist when the prompt jointly supplies exact counts or information gains, explicitly requires the best-scoring attribute, states that the user knows all values, and removes the naturalness exception?

  2. What evidence supports describing Mistral, GLM, and Yi as not reacting to table values, given their responses to flattening in Table 4?

  3. Why do Figure 4 and Table 26 report different follow-up results, including DeepSeek’s minimum-round success of 45.7% versus 34.5%?

2 comments
AuthorsOpenReview ↗

Response to Reviewer s9M3

Thanks so much for the close reading and for checking our numbers!

In customer support, agents routinely query databases that return multiple records, making a precise clarifying question essential. Since this happens before any task completion, a poor question leads to a silent, hard-to-debug failure where the conversation just moves on. Many engineering teams don't even realize how suboptimal their clarification turns are. Knowing these trade-offs helps teams plan cost, latency, and decide when a code rule should make the choice instead.

W1 and Q2 (reading claim overstated; Mistral reacts to the extreme edit): You are right. Text updates: The abstract and Section 4.3 no longer claim that four models "do not react" to the values or that weak models "do not read the table." They now clarify that six models, including Mistral-7B, stop asking about an attribute whose EIG is 0, while four continue asking about one that is merely second best.

W2 and Q1 (repairs do not rule out the explanations; GPT-4o): Experiment added: We evaluated the prompt you described (printed counts, the rule "ask the attribute with the largest count," a user knowing every value, and no naturalness exception) on Mistral-7B and Yi-1.5-9B (Section 4.4, "Stating the rule"). After the mild edit, Mistral-7B still asks the edited attribute in about three-quarters of the questions, whereas Yi-1.5-9B moves off it. We also added both count hints on the 2026 models. Table updates: Table 2 has been redrawn to show, per model and hint, which tables become optimal and which stop being optimal. Text updates: Finding (2) is now strictly restricted to the nine open-weight models, and Section 4.4 explicitly reports that both hints help GPT-4o.

W3 (practical significance): Analysis added (with no new model calls): Regret is now scored against the best attribute other than an exact price (see the new appendix table). The regret of Qwen2.5-32B and Qwen2.5-14B drops by more than half on the benchmark and by less than half on the second sample. Text updates: The Discussion now highlights that determining whether a user can actually answer a given attribute is a judgment call that a hard-coded rule cannot make.

Q3 (Figure 4 vs Table 26): Correction: Figure 4 was generated by an outdated script. It has now been redrawn directly from Table 26, and both captions clearly define their quantities.

Thank you for the detailed response and additional experiments. The narrower claims about responses to table values address my concern about the original framing. I also appreciate the explicit discussion of the successful GPT-4o repairs.

The analysis excluding exact-price attributes helps clarify their contribution to the measured regret. Whether the remaining questions are answerable or preferable for real users remains a limitation. Thank you also for explaining and correcting the discrepancy between Figure 4 and Table 26.

Based on the response and described revisions, I am maintaining my rating of 3: weak acceptance.

Reviewer_ydo7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether task-oriented dialogue agents ground their clarifying questions in the retrieved records. The authors introduce ATTRTRAP and score questions through regret relative to the expected-information-gain optimum. Evaluations on real candidate tables show that all tested models ask non-optimal questions, while paired single-column interventions on SGD and Olist tables reveal substantial differences in how models adapt to changes in attribute informativeness. Providing exact attribute statistics or explicitly stating the objective does not reliably correct these failures.

Critical strengths
  1. The paper introduces ATTRTRAP, which evaluates clarifying questions using expected-information-gain regret.

  2. The table-editing experiments reveal substantial differences in how models adapt their question choices to changes in attribute informativeness.

Critical weaknesses
  1. The table-reading claim exceeds the experimental evidence. In Section 1, the authors claim that “weak models do not read the table; strong ones do”. However, the experiments assess whether attribute choices adapt to changes in informativeness, without directly verifying whether models can accurately read table contents. The authors should include a direct evaluation of table-value extraction to substantiate this claim.

  2. The model selection limits generalizability. The evaluation mainly covers older 7-32B open-source models, with closed-source coverage limited to 2024 GPT-4o snapshots. Testing newer open-source and stronger closed-source models would clarify whether the reported failure patterns persist in more capable systems.

  3. The abstract and introduction lack clear logic. They introduce technical controls and detailed findings without clearly establishing the limitations of existing clarification evaluations and how ATTRTRAP addresses them. This dense presentation obscures the motivation and makes the contribution difficult to follow. The authors should first clarify the evaluation gap, then explain their approach and its advantages before presenting detailed results.

Questions for authors

see weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer ydo7

Thanks so much for the close reading and for checking our numbers!

Honestly, in customer support, agents query DBs that return multiple records all the time, and a solid clarifying question is everything. Since this happens way before task completion, a bad question just fails silently—the conversation moves on, and it's a nightmare to debug. A lot of teams don't even realize how suboptimal their clarification turns actually are. Mapping out these trade-offs helps everyone balance cost and latency, and figure out when a simple code rule should just handle it instead.

To address your points, here are the new experiments and insights we've added:

  • What we added: We tested newer models and reasoning configs across all benchmark and second-sample tables (including edits and count hints). Specifically, we ran Qwen3.8-27B and Claude Haiku 5.5 (with thinking off/on), Claude Sonnet 5.5, and GPT-6 Luna at high reasoning effort (new Section 4.6 and Table 1), plus large-table scaling using real MultiWOZ searches and synthetic sets up to 64 records.
  • Why heavy models are usually overkill (Cost & Latency): Sure, scaling up or turning on thinking (like Qwen with thinking) helps performance, but it blows up output tokens (e.g., 266 vs. 35) and latency (12.3s vs. 4.0s). In a standard tool-call pipeline, forcing a massive reasoning model for a simple disambiguation step is massive overkill and a total waste of resources. A lightweight code rule is way more practical, deterministic, and cost-effective.
  • The real-world catch: At the end of the day, picking the attribute with the highest expected information gain (EIG) is only half the battle. In practice, the real bottleneck is whether the customer can actually remember or provide that specific attribute (like an exact price), which is a judgment call that static rules or raw models need to weigh carefully.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.