acceptodds
Under review as a conference paper at ICLR 2027

LAmp: Identifying Load-Bearing Assumptions for Outcome-Sensitive Clarification

Abstract

Language agents routinely act on underspecified instructions by silently committing to one interpretation. Yet ambiguity alone does not determine whether clarification is necessary: different valid readings can lead to the same task outcome. We call an unresolved assumption load-bearing when resolving it differently can materially change the task outcome. Good clarification thus requires coverage (asking about load-bearing assumptions) and selectivity (not asking unnecessarily). We introduce LAmp (Load-bearing AssuMPtion), a clarification framework that detects unresolved assumptions, enumerates their readings, and decides which are load-bearing, using structured reasoning or counterfactual execution. Since ambiguity annotations do not reveal whether the outcome materially changes, we construct execution-grounded evaluation cases: starting from a task that a fixed agent solves correctly, we remove one specification detail and rerun the agent; an observed outcome change yields a load-bearing evaluation target, while outcome preservation yields a negative control for the tested context. On text-to-SQL (BIRD), LAmp recovers 95% of load-bearing evaluation targets, versus 72% for the strongest prior method, and counterfactual execution makes clarification more selective at nearly the same coverage. The formulation transfers to software engineering (SWE-bench) without domain-specific adaptation, and the clarification policy can be distilled into a 7B model while keeping the execution agent fixed. Together, these results suggest that agents should not request clarification whenever an instruction is ambiguous, but when resolving the ambiguity differently can change the task outcome.

Primary area:
foundation or frontier models, including LLMs
Keywords:
clarifying questions, underspecified instructions, ambiguity resolution, language agents, outcome-sensitive clarification, load-bearing assumptions, selective clarification, agentic reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JuPyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LAmp, which identifies unresolved assumptions, enumerates their interpretations, and assesses whether different interpretations change task outcomes to decide what to clarify. The framework is evaluated on BIRD and SWE-bench, with an additional study on distilling the clarification policy into a 7B model.

Critical strengths

Deciding whether to clarify based on task outcomes is practically relevant. The paper evaluates downstream task success in addition to assumption recovery, and the improvements on the main BIRD cohort suggest that the approach has potential.

Critical weaknesses
  1. The contribution and novelty are limited. LAmp mainly organizes assumption identification, interpretation enumeration, and clarification decisions into a simple framework, offering an incremental methodological advance.

  2. The evaluation lacks simple yet effective baselines, such as a general-purpose agent equipped with clarification skills, which might achieve similar results. Evidence for reducing unnecessary questions and generalizing across tasks also remains limited.

Questions for authors

The proposed steps of identifying assumptions, enumerating interpretations, and deciding what to clarify appear straightforward to standardize as a reusable workflow. Could similar results be achieved simply by using agentic skills? Please clarify what additional benefit LAmp provides over such an implementation.

Reviewer_onwvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LAmp, a framework that identifies ambiguities in user instructions and clarifies only those that can materially change task outcomes. Authors name such assumptions "load-bearing assumptions". To construct such ambiguous datasets, authors start with BIRD text-to-SQL dataset, then using removal/editing of constraints for successfully executed tasks, constructs the ambiguous instances. Authors come with three settings: Settings 1 have outcome-changing edits, Settings 2 has naturally occurring failures and Settings 3 have outcome-preserving edits. Their methods LAmp seems to perform well on recovering the target assumptions in settings 1 and 3 well (with competing precision to baselines). Execution accuracy seems to be higher than most baselines.

Critical strengths
  1. Overall, the definition and setup of load-bearing assumptions are clear (modulo to statement equivalency checking: see comments below).
  2. Results seem to be strong for two benchmarks compared to well-established baselines (modulo concerns). They also show distillation in smaller 7B models help.
  3. Writing is clear in most places.
Critical weaknesses
  1. I have serious concerns that the constructed baselines seem to have high similarity wiuth designs of the constructed dataset setups. This seems to favor the proposed method. For example, removing "by revenue" or just facing "top customers" are two different setups, may have originated from different user intents, under various demograpical factors. -- A real world setup is heavily missed from the evaluations. Without that, commenting on the validity of the method raises a question. -- Very small number of SWE-Bench tasks are evaluated, and so is BIRD tasks.
  2. LAmp-Verify states "When controlled execution is available," -- even there judging two logically equivalent statements using their resulting sets is a computationally extensive tasks. There are significant failure modes involved, as only approximations seem practical.
  3. Similarly, the taxonomy remains severley dependent of the domain knowledge of the model -- again, something that can severely hamper the final performance of the model.

If the method works on a real-world publicly available benchmark which models these ambiguities naturally, I am willing to raise my scores.


The benchmark suggestion has been addressed and authors have addressed by their evaluation on AMBROSIA. Even though this is somewhat similar to the text-to-SQL BIRD setup, but since this is independently done and results are okay -- I am raising my scores.

Questions for authors

These are targted questions, line-by-line. Some may have overlap with the comments in the weaknesses.

  1. L064: This is minor. The term load-bearing is hard to intuitively connect any related concepts as CS researcher. Might be confusing. I suggest authors to re-think or justify more clearly.
  2. L131: Detecting such equivalent statements logically is quite difficult. I guess one needs to detect again through LLMs (or some approximations). What about the noise in such detection?
  3. L218: While the use of this taxonomy and augmentation is good, this looks quite difficult in domains where the models may not have much knowledge about. Can you show an experiment, in which this heavy lifting assumption can be somewhat relaxed for such cases?
  4. L262: My concern about the comparison is not fully addressed in A.2 I will request the authors to shed some light here.
Reviewer_6Gm4OpenReview ↗
Rating
2(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that clarification should target unresolved assumptions whose alternative readings materially change task outcomes. LAmp detects candidate assumptions, enumerates readings, and evaluates their consequences through model judgment or execution. It constructs outcome-changing and outcome-preserving edits for BIRD, studies naturally occurring failures and a selected SWE-bench cohort, and distills clarification behavior into a 7B model. The strongest BIRD variant improves target recall and downstream execution accuracy over the tested prompted baselines.

Critical strengths

The paper studies an important problem: when should an agent ask for clarification? I think the idea of checking whether different interpretations lead to different task outcomes is interesting. The authors also provide a theoretical definition and evaluate the method on BIRD and SWE-bench, with improvements over several existing baselines.

Critical weaknesses

My main concern is that the actual method does not seem to follow its theoretical definition. In Section 4, the authors state that LAmp-Verify changes one assumption "while holding the remaining readings fixed at aπ−j". This is important because the method needs to know whether changing one assumption actually causes a different outcome. However, Appendix A.2 states that "The implementation does not explicitly enforce this condition or identify the individual readings in aπ."

I think this is a serious problem. If the other assumptions are not kept fixed, the outcome may change for many different reasons. For example, when testing whether "top customers" should mean revenue or order count, the agent might also change which customers are included. In this case, we cannot know whether the different outcome is really caused by the assumption being tested.

This also raises concerns about the theoretical contribution. Proposition 1 is based on a controlled comparison where only one assumption changes. However, the actual implementation does not guarantee this condition. Therefore, I am not convinced that the theoretical analysis supports the proposed method or that the experiments really identify load-bearing assumptions as defined in the paper.

I also have concerns about how the evaluation data is constructed. The authors remove one detail from an originally correct instruction and check whether the execution result changes. However, an execution failure after removing some information does not necessarily mean that the removed information is a load-bearing assumption. The agent may simply make a reasoning error or choose a different execution strategy. Similarly, getting the same result after removing information does not mean that clarification is unnecessary. Other valid interpretations may still produce different results. I think the current evaluation is based too much on the behavior of one execution agent, rather than independently checking whether an assumption truly changes the task outcome.

Another important concern is that the experiments do not clearly demonstrate selective clarification. One of the main motivations is to avoid asking unnecessary questions, as in real life the questions is costy. However, LAmp still asks quite a few questions, including on the outcome-preserving examples. In the BIRD experiments, LAmp-Verify asks 2.64 questions on average for outcome-changing cases, compared with 1.65 for AmbiSQL and 1.34 for CLAMBER. Although LAmp achieves better target recall, part of this improvement may simply come from asking more questions. The paper needs stronger evidence that LAmp is better at selecting important questions, rather than just generating more candidates.

Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.