acceptodds
Under review as a conference paper at ICLR 2027

Agents Verify Their Assumptions, Not the Specification: Interpretation Lock-in in Autonomous Coding Agents

Abstract

Large language models (LLMs) have become capable coding agents that complete long software tasks autonomously and test their own work. While LLMs excel at well-defined programming tasks such as code generation and bug fixing, their self-verification breaks down when a requirement is ambiguous, because the tests they write share the interpretation of the code they test. In this paper, we study this failure, which we call interpretation lock-in: agents commit to one reading of an ambiguous requirement, implement it, and validate their work against that same reading. In a controlled suite where every valid reading is known, and in pipelines rebuilt from long-horizon benchmark runs, the resulting programs crash or silently produce wrong output under other valid readings. The failure is largely one of commitment rather than knowledge: asked only to list plausible readings, the same models often name the one they missed. To remedy this failure, we propose reading-diverse self-training (RDST), which rewards an agent’s own so- lutions for the number of valid readings they handle and fine-tunes the agent on its best episodes. Experimental results show that, relative to rewarding success on a single reading, RDST substantially reduces lock-in, including on task families held out from training, at little cost on well-specified tasks, and that the behaviour carries over to the interaction protocol of another agent harness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, coding agents, long-horizon tasks, underspecified requirements, specification ambiguity, self-verification, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wsRWOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies an "interpretation lock-in" problem that when a task contains ambiguous requirements, LLMs tend to commit to one of the possible interpretations and test only this one. They find this behavior in three models and show that LLMs actually have the knowledge of other interpretations but fail to consider them when implementing the code.

To solve this issue, they propose RDST, which fine-tunes an agent on its own episodes that manage to handle the most interpretations, and reduces lock-in behavior on their experiments.

However, they find that, such finetuning does not improve agent performance on real long-horizon benchmarks.

Critical strengths
  1. a clear and well-measured problem. The paper clearly defines interpretation lock-in, and the controlled suite can check every reading automatically without human labels.

  2. Insightful findings. Models can often name the readings they fail to implement, which shows the problem is about behavior rather than missing knowledge. The authors also report negative results openly that RDST does not help on real benchmarks.

Critical weaknesses
  1. weak practical impact The proposed RDST does not improve results on real benchmarks. The paper also does not analyze how many real failures are actually caused by lock-ion. Therefore, the practical implication of it is unclear.

  2. Lack evidence of generalization ability On kinds of variations not seen in training, RDST is no better than simply selecting successful episodes. Therefore, the algorithm may learn some specific formats rather than a general habit of considering other interpretations

Questions for authors

No. See weaknesses.

1 comment

After reading other reviews, I decided to change my score.

Reviewer_zeDsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the problem of ambiguous requirements. In particular, an agent behavior of locking in to one reading of an ambiguous requirements and never look back to original requirements to verify fit. The authors propose reading-diverse self-training (RDST) which rewards an agent's own solution for the number of valid readings they handle. The experiment shows a substantial reduction of lock-in.

Critical strengths
  • Important topic of spec-test alignment
  • Well-written and easy to follow
Critical weaknesses

I think the work starts at a very important topic on spec-test alignment however slightly drifted towards a setting that I am not fully convinced at. In particular:

  1. I agree with the authors that real specifications are rarely complete and can often be ambiguous. However the paper focuses on a specific type of ambiguity where multiple explanations needs to be satisfied all at once, aka they are more like edge cases that needs to be considered. I believe another important type (arguably more prominent type) of ambiguity is when different readings of requirements actually CONFLICT with each other and cannot be realistically satisfied at the same time, often caused by underspecifications (i.e., missing statements). The paper did not sufficiently justify why such scenario was not the focus in the setup and why their focus of the edge case covering is more important or worth studying by itself.

  2. Other minor issues:

a. The example given in Fig 1, instead of an ambiguity, looks more like a direct conflict, where the task explicitly listed fields like discount_frac. Even the note itself only say discounts may be expressed as percentages or basis points instead of fractions, instead of saying there might be very different keys exist in data.

b. the statement that autonomous agent in the middle of a long task usually has no one to ask for clarification is not accurate. As the paper pointed out for related work, upfront clarifications is a critical capability in modern llm training and modern harnesses all come with tools designed for agents to escalate problems to human.

Questions for authors

Can ambiguity be conflicting as well? How can your current experiment design and proposed method deal with that?

Reviewer_zqC7OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper looks at a specific failure mode of coding agents, namely committing to a particular interpretation of an ambiguous task specification. This specific failure is termed interpretation lock-in and the paper proposes a new method called reading-diverse self-training (RDST) to avoid this failure mode in practice. Essentially, the method rewards the agent's own solutions corresponding to valid interpretations it handles and therefore it fine-tunes the model on the best episodes. The empirical evaluation is carried out on a synthetic dataset as well as two realistic coding datasets. The results show that the proposed RDST method improves performance compared with the baselines.

Critical strengths
  • The paper considers an important failure mode of coding agents and proposes a method to address it.
Critical weaknesses
  • Quality of the presentation. The paper content looks heavily AI-assisted. While the first 2 sections are fairly easy to follow, the experimental section is really hard to follow. There are too many numbers floating around and it is very difficult to pinpoint the performance gains. I would suggest including a high-level summary of the main takeaways before delving into the details.

  • The scope of the evaluation appears fairly limited. There is only one kind of model families that is evaluated and I think that it's not enough to draw more general conclusions.

Questions for authors
  • I was wondering if the frontier models like Claude or GPT-6 suffer from the same issues? Have you tried them?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.