- Rating
- 1(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies an "interpretation lock-in" problem that when a task contains ambiguous requirements, LLMs tend to commit to one of the possible interpretations and test only this one. They find this behavior in three models and show that LLMs actually have the knowledge of other interpretations but fail to consider them when implementing the code.
To solve this issue, they propose RDST, which fine-tunes an agent on its own episodes that manage to handle the most interpretations, and reduces lock-in behavior on their experiments.
However, they find that, such finetuning does not improve agent performance on real long-horizon benchmarks.
Critical strengths
-
a clear and well-measured problem. The paper clearly defines interpretation lock-in, and the controlled suite can check every reading automatically without human labels.
-
Insightful findings. Models can often name the readings they fail to implement, which shows the problem is about behavior rather than missing knowledge. The authors also report negative results openly that RDST does not help on real benchmarks.
Critical weaknesses
-
weak practical impact The proposed RDST does not improve results on real benchmarks. The paper also does not analyze how many real failures are actually caused by lock-ion. Therefore, the practical implication of it is unclear.
-
Lack evidence of generalization ability On kinds of variations not seen in training, RDST is no better than simply selecting successful episodes. Therefore, the algorithm may learn some specific formats rather than a general habit of considering other interpretations
Questions for authors
No. See weaknesses.
After reading other reviews, I decided to change my score.