- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper picks a couple of test calls that reveal which setting does what (setting here is about standard or strict for tools where the exact effects arent documented). It then gives the agent a short note with the answer, and with that note agents almost always choose the right setting.
The puzzle being solved is very small. There are three settings, three known behaviours, and the right answer is guaranteed to be on the list, so two well-chosen test calls always solve it. The authors’ own comparison shows that letting the AI design its own test calls works just as well. So the gain comes from handing the agent the answer, not from the paper’s method of choosing the calls
Several headline numbers follow automatically from the setup rather than measuring anything. One table also doesn’t match another, in a study where the authors admit some agents could read the answer files.
Critical strengths
The examples drawn from real software, such as different languages rounding numbers differently, show the problem exists in practice. The authors also ran a comparison that undercuts their own main claim, and they report it directly.
Critical weaknesses
Addressed in the summary.
Questions for authors
- Why do the per-family results for one model average 43% when the main table reports 32%?
- Is there any situation where your method of choosing test calls beats simply letting the AI design its own?