- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies bias from candidate order in LLM-generated agent environments. In a record chain lookup task, the authors find that the generator almost always places the correct candidate first, and through three independent experiments they show that the position of the format example in the generation prompt affects this arrangement, that the original order raises traversal completion of the small model solver, and that fine-tuning on trajectories with this order leads the model to learn to rely on position. The authors therefore suggest randomly shuffling and reporting candidate order when generating environments.
Critical strengths
The experimental design is relatively tightly controlled, each study sets an analysis plan and a multiple comparison correction in advance, and reproducible analysis materials are provided.
Critical weaknesses
- The bias studied in the paper comes mainly from the format example in the authors' own prompt, and the paper does not show that it exists in published generated environments. The authors also acknowledge that the three conclusions are not new, and reproducing them on a single toy task gives a limited contribution.
- The effect appears only on weaker solvers, stronger models are already close to a full score, and the final answer of 4B also shows no improvement. The training experiment uses extreme data that is one hundred percent ordered, no effect is observed on an external benchmark either, and the practical impact is very small.
Questions for authors
- Since the generator places the correct candidate first mainly by copying the format example, does this bias still exist under a prompt that contains no fixed position example, or in published LLM-generated environments?
- Since order affects only the traversal of weaker solvers and not the final answer, and the training experiment uses data that is one hundred percent ordered, how much impact would this bias still have on stronger models and on more realistic data that is only partly ordered?
- Since rewriting the instruction wording in the appendix lowers traversal completion from 0.861 to 0.713, with a drop larger than the effect of order, does candidate order deserve to be treated on its own as a main issue in environment construction compared with wording and other presentation factors?