acceptodds
Under review as a conference paper at ICLR 2027

Candidate Order in LLM-Generated Agent Environments: Prompted Placement, Measured Difficulty and Learned Reliance

Abstract

Language models increasingly write the environments used to evaluate and train tool-using agents. Such environments are checked for solvability and difficulty, not for where the correct option sits among an agent's candidates. We study this in a record-chain lookup task in which each record lists one labelled active candidate and two decoys. In fresh generated worlds, Sonnet 5 listed the active candidate first in 85.0% of these choices and two GPT-5.6 generators in all of them; the format example in the generation prompt also listed it first. Three separate controlled studies examine this regularity. Moving the active object to second or third place in the format example reduced first-position placement for all three generators (Holm-adjusted p ≤ 3×10⁻⁵ each), almost completely for one and partially for the other two. On identical worlds, the generators' own order raised traversal completion by 6.4 to 15.2 percentage points for Qwen3-1.7B and by 3.4 to 12.4 points for Qwen3-4B, while re-permuting an already shuffled reference produced no detectable change; the 4B solver's final answers showed no corresponding advantage. In matched fine-tuning of Qwen3-1.7B, permuting the candidate lists inside otherwise identical training trajectories raised held-out traversal completion by 58.6 to 86.8 points across three seed pairs and final-answer success by +10.46 points on average; models trained on the generators' order mostly followed a decoy once the order changed. The studies share one task mechanism and one model family and do not test the full prompt-to-learner chain. Generated environments should randomise and report candidate order.

Primary area:
datasets and benchmarks
Keywords:
agentic environments, synthetic environment generation, candidate order, position bias, tool use, benchmark validity, supervised fine-tuning, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FgXROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies bias from candidate order in LLM-generated agent environments. In a record chain lookup task, the authors find that the generator almost always places the correct candidate first, and through three independent experiments they show that the position of the format example in the generation prompt affects this arrangement, that the original order raises traversal completion of the small model solver, and that fine-tuning on trajectories with this order leads the model to learn to rely on position. The authors therefore suggest randomly shuffling and reporting candidate order when generating environments.

Critical strengths

The experimental design is relatively tightly controlled, each study sets an analysis plan and a multiple comparison correction in advance, and reproducible analysis materials are provided.

Critical weaknesses
  1. The bias studied in the paper comes mainly from the format example in the authors' own prompt, and the paper does not show that it exists in published generated environments. The authors also acknowledge that the three conclusions are not new, and reproducing them on a single toy task gives a limited contribution.
  2. The effect appears only on weaker solvers, stronger models are already close to a full score, and the final answer of 4B also shows no improvement. The training experiment uses extreme data that is one hundred percent ordered, no effect is observed on an external benchmark either, and the practical impact is very small.
Questions for authors
  1. Since the generator places the correct candidate first mainly by copying the format example, does this bias still exist under a prompt that contains no fixed position example, or in published LLM-generated environments?
  2. Since order affects only the traversal of weaker solvers and not the final answer, and the training experiment uses data that is one hundred percent ordered, how much impact would this bias still have on stronger models and on more realistic data that is only partly ordered?
  3. Since rewriting the instruction wording in the appendix lowers traversal completion from 0.861 to 0.713, with a drop larger than the effect of order, does candidate order deserve to be treated on its own as a main issue in environment construction compared with wording and other presentation factors?
Reviewer_bbcAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies candidate order in LLM-generated record-chain tasks, where agents follow labelled active candidates and sum the records' amounts. Three controlled studies show that format-example position changes generated placement, native order improves two Qwen3 solvers' traversal on identical worlds, and changing only candidate order in fine-tuning trajectories alters robustness to ordering. Across three seed pairs, randomized-order training improves reference-set traversal by 77.13 percentage points and strict success by 10.46 points. The paper distinguishes these outcomes and does not test the full causal chain from a prompt change to a learning outcome.

Critical strengths
  1. The controlled comparisons isolate presentation order. P1–P3 change object positions within the same format example; evaluation compares two presentations of identical worlds and includes an already randomized reference. Both solvers show a native-order traversal advantage. This directly supports an effect of presentation on measured lookup capability and is useful for validating generated environments.
  2. Training and behavioral probes provide converging evidence. Fine-tuning conditions match trajectory content, targets, loss masks, token counts, and example order within each seed pair. All three pairs show large traversal gains; restoring active-first test presentation restores native performance; and first-decision probes demonstrate positional dependence. Together with improved strict success, these results substantiate candidate randomization as a training-data design choice.
Critical weaknesses
  1. Empirical coverage limits generality. The eight domains share one record-chain mechanism, the main solvers belong to the Qwen3 family, and training studies one 1.7B model with entirely active-first data. The external benchmark already has random order, so re-permuting it does not establish the prevalence of naturally generated shortcuts across tasks or transfer of the trained policies. Practical effects in other environments and under milder positional skew remain uncertain. This limits the contribution's breadth without invalidating the controlled findings.
Questions for authors
  1. Decoy resolvability. Section 5 and Appendix C provide the 4B sensitivity. For the main 1.7B fresh-world experiment, how do native-minus-shuffled traversal gains change when restricted to worlds where all advertised gold-path decoys resolve? Per-generator exclusion counts and paired gains from existing data would clarify robustness to this requirement.
Reviewer_rEZoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies candidate-order bias in LLM-generated agent environments, where the placement of correct candidates may affect agent evaluation and training. Using a synthetic record-chain lookup task, this paper presents three findings: (1) the position of the "active" candidate in the format example of the generation prompt influences its placement in generated environments; (2) the native candidate order of generators leads to higher traversal completion rates than shuffled order on identical worlds; and (3) fine-tuning on trajectories where the "active" candidate always appears first causes agents to develop strong positional reliance. These findings highlight the risk of learned positional shortcuts and motivate randomizing candidate order. However, the study extends known positional-bias phenomena to a specific synthetic task, with limited evidence of broader practical significance and generalizability, and limited additional insight into the underlying mechanisms.

Critical strengths

S1. Unified investigation with well-controlled experiments. The paper incorporates three phenomena studied in prior work within a unified multi-step agent task: generator position bias, evaluation shortcuts, and learned positional reliance. The well-controlled experiments and distinct evaluation metrics isolate the effects of candidate order on agent evaluation and training, providing a coherent and empirically grounded perspective on these findings.

Critical weaknesses

W1. Limited practical significance and generalizability. The paper primarily investigates candidate-order bias in a specific constructed synthetic record-chain lookup task, where the correct candidate is explicitly identified by an "active" label. Although the controlled prompt experiments show that format examples influence candidate placement, there is limited evidence that such biases are prevalent in existing LLM-generated agent environments or meaningfully affect more realistic agent tasks. Moreover, the eight domains share the same task mechanism, and the external AgentCE-Bench evaluation, where record order is already randomized, does not establish broader applicability. Hence, the practical significance and generalizability of the findings remain unclear.

W2. Limited novelty and insufficient investigation of underlying mechanisms. The three main findings closely relate to phenomena established in prior work (Tang et al., 2026; Shao et al., 2026; Maniyar et al., 2026). Although the paper provides additional controlled evidence in generated interactive environments, it does not sufficiently establish what new insights this setting provides beyond those prior findings. For example, Study 1 only varies the position of the active candidate in a single format example, without examining how multiple examples with different positions or their presentation order influence generation. Study 3 uses an extreme training distribution where the active candidate always appears first. Although this distribution arises naturally in the studied generators’ outputs, the paper does not investigate other fixed positions or more realistic partial biases. Furthermore, despite adopting a multi-step agent task, the paper does not adequately characterize how positional errors accumulate or how agents recover across steps, nor does it validate the end-to-end propagation of bias from generation prompts to downstream training. These gaps limit the mechanistic understanding and incremental scientific contribution of this work beyond existing positional-bias studies.

Questions for authors

Q1. The paper demonstrates candidate-order bias in a synthetic record-chain lookup task, but its prevalence in real-world agent environments remains unclear. Is there evidence that existing LLM-generated agent environment frameworks exhibit similar biases, particularly when generation prompts do not contain fixed-position examples?

Q2. Study 1 only varies the position of the active candidate in a single format example. How would the generated position distribution change when multiple examples are provided, particularly when they have the same positional distribution but different presentation orders (e.g., [1, 3] vs. [3, 1], or [1, 2, 3] vs. [3, 2, 1])?

Q3. Study 3 uses an extreme training distribution where the active candidate always appears first, as observed in the outputs of studied generators. How does learned positional reliance change under other fixed positions (second or third) or more realistic partial biases (e.g., 50% active-first)?

Q4. Although the paper examines depth-dependent effects and provides trajectory categories and first-decision probes, it does not directly characterize how positional errors propagate through multi-step tool use. Can the authors analyze per-step candidate-selection errors and subsequent recovery behavior to explain how positional bias affects traversal completion as task depth increases?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.