- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LEGO-UX, a framework for using real-world human–agent coding trajectories for both training and multi-turn evaluation of coding agents. For training, the framework filters trajectories with deterministic rules (HardDrop) and with annotation-based criteria that retain failure and recovery behavior (FineHard). It then ranks the remaining trajectories with a weighted Agent Productivity Score to select SFT data. For evaluation, it anchors each trajectory to a GitHub commit, using the parent commit as the base state and the commit diff as the reference solution, and admits only candidates that satisfy the fail-to-pass criterion. Agents are evaluated in stateful multi-turn sessions driven by a GPT-5.5 user simulator, and the benchmark reports per-turn pass rates together with an interaction-quality score judged by GPT-5.5.
Critical strengths
-
The paper offers a useful problem framing. Separating test-verified completion from interaction quality is a sensible lens for evaluating multi-turn coding agents. The results also include concrete cases where the two diverge, such as models with the same final pass rate but noticeably different interaction-quality rankings.
-
The paper delivers a practically useful set of resources, covering both evaluation and training. On the evaluation side, the benchmark verifies every turn with deterministic fail-to-pass tests. This allows analysis of intermediate progress, such as how success changes across turns, how often agents recover from initial failures, and how consistent they are across independent rollouts. On the training side, the trajectory curation pipeline yields SFT data that improves a base model across several benchmarks.
Critical weaknesses
-
The novelty of the work is limited, and the empirical findings are somewhat expected. The benchmark builds largely on established components. Commit-anchored task construction with fail-to-pass validation follows the SWE-bench pipeline, and multi-turn evaluation with LLM-simulated user feedback has precedent in MINT [1]. The experimental findings do not clearly add new insight that would compensate for this. Pass rate and interaction quality are positively related overall. The divergences highlighted by the authors involve small differences in interaction-quality scores and are reported without uncertainty estimates, so it is unclear how robust they are. The gain from multi-turn interaction is modest, and only a small fraction of initially failed tasks is recovered by the final turn. The paper does not examine why simulated feedback resolves so few failures, although such an analysis could have been a distinctive contribution of a multi-turn benchmark. Similarly, the trajectory analysis shows that longer sessions are associated with weaker completion. This is somewhat expected, since successful sessions tend to end early, as the authors themselves note.
-
Multi-turn interaction, the central theme of the paper, relies on a simulator that is neither grounded in nor validated against real user behavior, and that is only briefly specified. The paper motivates its approach by how real users refine requirements, report failures, and correct the agent across turns, and it presents multi-turn evaluation as one of its core components. However, the original follow-up turns are not replayed during evaluation. The task instruction is reconstructed from the user query, and all subsequent turns are generated by a GPT-5.5 simulator with four generic intents. This simulator is not validated against real users. The design of the simulator is also described only briefly. Several settings are left unspecified, including how an intent is chosen at each turn, the maximum number of turns, the stopping criterion, and whether the simulator receives the verification signal in the reported runs. In addition, the same model serves as the simulator and the judge, and it also appears among the evaluated models.
[1] Wang, Xingyao, et al. "Mint: Evaluating llms in multi-turn interaction with tools and language feedback." International Conference on Learning Representations. Vol. 2024. 2024.
Questions for authors
-
How faithfully does the user simulator reflect real user behavior? Have the simulated follow-up turns been compared with the original users’ follow-up turns in the source trajectories, or assessed by human annotators? Do model rankings remain stable when a different model is used as the simulator? Since the gain from multi-turn interaction is modest, it would also help to understand whether this reflects the simulator design or the agents themselves, for example through a breakdown of recovery by feedback type.
-
Does training on the selected multi-turn trajectories improve interaction-specific behavior rather than general coding ability? For example, compared with SFT on the same tasks using only first-turn data, or on a randomly selected set of the same size, does the selected data lead to larger gains in recovery across turns and in interaction quality? The appendix notes that such a controlled comparison would be needed, and its result would clarify whether the value of real interaction trajectories is realized in training.
Thank you for the valuable comments. Please find our point-by-point response below.
While individual components have precedents in SWE-bench and MINT, LEGO-UX contributes a unified studies that connects interaction-aware trajectory selection, evidence-bounded task reconstruction, and stateful multiturn evaluation. Rather than replaying fixed feedback from source trajectories, our simulator generates follow-up requests conditioned on each interaction and workspace state. Moreover, LEGO-UXBench comprises tasks that better reflect the diversity of real-world usage.
Regarding empirical findings, our results reveal complementary aspects of coding-agent performance beyond task completion: models with identical final pass rates can exhibit different interaction quality. Additional human evaluation shows positive agreement between MeanIQ and human rankings (Spearman's ρ = 0.7), while cross-judge evaluation yields identical rankings (Kendall's τ_b = 1.0), providing further evidence of aggregate ranking consistency.
We further analyze recovery across feedback types in our response to R3Q1.
As described in Section 5.1, we use GPT-5.5 to simulate user feedback based on the task instruction, interaction history, and execution evidence. Four feedback intents are assigned using a deterministic hash-based procedure with a fixed seed (42), ensuring consistency across agents while allowing variation across tasks. Feedback is generated at temperature 0.2, with access to sanitized verification signals but not hidden tests or raw verifier outputs. Interactions terminate when all tests pass, or the predefined turn budget is reached.
Human evaluation of feedback realism. To assess the realism of simulated user feedback, we developed a human annotation platform following the design of our LLM-based evaluation framework, allowing annotators to review interaction histories and available execution evidence. We randomly sampled 20 tasks and collected 100 trajectories from 5 models, with model identities masked. Two independent annotators rated 86% and 84% of simulated follow-ups as fairly or realistic (scores 4 out of 5), resp., with inter-annotator agreement (quadratic-weighted Cohen's κ = 0.65). These results support the contextual plausibility of simulated user feedback.
Recovery analysis. We evaluated five agents on each of 20 tasks. All five agents received the same type of follow-up for a given task. We counted a task as having a recovery if at least one agent failed a test on its first attempt and then passed all tests after one follow-up. Recovery occurred on 4 of the 20 tasks, with exactly one agent recovering on each.
Evaluation robustness. To assess sensitivity to the choice of judge, we additionally evaluated 96 archived trajectories from 24 randomly sampled tasks (25% of the benchmark), spanning four representative models, using GPT-5.6 Sol and DeepSeek-V4-Pro as alternative judges to GPT-5.5, with identical prompts. Across three IQ scoring repetitions, both alternative judges produced identical MeanIQ rankings (Kendall's τ_b = 1.00) and highly correlated trajectory-level scores (Pearson r = 0.99), indicating strong cross-judge consistency.
We clarify that our work focuses on coding-agent user experience across both single-turn and multi-turn interactions. Accordingly, our selection strategy prioritizes supervision quality and interaction relevance, rather than requiring a fixed number of user turns, thereby retaining the diversity of real-world human–agent interactions.
Our evaluation covers both multi-turn benchmarks (LEGO-UXBench, SWE-Together, EvoCodeBench) and single-turn evaluation (SWE-bench Verified). As shown in Table 2, compared with the pretrained base model, APS50 improves performance on multi-turn benchmarks: LEGO-UXBench MeanIQ increases from 0.18 to 0.34, final-turn pass rate from 16.6% to 28.6%, and SWE-Together Pass_1 from 0.0% to 15.9%. On the single-turn SWE-bench Verified, Pass_1 increases from 2.0% to 47.0%. These results show that our selection improves measured interaction quality and task completion across multi-turn and single-turn evaluation settings.