acceptodds
Under review as a conference paper at ICLR 2027

Lego-UX: Unlocking the Value of Human-Agent Interaction Trajectories

Abstract

Recent advances in Large Language Models (LLMs) have substantially improved coding agents on single-turn benchmarks such as SWE-bench. However, real-world coding assistance is naturally interactive: users refine requirements, report failures, and correct unsuccessful attempts across turns. In this setting, user experience (UX) encompasses both test-verified task completion and interaction quality, including how effectively the agent follows and adapts to user feedback. Improving this experience calls for learning from real-world human–agent trajectories, which provide valuable supervision for multi-turn interactions but are difficult to leverage reliably. Their quality varies widely, complicating training data selection, while their underlying tasks often lack reproducible environments for verification and replay. We introduce Lego-UX, a unified framework that uses interaction-aware trajectory selection and evidence-bounded task reconstruction for training and multi-turn evaluation. It identifies reliable supervision using behavioral and execution signals, while recovering executable tasks that facilitate multi-turn evaluation and controlled training rollouts. Using the reconstruction pipeline, we construct Lego-UXBench, a 95-task benchmark for multi-turn agent user-experience. In the experiments, we evaluate 11 frontier models on Lego-UXBench and observe that similar task-completion performance can correspond to markedly different interaction quality. Fine-tuning Qwen3.5-35B-A3B on selected trajectories improves both measured dimensions and raises the unweighted eight-metric macro score across four benchmarks by 15.2 points (7.75 to 22.91).

Primary area:
generative models
Keywords:
Coding Agents, Large Language Models, User–Agent Interaction, Data Curation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wUMvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LEGO-UX, a framework for using real-world human–agent coding trajectories for both training and multi-turn evaluation of coding agents. For training, the framework filters trajectories with deterministic rules (HardDrop) and with annotation-based criteria that retain failure and recovery behavior (FineHard). It then ranks the remaining trajectories with a weighted Agent Productivity Score to select SFT data. For evaluation, it anchors each trajectory to a GitHub commit, using the parent commit as the base state and the commit diff as the reference solution, and admits only candidates that satisfy the fail-to-pass criterion. Agents are evaluated in stateful multi-turn sessions driven by a GPT-5.5 user simulator, and the benchmark reports per-turn pass rates together with an interaction-quality score judged by GPT-5.5.

Critical strengths
  1. The paper offers a useful problem framing. Separating test-verified completion from interaction quality is a sensible lens for evaluating multi-turn coding agents. The results also include concrete cases where the two diverge, such as models with the same final pass rate but noticeably different interaction-quality rankings.

  2. The paper delivers a practically useful set of resources, covering both evaluation and training. On the evaluation side, the benchmark verifies every turn with deterministic fail-to-pass tests. This allows analysis of intermediate progress, such as how success changes across turns, how often agents recover from initial failures, and how consistent they are across independent rollouts. On the training side, the trajectory curation pipeline yields SFT data that improves a base model across several benchmarks.

Critical weaknesses
  1. The novelty of the work is limited, and the empirical findings are somewhat expected. The benchmark builds largely on established components. Commit-anchored task construction with fail-to-pass validation follows the SWE-bench pipeline, and multi-turn evaluation with LLM-simulated user feedback has precedent in MINT [1]. The experimental findings do not clearly add new insight that would compensate for this. Pass rate and interaction quality are positively related overall. The divergences highlighted by the authors involve small differences in interaction-quality scores and are reported without uncertainty estimates, so it is unclear how robust they are. The gain from multi-turn interaction is modest, and only a small fraction of initially failed tasks is recovered by the final turn. The paper does not examine why simulated feedback resolves so few failures, although such an analysis could have been a distinctive contribution of a multi-turn benchmark. Similarly, the trajectory analysis shows that longer sessions are associated with weaker completion. This is somewhat expected, since successful sessions tend to end early, as the authors themselves note.

  2. Multi-turn interaction, the central theme of the paper, relies on a simulator that is neither grounded in nor validated against real user behavior, and that is only briefly specified. The paper motivates its approach by how real users refine requirements, report failures, and correct the agent across turns, and it presents multi-turn evaluation as one of its core components. However, the original follow-up turns are not replayed during evaluation. The task instruction is reconstructed from the user query, and all subsequent turns are generated by a GPT-5.5 simulator with four generic intents. This simulator is not validated against real users. The design of the simulator is also described only briefly. Several settings are left unspecified, including how an intent is chosen at each turn, the maximum number of turns, the stopping criterion, and whether the simulator receives the verification signal in the reported runs. In addition, the same model serves as the simulator and the judge, and it also appears among the evaluated models.

[1] Wang, Xingyao, et al. "Mint: Evaluating llms in multi-turn interaction with tools and language feedback." International Conference on Learning Representations. Vol. 2024. 2024.

Questions for authors
  1. How faithfully does the user simulator reflect real user behavior? Have the simulated follow-up turns been compared with the original users’ follow-up turns in the source trajectories, or assessed by human annotators? Do model rankings remain stable when a different model is used as the simulator? Since the gain from multi-turn interaction is modest, it would also help to understand whether this reflects the simulator design or the agents themselves, for example through a breakdown of recovery by feedback type.

  2. Does training on the selected multi-turn trajectories improve interaction-specific behavior rather than general coding ability? For example, compared with SFT on the same tasks using only first-turn data, or on a randomly selected set of the same size, does the selected data lead to larger gains in recovery across turns and in interaction quality? The appendix notes that such a controlled comparison would be needed, and its result would clarify whether the value of real interaction trajectories is realized in training.

1 comment

Thank you for the valuable comments. Please find our point-by-point response below.

R3W1: The reviewer considers the methodological contributions incremental and argues that stronger empirical analysis, particularly of interaction-quality robustness and failure recovery.

While individual components have precedents in SWE-bench and MINT, LEGO-UX contributes a unified studies that connects interaction-aware trajectory selection, evidence-bounded task reconstruction, and stateful multiturn evaluation. Rather than replaying fixed feedback from source trajectories, our simulator generates follow-up requests conditioned on each interaction and workspace state. Moreover, LEGO-UXBench comprises tasks that better reflect the diversity of real-world usage.

Regarding empirical findings, our results reveal complementary aspects of coding-agent performance beyond task completion: models with identical final pass rates can exhibit different interaction quality. Additional human evaluation shows positive agreement between MeanIQ and human rankings (Spearman's ρ = 0.7), while cross-judge evaluation yields identical rankings (Kendall's τ_b = 1.0), providing further evidence of aggregate ranking consistency.

We further analyze recovery across feedback types in our response to R3Q1.

R3W2, and R3Q1: The reviewers request validation of simulated feedback against real human behavior, a complete simulator specification, and robustness checks across models.

As described in Section 5.1, we use GPT-5.5 to simulate user feedback based on the task instruction, interaction history, and execution evidence. Four feedback intents are assigned using a deterministic hash-based procedure with a fixed seed (42), ensuring consistency across agents while allowing variation across tasks. Feedback is generated at temperature 0.2, with access to sanitized verification signals but not hidden tests or raw verifier outputs. Interactions terminate when all tests pass, or the predefined turn budget is reached.

Human evaluation of feedback realism. To assess the realism of simulated user feedback, we developed a human annotation platform following the design of our LLM-based evaluation framework, allowing annotators to review interaction histories and available execution evidence. We randomly sampled 20 tasks and collected 100 trajectories from 5 models, with model identities masked. Two independent annotators rated 86% and 84% of simulated follow-ups as fairly or realistic (scores 4 out of 5), resp., with inter-annotator agreement (quadratic-weighted Cohen's κ = 0.65). These results support the contextual plausibility of simulated user feedback.

Recovery analysis. We evaluated five agents on each of 20 tasks. All five agents received the same type of follow-up for a given task. We counted a task as having a recovery if at least one agent failed a test on its first attempt and then passed all tests after one follow-up. Recovery occurred on 4 of the 20 tasks, with exactly one agent recovering on each.

Feedback typeTasks evaluatedTasks with recovery
Failure report72
Debugging request41
Requirement correction21
Request for additional tests70
Total204

Evaluation robustness. To assess sensitivity to the choice of judge, we additionally evaluated 96 archived trajectories from 24 randomly sampled tasks (25% of the benchmark), spanning four representative models, using GPT-5.6 Sol and DeepSeek-V4-Pro as alternative judges to GPT-5.5, with identical prompts. Across three IQ scoring repetitions, both alternative judges produced identical MeanIQ rankings (Kendall's τ_b = 1.00) and highly correlated trajectory-level scores (Pearson r = 0.99), indicating strong cross-judge consistency.

R3Q2: Does training on the selected multi-turn trajectories improve interaction-specific behavior rather than general coding ability?

We clarify that our work focuses on coding-agent user experience across both single-turn and multi-turn interactions. Accordingly, our selection strategy prioritizes supervision quality and interaction relevance, rather than requiring a fixed number of user turns, thereby retaining the diversity of real-world human–agent interactions.

Our evaluation covers both multi-turn benchmarks (LEGO-UXBench, SWE-Together, EvoCodeBench) and single-turn evaluation (SWE-bench Verified). As shown in Table 2, compared with the pretrained base model, APS50 improves performance on multi-turn benchmarks: LEGO-UXBench MeanIQ increases from 0.18 to 0.34, final-turn pass rate from 16.6% to 28.6%, and SWE-Together Pass_1 from 0.0% to 15.9%. On the single-turn SWE-bench Verified, Pass_1 increases from 2.0% to 47.0%. These results show that our selection improves measured interaction quality and task completion across multi-turn and single-turn evaluation settings.

Reviewer_Fv45OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LEGO-UX, a framework that uses real-world human–agent coding trajectories to improve and evaluate the user experience of coding agents. Using the reconstruction pipeline, the authors build LEGO-UXBench and evaluate 11 frontier models. They find that models with similar task completion can differ in interaction quality.

Critical strengths
  1. The problem is timely and important. Real coding assistance is multi-turn, and evaluating interaction quality separately from test-based success is a useful idea.

  2. I found the corpus analysis very informative: completion quality drops as the number of user turns grows, and interaction-rich trajectories are not necessarily reliable supervision.

Critical weaknesses
  1. The gains of the selection method are not isolated. Training starts from a base model Qwen3.5-35B-A3B-Base, which scores 2.0% on SWE-bench Verified, and HardDrop alone gives +11.0 of the +15.2 macro gain. How much of this comes from SFT itself rather than from filtering? How does the official post-trained Qwen3.5-35B-A3B perform?

  2. The interaction-quality metric is unstable. With the same GPT-5.5 judge, GPT-5.6 Sol ranks 1st by MeanIQ, 7th by Mean judge, and 6th by Elo. How do the rankings change with judges from other model families? How well does the judge agree with human ratings?

Questions for authors

See weaknesses.

1 comment

Thank you for the valuable comments. Please find our point-by-point response below.

R2W1: The gains of the selection method are not isolated. Training starts from a base model Qwen3.5-35B-A3B-Base, which scores 2.0% on SWE-bench Verified, and HardDrop alone gives +11.0 of the +15.2 macro gain. How much of this comes from SFT itself rather than from filtering? How does the official post-trained Qwen3.5-35B-A3B perform?

Table 2 provides controlled comparisons among SFT models that isolate the incremental benefits of subsequent selection stages.

We want to clarify that our experiments aim to evaluate the effectiveness and effects of different selection strategies under a fixed model initialization and SFT configuration. As shown in Table 2, adding FineHard to HardDrop-SFT increases the macro-average score from 18.8 to 21.8, while further applying APS50 raises it to 22.9, yielding a cumulative improvement of 4.1 points over HardDrop-SFT. These results demonstrate that finer-grained trajectory selection provides additional performance gains beyond the initial SFT stage.

These experiments aim to assess the effectiveness and incremental contributions of different filtering strategies under a controlled, limited-data SFT setting, rather than to outperform the officially post-trained Qwen3.5-35B-A3B model.

R2W2: The interaction-quality metric is unstable. With the same GPT-5.5 judge, GPT-5.6 Sol ranks 1st by MeanIQ, 7th by Mean judge, and 6th by Elo. How do the rankings change with judges from other model families? How well does the judge agree with human ratings?

MeanIQ, Mean judge, and Elo measure different aspects of performance; differences in their rankings do not necessarily indicate judge instability. Specifically, MeanIQ measures overall interaction quality, considering task success, requirement following, verification, and user effort. Mean judge focuses on task success, averaging scores across valid trials and tasks. Elo measures relative agent performance through pairwise comparisons aggregated using a Bradley–Terry model (Section 5.3 and Appendix C.1).

To assess judge sensitivity, we conducted an additional evaluation on archived trajectories from 24 randomly sampled tasks (25% of the benchmark), spanning four representative models, using GPT-5.5, GPT-5.6 Sol, and DeepSeek-V4-Pro as judges with identical prompts. Across three IQ scoring repetitions, the judges produced the same MeanIQ ranking: GPT-5.6 Sol > Claude Opus 4.8 > Kimi K3 > DeepSeek V3.2. Cross-judge agreement was high (Kendall's τ_b = 1.00; Pearson r = 0.99), indicating identical agent rankings and strong correlation in trajectory-level IQ scores.

For pairwise assessment, the Bradley–Terry rankings differed only in the relative positions of Claude and Kimi (Kendall's τ_b = 0.6). Reversing candidate order yielded 87% and 85.65% judgment consistency for GPT-5.6 Sol and DeepSeek-V4-Pro, respectively. These results suggest that MeanIQ rankings are robust to judge choice.

Reviewer_mzUMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a framework for improving and evaluating coding-agent user experience using real-world human-agent interaction trajectories. It consists of three components: (1) interaction-aware trajectory selection for training data curation, (2) evidence-bounded task reconstruction for converting trajectories into executable coding tasks, and (3) a multi-turn evaluation protocol measuring both functional correctness and interaction quality. The authors also construct a 95-task benchmark derived from real coding-agent interactions, and evaluate 11 frontier models. They show that task success and interaction quality are only moderately correlated and that supervised fine-tuning on carefully selected trajectories improves performance across multiple benchmarks.

Critical strengths
  • Addresses an important problem by focusing on the user experience of coding agents in realistic multi-turn interactions rather than only measuring task completion.
  • Introduces a benchmark derived from real-world human-agent interaction traces, providing stronger ecological validity than many existing coding-agent benchmarks.
  • Provides an important empirical finding that functional correctness and interaction quality are only moderately correlated, highlighting a limitation of current evaluation practices.
Critical weaknesses
  • The evaluation framework relies heavily on LLM-based user simulation and interaction-quality judging, making it unclear whether the reported UX metrics accurately reflect real user preferences and experiences.
  • The benchmark is relatively small, containing only 95 tasks, which raises questions about the robustness and generalizability of the conclusions across the broader software engineering domain.
  • While contamination mitigation is discussed, stronger analysis is needed to rule out potential overlap between reconstructed benchmark tasks and trajectory-derived training data.
  • The Aggregate Preference Score (APS) depends on manually selected component weights, and the paper does not provide sufficient justification or sensitivity analysis for these design choices.
  • The realism of the simulated user feedback patterns is not thoroughly validated against real-world user behavior, which may limit confidence in the benchmark's ability to measure practical coding-agent UX.
Questions for authors
  • How well does the proposed MeanIQ metric correlate with human judgments of coding-agent quality, and have the authors conducted any human evaluation to validate the reliability of the LLM-based interaction assessment?
  • How robust are the benchmark results to the choice of evaluator model? Would the relative ranking of coding agents remain consistent when using different judge models or evaluation frameworks?
  • The Aggregate Preference Score (APS) uses manually assigned weights for its components. How sensitive are the results to these weight choices, and was a systematic sensitivity analysis performed?
  • Given that LEGO-UXBENCH contains only 95 tasks, how confident are the authors that the benchmark is sufficiently representative of real-world coding-agent interactions and software engineering workflows?
  • Can the authors disentangle the impact of trajectory quality from the impact of data quantity? For example, how would randomly sampled trajectories of the same size compare against the proposed selection strategy?
  • Can the authors provide stronger contamination and data leakage analysis to demonstrate that benchmark tasks and training trajectories are sufficiently separated?
  • Since interaction quality and task success are only moderately correlated, do the authors envision MeanIQ becoming a primary evaluation metric, or should it be viewed as a complementary measure alongside traditional correctness metrics?
2 comments

Thank you for the valuable comments. Please find our point-by-point response below.

R1W1 and R1Q1: whether UX metrics based on simulated users and LLM judges reflect real users’ preferences and experiences. The reviewer requests human evaluation showing how closely MeanIQ agrees with human ratings of coding-agent interaction quality.

To assess the agreement between LLM-based and human evaluations of interaction quality, we developed a human annotation platform following our LLM-based evaluation framework, allowing annotators to review interaction histories and execution evidence. We randomly sampled 100 trajectories across 20 tasks and 5 models, with model identities masked. Two independent annotators achieves high inter-annotator agreement (quadratic-weighted Cohen's κ = 0.73).

To validate MeanIQ, we compared LLM-judged MeanIQ with mean human ratings on the same set of tasks with complete evaluations across all 5 agents. The correlations were Pearson's r = 0.79 and Spearman's ρ = 0.70, indicating positive agreement in aggregate assessments. This suggests that MeanIQ aligns closely with human judgments.

Using the same platform, we also assessed the realism of simulated feedback, with 85% of human ratings indicating fairly or realistic feedback (scores 4 out of 5). Refer to our response to R1W5.

R1W2 and R1Q4: The reviewer questions whether the 95-task benchmark adequately represents real-world coding-agent interactions and supports robust, generalizable conclusions across the broader software engineering domain.

LEGO-UXBench is designed to provide a diverse and realistic evaluation of multi-turn human-agent interactions. We address the reviewer's concerns from three perspectives: real-world task provenance, diverse task coverage, and empirical evaluation findings.

Real-world task provenance. LEGO-UXBench is constructed from three sources of real-world human–agent trajectories: CC-Bench Trajectories, GitHub PR Agent Trajectories, and SWE-chat. Using our task reconstruction pipeline, we first constructed over 400 candidate tasks from these trajectories, then selected 95 tasks based on executable validity and coverage considerations, with each task satisfying fail-to-pass verification. See Sec. 4 and App. B.

Diverse and complementary task coverage. As shown in Fig. 2, LEGO-UXBench's task distribution most closely resembles that of real-world trajectories among the compared benchmarks. Covering six categories and four languages, it better reflects the diversity of real-world tasks while providing complementary coverage to existing benchmarks.

Value for multi-turn evaluation. Across 11 models, LEGO-UXBench reveals that models with similar final pass rates can exhibit different interaction quality. This shows the value in assessing complementary aspects of model performance beyond functional correctness. See Sec. 6.2.

R1W3 and R1Q6: Stronger contamination and data leakage analysis to verify sufficient separation between benchmark tasks and training trajectories.

We conducted an additional contamination analysis against benchmarks that we used as requested by the reviewer. Following an N-gram-based overlap detection approach, we (1) normalized texts using NFKC; (2) extracted 50-character windows at word boundaries from evaluation instructions; and (3) used Aho–Corasick matching to detect overlap with training trajectories. Tasks were classified as Clean (<20%), Suspicious (20–80%), or Dirty (≥80%) based on overlap ratios.

We audited our APS_{50} data against the evaluated benchmarks. All were classified as clean, with no suspicious or dirty matches, providing further evidence against benchmark contamination.

CleanSuspiciousDirty
Lego-UXBench100%0%0%
SWE-Together100%0%0%
EvoCodeBench100%0%0%

R1W4 and R1Q3: The reviewer requests justification for the APS component weights and a sensitivity analysis assessing their impact on the results.

We assign APS weights based on the strength of evidence each component provides for correctness. We prioritize (1) verification quality because it provides the most direct evidence of whether code changes behave as intended, followed by (2) execution cleanliness, which captures execution failures and subsequent recovery, and (3) change productivity, which reflects successful code modifications but provides weaker evidence of correctness.

We conducted ablation of components in APS in Sec. 6.3, removing verification quality, execution cleanliness, or change productivity reduces the macro-average score from 22.9 to 21.2, 21.3, and 21.6, respectively. These results show the contribution of all components. A comprehensive weight-sensitivity analysis requires substantial additional training and evaluation costs. We thus focus on component-level ablations in the current study and leave a systematic exploration of alternative weight configurations for future work.

R1W4 and R1Q3: Justification for the APS component weights and a sensitivity analysis assessing their impact on the results.

We assign APS weights based on the strength of evidence each component provides for correctness. We prioritize (1) verification quality because it provides the most direct evidence of whether code changes behave as intended, followed by (2) execution cleanliness, which captures execution failures and subsequent recovery, and (3) change productivity, which reflects successful code modifications but provides weaker evidence of correctness.

We conducted ablation of components in APS in Sec. 6.3, removing verification quality, execution cleanliness, or change productivity reduces the macro-average score from 22.91 to 21.21, 21.30, and 21.55, resp.. These results demonstrate the contribution of all three components. A comprehensive weight-sensitivity analysis requires substantial additional training and evaluation costs. We thus focus on component-level ablations in the current study and leave a systematic exploration of alternative weight configurations for future work.

R1W5: The realism of the simulated user feedback patterns is not thoroughly validated against real-world user behavior.

To assess the realism of simulated user feedback, we developed an annotation platform following the design of our LLM-based evaluation framework, allowing annotators to review interaction histories and available execution evidence. We randomly sampled 20 tasks and collected 100 trajectories from 5 models, with model identities masked. Two annotators rated 86% and 84% of simulated follow-ups as fairly or realistic (scores 4 out of 5), respectively, with inter-annotator agreement (quadratic-weighted Cohen's κ = 0.65). It supports the contextual plausibility of simulated user feedback.

We also assessed the robustness of interaction-quality evaluation across judge models. The results show consistent MeanIQ rankings across different judges, as detailed in our response to R1Q2.

R1Q2: How robust are the benchmark results to the choice of evaluator model? Would the relative ranking of coding agents remain consistent when using different judge models or evaluation frameworks?

To assess judge sensitivity, we conducted an evaluation on trajectories from 24 randomly sampled tasks, spanning 4 representative models, using GPT-5.5, GPT-5.6 Sol and DeepSeek-V4-Pro as judges with identical prompts. Across three IQ scoring repetitions, the judges produced the same MeanIQ ranking: GPT-5.6 Sol > Claude Opus 4.8 > Kimi K3 > DeepSeek V3.2. Cross-judge agreement was high (Kendall's τ_b = 1.00; Pearson r = 0.99), indicating identical agent rankings and strong correlation in trajectory-level IQ scores.

For pairwise assessment, the Bradley–Terry rankings differed only in the relative positions of Claude and Kimi (Kendall's τ_b = 0.6). Reversing candidate order yielded 87% and 85.65% judgment consistency for GPT-5.6 Sol and DeepSeek-V4-Pro, respectively. It suggests that MeanIQ rankings are robust to judge choice.

R1Q5: Can the authors disentangle the impact of trajectory quality from the impact of data quantity? For example, how would randomly sampled trajectories of the same size compare against the proposed selection strategy?

We trained two models on 1,000 trajectories each using the same training configuration. For the random baseline, we sampled 1,000 trajectories from the data that passed HardDrop. For the proposed strategy, we further filtered this pool using FineHard, ranked the remaining trajectories by APS_50, and randomly sampled 1,000 data. Compared to the baseline, APS_50 improved LEGO-UXBench MeanIQ from 0.32 to 0.34 and Pass_1 from 18.4% to 24.0%. Pass_1 also rose from 4.6% to 15.9% on SWE-Together and from 39.0% to 47.0% on SWE-bench Verified.

R1Q7: Since interaction quality and task success are only moderately correlated, do the authors envision MeanIQ becoming a primary evaluation metric, or should it be viewed as a complementary measure alongside traditional correctness metrics?

The Interaction quality is intended as a complementary measure alongside traditional correctness metrics, rather than a replacement. As described in Sec. 5.3, test-based metrics assess whether coding tasks are successfully completed, while IQ captures interaction quality, including requirement following, correction handling, and verification. This evaluation design is consistent with prior work such as SWE-Together, which also considers interaction quality alongside task correctness.

Across 11 frontier models (in Sec. 6.2), MeanIQ and pass_F show a strong positive correlation (Pearson r = 0.94) and moderate agreement in rankings (Spearman ρ = 0.68). For example, GPT-5.6 Sol and Opus 4.8 achieve identical final pass rates but different MeanIQ. These results indicate that the two metrics are related but capture complementary aspects of model performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.