acceptodds
Withdrawn from ICLR 2027

Trace2Eval: Contextualized Evaluation of Coding Agents from Real-World Traces

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Coding agents are increasingly used in real-world workflows across diverse domains, motivating evaluation grounded in real user tasks. Existing benchmarks, however, largely rely on static public tasks with reference patches or hidden tests. Next-generation coding-agent benchmarks should therefore move beyond correctness on fixed tasks toward continually renewed evaluation of overall artifact quality in real user workflows. We propose Trace2Eval, a dynamically updatable pipeline that constructs benchmarks from real-world interaction trajectories. Trace2Eval employs a multi-agent reconstruction system to analyze the filtered trajectory, recover the original user query, restore the workspace files to their pre-interaction state, and rebuild a trajectory-equivalent execution environment. A candidate agent then executes the reconstructed query to produce an initial artifact. To evaluate these artifacts, the pipeline iteratively refines human rubrics through inter-annotator alignment, and calibrates judge agents against human assessments using source-code and runtime evidence. Guided by feedback from the Trace2Eval pipeline, models revise their initial artifacts, yielding a mean score gain of 10.05 percentage points across 19 model-harness configurations and a maximum of 19.28 percentage points for DeepSeek-V4-Pro with Claude Code, which supports rubric feedback as actionable guidance for artifact improvement.

Primary area:
datasets and benchmarks
Keywords:
Coding Agent Evaluation, Contextualized Benchmark Construction, Rubric-Guided Feedback

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9gfFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Trace2Eval, the pipeline that constructs coding benchmarks from real-world interaction trajectories. Existing benchmarks highly rely on reference patches and hidden tests, but the authors argue that this static public tasks are limited to evaluate next-generation coding agent. They propose four properties that good benchmark should satisfy, authenticity, contamination resistance, dynamism, and measurability. The proposed method firstly select the trace, recover the context, execute the agent, and evaluate the artifact. Across 19 model-harness configurations, rubric-derived feedback improves artifact scores by 10.05 on average.

Critical strengths
  • The problem setting is well-motivated and important. Evaluation of agents on real user requests with recovered workspace effectively addresses narrow task distribution of existing benchmarks such as SWE-bench

  • The authors conducted a wide scope of experiments with 2 harnesses and 19 model-harness configurations. They also included ablation study, paired bootstrap CI, per task improvement and regression counts

Critical weaknesses
  • The LLM judge reliability is too limited. Agent-GT kappa in Table 13 is only 0.173, compared with 0.542 of Human-GT. Furthermore, for code quality, which is 35% of the total weight, the kappa is only 0.070. Therefore, the difference between model of Table 1 can be lied in noise.

  • The rubric-derived feedback is circular. The judge's score generate the feedback, and the same judge and rubric then score R2. Therefore, the gains may reflect optimizing toward the judge, not the real quality improvement.

Questions for authors
  • GLM models serve as judge, feedback generator, and candidates. Do model rankings and R2 gains hold with a judge from a different model family?
Reviewer_DVPROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Trace2Eval, a framework that constructs coding-agent benchmarks from real-world interaction traces. It reconstructs user requests, initial workspaces, and execution environments, then evaluates generated artifacts using human-grounded rubrics and LLM judges. Experiments on 129 tasks across 19 model-harness configurations report an average improvement of 10.05 percentage points after rubric-guided revision.

Critical strengths
  1. The paper addresses an important limitation of existing coding benchmarks by constructing evaluation tasks from real-world user interactions rather than relying solely on static programming problems.
  2. The evaluation considers multiple dimensions of artifact quality, including functionality, code quality, and user experience, providing a broader assessment than correctness-only benchmarks.
Critical weaknesses
  1. The LLM judge shows poor agreement with human evaluations. Table 13 reports a weighted agent–GT Cohen's k of only 0.173, compared with 0.542 for human–GT agreement, raising serious concerns about scoring reliability.
  2. The reported average improvement of 10.05 percentage points lacks independent human validation. Although Table 16 provides feedback ablations and the case studies demonstrate functional improvements, the overall gains are still measured by the same evaluation framework used to generate the feedback.
Questions for authors

In addition to the weaknesses, the following issues also exist.

  1. Table 6 reports distribution alignment for 300 candidate tasks rather than the final 129 tasks. Can you provide distribution alignment and recovery success rates for the final benchmark?
  2. The rubric weights are determined by expert judgments rather than direct user evaluations. How do you verify that these scores reflect actual user preferences?
  3. The paper claims that the benchmark supports dynamic updates, but only one collection period is evaluated. Can you demonstrate benchmark updates over time and assess model ranking stability?
Reviewer_mf5zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a pipeline for constructing agent evaluation tasks from real user trajectories. It includes retrieving, filtering, and annotating online trajectories; restoring the initial workspace and runtime environment; developing scoring criteria and calibrating them with human evaluation; and conducting automated scoring and feedback-driven iteration using both code and execution evidence. The authors ultimately construct 129 tasks, compare 19 model-and-runtime-framework combinations, report an average 10.05-percentage-point improvement from feedback-guided revisions, and provide controlled experiments demonstrating the value of feedback.

Critical strengths
  1. Grounded in real business and development scenarios, the tasks are derived from online user requests and cover a diverse range of practical needs.

  2. Transforms online tasks into reproducible evaluation environments. By restoring each task along with its initial workspace and runtime environment, it enables capability evaluation and feedback iteration independent of the original user session, offering a practical path for evaluating continually updated models on real-world tasks.

  3. Provides controlled experiments demonstrating the value of feedback.

Critical weaknesses
  1. The workflow still depends on human effort, and its cost remains unclear. The scoring dimensions, weights, and reference judgments rely on expert design, multiple rounds of human annotation, and adjudication; however, the paper does not quantify the initial investment or the recalibration cost when updates are needed. Its sustainable scalability is therefore difficult to assess. Moreover, the authors acknowledge that the standards are shaped by the preferences of particular user groups and experts. Even after calibration, agents remain noticeably weaker than humans on dimensions such as visual explanation, and the need for human review cannot be ignored.

  2. The approach applies upfront filtering before reconstructing environments. While this improves evaluability, it limits coverage of real online requests. Table 4 excludes trivial edits, purely informational queries, and tasks that depend on indispensable but irreproducible authenticated services, remote systems, or internal platforms. Such requests are themselves realistic use cases. Although excluding them is operationally reasonable, it biases the benchmark toward tasks that can be restored and executed locally. The distribution-matching results in Table 6 apply only to the 300 candidate tasks before filtering and cannot directly establish that the final set of 129 tasks is equally representative.

  3. The incremental contribution of the core environment-reconstruction method over prior work remains unclear. Restoring real trajectories into executable contexts is the key technique supporting the evaluation pipeline. Section 2 discusses TerminalWorld, which reconstructs terminal tasks from public records, as well as TerminalTraj and TermiGen, which construct container environments for trajectory synthesis. However, the comparison largely remains at a conceptual level, such as task sources and evaluation approaches. The paper does not yet systematically explain the substantive differences between its method and prior approaches. Existing experiments mainly validate whether the proposed pipeline is usable; they do not directly compare it, under the same set of trajectories and comparable resource budgets, against adapted prior methods or reconstruction baselines.

Questions for authors
  1. How much human effort is required for benchmark construction and subsequent updates, respectively, in terms of annotation, calibration, and review? The paper should report personnel and time costs, and explain how the standards would be recalibrated when user populations or expert preferences change, so that the scalability of the pipeline and the reliability of its evaluation results can be assessed.

  2. How many tasks are excluded by each filtering criterion, and how does the final set of 129 tasks differ in distribution from the original coding traffic? The paper should report the sample-retention rate at each stage and analyze the scenarios not covered. It should also clarify whether the filtering biases the benchmark toward tasks that are easier to restore and execute, and how this affects the scope of applicability of the model-ranking and feedback-benefit conclusions.

  3. What are the key technical differences between this work and prior environment-reconstruction methods? The paper should add a methodological comparison or controlled baseline experiments to clarify the additional benefit of the proposed design beyond the combination of “environment reconstruction with LLM-as-judge” This would help further delineate the paper’s contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.