- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Trace2Eval, the pipeline that constructs coding benchmarks from real-world interaction trajectories. Existing benchmarks highly rely on reference patches and hidden tests, but the authors argue that this static public tasks are limited to evaluate next-generation coding agent. They propose four properties that good benchmark should satisfy, authenticity, contamination resistance, dynamism, and measurability. The proposed method firstly select the trace, recover the context, execute the agent, and evaluate the artifact. Across 19 model-harness configurations, rubric-derived feedback improves artifact scores by 10.05 on average.
Critical strengths
-
The problem setting is well-motivated and important. Evaluation of agents on real user requests with recovered workspace effectively addresses narrow task distribution of existing benchmarks such as SWE-bench
-
The authors conducted a wide scope of experiments with 2 harnesses and 19 model-harness configurations. They also included ablation study, paired bootstrap CI, per task improvement and regression counts
Critical weaknesses
-
The LLM judge reliability is too limited. Agent-GT kappa in Table 13 is only 0.173, compared with 0.542 of Human-GT. Furthermore, for code quality, which is 35% of the total weight, the kappa is only 0.070. Therefore, the difference between model of Table 1 can be lied in noise.
-
The rubric-derived feedback is circular. The judge's score generate the feedback, and the same judge and rubric then score R2. Therefore, the gains may reflect optimizing toward the judge, not the real quality improvement.
Questions for authors
- GLM models serve as judge, feedback generator, and candidates. Do model rankings and R2 gains hold with a judge from a different model family?