- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TraceDance constructs behavior-specific agent benchmarks from real deployment traces. Given a natural-language behavior query, it selects or synthesizes an executable retrieval anchor, a confirmation criterion, a decision-point cut, and a grading rubric. Anchor-and-Confirm combines programmatic retrieval with LLM-based validation, while an Anchor Synthesis Loop revises specifications for new behaviors. Evaluated models generate one assistant turn from the recorded prefix without environment execution. Experiments use 252,557 sessions from two agent harnesses, producing 107 benchmarks with 4,125 instances and evaluating nine models.
Critical strengths
- Relevant problem and practical design. Task completion can conceal undesirable execution behavior. Reusing deployment prefixes enables targeted evaluation when private repositories, tools, or accounts make environment reconstruction impractical.
- Coherent construction framework. The separation of retrieval, confirmation, context construction, and grading is sensible. The action, failure, and claim frames clarify the decision being evaluated, and rubrics accommodate multiple appropriate responses.
Critical weaknesses
W1. Strict rubric failure is not equivalent to exhibiting the target behavior
The central measurement concern is that a response can avoid the undesirable behavior and still fail the rubric, as explicitly acknowledged in Appendix C.2 and Appendix F. Consequently, the 26.7% mean pass rate should not be interpreted as a corresponding rate of undesirable behavior.
W2. The construction method lacks a controlled comparison
The experiments establish feasibility but do not isolate the benefits of Anchor-and-Confirm or iterative specification synthesis. Table 23 reports averages of 552 LLM calls and 25.14 million input/output tokens per successful query, without a matched cost-quality baseline. These counts alone establish neither efficiency nor inefficiency. Confirmation of retrieved candidates also leaves retrieval coverage unmeasured.
Questions for authors
- Among failed responses, what fraction actually exhibits the target behavior, versus avoiding it without satisfying the preferred next-action rubric?
- What quality-cost advantage does the full construction pipeline provide over simpler alternatives under the same budget?