- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work constructs Trace‑MT, a multi‑turn dialogue benchmark inspired by software control‑flow testing. It builds dialogue traces via N‑gram coverage‑guided pruning and adopts dynamic simulation with observer logs to achieve both reproducibility and state‑dependent evaluation. Empirical results reveal that strong performance under simple dialogue conditions does not transfer to scenarios with stacked interference.
Critical strengths
- This work adapts coverage criteria from software testing to formalize the behavioral space of multi‑turn dialogue, overcomes key drawbacks of static scripted benchmarks and non‑reproducible purely dynamic evaluations.
- This work leverages N-gram pruning to maintain the number of target behavior combinations at a manageable scale.
Critical weaknesses
-
While extensive coverage efforts have been undertaken, the covered behavior combinations may not manifest in real‑world multi‑turn conversations. Specifically, the real‑world occurrence probability of higher‑difficulty combinations remains unclear, which may limit the practical impact of this benchmark for the broader research community.
-
Difficulty tiers are defined based on interference density within dialogue traces rather than the intrinsic difficulty of seed questions.
Questions for authors
-
Design choices including category merging during pruning and the selection of the 6‑gram window appear largely empirical. The validity of such pruning operations has not been sufficiently justified.
-
The figures suffer from small fonts and poor readability. Moreover, all prompts provided in the appendix are written in Chinese, which raises concerns regarding the generalizability of the benchmark.