- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies behavioral evaluation in multi-turn LLM interactions. It introduces RHCA, a turn-level rubric covering reasoning transparency, helpfulness, consistency, and context alignment, and BTV, a trajectory-level metric that penalizes low scores and declines across turns. Evaluation includes 13 music-programming scenarios, controlled late-failure trajectories based on 16 practitioner-defined tasks, and selected real-model conversations. The results show discrepancies between LLM judges and human ratings, and cases where BTV assigns failing scores to trajectories that pass under arithmetic averaging.
Critical strengths
- The paper addresses an important evaluation problem: task success alone may overlook behavioral failures in multi-turn LLM interactions, motivating assessment across multiple dimensions.
- The RHCA dimensions are reasonable, and BTV is motivated by the concern that averaging can mask severe failures and quality declines across turns.
- The experiments illustrate applications of RHCA and demonstrate how BTV captures failures masked by averaging in both controlled trajectories and real-model conversations.
Critical weaknesses
The distinction from prior evaluation methods needs clarification.
Prior work has motivated fine-grained evaluation of multi-turn dialogues [a,b], including minimum-based aggregation [a] and assessment across multiple quality dimensions [b]. The paper contributes a literature- and interaction-informed RHCA rubric and a trajectory metric combining sensitivity to low scores with a decline penalty. However, it should clarify what these choices add beyond existing evaluation frameworks and aggregation methods.
The assumptions underlying the decline penalty are unclear.
Later turns may receive lower scores simply because the user asks harder questions or introduces a new subtask. It is unclear why such drops should receive an additional penalty. BTV also gives the same score of 0.8 to [3, 1, 3] and [3, 3, 1], although the first interaction recovers from its error and the second ends in failure. The paper should explain when this penalty is appropriate and whether recovery should affect the final score.
The validation of BTV is limited.
Figure 4 and Table 3 use constructed final-turn failures that closely match Proposition 1. These results do not show how often BTV incorrectly rejects acceptable interactions. Comparison with mean- and minimum-based aggregation against independent human judgments of whole conversations would better establish its value.
[a] Bai et al. “MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues.” ACL, 2024.
[b] Zhang et al. “FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation.” EMNLP, 2022.
Questions for authors
See weaknesses
3 comments
Thank you for the reply. Here are my thoughts:
- [Partially addressed] The paper’s focus on workflows and its approach to trajectory aggregation are clearer. However, multidimensional and dialogue-level evaluation still overlap with prior work, and MT-Bench-101 also covers multi-turn reasoning.
- [Partially addressed] I understand the intent to penalize severe failures, but it remains unclear when an additional decline penalty is appropriate, particularly across subtask changes or after successful recovery.
- [Partially addressed] The added comparisons and independent assessment of one case are helpful. However, validation against independent whole-conversation judgments, including successful and recovered interactions, is still needed.
I therefore maintain my current rating.
Thank you for the further comments.
- Multidimensional and dialogue-level evaluation are established, and we build on them. Our additions are practitioner-defined criteria for professional workflows and a trajectory rule that responds to both low scores and declines (Section 2). MT-Bench-101’s minimum rule ignores repeated low scores and trends, and in Appendix Table 12 it passes the Gemini 3 Flash trajectory that BTV flags.
- The decline penalty targets drops in response quality, such as losing earlier context or constraints; a well-handled new subtask triggers none. In professional workflows, users may act on an incorrect intermediate answer before it is corrected, so BTV does not let later recovery cancel a severe failure. For trial-and-error tasks such as debugging, we plan recovery-aware variants (Section 6.2).
- Whether BTV’s flags match expert judgments of whole conversations is the next test: we plan to collect expert whole-conversation labels and report BTV’s agreement with them at 2.0 and nearby thresholds, and how often BTV fails conversations that experts label successful (Section 6.2), with recovered interactions included.
Thank you for your review. Our replies follow.
W1. We revised Section 2 to clarify two main differences from MT-Bench-101 (Bai et al., 2024) and FineD-Eval (Zhang et al., 2022). (a) Evaluation focus: these methods assess general dialogue quality, while RHCA uses criteria grounded in learning and professional workflows. (b) Trajectory scoring: MT-Bench-101 takes the minimum turn score, overlooking repeated low scores and score trends. FineD-Eval combines dimension-level sub-metrics for the whole dialogue, without tracking how quality changes across turns. BTV accounts for low-scoring turns and drops from earlier peaks.
W2. Our 1-3 rubric scores response quality, not task difficulty: accurate, appropriate responses to harder questions or new subtasks still score 2 or 3. The decline penalty targets drops in response quality, such as losing a constraint or context that the model handled earlier. A score of 1 marks a serious failure, such as forgotten constraints, hallucinations, or misleading guidance, whose costs remain even after recovery. Averaging passes [3, 1, 3] at 2.33, while BTV's strict default flags it. However, recovery may deserve more credit in trial-and-error tasks such as debugging or brainstorming. We plan to explore recovery-aware variants (Section 6.2).
W3. We agree that these failures are constructed and that their detection follows from BTV's definition. We have softened the claim and revised the Table 3 caption and the discussion of Figure 4 accordingly. We also assessed one flagged case, Gemini 3 Flash (Table 4): a domain evaluator who did not see scores, BTV values, or model names reviewed the full transcript and judged it a failure, consistent with BTV. In Setting 1 (Table 9), BTV also reverses the mean's preference for Model A in Scenario S3, matching the annotators' preference for Model B. In Appendix Table 12, we also add a comparison of BTV with other aggregators on the four real-model trajectories. Assessing the remaining flagged cases and measuring how often BTV rejects acceptable interactions, both against independent task-success labels, are in our plans (Section 6.2).