- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RepetitionBench, a benchmark for evaluating inappropriate repetition in multi-turn conversational LLM agents, focusing on healthcare telephone conversations. The authors propose a taxonomy of 13 repetition types, including seven context-dependent semantic categories and six deterministic categories, together with corresponding automated verifiers.
The benchmark contains 6,892 adversarially selected conversation contexts, accompanied by RepetitionBench-Gold, a human-annotated dataset used to calibrate the subjective verifiers. The authors evaluate 13 frontier LLMs and demonstrate that semantic repetition remains prevalent even when surface-level repetition is relatively limited.
Additionally, the paper investigates using verifier-based rewards for reinforcement learning. A post-trained DeepSeek-V3.1 model achieves substantially lower verifier-measured repetition failure rates than the evaluated frontier models. The results suggest that the proposed verifiers can serve both as evaluation tools and as training signals for mitigating repetition.
Critical strengths
1. Fine-grained and practically relevant evaluation framework.
The paper addresses an important but relatively underexplored failure mode in multi-turn conversational agents. Its main contribution is the distinction between inappropriate semantic repetition and superficial lexical repetition. The 13-category taxonomy provides a more interpretable and diagnostic evaluation framework than conventional repetition metrics based on lexical overlap or similarity. This is particularly valuable in healthcare dialogue, where repetition may either hinder conversational progress or serve a legitimate purpose, such as clarification or confirmation.
2. Substantial benchmark construction and experimental validation.
The authors construct a relatively large adversarial benchmark with 6,892 contexts and use expert human annotations to calibrate the subjective verifiers. The evaluation across 13 frontier models provides useful empirical evidence that semantic repetition remains challenging. Furthermore, the reinforcement learning experiments demonstrate a potential practical application of the proposed framework beyond benchmarking. The additional ablation studies, decoding sensitivity analysis, and statistical comparisons strengthen the empirical contribution.
Critical weaknesses
1. Limited reliability of some subjective verifiers and potential evaluation bias.
Although the authors report relatively strong overall agreement between the verifiers and human annotations, the reliability varies substantially across repetition categories. For example, in Table 1, the held-out Cohen's kappa for phrase reuse (category C) is only 0.26 for GPT-5.5 and 0.46 for the fine-tuned Qwen3.5-27B verifier. Furthermore, the held-out verifier evaluation contains only 150 response-category items rather than 150 independent conversation contexts, which raises concerns about the robustness of category-level conclusions.
This issue is particularly important because the proposed verifiers are used both to construct the adversarial benchmark and to evaluate model performance. The reinforcement learning model is also optimized using verifier-derived rewards. Consequently, improvements in verifier-measured failure rates may not fully correspond to improvements in human-perceived conversational quality. Stronger independent human validation of the benchmark-level model rankings would increase confidence in the results.
2. The reduction in unnecessary repetition appears to come at the cost of appropriate repetition and potentially other dialogue capabilities.
The reward-weight ablation in Table 2 reveals an important trade-off that is not sufficiently addressed. Increasing the repetition reward weight from 0% to 30% reduces unnecessary repetition from 59.7 to 30.7, but the appropriate-repetition score also decreases from 58.4 to 49.4. In addition, clinical correctness for the 15% reward checkpoint is 37.2, compared with 48.0 for the original base model.
These results raise concerns about whether the training objective effectively distinguishes harmful repetition from contextually necessary repetition. This distinction is especially consequential in healthcare conversations, where appropriate confirmation and restatement may be important for communication quality and safety.
While the paper argues that repetition optimization does not substantially compromise other capabilities, the reported ablation results suggest a more nuanced trade-off. Additional analysis is needed to establish whether the learned behavior represents a genuine improvement in dialogue quality rather than a general suppression of repetition.
Questions for authors
Q1. Independent validation of verifier-based evaluation.
Given the relatively low agreement for certain subjective categories, particularly phrase reuse, can the authors provide additional human evaluation of responses generated by the benchmarked models? Specifically, it would be helpful to know whether the reported model rankings and the substantial improvement of the RL-trained model remain consistent when evaluated by independent human annotators rather than the proposed verifiers.
Q2. Trade-off between unnecessary and appropriate repetition.
Table 2 shows that stronger repetition penalties reduce both unnecessary and appropriate repetition. Could the authors explain why appropriate-repetition performance decreases and whether this reduction is statistically significant? Additionally, how should the lower clinical correctness of the RL checkpoints relative to the original base model be interpreted? Clarifying these results would help establish whether the proposed reward improves overall dialogue behavior.
Q3. Independence of the verifier training and evaluation splits.
Appendix F.3 describes a random split of 1,611 response-category items into 1,461 training and 150 validation items. Were these splits performed at the conversation level to ensure that related contexts or responses from the same conversation did not appear in both sets? If not, could the authors evaluate verifier performance using conversation-disjoint held-out data?
4 comments
Thank you for the detailed rebuttal. I appreciate the clarification regarding the conversation-disjoint split and the additional discussion of human annotation agreement.
However, my main concerns are only partially addressed. First, while the low human–human agreement helps contextualize the difficulty of phrase reuse detection, the lack of independent human validation of the model leaderboard remains an important limitation.
Second, the response does not directly address the decrease in appropriate-repetition performance (58.4 to 49.4) or its statistical significance. The comparison of clinical correctness between the 0% and 30% repetition-reward checkpoints is informative, but all reported RL checkpoints still underperform the original base model on clinical correctness. Therefore, the potential trade-off between reducing unnecessary repetition and preserving appropriate repetition and clinical quality remains insufficiently characterized.
Nevertheless, I recognize the value of the proposed taxonomy, benchmark construction, and extensive empirical evaluation. Given these contributions and the authors' clarifications, I will maintain my original rating of 3.
Thank you for the follow-up. We appreciate your recognition of our benchmark contributions and agree that our earlier response did not sufficiently address the concerns about independent human validation and the trade-offs associated with RL training. Since then, we have conducted an independent nurse evaluation of the benchmark rankings and additional statistical analyses of appropriate repetition and clinical correctness. We present these results below.
1. Independent human evaluation reproduces the benchmark ranking across three representative models
To validate our benchmark rankings independently of the LLM-based verifiers, we conducted an additional evaluation with two independent nurse annotators. We selected three models representing different levels of benchmark performance: our RL model, which has the lowest failure rate; Claude Haiku 4.5, whose Hard-split failure rate (53.35%) is closest to the average across the 14 benchmarked models (53.09%); and the DeepSeek-V3.1 base model, from which our RL model was trained and which has one of the highest failure rates.
Each model generated responses to the same 43 randomly sampled Hard-tier contexts (129 responses in total). The nurses evaluated the responses across all seven subjective repetition categories, with each response assigned to one annotator. Model identities were hidden, and response order was randomized within each context.
| Model | Independent human-labeled failure rate | GPT-5.5 verifier-labeled failure rate |
|---|---|---|
| RL model | 14.0% (6/43) | 14.0% (6/43) |
| Claude Haiku 4.5 | 53.5% (23/43) | 62.8% (27/43) |
| DeepSeek-V3.1 base model | 74.4% (32/43) | 86.0% (37/43) |
The independent nurse annotations reproduce the model ordering observed on the full Hard-split benchmark. On the 43 sampled contexts, the nurse-labeled subjective failure rates are 14.0%, 53.5%, and 74.4% for the RL model, Claude Haiku 4.5, and DeepSeek-V3.1 base model, respectively. This ordering is consistent with the full-benchmark results (15.94%, 50.54%, and 61.87%, respectively), despite differences in sample size and evaluation method. Also, all pairwise differences are statistically significant (exact McNemar test: base model vs. RL model, p < 0.001; Claude Haiku 4.5 vs. RL model, p < 0.001; base model vs. Claude Haiku 4.5, p = 0.01). The nurse annotations also show strong agreement with the GPT-5.5 verifier, matching on 92.2% of the 903 category-level judgments (129 responses × 7 categories; κ = 0.64). Together, these results provide independent human evidence that the relative performance of the three models is consistent with the benchmark rankings, rather than being determined solely by the LLM-based verifier.
Given the limited annotation time available during the discussion period, each response was evaluated by one nurse. If accepted, we plan to extend this validation to three annotators per response and report inter-annotator agreement.
2. Additional analysis clarifies the trade-offs between repetition reduction and other capabilities
We agree that our earlier response did not sufficiently address the decrease in appropriate repetition or the decline in clinical correctness relative to the base model. We therefore conducted additional statistical analyses to better characterize these trade-offs.
The table summarizes performance across the base model and three RL checkpoints. The 0%-reward checkpoint is trained with task-quality RL but without the repetition reward, providing a reference for assessing the additional effect of the repetition reward at 15% and 30%. For these checkpoints, we report the difference relative to the 0%-reward checkpoint in percentage points and the corresponding p-value.
| Metric | Base model | 0% reward | 15% reward | 30% reward |
|---|---|---|---|---|
| Appropriate repetition | 58.4% (45/77) | 58.4% (45/77) | 51.9% (40/77; −6.5, p = 0.52) | 49.4% (38/77; −9.1, p = 0.33) |
| Clinical correctness | 48.0% | 40.0% | 37.2% (−2.8, p = 0.16) | 42.4% (+2.4, p = 0.22) |
The two metrics were evaluated differently. Appropriate repetition was measured using pass/fail outcomes on 77 cases in a single run, and we report Fisher’s exact test for comparisons between checkpoints. Clinical correctness was evaluated over three runs per checkpoint, and we report Welch’s t-test on the run-level scores.
Appropriate repetition decreases as the repetition reward increases, although the differences are not statistically significant. The 30%-reward checkpoint scores 9.1 percentage points below the 0%-reward checkpoint (p = 0.33). While the difference is not statistically significant, the downward trend across both reward weights suggests a potential trade-off. One possible explanation is that our repetition reward penalizes inappropriate repetition but does not explicitly reward appropriate repetition. As a result, the model may learn to avoid repetition more broadly, including cases where it is appropriate. Explicitly rewarding appropriate repetition could help address this trade-off, which we will explore in future work.
Clinical correctness remains below the base model, but we do not observe a consistent additional decline from the repetition reward. Clinical correctness is already 40.0% at the 0%-reward checkpoint, compared with 48.0% for the original base model. Relative to the 0%-reward checkpoint, the score decreases by 2.8 percentage points at 15% (p = 0.16) and increases by 2.4 points at 30% (p = 0.22). Neither difference is statistically significant, and we do not observe a consistent decline as the repetition reward weight increases. These results suggest that the decline relative to the base model is not solely attributable to the repetition reward, although we cannot rule out negative effects or isolate the cause of the decline during task-quality RL training.
We would also like to clarify that the primary contributions of our paper are the dataset, repetition taxonomy, benchmark, and verifiers. The RL experiment is intended to demonstrate that the repetition signal captured by the benchmark can be optimized through training, rather than to establish a clinically equal or better policy. We recognize the importance of preserving appropriate repetition and clinical correctness while reducing unnecessary repetition. We will therefore revise the abstract and Section 4.2 to explicitly acknowledge these trade-offs and include the additional statistical comparisons in Table 2. Future work will investigate how to better balance these capabilities during RL training.
We hope these additional results help address your concerns, and we would appreciate it if you could reconsider your score.
We appreciate your positive assessment and thoughtful questions. We address each concern and identify the corresponding revisions.
Phrase reuse is also the hardest category for human annotators, and its effect on the leaderboard is limited.
We would like to point out that phrase reuse is also the most challenging category for human annotators. For category C, human–human agreement is κ = 0.23 despite 80.8% raw agreement, the lowest among all categories (Table 1). This reflects ambiguity between reusable conversational scaffolds and templates. Consequently, category C has the longest "ignore" list in Appendix O.3.3. However, the ANY-failure aggregation limits the influence of overlapping category C flags, as category C changes a response’s overall failure status only when no other verifier flags it. We agree that the 150 held-out items are response–category pairs (Table 1 caption), making category-specific κ estimates imprecise.
Regarding human validation, RepetitionBench-Gold provides expert human annotations for 8,736 items from nurse role-played conversations. Verifier–human agreement is κ = 0.64 for GPT-5.5 and κ = 0.77 for the fine-tuned verifier, compared with human–human κ = 0.49. The GPT-5.5 and human–human values are computed on the same full gold set, whereas the fine-tuned verifier's κ = 0.77 is computed on the balanced held-out set and uses a different label distribution. A direct human evaluation of the 13-model leaderboard was not conducted. We also acknowledge the reuse of verifiers across benchmark construction, evaluation, and reward optimization. We will expand the limitations in Section 6 and report per-category held-out item counts in Table 1.
The repetition reward shows no consistent negative effect on clinical correctness.
We appreciate the reviewer's concern regarding the potential trade-off between reducing repetition and maintaining clinical correctness. To better understand this relationship, Table 2 compares checkpoints trained with different repetition reward weights. Clinical correctness is 48.0% for the base model and 40.0% for the checkpoint trained with task-quality RL alone (0% repetition reward). With the addition of the repetition reward, clinical correctness is 37.2% at 15% and 42.4% at 30%. Notably, the 30% checkpoint achieves higher clinical correctness than the 0% checkpoint (42.4% vs. 40.0%), suggesting that reducing repetition does not necessarily come at the expense of clinical correctness. While the results vary across reward weights, they do not indicate a consistent negative relationship between repetition reduction and clinical correctness.
The training and validation sets are conversation-disjoint.
All items belonging to a given conversation are assigned to the same split. To make this clear, we will clarify the conversation-disjoint split in Appendix F.3 and report the number of conversations in each partition.
We hope these clarifications address your concerns, and thank you again for your constructive feedback.