acceptodds
Under review as a conference paper at ICLR 2027

RepetitionBench: A Conversational Benchmark for Multi-Turn Agent Repetition

Abstract

Repetition is a pervasive failure mode in conversational Large Language Model (LLM) agents, manifesting as re-asked questions, redundant confirmations, and unnecessary restatements. Such behavior disrupts the natural flow of the dialogue and erodes user trust. Yet repetition remains under-measured, and existing benchmarks do not stress-test this behavior in realistic, conversational settings. To address this gap, we introduce RepetitionBench, a benchmark to quantify and classify different repetition modes in phone-based healthcare conversations between humans and AI agents. RepetitionBench comprises three components: (i) a fine-grained taxonomy spanning semantic, structural, and surface-level dimensions of repetition; (ii) 13 verifiers (6 deterministic and 7 subjective) to judge the repetition of an LLM agent; and (iii) an adversarial dataset of 6,892 samples drawn from healthcare conversations averaging over 20 turns. We publish a total of 7,516 labeled samples, including 18,372 human annotations of repetition types and 8,736 verifier outputs. Across 13 frontier models, surface-level repetition is rare, but semantic repetition remains common, especially in longer conversations. RepetitionBench is the first work to enable a systematic study of repetition in multi-turn conversational LLM agents, providing a basis for scalable evaluation prior to real-world deployment, while the verifiers in this work can be used to evaluate and analyze different modes of repetition.

Primary area:
datasets and benchmarks
Keywords:
Conversational AI, Repetition Detection, Multi-turn Dialogue Evaluation, Adversarial Dataset, Healthcare AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6eukOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RepetitionBench, a benchmark for evaluating inappropriate repetition in multi-turn conversational LLM agents, focusing on healthcare telephone conversations. The authors propose a taxonomy of 13 repetition types, including seven context-dependent semantic categories and six deterministic categories, together with corresponding automated verifiers.

The benchmark contains 6,892 adversarially selected conversation contexts, accompanied by RepetitionBench-Gold, a human-annotated dataset used to calibrate the subjective verifiers. The authors evaluate 13 frontier LLMs and demonstrate that semantic repetition remains prevalent even when surface-level repetition is relatively limited.

Additionally, the paper investigates using verifier-based rewards for reinforcement learning. A post-trained DeepSeek-V3.1 model achieves substantially lower verifier-measured repetition failure rates than the evaluated frontier models. The results suggest that the proposed verifiers can serve both as evaluation tools and as training signals for mitigating repetition.

Critical strengths

1. Fine-grained and practically relevant evaluation framework.

The paper addresses an important but relatively underexplored failure mode in multi-turn conversational agents. Its main contribution is the distinction between inappropriate semantic repetition and superficial lexical repetition. The 13-category taxonomy provides a more interpretable and diagnostic evaluation framework than conventional repetition metrics based on lexical overlap or similarity. This is particularly valuable in healthcare dialogue, where repetition may either hinder conversational progress or serve a legitimate purpose, such as clarification or confirmation.

2. Substantial benchmark construction and experimental validation.

The authors construct a relatively large adversarial benchmark with 6,892 contexts and use expert human annotations to calibrate the subjective verifiers. The evaluation across 13 frontier models provides useful empirical evidence that semantic repetition remains challenging. Furthermore, the reinforcement learning experiments demonstrate a potential practical application of the proposed framework beyond benchmarking. The additional ablation studies, decoding sensitivity analysis, and statistical comparisons strengthen the empirical contribution.

Critical weaknesses

1. Limited reliability of some subjective verifiers and potential evaluation bias.

Although the authors report relatively strong overall agreement between the verifiers and human annotations, the reliability varies substantially across repetition categories. For example, in Table 1, the held-out Cohen's kappa for phrase reuse (category C) is only 0.26 for GPT-5.5 and 0.46 for the fine-tuned Qwen3.5-27B verifier. Furthermore, the held-out verifier evaluation contains only 150 response-category items rather than 150 independent conversation contexts, which raises concerns about the robustness of category-level conclusions.

This issue is particularly important because the proposed verifiers are used both to construct the adversarial benchmark and to evaluate model performance. The reinforcement learning model is also optimized using verifier-derived rewards. Consequently, improvements in verifier-measured failure rates may not fully correspond to improvements in human-perceived conversational quality. Stronger independent human validation of the benchmark-level model rankings would increase confidence in the results.

2. The reduction in unnecessary repetition appears to come at the cost of appropriate repetition and potentially other dialogue capabilities.

The reward-weight ablation in Table 2 reveals an important trade-off that is not sufficiently addressed. Increasing the repetition reward weight from 0% to 30% reduces unnecessary repetition from 59.7 to 30.7, but the appropriate-repetition score also decreases from 58.4 to 49.4. In addition, clinical correctness for the 15% reward checkpoint is 37.2, compared with 48.0 for the original base model.

These results raise concerns about whether the training objective effectively distinguishes harmful repetition from contextually necessary repetition. This distinction is especially consequential in healthcare conversations, where appropriate confirmation and restatement may be important for communication quality and safety.

While the paper argues that repetition optimization does not substantially compromise other capabilities, the reported ablation results suggest a more nuanced trade-off. Additional analysis is needed to establish whether the learned behavior represents a genuine improvement in dialogue quality rather than a general suppression of repetition.

Questions for authors

Q1. Independent validation of verifier-based evaluation.

Given the relatively low agreement for certain subjective categories, particularly phrase reuse, can the authors provide additional human evaluation of responses generated by the benchmarked models? Specifically, it would be helpful to know whether the reported model rankings and the substantial improvement of the RL-trained model remain consistent when evaluated by independent human annotators rather than the proposed verifiers.

Q2. Trade-off between unnecessary and appropriate repetition.

Table 2 shows that stronger repetition penalties reduce both unnecessary and appropriate repetition. Could the authors explain why appropriate-repetition performance decreases and whether this reduction is statistically significant? Additionally, how should the lower clinical correctness of the RL checkpoints relative to the original base model be interpreted? Clarifying these results would help establish whether the proposed reward improves overall dialogue behavior.

Q3. Independence of the verifier training and evaluation splits.

Appendix F.3 describes a random split of 1,611 response-category items into 1,461 training and 150 validation items. Were these splits performed at the conversation level to ensure that related contexts or responses from the same conversation did not appear in both sets? If not, could the authors evaluate verifier performance using conversation-disjoint held-out data?

4 comments

We appreciate your positive assessment and thoughtful questions. We address each concern and identify the corresponding revisions.

Phrase reuse is also the hardest category for human annotators, and its effect on the leaderboard is limited.

We would like to point out that phrase reuse is also the most challenging category for human annotators. For category C, human–human agreement is κ = 0.23 despite 80.8% raw agreement, the lowest among all categories (Table 1). This reflects ambiguity between reusable conversational scaffolds and templates. Consequently, category C has the longest "ignore" list in Appendix O.3.3. However, the ANY-failure aggregation limits the influence of overlapping category C flags, as category C changes a response’s overall failure status only when no other verifier flags it. We agree that the 150 held-out items are response–category pairs (Table 1 caption), making category-specific κ estimates imprecise.

Regarding human validation, RepetitionBench-Gold provides expert human annotations for 8,736 items from nurse role-played conversations. Verifier–human agreement is κ = 0.64 for GPT-5.5 and κ = 0.77 for the fine-tuned verifier, compared with human–human κ = 0.49. The GPT-5.5 and human–human values are computed on the same full gold set, whereas the fine-tuned verifier's κ = 0.77 is computed on the balanced held-out set and uses a different label distribution. A direct human evaluation of the 13-model leaderboard was not conducted. We also acknowledge the reuse of verifiers across benchmark construction, evaluation, and reward optimization. We will expand the limitations in Section 6 and report per-category held-out item counts in Table 1.

The repetition reward shows no consistent negative effect on clinical correctness.

We appreciate the reviewer's concern regarding the potential trade-off between reducing repetition and maintaining clinical correctness. To better understand this relationship, Table 2 compares checkpoints trained with different repetition reward weights. Clinical correctness is 48.0% for the base model and 40.0% for the checkpoint trained with task-quality RL alone (0% repetition reward). With the addition of the repetition reward, clinical correctness is 37.2% at 15% and 42.4% at 30%. Notably, the 30% checkpoint achieves higher clinical correctness than the 0% checkpoint (42.4% vs. 40.0%), suggesting that reducing repetition does not necessarily come at the expense of clinical correctness. While the results vary across reward weights, they do not indicate a consistent negative relationship between repetition reduction and clinical correctness.

The training and validation sets are conversation-disjoint.

All items belonging to a given conversation are assigned to the same split. To make this clear, we will clarify the conversation-disjoint split in Appendix F.3 and report the number of conversations in each partition.

We hope these clarifications address your concerns, and thank you again for your constructive feedback.

Thank you for the detailed rebuttal. I appreciate the clarification regarding the conversation-disjoint split and the additional discussion of human annotation agreement.

However, my main concerns are only partially addressed. First, while the low human–human agreement helps contextualize the difficulty of phrase reuse detection, the lack of independent human validation of the model leaderboard remains an important limitation.

Second, the response does not directly address the decrease in appropriate-repetition performance (58.4 to 49.4) or its statistical significance. The comparison of clinical correctness between the 0% and 30% repetition-reward checkpoints is informative, but all reported RL checkpoints still underperform the original base model on clinical correctness. Therefore, the potential trade-off between reducing unnecessary repetition and preserving appropriate repetition and clinical quality remains insufficiently characterized.

Nevertheless, I recognize the value of the proposed taxonomy, benchmark construction, and extensive empirical evaluation. Given these contributions and the authors' clarifications, I will maintain my original rating of 3.

Thank you for the follow-up. We appreciate your recognition of our benchmark contributions and agree that our earlier response did not sufficiently address the concerns about independent human validation and the trade-offs associated with RL training. Since then, we have conducted an independent nurse evaluation of the benchmark rankings and additional statistical analyses of appropriate repetition and clinical correctness. We present these results below.

1. Independent human evaluation reproduces the benchmark ranking across three representative models

To validate our benchmark rankings independently of the LLM-based verifiers, we conducted an additional evaluation with two independent nurse annotators. We selected three models representing different levels of benchmark performance: our RL model, which has the lowest failure rate; Claude Haiku 4.5, whose Hard-split failure rate (53.35%) is closest to the average across the 14 benchmarked models (53.09%); and the DeepSeek-V3.1 base model, from which our RL model was trained and which has one of the highest failure rates.

Each model generated responses to the same 43 randomly sampled Hard-tier contexts (129 responses in total). The nurses evaluated the responses across all seven subjective repetition categories, with each response assigned to one annotator. Model identities were hidden, and response order was randomized within each context.

ModelIndependent human-labeled failure rateGPT-5.5 verifier-labeled failure rate
RL model14.0% (6/43)14.0% (6/43)
Claude Haiku 4.553.5% (23/43)62.8% (27/43)
DeepSeek-V3.1 base model74.4% (32/43)86.0% (37/43)

The independent nurse annotations reproduce the model ordering observed on the full Hard-split benchmark. On the 43 sampled contexts, the nurse-labeled subjective failure rates are 14.0%, 53.5%, and 74.4% for the RL model, Claude Haiku 4.5, and DeepSeek-V3.1 base model, respectively. This ordering is consistent with the full-benchmark results (15.94%, 50.54%, and 61.87%, respectively), despite differences in sample size and evaluation method. Also, all pairwise differences are statistically significant (exact McNemar test: base model vs. RL model, p < 0.001; Claude Haiku 4.5 vs. RL model, p < 0.001; base model vs. Claude Haiku 4.5, p = 0.01). The nurse annotations also show strong agreement with the GPT-5.5 verifier, matching on 92.2% of the 903 category-level judgments (129 responses × 7 categories; κ = 0.64). Together, these results provide independent human evidence that the relative performance of the three models is consistent with the benchmark rankings, rather than being determined solely by the LLM-based verifier.

Given the limited annotation time available during the discussion period, each response was evaluated by one nurse. If accepted, we plan to extend this validation to three annotators per response and report inter-annotator agreement.

2. Additional analysis clarifies the trade-offs between repetition reduction and other capabilities

We agree that our earlier response did not sufficiently address the decrease in appropriate repetition or the decline in clinical correctness relative to the base model. We therefore conducted additional statistical analyses to better characterize these trade-offs.

The table summarizes performance across the base model and three RL checkpoints. The 0%-reward checkpoint is trained with task-quality RL but without the repetition reward, providing a reference for assessing the additional effect of the repetition reward at 15% and 30%. For these checkpoints, we report the difference relative to the 0%-reward checkpoint in percentage points and the corresponding p-value.

MetricBase model0% reward15% reward30% reward
Appropriate repetition58.4% (45/77)58.4% (45/77)51.9% (40/77; −6.5, p = 0.52)49.4% (38/77; −9.1, p = 0.33)
Clinical correctness48.0%40.0%37.2% (−2.8, p = 0.16)42.4% (+2.4, p = 0.22)

The two metrics were evaluated differently. Appropriate repetition was measured using pass/fail outcomes on 77 cases in a single run, and we report Fisher’s exact test for comparisons between checkpoints. Clinical correctness was evaluated over three runs per checkpoint, and we report Welch’s t-test on the run-level scores.

Appropriate repetition decreases as the repetition reward increases, although the differences are not statistically significant. The 30%-reward checkpoint scores 9.1 percentage points below the 0%-reward checkpoint (p = 0.33). While the difference is not statistically significant, the downward trend across both reward weights suggests a potential trade-off. One possible explanation is that our repetition reward penalizes inappropriate repetition but does not explicitly reward appropriate repetition. As a result, the model may learn to avoid repetition more broadly, including cases where it is appropriate. Explicitly rewarding appropriate repetition could help address this trade-off, which we will explore in future work.

Clinical correctness remains below the base model, but we do not observe a consistent additional decline from the repetition reward. Clinical correctness is already 40.0% at the 0%-reward checkpoint, compared with 48.0% for the original base model. Relative to the 0%-reward checkpoint, the score decreases by 2.8 percentage points at 15% (p = 0.16) and increases by 2.4 points at 30% (p = 0.22). Neither difference is statistically significant, and we do not observe a consistent decline as the repetition reward weight increases. These results suggest that the decline relative to the base model is not solely attributable to the repetition reward, although we cannot rule out negative effects or isolate the cause of the decline during task-quality RL training.

We would also like to clarify that the primary contributions of our paper are the dataset, repetition taxonomy, benchmark, and verifiers. The RL experiment is intended to demonstrate that the repetition signal captured by the benchmark can be optimized through training, rather than to establish a clinically equal or better policy. We recognize the importance of preserving appropriate repetition and clinical correctness while reducing unnecessary repetition. We will therefore revise the abstract and Section 4.2 to explicitly acknowledge these trade-offs and include the additional statistical comparisons in Table 2. Future work will investigate how to better balance these capabilities during RL training.

We hope these additional results help address your concerns, and we would appreciate it if you could reconsider your score.

Reviewer_fjUDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RepetitionBench, a benchmark for inappropriate repetition in long, phone-based healthcare conversations between nurses role-playing patients and an AI agent. It contributes (i) a 13-category taxonomy (7 "subjective" categories judged by GPT-5.5 prompts, 6 "deterministic" string rules; Figure 5, Appendix O); (ii) RepetitionBench-Gold, 624 contexts × 2 responses × 7 categories = 8,736 items labelled by two annotators plus adjudication (18,372 labels), used for verifier–human agreement (Table 1) and to fine-tune a Qwen3.5-27B verifier; (iii) 6,892 adversarially selected contexts (hard 2,203 / easy 4,689) where GPT-4o and/or DeepSeek-V3.1 triggered a verifier; and (iv) ANY-failure rates for 13 frontier models (Figure 4a, Table 11, Tables 12–14). Section 4.2 post-trains DeepSeek-V3.1 with GRPO using the verifiers in the reward and reports the lowest failure rate among 14 models, a reward-weight ablation (Table 2), decoding-sensitivity runs (Figure 7) and a SODA-Eval check (Table 15). No code or data were submitted; a Kaggle release is promised without a link.

Critical strengths
  1. A large, transparently documented human-labelled resource. The Gold set (8,736 items, two annotators plus adjudication, per-category κ in Table 1), the full annotation protocol with edge cases and overrides (Appendix D), the complete judge prompts and rule pseudo-code (Appendix O), and per-verifier fire counts (Table 7, which reconciles row- and column-wise) are documentation benchmark papers often lack. This matters because the verifiers, not the leaderboard, are the reusable contribution.
  2. Uncertainty is reported properly. Paired-bootstrap CIs and Holm-adjusted McNemar tests for all 91 pairs (Tables 12–13) and rank-stability intervals (Table 14) go beyond typical leaderboards; the Figure 4a arithmetic (means, the 1.7–2.2× and 21–35% statements) reproduces.
Critical weaknesses
  1. The "semantic repetition" narrative is contradicted by the verifiers' own operational definitions. The introduction says models "still struggle with semantic repetition: they often vary their wording without meaningfully varying the content" (p.2), and §2.2 says verifier D "assesses whether they are semantically equivalent", giving a paraphrased medication question as a FAIL example (p.4). The D prompt (Appendix O.3.4, p.37) instead rules: "A question is NOT duplicative (PASS) if … It uses substantially different wording or sentence structure, even if the underlying intent or meaning is the same. Same meaning is NOT duplicative", listing "Is now a good time to talk?" vs "Do you have time to chat now?" as PASS; the E/G prompts state "Semantic similarity alone is NOT repetition" (p.43). The human protocol D3 (p.20) does contain a semantic clause ("a re-ask for information already requested … and already provided by the patient"), so the LLM verifiers and the gold labels measure different things. As written, the benchmark measures near-verbatim structural reuse, which undercuts the main conclusion and the "beyond surface-level" positioning (Table 3).
  2. The post-training result (§4.2) cannot be interpreted from what is reported, and Table 2 contradicts the text. (a) The hard split is by construction the set of contexts where one DeepSeek-V3.1 sample was flagged (p.6), yet neither the V3.1 base nor the 0%-reward checkpoint appears in Figure 4a/Table 11, so the RL model's 17.75% cannot be separated from regression to the mean. (b) The policy-RL setup has no appendix: Appendix G ("…details for verifier training") describes GRPO on the Qwen verifier, while the training prompts, their overlap with the 2,723 benchmark conversations, the "85% task-level dialogue quality" judge, and whether the 15% repetition reward uses the same GPT-5.5 judge that scores the benchmark are unspecified. (c) Table 2 shows "Clinical correctness" falling from 48.0 (base) to 37.2 (15%, the model in Figure 4) and "Appropriate rep." from 58.4 to 49.4, but the text claims "without reducing empathy (90.8%), clinical confirmation (42.4%), or task adherence (80.5%)" (there is no "clinical confirmation" row) and never mentions the appropriate-repetition drop, which concerns the paper's stated goal of preserving helpful repetition; Table 2's metrics are nowhere defined. (d) The deterministic verifiers used as a zero-reward trigger (J, L, H) have no exemption for a patient asking to repeat (O.4.3–O.4.6, p.44–45; Table 10 exempts only "All seven LLM verifiers"), contradicting Figure 1(a). Together these leave the "lowest failure rates among 14 models" claim unverifiable and the "no cost to other capabilities" claim inaccurate on the paper's own table, which most determines my recommendation.
Questions for authors
  1. Do verifiers D/E/F/G detect semantic equivalence (as §2.2 states) or only same-structure trivial edits (as the prompts state)? Under O.3.4, is the §2.2 medication example PASS or FAIL? How were the human D3 labels reconciled with the D prompt?
  2. Please report the ANY-failure rates (easy/hard, several samples) of the DeepSeek-V3.1 base and the 0%-reward checkpoint on RepetitionBench, and describe the policy-RL data (source, size, overlap with benchmark/Gold conversations), the reward judge, and whether GPT-5.5 was used in the reward.
  3. What are the Table 2 metrics (definition, evaluation set, judge, N)? How do you interpret the drops in appropriate repetition and clinical correctness?
  4. Is the "held-out eval set (N=150)" in Table 1 the same 150 "validation examples" used for checkpoint selection (F.3, G.4)? What are the per-category N (the accuracies imply B≈6 items) and confidence intervals for κ? On which set was the Human IRR column computed, given that κ=0.24 at 98.6% agreement cannot arise on a PASS/FAIL-balanced subset?
  5. Figure 6 labels the fourth model "GPT-4.1m" while the text says GPT-5.4 mini; which model was evaluated?
5 comments

We thank the reviewer for the careful reading of the appendices and for checking our definitions and tables against the text. We agree with several of the points raised and describe the corresponding changes regarding the semantic repetition definition and post-training result details.

The subjective verifiers capture reworded repetition beyond exact matching, and we will replace the term "semantic repetition."

The subjective verifiers D–G do go beyond exact matching, which is why we contrast them with the six deterministic string-matching verifiers. They capture questions or statements that repeat an earlier one in reworded form, not merely verbatim. In Figure 1(c), the agent asks, "Anything else you'd like to share about the discount offer?" and, after the patient declines, asks, "Is there anything else you'd like to share about your experience with the discount offer?" No exact-match rule captures this, but it is a FAIL under the verifier D prompt (Appendix O.3.4), which fails a question that "keeps the SAME sentence structure as the baseline with only minor changes."

Repetition of meaning alone is not sufficient to mark a repetition, because in clinical conversations re-asking is often appropriate, such as confirming a medication (Appendix O.3.3). The verifiers therefore also judge whether a repeated question "serves a clearly different purpose," which no string rule can assess.

Human rule D3 shares this core criterion, failing "an identical or trivially reworded question." It also fails a re-ask for information the patient already provided, which is broader than the prompt, but the two rarely diverge: verifier agreement with human labels on category D is κ = 0.73 and 0.85 (Table 1).

We appreciate the reviewer highlighting this ambiguity. We will adopt the term "near-paraphrase repetition with context-dependent exemptions," update Table 10, replace the Section 2.2 medication example, which would receive a PASS under the same prompt because it uses "substantially different wording or sentence structure," with the Figure 1(c) example, and note the difference between rule D3 and the D prompt.

We evaluated the base model and the 0%-reward checkpoint on RepetitionBench, and the improvement of the RL model holds.

We agree these two rows are needed to interpret the result, and we will add them to Figure 4a and Table 11. We report them here, rerunning the base model and the 0%-reward checkpoint on the same easy (E) and hard (H) tier contexts (ANY-failure, %):

ModelSubjective E / HDeterministic E / HBoth E / H
DeepSeek-V3.1 (base)38.13 / 61.8723.14 / 42.7640.93 / 67.18
0% reward43.57 / 66.7725.44 / 45.8547.52 / 72.49
RL model11.76 / 15.941.28 / 2.9912.55 / 17.75

We appreciate the reviewer raising this concern and agree that the benchmark construction process should be considered when interpreting the results. The hard split consists of contexts where both probe models (GPT-4o and DeepSeek-V3.1) produced responses flagged for repetition during benchmark construction (Section 3.3). Consequently, the original responses are flagged by construction, making a fresh evaluation of the base model the appropriate baseline. When evaluated on these same contexts, the base model exhibits a repetition rate of 67.18%, compared to 17.75% for the RL model. Since both models generate new responses to the same contexts, this comparison accounts for the context selection process and demonstrates the improvement attributable to RL training.

Furthermore, the 0%-repetition-reward checkpoint, trained solely with the task-quality reward, exhibits a repetition rate of 72.49%, indicating that task-quality RL alone does not reduce repetition. This is consistent with Table 2, where the unnecessary-repetition rates are 58.6% for the base model and 59.7% for the 0%-reward checkpoint.

We will add an RL post-training appendix covering the rewards, the training data, and the GRPO configuration.

We agree that the RL post-training setup is under-documented: Appendix G covers only verifier training. We will add an RL post-training appendix covering the four items the reviewer lists: the repetition reward and its judge, the task-quality reward, the training conversations and their overlap with the benchmark, and the GRPO configuration.

  1. Repetition reward (15%). The reward uses all 13 verifiers (six deterministic and seven subjective), with GPT-5.5 as the subjective judge. This is the same judge used to score the benchmark. Three results help address the potential bias. First, the six deterministic verifiers are judge-independent, and the RL model achieves deterministic ANY-failure rates of 1.28%/2.99% on Easy/Hard, compared with 9.27–20.24% and 20.61–41.62% for frontier models (Figure 4a). Second, the hard split is adversarial to the RL model's own base model, DeepSeek-V3.1 (Section 3.3). Third, we will re-score the full leaderboard using the fine-tuned Qwen3.5-27B verifier instead of GPT-5.5. The Qwen verifier was trained only on RepetitionBench-Gold (465 conversations), which is conversation-disjoint from the 2,723 benchmark conversations. This provides an evaluation independent of the reward judge. Independent human validation of the RL-versus-base-model gap is discussed in our response to Reviewer 6euk.
  2. Task-quality reward (85%). We will clarify in the manuscript that empathy, clinical correctness, and task adherence in Table 2 are evaluated by this same GPT-5.5-based judge. This is relevant to the concern about Table 2, as the decline in clinical correctness from the base model to the 0% checkpoint occurs under task-quality RL alone, before introducing the repetition reward.
  3. Training prompts and overlap. We will report a call_id-level audit of overlap between the policy training conversations and both the 2,723 benchmark conversations and the 465 gold conversations.
  4. GRPO configuration and compute. We will provide the full policy-training configuration, including group size, clipping, KL coefficient, learning rate and schedule, batch and mini-batch sizes, and number of training steps, following the format of Appendices G.1–G.3.

Most of the decline in clinical correctness occurs without the repetition reward.

We appreciate the reviewer highlighting these concerns. We agree that the terminology in Section 4.2 could be clearer: "clinical confirmation" should refer to "clinical correctness." We also acknowledge that the reduction in appropriate repetition is an important trade-off that deserves discussion (please see our response to Reviewer 6euk).

We would like to clarify that the decrease in clinical correctness from 48.0% to 37.2% compares the base model with the 15%-repetition-reward checkpoint. However, these models differ in both task-quality RL training and the addition of the repetition reward. To distinguish these effects, Table 2 includes a 0%-repetition-reward checkpoint trained with task-quality RL alone. Comparing the base model with this 0% checkpoint, clinical correctness decreases from 48.0% to 40.0%, indicating that much of the initial decline occurs even without the repetition reward. With the addition of the repetition reward, clinical correctness is 37.2% at 15% reward weight and 42.4% at 30% reward weight, showing no consistent decline as the repetition reward increases.

The concern specifically refers to the comparison between the 0% and 30% checkpoints, where empathy remains unchanged (90.8% vs. 90.8%), clinical correctness improves (40.0% vs. 42.4%), and task adherence remains comparable (80.0% vs. 80.5%). These results suggest that the repetition reward does not consistently degrade these quality metrics, although the reduction in appropriate repetition remains an important trade-off. We will report this trade-off in the limitation.

The repetition reward uses all 13 verifiers, with GPT-5.5 as the judge for the seven subjective verifiers.

We appreciate the reviewer raising this question and would like to clarify the reward composition. The repetition reward uses all 13 verifiers, including six deterministic verifiers that do not rely on an LLM judge and seven subjective verifiers evaluated by GPT-5.5. We acknowledge that using the same subjective judge for training and evaluation introduces a potential source of bias. However, the deterministic verifiers provide judge-independent measurements, and the hard split is constructed adversarially against the RL model's base model. The remaining 85% of the reward comes from a proprietary production judge that evaluates clinical correctness, empathy, and task adherence, without explicitly assessing repetition. We will provide further details on the reward judges, training-prompt sources, and GRPO configuration in a policy-RL appendix.

Validation and evaluation overlap. We confirm that the 150 items evaluated in Appendix F.3 were also used for checkpoint selection in Appendix G.4. This overlap does not affect GPT-5.5, which is evaluated zero-shot without checkpoint selection.

Per-category sample sizes and uncertainty. We agree with the reviewer's observation regarding the small per-category sample sizes. The 150 evaluation items represent response–category pairs and were randomly sampled from the balanced pool, resulting in an uneven distribution across categories (A: 6, B: 6, C: 26, D: 27, E: 23, F: 37, G: 25). Given these sample sizes, particularly for the hard categories A and B, the per-category κ values should be interpreted cautiously, while the overall κ provides a more stable summary of agreement. We also clarify that the ± values in Table 1 represent standard deviations across three verifier runs on the same items, reflecting run-to-run variability rather than sampling uncertainty.

Human IRR vs. verifier agreement. Human inter-rater reliability (IRR) and GPT-5.5 agreement are both computed on the full gold set of 8,736 items, whereas the fine-tuned verifier is evaluated on the smaller balanced subset. The full gold set is used because both human annotators labeled every item, while the balanced subset of 1,461 items was constructed using adjudicated labels for fine-tuning. We acknowledge that the N = 1,461 header in Table 1 was incorrect for the full-set columns. For example, in category A, only 9 of 1,248 items are labeled FAIL, resulting in κ = 0.24 despite 98.6% raw agreement due to the strong label imbalance.

Importantly, the comparison between the verifiers and Human IRR (κ = 0.49) is based on the same 8,736 items and is therefore directly comparable in terms of evaluation distribution. In contrast, κ values from the balanced 150-item evaluation set should not be directly compared with Human IRR because κ is sensitive to label prevalence. We will report the corrected table below in the revised manuscript.

The corrected table

Full gold set (N = 8,736, natural label mix):

ScenarioNGPT-5.5 κQwen 3.5-27B κHuman IRR κHuman Acc. (%)
AWithin-turn redundancy1,2480.53 ± 0.020.54 ± 0.020.2498.6
BUser echoing1,2480.56 ± 0.010.47 ± 0.030.3194.7
CPhrase reuse1,2480.57 ± 0.010.31 ± 0.040.2380.8
DRedundant questions1,2480.73 ± 0.000.68 ± 0.000.6491.6
ERepeated statements1,2480.61 ± 0.030.63 ± 0.020.5892.3
FRedundant questions (history)1,2480.65 ± 0.010.64 ± 0.000.5586.6
GRepeated statements (history)1,2480.57 ± 0.010.52 ± 0.030.4686.3
Overall (sample-wise)8,7360.64 ± 0.010.57 ± 0.000.4990.1

Balanced held-out evaluation set:

ScenarioN (eval)GPT-5.5 κGPT-5.5 Acc. (%)Qwen SFT κQwen SFT Acc. (%)
AWithin-turn redundancy61.00 ± 0.00100.0 ± 0.01.00 ± 0.00100.0 ± 0.0
BUser echoing60.67 ± 0.0083.3 ± 0.00.44 ± 0.1972.2 ± 9.6
CPhrase reuse260.26 ± 0.1262.8 ± 5.90.46 ± 0.1373.1 ± 6.7
DRedundant questions270.73 ± 0.0486.4 ± 2.10.85 ± 0.0092.6 ± 0.0
ERepeated statements231.00 ± 0.00100.0 ± 0.01.00 ± 0.00100.0 ± 0.0
FRedundant questions (history)370.55 ± 0.0078.4 ± 0.00.84 ± 0.0591.9 ± 2.7
GRepeated statements (history)250.68 ± 0.0884.0 ± 4.00.73 ± 0.1286.7 ± 6.1
Overall (sample-wise)1500.65 ± 0.0182.4 ± 0.00.77 ± 0.0488.7 ± 0.0

The dataset link is provided in Footnote 1.

We would like to point the reviewer to Footnote 1, which already links to the dataset through the hyperlinked word "Kaggle." The URL is https://www.kaggle.com/datasets/d73a1e75af77dea652f41554a346b96ef46bf30b891907ef5391e246b8c0ebbb

Reviewer_BPrZOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a benchmark for gauging the undesirable repetitions by LLMs in a multi-turn. The benchmark consists of two dataset, one human generated where nurses were recruited to talk to a model and then annotated the conversations. These conversations are then used to create a set of "verifiers". Some verifiers are hard-coded rules, and some are LLM-judge based evaluators (either by prompt design, or by human agreement). The verifiers are then used to create the "benchmarking" set which is obtained by filtering fully automatic conversations. This dataset is used for RL and as benchmark set for the frointer model. The paper shows that repetition rates go down when instructions and verifiers are used.

Critical strengths

good initial collection of the golden dataset initial work on the taxonomy valuable problem to solve

Critical weaknesses

lack of verification of the taxonomy and the filters low kappa scores (0.49 is lower than a usual 0.61) kappa is used for validity while it shows only the agreement not multi-turn (less than 2 turns in the golden dataset) not a fixed data (running a new model means regenerating all responses and all verifiers) some details missing for the human study (how many annotators in total, what was their work load, what was the study setup) verifiers were not tested on the natural distribution (similar to the golden dataset) RL reward is supplied by the same verifiers and this could lead to some bias

Questions for authors

How are you certain that the proposed taxonomy is a good representation of the repetition types? What was the distribution of kappa scores and how results change if you disregard all non-matching labels? Was there any prompt ablations done? Would results changed if the prompt was phrased differently or a different model used? How many times was the the generation of benchmark dataset run? Are the reported results on a single run or some aggregation? What are the verifiers vs human label precision/recall scores? What is the effect of changing the reward judge in RL?

8 comments

We appreciate your review and your recognition of the gold dataset, the taxonomy, and the value of the problem. However, we would like to clarify several points in the review that do not fully reflect the methodology and results presented in the manuscript.

κ = 0.49 measures agreement between the two human annotators, not verifier performance.

We would like to clarify that κ = 0.49 measures agreement between two human annotators, rather than verifier performance. Section 3.1 explicitly identifies it as "inter-rater reliability between the first two annotators." Disagreements were not discarded: a third annotator adjudicated 900 items (10.3%), and verifiers were evaluated against the resulting labels. We use Cohen's κ as a verifier evaluation metric because our goal is to develop a human-calibrated verifier whose judgments align with human annotations. Since the objective is to reproduce human judgments rather than simply maximize classification accuracy, κ is appropriate because it measures agreement while accounting for chance agreement. Verifier agreement against adjudicated human labels is reported separately in Section 3.2: κ = 0.64 for GPT-5.5 and 0.77 for the fine-tuned verifier, both above 0.61, the threshold referenced in footnote 3. Human κ is affected by class imbalance; category A achieves 98.6% raw agreement but κ = 0.24, which is why Table 1 reports both metrics.

The gold dataset is multi-turn: its conversations average 28.9 turns.

The "two" in Section 3.1 refers to the number of candidate responses per context, not conversation length. Section 3 defines each payload as containing "a conversation history of n ≥ 20 turns." Table 5 reports a mean of 28.9 turns per gold conversation (14.7 agent, 14.2 patient).

The benchmark contexts are fixed; only the evaluated model's response varies.

The dataset is fixed, the model's response is what varies per model. This is the standard way of benchmarking generative models: MT-Bench (Zheng et al., 2023), MT-Bench-101 (Bai et al., 2024), and MultiChallenge (Sirdeshmukh et al., 2025) do the same (Table 3). The deterministic verifiers are "pure functions" of the transcript and response (Appendix O.4), and the subjective verifiers are provided as a fixed, fine-tuned Qwen3.5-27B checkpoint (Section 3.2).

The RL model is also evaluated beyond the training verifiers, and we will discuss the reward overlap as a limitation.

We appreciate the reviewer raising this important concern, which we share. For this reason the paper includes evaluation results beyond the training verifiers on SODA-Eval, which differ in domain, taxonomy, and judgment (Appendix N). There, the 30%-reward checkpoint repeats no more than the base DeepSeek-V3.1 model (2.07% vs. 2.36%) and has higher judge-rated quality (4.77 vs. 4.54 out of 5). Furthermore, the deterministic verifiers involve no LLM judge, so they rule out judge bias, though not reward overlap; on them the RL model fails on 2.99% of hard-tier contexts, against 42.76% of the base model (please see our response to Reviewer fjUD). We will explicitly identify the reward judges in a new policy-RL appendix and discuss the overlap as a limitation.

The taxonomy is derived from prior frameworks, and its categories are distinct in the human labels.

Section 2 derives the categories from four prior frameworks. Appendix I reports pairwise Jaccard similarity of at most 0.14 between categories in the human labels, supporting their distinctiveness. The verifiers are validated against 8,736 adjudicated nurse labels (Table 1). We will move this validation evidence into Section 2 and acknowledge the coverage limitation in Section 6.

The gold set is sampled from patient–agent interaction logs at their natural repetition rate.

Gold payloads are sampled from patient–agent interaction logs, where inappropriate repetition occurs in 13.1% of contexts (footnote 2). The Human IRR column uses this full set. However, the verifier columns use balanced subsets (Appendix F.3), reporting κ and accuracy rather than precision and recall. We will specify the evaluation set and label distribution for each Table 1 column and discuss this limitation in Section 6.

Three nurse annotators labeled the gold set, contributing about 380 annotation hours in total.

Section 3.1 already specifies three nurse annotators: two independently labeled all 8,736 items, and a third annotator adjudicated 900 items. The Ethics Statement reports professional-rate compensation and the right to decline tasks. We will clarify in Appendix D that annotations were conducted through registered nurses who are professionally trained for data annotation. In total, we estimate that about total of six registered nurses contributed roughly 380 annotation hours: about 330 hours for the two independent primary passes and about 50 hours for third-annotator adjudication.

Section 3.2 compares four judge models, and verifier prompts were selected among paraphrased variants.

Section 3.2 already compares four judge models under identical prompts. GPT-5.5 achieves κ = 0.64, compared with 0.36, 0.24, and 0.21 for the alternatives. Verifier prompts were selected by comparing paraphrased variants against the human annotations and keeping the variant with the highest Cohen's kappa.

The reported results are aggregated over multiple runs.

The reported results are aggregated, not from a single run. Each main result is the mean over three runs with its standard deviation, as shown in Table 1. Robustness to decoding is measured over seven decoding runs, as shown in Appendix M, and model differences are tested with paired-bootstrap confidence intervals and McNemar tests, as shown in Appendix K.

We wanted to follow up on our response, where we clarify the distinction between human annotation agreement and verifier performance, the multi-turn nature of our evaluation data, and how we assess the RL model beyond its training verifiers. We hope these clarifications help address your concerns, and we would appreciate it if you would consider updating your score. We are also happy to answer any remaining questions before the discussion period ends.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

General Response to All Reviewers

We thank all reviewers for their time and constructive feedback. We appreciate their recognition of the importance of the problem (Reviewer BPrZ), the scale and transparency of our human-annotated dataset (Reviewer fjUD), and the value of our taxonomy for diagnosing repetition (Reviewer 6euk). Many of the concerns raised involve details of our methodology and evaluation that we could have explained more clearly. We also ran additional experiments to address the reviewers' questions, and the results further support our main findings. The central claims of the paper remain unchanged.

1. Overlap between the RL reward and benchmark judge (Reviewers BPrZ, fjUD, 6euk)

The repetition reward (15%) uses all 13 verifiers: six deterministic verifiers that do not rely on an LLM judge and seven subjective verifiers evaluated by GPT-5.5, which also scores the benchmark. We agree that using the same subjective judge for training and evaluation could introduce bias, and we will make this limitation clear in the revision.

There are three points worth highlighting:

  • Judge-independent results. The RL model has deterministic ANY-failure rates of 1.28% / 2.99% on Easy / Hard, compared with 9.27–20.24% and 20.61–41.62% for frontier models (Figure 4a). These results do not depend on GPT-5.5 judgments.
  • Adversarial benchmark construction. The Hard split consists of contexts where DeepSeek-V3.1, the RL model's base, was flagged for repetition during benchmark construction (Section 3.3).
  • Evaluation with a different judge and taxonomy. On SODA-Eval, which uses an identity-blind GPT-5.6-terra judge without repetition-specific instructions, repetition decreases from 2.36% (base) to 2.07% (30%-reward checkpoint; Table 2, Appendix N). This suggests that the improvement is not limited to the benchmark judge.

The remaining 85% of the reward comes from a proprietary production judge that evaluates clinical correctness, empathy, and task adherence without explicitly evaluating repetition. We will add a policy-RL appendix with details on the reward judges, training-prompt sources and their overlap with the benchmark, and GRPO configuration.

2. Additional baseline evaluations on RepetitionBench (Reviewer fjUD)

We ran additional evaluations of DeepSeek-V3.1 (base) and the 0%-repetition-reward checkpoint on the same Easy (E) and Hard (H) contexts. The table shows ANY-failure rates (%; lower is better).

ModelBoth E / H
DeepSeek-V3.1 (base)40.93 / 67.18
0% repetition reward47.52 / 72.49
RL model (15%)12.55 / 17.75

All three models generate new responses to the same contexts, so the comparison is not affected by differences in context selection. The 0%-reward checkpoint, trained with task-quality RL alone, has higher repetition than the base model (72.49% vs. 67.18% on Hard). With the repetition reward, the failure rate drops to 17.75%, supporting its contribution beyond task-quality RL alone. We will add these baselines to Figure 4a and Table 11.

3. Effects on other capabilities (Reviewers fjUD, 6euk)

We agree that reducing inappropriate repetition should not come at the expense of other capabilities. To examine this, Table 2 includes a 0%-reward checkpoint trained with task-quality RL alone.

Clinical correctness drops from 48.0% (base) to 40.0% (0%), before the repetition reward is introduced. The scores are 37.2% at 15% and 42.4% at 30%, with no consistent decline as the repetition reward increases. Since our current ablations do not isolate the cause of the initial drop, we will report this as an observation rather than make a causal claim.

Between the 0% and 30% checkpoints, empathy stays at 90.8%, while task adherence changes from 80.0% to 80.5%. Appropriate repetition also decreases from 58.4% to 49.4%. We recognize this as a trade-off, since some repetition is useful for clarification and confirmation, and will discuss it in the limitations.

4. Human agreement and verifier reliability (Reviewers BPrZ, fjUD)

The κ = 0.49 reported in Section 3.1 measures agreement between the two human annotators, not verifier performance. This value is partly affected by label imbalance. For example, in category A, only 9 of 1,248 items are labeled FAIL, resulting in κ = 0.24 despite 98.6% raw agreement. Verifier performance is reported separately: on the full gold set, GPT-5.5 achieves κ = 0.64 against adjudicated human labels.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.