- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces the Chinese Competitive Debate Dataset (CCDD), which provides live transcripts and rubric scoring of competitive debates in Chinese. It covers 148 matches with over 20,000 speech turns, all transcribed from live recordings, as well as all ballots and comments of each match. Based on these data the authors propose three benchmark tasks: winner prediction, stage-score prediction, and best-debater prediction. Experiments on mainstream LLMs reveals existing challenges in applying LLMs directly on debate judgement.
Critical strengths
- The dataset completes the missing piece of full-debate assessment in Chinese. Existing datasets focus on either English debates or fine-grain analysis of the debate contents, while CCDD provides new language resources and unique debate formations that can enrich the family of debate judging samples.
- The benchmark supports multiple judging tasks that covers various aspects. CCDD proposes prediction tasks on winner, stage-score, and best-debater, which helps measure the ability of LLMs on understanding whole debates from different aspects and granularities. Also, each task is backed with professional rubrics and adjudication, so the benchmark can be seen as a reliable assessment.
Critical weaknesses
- The insights from the given experiments and analysis are limited. Although the current set of tasks does cover various levels and aspects, the authors only report the metrics on the three defined tasks, and gives little further analysis nor insight suggestions. Basically I would expect a much richer Section 5.3 than the current version.
- The last paragraph says "Together, these tasks demonstrate how linked adjudication records support evaluation beyond final outcomes. Stage-score results particularly motivate joint consideration of numerical accuracy, ranking, and role structure. Small differences between models should be interpreted cautiously." However, I don't see a clear analysis chain for the first two statements, and the third provides little information.
- Also the tasks don't fully utilize the rubric results: the stage-score prediction merges results from three different stages and report overall metrics. Given that stage score labels are available, I'd be more interested in how LLMs perform in each of the stage, and whether it matters the winner and best-debater predictions.
- The current experiments are based on zero-shot prompting, which is relatively weak. This issue is significant on the non-tie agreement column in Table 5, where structure-aware random beats most LLM predictors. It would be more interesting to see how existing agentic or fine-grain frameworks like Debatrix or recent general harness tools like Claude Code would perform on the benchmark.
Questions for authors
- Can you give a clearer analysis regarding the last paragraph of Section 5.3, mentioned above?
- What are the most important factors that affects the performance of LLM debate judges on CDDC, either positive or negative?
- How are the debate motions distributed, and do LLMs perform better or worse on certain debate motions or judge stages?
- How much do you think that agentic workflows can help improve performance on the three tasks, respectively?
4 comments
A Gentle Follow-up on Our Response
Dear Reviewer WMZ1,
Thank you sincerely for your time and effort. ICLR received a record number of submissions this year, roughly twice last year's, and the burden on every reviewer is very heavy. It is reviewers' careful work that keeps the peer-review process running.
The reviewers of our paper hold differing views. As the discussion period closes on November 18, we would greatly value further feedback from you. Our reply of November 9 answered each of the points you raised, and the revised PDF has been uploaded. If any concern remains, even a single line would help us, and we will respond before the deadline. Thank you.
Best regards, The Authors
Thank you for the response and revisions. The added analyses by stage, speaker role, and judge agreement provide useful insights and address my concern about the limited analysis. The clarification of the evaluation metrics is also helpful.
I understand the rationale for using a common zero-shot baseline. The benefits of stronger judging workflows, and how stage-level prediction errors relate to winner and best-debater predictions, remain open questions.
Considering these improvements alongside the dataset’s contribution, I have adjusted my ratings.
Thank You for the Updated Assessment
Thank you for reviewing our revisions and for updating your assessment.
We agree that the benefits of stronger judging workflows, and how stage-level prediction errors relate to winner and best-debater predictions, are important open questions. We hope that CCDD, both as a dataset and as a benchmark, can serve as a basis for future work on them.
As a further update, on November 17 we added a paragraph in Section 5.3 noting that the correlation metrics in Task 2 are influenced by role and stage type, which makes the analysis in the main text more complete.
As the discussion period draws to a close, we would like to thank you once again for the time and support you have given throughout the review process.
Thank you for reviewing our paper.
As you noted in the Strengths section, CCDD fills a gap in the comprehensive evaluation of Chinese-language debate and supports several tasks that assess different aspects of debate performance. Our aim is to build a dataset and benchmark. The initial submission therefore focused on how to interpret the metrics and use the dataset, leaving a broader investigation of what shapes current models’ performance to future work. We address your points in turn.
(W2) You noted that zero-shot prompting is relatively simple. We chose it to evaluate base models under the same conditions. Once a study introduces task-specific prompts or an agentic workflow, its results reflect the contributions of the base model, the prompts, and the workflow. A common zero-shot baseline gives future studies a reference against which to measure the improvements these designs bring.
Your observation that the structure-aware baseline achieves a non-tie agreement of 0.634 illustrates why we report several metrics. Non-tie agreement measures how often a model identifies the higher-scoring side when the judges give the two sides different scores. Even when a model identifies that side correctly, its predicted scores may still differ substantially from the judges’ mean scores. We therefore report MSE, correlation coefficients, full agreement, and non-tie agreement. Appendix F.1 further examines performance within each speaker role.
(Q4, W2) You asked how much agentic workflows might improve performance on each of the three tasks. We believe this question is worth investigating, and we hope researchers will use CCDD to develop debate models and agents. The debate experts we consulted offered one idea. They described judging a debate as keeping a mental scale that tips toward the affirmative when that side makes a strong argument and toward the negative when the negative side argues well. This suggests a design for a judging agent that updates its assessment as the debate unfolds.
(Q3, Q2, W1) You asked about the distribution of motions and stages, and about the factors that most affect LLM judges’ performance on CCDD. Given the paper’s focus on introducing the dataset and benchmark, the initial submission did not break down results by these factors.
CCDD covers 28 distinct motions. Six appeared in at least eight matches each. For each of these motions, we averaged winner-prediction accuracy across the nine model configurations. These averages ranged from 48.6% to 64.8%. With few matches per motion, we report the observed differences but remain cautious about concluding that particular motions are easier or harder to judge.
The additional analysis yielded two observations. The first concerns agreement among human judges. Mean winner-prediction accuracy was about 14.4% higher in matches decided unanimously (3–0) than in matches decided by a split vote (2–1). The second concerns debate stages and speaker roles. Spearman’s ρ between model scores and judges’ scores was roughly 0.1 for single-speaker stages, about 0.2 for questioners, and 0.07 for respondents. We report these results in Appendix F.1 and hope they provide a starting point for further study.
(Q1, W1) In response to your comments, we revised the wording and structure of the text. What we meant by that sentence was that, by linking the stage scores, best-debater ballots, and final outcome of the same match, we can assess whether a model can correctly identify the winner, accurately score a particular stage, and accurately evaluate an individual debater. For stage scoring, for example, we need to consider how far the model’s scores are from the judges’ scores (MSE), whether the model ranks stronger and weaker performances similarly to the judges (Spearman’s ρ), and how to interpret the results in light of the debater’s role in that stage.
We have reorganized the text, removed the original sentence, and addressed its points in separate sections. Section 5.1 explains the purpose of each task. Section 5.3 explains the importance of the different metrics in more detail. Finally, we have added a confidence interval analysis in Appendix F.2 to address whether the observed differences between models support a ranking.
Finally, we ask you to consider the paper’s contribution as a dataset and benchmark. To our knowledge, no previous large-scale Chinese-language debate dataset has provided stage-level supervision or exchange-level transcript segmentation that identifies the side speaking. These resources can support research applications such as supervised or unsupervised fine-tuning of LLMs. The scoring framework and tasks we defined with debate experts also provide a reference for future studies. We would appreciate it if you reconsidered your score in light of this contribution and the additional analyses in the revised paper. If you have any further questions or comments, please feel free to share them. We will respond within 48 hours.