acceptodds
Under review as a conference paper at ICLR 2027

Chinese Competitive Debate: A Dataset and Benchmark with Professional Adjudication

14 views

Abstract

Evaluating debate requires interpreting contributions in context and relating them to local assessments and overall outcomes. We introduce the Chinese Competitive Debate Dataset (CCDD), linking competition transcripts to original adjudication under a predefined rubric. CCDD contains 148 matches, 2,690 stage–side scoring instances, and 20,542 turns, with reviewed transcripts, aligned individual judge scores, match decisions, best-debater ballots, and commentary. Linked appeal records preserve reassessments. Three benchmark tasks—winner prediction, stage-score prediction, and best-debater prediction—demonstrate the use of this supervision. Zero-shot results reveal challenges in reproducing stage assessments, while judge-agreement analyses and structural references contextualize performance. CCDD supports research on contextual debate evaluation and judge disagreement. The dataset is publicly available at https://anonymous.4open.science/w/CCDD-Page.

Primary area:
datasets and benchmarks
Keywords:
Dabate, Benchmark, Dataset, LLM, NLP

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WMZ1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the Chinese Competitive Debate Dataset (CCDD), which provides live transcripts and rubric scoring of competitive debates in Chinese. It covers 148 matches with over 20,000 speech turns, all transcribed from live recordings, as well as all ballots and comments of each match. Based on these data the authors propose three benchmark tasks: winner prediction, stage-score prediction, and best-debater prediction. Experiments on mainstream LLMs reveals existing challenges in applying LLMs directly on debate judgement.

Critical strengths
  1. The dataset completes the missing piece of full-debate assessment in Chinese. Existing datasets focus on either English debates or fine-grain analysis of the debate contents, while CCDD provides new language resources and unique debate formations that can enrich the family of debate judging samples.
  2. The benchmark supports multiple judging tasks that covers various aspects. CCDD proposes prediction tasks on winner, stage-score, and best-debater, which helps measure the ability of LLMs on understanding whole debates from different aspects and granularities. Also, each task is backed with professional rubrics and adjudication, so the benchmark can be seen as a reliable assessment.
Critical weaknesses
  1. The insights from the given experiments and analysis are limited. Although the current set of tasks does cover various levels and aspects, the authors only report the metrics on the three defined tasks, and gives little further analysis nor insight suggestions. Basically I would expect a much richer Section 5.3 than the current version.
    • The last paragraph says "Together, these tasks demonstrate how linked adjudication records support evaluation beyond final outcomes. Stage-score results particularly motivate joint consideration of numerical accuracy, ranking, and role structure. Small differences between models should be interpreted cautiously." However, I don't see a clear analysis chain for the first two statements, and the third provides little information.
    • Also the tasks don't fully utilize the rubric results: the stage-score prediction merges results from three different stages and report overall metrics. Given that stage score labels are available, I'd be more interested in how LLMs perform in each of the stage, and whether it matters the winner and best-debater predictions.
  2. The current experiments are based on zero-shot prompting, which is relatively weak. This issue is significant on the non-tie agreement column in Table 5, where structure-aware random beats most LLM predictors. It would be more interesting to see how existing agentic or fine-grain frameworks like Debatrix or recent general harness tools like Claude Code would perform on the benchmark.
Questions for authors
  1. Can you give a clearer analysis regarding the last paragraph of Section 5.3, mentioned above?
  2. What are the most important factors that affects the performance of LLM debate judges on CDDC, either positive or negative?
  3. How are the debate motions distributed, and do LLMs perform better or worse on certain debate motions or judge stages?
  4. How much do you think that agentic workflows can help improve performance on the three tasks, respectively?
4 comments

Thank you for reviewing our paper.

As you noted in the Strengths section, CCDD fills a gap in the comprehensive evaluation of Chinese-language debate and supports several tasks that assess different aspects of debate performance. Our aim is to build a dataset and benchmark. The initial submission therefore focused on how to interpret the metrics and use the dataset, leaving a broader investigation of what shapes current models’ performance to future work. We address your points in turn.

(W2) You noted that zero-shot prompting is relatively simple. We chose it to evaluate base models under the same conditions. Once a study introduces task-specific prompts or an agentic workflow, its results reflect the contributions of the base model, the prompts, and the workflow. A common zero-shot baseline gives future studies a reference against which to measure the improvements these designs bring.

Your observation that the structure-aware baseline achieves a non-tie agreement of 0.634 illustrates why we report several metrics. Non-tie agreement measures how often a model identifies the higher-scoring side when the judges give the two sides different scores. Even when a model identifies that side correctly, its predicted scores may still differ substantially from the judges’ mean scores. We therefore report MSE, correlation coefficients, full agreement, and non-tie agreement. Appendix F.1 further examines performance within each speaker role.

(Q4, W2) You asked how much agentic workflows might improve performance on each of the three tasks. We believe this question is worth investigating, and we hope researchers will use CCDD to develop debate models and agents. The debate experts we consulted offered one idea. They described judging a debate as keeping a mental scale that tips toward the affirmative when that side makes a strong argument and toward the negative when the negative side argues well. This suggests a design for a judging agent that updates its assessment as the debate unfolds.

(Q3, Q2, W1) You asked about the distribution of motions and stages, and about the factors that most affect LLM judges’ performance on CCDD. Given the paper’s focus on introducing the dataset and benchmark, the initial submission did not break down results by these factors.

CCDD covers 28 distinct motions. Six appeared in at least eight matches each. For each of these motions, we averaged winner-prediction accuracy across the nine model configurations. These averages ranged from 48.6% to 64.8%. With few matches per motion, we report the observed differences but remain cautious about concluding that particular motions are easier or harder to judge.

The additional analysis yielded two observations. The first concerns agreement among human judges. Mean winner-prediction accuracy was about 14.4% higher in matches decided unanimously (3–0) than in matches decided by a split vote (2–1). The second concerns debate stages and speaker roles. Spearman’s ρ between model scores and judges’ scores was roughly 0.1 for single-speaker stages, about 0.2 for questioners, and 0.07 for respondents. We report these results in Appendix F.1 and hope they provide a starting point for further study.

(Q1, W1) In response to your comments, we revised the wording and structure of the text. What we meant by that sentence was that, by linking the stage scores, best-debater ballots, and final outcome of the same match, we can assess whether a model can correctly identify the winner, accurately score a particular stage, and accurately evaluate an individual debater. For stage scoring, for example, we need to consider how far the model’s scores are from the judges’ scores (MSE), whether the model ranks stronger and weaker performances similarly to the judges (Spearman’s ρ), and how to interpret the results in light of the debater’s role in that stage.

We have reorganized the text, removed the original sentence, and addressed its points in separate sections. Section 5.1 explains the purpose of each task. Section 5.3 explains the importance of the different metrics in more detail. Finally, we have added a confidence interval analysis in Appendix F.2 to address whether the observed differences between models support a ranking.

Finally, we ask you to consider the paper’s contribution as a dataset and benchmark. To our knowledge, no previous large-scale Chinese-language debate dataset has provided stage-level supervision or exchange-level transcript segmentation that identifies the side speaking. These resources can support research applications such as supervised or unsupervised fine-tuning of LLMs. The scoring framework and tasks we defined with debate experts also provide a reference for future studies. We would appreciate it if you reconsidered your score in light of this contribution and the additional analyses in the revised paper. If you have any further questions or comments, please feel free to share them. We will respond within 48 hours.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Response

Dear Reviewer WMZ1,

Thank you sincerely for your time and effort. ICLR received a record number of submissions this year, roughly twice last year's, and the burden on every reviewer is very heavy. It is reviewers' careful work that keeps the peer-review process running.

The reviewers of our paper hold differing views. As the discussion period closes on November 18, we would greatly value further feedback from you. Our reply of November 9 answered each of the points you raised, and the revised PDF has been uploaded. If any concern remains, even a single line would help us, and we will respond before the deadline. Thank you.

Best regards, The Authors

Thank you for the response and revisions. The added analyses by stage, speaker role, and judge agreement provide useful insights and address my concern about the limited analysis. The clarification of the evaluation metrics is also helpful.

I understand the rationale for using a common zero-shot baseline. The benefits of stronger judging workflows, and how stage-level prediction errors relate to winner and best-debater predictions, remain open questions.

Considering these improvements alongside the dataset’s contribution, I have adjusted my ratings.

AuthorsOpenReview ↗

Thank You for the Updated Assessment

Thank you for reviewing our revisions and for updating your assessment.

We agree that the benefits of stronger judging workflows, and how stage-level prediction errors relate to winner and best-debater predictions, are important open questions. We hope that CCDD, both as a dataset and as a benchmark, can serve as a basis for future work on them.

As a further update, on November 17 we added a paragraph in Section 5.3 noting that the correlation metrics in Task 2 are influenced by role and stage type, which makes the analysis in the main text more complete.

As the discussion period draws to a close, we would like to thank you once again for the time and support you have given throughout the review process.

Reviewer_45kYOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper introduces CCDD, a dataset of 148 Chinese competitive debate matches (2,690 stage–side scoring instances, 20.5k turns) with each judge’s individual stage scores under a fixed rubric, match decisions, best-debater ballots, commentary and appeal records. It defines three tasks (predicting the winner, the stage scores and the best debater) and reports zero-shot LLM results with structural and human-agreement baselines.

Critical strengths
  • The data is rare. Individual judges’ ballots, stage-level scores and commentary for Chinese debate formats aren’t available elsewhere, and the appeal records are a nice extra.
  • It’s recent and timely. There is very little chance that there is contamination.
Critical weaknesses
  • No confidence intervals, and the results are close together. With n = 148, a 95% interval on winner or best-debater accuracy is about ±8pp. The model spread (.56–.66 on winner) mostly falls within that. CIs, ideally from resampling whole matches, are needed before any ranking claims.
  • The human reference isn’t comparable to the model scores. The leave-one-out human winner agreement (.788) drops comparisons where the other two judges split (312 of 444 comparisons are kept). Models, though, are scored on all 148 matches, including the 66 split panels. Either score models on the same eligible subset, or use a pairwise comparison: treat the model as a fourth judge and compare its agreement with each individual judge to judge–judge agreement (70.3%).
  • The analysis is thin for what the dataset allows. Natural breakdowns are missing: by stage type, by format (4v4 vs 2v2), and side bias, i.e. how often models predict affirmative compared with the .507 base rate and with judges. This kind of analysis was standard in pre-LLM debate work (e.g., Zhang et al., NAACL 2016; Durmus & Cardie, 2019).
  • Task 2 leaves out the strongest models. Only 5 of the 9 configurations are evaluated on stage scoring (Table A5). Missing are opus-5 (best on winner), gpt-5.6-sol (best on best-debater), sonnet-5 and haiku-4.5. No reason is given. Since stage scoring is the paper’s main contribution, this gap matters most.
  • Useful guidance is buried in Appendix D.4. It recommends keeping whole matches in one split, grouping repeated motions, and resampling at the match level to get uncertainty. These would strengthen the paper as benchmark protocol in the main text, and the authors should apply the resampling to their own results (ties to point 1).
Questions for authors
  1. Why were the models left out of task 2, when these models were also performing well on tasks 1 and 3?
  2. Can you report the intervals from resampling whole matches for tables 5 and 6? With n=148, it looks like the spread across models is within noise.
  3. The structure-aware baseline is almost the same as the best model (.634 vs .638). Could you report metrics that account for the role; this would help show if models use anything beyond the role of the debater.
4 comments

Thank you for your review. We address your points in turn.

(W2) You raised a concern about the comparability of human and LLM accuracy in predicting match winners. We would first like to clarify that the model-to-model comparisons in the main text are directly comparable and are unaffected by this issue. We deliberately placed the human reference in a separate appendix section, Human and Oracle References, precisely because of this limitation. As explained there, when all three judges are included, each judge’s own vote contributes to the target against which that judge is evaluated. Under leave-one-out evaluation, the target is determined by the other two judges. If they vote for opposite sides, their votes do not determine a winner, so that comparison is excluded. Consequently, no human winner-prediction accuracy reference preserves the same labels, evaluation subset, and task as the main evaluation. The first approach you suggest would give less weight to matches decided by a 2–1 vote. Since models predict winners more accurately in matches decided unanimously (3–0), this change in weighting would also affect the comparison. Your second approach measures agreement between a model and an individual judge, which addresses a different question from comparing human and model accuracy on Task 1. This measure conveys some of the same information as winner-prediction accuracy broken down by whether the judges reached a unanimous or split decision. We have nevertheless followed your second suggestion and reported model agreement with individual judges in Appendix D.3.

(Q2, W1, W5) You noted that we had not reported confidence intervals or resampling results and raised concerns about model rankings. Many related studies, including DebateBench, Debatrix, Debatable Intelligence, and DEFINED, report their main model evaluation results without confidence intervals. Since establishing the relative ranking of models is outside the main objective of this benchmark, quantifying uncertainty was not a primary focus of our initial evaluation. Following your suggestion, we have now conducted a resampling analysis and reported the resulting confidence intervals in Appendix F.2.

(Q3, W3) You pointed out that the structure-aware baseline achieves a non-tie agreement of 0.634, close to the results of several models. We believe this observation illustrates why it is important to preserve each side’s role and score within the same exchange and to consider several metrics together, including MSE, correlation, full agreement, and non-tie agreement. You also identified opportunities for further analysis. The scope of this paper is to introduce a dataset and benchmark; explaining why models perform poorly on particular debate tasks is a broader research question. Following your suggestion, we have added an initial breakdown by stage type, competition format, and level of agreement among judges in Appendix F.1. We hope these results will provide a useful starting point for other researchers.

(Q1, W4) We initially evaluated only a subset of the models on Task 2, primarily because of the evaluation cost. The paper’s main objective is to provide a dataset and an evaluation methodology. Data collection had already required substantial resources, and Task 2 cost approximately 13 times as much to run as the other evaluation tasks. This is why we initially limited the number of models evaluated on this task. After you raised the issue, we ran the remaining experiments and added the results to Table 5.

Thank you for your suggestions. The main contribution of this work is a large-scale dataset and benchmark for professionally adjudicated debates. We appreciate your recognition of the dataset’s value. Several of your concerns relate to analyses beyond the paper’s central contribution, and we have expanded the analysis in response to your suggestions. In light of the dataset’s contribution and these additions, we would be grateful if you would reconsider your score. If you have any further concerns or suggestions, please feel free to raise them at any time. We will respond within 48 hours.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Response

Dear Reviewer 45kY,

Thank you sincerely for your time and effort. ICLR received a record number of submissions this year, roughly twice last year's, and the burden on every reviewer is very heavy. It is reviewers' careful work that keeps the peer-review process running.

The reviewers of our paper hold differing views. As the discussion period closes on November 18, we would greatly value further feedback from you. Our reply of November 9 answered each of the points you raised, and the revised PDF has been uploaded. If any concern remains, even a single line would help us, and we will respond before the deadline. Thank you.

Best regards, The Authors

Thank you for the response and reminder! Many of my concerns were addressed, so I raised the soundness and score to a 2.

However to follow up: the role correlations in table A13 look weak. This reads that a lot of task 2 performance reflect sjust the role and stage type rather than in-role quality. this feels like this should be stated in the maintext alongside table 5.

AuthorsOpenReview ↗

Response to Follow-up and New Paragraph in Section 5.3

Thank you for the follow-up and for reassessing our paper.

Following your suggestion, we have added a paragraph in Section 5.3, immediately after the agreement metrics and before Table 6, that explicitly states the interpretation above and refers to Table A13. The revised PDF has been uploaded.

Yes, the correlation metrics in Task 2 are influenced by role and stage type. When computed separately by stage type and speaker role, the mean Spearman's ρ across the nine configurations is highest for summaries at 0.180 and lowest for closing speeches at 0.036, both below the overall mean of 0.238. This indicates that part of the overall correlation comes from between-group alignment: mean judge scores already differ across roles and stage types, and the models reproduce these differences. Therefore, a single metric should be read together with the other metrics and understood alongside the within-role decomposition; current models mainly reproduce the role structure of the scores (and even so, the overall mean ρ is only 0.238), while their ability to discriminate performance within the same role remains limited.

The insufficient capability of current models is precisely one of the purposes for which we propose this benchmark: by constructing a dataset and benchmark from high-quality data in a professional domain, we enable future researchers to build on it to develop better agents or models and to improve stage-level discrimination in debate.

If you have any further concerns, please feel free to point them out. If this revision has resolved your question, we would be grateful if you would take it into account in your assessment. We know your time is very valuable, and we thank you again for it.

Reviewer_HbdROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The study examines how well LLMs can evaluate chinese competitive debates. The challenge is that a debate cannot be evaluated simply based on individual statements or linguistic quality. It is also crucial to consider which arguments a person responds to, what role they are currently playing, and whether they are actually advancing the discussion. To this end, the authors introduce the Chinese Competitive Debate Dataset (CCDD). It links authentic Chinese-language debates with the evaluations of professional judges and documents not only the outcome of a match but also evaluations of individual debate phases and votes on the best debater.

Critical strengths

The paper provides a well-structured and well documented dataset for a challenging problem.

Critical weaknesses

For someone unfamiliar with competitive debates and the structure of such competitions, the paper was difficult to follow at times. Another point I would like to raise is that, in order to judge the debate, the dataset includes transcripts of the spoken debate rather than real audio recordings. The dataset contains a great deal of detailed information. I am wondering whether it would be possible to derive judge scores from professional judges based on the transcribed version alone, rather than the audio recordings. Perhaps this missing information also explains the discrepancy between human labels and LLM performance.

Questions for authors

Can you reflect on how much relevant information is lost in the transcript compared to audio or video recordings of a debate? Do factors such as facial expressions, gestures, tone of voice and pauses play a significant role in debates and their outcomes?

1 comment

Dear Reviewer,

Thank you very much for your review and suggestions!

First, we would like to address your question about how much information is lost when a debate is transcribed from audio or video. Before organizing the competitions, we consulted several prominent debate experts and conducted more than 20 hours of interviews in total. The experts described some differences in judging practices across Chinese-language debate communities in China, Singapore, and Malaysia. For example, they told us that judges in Malaysia tend to emphasize how debaters build their arguments, while judges in China place greater weight on how debaters challenge and defend arguments in direct exchanges. Despite these differences, almost all the experts said that facial expressions and tone of voice rarely affect their own decisions. Asking judges to score the debates again from transcripts alone would cost nearly as much as building the dataset itself. Based on the advice we have received so far, we would expect the change in judgments to be limited.

We have not evaluated models on the audio recordings. We designed this as a text-based benchmark and evaluated commonly used models on the transcripts. We would be delighted to see other research teams investigate the differences between judging a debate from a recording and judging it from a transcript. This is exactly the kind of research we hoped to encourage by releasing the dataset. Thank you for this suggestion. The audio recordings are now available by application for academic research only. Access is restricted because participants may be identifiable from their voices. Several research institutions have already emailed us to request access.

You also mentioned that parts of the paper were difficult to follow for readers unfamiliar with competitive debate and its formats. Given the space constraints in the main text, we provide a detailed description of the formats in Appendix A. Figure A1 illustrates how a 4v4 Chinese-language debate unfolds, and Table A3 in Appendix B explains terms such as stage and turn. We will also revise the description in the main text to make the paper easier to follow for readers who are new to competitive debate.

If you have any further concerns, or if our response raises new questions, please feel free to let us know. We will respond within 48 hours. If you feel that our response has addressed your concerns, we would be grateful if you would reconsider your score. Every reviewer’s assessment matters to us. Thank you again!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Initial Author Response

We thank all three reviewers for their careful reading and the Area Chair for coordinating and contributing to the review process. We have responded to all reviewer comments and uploaded the revised PDF.

We hope the reviewers will consider the paper’s contribution as a dataset and benchmark. Prior to this work, we were unaware of any large-scale Chinese-language debate dataset that provided stage-level supervision or exchange-level transcript segmentation with side labels. These resources can support research applications such as supervised or unsupervised fine-tuning of LLMs. The scoring framework and tasks we developed with debate experts can also serve as a reference for future research.

We made the following changes in the revised PDF.

  1. We corrected a transcription error in Table 6. While computing the confidence intervals requested by the reviewers, we discovered that the winner-prediction accuracies listed for opus-5, sonnet-5, and haiku-4.5 were actually the values for sonnet-5, gemini-3.5-flash-lite, and deepseek-v4-flash, respectively. We apologize for this error and have corrected it.
  2. We revised the wording and visual presentation of the paper.
  3. We added four model rows to Table 5 and included the analyses suggested by the reviewers in the appendix. Appendix F.1 presents the subgroup analyses, and Appendix F.2 presents the uncertainty analysis.

We have addressed each reviewer’s concerns in our individual replies. If the Area Chair, reviewers, or readers have any further questions or suggestions, please feel free to raise them during the discussion period. We will respond within 48 hours.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.