- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates epistemic reliability degradation in debate distillation approaches, which aim to improve a weaker verifier using debate transcripts generated by two stronger LLMs. This important gap arises when an adversarial attacker among the debaters can successfully satisfy the monitored task's correctness criteria while simultaneously pursuing a hidden objective. The authors propose a two-stage black-box auditing framework that first searches for counterexamples and, if none are found, provides statistical bounds on non-degradation. They claim that this framework can detect reliability degradation that conventional evaluation metrics often fail to identify or distinguish from catastrophic forgetting. Their findings suggest that ER-AUDIT can uncover degradation on hidden tasks induced by the attacker debater, which remains concealed by apparent performance gains on monitored tasks.
Critical strengths
Important safety problem with promising empirical observations. The paper investigates an important gap in debate-based oversight, where improvements in monitored-task performance may conceal degradation on related hidden tasks. The results provide some preliminary evidence of this phenomenon.
Principled auditing framework with theoretical guarantees. The proposed two-stage auditing framework combines counterexample search followed by a statistical certification of non-degradation when no counterexample is found. It provides anytime-valid confidence bounds and extensions across multiple paraphrase distributions. Empirically, ER-Audit identifies higher counterexample ratios under adversarial adaptation across all six model–dataset combinations. These results demonstrate the potential value of the framework for detecting localized degradation that standard evaluation may overlook.
Critical weaknesses
-
Although the paper introduces two benchmarks pairing monitored and hidden questions, the relationship between these tasks is not sufficiently justified or evaluated. I also read Appendix B, on how each benchmark is constructed. Sharing a context or scientific subdomain does not necessarily imply meaningful evidential overlap or dependency between the questions.
More broadly, the paper does not provide direct evidence explaining how misleading information in adversarial debates leads to persistent changes in the verifier's hidden-task predictions. The observed effects could arise from misleading factual associations, the optimized adversarial suffix, increased prompt sensitivity, or more general fine-tuning interference. Although Appendix F proposes shortcut learning as a possible explanation, this remains a hypothesis without supporting causal experiments. I believe a more systematic analysis of the relationship between monitored and hidden tasks, together with controlled interventions on adversarial transcripts, would substantially strengthen the paper's central claim about adversarially induced epistemic degradation. -
One of the paper's central motivating questions, emphasized in the abstract and introduction, is whether epistemic reliability degradation represents a distinct risk beyond catastrophic forgetting. However, I find the experimental evidence insufficient to establish this distinction. The authors primarily rely on the observation that higher aggregate hidden-task accuracy can coexist with more localized degradation counterexamples. For example, in QUALITY-H, the GPT-20B adversarial verifier achieves higher hidden-task accuracy than the honest verifier (63.96% vs. 59.91%) while exhibiting more counterexamples. Although this observation challenges explanations based on uniform performance degradation, it does not rule out selective catastrophic forgetting, where a model improves on some tasks while losing previously acquired knowledge on others. The authors acknowledge that selective forgetting may contribute to the observed phenomenon (Section 4.3 and Appendix F), but do not provide experiments that separate it from the proposed epistemic degradation mechanism. Given the importance of this distinction to the paper's motivation, stronger evidence would be valuable.
-
Several aspects of the experimental design make it difficult to isolate the contribution of debate distillation and the additional effects of adversarial transcripts. First, the verifier's fine-tuning objective includes both a language-modeling loss over debate transcripts and direct supervision on the correct monitored answers (Appendix D.1). While Table 1 shows substantial improvements in monitored-task accuracy after honest debate distillation, the paper does not include an answer-only supervised fine-tuning baseline. Consequently, it is unclear how much of the improvement comes from the debate transcripts rather than ordinary supervised fine-tuning. Second, the honest and adversarial adaptation settings use different numbers of training transcripts (222 vs. 184 for QuALITY-H and 101 vs. 83 for GPQA-H). These differences, together with the different transcript-selection procedures, introduce potential confounding factors when attributing hidden-task degradation specifically to adversarial content. Finally, the paper does not clearly specify whether the reported results are averaged across multiple independent fine-tuning runs or obtained from single runs. No run-to-run variance or confidence intervals are provided. Given that some honest–adversarial differences in Figure 2(a) are relatively small, additional statistical analysis and repeated experiments are necessary to establish their robustness.
-
I have two concerns regarding the proposed measurement and auditing approach. First, the degradation score in Equation (2) normalizes the reduction in correct hidden-answer probability by the improvement in monitored-task probability. Consequently, two verifiers experiencing the same absolute hidden-task degradation may receive different degradation scores depending on their monitored-task improvements. This makes comparisons between honest and adversarial adaptation potentially sensitive to the denominator rather than solely to hidden-task reliability. Moreover, changes in correct-answer probabilities do not necessarily correspond to meaningful changes in correctness or epistemic reliability. Second, although the anytime-valid theoretical guarantees appear sound under the stated assumptions, the proposed lower bound in Theorem 1 becomes zero upon observing a single counterexample, regardless of how many previous samples satisfied the non-degradation criterion. This makes the bound conservative and potentially uninformative when degradation is rare but nonzero. Importantly, a lower confidence bound under adversarial adaptation does not necessarily imply a lower true non-degradation probability. The paper would benefit from comparisons with alternative degradation metrics, more informative confidence-sequence methods, and simpler auditing baselines under matched query budgets. Such comparisons would help clarify the practical advantage of ER-Audit beyond evaluating additional paraphrases.
Overall, I find the research problem important and the proposed auditing framework promising, but I remain unconvinced that the current experimental evidence sufficiently supports the paper's broader claims, particularly regarding the underlying degradation mechanism, its distinction from catastrophic forgetting, and its practical relevance to safety-critical oversight.
Questions for authors
Please first refer to the concerns I mentioned earlier.
-
What motivates normalizing hidden-task degradation by monitored-task improvement in Equation (2)? Have the authors evaluated whether their findings remain consistent using unnormalized changes in correct-answer probabilities or correctness-based measures?
-
Given that the paper motivates epistemic reliability degradation as a risk to trustworthy and safety-critical oversight (line 60 and Figure 1), while the proposed benchmarks are derived from general question-answering datasets (QUALITY and GPQA), could the authors clarify to what extent the benchmarks capture safety- or security-relevant scenarios? Does Figure 1 represent an actual experimental example or is purely illustrative? Have the authors assessed whether the observed degradation extends to consequential safety or security judgments?
6 comments
Response to Reviewer bodJ (Part 2/3)
Second, we measure whether Replay mitigates ERD. In Table W2, Acc denotes monitored-task accuracy and denotes the mean post- minus pre-adaptation change in correct-answer negative log-likelihood on , with positive values indicating degradation. We denote the forgetting rates on these non-target questions, target hidden questions, and held-out background questions in by , , and , respectively.
Table W2. Effect of Replay on selective forgetting and ERD during verifier adaptation.
| Verifier | Replay | Acc | ||||
|---|---|---|---|---|---|---|
| No | 0.9645 | 0.2777 | 20.79% | 14.71% | 13.85% | |
| Yes | 0.9408 | 0.0680 | 3.96% | 5.88% | 4.62% | |
| No | 0.9172 | 0.2450 | 23.76% | 27.45% | 16.92% | |
| Yes | 0.9112 | 0.0614 | 4.95% | 23.53% | 6.15% |
Replay reduces by approximately 75% and by approximately 80% for both verifiers, while Acc remains above . Background forgetting rates also fall to 4.62% for and 6.15% for . However, on target hidden questions, Replay reduces from 14.71% to 5.88% for , but only from 27.45% to 23.53% for , corresponding to relative reductions of 60.0% and 14.3%, respectively. Thus, substantial adversarial target degradation persists despite similarly low control and background forgetting rates after Replay. These results support an empirical distinction between ERD and SCF, showing that an intervention that alleviates forgetting has limited effectiveness against the targeted degradation associated with ERD.
We believe these experiments distinguish the proposed ERD mechanism from selective forgetting. We have added these results to the manuscript in a new Section 4.4.
[A] Xisen Jin and Xiang Ren. "Demystifying language model forgetting with low-rank example associations." In NeurIPS, 2025.
W3. Answer-only supervised fine-tuning baseline and additional statistical analysis.
We evaluate the accuracy scores of a GPT-20B verifier obtained via answer-only fine-tuning () and show the comparison in Table W3.A. It can be observed that the accuracy of is higher than that of in all comparisons, which indicates the effectiveness of debate transcripts. The table and analysis are included in Appendix D of the revised manuscript.
Table W3.A. Effectiveness of debate transcripts for GPT-20B verifier adaptation.
| Dataset | Architecture | Question | Acc | Acc | Acc |
|---|---|---|---|---|---|
| QuALITY-H | GPT-20B | 0.5360 | 0.7432 | 0.7973 | |
| QuALITY-H | GPT-20B | 0.6306 | 0.5676 | 0.5991 | |
| GPQA-H | GPT-20B | 0.5743 | 0.6237 | 0.7822 | |
| GPQA-H | GPT-20B | 0.6238 | 0.4455 | 0.6337 |
We also repeat our experiments multiple times to address the concern about the statistical significance of our results in Table W3.B.
Table W3.B. Counterexample ratios (CR) from repeated fine-tuning runs on QuALITY-H and GPQA-H.
| Run / Statistic | QuALITY-H CR | QuALITY-H CR | QuALITY-H | GPQA-H CR | GPQA-H CR | GPQA-H |
|---|---|---|---|---|---|---|
| 1 | 0.4054 | 0.4640 | 0.0586 | 0.3465 | 0.3960 | 0.0495 |
| 2 | 0.3964 | 0.4685 | 0.0721 | 0.3564 | 0.4158 | 0.0594 |
| 3 | 0.4144 | 0.4640 | 0.0495 | 0.3267 | 0.3861 | 0.0594 |
| 4 | 0.4009 | 0.4595 | 0.0586 | 0.3366 | 0.4059 | 0.0693 |
| 5 | 0.4505 | 0.4640 | 0.0135 | 0.3267 | 0.3960 | 0.0693 |
| Mean | 0.4135 | 0.4640 | 0.0505 | 0.3386 | 0.4000 | 0.0614 |
| SD | 0.0217 | 0.0032 | — | 0.0129 | 0.0113 | — |
| 95% CI | — | — | [0.0229, 0.0780] | — | — | [0.0511, 0.0717] |
Based on Table W3.B, has a CR consistently higher than that of , which demonstrates the robustness of our framework.
Response to Reviewer bodJ (Part 3/3)
W4 and Q1. Motivations behind Equation (2) and Theorem 1.
The design of Equation 2 is motivated by the uniqueness of epistemic reliability degradation. There are two motivations behind this design.
First, normalisation reduces sensitivity to shared response biases when monitored-task gain and hidden-task loss grow proportionally. For example, consider an unstable or poisoned training run that makes the verifier increasingly favour "Yes" across tasks. If "Yes" is correct for the monitored task but "No" is correct for the hidden task, the same bias increases the probability of a correct answer on the monitored task and decreases the probability of a correct answer on the hidden task. As this bias strengthens, the raw hidden task decrease grows, even though the change affects both tasks. When the monitored-task gain and hidden-task loss grow proportionally, normalisation keeps the score unchanged instead of treating it as a sign of stronger relative degradation. This motivates assessing hidden-task losses together with the accompanying monitored-task gains in Equation (2). Second, correct-answer probabilities capture changes in predictive support that top-1 accuracy can miss. A decrease in the probability of the reference-correct answer shifts probability mass away from that answer and increases its log loss. If answers are sampled from the evaluated distribution, this decrease also directly reduces expected correctness. Even when the highest-probability answer remains unchanged, the verifier assigns less support to the correct response, which is the aspect of epistemic reliability that our audit measures.
To further address your concern, we add a comparison with an alternative measurement that retains only the hidden-task probability decrease (denoted as Raw in the following Table W4). The adversarial verifier still exhibits a higher counterexample ratio on both datasets. Thus, the observed phenomenon persists after removing the denominator.
Table W4. Normalisation of the hidden task degradation score.
| Dataset | Degradation Metric | Threshold | CR | CR | Adv-Honest CR Gap |
|---|---|---|---|---|---|
| QuALITY-H | Normalised | 1.2088 | 0.4054 | 0.4640 | 0.0586 |
| QuALITY-H | Raw | 0.1931 | 0.5180 | 0.6036 | 0.0856 |
| GPQA-H | Normalised | 2.8901 | 0.3465 | 0.3960 | 0.0495 |
| GPQA-H | Raw | 0.0722 | 0.6238 | 0.6832 | 0.0594 |
These justifications and experimental results are included in Appendix D of the revised manuscript.
On the other hand, Theorem 1 aims to provide the complementary guarantee when no counterexample is found, with increasing towards one as passing samples accumulate. The lower bound is intended to quantify the non-degradation guarantee supported by a finite audit. A smaller bound indicates weaker statistical assurance, rather than necessarily a lower true non-degradation probability.
Q2. Safety- or security-relevant scenarios in the proposed benchmarks.
Thank you for raising this point. Our benchmarks primarily assess epistemic reliability in general QA, and they partially cover safety-relevant scenarios.
Using a conservative criterion, we classify a hidden question as safety-related only when the question or its correct answer explicitly concerns danger, harm, protection, rescue, disease, pathogen detection, or antibiotic resistance. A wartime setting or a biology/chemistry topic alone does not qualify. Under this criterion, 18 of 222 QuALITY-H questions and 7 of 101 GPQA-H questions relate to safety/security issues. QuALITY-H examples involve evacuation decisions, travel hazards, shortages of air or drinking water, and rescue or protection outcomes. GPQA-H examples concern pathogen detection, antibiotic resistance, and genetic diseases. Constructing our benchmarks requires multiple distinct questions grounded in a shared context. The limited availability of safety- and security-specific datasets with this structure poses a major challenge and motivates our use of QuALITY and GPQA. We will seek to address this data gap in future work.
Despite the inclusion of safety and security questions, we acknowledge that Figure 1 is an illustrative example of our threat model for an upfront explanation of why ERD is considered a risk.
Response to Reviewer bodJ (Part 1/3)
We sincerely appreciate your careful review and feedback. A set of new experiments has been completed to address the weaknesses and answer the questions.
W1. Relationship between monitored and hidden tasks and controlled interventions on adversarial transcripts.
Our benchmarks are constructed around the principle of shared evidence with distinct target propositions. Paired questions are designed to draw on overlapping facts or scientific premises while requiring different conclusions, but are not connected only by topic. Example B.1 from QuALITY-H illustrates this principle. The statement about Wednesday’s debt notice and the complaint written “yesterday” directly supports the monitored answer and supplies a temporal premise needed to derive the hidden answer. The example demonstrates the shared evidential basis required by our benchmark construction procedure and does not merely show that both questions concern the same story.
Moreover, we provide an ablation study on adversarial transcripts to separate the effects of misleading arguments and the GCG suffix. The results are given in Table 3.A below. The results suggest that adversarial transcripts can lead to higher hidden-task accuracy than honest transcripts do, with or without GCG. Notably, our method identified more counterexamples from verifiers adapted on both versions of adversarial transcripts.
Table W1.A. Ablation study on adversarial transcripts.
W2. Epistemic reliability degradation vs. selective catastrophic forgetting.
Thank you for highlighting this point. To address this weakness, we provide extra experiments to separate epistemic reliability degradation (ERD) from selective catastrophic forgetting (SCF). We follow the previous work of Jin & Ren [A] to identify associations between learned tasks and forgotten upstream examples on QuALITY-H, and test whether mitigating SCF also reduces ERD based on the method (i.e., Replay) from Jin & Ren.
We construct a background replay set, Kreplay, and a held-out evaluation set, Keval, using articles and their labelled questions from the training and development splits of QuALITY. The context articles associated with the original 222 question pairs in QuALITY-H are excluded. We then split the selected articles into disjoint Kreplay and Keval containing 17 articles each. We sample up to five questions per article and pair each original correct answer with one distractor from the original incorrect options in QuALITY, and screen for ambiguity and duplication, yielding 79 replay candidates and 84 evaluation questions in the end. Next, for each target hidden question, we select a non-target question from the original labelled questions for the same QuALITY article, excluding monitored questions and known attack targets. Comparing forgetting on target and non-target questions allows us to assess whether degradation is concentrated on the attack targets. We use the background questions in Keval to assess whether Replay improves general knowledge retention beyond the replayed examples. Every eighth training update, we jointly optimise the original fine-tuning loss and a cross-entropy replay loss computed using the ground-truth answers for a batch from Kreplay, keeping the total number of optimiser steps unchanged.
First, we filter out a set of easy-to-forget examples of hidden questions via fine-tuning on honest transcripts, and then compare this set of examples to those hidden task questions exhibiting the greatest ERD after fine-tuning with adversarial transcripts and Replay. We identify 25 hidden questions answered correctly by Vbase but incorrectly by Vhonest, and then select the top 10% of hidden questions with the largest decreases in Pr[Vadv(qh)=ah]. The intersection over union of the two sets is merely 16.13%, suggesting that questions exhibiting the greatest ERD largely differ from those forgotten after honest adaptation.