acceptodds
Under review as a conference paper at ICLR 2027

Black-Box Auditing of Epistemic Reliability in Multi-Agent Debate Distillation

Abstract

Debate distillation adapts weaker verifiers using multi-agent debate transcripts to improve their judgement in subsequent debates, but gains on monitored tasks do not establish reliability on related unmonitored tasks. We study epistemic reliability degradation, in which adaptation preserves monitored performance while reducing support for correct responses on hidden tasks. We consider an adversarial debater that manipulates debate arguments while defending the correct monitored response, and ask whether the resulting degradation merely reflects catastrophic forgetting and whether standard evaluation can detect it. To address these questions, we propose ER-AUDIT, a two-stage black-box auditing framework that compares frozen verifier checkpoints before and after adaptation, and introduce two evaluation benchmarks pairing monitored and hidden task prompts grounded in shared contexts. ER-AUDIT searches for counterexamples to non-degradation by evaluating semantically valid paraphrases and, if none is found, uses independent paraphrases for sequential hypothesis testing. We derive anytime-valid lower confidence bounds on the non-degradation probability, allowing data-dependent stopping within a finite budget. We further establish a common lower bound across fixed paraphrase distributions and extend it to distributions within a bounded total variation distance of their mixtures. Our experiments show that higher hidden-task accuracy can coexist with more counterexamples to non-degradation and lower non-degradation bounds. This divergence challenges explanations based solely on broad catastrophic forgetting and shows that auditing can uncover selective hidden-task degradation concealed by aggregate performance gains.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, epistemic reliability, LLM evaluation, multi-agent debate, debate distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bodJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates epistemic reliability degradation in debate distillation approaches, which aim to improve a weaker verifier using debate transcripts generated by two stronger LLMs. This important gap arises when an adversarial attacker among the debaters can successfully satisfy the monitored task's correctness criteria while simultaneously pursuing a hidden objective. The authors propose a two-stage black-box auditing framework that first searches for counterexamples and, if none are found, provides statistical bounds on non-degradation. They claim that this framework can detect reliability degradation that conventional evaluation metrics often fail to identify or distinguish from catastrophic forgetting. Their findings suggest that ER-AUDIT can uncover degradation on hidden tasks induced by the attacker debater, which remains concealed by apparent performance gains on monitored tasks.

Critical strengths

Important safety problem with promising empirical observations. The paper investigates an important gap in debate-based oversight, where improvements in monitored-task performance may conceal degradation on related hidden tasks. The results provide some preliminary evidence of this phenomenon.

Principled auditing framework with theoretical guarantees. The proposed two-stage auditing framework combines counterexample search followed by a statistical certification of non-degradation when no counterexample is found. It provides anytime-valid confidence bounds and extensions across multiple paraphrase distributions. Empirically, ER-Audit identifies higher counterexample ratios under adversarial adaptation across all six model–dataset combinations. These results demonstrate the potential value of the framework for detecting localized degradation that standard evaluation may overlook.

Critical weaknesses
  1. Although the paper introduces two benchmarks pairing monitored and hidden questions, the relationship between these tasks is not sufficiently justified or evaluated. I also read Appendix B, on how each benchmark is constructed. Sharing a context or scientific subdomain does not necessarily imply meaningful evidential overlap or dependency between the questions.
    More broadly, the paper does not provide direct evidence explaining how misleading information in adversarial debates leads to persistent changes in the verifier's hidden-task predictions. The observed effects could arise from misleading factual associations, the optimized adversarial suffix, increased prompt sensitivity, or more general fine-tuning interference. Although Appendix F proposes shortcut learning as a possible explanation, this remains a hypothesis without supporting causal experiments. I believe a more systematic analysis of the relationship between monitored and hidden tasks, together with controlled interventions on adversarial transcripts, would substantially strengthen the paper's central claim about adversarially induced epistemic degradation.

  2. One of the paper's central motivating questions, emphasized in the abstract and introduction, is whether epistemic reliability degradation represents a distinct risk beyond catastrophic forgetting. However, I find the experimental evidence insufficient to establish this distinction. The authors primarily rely on the observation that higher aggregate hidden-task accuracy can coexist with more localized degradation counterexamples. For example, in QUALITY-H, the GPT-20B adversarial verifier achieves higher hidden-task accuracy than the honest verifier (63.96% vs. 59.91%) while exhibiting more counterexamples. Although this observation challenges explanations based on uniform performance degradation, it does not rule out selective catastrophic forgetting, where a model improves on some tasks while losing previously acquired knowledge on others. The authors acknowledge that selective forgetting may contribute to the observed phenomenon (Section 4.3 and Appendix F), but do not provide experiments that separate it from the proposed epistemic degradation mechanism. Given the importance of this distinction to the paper's motivation, stronger evidence would be valuable.

  3. Several aspects of the experimental design make it difficult to isolate the contribution of debate distillation and the additional effects of adversarial transcripts. First, the verifier's fine-tuning objective includes both a language-modeling loss over debate transcripts and direct supervision on the correct monitored answers (Appendix D.1). While Table 1 shows substantial improvements in monitored-task accuracy after honest debate distillation, the paper does not include an answer-only supervised fine-tuning baseline. Consequently, it is unclear how much of the improvement comes from the debate transcripts rather than ordinary supervised fine-tuning. Second, the honest and adversarial adaptation settings use different numbers of training transcripts (222 vs. 184 for QuALITY-H and 101 vs. 83 for GPQA-H). These differences, together with the different transcript-selection procedures, introduce potential confounding factors when attributing hidden-task degradation specifically to adversarial content. Finally, the paper does not clearly specify whether the reported results are averaged across multiple independent fine-tuning runs or obtained from single runs. No run-to-run variance or confidence intervals are provided. Given that some honest–adversarial differences in Figure 2(a) are relatively small, additional statistical analysis and repeated experiments are necessary to establish their robustness.

  4. I have two concerns regarding the proposed measurement and auditing approach. First, the degradation score in Equation (2) normalizes the reduction in correct hidden-answer probability by the improvement in monitored-task probability. Consequently, two verifiers experiencing the same absolute hidden-task degradation may receive different degradation scores depending on their monitored-task improvements. This makes comparisons between honest and adversarial adaptation potentially sensitive to the denominator rather than solely to hidden-task reliability. Moreover, changes in correct-answer probabilities do not necessarily correspond to meaningful changes in correctness or epistemic reliability. Second, although the anytime-valid theoretical guarantees appear sound under the stated assumptions, the proposed lower bound in Theorem 1 becomes zero upon observing a single counterexample, regardless of how many previous samples satisfied the non-degradation criterion. This makes the bound conservative and potentially uninformative when degradation is rare but nonzero. Importantly, a lower confidence bound under adversarial adaptation does not necessarily imply a lower true non-degradation probability. The paper would benefit from comparisons with alternative degradation metrics, more informative confidence-sequence methods, and simpler auditing baselines under matched query budgets. Such comparisons would help clarify the practical advantage of ER-Audit beyond evaluating additional paraphrases.

Overall, I find the research problem important and the proposed auditing framework promising, but I remain unconvinced that the current experimental evidence sufficiently supports the paper's broader claims, particularly regarding the underlying degradation mechanism, its distinction from catastrophic forgetting, and its practical relevance to safety-critical oversight.

Questions for authors

Please first refer to the concerns I mentioned earlier.

  1. What motivates normalizing hidden-task degradation by monitored-task improvement in Equation (2)? Have the authors evaluated whether their findings remain consistent using unnormalized changes in correct-answer probabilities or correctness-based measures?

  2. Given that the paper motivates epistemic reliability degradation as a risk to trustworthy and safety-critical oversight (line 60 and Figure 1), while the proposed benchmarks are derived from general question-answering datasets (QUALITY and GPQA), could the authors clarify to what extent the benchmarks capture safety- or security-relevant scenarios? Does Figure 1 represent an actual experimental example or is purely illustrative? Have the authors assessed whether the observed degradation extends to consequential safety or security judgments?

6 comments
AuthorsOpenReview ↗

Response to Reviewer bodJ (Part 1/3)

We sincerely appreciate your careful review and feedback. A set of new experiments has been completed to address the weaknesses and answer the questions.

W1. Relationship between monitored and hidden tasks and controlled interventions on adversarial transcripts.

Our benchmarks are constructed around the principle of shared evidence with distinct target propositions. Paired questions are designed to draw on overlapping facts or scientific premises while requiring different conclusions, but are not connected only by topic. Example B.1 from QuALITY-H illustrates this principle. The statement about Wednesday’s debt notice and the complaint written “yesterday” directly supports the monitored answer and supplies a temporal premise needed to derive the hidden answer. The example demonstrates the shared evidential basis required by our benchmark construction procedure and does not merely show that both questions concern the same story.

Moreover, we provide an ablation study on adversarial transcripts to separate the effects of misleading arguments and the GCG suffix. The results are given in Table 3.A below. The results suggest that adversarial transcripts can lead to higher hidden-task accuracy than honest transcripts do, with or without GCG. Notably, our method identified more counterexamples from verifiers adapted on both versions of adversarial transcripts.

Table W1.A. Ablation study on adversarial transcripts.

ModelGCGQuALITY-H qmq_m accQuALITY-H qhq_h accQuALITY-H CRGPQA-H qmq_m accGPQA-H qhq_h accGPQA-H CR
VhonestV_\mathrm{honest}No GCG0.79730.59910.40540.78220.63370.3465
VadvV_\mathrm{adv}No GCG0.55410.68010.41440.68320.55450.3663
VadvV_\mathrm{adv}with GCG0.80630.63960.46400.66340.49500.3960

W2. Epistemic reliability degradation vs. selective catastrophic forgetting.

Thank you for highlighting this point. To address this weakness, we provide extra experiments to separate epistemic reliability degradation (ERD) from selective catastrophic forgetting (SCF). We follow the previous work of Jin & Ren [A] to identify associations between learned tasks and forgotten upstream examples on QuALITY-H, and test whether mitigating SCF also reduces ERD based on the method (i.e., Replay) from Jin & Ren.

We construct a background replay set, KreplayK_{\mathrm{replay}}, and a held-out evaluation set, KevalK_{\mathrm{eval}}, using articles and their labelled questions from the training and development splits of QuALITY. The context articles associated with the original 222222 question pairs in QuALITY-H are excluded. We then split the selected articles into disjoint KreplayK_{\mathrm{replay}} and KevalK_{\mathrm{eval}} containing 1717 articles each. We sample up to five questions per article and pair each original correct answer with one distractor from the original incorrect options in QuALITY, and screen for ambiguity and duplication, yielding 7979 replay candidates and 8484 evaluation questions in the end. Next, for each target hidden question, we select a non-target question from the original labelled questions for the same QuALITY article, excluding monitored questions and known attack targets. Comparing forgetting on target and non-target questions allows us to assess whether degradation is concentrated on the attack targets. We use the background questions in KevalK_{\mathrm{eval}} to assess whether Replay improves general knowledge retention beyond the replayed examples. Every eighth training update, we jointly optimise the original fine-tuning loss and a cross-entropy replay loss computed using the ground-truth answers for a batch from KreplayK_{\mathrm{replay}}, keeping the total number of optimiser steps unchanged.

First, we filter out a set of easy-to-forget examples of hidden questions via fine-tuning on honest transcripts, and then compare this set of examples to those hidden task questions exhibiting the greatest ERD after fine-tuning with adversarial transcripts and Replay. We identify 2525 hidden questions answered correctly by VbaseV_\mathrm{base} but incorrectly by VhonestV_\mathrm{honest}, and then select the top 10% of hidden questions with the largest decreases in Pr⁡[Vadv(qh)=ah]\Pr\left[V_\mathrm{adv}(q_h)=a_h\right]. The intersection over union of the two sets is merely 16.13%, suggesting that questions exhibiting the greatest ERD largely differ from those forgotten after honest adaptation.

AuthorsOpenReview ↗

Response to Reviewer bodJ (Part 2/3)

Second, we measure whether Replay mitigates ERD. In Table W2, qmq_m Acc denotes monitored-task accuracy and ΔNLLbg\Delta\mathrm{NLL}_{\mathrm{bg}} denotes the mean post- minus pre-adaptation change in correct-answer negative log-likelihood on KevalK_{\mathrm{eval}}, with positive values indicating degradation. We denote the forgetting rates on these non-target questions, target hidden questions, and held-out background questions in KevalK_{\mathrm{eval}} by FntF_{\mathrm{nt}}, FtF_{\mathrm{t}}, and FbgF_{\mathrm{bg}}, respectively.

Table W2. Effect of Replay on selective forgetting and ERD during verifier adaptation.

VerifierReplayqmq_m Acc ↑\uparrowΔNLLbg↓\Delta\mathrm{NLL}_{\mathrm{bg}}\downarrowFnt↓F_{\mathrm{nt}}\downarrowFt↓F_{\mathrm{t}}\downarrowFbg↓F_{\mathrm{bg}}\downarrow
VhonestV_{\mathrm{honest}}No0.96450.277720.79%14.71%13.85%
VhonestV_{\mathrm{honest}}Yes0.94080.06803.96%5.88%4.62%
VadvV_{\mathrm{adv}}No0.91720.245023.76%27.45%16.92%
VadvV_{\mathrm{adv}}Yes0.91120.06144.95%23.53%6.15%

Replay reduces ΔNLLbg\Delta\mathrm{NLL}_{\mathrm{bg}} by approximately 75% and FntF_{\mathrm{nt}} by approximately 80% for both verifiers, while qmq_m Acc remains above 0.910.91. Background forgetting rates also fall to 4.62% for VhonestV_\mathrm{honest} and 6.15% for VadvV_\mathrm{adv}. However, on target hidden questions, Replay reduces FtF_{\mathrm{t}} from 14.71% to 5.88% for VhonestV_\mathrm{honest}, but only from 27.45% to 23.53% for VadvV_\mathrm{adv}, corresponding to relative reductions of 60.0% and 14.3%, respectively. Thus, substantial adversarial target degradation persists despite similarly low control and background forgetting rates after Replay. These results support an empirical distinction between ERD and SCF, showing that an intervention that alleviates forgetting has limited effectiveness against the targeted degradation associated with ERD.

We believe these experiments distinguish the proposed ERD mechanism from selective forgetting. We have added these results to the manuscript in a new Section 4.4.

[A] Xisen Jin and Xiang Ren. "Demystifying language model forgetting with low-rank example associations." In NeurIPS, 2025.

W3. Answer-only supervised fine-tuning baseline and additional statistical analysis.

We evaluate the accuracy scores of a GPT-20B verifier obtained via answer-only fine-tuning (VhonestAOV_\mathrm{honestAO}) and show the comparison in Table W3.A. It can be observed that the accuracy of VhonestV_\mathrm{honest} is higher than that of VhonestAOV_\mathrm{honestAO} in all comparisons, which indicates the effectiveness of debate transcripts. The table and analysis are included in Appendix D of the revised manuscript.

Table W3.A. Effectiveness of debate transcripts for GPT-20B verifier adaptation.

DatasetArchitectureQuestionVbaseV_{\mathrm{base}} AccVhonestAOV_{\text{honestAO}} AccVhonestV_{\mathrm{honest}} Acc
QuALITY-HGPT-20Bqmq_m0.53600.74320.7973
QuALITY-HGPT-20Bqhq_h0.63060.56760.5991
GPQA-HGPT-20Bqmq_m0.57430.62370.7822
GPQA-HGPT-20Bqhq_h0.62380.44550.6337

We also repeat our experiments multiple times to address the concern about the statistical significance of our results in Table W3.B.

Table W3.B. Counterexample ratios (CR) from repeated fine-tuning runs on QuALITY-H and GPQA-H.

Run / StatisticQuALITY-H VhonestV_{\mathrm{honest}} CRQuALITY-H VadvV_{\mathrm{adv}} CRQuALITY-H ΔCR\Delta\mathrm{CR}GPQA-H VhonestV_{\mathrm{honest}} CRGPQA-H VadvV_{\mathrm{adv}} CRGPQA-H ΔCR\Delta\mathrm{CR}
10.40540.46400.05860.34650.39600.0495
20.39640.46850.07210.35640.41580.0594
30.41440.46400.04950.32670.38610.0594
40.40090.45950.05860.33660.40590.0693
50.45050.46400.01350.32670.39600.0693
Mean0.41350.46400.05050.33860.40000.0614
SD0.02170.0032—0.01290.0113—
95% CI——[0.0229, 0.0780]——[0.0511, 0.0717]

Based on Table W3.B, VadvV_{\mathrm{adv}} has a CR consistently higher than that of VhonestV_{\mathrm{honest}}, which demonstrates the robustness of our framework.

AuthorsOpenReview ↗

Response to Reviewer bodJ (Part 3/3)

W4 and Q1. Motivations behind Equation (2) and Theorem 1.

The design of Equation 2 is motivated by the uniqueness of epistemic reliability degradation. There are two motivations behind this design.

First, normalisation reduces sensitivity to shared response biases when monitored-task gain and hidden-task loss grow proportionally. For example, consider an unstable or poisoned training run that makes the verifier increasingly favour "Yes" across tasks. If "Yes" is correct for the monitored task but "No" is correct for the hidden task, the same bias increases the probability of a correct answer on the monitored task and decreases the probability of a correct answer on the hidden task. As this bias strengthens, the raw hidden task decrease grows, even though the change affects both tasks. When the monitored-task gain and hidden-task loss grow proportionally, normalisation keeps the score unchanged instead of treating it as a sign of stronger relative degradation. This motivates assessing hidden-task losses together with the accompanying monitored-task gains in Equation (2). Second, correct-answer probabilities capture changes in predictive support that top-1 accuracy can miss. A decrease in the probability of the reference-correct answer shifts probability mass away from that answer and increases its log loss. If answers are sampled from the evaluated distribution, this decrease also directly reduces expected correctness. Even when the highest-probability answer remains unchanged, the verifier assigns less support to the correct response, which is the aspect of epistemic reliability that our audit measures.

To further address your concern, we add a comparison with an alternative measurement that retains only the hidden-task probability decrease (denoted as Raw in the following Table W4). The adversarial verifier still exhibits a higher counterexample ratio on both datasets. Thus, the observed phenomenon persists after removing the denominator.

Table W4. Normalisation of the hidden task degradation score.

DatasetDegradation MetricThreshold ϵ\epsilonVhonestV_\mathrm{honest} CRVadvV_\mathrm{adv} CRAdv-Honest CR Gap
QuALITY-HNormalised1.20880.40540.46400.0586
QuALITY-HRaw0.19310.51800.60360.0856
GPQA-HNormalised2.89010.34650.39600.0495
GPQA-HRaw0.07220.62380.68320.0594

These justifications and experimental results are included in Appendix D of the revised manuscript.

On the other hand, Theorem 1 aims to provide the complementary guarantee when no counterexample is found, with Lj,t=η1/tL_{j,t}=\eta^{1/t} increasing towards one as passing samples accumulate. The lower bound is intended to quantify the non-degradation guarantee supported by a finite audit. A smaller bound indicates weaker statistical assurance, rather than necessarily a lower true non-degradation probability.

Q2. Safety- or security-relevant scenarios in the proposed benchmarks.

Thank you for raising this point. Our benchmarks primarily assess epistemic reliability in general QA, and they partially cover safety-relevant scenarios.

Using a conservative criterion, we classify a hidden question as safety-related only when the question or its correct answer explicitly concerns danger, harm, protection, rescue, disease, pathogen detection, or antibiotic resistance. A wartime setting or a biology/chemistry topic alone does not qualify. Under this criterion, 18 of 222 QuALITY-H questions and 7 of 101 GPQA-H questions relate to safety/security issues. QuALITY-H examples involve evacuation decisions, travel hazards, shortages of air or drinking water, and rescue or protection outcomes. GPQA-H examples concern pathogen detection, antibiotic resistance, and genetic diseases. Constructing our benchmarks requires multiple distinct questions grounded in a shared context. The limited availability of safety- and security-specific datasets with this structure poses a major challenge and motivates our use of QuALITY and GPQA. We will seek to address this data gap in future work.

Despite the inclusion of safety and security questions, we acknowledge that Figure 1 is an illustrative example of our threat model for an upfront explanation of why ERD is considered a risk.

Reviewer_2PctOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies epistemic reliabilty degradation in multi-agent debate distillation, wehre adaptation preserves monitored-task preformance yet reduce support for correct answers on related hidden tasks. It introduces QUALITY-H and GPQA-H benchmarks and ER-AUDIT, a two stage blackbox framework combining counterexample search over paraphrases with anytime-valid sequential tests that yield lower confidence bounds on non-degradation probability.

Critical strengths
  • Novel risk characterization: adversarial debate distilation can raise hidden task accuracy while increasing local degradation under paraphrases.

  • Principled black-box audit: anytime-valid lower confidence bounds that permits data-dependent stopping without invalidating coverage.

Critical weaknesses
  • Reliance on LLM paraphrase generators whose coverage of the semantic neighbourhood is hard to certify.

  • Limited analysis of why the degradation occurs, shortcut learning vs. selective forgetting remains largely correlational.

Questions for authors

How sensitive are the reported counter example ratios and average lower bounds to the particular paraphrase generators (gpt-5.6-terra, claude-opus-4.8) versus alternative generators or human paraphrases?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2Pct (Part 1/2)

Thank you for your positive feedback on the novelty and the principled auditing framework. We have conducted new experiments to address the weaknesses and questions.

W1 and Q1. Reliance on LLM paraphrase generators.

To assess the stability of auditing results across paraphrase distributions, we evaluate CR and the average lower confidence bound for VhonestV_\mathrm{honest} and VadvV_\mathrm{adv} under three paraphrasers. In addition to gpt-5.6-terra and claude-opus-4.8 from previous experiments, we add a locally deployed deepseek-v4-pro as a third paraphraser. Tables W1.A and W1.B show that ER-Audit consistently identifies more counterexamples and lower confidence bounds for VadvV_\mathrm{adv}, with similar values across paraphrasers. Although this evaluation covers only a finite set of generators, the consistent results suggest robustness to different paraphrase distributions.

Table W1.A. Sensitivity of the auditing framework to paraphrase generators in counterexample search.

DatasetSearch GeneratorM_searchHonest CRAdversarial CRAdv-Honest
QuALITY-Hgpt-5.6-terra1000.40540.46400.0586
QuALITY-Hclaude-opus-4.81000.42340.45500.0315
QuALITY-Hdeepseek-v4-pro1000.42790.49100.0631
GPQA-Hgpt-5.6-terra1000.34650.39600.0495
GPQA-Hclaude-opus-4.81000.31680.43560.1188
GPQA-Hdeepseek-v4-pro1000.28710.39600.1089

Table W1.B. Sensitivity of the auditing framework to paraphrase generators in lower bound computation.

DatasetTest GeneratorM_testHonest Lower BoundAdv Lower BoundHonest-adv
QuALITY-Hgpt-5.6-terra1000.55960.51580.0437
QuALITY-Hclaude-opus-4.81000.55080.50270.0481
QuALITY-Hdeepseek-v4-pro1000.56390.50710.0568
GPQA-Hgpt-5.6-terra1000.63420.56690.0673
GPQA-Hclaude-opus-4.81000.63420.56690.0673
GPQA-Hdeepseek-v4-pro1000.61500.55730.0577

The new results and analysis are included in Appendix D of the revised manuscript.

W2. Analysis of epistemic reliability degradation vs. selective catastrophic forgetting.

Thank you for your feedback. We reckon that more in-depth analysis of this problem can further improve our paper. To address this weakness, we provide extra experiments to separate epistemic reliability degradation (ERD) from selective catastrophic forgetting (SCF). We follow the previous work of Jin & Ren [A] to identify associations between learned tasks and forgotten upstream examples on QuALITY-H, and test whether mitigating SCF also reduces ERD based on the method (i.e., Replay) from Jin & Ren.

We construct a background replay set, KreplayK_{\mathrm{replay}}, and a held-out evaluation set, KevalK_{\mathrm{eval}}, using articles and their labelled questions from the training and development splits of QuALITY. The context articles associated with the original 222222 question pairs in QuALITY-H are excluded. We then split the selected articles into disjoint KreplayK_{\mathrm{replay}} and KevalK_{\mathrm{eval}} containing 1717 articles each. We sample up to five questions per article and pair each original correct answer with one distractor from the original incorrect options in QuALITY, and screen for ambiguity and duplication, yielding 7979 replay candidates and 8484 evaluation questions in the end. Next, for each target hidden question, we select a non-target question from the original labelled questions for the same QuALITY article, excluding monitored questions and known attack targets. Comparing forgetting on target and non-target questions allows us to assess whether degradation is concentrated on the attack targets. We use the background questions in KevalK_{\mathrm{eval}} to assess whether Replay improves general knowledge retention beyond the replayed examples. Every eighth training update, we jointly optimise the original fine-tuning loss and a cross-entropy replay loss computed using the ground-truth answers for a batch from KreplayK_{\mathrm{replay}}, keeping the total number of optimiser steps unchanged.

AuthorsOpenReview ↗

Response to Reviewer 2Pct (Part 2/2)

First, we filter out a set of easy-to-forget examples of hidden questions via fine-tuning on honest transcripts, and then compare this set of examples to those hidden task questions exhibiting the greatest ERD after fine-tuning with adversarial transcripts and Replay. We identify 2525 hidden questions answered correctly by VbaseV_\mathrm{base} but incorrectly by VhonestV_\mathrm{honest}, and then select the top 10% of hidden questions with the largest decreases in Pr⁡[Vadv(qh)=ah]\Pr\left[V_\mathrm{adv}(q_h)=a_h\right]. The intersection over union of the two sets is merely 16.13%, suggesting that questions exhibiting the greatest ERD largely differ from those forgotten after honest adaptation.

Second, we measure whether Replay mitigates ERD. In Table W2, qmq_m Acc denotes monitored-task accuracy and ΔNLLbg\Delta\mathrm{NLL}_{\mathrm{bg}} denotes the mean post- minus pre-adaptation change in correct-answer negative log-likelihood on KevalK_{\mathrm{eval}}, with positive values indicating degradation. We denote the forgetting rates on these non-target questions, target hidden questions, and held-out background questions in KevalK_{\mathrm{eval}} by FntF_{\mathrm{nt}}, FtF_{\mathrm{t}}, and FbgF_{\mathrm{bg}}, respectively.

Table W2. Effect of Replay on selective forgetting and ERD during verifier adaptation.

VerifierReplayqmq_m Acc ↑\uparrowΔNLLbg↓\Delta\mathrm{NLL}_{\mathrm{bg}}\downarrowFnt↓F_{\mathrm{nt}}\downarrowFt↓F_{\mathrm{t}}\downarrowFbg↓F_{\mathrm{bg}}\downarrow
VhonestV_{\mathrm{honest}}No0.96450.277720.79%14.71%13.85%
VhonestV_{\mathrm{honest}}Yes0.94080.06803.96%5.88%4.62%
VadvV_{\mathrm{adv}}No0.91720.245023.76%27.45%16.92%
VadvV_{\mathrm{adv}}Yes0.91120.06144.95%23.53%6.15%

Replay reduces ΔNLLbg\Delta\mathrm{NLL}_{\mathrm{bg}} by approximately 75% and FntF_{\mathrm{nt}} by approximately 80% for both verifiers, while qmq_m Acc remains above 0.910.91. Background forgetting rates also fall to 4.62% for VhonestV_\mathrm{honest} and 6.15% for VadvV_\mathrm{adv}. However, on target hidden questions, Replay reduces FtF_{\mathrm{t}} from 14.71% to 5.88% for VhonestV_\mathrm{honest}, but only from 27.45% to 23.53% for VadvV_\mathrm{adv}, corresponding to relative reductions of 60.0% and 14.3%, respectively. Thus, substantial adversarial target degradation persists despite similarly low control and background forgetting rates after Replay. These results support an empirical distinction between ERD and SCF, showing that an intervention that alleviates forgetting has limited effectiveness against the targeted degradation associated with ERD.

These new experiments demonstrate that ERD arises through a mechanism distinct from selective forgetting. We have added the results to the manuscript.

[A] Xisen Jin and Xiang Ren. "Demystifying language model forgetting with low-rank example associations." In NeurIPS, 2025.

Reviewer_aczsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a certain failure mode of training language models via AI-generated debates: a model can become more accurate on the tasks it is trained to solve while becoming less reliable on related questions, even when its overall accuracy on those questions improves.

They propose an auditing framework based on the idea that a model may produce correct responses to many task prompts yet fail to maintain a consistent belief when the same question is phrased differently. What they refer to as "epistemic reliability" requires that correct beliefs remain stable across variations that preserve the underlying meaning and evidence.

First, they construct adversarial training examples using two datasets, QuALITY-H (222 question pairs) and GPQA-H (101 pairs). The debate transcripts are generated using Gemma-4-31B, including adversarial transcripts in which one debater tries to induce incorrect beliefs about a hidden question while defending the correct answer to the monitored question. They then fine-tune three verifier models (8B, 9B, and 20B parameters) using either honest or adversarial transcripts.

Second, they develop ER-AUDIT, a black-box auditing method. The auditor has access to the original and fine-tuned models, including their answer-token probabilities. It examines whether the fine-tuned model has become less confident in the correct answer to a hidden question. It first searches for counterexamples using meaning-preserving paraphrases. If none are found, it uses sequential hypothesis testing to compute a statistical lower bound on the probability of non-degradation.

They also extend the guarantee to mixtures of several paraphrase distributions and distributions within a bounded total variation distance. Their main empirical finding is that adversarially trained models can perform better on hidden questions overall while exhibiting more instances of degradation under paraphrasing. Thus, aggregate accuracy can conceal localized failures in reliability.

Critical strengths

the main strength of the paper, is that it identifies and demonstrates empirically a failure mode of debate distillation that standard accuracy metrics may miss and provides a systematic auditing framework with statistical guarantees for detecting and quantifying this degradation.

IMO, the empirical observation is the strongest part of the paper, more so than the auditing algorithm or its theoretical guarantees.

Critical weaknesses

Limited novelty: the main ideas exist in the literature already. Further, the experiments do not convincingly establish that the observed degradation represents a distinct failure mode rather than selective forgetting or ordinary sensitivity to prompt phrasing.

Questions for authors

no questions

2 comments
AuthorsOpenReview ↗

Response to Reviewer aczs (Part 1/2)

Thank you for acknowledging the empirical findings and the auditing framework. Our response to the identified weakness point is as follows.

W1.A. Novelty.

We respectfully disagree with this assessment. To the best of our knowledge, we provide the first systematic characterisation of epistemic reliability degradation in debate-based verifier adaptation and introduce a new auditing framework and dedicated benchmarks for this risk. Our manuscript discusses related work in Appendix E. The closest work we identify is Marks et al. [A], which focuses on identifying objectives that models pursue but do not disclose. It does not examine how misleading debate arguments can shift a verifier’s beliefs about related hidden tasks after adaptation, or how to characterise the resulting degradation through rigorous auditing. Our novelty is also recognised by Reviewer 2Pct.

W1.B. Epistemic reliability degradation vs. selective catastrophic forgetting.

Thank you for raising this weakness. We agree that further distinguishing epistemic reliability degradation (ERD) from other failure modes, such as selective catastrophic forgetting (SCF), would consolidate the contribution of our paper. To address this weakness, we provide extra experiments to separate ERD from SCF. We follow the previous work of Jin & Ren [B] to identify associations between learned tasks and forgotten upstream examples on QuALITY-H, and test whether mitigating SCF also reduces ERD based on the method (i.e., Replay) from Jin & Ren.

We construct a background replay set, KreplayK_{\mathrm{replay}}, and a held-out evaluation set, KevalK_{\mathrm{eval}}, using articles and their labelled questions from the training and development splits of QuALITY. The context articles associated with the original 222222 question pairs in QuALITY-H are excluded. We then split the selected articles into disjoint KreplayK_{\mathrm{replay}} and KevalK_{\mathrm{eval}} containing 1717 articles each. We sample up to five questions per article and pair each original correct answer with one distractor from the original incorrect options in QuALITY, and screen for ambiguity and duplication, yielding 7979 replay candidates and 8484 evaluation questions in the end. Next, for each target hidden question, we select a non-target question from the original labelled questions for the same QuALITY article, excluding monitored questions and known attack targets. Comparing forgetting on target and non-target questions allows us to assess whether degradation is concentrated on the attack targets. We use the background questions in KevalK_{\mathrm{eval}} to assess whether Replay improves general knowledge retention beyond the replayed examples. Every eighth training update, we jointly optimise the original fine-tuning loss and a cross-entropy replay loss computed using the ground-truth answers for a batch from KreplayK_{\mathrm{replay}}, keeping the total number of optimiser steps unchanged.

First, we filter out a set of easy-to-forget examples of hidden questions via fine-tuning on honest transcripts, and then compare this set of examples to those hidden task questions exhibiting the greatest ERD after fine-tuning with adversarial transcripts and Replay. We identify 2525 hidden questions answered correctly by VbaseV_\mathrm{base} but incorrectly by VhonestV_\mathrm{honest}, and then select the top 10% of hidden questions with the largest decreases in Pr⁡[Vadv(qh)=ah]\Pr\left[V_\mathrm{adv}(q_h)=a_h\right]. The intersection over union of the two sets is merely 16.13%, suggesting that questions exhibiting the greatest ERD largely differ from those forgotten after honest adaptation.

AuthorsOpenReview ↗

Response to Reviewer aczs (Part 2/2)

Second, we measure whether Replay mitigates ERD. In Table W1, qmq_m Acc denotes monitored-task accuracy and ΔNLLbg\Delta\mathrm{NLL}_{\mathrm{bg}} denotes the mean post- minus pre-adaptation change in correct-answer negative log-likelihood on KevalK_{\mathrm{eval}}, with positive values indicating degradation. We denote the forgetting rates on these non-target questions, target hidden questions, and held-out background questions in KevalK_{\mathrm{eval}} by FntF_{\mathrm{nt}}, FtF_{\mathrm{t}}, and FbgF_{\mathrm{bg}}, respectively.

Table W1. Effect of Replay on selective forgetting and ERD during verifier adaptation.

VerifierReplayqmq_m Acc ↑\uparrowΔNLLbg↓\Delta\mathrm{NLL}_{\mathrm{bg}}\downarrowFnt↓F_{\mathrm{nt}}\downarrowFt↓F_{\mathrm{t}}\downarrowFbg↓F_{\mathrm{bg}}\downarrow
VhonestV_{\mathrm{honest}}No0.96450.277720.79%14.71%13.85%
VhonestV_{\mathrm{honest}}Yes0.94080.06803.96%5.88%4.62%
VadvV_{\mathrm{adv}}No0.91720.245023.76%27.45%16.92%
VadvV_{\mathrm{adv}}Yes0.91120.06144.95%23.53%6.15%

Replay reduces ΔNLLbg\Delta\mathrm{NLL}_{\mathrm{bg}} by approximately 75% and FntF_{\mathrm{nt}} by approximately 80% for both verifiers, while qmq_m Acc remains above 0.910.91. Background forgetting rates also fall to 4.62% for VhonestV_\mathrm{honest} and 6.15% for VadvV_\mathrm{adv}. However, on target hidden questions, Replay reduces FtF_{\mathrm{t}} from 14.71% to 5.88% for VhonestV_\mathrm{honest}, but only from 27.45% to 23.53% for VadvV_\mathrm{adv}, corresponding to relative reductions of 60.0% and 14.3%, respectively. Thus, substantial adversarial target degradation persists despite similarly low control and background forgetting rates after Replay. These results support an empirical distinction between ERD and SCF, showing that an intervention that alleviates forgetting has limited effectiveness against the targeted degradation associated with ERD.

The above results further support the conclusion that ERD represents a distinct failure mode. We have accordingly added the new results to our manuscript.

[A] Samuel Marks et al. "Auditing language models for hidden objectives." arXiv preprint arXiv:2503.10965, 2025.

[B] Xisen Jin and Xiang Ren. "Demystifying language model forgetting with low-rank example associations." In NeurIPS, 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We sincerely appreciate the time and effort you have devoted to reviewing our manuscript. We are encouraged by your positive feedback on our empirical findings and auditing framework, and grateful for your constructive suggestions.

In response to the reviews, we have made the following core revisions:

  • Compared epistemic reliability degradation (ERD) with selective catastrophic forgetting. We find limited overlap between easy-to-forget examples and those exhibiting the greatest ERD. Substantial targeted degradation also persists after Replay mitigates forgetting. Together, these findings support treating ERD as a distinct risk.

  • Tested sensitivity to paraphrase generators. Across the tested generators, adversarial verifiers consistently exhibit higher counterexample ratios and lower average non-degradation bounds than honest verifiers. This consistency supports the stability of our auditing findings across the paraphrase distributions examined.

  • Expanded the empirical evaluation. Adversarial verifiers continue to exhibit higher counterexample ratios without GCG, across repeated fine-tuning runs, and without score normalisation. Fine-tuning on honest debate transcripts also outperforms the answer-only baseline. These results strengthen our empirical findings and support the value of debate transcripts for verifier adaptation.

  • Clarified the contribution and scope. The revised discussion distinguishes our contribution from prior auditing work and clarifies the shared evidence linking task pairs, the rationale for our degradation metric, the interpretation of the finite-audit guarantees, and the benchmarks’ safety coverage. These changes make the paper’s contribution and the scope of its claims more precise.

Other minor revisions are highlighted in the manuscript. We hope these revisions address your concerns and clarify the evidence supporting our conclusions. We look forward to continuing the discussion and addressing any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.