acceptodds
Under review as a conference paper at ICLR 2027

CAT-alyzing Honesty: Continuous Adversarial Training for Transferable LLM Honesty

Abstract

Large language models (LLMs) are becoming prevalent in scenarios where honesty is critical. However, despite alignment training, these models still exhibit dishonest behaviour, such as outputting claims that contradict some knowledge or are not supported by what is accessible in context. We introduce Continuous Adversarial Training for Consistent Honesty (CATCH), inspired by a method originally developed to robustly refuse harmful queries. CATCH trains the model towards an honest answer and away from a dishonest one while maintaining the model's capabilities. We evaluate two architectures (Llama-3.3-70B-Instruct and Qwen3-3.6-27B). We show that trained with only hundreds of targets built from self-reporting scenarios, CATCH learns an honest behaviour that robustly generalizes and transfers across distinct benchmarks. It consistently reduces dishonesty under pressure on MASK, increases confessions in the multi-agent game of Among Us, and improves the discovery rates of the model's hidden behaviours in AuditBench. We show that CATCH outperforms prompting and supervised fine-tuning in improving honesty, and it is on par with state-of-the-art honesty activation steering. However, CATCH surpasses honesty steering in maintaining utility, thanks to a novel approach to generating utility data.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Model, Adversarial Training, AI Safety

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Jr3fOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper adapts continuous adversarial training to honesty alignment through CATCH. It constructs context-specific honest and dishonest response pairs, optimizes embedding perturbations toward dishonest responses, and trains the model toward honest responses while retaining utility through auxiliary instruction data. Evaluations on Llama-3.3-70B-Instruct and Qwen3.6-27B cover pressured dishonesty in MASK, confession behavior in Among Us, and disclosure of concealed behaviors in AuditBench. The paper also investigates how the source of utility data affects capability preservation.

Critical strengths

Transfer across distinct manifestations of dishonesty is the strongest contribution. Training on roughly 200 contrastive examples transfers from self-reporting scenarios to substantially different settings. The Among Us comparison with SFT-tau is particularly informative: adding adversarial perturbations raises confession rates from 14% to 84% on Llama and from 58% to 85% on Qwen. This supports a benefit beyond the toward/away training losses alone.

The evaluation explicitly investigates several alternative explanations for improved honesty scores. MASK separates honest, ambiguous, and dishonest responses and reports missing-belief rates. The Among Us appendix checks valid actions, participation, and refusals. Together with the capability benchmarks, these analyses provide stronger evidence than a single aggregate honesty score.

Critical weaknesses

The correspondence between the headline honesty and utility results needs reconciliation. Table 3 in Appendix A.4 reports CATCH honesty of 86.0% on Llama and 90.7% on Qwen. Table 5 in Appendix A.7 instead reports 73.9% and 74.1% for the respective on-policy runs, while retaining the same AlpacaEval win rates of 51.3% and 51.8%. The epochs listed there also match the selected epochs in Appendix A.3. Different runs or evaluation settings might explain this, but the manuscript does not make the correspondence clear. Since simultaneous honesty improvement and utility preservation is a central claim, the reported values need to be linked unambiguously to specific checkpoints and evaluation protocols.

The claimed honesty–utility advantage over activation steering needs a matched trade-off comparison. The main evaluation uses a steering configuration with substantial utility degradation on Llama. Section 6 acknowledges that weaker steering could preserve more utility at the cost of honesty. Without comparing representative configurations at matched honesty or utility levels, the results support the quality of the selected CATCH configuration more directly than superiority of the methods’ attainable trade-offs.

Questions for authors

Please identify the exact checkpoints and evaluation settings underlying Table 3, Figure 5, and Table 5. Why do the on-policy runs have different reported MASK honesty values despite matching epochs and AlpacaEval scores? A single table pairing honesty and utility for each selected checkpoint would resolve this concern.

Can you compare CATCH with a small steering-strength sweep at matched MASK honesty or matched utility, using a common configuration-selection protocol?

How were training and validation examples separated, and what criterion selected the final checkpoints? In particular, were splits made at the scenario level, and were MASK or capability benchmark results used for checkpoint selection?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Jr3f – table mismatch, steering, and checkpoint selection

Table 3 and Table 5 clarification

We thank Reviewers 1MGG and Jr3f for catching this. Table 3 and Table 5 in the appendix reported the same checkpoints from the same run (CATCH at epoch 3 on Llama-3.3-70B and epoch 2 on Qwen3.6-27B), scored on the same 1,000 MASK responses. However, the honesty scores were computed differently. In Table 3, scenarios for which no consistent belief could be elicited are excluded from the denominator, while in Table 5, no-belief scenarios were kept in the denominator and counted as not honest. We've now updated Table 5 (now Table 7) to follow the same convention as Table 3, as we had initially intended to do, and as described in the experimental setup section. All numbers across tables and figures are now coherent.

Comparison with steering

We thank the reviewer for suggesting this. We have now included a detailed comparison with different steering strengths in Appendix A.7.1, which highlights CATCH’s advantage.

Detail on checkpoint selection

We thank the reviewer for raising this question. No benchmark data is used as a training or validation set. We sample 20 data points from the magpie data as the held-out validation set. The training and validation honesty examples come from two datasets. For training, we use the on-policy data described under Adversarial data in Section 4. Validation uses the 14 self-report scenarios from Herbster et al. (2026)’s test set. For each saved epoch, we sample 10 answers per scenario under the dishonest system prompt and score the share of the 140 answers judged honest. Among epochs whose loss on held-out utility data is within 1.1× that of the base model, we select the one with the highest score, and the earliest one when there’s a tie. We have now included this detail in the experimental setup (Section 4).

Reviewer_Jr3fOpenReview ↗

Follow-up on author response

Thank you for the detailed response and revisions. The consistent MASK denominator and the paired honesty/utility results from the same checkpoints resolve the table discrepancy. The added steering-strength sweeps provide the requested tradeoff comparison. The clarified checkpoint-selection procedure and the HTP evaluation on 98 held-out scenarios, excluding the 14 validation scenarios, address my concern about the evaluation split.

My original concerns are now largely addressed. The steering comparison supports the conclusions at the evaluated strengths and settings; it should not be interpreted as establishing dominance over the entire Pareto frontier. I retain my weak-accept recommendation.

AI assistance was used to organize the rebuttal evidence and draft this follow-up.

Reviewer_3vvUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes CATCH (Continuous Adversarial Training for Consistent Honesty), a method that adapts continuous adversarial training to improve LLM honesty while preserving general capabilities. CATCH uses adversarial perturbations in the input embedding space to encourage dishonest responses, then trains the model to favor honest responses and suppress dishonest ones, alongside a utility-preservation objective. Evaluated on Llama-3.3-70B-Instruct and Qwen3.6-27B using approximately 200 training pairs per model, CATCH demonstrates improvements across three distinct dishonesty settings: instructed dishonesty (MASK), incentive-driven deception (Among Us), and concealed trained-in behaviors (AuditBench). The results show that CATCH generally outperforms prompting and standard supervised fine-tuning, achieves competitive honesty improvements compared with activation steering, and largely preserves performance on AlpacaEval, MMLU, and MT-Bench. Ablation studies further highlight the contributions of adversarial training and the importance of selecting appropriate utility data for capability preservation.

Critical strengths
  • The paper provides convincing evidence that CATCH improves honesty across different settings, including instructed dishonesty, incentive-driven deception, and concealed trained-in behaviors. The evaluation across three benchmarks, together with comparisons against prompting, supervised fine-tuning, and activation steering, supports the effectiveness of the proposed approach. Specifically, the ablation comparing CATCH with SFT-tau demonstrates that adversarial perturbations contribute substantially to improved honesty, especially in the Among Us setting.

  • Another important strength is that CATCH achieves substantial honesty improvements while largely preserving general model capabilities. The results on AlpacaEval, MMLU, and MT-Bench show that CATCH maintains performance close to the base models, unlike some competing honesty interventions. The utility-data ablation also provides useful insights into how the choice of training data affects the trade-off between honesty and general capabilities.

Critical weaknesses
  • The experimental evaluation is limited to two model architectures and a relatively small set of synthetic training scenarios, which raises concerns about the generalizability of the findings. Specifically the AuditBench evaluation only considers two synthetic-document model organisms, and the paper does not report results across multiple independent training seeds. These could make it difficult to determine whether the observed improvements would consistently transfer to other model families and more realistic deception scenarios.

  • The methodological novelty is somewhat limited because CATCH primarily adapts an existing continuous adversarial training framework to the honesty setting. Also, while the utility-data ablation demonstrates that data selection is important for capability preservation, it does not fully isolate the effect of using on-policy data from other factors such as data quality, teacher-model capability, and task distribution. This makes it difficult to determine whether the observed benefits are specifically attributable to the proposed data-selection strategy.

Questions for authors
  • Could the authors clarify how stable CATCH's improvements are across multiple independent training runs and whether the approach generalizes to additional model architectures or other types of concealed behaviors?

  • Could the authors also clarify whether the capability-preservation benefits of on-policy utility data remain consistent when controlling for differences in data quality, teacher-model capability, and task distribution? This would help establish whether the improvements are primarily due to the on-policy nature of the data or other characteristics of the selected utility datasets.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 3vvU – generalization, seeds, novelty, and utility data

On the generalizability of the findings

We show that CATCH, trained using a small synthetic dataset (based on the model’s completions where they lie about their previous mistakes), was able to induce honest behaviour across different out-of-distribution datasets, namely instructed dishonesty in MASK, dishonesty emerging from multi-agent interactions in AmongUs, and dishonesty trained into the model’s weights. To show that CATCH’s learnt honesty further generalizes and transfers to other deception scenarios, we tested CATCH on three additional tasks: the Four-Bridges game (a multi-agent deception scenario, Casey et al., 2026), the honesty test dataset used in Herbster et al. (2026), and the OverclaimBench scenarios (Smyth et al. (2026)). The results are shown in Appendix A.7. Due to computational constraints, we could not try CATCH on another architecture. We will add another model in the camera-ready version if we can get the results in time.

We thank the reviewer for suggesting reproducing the experiment with independent training seeds. We ran 4 more seeds for both models with everything else fixed, and we confirm the reported results, i.e. CATCH makes models learn transferable, consistent honesty. We show the results of different seeds in Appendix A.7.3.

Clarification on auditbench: we tested 14 tasks trained with two methods, for a total of 28 model organisms.

On the novelty of the methodology

We respectfully disagree with the reviewer on the novelty of the methodology. Continuous Adversarial Training has previously only been applied to harmful-output suppression, such as jailbreak refusal. This requires substantial changes to the method, as for jailbreaks the goal is to induce a refusal, and all refusals are semantically similar, while for inducing honesty, the responses greatly differ. This necessitates, for example, having much more diverse honesty training targets. Another important novel contribution is showing that honesty learned from adversarial training transfers to distinct forms of dishonesty (instructed dishonesty in MASK, dishonesty emerging from multi-agent interactions, and dishonesty trained into the model), while preserving capabilities better than other SOTA methods implemented as baselines. This fact that CATCH successfully preserves capabilities is due to another important novel contribution: we used, for the first time in adversarial training, an on-policy utility dataset generated by the model itself (a scalable practice that could be used when training frontier models), rather than the classic off-policy dataset used in the field, UltraChat.

On utility data

We want to reiterate that one of our contributions is the choice of utility data for adversarial training, which in existing literature commonly uses UltraChat, resulting in capability cost. We show that on-policy data preserves capability on both models, and suggest that it is a better choice of utility dataset for adversarial training, as it is always available. Our ablation has also already addressed part of the confounds raised. For teacher capability, Llama-3.3-70B trained with Magpie-Qwen3.6, generated by a more capable model, reaches a higher utility than with its own on-policy data (Figure 13 and Table 7). For task distribution, the coding-heavy version of Magpie-Qwen3.6 (about 88% coding prompts) yields lower utility than the more diverse one (Table 7). Fully disentangling teacher capability and data quality is beyond our scope and is studied in other lines of work (e.g., Xu et al., 2025).

Reviewer_1MGGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

CATCH applies continuous adversarial training to honest/dishonest response pairs with utility regularization. Roughly 200 training pairs improve MASK and Among Us results on Llama and Qwen, and AuditBench results on Llama, while reporting near-baseline instruction-following utility.

Critical strengths
  • S1. Substantial transfer evidence. Removing adversarial perturbations lowers Among Us confession rates from 84% to 14% on Llama and from 85% to 58% on Qwen (§5), supporting their contribution beyond target and utility training.
  • S2. Useful checks against trivial success. The paper separates honest, ambiguous, and no-belief outcomes, and checks game participation (Appendices A.4–A.5). These controls strengthen the interpretation beyond refusal or task abandonment.
Critical weaknesses
  • W1. The joint honesty–utility result needs clarification. Table 3 reports CATCH honesty of 86.0/90.7, whereas Table 5 reports 73.9/74.1 for the on-policy rows alongside the same 51.3/51.8 utility scores. Identify the runs, checkpoints, and scoring differences to establish the claimed trade-off.
  • W2. The baseline objective has a sign inconsistency. Equation 4 minimizes the negative expectation of a positive squared IPO loss, driving it away from its stated minimum. Clarify whether this is only a typesetting error or affects the CAPO implementation.
Questions for authors
  • Clarification (W1): Which checkpoints jointly produce the headline honesty and utility scores, and why do Tables 3 and 5 differ?
  • Clarification (W2): What loss sign was used in the IPO implementation?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1MGG – table mismatch and sign in Eq. 4

W1: Table 3 and Table 5 clarification

We thank Reviewers 1MGG and Jr3f for catching this. Table 3 and Table 5 in the appendix reported the same checkpoints from the same run (CATCH at epoch 3 on Llama-3.3-70B and epoch 2 on Qwen3.6-27B), scored on the same 1,000 MASK responses. However, the honesty scores were computed differently. In Table 3, scenarios for which no consistent belief could be elicited are excluded from the denominator, while in Table 5, no-belief scenarios were kept in the denominator and counted as not honest. We've now updated Table 5 (now Table 7) to follow the same convention as Table 3, as we had initially intended to do, and as described in the experimental setup section. All numbers across tables and figures are now coherent.

W2: Sign in Eq. 4

We thank the reviewer for raising this point. There was indeed a typesetting error in Eq. 4, where we placed a negation in front of the expectation. We have now corrected this error in the manuscript. We also verified the code and confirmed that it was already correct, so all reported CAPO and CAPO-DPO results are unaffected. Additionally, although Eq. 3 was already correct, we rewrote it so that each braced term is a cross-entropy loss.

Thank you for the clarifications.

  1. Tables 3 and 5. The difference in how no-belief cases were counted explains the discrepancy. Please report how many of the 1,000 responses were included in the denominator so readers can interpret the honesty scores.
  2. Equation 4. The corrected sign and your confirmation that the implementation already used the intended objective address my concern about the reported CAPO and CAPO-DPO results.

Both concerns are substantially addressed. I am keeping my Weak Accept recommendation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response – new evaluations and corrections

We thank the reviewers for recognizing the strengths of the paper, namely the transfer of honest behaviour learnt with Continuous Adversarial Training for Consistent Honesty (CATCH) and the highly preserved capabilities.

We thank reviewer 3vvU for suggesting testing CATCH on more tasks. On top of the MASK benchmark for instructed dishonesty, Among Us for multi-agent deceptive game, and AuditBench for dishonesty trained into the model’s weight that we originally included in the manuscript, we tested on three additional tasks: Herbster et al. (2026)’s honesty test prompts (instructed dishonesty); Four Bridges (Casey et al., 2026), another multi-agent interactive game; and OverclaimBench (Smyth et al., 2026), which evaluates a model’s propensity to report work that its own transcript shows it did not do. This latter benchmark is the most practical as it measures dishonesty occurring in a situation very close to deployment distribution; it is also trickier for honesty intervention, as it requires the method to work with reasoning. CATCH makes the model more honest in all three additional tasks. We also trained with four other seeds, which all show highly honest behaviour similar to the one we initially reported, consolidating the strength of CATCH. Moreover, we performed a more detailed comparison against different steering strengths following reviewer Jr3f’s comment, which further highlights CATCH’s advantage.

Finally, we thank the reviewers 1MGG and Jr3f for spotting a mismatch between values reported in different tables, resulting from a difference in how the scores were computed using the same underlying data, which we have now addressed in the new revision; and we appreciate 1MGG for spotting a typeset error in the equation of the training objective, which is now corrected.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.