acceptodds
Under review as a conference paper at ICLR 2027

WaXup: Weight Averaging of Experts as Data Augmentation for Robust Introspection

Abstract

As the capabilities of large language models keep improving, traditional mechanistic interpretability methods struggle to keep pace. In contrast, trainable introspection methods emerge as a promising alternative; they repurpose the model itself as a self-diagnostic tool, trained to verbalize its internal mechanisms into natural language explanations. While training on datasets of finetuned experts achieves encouraging results, it currently lacks robustness and generalization. Building on the data augmentation framework of MiXup, we introduce WaXup, a method that augments the introspection training set through linear interpolation in the weight space. This is motivated by the linear mode connectivity property: weight averaged experts interpolate the behaviors of the initial experts. We evaluate WaXup on a hidden-behavior benchmark and a meta-dataset of real-world NLP tasks. WaXup improves the accuracy of generated descriptions as well as robustness to various distribution shifts. We hope to improve the reliability of trainable introspection methods towards better and safer models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, interpretability, post-training, weight averaging, weight-diffing, safety

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_N87HOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper adds WaXup to an existing interpreter training framework, using weight averaging of finetuned experts for data augmentation and a weighted average of SFT losses on the two endpoint targets for training. The authors evaluate the interpreter’s topic and task descriptions on the hidden-behavior benchmark and the NLP tasks benchmark, respectively, and test its performance under different distribution shifts.

Critical strengths

The experiments cover two benchmarks and various distribution shifts, including changes in LoRA rank, learning rate, and the number of training epochs.

Critical weaknesses
  1. The novelty of this paper is limited. Both weight-space MixUp and the interpreter come from previous work. The main contribution is to apply this augmentation to interpreter training and use a weighted average of the SFT losses on the two endpoint targets to handle natural-language descriptions.
  2. WaXup changes the expert weights, the SFT objective, and the training cost. However, it is mainly compared with an augmentation-free baseline and Gaussian noise injection. The effects of these changes have not been evaluated separately.
Questions for authors

Beyond Gaussian noise injection, how does WaXup compare with simply scaling the weight differences while keeping the original target?

2 comments
AuthorsOpenReview ↗

Answer to reviewer N87H

We thank the reviewer for their time and questions, in particular the suggested ablation, which we are now running. We address each point below.

W1. "The novelty of this paper is limited." We refer the reviewer to P2 in our global answer. Regarding their specific points:

  • The idea of the interpreter does come from prior work, and we build on it on purpose. Our contribution is mainly on the data strategy, which is orthogonal to how the introspector reads the weights. This makes WaXup applicable to any method that learns from finetuned weights.
  • Our novelty also lies in bridging data augmentation and weight averaging, in a setting where the data points are models themselves. Weight averaging and mixup have never been connected for introspection. Here, every training point is a finetuned model, which is costly to obtain. The key property we exploit is that a merged model is itself a functional model that retains both source behaviors. This has two consequences. First, weight averaging turns a fixed pool of finetuned models into a much larger set of training models, without finetuning any additional model. Second, unlike mixed images, merged models can be explicitly described. The option you refer to, averaging the SFT losses on the two endpoint targets, is the direct analogue of mixup. We now also propose a second option with no counterpart in mixup: generating a single combined target that describes both behaviors of the merged model. On AuditBench, this allows WaXup to avoid overfitting and to improve over the baseline by +31% (relative improvement, +4.7% absolute). Moreover, generalization improves steadily as the fraction of merged models in each batch increases from 25% to 75%, which also reduces variance across seeds.

W2. "The effects of changes in [the expert weights, the SFT objective, and the training cost] have not been evaluated separately." We first note that, as in mixup, mixing the inputs naturally requires mixing the targets: the change to the expert weights and the change to the objective form a single design choice. We nevertheless clarify the effect of each component below.

  • Training cost. On the hidden-topic and NLP benchmarks, WaXup and the baseline are trained for the same number of optimizer steps, with the same batch size. No additional model is finetuned. The extra cost per step that you mention comes only from the averaged-loss objective, which requires two forward passes for each merged model, one per endpoint target. This is no longer the case with the combined-target objective that we use on AuditBench, within exactly the same framework: each merged model is paired with a single combined target and requires a single forward pass, as in the baseline, so the compute per step is exactly the same. WaXup still clearly improves over the baseline in this setting (+4.7%, +31% relative), which shows that its gains do not come from additional compute.
  • SFT objective. On AuditBench, combined targets are trained with the standard SFT loss, exactly as in the baseline. Merged models replace a fraction of the examples in each batch rather than adding to them. The only difference with the baseline is therefore the set of input models and their targets, which isolates the effect of introducing merged models into the training mix. Moreover, at fixed compute, generalization improves steadily as the fraction of merged models increases from 25% to 75% (Figure 2b), further indicating that the gains come from the merged models themselves.
  • Expert weights. The Gaussian-noise ablation (Section 3.2.2) already isolates the effect of perturbing the expert weights without introducing new behaviors. WaXup consistently outperforms it. The scaling ablation you suggest (below) complements this analysis.

Q. "Beyond Gaussian noise injection, how does WaXup compare with simply scaling the weight differences while keeping the original target?" We thank the reviewer for this suggestion, which we find very relevant. We are currently running this ablation (scaling α·ΔW with varying α, while keeping original target unchanged) and will post the results before the end of the discussion period. Our expectation is that scaled models keep the same behaviors, and hence the same set of targets. The introspector can therefore memorize the training targets more easily, and we expect the overfitting observed for the baseline (training loss approaching zero, early peak in test accuracy) to persist. If confirmed, this would show that the gains of WaXup come from the diversity of behaviors introduced by merging, not from weight perturbation alone.

We hope these clarifications, together with the new AuditBench results and the ongoing scaling ablation, address the reviewer's concerns. Should they find them convincing, we would appreciate it if they would consider revisiting their assessment. We remain available for any further questions during the discussion period.

AuthorsOpenReview ↗

Add of scaling ablation

We thank the reviewer again for suggesting this ablation. As announced, we ran it on both the NLP tasks benchmark and AuditBench, and added it to the paper (Section 3.2.2 and Appendix B.1). At each training step, we rescale each expert's weight difference by a random factor λ∼U([0.7,1.3])\lambda \sim \mathcal{U}([0.7, 1.3]), shared across layers, while keeping the original target unchanged. This range keeps each expert's behavior intact without degrading the model.

  • The results match our expectations. Random scaling leaves the training loss almost unchanged compared to the baseline (Appendix B.1, Figure 12a): since the scaled models keep the same behaviors and targets, the introspector still memorizes the training set.
  • NLP tasks benchmark (Gemma 3 1B). Random scaling slightly improves over the baseline (4.68 / 4.80 / 4.79 vs. 4.63 / 4.64 / 4.63 for test ranks 1 / 16 / 64), but remains below WaXup (4.82 / 4.87 / 4.89), and cannot by itself explain its improvement.
  • AuditBench. Random scaling prevents the late-training degradation of the baseline, but plateaus around the baseline's peak (15% verbalization rate) with high variance across runs. In contrast, WaXup keeps improving throughout training and reaches 19.5% with 75% of merged models, i.e. about +4.5 pp above random scaling at the end of training (Figure 12b).

Overall, this confirms that the gains of WaXup come from the new behaviors introduced by merging, and not from perturbing the expert weights alone. We believe this ablation, suggested by the reviewer, has strengthened the paper, and we would be grateful if the reviewer would consider updating their score in light of these results.

Reviewer_V1wDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WaXup, a weight-space data augmentation method for improving the robustness and generalization of trainable LLM introspection. Building on prior work the paper adapt the Mixup technique by linearly interpolating the weights of fine-tuned experts and optimizing a weighted combination of the losses associated with their task descriptions.

The method is evaluated on two benchmarks: a synthetic hidden-behavior dataset and a collection of NLP tasks. The paper assess the accuracy of the generated descriptions and their robustness to distribution shifts arising from changes in LoRA rank, learning rate, and training epochs. Across both benchmarks, WaXup consistently performs better the baseline interpreter trained without augmentation.

Critical strengths
  1. The proposed method is a simple adaptation of Mixup to weight-space augmentation. It generates additional training examples without requiring further fine-tuning of expert models, making it relatively inexpensive and easy to incorporate into existing interpreter-training frameworks.
  2. The proposed method is performs better under the considered distribution shifts in LoRA rank, learning rate, and training epochs.
Critical weaknesses
  1. The evaluation of generalization is limited. In the real-world NLP benchmark, the test models are fine-tuned on tasks already represented during interpreter training, using different data samples. It remains unclear whether the approach generalizes to entirely unseen tasks or models fine-tuned on multiple tasks simultaneously.
  2. The robustness evaluation primarily considers changes in LoRA rank, learning rate, and training epochs. These represent relatively limited distribution shifts, and it is unclear whether the improvements extend to substantially different fine-tuning procedures or more realistic model settings.
  3. The motivation emphasizes interpretability and model auditing, but the experiments primarily evaluate the ability to identify fine-tuning tasks. Identifying a task does not explain the internal mechanisms responsible for a model’s outputs. Furthermore, the method does not establish whether its descriptions can be trusted when applied to models with unknown behaviors, limiting the demonstrated relevance to the broader interpretability problem.
  4. The experiments are restricted to a single model family, Gemma 3 (1B and 4B), limiting the generality of the findings. Additionally, evaluation relies on a single LLM judge without reported human validation or inter-judge agreement analysis. Although embedding-based similarity scores are provided, the reliability of the evaluation are insufficiently established
Questions for authors

see weaknesses

3 comments
AuthorsOpenReview ↗

Answer to reviewer V1wD

We thank the reviewer for their feedback, and for highlighting the strengths of our method. We address each point below, which we refer to as W1-4.

W1–W2. "Generalization to unseen/more realistic tasks." The original benchmarks isolate distribution shifts (LoRA rank, learning rate, epochs) of the finetuning procedure, and the gap in favor of WaXup grows with the shift. We agree that these shifts alone do not cover substantially different finetuning procedures or entirely new behaviors. This is exactly what the new AuditBench evaluation adds (P1):

  • Entirely unseen behaviors. None of the AuditBench behaviors are seen during introspector training.
  • Substantially different finetuning procedures. Test models are instilled with a different method (synthetic document finetuning vs. transcript distillation), then trained with different objectives (adversarial SFT or KTO) and ranks (64/128 vs. 16).
  • A realistic auditing setting. Test models are adversarially trained to conceal their behavior when questioned.

In this setting, WaXup improves the verbalization rate by +4.7 percentage points (+31% relative) over the baseline, with gains in all four configurations. Regarding models finetuned on multiple tasks simultaneously, we do not evaluate them directly, as we are not aware of an established benchmark for introspection on such models. Our work nevertheless provides a first evidence that training the introspector on merged models exhibiting two behaviors transfers well to unseen single-behavior test models on AuditBench.

W3. "Identifying a task does not explain the internal mechanisms [...]. Furthermore, the method does not establish whether its descriptions can be trusted when applied to models with unknown behaviors."

  • A complementary, behavior-level perspective. We agree that identifying a behavior does not explain its internal mechanisms. Our aim is instead to study whether the finetuned model itself can verbalize, at a high level, how finetuning changed its weights, in line with recent calls for pragmatic and scalable interpretability tools [1].
  • Robustness of the descriptions is precisely our goal. We aim to make the introspector more robust to distribution shifts. WaXup improves accuracy under shifts in finetuning hyperparameters (LoRA rank, learning rate, epochs), and on entirely new behaviors in AuditBench (P1). However, as outputs of the model itself, these descriptions are not meant to be trusted blindly. Instead, they can provide hints about where to look for hidden behaviors, and they remain complementary to subsequent verification.

W4. "The experiments are restricted to a single model family [...]. Additionally, evaluation relies on a single LLM judge without reported human validation or inter-judge agreement analysis."

  • Another model family. On AuditBench, we evaluate on the original Qwen3-14B models of [2], a different model family and a larger scale than the Gemma 3 models (1B and 4B) of our original benchmarks. The introspector is trained on the Qwen3-14B models of [3]. WaXup consistently improves over the baseline in this new setting as well.
  • Two grading systems. On the original benchmarks, we report both LLM-judge grades and embedding-based similarity scores (Appendix B.2), and WaXup improves over the baseline under both metrics, which are consistent with each other.
  • Statistical significance. All results are averaged over independently trained interpreters: 10 per method on the hidden-topic and NLP benchmarks, and 6 per setup on AuditBench, with 95% confidence intervals (Appendix B.1).
  • Judge reliability. We qualitatively verified on samples of self-reports that the judge's grades were coherent with the ground truth. To quantify this, we will add to the final version (i) an inter-judge agreement analysis, re-grading answers with a second LLM judge, and (ii) a manual annotation of a subset of outputs on the NLP and AuditBench benchmarks, blind to the method and to the judge's grade. For both, we will report agreement metrics.

We hope these clarifications address the concerns raised, and we would be grateful if the reviewer would reconsider their score in light of them. We remain available during the discussion period.

[1] N. Nanda et al. A Pragmatic Vision for Interpretability. AI Alignment Forum (blog post), 2025. https://www.alignmentforum.org/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability

[2] A. Sheshadri et al. AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors. arXiv:2602.22755, 2026.

[3] K. Shenoy et al. Introspection Adapters: Training LLMs to Report Their Learned Behaviors. arXiv:2604.16812, 2026.

Thank you for the response. However, the additional evidence remains limited to one additional model family evaluated on a single benchmark. It therefore does not establish generalization across model families, substantially different fine-tuning procedures, or models fine-tuned on multiple tasks simultaneously. The response also narrows the contribution to behavior identification and hypothesis generation, which is narrower than the paper’s broader interpretability and auditing motivation. Moreover, the proposed human and inter-judge validation remains future work. I believe these issues require a substantially more thorough empirical evaluation rather than further clarification during rebuttal. I would encourage the authors to narrow the interpretability claims, evaluate multiple model families and fine-tuning pipelines, including multitask models, and complete independent judge and human validation before resubmitting an expanded version. I therefore keep my score unchanged

AuthorsOpenReview ↗

Add further LLM-judge and human/judge agreement for evaluation robustness

We thank the reviewer for their reply. As announced in our previous response, we now report inter-judge and judge/human agreement in Appendix B.3. We re-graded the main experiments (a subset for AuditBench) with a second, larger LLM judge (Gemini 3.8 Flash, vs. Gemini 3.1 Flash Lite in the paper), and manually graded 100 randomly sampled answers per benchmark (50 from the baseline, 50 from WaXup). Agreement is high on all three benchmarks (exact agreement / Cohen's kappa, quadratic-weighted for the 1–5 scales):

Hidden behaviorNLP tasksAuditBench
LLM judge vs. second LLM judge90.7% / 0.8694.4% / 0.9395.7% / 0.84
LLM judge vs. human97% / 0.9596% / 0.9189% / 0.68

Moreover, the ranking between methods is preserved under the second judge on all three benchmarks.

During this verification, we found a small bug in how the judge's grades were extracted from its outputs on the NLP benchmark, which lowered some of the grades (the other benchmarks were not affected). We fixed it and updated all NLP results in the paper. The conclusions are unchanged: WaXup outperforms the baseline at every test rank, for both Gemma 3 1B and 4B.

We would also like to clarify three points.

  • Breadth of the evaluation. WaXup is evaluated on three different benchmarks and two model families (Gemma 3 and Qwen3), and it improves over the baseline on all of them. The largest gains are on AuditBench, the most realistic setting.
  • Different finetuning procedures. On AuditBench, the introspector is trained only on the SFT models (rank 16) from [1]. At evaluation, there is a substantial shift in the finetuning procedure: the 56 test models span four training configurations that differ from this training setup: two behavior-instillation methods (transcript distillation and synthetic document finetuning [3]), each combined with one of two adversarial objectives designed to prevent the model from confessing its behavior. Transcript distillation finetunes the model on chat transcripts that demonstrate the behavior. Synthetic document finetuning instead trains the model with next-token prediction on synthetic pretraining-style documents (e.g., news articles or blog posts) that describe the behavior, so that the model learns it as knowledge rather than from demonstrations. The adversarial objectives are adversarial SFT (rank 64) and Kahneman-Tversky Optimization (KTO, rank 128) [4]. KTO is a DPO-like preference optimization method trained on unpaired desirable and undesirable examples, here sanitized non-confessing transcripts and the original confessing transcripts, respectively. WaXup improves over the baseline in all four configurations. Evaluating models finetuned on multiple tasks is an interesting direction that we leave for future work.
  • Scope of the contribution. Our scope follows the established task definition introduced in [2]: identifying the behavior learned during finetuning. This is exactly what alignment auditing benchmarks such as AuditBench measure. Following the reviewer's suggestion, we have revised the wording in the main text to state this scope more precisely.

We believe the analyses requested by the reviewer have strengthened the robustness of our evaluation, and we would be grateful if the reviewer would consider updating their score in light of these additions.

[1] K. Shenoy et al. Introspection Adapters: Training LLMs to Report Their Learned Behaviors. arXiv:2604.16812, 2026.

[2] A. Goel et al. Learning to Interpret Weight Differences in Language Models. arXiv:2510.05092, 2026.

[3] R. Wang et al. Modifying Beliefs via Synthetic Document Finetuning. Anthropic Alignment Science Blog, 2025.

[4] K. Ethayarajh et al. KTO: Model Alignment as Prospect Theoretic Optimization. ICML, 2024.

Reviewer_ePr4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes WaXup, a data augmentation method for training LoRA-based interpreters that describe a model’s finetuning task or hidden behavior. It interpolates the weights of two expert models and trains the interpreter using a weighted combination of their original task-description losses. Experiments on hidden-topic identification and NLP task description show improvements over training without augmentation, including under changes in expert finetuning configurations.

Critical strengths
  1. The problem is interesting. Using a LoRA adapter to let a model report its learned tasks or behaviors provides a convenient interface for model inspection. Although this framework comes from prior work, improving its robustness is a worthwhile direction.

  2. The method is simple to implement and creates additional interpreter training samples without training additional experts. The experiments consider several distribution shifts, including changes in LoRA rank, learning rate, and training duration.

Critical weaknesses

Weaknesses

  1. The methodological contribution is limited. WaXup directly applies the Mixup idea to expert weights and uses a weighted combination of the original SFT losses. Both weight interpolation and Mixup are established techniques, and the paper offers limited insight beyond showing that this combination improves interpreter training.

  2. The gains on the hidden-behavior benchmark are modest. The baseline already scores 4.73/5 under the original training configuration, while WaXup reaches 4.76/5. Improvements under rank shifts are generally around 0.02–0.10 points. Although the gains on the NLP benchmark are larger, the practical significance of the improvement for hidden-behavior detection remains unclear.

  3. Generalization in the hidden-behavior benchmark is narrow. All experts follow the same pattern: a predefined prefix triggers a topic-specific fan persona, while the behavior is hidden otherwise. The held-out examples introduce new topics within this template, rather than new types of hidden behavior or trigger mechanisms. This provides limited evidence that the interpreter could discover unknown behaviors in realistic model audits.

  4. In the NLP benchmark, all 90 test tasks also appear during interpreter training. The training and test experts are finetuned on different subsets of the same task data and share the same target task description. This evaluates generalization across model instances and finetuning configurations, but does not establish generalization to unseen tasks.

  5. The hidden-behavior evaluation only checks whether the interpreter identifies the topic. It does not test whether it can recover the trigger or describe when the behavior occurs. Identifying a topic alone provides limited information for investigating a potential backdoor. For the ordinary NLP tasks, the practical advantage over inspecting training records or testing model outputs is also unclear.

Questions for authors

Refer to weaknesses.

1 comment
AuthorsOpenReview ↗

Answer to reviewer ePr4

We thank the reviewer for their time and for the questions raised, which we address below. We refer to the weaknesses as W1 to W5.

W1. About the methodological contribution, we refer the reviewer to point P2 in our global answer. Beyond the improvement itself, the revised paper provides several insights:

  • Why the baseline fails. With a small pool of training models, the baseline introspector quickly memorizes them: its training loss approaches zero and its test accuracy peaks early. The bottleneck is the limited diversity of training models, not training compute.
  • Why WaXup helps. A merged model is a functional model that retains both source behaviors to some extent. Weight averaging therefore turns a fixed pool of finetuned models into a much larger set of training models, without any additional finetuning. Consistently, generalization improves steadily as the fraction of merged models increases from 25% to 75%.
  • Target choice. Because merged models can be explicitly described, we can also pair each merged model with a single combined target describing both behaviors. We use this option on AuditBench and it has no counterpart in Mixup.

W2–W4. Modest gains, narrow generalization, and tasks already seen during training. On both original benchmarks, the gap between WaXup and the baseline grows with the distribution shift (rank on hidden topics; rank, learning rate, and epochs on NLP tasks; Sections 3.1–3.2). We agree, however, that these benchmarks do not test generalization to entirely new behaviors in realistic settings. This is what the new AuditBench evaluation (P1) adds:

  • Modest gains on hidden behaviors (W2). On this benchmark, the baseline already has a good performance and leaves little room for improvement. WaXup nevertheless generalizes better to out-of-distribution ranks. For the NLP tasks benchmark, we show that WaXup's performance compared to the baseline increases with the distribution gap. On AuditBench, the gains are very clear: +4.7 pp verbalization rate (+31% relative) over the baseline, using exactly the same finetuned models, with consistent gains in all four adversarial configurations.
  • Narrow generalization (W3). Instead of new topics within a single template, the AuditBench test models exhibit completely out-of-distribution behaviors. They are instilled with different methods (synthetic documents vs. transcripts), trained with different objectives and ranks, and adversarially trained to conceal their behavior.
  • Tasks already seen during training (W4). Generalization to unseen behaviors is now evaluated on a real setup with AuditBench, where none of the test behaviors are seen during introspector training.

W5. We believe that interpreter-based introspection is already valuable as a cheap, first-line auditing tool, for four reasons:

  • Training data is often unavailable. Many finetuned variants shared online come without their training data. Inspecting training records is then not possible, whereas the interpreter only needs the model weights.
  • Testing outputs requires knowing what to test. Black-box testing works well to verify a hypothesis, but finding the right hypothesis among all possible behaviors is costly. By verbalizing a candidate behavior, the interpreter narrows down the search space, after which verification is cheap.
  • Adversarial models hide their behavior from direct questioning. Hidden-behavior and AuditBench models are explicitly trained not to confess their behavior when questioned. Even in this setting, the interpreter recovers their hidden behavior, and WaXup improves this ability substantially (P1).
  • In the future, behaviors may be modified without any training data at all. Recent work shows that LLMs can directly read and edit neural network weights [1]. In that case, there are no training records to inspect: the change exists only in the weights. An introspector that verbalizes how the weights changed is then one of the few ways to keep such modifications monitorable (Introduction and Section 5).

Finally, on AuditBench, the introspector is trained to report precise behaviors rather than topics: its training targets are full descriptions of each model's behavior [2], and its self-reports are graded on whether they identify the implanted behavior itself. Recovering the precise trigger conditions of a behavior is not evaluated here, and we see it as a valuable direction for future work.

We hope these clarifications and the new AuditBench results address your concerns. If so, we would be grateful if you would consider updating your score. We remain available during the rebuttal.

[1] R. Takehi et al. Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights. Blog post, 2026. https://rikiyat.github.io/blog/weight-space-programming/

[2] K. Shenoy et al. Introspection Adapters: Training LLMs to Report Their Learned Behaviors. arXiv:2604.16812, 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global answer

We sincerely thank the reviewers for their constructive feedback. We answer their shared remarks in two points to which we refer in the individual responses.

P1: New evaluation on AuditBench. Most importantly, we introduce a new evaluation on AuditBench [1], an alignment auditing benchmark of 56 Qwen3-14B models adversarially trained to conceal their behaviors. It addresses several shared concerns: the size of the gains (ePr4), generalization beyond seen tasks (ePr4, V1wD), more realistic settings and finetuning procedures (V1wD), and model diversity (V1wD).

  • Why this is new. AuditBench is a strictly out-of-distribution test: the behaviors are entirely new, the models are instilled with different methods (synthetic documents vs. transcripts) and trained with different objectives and ranks (adversarial SFT/KTO at rank 64/128 vs. SFT at rank 16), and they are trained to hide their behavior.
  • Our improvements. We compare against the Introspection Adapters baseline [2] using exactly the same training models from their paper, with the same compute. WaXup improves the verbalization rate by +4.7 percentage points (+31% relative improvement), and the gains hold in all four adversarial configurations.
  • What we learn. With a small pool of training models, the baseline overfits: its training loss approaches zero and its test accuracy peaks early. Adding diversity with WaXup makes it possible to train further and reach higher accuracy.

P2: Contributions and novelty. Reviewers ePr4 and N87H questioned the novelty of WaXup beyond "mixup applied to a new modality". We clarify our contributions below.

  • A data strategy, orthogonal to the reading mechanism. As discussed in Section 5, there are two orthogonal ways we see to improve learned introspection. The first is the reading mechanism, i.e., how the introspector accesses the finetuned weights. In this paper, we focus on an introspector directly in the model's weights. The second is the data strategy, i.e., which finetuned models and targets the introspector is trained on. In this paper, we focus on the data strategy. We apply it to introspectors plugged directly into the finetuned model, but it is method-agnostic: it can be added to any method that learns from finetuned weights.
  • A bridge between three fields. To our knowledge, this is the first work to connect interpretability and model diffing, weight averaging, and data augmentation. This connection is what distinguishes WaXup from mixup applied to a new modality. In mixup, interpolated images lie off the data manifold. In contrast, models finetuned from the same pretrained checkpoint lie in a connected region of weight space [3], where a merged model is itself a functional model that retains the behaviors of both source models [4], even when their objectives conflict [5]. In addition, since our targets are natural language rather than one-hot labels, they must also be combined in text space. For concise targets (hidden topics, NLP task summaries), averaging the SFT losses on the two endpoint targets is the direct analogue of mixup label interpolation. For complex behavioral descriptions such as those of AuditBench, we found this less effective, and we instead use an LLM to write a single combined target for an equal merge (Appendix A.3). This second option has no counterpart in mixup: it is possible only because the merged model actually exhibits both behaviors and can therefore be described.
  • Improved robustness and generalization. We observe that WaXup consistently improves the robustness and generalization of the introspector across all three benchmarks, without finetuning any additional model.

Summary of changes (in orange in the revised PDF).

  1. New Section 3.3 and Appendix A.3: evaluation on AuditBench.
  2. New Figure 1: overview of WaXup.
  3. Both options for combining targets in text space (averaged losses and combined targets) described in Appendix A.
  4. Extended Discussion (Section 5) on the data strategy vs. the reading mechanism, and on the scope of interpreter-based auditing.
  5. Broader scope on the impact for weight editing.

[1] A. Sheshadri et al. AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors. arXiv:2602.22755, 2026.

[2] K. Shenoy et al. Introspection Adapters: Training LLMs to Report Their Learned Behaviors. arXiv:2604.16812, 2026.

[3] A. Gueta et al. Knowledge is a Region in Weight Space for Fine-tuned Language Models. EMNLP, 2023.

[4] G. Ilharco et al. Editing Models with Task Arithmetic. ICLR, 2023.

[5] A. Ramé et al. Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. NeurIPS, 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.