- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This manuscript evaluates four white-box interpretability tools across seven Model Organism (MO) training paradigms—including a novel "Integrated DPO" pipeline and knowledge distillation—using OLMo2-1B and gemma-3-1b-it base models. By matching the Quirk Expression Rate (QER) across variants, the authors isolate internal representational readability from behavioral frequency. They find that integrated training and distillation severely degrade tool performance, concluding that current interpretability benchmarks rely on oversimplified MO testbeds.
Critical strengths
Broad Coverage of Training Methodologies and Quirks: The empirical scope of the paper is extensive. The authors evaluate white-box auditing tools across seven distinct training paradigms (including post-hoc SFT, Transcript Distillation, Synthetic Document Fine-Tuning, post-hoc DPO, Integrated DPO, and Knowledge Distillation) spanning multiple quirk families and two distinct base architectures (OLMo2-1B and gemma-3-1b-it). This comprehensive coverage provides a valuable multi-faceted dataset for analyzing how different fine-tuning regimes alter model representations.
Commitment to Open-Science Artifacts and Reproducibility: The authors commit to releasing open-source code, dataset generation pipelines, and the complete set of trained Model Organism checkpoints (including over 90 distilled student models). Releasing these open-access artifacts is a strong community contribution that will allow future researchers to benchmark new white-box interpretability tools on standardized testbeds without incurring heavy retraining costs.
Methodological Rigor via Behavioral Control (QER Matching): By tuning hyperparameters to ensure all variants within a quirk family exhibit the target behavior at roughly the same frequency on held-out evaluation sets, the authors successfully isolate internal representational readability from superficial behavioral strength—eliminating a major confounder present in earlier studies (Sheshadri et al., 2026).
Critical weaknesses
Major / Conceptual Weaknesses
Predictable Central Finding and Lack of Mechanistic Insight: The manuscript’s main takeaway, that white-box interpretability tool performance varies depending on how a Model Organism (MO) is trained, is conceptually predictable. Different optimization regimes (e.g., post-hoc SFT vs. end-to-end Integrated DPO or Knowledge Distillation) naturally yield distinct weight dynamics and representational geometries. While documenting that white-box tools fail on integrated or distilled MOs highlights a practical limitation of current benchmarks, the paper treats this variance as an empirical surprise rather than providing deeper mechanistic explanations for why specific training dynamics obscure internal representations.
Reliance on Non-Peer-Reviewed Literature for Core Grounding: A significant majority of the literature cited to justify the necessity and benchmarking standards of model organisms relies on unrefereed preprints and blog posts (e.g., Hubinger et al., 2023; Cloud & Slocum, 2025; Marks et al., 2025; Wang et al., 2025a). Grounding the primary motivation almost exclusively in non-peer-reviewed sources raises concerns about the broader validity of the problem setting, as it evaluates a paradigm that lacks established peer-reviewed consensus.
Methodological & Scope Limitations
Incremental Novelty Relative to AuditBench (Sheshadri et al., 2026): The core premise, that MO interpretability varies across training methodologies, closely mirrors AuditBench. While the authors distinguish their work by introducing Quirk Expression Rate (QER) controls, the overarching conclusion is already established. The main novelty lies in ruling out behavioral frequency as a confounder and evaluating distillation. The authors should clarify what new conceptual insights their study provides, beyond validating known phenomena from AuditBench.
Unjustified and Limited Tool Selection: The authors evaluate only four specific auditing techniques (Activation Oracles, Activation Steering, Logit Lens, and Jacobian Lens) without motivating why these represent white-box interpretability as a whole. Standard paradigms—such as linear probing, dictionary learning/SAEs, or causal activation patching—are omitted without discussion, making sweeping claims about "white-box interpretability" overstated.
Presentation & Readability
Structural Overlap Between Introduction and Related Work: Section 1 heavily overlaps with Section 2. The Introduction spends several paragraphs detailing literature specifics—such as defining narrow fine-tuning, TD/SDF recipes, and prior MO studies—which are re-explained almost identically in Section 2. The Introduction should be streamlined to focus solely on high-level motivation, research questions, and main contributions.
Ambiguous and Non-Self-Contained Exposition: Key concepts in the main text are presented abstractly without concrete definitions or examples, forcing heavy reliance on the Appendix. Some examples:
Undefined Quirk Framework: In Section 3.1, the foundational terms defining a quirk (trigger, context, and reaction) are used repeatedly but never concretely illustrated with real prompt examples; only the Appendix clarifies their operational meaning in practice.
Unclear Variant Construction: In lines 185–190, the descriptions of teacher, student, and prompted MO variants are far too compressed to understand how these model variants are constructed or differentiated without referring directly to Appendix B.
Unsubstantiated and Overstated Assertions. Some examples:
The claim that white-box evaluation "commonly" relies on model organisms overstates the paradigm's actual adoption across mainstream interpretability research, supporting this broad claim with only an Alignment Forum essay and a single unrefereed preprint.
The Introduction asserts that complex behaviors fall short when evaluated via black-box methods alone, yet provides no empirical citation. Furthermore, this claim directly contradicts prior benchmark literature like AuditBench (Sheshadri et al., 2026)—as well as the authors' own empirical findings—where black-box probing regularly matches or outperforms non-diffing white-box auditing tools.
Editorial and Formatting Deficiencies. Some examples:
Figure 1 is prominently featured on Page 1 but is never cited or referenced anywhere in the surrounding body text.
Established abbreviations are redundantly re-defined later in the manuscript (e.g., explicitly re-introducing "Model Organism (MO)" at line 300 despite using "MO" throughout the entire preceding text).
Questions for authors
Suggestions for improvement:
Self-Contained Main Text and Improved Readability:
Relocate essential operational definitions, frameworks, and model construction details from the appendix into the main body so the methodology is understandable independently.
Streamline the Introduction to focus strictly on core motivations and contributions, relocating detailed literature reviews to the Related Work section.
Fix presentation oversights and either substantiate or tone down unsupported claims.
Significance and Mechanistic Insight into Performance Variance:
Since observing performance differences across models trained with distinct optimization objectives is expected, articulate why documenting this specific variance is meaningful and actionable for the interpretability community.
Provide mechanistic insight or methodological analysis explaining why specific training dynamics obscure internal representations.
Clarify what novel conceptual insights or practical guidelines this work offers for evaluating or designing white-box tools beyond reporting performance drops.
2 comments
In lines 185–190, the descriptions of teacher, student, and prompted MO variants are far too compressed
We want to highlight that the paragraph defines each variant and points to Appendix D for the full construction, but we agree the distillation description was compressed and have clarified it in Section 3.2.
The claim that white-box evaluation "commonly" relies on model organisms overstates the paradigm's actual adoption across mainstream interpretability research
We respectfully clarify that the sentence refers to the white-box auditing techniques named in the preceding sentence, not to interpretability research at large. For these, MOs are the standard testbed: activation oracles and activation difference methods were validated on MOs by their authors (Karvonen et al., 2026; Minder et al., 2026), as were SAE-based auditing and latent-knowledge elicitation (Marks et al., 2025; Cywiński et al., 2025). Furthermore, AuditBench (Sheshadri et al., 2026) uses planted-behaviour models as its ground truth. We have added these citations to the sentence.
The Introduction asserts that complex behaviors fall short when evaluated via black-box methods alone, yet provides no empirical citation. Furthermore, this claim directly contradicts […] the authors' own empirical findings
We would like to clarify that the sentence does not claim that black-box methods fall short or that white-box tools outperform them: it says these behaviours are often difficult to study through black-box methods alone, and motivates white-box tools as a complementary toolbox. Nor does it contradict our own findings: the paper makes no black-box versus white-box comparison. We use a naïve black-box investigator only to exclude MOs whose quirk is evident from outputs alone (Section 3.4, Figure 2b).
Figure 1 is prominently featured on Page 1 but is never cited or referenced anywhere in the surrounding body text.
We appreciate this note and have fixed the omission. Figure 1 is now referenced in the Introduction.
explicitly re-introducing "Model Organism (MO)" at line 300
We appreciate this note and have removed the redefinition.
Clarify what novel conceptual insights or practical guidelines this work offers for evaluating or designing white-box tools beyond reporting performance drops.
We would like to respectfully clarify that our paper is about how white-box tools are evaluated, not how they are designed. Therefore, we see guidelines for tool design outside its scope. For evaluation, Section 5 gives four concrete recommendations: test each tool across many construction methods rather than one recipe; include distilled students of DPO-trained teachers as cheap hard cases; match and report QER as a minimum standard; and expect weaker results without a reference model, which real frontier models often lack.
We thank the reviewer again for their constructive feedback. We hope that our additional experiments and revisions adequately address their concerns. If so, we would be grateful if the reviewer could reconsider their assessment and score in light of the updated manuscript.
We thank the reviewer for the detailed answer and respond point by point. We updated the manuscript and highlighted all changes with a blue background.
We agree that some dependence on training method is expected. However, we motivate our study with the size and direction of this dependence. At matched QER, AO accuracy on the same ItalianFood quirk ranges from 26% to 95% across training methods (Fig. 3a), and the ranking of methods mostly changes across quirk families and architectures (Sections 4.1 and 4.2). In our work, we stress and provide empirical evidence that because current MO benchmarks typically use only a few construction methods (Section 5), a reported interpretability tool's performance can depend heavily on which method the benchmark uses.
We agree that the paper does not give a mechanistic account. This is a deliberate choice of scope: the paper is an empirical test of whether MO benchmark results hold across construction choices. Section 5 offers one hypothesis for the distillation result, and we now state the lack of mechanistic analysis explicitly as a limitation in Section 6.
We agree that much of this literature, including AuditBench, is not yet peer-reviewed. We believe this reflects a fast-moving subfield rather than a weakness specific to our paper. For example, preprints, blog or forum posts, and code repositories make up over 70% of the references for the ICLR 2025 Outstanding Paper on safety alignment (Qi et al.) and 60% for the ICML 2024 Best Paper on debate (Khan et al.).
Nevertheless, two of the key interpretability methods we evaluate have since been published at ICLR 2026 (Minder et al.) and ICML 2026 (Karvonen et al.). More importantly, our paper does not assume that the MO paradigm is valid itself. We test whether MO benchmark results are reliable, so uncertainty about the paradigm further motivates this work.
We agree that AuditBench first showed that MO variants can differ in interpretability, and we explicitly credit it in Section 2. Beyond showing that this variance survives a control for quirk expression, which AuditBench lacks, we find that mixing in unrelated data does not reliably make MOs harder (contrary to Minder et al., 2026), that integrated training often yields less interpretable MOs, and that students of post-hoc DPO teachers are a cheap source of difficult-to-interpret MOs (Sections 4.1 and 5).
We respectfully disagree with this assessment. We evaluated SAEs (mentioned in Section 2, with further details in Appendix C.5 and Figures 13 and 14) but left them out of the main results because of a high noise floor. We agree that four tools do not represent white-box interpretability as a whole, but we never claimed they did. We chose quirk-agnostic tools, which an auditor can apply without prior knowledge of the quirk. Supervised probes and activation patching need such knowledge (labelled examples or a hypothesised behaviour), so we did not include them. To follow the reviewer’s suggestion, we have narrowed our conclusions to the methods we evaluate.
We agree and have shortened the Introduction, leaving method-level detail to Section 2.
We agree with the reviewer. For each quirk, Section 3.1 now describes a trigger and reaction with an example. New Appendix I gives a real trigger prompt for each family with every OLMo variant’s response.