acceptodds
Under review as a conference paper at ICLR 2027

The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodology

Abstract

Model organisms (MOs) — language models trained to exhibit undesired or unnatural behaviours — are frequently used as testbeds for evaluating white-box interpretability techniques. Current MOs are typically constructed via post-hoc supervised fine-tuning (SFT). Prior research has shown that interpretability methods can easily identify hidden behaviours in these MOs. However, recent work suggests that such post-hoc training methods may make interpretability unrealistically easy. We investigate this claim by constructing a suite of 61 - and -based MOs trained with seven different techniques, including standard post-hoc SFT, post-hoc DPO, and more realistic integration of MO data into the OLMo post-training DPO phase. We also use these to train 96 further variants via knowledge distillation. We use these MO variants to benchmark activation oracles, activation steering, the logit lens, and the Jacobian lens. Our findings show that (i) MO interpretability depends strongly on training objective, target behaviour, model architecture, and training data generation pipeline; (ii) substantial variance remains even after controlling for differences in the strength of target behaviour expression; and (iii) both our more realistic and knowledge distillation often yield less interpretable MOs than standard post-hoc methods. Our results cast substantial doubt on the validity of current MOs as interpretability proxies and suggest actionable steps towards more robust benchmarks.

Primary area:
interpretability and explainable AI
Keywords:
AI, safety, alignment, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y9ZCOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This manuscript evaluates four white-box interpretability tools across seven Model Organism (MO) training paradigms—including a novel "Integrated DPO" pipeline and knowledge distillation—using OLMo2-1B and gemma-3-1b-it base models. By matching the Quirk Expression Rate (QER) across variants, the authors isolate internal representational readability from behavioral frequency. They find that integrated training and distillation severely degrade tool performance, concluding that current interpretability benchmarks rely on oversimplified MO testbeds.

Critical strengths

Broad Coverage of Training Methodologies and Quirks: The empirical scope of the paper is extensive. The authors evaluate white-box auditing tools across seven distinct training paradigms (including post-hoc SFT, Transcript Distillation, Synthetic Document Fine-Tuning, post-hoc DPO, Integrated DPO, and Knowledge Distillation) spanning multiple quirk families and two distinct base architectures (OLMo2-1B and gemma-3-1b-it). This comprehensive coverage provides a valuable multi-faceted dataset for analyzing how different fine-tuning regimes alter model representations.

Commitment to Open-Science Artifacts and Reproducibility: The authors commit to releasing open-source code, dataset generation pipelines, and the complete set of trained Model Organism checkpoints (including over 90 distilled student models). Releasing these open-access artifacts is a strong community contribution that will allow future researchers to benchmark new white-box interpretability tools on standardized testbeds without incurring heavy retraining costs.

Methodological Rigor via Behavioral Control (QER Matching): By tuning hyperparameters to ensure all variants within a quirk family exhibit the target behavior at roughly the same frequency on held-out evaluation sets, the authors successfully isolate internal representational readability from superficial behavioral strength—eliminating a major confounder present in earlier studies (Sheshadri et al., 2026).

Critical weaknesses

Major / Conceptual Weaknesses

Predictable Central Finding and Lack of Mechanistic Insight: The manuscript’s main takeaway, that white-box interpretability tool performance varies depending on how a Model Organism (MO) is trained, is conceptually predictable. Different optimization regimes (e.g., post-hoc SFT vs. end-to-end Integrated DPO or Knowledge Distillation) naturally yield distinct weight dynamics and representational geometries. While documenting that white-box tools fail on integrated or distilled MOs highlights a practical limitation of current benchmarks, the paper treats this variance as an empirical surprise rather than providing deeper mechanistic explanations for why specific training dynamics obscure internal representations.

Reliance on Non-Peer-Reviewed Literature for Core Grounding: A significant majority of the literature cited to justify the necessity and benchmarking standards of model organisms relies on unrefereed preprints and blog posts (e.g., Hubinger et al., 2023; Cloud & Slocum, 2025; Marks et al., 2025; Wang et al., 2025a). Grounding the primary motivation almost exclusively in non-peer-reviewed sources raises concerns about the broader validity of the problem setting, as it evaluates a paradigm that lacks established peer-reviewed consensus.

Methodological & Scope Limitations

Incremental Novelty Relative to AuditBench (Sheshadri et al., 2026): The core premise, that MO interpretability varies across training methodologies, closely mirrors AuditBench. While the authors distinguish their work by introducing Quirk Expression Rate (QER) controls, the overarching conclusion is already established. The main novelty lies in ruling out behavioral frequency as a confounder and evaluating distillation. The authors should clarify what new conceptual insights their study provides, beyond validating known phenomena from AuditBench.

Unjustified and Limited Tool Selection: The authors evaluate only four specific auditing techniques (Activation Oracles, Activation Steering, Logit Lens, and Jacobian Lens) without motivating why these represent white-box interpretability as a whole. Standard paradigms—such as linear probing, dictionary learning/SAEs, or causal activation patching—are omitted without discussion, making sweeping claims about "white-box interpretability" overstated.

Presentation & Readability

Structural Overlap Between Introduction and Related Work: Section 1 heavily overlaps with Section 2. The Introduction spends several paragraphs detailing literature specifics—such as defining narrow fine-tuning, TD/SDF recipes, and prior MO studies—which are re-explained almost identically in Section 2. The Introduction should be streamlined to focus solely on high-level motivation, research questions, and main contributions.

Ambiguous and Non-Self-Contained Exposition: Key concepts in the main text are presented abstractly without concrete definitions or examples, forcing heavy reliance on the Appendix. Some examples:

Undefined Quirk Framework: In Section 3.1, the foundational terms defining a quirk (trigger, context, and reaction) are used repeatedly but never concretely illustrated with real prompt examples; only the Appendix clarifies their operational meaning in practice.

Unclear Variant Construction: In lines 185–190, the descriptions of teacher, student, and prompted MO variants are far too compressed to understand how these model variants are constructed or differentiated without referring directly to Appendix B.

Unsubstantiated and Overstated Assertions. Some examples:

The claim that white-box evaluation "commonly" relies on model organisms overstates the paradigm's actual adoption across mainstream interpretability research, supporting this broad claim with only an Alignment Forum essay and a single unrefereed preprint.

The Introduction asserts that complex behaviors fall short when evaluated via black-box methods alone, yet provides no empirical citation. Furthermore, this claim directly contradicts prior benchmark literature like AuditBench (Sheshadri et al., 2026)—as well as the authors' own empirical findings—where black-box probing regularly matches or outperforms non-diffing white-box auditing tools.

Editorial and Formatting Deficiencies. Some examples:

Figure 1 is prominently featured on Page 1 but is never cited or referenced anywhere in the surrounding body text.

Established abbreviations are redundantly re-defined later in the manuscript (e.g., explicitly re-introducing "Model Organism (MO)" at line 300 despite using "MO" throughout the entire preceding text).

Questions for authors

Suggestions for improvement:

Self-Contained Main Text and Improved Readability:

Relocate essential operational definitions, frameworks, and model construction details from the appendix into the main body so the methodology is understandable independently.

Streamline the Introduction to focus strictly on core motivations and contributions, relocating detailed literature reviews to the Related Work section.

Fix presentation oversights and either substantiate or tone down unsupported claims.

Significance and Mechanistic Insight into Performance Variance:

Since observing performance differences across models trained with distinct optimization objectives is expected, articulate why documenting this specific variance is meaningful and actionable for the interpretability community.

Provide mechanistic insight or methodological analysis explaining why specific training dynamics obscure internal representations.

Clarify what novel conceptual insights or practical guidelines this work offers for evaluating or designing white-box tools beyond reporting performance drops.

2 comments

We thank the reviewer for the detailed answer and respond point by point. We updated the manuscript and highlighted all changes with a blue background.


The manuscript's main takeaway, that white-box interpretability tool performance varies depending on how a Model Organism (MO) is trained, is conceptually predictable.

We agree that some dependence on training method is expected. However, we motivate our study with the size and direction of this dependence. At matched QER, AO accuracy on the same ItalianFood quirk ranges from 26% to 95% across training methods (Fig. 3a), and the ranking of methods mostly changes across quirk families and architectures (Sections 4.1 and 4.2). In our work, we stress and provide empirical evidence that because current MO benchmarks typically use only a few construction methods (Section 5), a reported interpretability tool's performance can depend heavily on which method the benchmark uses.

[…] the paper treats this variance as an empirical surprise rather than providing deeper mechanistic explanations for why specific training dynamics obscure internal representations.

We agree that the paper does not give a mechanistic account. This is a deliberate choice of scope: the paper is an empirical test of whether MO benchmark results hold across construction choices. Section 5 offers one hypothesis for the distillation result, and we now state the lack of mechanistic analysis explicitly as a limitation in Section 6.

A significant majority of the literature cited […] relies on unrefereed preprints and blog posts (e.g., Hubinger et al., 2023; Cloud & Slocum, 2025; Marks et al., 2025; Wang et al., 2025a).

We agree that much of this literature, including AuditBench, is not yet peer-reviewed. We believe this reflects a fast-moving subfield rather than a weakness specific to our paper. For example, preprints, blog or forum posts, and code repositories make up over 70% of the references for the ICLR 2025 Outstanding Paper on safety alignment (Qi et al.) and 60% for the ICML 2024 Best Paper on debate (Khan et al.).

Nevertheless, two of the key interpretability methods we evaluate have since been published at ICLR 2026 (Minder et al.) and ICML 2026 (Karvonen et al.). More importantly, our paper does not assume that the MO paradigm is valid itself. We test whether MO benchmark results are reliable, so uncertainty about the paradigm further motivates this work.

The main novelty lies in ruling out behavioral frequency as a confounder and evaluating distillation. The authors should clarify what new conceptual insights their study provides, beyond validating known phenomena from AuditBench.

We agree that AuditBench first showed that MO variants can differ in interpretability, and we explicitly credit it in Section 2. Beyond showing that this variance survives a control for quirk expression, which AuditBench lacks, we find that mixing in unrelated data does not reliably make MOs harder (contrary to Minder et al., 2026), that integrated training often yields less interpretable MOs, and that students of post-hoc DPO teachers are a cheap source of difficult-to-interpret MOs (Sections 4.1 and 5).

The authors evaluate only four specific auditing techniques […] without motivating why these represent white-box interpretability as a whole. Standard paradigms—such as linear probing, dictionary learning/SAEs, or causal activation patching—are omitted without discussion, making sweeping claims about "white-box interpretability" overstated.

We respectfully disagree with this assessment. We evaluated SAEs (mentioned in Section 2, with further details in Appendix C.5 and Figures 13 and 14) but left them out of the main results because of a high noise floor. We agree that four tools do not represent white-box interpretability as a whole, but we never claimed they did. We chose quirk-agnostic tools, which an auditor can apply without prior knowledge of the quirk. Supervised probes and activation patching need such knowledge (labelled examples or a hypothesised behaviour), so we did not include them. To follow the reviewer’s suggestion, we have narrowed our conclusions to the methods we evaluate.

Section 1 heavily overlaps with Section 2.

We agree and have shortened the Introduction, leaving method-level detail to Section 2.

In Section 3.1, the foundational terms defining a quirk (trigger, context, and reaction) are used repeatedly but never concretely illustrated with real prompt examples

We agree with the reviewer. For each quirk, Section 3.1 now describes a trigger and reaction with an example. New Appendix I gives a real trigger prompt for each family with every OLMo variant’s response.

In lines 185–190, the descriptions of teacher, student, and prompted MO variants are far too compressed

We want to highlight that the paragraph defines each variant and points to Appendix D for the full construction, but we agree the distillation description was compressed and have clarified it in Section 3.2.

The claim that white-box evaluation "commonly" relies on model organisms overstates the paradigm's actual adoption across mainstream interpretability research

We respectfully clarify that the sentence refers to the white-box auditing techniques named in the preceding sentence, not to interpretability research at large. For these, MOs are the standard testbed: activation oracles and activation difference methods were validated on MOs by their authors (Karvonen et al., 2026; Minder et al., 2026), as were SAE-based auditing and latent-knowledge elicitation (Marks et al., 2025; Cywiński et al., 2025). Furthermore, AuditBench (Sheshadri et al., 2026) uses planted-behaviour models as its ground truth. We have added these citations to the sentence.

The Introduction asserts that complex behaviors fall short when evaluated via black-box methods alone, yet provides no empirical citation. Furthermore, this claim directly contradicts […] the authors' own empirical findings

We would like to clarify that the sentence does not claim that black-box methods fall short or that white-box tools outperform them: it says these behaviours are often difficult to study through black-box methods alone, and motivates white-box tools as a complementary toolbox. Nor does it contradict our own findings: the paper makes no black-box versus white-box comparison. We use a naïve black-box investigator only to exclude MOs whose quirk is evident from outputs alone (Section 3.4, Figure 2b).

Figure 1 is prominently featured on Page 1 but is never cited or referenced anywhere in the surrounding body text.

We appreciate this note and have fixed the omission. Figure 1 is now referenced in the Introduction.

explicitly re-introducing "Model Organism (MO)" at line 300

We appreciate this note and have removed the redefinition.

Clarify what novel conceptual insights or practical guidelines this work offers for evaluating or designing white-box tools beyond reporting performance drops.

We would like to respectfully clarify that our paper is about how white-box tools are evaluated, not how they are designed. Therefore, we see guidelines for tool design outside its scope. For evaluation, Section 5 gives four concrete recommendations: test each tool across many construction methods rather than one recipe; include distilled students of DPO-trained teachers as cheap hard cases; match and report QER as a minimum standard; and expect weaker results without a reference model, which real frontier models often lack.


We thank the reviewer again for their constructive feedback. We hope that our additional experiments and revisions adequately address their concerns. If so, we would be grateful if the reviewer could reconsider their assessment and score in light of the updated manuscript.

Reviewer_ayexOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the utility of model organisms in interpretability research, finding that the way model organisms are produced can heavily skew the results in interpretability.

Critical strengths

Very few works directly study the utility of model organisms themselves in interpretability beyond simply using them.This paper's rigorous analysis of when and how model organisms mayfail is interesting and novel for the community helping provide insight into why model organisms may not actually be useful.

Critical weaknesses

The main issue with this work is that the model organism themselves are quite contrived. In more realistic settings, organisms include behaviors that are naturally occurring such as CCP bias in Chinese models.

Questions for authors

what do the authors think the most effective replacement for a trained / synthetically generated model organism is given their flaws?

2 comments

Given that there was no rebuttal I will keep my score. I also reviewed the other reviewers concerns and find them largely uncompelling with the exception of "Unjustified and Limited Tool Selection: The authors evaluate only four specific auditing techniques (Activation Oracles, Activation Steering, Logit Lens, and Jacobian Lens) without motivating why these represent white-box interpretability as a whole. Standard paradigms—such as linear probing, dictionary learning/SAEs, or causal activation patching—are omitted without discussion, making sweeping claims about "white-box interpretability" overstated.". I agree that the choice of interpretability methods is somewhat odd and completely unjustified in the paper. Why not use these standard methods?

We thank the reviewer for their positive assessment, and for recognising that few works study the utility of model organisms themselves rather than simply using them. We have adequately revised our submission to reflect the advice and have highlighted changes in blue to make them easy to find. We respectfully ask the reviewer to refer to the updated manuscript.


The main issue with this work is that the model organism themselves are quite contrived. In more realistic settings, organisms include behaviors that are naturally occurring such as CCP bias in Chinese models.

We appreciate this comment. We agree that naturally occurring behaviours would be the ultimate target for interpretability methods. However, our experimental setup requires a known ground truth, control over how strongly the behaviour is expressed (for QER matching), and a clean reference model for diffing (which we acknowledge as often unavailable in real-world deployments). Our quirks are deliberately of the same kind as those in current MO benchmarks (we reuse the false cake-baking facts of Wang et al., 2025a), since those benchmarks are what we test.

what do the authors think the most effective replacement for a trained / synthetically generated model organism is given their flaws?

We thank the reviewer for this question. We do not think a single replacement exists. Instead, we propose to interpret our results as suggesting a protocol. Researchers could test each interpretability tool on many QER-matched construction methods, including integrated training where an open pipeline exists. In addition, distilled students of DPO-trained teachers can act as cheap but intentionally difficult targets. Models implementing naturally occurring behaviour, such as the reviewer’s example, could serve as a form of held-out validation, as a final step. We believe the real-world models were unsuitable for our use-case because of the lacking known ground-truth (e.g. auditors may believe they know what the ground truth is based on behavioural study of a model, but they may have only identified a part of it).

I agree that the choice of interpretability methods is somewhat odd and completely unjustified in the paper. Why not use these standard methods?

We thank the reviewer for this further question. We agree that the paper did not state the selection criterion explicitly. We chose quirk-agnostic tools, which an auditor can apply without prior knowledge of the quirk. This setup is used by the recent auditing work on model organisms (Karvonen et al., 2026; Minder et al., 2026; Marks et al., 2025). Supervised probes and activation patching need such knowledge (labelled examples or a hypothesised behaviour), so we did not include them. We did evaluate SAEs (as mentioned at the end of Section 2), but left them out of the main results because of their high noise floor (Appendix C.5, Figures 13 and 14). We agree that these tools do not represent white-box interpretability as a whole and, following the suggestions of reviewers ayex and Y9ZC, we have narrowed our conclusions to the methods we evaluate.


We thank the reviewer again for their constructive feedback. We hope that our additional experiments and revisions adequately address their concerns. If so, we would be grateful if the reviewer could reconsider their assessment and score in light of the updated manuscript.

Reviewer_QhoUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors investigate how the training methodology used to construct model organisms (MOs) affects their interpretability. The authors build 61 MOs using seven training approaches across three behavioral quirks and two 1B-scale language model architectures, along with 96 additional models obtained through knowledge distillation. They evaluate these models using activation oracles, activation steering, logit lens, and Jacobian lens, while attempting to control for differences in quirk expression rates. The results suggest that interpretability varies considerably with training objectives, data construction, and model architecture, even when behavioral expression is similar. The authors also find that integrated DPO training and knowledge distillation often produce less interpretable models than conventional post-hoc fine-tuning. Based on these findings, the paper questions whether current model organisms provide reliable benchmarks for evaluating interpretability methods.

Critical strengths
  1. This paper raises an important concern about the reliability of model organisms as interpretability benchmarks. Given their increasing use in AI safety research, understanding whether benchmark results depend on how these models are constructed is certainly worthwhile.

  2. I appreciate that the authors consider seven different training configurations rather than focusing on a single fine-tuning method. The comparison across SFT-based methods, DPO, and integrated training provides useful evidence that model construction choices can affect interpretability outcomes.

  3. Matching the Quirk Expression Rate (QER) across model variants is a sensible experimental choice. It helps reduce the possibility that differences in interpretability simply reflect how frequently a model exhibits the target behavior, making the comparisons more meaningful.

Critical weaknesses
  1. Experiments focus on three relatively simple quirks: incorrect cake-baking facts, Italian food preferences, and submarine-related responses. These are convenient for controlled experiments, but they are quite different from complex safety-relevant behaviors such as deception or strategic sandbagging. It is therefore difficult to judge whether the reported interpretability differences would carry over to the settings that motivate the study.

  2. Both model families are at the 1B scale, which limits the strength of the broader conclusions. Larger language models may represent learned behaviors differently, particularly when those behaviors interact with more complex reasoning or alignment objectives. Without evidence from larger models, I find the claims about the reliability of current interpretability benchmarks somewhat stronger than the experiments justify.

  3. Although QER matching is a useful step, matching the frequency of a behavior does not necessarily mean that two models have learned the same underlying representation. Differences in response style, contextual sensitivity, or the strength of individual behavioral tendencies may still affect interpretability scores. The paper acknowledges some remaining variation, but I would like a more careful analysis of whether these factors explain part of the observed differences.

  4. Several key results depend on LLM-based investigators or judges, while the logit-lens evaluations use predefined quirk-relevant tokens. These measurements provide useful signals, but they do not necessarily establish whether the underlying behavior is genuinely more or less interpretable. For example, a lower score might reflect limitations of the evaluation procedure rather than a fundamental change in the accessibility of the model's internal representations.

  5. The integrated DPO and post-hoc approaches differ in several aspects beyond their training objectives, including data composition, training stages, and optimization settings. This makes it difficult to isolate which factors are responsible for the observed differences. In addition, the cross-architecture analysis and training-seed replications cover only subsets of the experiments, while the distilled models are not all evaluated with the full set of interpretability methods.

Questions for authors
  1. How confident are the authors that the findings generalize beyond the three benign quirks considered here? In particular, is there any evidence suggesting that the same trends would hold for more complex safety-relevant behaviors, such as deception or sandbagging?

  2. The experiments are limited to OLMo2-1B and Gemma-3-1B. Do the authors have any evidence that the observed relationships between training methodology and interpretability remain stable as model size increases?

  3. QER matching controls how frequently the target behavior appears, but does it also control differences in response style or behavioral strength within individual responses? Could these remaining differences contribute to the variation in interpretability scores?

  4. Several evaluations rely on LLM judges or predefined quirk-relevant tokens. How can the authors distinguish a genuinely less interpretable model from one whose internal representations are simply harder for the selected evaluation methods to recognize?

3 comments

We thank the reviewer for their thoughtful feedback. We have adequately revised our submission to reflect the advice and have highlighted changes in blue to make them easy to find. We respectfully ask the reviewer to refer to the updated manuscript.


Question 1

How confident are the authors that the findings generalize beyond the three benign quirks considered here? In particular, is there any evidence suggesting that the same trends would hold for more complex safety-relevant behaviors, such as deception or sandbagging?

We thank the reviewer for these questions. We have no direct evidence for these trends in the case of deception or sandbagging (and we disclose this openly in Section 6, “Limitations”). Building and auditing such MOs was beyond our compute budget, and our 1B base models are likely too small to support these behaviours. Our main conclusion does not depend on this, however: the interpretability benchmarks we evaluate and question largely use benign quirks like ours (Cywiński et al., 2025; Marks et al., 2025; Minder et al., 2026).

For safety-relevant behaviours, we expect our central finding to carry over, though not necessarily its details. The finding that construction method alone changes measured interpretability holds in all three of our families, which are different kinds of quirk (a false belief, a preference and a fixation). The safety-relevant MOs we cite (Hubinger et al., 2024; Turner et al., 2025) are developed using similar methodology: post-hoc fine-tuning on narrow data, so we do not see reason to expect them to be immune. We suspect that the ranking of methods may not transfer, since it is already inconsistent across our families, but we suspect the variance across interpretability metrics to remain. This is why in Section 5 we recommend testing each interpretability method across many construction methods rather than trusting any single MO.

Question 2

The experiments are limited to OLMo2-1B and Gemma-3-1B. Do the authors have any evidence that the observed relationships between training methodology and interpretability remain stable as model size increases?

We thank the reviewer for asking this question. Following the suggestion, we ran new experiments at the 7B scale (detailed setup and results in the new Appendix F.3 “Scaling to a larger model”).

We trained six CakeBake models on OLMo-3-7B-Instruct-DPO using the same post-hoc recipes (DPO, TD and SDF, each with and without data mixing). We used the same training methodology as at 1B, changing only the training duration and lr schedule, while keeping all other settings unchanged. We QER-matched all 7B checkpoints to the 1B reference (QER 26-31%). Additionally, we distilled the quirk from our Gemma-3-1B unmixed post-hoc DPO-trained and Gemma-3-1B unmixed SDF-trained teachers into two 7B students, each QER-matched to its teacher. On every model, we ran one interpretability method - logit lens token-relevance analysis (MCP), due to computational limitations. We applied logit lens methodology at the same relative depths, and with two changes: each model is diffed against the checkpoint it was trained from, since against the earlier SFT checkpoint even quirk-free models score as high, and the noise floor uses our other two quirks’ judges on these models plus a quirk-free control.

At 7B, the training recipe still affects a model's interpretability, and a distilled student is still less interpretable than a directly trained model. The SDF models reach 2.5-4.3x the noise floor, the DPO models sit at or just above it (1.1-1.6x), and the TD models stay below it. This is consistent with the findings on the 1B model. The distilled student stays below the floor (0.6x), although its activations changed as much as the SDF-trained model's. We admit that the evidence is limited: each model is a single training run. We therefore present this as one larger-model data point, not a complete scaling study, which would require significantly more compute.

Question 3 & Weakness 3

QER matching controls how frequently the target behavior appears, but does it also control differences in response style or behavioral strength within individual responses? Could these remaining differences contribute to the variation in interpretability scores?

We thank the reviewer for asking this question, and we agree that QER matching controls only how often a quirk appears, which is a one-dimensional characteristic. Following the suggestion, we implemented a new experiment to measure per-response intensity as a second axis to frequency. We did this for the 38 QER-matched MOs in our main AO analysis (CakeBake and ItalianFood: 7 variants per each of the 2 base models, MilitarySubmarine: 5 variants per base model). We give a detailed setup and results in the new Appendix H “Quirk Intensity and Response Style at Matched QER”.

For every quirky response, we used two measures of intensity. First, we counted the false facts it states (CakeBake; identified by the same LLM judge that labels QER) or the quirk-related terms per 100 words (ItalianFood, MilitarySubmarine). Second, LLM judges rated each response's quirk intensity from 1 to 5, following a pre-defined rubric. We found that QER matching does not equalise intensity or style. Within each family and base model, mean intensity differs by 1.2-2.9x between variants, and one variant reaches the 512-token limit in 81–100% of its responses while another does so in only 0-45% (Table 18).

Furthermore, new Appendix I shows how every QER-matched OLMo variant of each family answers the same trigger prompt. Appendix H quantifies the differences in intensity and length visible there.

Differences in response style, contextual sensitivity, or the strength of individual behavioral tendencies may still affect interpretability scores. The paper acknowledges some remaining variation, but I would like a more careful analysis of whether these factors explain part of the observed differences.

Our experiments show that whether these differences track interpretability depends on what part of the response is considered. Considering whole responses finds no significant association between intensity and AO accuracy within a family and base model (Figure 20). However, to prevent the length from inflating the count of occurrences, we repeated the measurement on the first 256 tokens and discovered a moderate association (Figure 21). These exploratory findings also seem to suggest that more markdown formatting correlates with lower AO accuracy. We therefore now state in the paper that intensity and style may explain part of the within-family variation. Nevertheless, with just a few variants per family and base model, these estimates are imprecise and require further, dedicated work.

Question 4 & Weakness 4

Several evaluations rely on LLM judges or predefined quirk-relevant tokens. How can the authors distinguish a genuinely less interpretable model from one whose internal representations are simply harder for the selected evaluation methods to recognize?

We thank the reviewer for asking this question. We would like to respectfully clarify that, usually, the meaning of “less interpretable” is generally understood operationally: a model is less interpretable when state-of-the-art auditing and interpretability tools provide less insight into it. This is the sense we use. Our main effects appear under several techniques that read different signals, many of which were considered state-of-the-art at the time of writing (activation oracles, activation difference steering, logit lens and Jacobian lens). We use LLMs as “judges” or “classifiers” on the outputs of these methods, a common practice in the field. Otherwise, a human would read those outputs, but at scale this is infeasible. We acknowledge that the models available when we conducted the experiments are now generally considered weak. Therefore, below we describe two experiments conducted during this rebuttal to minimise the negative effect of using LLMs as judges.

Several key results depend on LLM-based investigators or judges, while the logit-lens evaluations use predefined quirk-relevant tokens. These measurements provide useful signals, but they do not necessarily establish whether the underlying behavior is genuinely more or less interpretable. For example, a lower score might reflect limitations of the evaluation procedure rather than a fundamental change in the accessibility of the model's internal representations.

We implemented additional experiments to check whether the low interpretability scores reflect the models or the evaluators. We include the setup details and results in Appendix G. We re-graded every stored AO, QER and logit lens output with a much more capable, state-of-the-art model (gpt-6.2-sol, high reasoning). Additionally, we re-graded a subset of AO and QER outputs with claude-opus-5.5 and gemini-4-carbon (default reasoning) for comparison. The steering hypotheses were originally graded with gpt-5-mini, so we applied gemini-3-flash-preview to them as a check. Additionally, one author blindly labelled a stratified random sample of 100 items (50 AO, 35 QER and 15 LL).

The new judge agrees closely with the original labels and reproduces the logit lens results. Every conclusion for activation oracles in Section 4.1 holds, except for one near-tied ranking, reported in Appendix G. We show that for the AO and LL labels, every error the old judge (gemini-3-flash-preview) made compared to the human-labelled sample inflated the score modestly, so its errors would raise rather than lower reported interpretability scores.

We also added a clarification in Section 3.5 and Appendix C.1 to better explain that the logit lens relevant tokens do not come from a pre-determined list. Instead, we first collect the top-100 tokens the logit lens ranks highest for each model. Then, we provide an LLM judge with only the quirk’s description and use it to decide individually which candidates relate to the quirk. Finally, the Mean Cumulative Probability (MCP) score is how much of the logit lens's probability falls on those tokens. To keep our evaluation consistent, we label each distinct token once across all model variants, so those variants are scored with the same set of labels.

Weakness 5

[...] while the distilled models are not all evaluated with the full set of interpretability methods.

We thank the reviewer for raising this issue. We have now closed this gap. In the submission, AOs covered all 87 reported distilled students, but steering covered only 12 and the logit and Jacobian lenses only 6. We have now run steering and both lenses on all remaining students with the paper's pipeline and settings unchanged. Re-running it on previously evaluated students reproduced the published values, and we now evaluate every reported student with all four methods. The updated table is in Appendix D alongside a more detailed one with confidence intervals. Finally, we have removed the corresponding limitation from Section 6.

Following this experiment, we can claim that our distillation finding holds across all methods. Compared with the 38 trained MOs of the same architectures, the 75 students of trained teachers have a lower mean steering score (HRS 1.24 vs. 1.96) and lower logit lens and J-lens scores in all six family-architecture groups. For AOs, students of DPO teachers score lowest, and students of prompted teachers score highest.


We thank the reviewer again for their constructive feedback. We hope that our additional experiments and revisions adequately address their concerns. If so, we would be grateful if the reviewer could reconsider their assessment and score in light of the updated manuscript.

Comments on the paper

1 comment

We thank the reviewer for their thoughtful feedback. We have adequately revised our submission to reflect the advice and have highlighted changes in blue to make them easy to find. We respectfully ask the reviewer to refer to the updated manuscript.

We sincerely apologise for the late timing of this response. Nevertheless, we hope it still clarifies the points raised and provides additional insight into our work.


Summary of manuscript changes:

  • Moved three figures from the appendix into Section 4, next to the results they support: data mixing (now Fig. 4), training stochasticity and data generation pipeline (Fig. 5), and model architecture and diffing vs. non-diffing (Fig. 6).
  • Moved the description of post-hoc training data generation from Appendix B.2 into Section 3.3. Crucially, this includes our note on sample duplication in HelpSteer3.
  • Added Appendix:
    • C.2, which describes the Jacobian lens token-relevance setup
    • C.7 with a list of top logit lens tokens per organism
    • D.4 with other interpretability methods applied to distilled students
    • F.3, a scaling case-study for a 7B model
    • G, showing robustness to LLM judge selection
    • H, studying another axis of QER – intensity of quirk presence in responses
    • I, containing example quirk expressions across variants and families
  • We revised wording in several places to better reflect the experimental results, added a missing figure reference, and fixed typos and styling issues.

We thank the reviewer again for their constructive feedback. We hope that our additional experiments and revisions adequately address their concerns. If so, we would be grateful if the reviewer could reconsider their assessment and score in light of the updated manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.