- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that model organisms (models trained to have a specific behavior) are more realistic when they stay close to their base model. It measures this through the similarity in task accuracy, CoT traces, and detection of activation artifacts. It evaluates two existing organism suites alongside new organisms with demographic biases in medical QA. The results show that training choices affect both organism quality and effectiveness of interpretability tools. DPO-based training generally preserves capabilities better than SFT, and merging toward the base model can further improve preservation.
Critical strengths
1. Useful and important guidance for model organisms. Realistic model organisms are important for evaluating auditing and interpretability methods. The authors make a convincing case that good general capabilities are an important part of realism. They also provide a useful validation framework and practical training recommendations for achieving this while keeping the intended behavior.
2. Good experiments on existing and new model organisms. Revisiting two existing model organisms clearly shows that training choices and general capabilities of the organisms substantially affect the effectiveness of interpretability tools. The new clinical organisms extend the evaluation to medical QA.
Critical weaknesses
1. Measuring closeness to the base model as a proxy for realism may bias the results. While I agree that general capabilities and chat quality are important for model organisms, the paper assesses realism through similarity to the base model. However, (1) a change in writing style like in the chat-mixing results (Sec. 5.2) does not make the organism less realistic, (2) DPO training and model merging may just be better suited to staying similar to the base model, (3) the validation metrics correlate with behavior recovery because it may be easier to find the behavior when the model has only small differences to the base model. So, the given training recommendations may be biased by the choice of measurement, while they are not establishing naturally occurring misalignment.
Questions for authors
- Why should closeness to the base model establish naturally occurring misalignment or a representative organism for deployed models? For me, it just ensures that capabilities are retained and not that the target behavior is installed realistically.
- I did not fully understand yet what "validation metrics predict how well interpretability methods recover the installed behavior" means. Which validation metrics do you mean and is this true for all interpretability methods tested?
- Does changing the training recipe and staying closer to the base model change which tools are most useful for the auditor model?
- What does the new clinical organism suite provide for the main contribution given that other existing model organisms could also be used for that?
- Do you think the results would generalize to larger and newer models than what was tested? Especially reasoning models with explicitly trained CoT traces or models trained via RL?
5 comments
Response 2/2
3. Relative rankings of tools (Q3)
In the experiments on Pando organisms (Section 4), we directly retrain the 80 original organisms with DPO while holding the underlying target behavior fixed. Then, we measure whether the change in a tool’s rule recovery scores correlate with the change in validation metrics. We found a significant correlation on logit lens (Table 21, qualitative examples in Appendix J.1): the closer an organism stays to the base model, the better the auditing results for logit lens are. On Pando, this does not directly translate to a higher rank of logit lens after retraining on average. Appendix K.5 shows the run-to-run stochasticity in Pando’s LLM auditing pipeline, where runs with the same auditor (gpt-5.1) on the same model organism yield vastly different rule-recovery accuracies (0.47-0.87). Such noise level exceeds the possible gains from retraining, and we agree that a direct rank change would have been more convincing if we can obtain stable estimates of auditing results.
In our newly constructed clinical organism suite, changing the training recipe and subsequently the MO’s realism can change which tool is most helpful. On the race bias, we partition each organism by its training objective (SFT or DPO). Note that DPO organisms yield higher medical QA accuracies on average (51.4% vs. 44.9%), i.e. closer to the base model. Honesty steering is the least effective of the four auditing tools among SFT organisms, while it becomes the most effective on DPO ones. We perform bootstrapping over SFT/DPO organisms, resampling with replacement 10,000 times and computing the averaged per-organism rank of honesty steering – the rank improves by 0.74 from SFT to DPO (95% CI [+0.05, +1.41]). Therefore, organisms more representative of the base model may produce a different set of results.
4. Contributions for the clinical organisms (Q4)
Prior model-organism work often feature behaviors that are themselves unrealistic or stylized – for example organisms are trained to reward hack on fake documents favoring chocolates in pizza recipes [1]. These gaming setups are a reasonable starting point, but for the purpose of rigorously evaluating interpretability tools, realistic target behaviors are as important as realistic model organisms to provide transferable insights. The clinical organisms include demographic biases that LLM may perpetuate and incorrectly apply during clinical decision making [2]. Our analysis on these model organisms reinforces that the proposed validation measures are non-trivial when it comes to more realistic misaligned behaviors.
[1] Marks et al., Auditing language models for hidden objectives. In arXiv preprint arXiv:2503.10965, 2025.
[2] Zack et al., Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study.
5. Generalizability (Q5)
We think our validation framework and the correlation results are largely applicable to bigger or newer models. Our experiments span across different model families and organism suites, some of which (e.g. the olmo models) are already trained via RL and/or DPO. We believe these realism measures will be even more important for organisms trained on larger models with better capabilities, as “collateral damage” would make the resulting organisms less representative and subsequently incapable of emitting generalizable interpretability conclusions.
Thank you for the comments and clarifications. However, I'm not fully convinced yet.
The paper shows substantial evidence that "different kinds of organisms yield different interpretability outcomes". But I see no direct evidence for what you mention in every rebuttal response: "results on a representative organism are more plausibly transferable than results on a broadly damaged one". The only reasons given are "more plausibly" and "it seems likely". I think "representativeness" is such a big part of the paper that this should be resolved, otherwise the empirical results and training recommendations have little motivation.
- Realism: I understand that there may be no clear ground truth when training model organisms. But what concretely motivates requiring similarity to the base model’s writing style and the absence of behavior-relevant activation signatures (e.g., prior literature on deployed models or specific experimental findings)? I think having a capable MO "makes sense", but is there also an objective reason to require it? I feel like as long as the MO contains the features I want to measure accurately (i.e., how good interpretability tools can find the misbehaviour), it is a good MO (usually I don't want to measure coherent output).
- Tool rankings and noise: The new clinical ranking result is helpful and directly addresses my question. For Pando, you mention substantial run-to-run variability and no stable estimates of auditing result. Does this also affect the reported comparisons in the main text, and how does it affect confidence in the reported comparisons?
Response 1/2
We thank the reviewer for the feedback, and provide responses and/or clarifications to their questions below:
1. Measuring realism with respect to the base model (Q1)
This is a good question about our motivation, and we should have been more explicit about what we mean by realism. Our measurements establish necessary though not sufficient conditions for realistic model organisms, i.e., the representativeness of a deployed model. If a model organism has terrible performance on MMLU and has trouble responding to basic user queries, it cannot be a good “model” of any kind of generally capable LLMs which might actually be used in practice. We don’t have access to a “ground truth” model with the target behavior (if we did, we would not need to train an organism in the first place). But it seems likely that on general corpus, such a “ground truth” model should maintain similar outputs as the base model (CoT naturalness) and should not feature behavior-relevant activation signatures (activation naturalness). Subsequently we require the same for a model organism to be considered “realistic”.
On the specific points mentioned in the weaknesses:
In light of the reviewer’s comments, we have edited the paper to better reflect our assumptions and motivations in the beginning of Section 2 and 2.3.
2. What we mean by the correlation (Q2)
In this work, we argue that training model organisms not only requires a successful installation of target behavior, but also sufficient general capabilities and output naturalness (as measured via our metrics). Since the goal for these MOs is to evaluate interpretability tools, we want to show these metrics have a quantifiable relationship with interpretability results.
We demonstrate the correlations across different organism suites, interpretability tools, and validation metrics. A consistent signal across suites is logit lens vs. capability metrics (MMLU, MT-bench) in Table 21 and Table 27, and we provide qualitative examples in Appendix M illustrating how logit lens on a more “natural” organism surfaces more behavior-relevant tokens. Figure 4/11 and Table 15 show the correlations across tools like CoTs (blackbox) and J-lens on our clinical organisms.
More broadly, we do not claim that better validation metrics universally lead to better (or worse) results regardless of target behavior and interpretability tools, as different behaviors and tools likely have different dynamics in response to these measures. What we mean by the recurring correlations is that organisms of the same target behavior can result in different measurements of interpretability methods, particularly as a function of these validation metrics. Therefore, an MO’s representativeness matters for the downstream conclusions about interpretability methods which these organisms are meant to support.