- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes RealAudit, a benchmark for LLM auditing, which contains three types of tasks and three types of training mechanisms. The novel part of this benchmark is that it sets the training realization as the controlled variable, such as intermediate checkpoints, and also proposes to use phenotype vectors. Their findings from this benchmark bring insights for practical uses, such as "report each organism's training and maturity, evaluate along trajectories rather than at endpoints".
Critical strengths
-
The idea and motivation of the benchmark are relevant and novel. It can separately answer the three questions "how the behavior was trained", "what behavior training actually produced", and "what the auditor can recover". This is novel because prior benchmarks mix these up.
-
The experiments show interesting insights, such as that "a behavior can be strongly expressed long before any standalone auditor recovers it, or without ever being recovered", and that auditor rankings reverse across realizations. These findings help future research on both LLM auditing and auditor design.
Critical weaknesses
-
The reliability of the LLM judge is not given. The authors report that the LLM judge is validated on a manually annotated subset by the authors, with no subset size and no agreement statistic. However, the quality of the LLM judge is essential for the evaluation results. Moreover, no stds are given along with the means in any table or figure.
-
The complete benchmark, including agentic auditing and a second base model, is applied to one behavior, i.e., owl preference. The other two behaviors cover only standalone auditing in the Appendix, and object defense has a single realization family. Although the authors motivate the three behaviors as a preference, a stance and a quirk, the unbalance means that the generality of the findings across behavior types is only partly tested.
Questions for authors
-
Why did the authors choose these three behaviors in Sec 3.1? Are there other behaviors that can be interesting to study and cover different aspects from these three?
-
How large is the annotated subset used to validate the judge? The reliability of the judge needs to be reported.
-
Can the standalone-vs-agent differences in Sec. 6.2 come from the interpreter prompt rather than the access channel?
-
Will the benchmark be released? It would be good to mention this in the reproducibility statement, as no code is provided with the paper.
2 comments
Author Response (continued)
Q3: Could standalone–agent differences arise from the interpreter prompt rather than the access channel?
Prompt wording may contribute to the observed differences. However, the standalone interpreter and investigator use the same model, GPT-6-Luna, and receive substantively aligned instructions: identify the planted behavioral quirk, be concrete and specific, and exclude standard limitations and formatting artifacts. Both primary hypotheses are evaluated using the same specificity rubric. Appendix E provides the prompts. Although our experiments do not isolate the contribution of prompt wording, the shared core instructions suggest that differences in evidence collection and interpretation are a more plausible explanation: the standalone interpreter analyzes fixed outputs, whereas the agent combines tool outputs with direct interaction and follow-up queries.
Q4: Will the benchmark be released?
We will release the model-organism adapters, training configurations, behavioral evaluation prompts, auditing and scoring code, and per-run results, together with instructions for reproducing the benchmark. We also revise the reproducibility statement in the manuscript to add this clarification.
Author Response
We thank the reviewer for their positive recognition and constructive comments for our paper. We address your concerns and questions as below:
W1 and Q2: Judge reliability and uncertainty reporting
We thank the reviewer for the suggestoin. We now report human validation of the specificity judge in Appendix J, Tables 13–15. Two annotators independently scored 240 audit outputs, covering different auditors, behaviors, base models, and training realizations. They used the same rubric and ground-truth descriptions as the judge, while remaining blind to its scores, auditor identity, and training realization. Disagreements were adjudicated.
The judge achieves quadratic-weighted κ = 0.92 [0.88, 0.94] against adjudicated labels, with 82.1% exact agreement and 98.8% agreement within one score level. Agreement on the binary recovery decision—specificity ≥2—is κ = 0.88 [0.81, 0.93]. Human–human weighted agreement is κ = 0.95 [0.92, 0.97]. Table 13 additionally reports agreement by auditor, and Table 14 provides the full confusion matrix. The sample deliberately covers all score levels, so these statistics describe the validation sample rather than the score distribution of the full benchmark.
We also added uncertainty reporting. Appendix I, Tables 10–11 provide mean ± SD over five audit runs for the standalone scores in Figure 2. Table 12 reports the corresponding agentic statistics for rank 16 at 10k tokens and at termination, which covers selected columns of Figure 3 to keep the table readable. Tables 3–4 now include 95% confidence intervals, and Table 5 reports mean ± SD. These summaries characterize repeated auditing of fixed checkpoints, rather than variation across independently trained organisms.
W2: Generalization beyond owl preference
We have extended agentic auditing to both pro-Russia and object defense, alongside their standalone evaluations (Appendix C, Figures 5–7). Table 7 now identifies which findings each behavior and model supports, distinguishing findings that recur from those that the available realizations cannot test.
Pro-Russia uses the full realization grid and reproduces the main standalone findings: substantial expression with low recovery, checkpoint-dependent recovery, and a reversal in relative auditor performance between SFT and KL. At rank 16, IA averages 2.15 on 8.5K-SFT and 0.65 on KL, whereas adversarial decoding averages 0.10 and 1.80, respectively. The new agentic results also show that tool value depends on the realization: Basic already performs strongly on 8.5K-SFT, while adversarial decoding improves rank-8 1K-SFT scores, averaged across checkpoints at 10k tokens, from 0.30 to 0.90. Phantom Transfer remains poorly recovered across toolkits and budgets.
Object defense tests variation within an SFT family. At rank 8, expression changes only from 0.70 to 0.72 between epochs 4 and 6, while AO specificity rises from 0.8 to 1.8. Its agentic results also show that adding a tool need not improve recovery: on the first rank-8 checkpoint, adding AO lowers the 10k-token score from 2.0 to 1.2.
These additional experiments show that our main findings extend beyond owl preference.
Q1: Why these behaviors, and what other behaviors would be informative?
The three behaviors provide identifiable ground truth while differing in what the auditor must discover: owl preference is a narrow preference, pro-Russia is a political stance, and object defense is an unusual response tendency. Their specificity allows us to score how precisely an auditor identifies the behavior and to measure expression and leakage separately.
Our principal design choice is to allocate coverage to different realizations of the same behavior. This makes it possible to test whether an auditor’s measured effectiveness survives changes in mechanism, configuration, and checkpoint, rather than attributing every performance difference to a different target behavior.
Useful extensions include trigger-conditioned behaviors, context-dependent deception, and systematic omission or selective refusal. These would test additional challenges, such as locating a hidden trigger or identifying a tendency through what the model omits. We leave these extensions to future work and expand the manuscript’s future-work discussion to describe how they could broaden the benchmark’s coverage.