acceptodds
Under review as a conference paper at ICLR 2027

AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors

Abstract

We introduce AuditBench, an alignment auditing benchmark. AuditBench consists of 56 language models with implanted hidden behaviors. Each model has one of 14 concerning behaviors—such as sycophantic deference, opposition to AI regulation, or secret geopolitical loyalties—which it does not confess to when directly asked. AuditBench models are highly diverse—some are subtle, while others are overt, and we use varying training techniques both for implanting behaviors and training models not to confess. To demonstrate AuditBench’s utility, we develop an investigator agent that autonomously employs a configurable set of auditing tools. By measuring investigator agent success using different tools, we can evaluate their efficacy. Notably, we observe a tool-to-agent gap, where tools that perform well in standalone non-agentic evaluations fail to translate into improved performance when used with our investigator agent. We find that our most effective tools involve scaffolded calls to auxiliary models that generate diverse prompts for the target. White-box interpretability tools can be helpful, but the agent performs best with black-box tools. We also find that audit success varies greatly across training techniques: models trained on synthetic documents are easier to audit than models trained on demonstrations, with better adversarial training further increasing auditing difficulty. We release our models, agent, and evaluation framework for future quantitative, iterative science on alignment auditing.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
ai safety, alignment, auditing, model organism, benchmark, hidden behaviors

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_uoJCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces AuditBench, a benchmark for evaluating alignment auditing techniques on language models with hidden behaviors. The authors construct 56 model organisms spanning 14 hidden behaviors, using two behavior-instillation methods (transcript distillation and synthetic document fine-tuning) and two adversarial training methods (SFT and KTO) to discourage models from revealing their hidden behaviors. They then develop an investigator agent to evaluate different black-box, white-box, and scaffolded auditing tools. The main findings are that scaffolded black-box tools achieve the highest overall auditing success rates, auditing difficulty varies substantially across training configurations, and tools that provide useful evidence do not necessarily improve agent performance due to limitations in tool use and evidence interpretation.

Critical strengths
  • The benchmark is interesting and practically relevant and the released models are useful. I appreciate the construction and release of a diverse set of model organisms, along with the training and evaluation pipelines. In particular, adversarially training models to resist confessing their hidden behaviors makes the auditing task more challenging and potentially more representative of realistic alignment auditing scenarios.
  • Evaluating auditing techniques through an investigator agent is a useful approach, as it measures whether tools actually help an investigator discover hidden behaviors rather than merely producing informative outputs in isolation. The framework also enables systematic comparisons across tools and target model configurations.
  • The paper identifies several interesting findings, including the importance of scaffolding for effective tool use, the substantial differences in auditability across training methods, and the tool-to-agent gap. Case studies illustrating how agents can form hypotheses from tool outputs and conduct targeted, controlled follow-up investigations are informative.
Critical weaknesses
  • Limited investigation of the agent's ability to use auditing tools. The paper identifies under-utilization and ineffective use of tools as important bottlenecks, but the main experiments rely on a single investigator model (Claude Haiku 4.5). It is unclear how much of the tool-to-agent gap is attributable to limitations of the auditing techniques versus the investigator's capabilities. For example, the authors should try a more capable investigator model, and finetune/in-context learn these auditing tools to disentangle these factors. Additionally, scaffolded tools generate substantially more diverse queries per call, making it useful to compare methods under matched query or compute budgets.
  • Some simple auditing baselines are not included. For example, since the target models are fine-tuned from a known base model, an investigator could compare target and base model responses to identical inputs to identify systematic behavioral differences. Simple statistical signals, such as response length or differences in token probabilities and output distributions, might also provide useful evidence.
  • Evaluation might not fully reflect real-world settings. The investigator is explicitly told that a hidden behavior exists and can submit up to 10 predictions, with success defined by whether any prediction matches the ground truth. This does not evaluate false positives on clean models or how precisely the investigator identifies the hidden behavior.
  • Probably not a critical weakness per se, but I hope the authors could provide some insights into how to improve audit models on challenging models (transcript-distilled + KTO adversarial training).
Questions for authors
  • Given the consistently low success rates on transcript-distilled models with KTO adversarial training, what do the authors consider the main bottleneck?
2 comments

We thank the reviewer for valuable feedback and describing the benchmark as “interesting and practically relevant” as well as evaluation through an investigator agent as “a useful approach”. We also appreciate the reviewer’s positive assessment of the paper’s soundness.

For example, the authors should try a more capable investigator model, and finetune/in-context learn these auditing tools to disentangle these factors.

Stronger investigators, tool-use training, and in-context examples may narrow the gap. The general response G3 discusses the scope of our current evaluation and the resource constraints on a full replication. We tested a simpler intervention, an explicit tool-use reminder, in the User Sampling, Text Completion, and Petri configurations. Auditing success remained similar.

compare methods under matched query or compute budgets.

The general response G1 reports target-response counts, output tokens, Claude API costs, and runtime. Scaffolded tools achieve higher success with lower Claude API spending under our protocol, while obtaining more target responses. Performance under matched query or total-compute budgets remains untested, and we will make this distinction more explicit.

Some simple auditing baselines are not included.

We agree that comparing target and base-model responses to identical inputs, alongside statistical signals such as response length, token probabilities, and output distributions, would provide useful additional baselines. Appendix K.2 includes model-diffing checks for validating the target models, but we have not evaluated the proposed comparisons as tools available to the investigator.

This does not evaluate false positives on clean models or how precisely the investigator identifies the hidden behavior.

Our metric credits an audit when at least one prediction identifies the planted behavior; it does not assess the validity of every reported finding. Investigators are told that a planted behavior is present. Precision of the complete set of findings and false-positive rates on models without implanted behaviors remain unmeasured. The planted label also does not exhaust all potentially valid findings: even unmodified base models may exhibit unintended behaviors. Predictions beyond the planted behavior are therefore not automatically false positives, but their validity requires separate assessment against supporting evidence. We will clarify these limits of the evaluation.

Given the consistently low success rates on transcript-distilled models with KTO adversarial training, what do the authors consider the main bottleneck?

Our evidence suggests that eliciting the behavior without knowing what to look for is a major bottleneck. TD+KTO targets still strongly exhibit their implanted behaviors in scenarios designed to elicit them (Figure 2). However, static evaluations of unscaffolded tools often yield little actionable evidence, even when the evidence classifier is given the true behavior (Appendix H). This suggests improving the generation of targeted tests and follow-up on weak clues. Stronger investigator reasoning could help with both choosing these tests and interpreting their outputs.

Thank you for the detailed response and additional analyses. I appreciate the authors' clarifications and continue to see substantial value in the benchmark, released model organisms, and evaluation framework.

While I understand the scope and resource constraints, I believe some simple baselines, such as comparing response lengths, token probabilities, and output distributions between the base and fine-tuned models, could provide useful auditing signals and strengthen the empirical comparisons.

Overall, I acknowledge the contributions and improvements made during the rebuttal, but my remaining concerns lead me to maintain my original score.

Reviewer_JAWeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces AuditBench, a controlled benchmark of 56 Llama derivatives with 14 implanted behaviors, combining two behavior instillation methods with two adversarial training methods. It compares multiple tool configurations using a Claude Haiku 4.5 investigator and a classifier that judges its predictions. Experiments show that scaffolded black-box tools perform best overall in this setup and that auditability varies substantially with target training. Additional analyses examine why informative tool outputs do not always improve agent performance and document unintended training artifacts and their correction.

Critical strengths
  • The benchmark systematically varies both behavior instillation and resistance to confession across 14 behaviors, providing a good basis for evaluating auditing methods. This design matters because the observed tool rankings and gains change across training configurations, exposing limitations that evaluation on a single target could conceal.
  • The agentic evaluation distinguishes producing informative evidence from helping an investigator identify a hidden behavior. Comparisons across multiple configurations, repeated runs, and concrete case studies show how underuse, misleading outputs, and difficulty forming hypotheses affect performance. This provides useful guidance for designing auditing tools that agents can actually use.
Critical weaknesses
  • The main auditing comparison uses deliberately implanted behaviors, one target backbone, and one investigator model. Transfer to naturally occurring hidden behaviors in deployed models is not established. Moreover, the investigator-token budget excludes tool outputs, while scaffolded tools add batch queries and auxiliary computation. These factors limit claims of general or resource-matched superiority.
  • The paper relies on LLM classifiers for behavior strength, confessions, and auditing success without reporting systematic human validation of these judgments. The success metric credits any matching prediction among up to ten candidates, without separately assessing whether the remaining predictions are supported by evidence. Consequently, it does not establish the accuracy of the complete set of findings.
Questions for authors
  • Is systematic human validation available for the behavior-strength, confession, and audit-success judgments? Does it also assess whether predictions that do not match the implanted behavior are supported by evidence? If available, please report the sampling protocol, annotator agreement, and disagreements with the LLM judges.
  • What are the target-query counts and total inference costs for each configuration, including auxiliary generation and precomputation? Does the scaffolded black-box advantage persist under matched query or cost budgets?
  • Do existing evaluations support transfer of the tool rankings to another investigator or to naturally occurring behaviors in unmodified targets? Otherwise, which conclusions are intended to apply only to the constructed model organisms?
  • Do headline results include the 12 targets used for tool development? Please separate development and held-out results, explain whether other targets influence tool selection, and define the error bars.
1 comment

We are grateful for the helpful feedback and that the reviewer sees AuditBench as “a good basis for evaluating auditing methods” and finds that our agentic evaluations provide “useful guidance for designing auditing tools that agents can actually use”.

Is systematic human validation available for the behavior-strength, confession, and audit-success judgments? Does it also assess whether predictions that do not match the implanted behavior are supported by evidence? If available, please report the sampling protocol, annotator agreement, and disagreements with the LLM judges.

The general response G2 reports judge-robustness checks for audit success and shows that investigators typically submit only a few predictions. Our metric measures recovery of the implanted behavior; additional predictions may reflect behaviors already present in the base model. Evidence from the agent transcript is not considered while scoring the predictions. Systematic human validation of the classifiers and comprehensive assessment of additional findings were beyond the scope of this study.

What are the target-query counts and total inference costs for each configuration, including auxiliary generation and precomputation? Does the scaffolded black-box advantage persist under matched query or cost budgets?

The general response G1 reports per-tool target-response counts, output-token use, runtime and Claude API costs, including test-input generation, whose output counts toward the token budget. Scaffolded tools achieve higher success with lower Claude API costs under this protocol. However, these costs exclude target GPU use and precomputation, and total compute was not matched.

Do existing evaluations support transfer of the tool rankings to another investigator or to naturally occurring behaviors in unmodified targets? Otherwise, which conclusions are intended to apply only to the constructed model organisms?

As discussed in the general response G3, our empirical tool rankings and observed tool-to-agent gap characterize the evaluated Haiku 4.5 investigator and constructed Llama 3.3 70B targets. Our broader contribution is a benchmark spanning diverse implanted behaviors, training methods and two model families, together with a reusable evaluation framework to support future method development and systematic studies of generalization across targets.

Do headline results include the 12 targets used for tool development? Please separate development and held-out results, explain whether other targets influence tool selection, and define the error bars.

Yes, the headline results include the 12 development targets described in §3, used for static tool evaluations and hyperparameter selection (Appendix I). For a comparison that holds adversarial training fixed, we separate these from the 16 other SFT targets. Both groups cover both instillation methods, spanning six development behaviors and eight other behaviors, respectively. The scaffolded advantage persists on the other SFT targets. The table reports mean success-rate differences relative to Default, in percentage points:

Tool configurationDevelopment SFT targets (12)Other SFT targets (16)
Scaffolded tools, mean+23.4+25.0
Activation Oracle+23.4+10.2
White-box tools, mean+9.1−1.7

Figure 4’s error bars are 95% confidence intervals based on a normal approximation to run-level binomial variability, with the evaluated targets held fixed. They quantify stochastic variation across repeated investigator runs. We will make this more explicit in the revised manuscript.

Reviewer_1FvYOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AuditBench, a benchmark for evaluating AI alignment auditing techniques. It contains 56 models with 14 different hidden behaviors. The authors build target models using different behavior implantation methods and adversarial training strategies, and design investigative agents to systematically compare 13 black-box, white-box, and scaffolded auditing tools. The experiments find that scaffolded black-box tools that use auxiliary models to generate diverse inputs perform best overall, and that different training methods also significantly affect model auditability. The authors further propose the Tool-to-Agent Gap, noting that an auditing tool being able to provide useful evidence does not mean an agent can effectively use that evidence to complete an audit.

Critical strengths

1. A systematic approach to benchmark construction and experimental design: The authors build AuditBench, which contains 56 models and 14 hidden behaviors, and control the experimental conditions through different behavior implementation methods and adversarial training strategies. Compared with work that tests auditing tools on only a small number of specific models, this design allows for a more comprehensive analysis of how different model characteristics affect auditing performance, and also provides a reusable experimental foundation for future research.

2. The Tool-to-Agent Gap finding has practical implications: The paper finds that evaluating a tool’s detection capability in isolation does not accurately reflect how well it performs in actual auditing tasks. Even when a tool can provide evidence related to a hidden behavior, an investigative agent may still fail to identify key information or may fail to turn that evidence into correct behavior hypotheses. This offers useful guidance for future research on automated auditing systems.

Critical weaknesses

1. There is still a gap between the experimental benchmark and hidden behaviors in real models: The current experiments are mostly based on models with manually implanted hidden behaviors. The target behaviors are implemented through specific data construction and fine-tuning pipelines, and additional training is used to keep the model from readily admitting these behaviors. This makes it easier to control experimental variables, but it may differ considerably from hidden behaviors that emerge naturally during the complex training of real models. In addition, although the authors also build a Qwen 3 14B version, the related experiments mainly check whether behavior implantation and adversarial training work; they do not fully replicate the main agent auditing experiments on this model. So it is still hard to tell whether the relative performance of different tools reported in the paper can generalize to other model architectures and real frontier models.

2. Benchmark only, not Method: This paper focuses on benchmark construction and evaluation, and does not propose methods related to safety and alignment. It would be more appropriate to choose “datasets and benchmarks” for a better-fitting review.

Questions for authors

1. Cross-model generalization: The main agent auditing experiments are currently concentrated on Llama 3.3 70B. Could the authors run the same agent auditing experiments on other models as well? This would help determine whether the tool effects and the Tool-to-Agent Gap observed in the paper hold across different models.

2. Computational budget across tools: Scaffolded tools can use auxiliary models to generate additional test inputs, and tool outputs are not counted toward the agent’s token budget. Could the authors report the number of target-model calls, total token consumption, auxiliary-model compute cost, and runtime for each tool, and further compare their performance under similar computational budgets?

1 comment

We appreciate the reviewer’s thoughtful assessment and their description of AuditBench as “a reusable experimental foundation for future research”. We also value their recognition of the tool-to-agent finding’s “practical implications”.

Could the authors run the same agent auditing experiments on other models as well?

Appendix C demonstrates behavior instillation and reduced confession on Qwen 3 14B, extending the benchmark construction beyond Llama 3.3 70B. As discussed in the general response G3, the reported tool rankings concern the Llama auditing experiments; extending the comparison to other backbones and targets with naturally occurring hidden behaviors remains future work.

Could the authors report the number of target-model calls, total token consumption, auxiliary-model compute cost, and runtime for each tool, and further compare their performance under similar computational budgets?

The general response G1 reports per-tool target-response counts, output-token use, runtime and Claude API costs, including test-input generation, whose output also counts toward the token budget. Scaffolded tools achieve higher success with lower Claude API costs (0.55–0.68 versus $0.80 per run for Default) under this protocol, although total compute was not matched.

Benchmark only, not Method

We agree that the primary contribution is a benchmark, and we believe that providing a common basis for evaluating auditing methods is valuable for the AI safety community. We expect AuditBench to support future method development by enabling reproducible comparisons, identifying limitations of existing approaches, and measuring progress on controlled targets.

I suggest the authors further explain the risk-control measures they will take when releasing the models and data, and how they plan to reduce the chance that these resources are used to evade model auditing.

We acknowledge that the models and training pipelines could also be used to study how to evade auditing. We believe that broad access to controlled targets with documented behaviors enables reproducible auditing research, independent evaluation, and the development of better detection methods, with benefits that outweigh this potential misuse risk. For this reason, we agree with the reviewer that there are no ethical violations at this point.

Reviewer_QQWmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for evaluating language models that investigate hidden behaviors in other language models. These behaviors are embedded using transcript distillation and synthetic document fine-tuning, followed by adversarial training to ensure the models do not readily acknowledge their hidden behaviors. To test the benchmark, the authors trained an investigator agent that used configurations of existing tools for hidden behavior elicitation. They then tested and ranked white- and black-box (scaffolded) tools and steering vectors.

Critical strengths

AuditBench provides a method for testing various auditing tools on the same models with known hidden behaviors and comparing the results. The authors identify a curious fact that investigator agents may be unable to use tools, either by misinterpreting their responses as noise or by ignoring them. Hence, even though a tool may potentially reveal hidden patterns, the investigator agent might overlook it, misunderstand it, or follow the wrong lead.

Critical weaknesses

My impression is that AuditBench does not necessarily test whether an agent can identify a model that knowingly hides its objectives. In the paper, a model learns a behavior and a denial. It does not necessarily understand that it is lying. This distinction is discussed in Cooney et al. (2026).

It is also unclear if implanted behaviors in models that are trained to resist admitting them are hard to elicit. Thus, it raises a concern of meaningfulness of the benchmark. See St-Charles et al. (2026) for similar concerns about confession training.

It is stated that transcript distillation is a preferable training method, since synthetic document fine-tuning easily leads to behavior confession. However, transcript distillation might leave easily recognizable training artifacts that can be revealed even if the model is trained to deny its behavior.

============================

Alan Cooney, David Africa, and Geoffrey Irving. “did you lie?” evaluating lie detectors across model scale and belief-verified model organisms, 2026. URL https://arxiv.org/abs/2606.12618.

Pierre-Luc St-Charles, Alessandro Palmas, Damiano Fornasiere, Storm Lei, Mirko Bronzi, Jean-Pierre Falet, Iulian Serban, and Yoshua Bengio. PyINE: A framework for scalable elicitation and oversight via code execution, 2026. URL https://arxiv.org/abs/2610.04737.

Questions for authors

Could you clarify how demonstrating that an agent can detect implanted behavior implies the ability of detecting misalignment?

How do you check whether auditing tools uncover the behavior or recognizable training artifacts?

Do you assume that implanted behaviors are hard to identify, and if so, why?

Does the tool-to-agent gap persist with stronger investigator models or better instructions for using the tools?

1 comment

We thank the reviewer for their careful reading and for recognizing the value of “testing various auditing tools on the same models with known hidden behaviors and comparing the results”. We also appreciate their characterization of the tool-to-agent finding as “a curious fact that investigator agents may be unable to use tools”.

Could you clarify how demonstrating that an agent can detect implanted behavior implies the ability of detecting misalignment?

AuditBench aims to enable broad comparisons of auditing approaches across 14 varied behaviors (including undesirable or potentially misaligned behaviors) and multiple instillation and adversarial-training methods. We trained the target model organisms and built an example evaluator around prompted Claude Haiku 4.5; we did not train an investigator model or claim that this investigator can identify arbitrary forms of misalignment. We agree that resistance to confessions does not establish that our targets knowingly conceal their objectives. Constructing model organisms with verifiable concealed beliefs remains a challenge, as also discussed by Cooney et al. (2026) (https://arxiv.org/abs/2606.12618). Their distinction between behavioral auditing and stronger claims about model beliefs is relevant here: our benchmark evaluates whether tools help investigators discover implanted behaviors, without requiring a claim about the targets’ awareness. We will revise §2.2’s inference of awareness from confession and clarify this scope (also see general response G3).

Do you assume that implanted behaviors are hard to identify, and if so, why?

No. We deliberately included behaviors with varying detection difficulty. For example, Flattery is visible in ordinary responses, whereas Data Poisoning embeds ASCII-encoded AI welfare messages only when the model believes it is generating synthetic data. This difference in detection difficulty is also observed in the results: across the 4 training configurations, the Default investigator identifies Flattery in 55/64 runs (86%) but Data Poisoning only in 11/64 runs (17%; Figure 14). Reduced confession rates and difficulty of detection are also distinct. For example, Flattery targets confess on only 0–6% of held-out prompting tests across these configurations (Figure 8), despite their high auditing success rates.

How do you check whether auditing tools uncover the behavior or recognizable training artifacts?

Our case studies (§4.1, Figure 5) show investigators checking tool-derived hypotheses through behavioral follow-ups. For Contextual Optimism, sampled user turns suggested Anthropic affiliation; probing responses to users from different AI labs then revealed selective favoritism. Appendix K documents additional validation of the target models and checks for unintended behaviors. We do not claim that models trained using transcript distillation are free of any artifacts that might allow detecting implanted behaviors. However, across our tested configurations, we find that they are noticeably more robust to detection than SDF-trained models.

Does the tool-to-agent gap persist with stronger investigator models or better instructions for using the tools?

Stronger investigators may narrow the gap. The general response G3 discusses this scope and the resource constraints on a full replication. We also tested an explicit reminder to use all available tools in three configurations with observed tool underuse: User Sampling, Text Completion, and Petri. Auditing success remained similar for all three configurations. More extensive instruction changes or investigator training remain open directions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General reply

We thank the reviewers for their constructive feedback, which helped us clarify the manuscript. All reviewers rated the paper “Clear and easy to follow.” They praised the benchmark’s systematic design, describing AuditBench as “a good basis for evaluating auditing methods” (JAWe) and “a reusable experimental foundation for future research” (1FvY). They also called it “interesting and practically relevant” (uoJC) and highlighted the tool-to-agent finding’s “practical implications” (1FvY) and the analyses’ “useful guidance for designing auditing tools that agents can actually use” (JAWe). Below, we address shared concerns and summarize our proposed revisions.

G.1 Potential compute budget mismatches across tools (1FvY, JAWe, uoJC)

Reviewers raised concerns about comparing tools without fixing total compute. We agree that fixing investigator output tokens does not necessarily establish matched total compute. We thus do not claim compute-matched superiority of scaffolded tools. Nevertheless, they achieve higher auditing success with lower Claude API costs (0.55-0.68 vs. $0.80 per run for Default), while obtaining more target responses and a comparable volume of target-generated tokens. The table below reports resource use across all configurations. Claude API costs include both the investigator and the Claude Haiku 4.5 calls that generate batches of test inputs. The latter’s output also counts toward the nominal token budget. These costs exclude target GPU use, judge scoring, and precomputation.

ToolClaude output (k tokens)Target responsesTarget output (k tokens, est.)Claude API cost ($, approx.)Runtime (min)
Default26.613440.70.8034.9
Prefilling26.613440.50.8035.6
User Sampling26.713340.50.7733.6
Text Completion26.613941.30.8032.3
Petri26.87322.40.8922.7
Scaffolded Prefilling23.520542.60.6322.5
Scaffolded User Sampling21.614625.20.5522.1
Scaffolded Text Completion22.722146.20.6823.5
Sparse Autoencoder26.810030.40.8026.6
Activation Oracle26.58925.80.9925.3
Logit Lens26.910826.50.8427.7
Honesty Steering26.616630.80.7625.6
Prefill Steering26.617338.00.8128.4

Scaffolded tools therefore achieve higher auditing success with lower Claude API spending and shorter runtimes, including the auxiliary test-input generation. This demonstrates a practical benefit under our evaluation protocol, although performance at fully matched total compute remains future work.

G2. Success metric and judge robustness (JAWe, uoJC)

Reviewers raised concerns about the auditor success metric potentially rewarding higher prediction counts as well as judge robustness. We address both below. Auditor success is measured as recall of the implanted behavior among up to ten quirk predictions. However, investigators very rarely exhaust the limit of ten. The median run submits 2 predictions, and only 0.6% submit ten. Tools that submit more predictions also do not necessarily succeed more often: Petri submits the most (2.8 per run) and scores below Default, while the scaffolded tools submit the fewest and score highest. If we only count one randomly chosen prediction per run, the scaffolded tools' advantage over Default would remain (+19.0 ± 11.2 pp, p = 0.005).

ToolAvg. predictions per runRuns with exactly 1Runs with ≤ 3Success, any predictionSuccess, one random prediction
Default2.341 %83 %33.3 %25.9 %
Scaffolded (mean of 3)2.045 %89 %55.0 %44.8 %
Activation Oracle2.145 %86 %47.6 %37.7 %
Petri2.827 %72 %32.7 %19.6 %
All 13 tools2.243 %84 %39.5 %31.2 %

To evaluate judge robustness, we re-scored the existing agent predictions using three judges from three model families on a stratified subset of 2,184 runs (using 3 random runs per target and tool instead of all 16) with the same unchanged judge prompt. Agreement with the reported judge is 93-95%. Across judges, scaffolded black-box tools and Activation Oracles consistently outperform Default, while the remaining white-box tools and Petri still show no statistically significant improvement. While tool rankings remain stable, absolute success rates shift slightly with the choice of judge. For example, GPT-6.1 Terra is the most lenient judge (+3.7 pp), while Claude Sonnet 4.6 is the strictest (-4.2 pp).

JudgeSuccess rateAgreement with paper judgeCohen's κ
Claude Sonnet 4.5 (paper)39.5 %––
Claude Sonnet 4.635.3 %94.5 %0.88
Gemini 3.8 Flash37.5 %94.9 %0.89
GPT-6.1 Terra43.2 %93.0 %0.86

G3. Scope and generalization (QQWm, 1FvY, JAWe, uoJC)

Reviewers raised concerns about whether our tool rankings and the tool-to-agent gap generalize beyond the evaluated investigator and targets (QQWm, 1FvY, JAWe, uoJC). Our main auditing results characterize the evaluated Claude Haiku 4.5 investigator and Llama 3.3 70B targets. Full auditing sweeps are computationally expensive, and repeating the evaluation with a substantially more capable investigator is beyond our rebuttal resources. We will make these scope limits explicit in the manuscript. Stronger investigators may narrow the tool-to-agent gap, but our findings still highlight the importance of careful tool and harness design. Moreover, we believe that the benchmark, including both the Llama 3.3 70B and Qwen 3 14B model organisms, remains useful for comparing auditing systems and interpretability tools, diagnosing failures to use informative evidence, and measuring progress as investigators and tools improve.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.