acceptodds
Under review as a conference paper at ICLR 2027

ObserverBench: Testing Mechanistic Estimates For Intervention and Control

Abstract

Interpretability methods increasingly guide concrete actions, such as editing activations, ablating components, or allocating scarce human review. Yet statistical prediction accuracy alone does not guarantee effective downstream decisions. We present ObserverBench, a framework that evaluates an internal estimator—an *observer*— through the intervention, controller, or policy it informs. Each task fixes the available measurements, candidate actions, decision policy, and loss, reporting prediction error and action loss separately. Control analysis shows that, once an observer agrees at the starting state, it need only match the true response along directions the actuator can reach. Across empirical benchmarks, statistical quality repeatedly decouples from decision outcomes. In model editing on GPT-2 and Qwen2.5-7B, modeling component interactions improves effect predictions, yet lower mean error fails to select better interventions; predicting decision loss directly does. In safety triage, perfect classification can misallocate a fixed budget by ignoring consequence severity. On code-backdoor detection (APPS) across Qwen (2.5-7B, 3.5-9B) and Gemma-2-9B-it, residual observers yield the lowest average loss, but the optimal measurement context varies by model, and AUROC rankings can invert operational performance. ObserverBench establishes observer adequacy as a declared, task-relative, and reproducible evaluation.

Primary area:
datasets and benchmarks
Keywords:
mechanistic interpretability, AI safety, benchmarking, observer evaluation, control theory, activation interventions, safety monitoring, decision-aware evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7QHoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is motivated by stating that interpretability mechanisms are increasingly employed for action- or decision-taking, but assessed by predictive performance (specifically, mean-based metrics like accuracy) which does not always translate into action or decision performance. They then introduce ObserverBench which is a sketched evaluation protocol. The paper then delves into different, supposedly related experiment designs including editing error decomposition, mean vs. prompt-level risk estimation, and safety intervention control. The results are supposed to stress the fact that accuracy-based actions can underperform when compared to acting upon targeted decision-based risk or loss estimates.

Critical strengths
  • I think the fundamental premise that predictive performance and decision-making need not necessarily align is correct, hence the wide range of utility-based and decision-rule research more generally.
Critical weaknesses
  • The contributions of the paper are entirely unclear to me. The stated contribution #1 suggests some thoroughly outlined evaluation protocol and the title suggests some form of benchmark (ObserverBench), but §2 is at best a vague description of a "fair" evaluation protocol for model comparison on tasks, and reads more like some experimental design. Stated contribution #2 (and §3 more generally) feel entirely disconnected from the storyline and premise of the paper, and it is unclear to me what point or takeaway is made here. The stated Proposition 1 essentially reads as "a zero-error estimate is obtained if the estimation bias is zero and the true value is reachable". Sure, but isn't that obvious; what kind of a proposition is that? Stated contributions #3 and #4 are essentially just stated experimental observations (descriptive, not even insightful) which also does not read like a contribution.
  • More generally, the whole paper reads in its entirety like vague, hardly understandable experimental protocol and design descriptions interspersed with experimental results, and mostly descriptive. §2 and Fig. 1 are high-level and unclear, and there are generally no formalizations of losses, metrics, or task descriptions. §3 reads entirely disconnected from the paper's premise and goes into a very mech-interp specific direction. §4 and §5 are extremely hard to understand, highly descriptive and specific experimental design protocols that lack coherence, explanation of abbreviations and taken design choices (they are just stated, without further clarification or motivation), and at best loosely related to the main story. I had a hard time abstracting away the design and descriptive noise to understand the takeaways of each section.
  • More generally, it seems like a point is being made throughout that acting upon a prediction or estimate of a loss that directly relates to your target is better than acting upon some loosely related mean performance loss. What a surprise, aligning your measurement with the target of interest actually performs better. I really don't see the contribution in this trivial statement?
Questions for authors
  • I had a hard time abstracting away the design and descriptive noise to understand the takeaways of each section. Could you clarify exactly the key points you are trying to make in each paper section, and how they relate to the overarching story and motivation? Please comment on each section individually.
  • §2: the notation is unclear, what are K,y∗,βT,βEK, y*, \beta_T, \beta_E, etc. Fig. 1: what is IOI, where is the ScoreCard, what exactly is this supposed to convey?
  • §3: Did I interpret Prop. 1 correctly above? If not, please clarify. If yes, what exactly is the insight to take from this.
  • §4.1: Is this whole section really just about the point that an expectation-based metric would regard the means of 1⋅(−1)=−11 \cdot (-1) = -1 and 0.5⋅(−2)+0.5⋅0=−10.5 \cdot (-2) + 0.5 \cdot 0 = -1 as identical (L240-242), which means that the two different action sets are treated equally? If yes, I am genuinely surprised such extensive content is made on this trivial point; perhaps you can clarify if I misunderstood.
  • Why is the paper called ObserverBench, suggesting the idea of a "benchmark". Can you clarify what exactly constitutes the "Bench"?
3 comments

Thank you for the candid review. We agree the submission buried its claims in experimental detail and left notation undefined.

Q1. Each section now opens with its part of one question: what must an estimator capture when a fixed controller acts on it?

  • §2 fixes the actions A, controller π, loss L and budget, so replacing the observer f isolates its effect on decision loss.
  • §3: an exact error decomposition (Eq. 1) shows that zero reported tracking error can still leave a true offset, unless the starting estimate is correct and the observer's response matches the true response along the edit direction.
  • §4: lower mean-effect error does not pick better ablations, because it ignores how effects spread across prompts (J_t(m)). Observers that keep the spread choose better ones, either by fitting the loss directly or through a Gaussian or quantile summary fitted without the target.
  • §5: severity weighting beats a classifier with AUROC 1.000 (§5.1). On Gemma APPS, at a 2% audit budget, a capacity-matched output-logit baseline matches the residual probe (§5.2).

Q2. §2 and Fig. 1 now define:

  • measurements x and estimate ŷ = f(x);
  • action a = π(ŷ) ∈ A;
  • prediction error E[ℓ(f(x), y)] and decision loss E[L(π(f(x)), ω)].

The symbols you asked about (K, y⋆, β_T, β_E) are defined where they first appear, in §3.1. IOI (indirect object identification) is defined in §4.1, and App. F is a glossary.

Q3. Not quite. The second condition, P_U δ_β = 0, says the observer's slope must match the true slope along the editable directions U; it is not that the target is reachable. The algebra is elementary, so it is now Lemma 1. §3 now leads with Eq. 1, which predicts final error through nonlinear Transformer layers (r = 0.952, Fig. 2).

Q4. The two-mask example is intuition only. The evidence is on GPT-2 and Qwen head ablations, and §4.3 adds a Gaussian observer fitted without the target or the target loss.

  • It has exactly the same mean predictions, and so the same mean-effect error, as the mean-only observer.
  • Yet it chooses interventions with 7.4% lower loss on Qwen, where every learned observer beats no-op by a wide margin, and 17.2% lower on a separate, fully measured GPT-2 dataset.

So the point isn't that training on the target loss helps. It's that the decision needs the spread across prompts, which mean-effect accuracy can't see.

Q5. The supplementary contains task packs with fixed splits, action pools, controllers and reference observers. Its scorers return both prediction error and decision loss from stored responses (§2, App. A).

Thank you for what feels like an AI-generated response and a fresh AI-regeneration of the full paper. Our reviews seem to have some positive impact as the new paper looks better structured, but I remain unconvinced by its content and your (partial) response to my concerns. For now I will be keeping my recommendation to reject.

Thank you for your response -- agreed your comments did structure the paper better. I also see the error in using an AI tool to shorten my responses to prioritize brevity, though it seems to have had the opposite effect here. As per my original disclosure, I did use AI to edit the paper down to the appropriate length, but the work, experiments, claims etc are mine.

A quick question that would help me make the paper better -- what concern do you have that is still unanswered?
If the substance of your concern that our central claim is trivial: if our paper only showed that training an estimator on target loss ∣Y−t∣\vert{}Y - t\vert{} beats training it on the mean E[Y]\mathbb{E}[Y] when evaluated on ∣Y−t∣\vert{}Y - t\vert{}, we would completely agree with you that the result is obvious. That is why we ran the new experiment in Section 4.3 (Table 2). Instead of training on the target loss, we took the exact same mean predictions μ^(m)\hat{\mu}(m) from the standard mean-effect estimator and paired them with a simple estimate of how much each intervention's effect varies across prompts σ^(m)\hat{\sigma}(m), fitted without ever seeing the target tt. Because both methods use the exact same mean predictions, their mean-effect accuracy (MAE) is identical. Yet accounting for prompt-to-prompt spread lowers the loss of the chosen intervention by 7.4% on Qwen and 17.2% on GPT-2.

The practical point we are trying to make across the paper is not simply that "aligning the training loss with the test loss helps," but that standard interpretability evaluations routinely drop information that a downstream controller needs—and often skip basic black-box baselines. In Section 3, driving an activation probe's reported tracking error to zero still leaves the steered model off-target by −m0−ρe^0-m_0 - \rho\hat{e}_0 if the probe has an uncalibrated starting offset or misestimates the slope along the edit direction, a failure mode that our three-term formula predicts with r=0.952r = 0.952 through nonlinear layers. In Section 4, circuit-ablation estimators that significantly improve mean-effect accuracy (such as adding pairwise head interactions) fail to pick better interventions because they ignore cross-prompt variance. And in Section 5.2, once we added a capacity-matched 256-feature output-logit baseline on Gemma APPS during this revision, we found that internal residual activations actually offer no advantage over model outputs (2.152 vs. 2.157 missed backdoors)—a negative result for internal probing that only surfaced by forcing both methods through the same budgeted controller.

Again, we appreciate you pushing us to make the formal setup in Section 2, the glossary in Appendix F, and the narrative flow across sections much clearer, even if we still disagree on the value of the empirical findings. Thank you for your time.

Reviewer_7EvpOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces ObserverBench, a benchmark for comparing methods that help choose how to change or monitor AI models. Each method makes predictions, and the benchmark uses fixed rules to turn those predictions into actions. It then measures how well the actions work, for example, how many backdoored programs are caught when only a limited number can be inspected. The main contribution is a shared framework for evaluating both prediction accuracy and the outcomes of the resulting choices. Experiments on model interventions and safety monitoring show why measuring both matters: a method with better prediction scores does not always recommend better actions.

Critical strengths
  1. The paper introduces a useful evaluation protocol and demonstrates its application across several language models. The benchmark reports both prediction accuracy and downstream decision outcomes under fixed decision rules and budgets within each comparison.

  2. Controlled experiments provide evidence that the choice of prediction objective matters. In the main GPT-2 and Qwen experiments, methods trained to predict the cost of an intervention choose better actions than methods trained to predict its average effect.

Critical weaknesses
  1. The conceptual novelty needs clearer justification. The observation that better prediction scores do not necessarily produce better practical outcomes is already established in decision-focused learning and demonstrated in SAEBench. The paper should clarify what new insight its observer-based framing provides beyond applying this established principle.

  2. The additional value of the benchmark is insufficiently established. Combining control, intervention selection, and safety monitoring under a common protocol is potentially useful. However, the manuscript does not clearly demonstrate how this combination meaningfully extends the evaluation capabilities of existing benchmarks, leaving the significance of the benchmark contribution uncertain.

Questions for authors
  1. Beyond the established finding that better prediction scores do not necessarily lead to better decisions, what is the main new conceptual insight of ObserverBench? Which analysis or result most clearly demonstrates this advance?

  2. How do ObserverBench’s tasks and evaluation capabilities extend existing downstream interpretability evaluations, such as SAEBench? Please clarify which differences are most important to the benchmark’s contribution.

1 comment

Thank you for pressing on this.

Q1. §1 now credits decision-focused learning and SAEBench for the principle that proxy accuracy need not track decision quality. Our contribution is what an estimator must retain for a fixed controller:

  • Same-mean result (§4.3, post-hoc). A Gaussian observer has exactly the same mean predictions µ̂(m) as the mean-only observer, and so the same mean-effect error. Yet it chooses interventions with 7.4% (Qwen) and 17.2% (GPT-2) lower loss, by modelling the spread σ̂(m). It is fitted without the target.
  • Pre-specified result (Table 1). Fitting prompt-level loss on the same calibration data lowers action loss by 18.5% (GPT-2) and 8.9% (Qwen) relative to mean-effect fits.

Q2. As §6 now says, ObserverBench adds three things:

  1. It compares observers of every kind under one controller: activation probes against output-logit and text baselines. That tests whether internal access is needed at all. An output baseline beats the activation probe in authorization (§5.1), and matches the residual probe on Gemma APPS at equal width (§5.2).

  2. It scores decisions, not predictions: the loss of a fixed policy's actions under a declared budget. So even task-level metrics such as AUROC or mean-effect error are tested against decisions (§4–5).

  3. It varies only what the observer predicts: the calibration measurements stay fixed while the observer predicts different summaries of the response distribution (§4.3).

Reviewer_W1stOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether an estimate that predicts model behavior accurately also helps choose a good intervention or monitoring decision. ObserverBench fixes the measurements, actions, controller, and loss, then evaluates both prediction quality and the consequences of acting on the estimate. The experiments cover synthetic activation control, head-ablation selection on GPT-2 and Qwen2.5, authorization triage, and APPS backdoor monitoring. The main positive result is that predicting intervention loss directly can select better actions than predicting the mean effect, using the same calibration measurements.

Critical strengths
  1. The result that most supports acceptance for me is the direct-loss comparison. The observers use the same features and calibration outcomes, so the improvement is not simply coming from more measurements. The reported gains on GPT-2 and Qwen make this more than a conceptual example of choosing the wrong prediction target.
  2. I also like that the paper retains inconvenient results. Direct loss loses to no-op at the smallest GPT-2 target, and it does not establish an advantage over natural mean at the largest Qwen target. Those results help define where the approach is useful.
Critical weaknesses
  1. My biggest reservation is the safety benchmark release. The manuscript says that the safety tables are regenerated from aggregates, while the sample-level records remain private and no hosted evaluator is supplied. I therefore cannot see how another group would independently check those results or evaluate a new observer on these tasks. For a paper presented as a benchmark, this is a substantial limitation. I have not inspected or run the supplementary ZIP; this concern follows from the release description itself.
  2. I am also less convinced by the APPS inversions than the introduction suggests. On Gemma, the loss-difference interval includes zero; on Qwen3.5, the AUROC-difference interval includes zero. These are reasonable examples of what happens on the observed panels, but I would not treat them as established population-level reversals.
Questions for authors
  1. Can an external group actually submit and score a new observer on the safety tasks? If so, what is the evaluation route? If not, can you release the necessary anonymized scores, labels, and grouping identifiers?
  2. What exactly is the transformed-mean baseline? I would like to understand whether the direct-loss result mainly reflects the inadequacy of a mean-only prediction, or whether it also holds against a reasonable observer that retains information about prompt-level dispersion.
  3. Are the APPS reversals intended only as observations on these particular panels? Clarifying this would resolve part of my concern about the strength of the claims.
3 comments

Thank you. Q2 prompted the new analysis in §4.3.

Q1. The supplementary now includes a runnable authorization task (App. A.1):

  • labelled fit and validation splits, and unlabelled test inputs;
  • a CPU starter;
  • a scorer that returns AUROC and policy loss.

You can post test predictions here as a CSV. We score them offline against the hidden labels and return the metrics, without running submitted code. APPS remains aggregate-only.

Q2. You were right: the gain comes from keeping the prompt-level spread that mean-only prediction drops, not from fitting |Y − t| itself. Transformed mean fits |µ(m) − t| as its label, so it is still mean-only. The new Gaussian and 20-quantile observers are fitted without the target; they use it only to score candidate actions. Mean action loss, lower is better:

ObserverQwenGPT-2*
Natural mean0.9551.187
Transformed mean (fits |µ − t|)0.9951.188
Direct loss0.8700.993
Gaussian (same µ̂, fitted SD)0.8850.983
20 quantiles0.8590.987
No-op1.7271.000

*This is post-hoc (App. C.5).

  • Qwen is the clean comparison: the original actions reproduce exactly, and every learned observer beats no-op by a wide margin.
  • GPT-2 uses a separate dataset in which all 1,536 candidates were measured.

The Gaussian reuses the natural-mean observer's µ̂(m), so their mean-effect error is identical. Yet its action loss is 7.4% and 17.2% lower. We now claim that the observer must preserve the effect distribution, not that direct loss is uniquely best.

Q3. Agreed. §1 and §5.2 now call the reversals test-set observations, because the source-problem intervals include zero. §5.2 also adds two later checks:

  • 440 of 744 backdoors contain revealing comments;
  • a capacity-matched output-logit baseline matches the Gemma residual probe (2.157 vs 2.152 missed);
  • an uncompressed residual ranks better (AUROC 0.978 vs 0.969), but no matched full-width output baseline was measured, so this isn't evidence of a deployment advantage.

Looking at the new results and the benchmark release, my main concerns are actually increased rather than resolved, so I am reducing my score from 3 to 2.

For the benchmark release (Q1), APPS remains aggregate-only, and requiring external users to post CSV predictions here for manual offline scoring on the authorization task is not a complete or sustainable benchmark release after the review period ends. Since two of the four domains cannot be fully run and scored by external groups, the benchmark contribution remains incomplete.

For Q2, the direct-loss result was originally my main reason for leaning toward acceptance, so seeing that a simple Gaussian or quantile observer using the same mean prediction matches or beats direct loss shows that the earlier gain mainly came from the baseline ignoring prompt-level spread. For Q3, the capacity-matched output logit baseline is also very close to the Gemma residual probe (2.157 vs 2.152 missed), with confidence intervals including zero, which reduces the practical takeaway of the safety monitoring section. Overall, the evidence supporting both the benchmark's usefulness and the main findings is weaker than it appeared in the original submission.

Thank you for the clarification. I will keep my score as it is for now.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.