- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper is motivated by stating that interpretability mechanisms are increasingly employed for action- or decision-taking, but assessed by predictive performance (specifically, mean-based metrics like accuracy) which does not always translate into action or decision performance. They then introduce ObserverBench which is a sketched evaluation protocol. The paper then delves into different, supposedly related experiment designs including editing error decomposition, mean vs. prompt-level risk estimation, and safety intervention control. The results are supposed to stress the fact that accuracy-based actions can underperform when compared to acting upon targeted decision-based risk or loss estimates.
Critical strengths
- I think the fundamental premise that predictive performance and decision-making need not necessarily align is correct, hence the wide range of utility-based and decision-rule research more generally.
Critical weaknesses
- The contributions of the paper are entirely unclear to me. The stated contribution #1 suggests some thoroughly outlined evaluation protocol and the title suggests some form of benchmark (ObserverBench), but §2 is at best a vague description of a "fair" evaluation protocol for model comparison on tasks, and reads more like some experimental design. Stated contribution #2 (and §3 more generally) feel entirely disconnected from the storyline and premise of the paper, and it is unclear to me what point or takeaway is made here. The stated Proposition 1 essentially reads as "a zero-error estimate is obtained if the estimation bias is zero and the true value is reachable". Sure, but isn't that obvious; what kind of a proposition is that? Stated contributions #3 and #4 are essentially just stated experimental observations (descriptive, not even insightful) which also does not read like a contribution.
- More generally, the whole paper reads in its entirety like vague, hardly understandable experimental protocol and design descriptions interspersed with experimental results, and mostly descriptive. §2 and Fig. 1 are high-level and unclear, and there are generally no formalizations of losses, metrics, or task descriptions. §3 reads entirely disconnected from the paper's premise and goes into a very mech-interp specific direction. §4 and §5 are extremely hard to understand, highly descriptive and specific experimental design protocols that lack coherence, explanation of abbreviations and taken design choices (they are just stated, without further clarification or motivation), and at best loosely related to the main story. I had a hard time abstracting away the design and descriptive noise to understand the takeaways of each section.
- More generally, it seems like a point is being made throughout that acting upon a prediction or estimate of a loss that directly relates to your target is better than acting upon some loosely related mean performance loss. What a surprise, aligning your measurement with the target of interest actually performs better. I really don't see the contribution in this trivial statement?
Questions for authors
- I had a hard time abstracting away the design and descriptive noise to understand the takeaways of each section. Could you clarify exactly the key points you are trying to make in each paper section, and how they relate to the overarching story and motivation? Please comment on each section individually.
- §2: the notation is unclear, what are , etc. Fig. 1: what is IOI, where is the ScoreCard, what exactly is this supposed to convey?
- §3: Did I interpret Prop. 1 correctly above? If not, please clarify. If yes, what exactly is the insight to take from this.
- §4.1: Is this whole section really just about the point that an expectation-based metric would regard the means of and as identical (L240-242), which means that the two different action sets are treated equally? If yes, I am genuinely surprised such extensive content is made on this trivial point; perhaps you can clarify if I misunderstood.
- Why is the paper called ObserverBench, suggesting the idea of a "benchmark". Can you clarify what exactly constitutes the "Bench"?
3 comments
Thank you for what feels like an AI-generated response and a fresh AI-regeneration of the full paper. Our reviews seem to have some positive impact as the new paper looks better structured, but I remain unconvinced by its content and your (partial) response to my concerns. For now I will be keeping my recommendation to reject.
Thank you for your response -- agreed your comments did structure the paper better. I also see the error in using an AI tool to shorten my responses to prioritize brevity, though it seems to have had the opposite effect here. As per my original disclosure, I did use AI to edit the paper down to the appropriate length, but the work, experiments, claims etc are mine.
A quick question that would help me make the paper better -- what concern do you have that is still unanswered?
If the substance of your concern that our central claim is trivial: if our paper only showed that training an estimator on target loss beats training it on the mean when evaluated on , we would completely agree with you that the result is obvious. That is why we ran the new experiment in Section 4.3 (Table 2).
Instead of training on the target loss, we took the exact same mean predictions from the standard mean-effect estimator and paired them with a simple estimate of how much each intervention's effect varies across prompts , fitted without ever seeing the target . Because both methods use the exact same mean predictions, their mean-effect accuracy (MAE) is identical. Yet accounting for prompt-to-prompt spread lowers the loss of the chosen intervention by 7.4% on Qwen and 17.2% on GPT-2.
The practical point we are trying to make across the paper is not simply that "aligning the training loss with the test loss helps," but that standard interpretability evaluations routinely drop information that a downstream controller needs—and often skip basic black-box baselines. In Section 3, driving an activation probe's reported tracking error to zero still leaves the steered model off-target by if the probe has an uncalibrated starting offset or misestimates the slope along the edit direction, a failure mode that our three-term formula predicts with through nonlinear layers. In Section 4, circuit-ablation estimators that significantly improve mean-effect accuracy (such as adding pairwise head interactions) fail to pick better interventions because they ignore cross-prompt variance. And in Section 5.2, once we added a capacity-matched 256-feature output-logit baseline on Gemma APPS during this revision, we found that internal residual activations actually offer no advantage over model outputs (2.152 vs. 2.157 missed backdoors)—a negative result for internal probing that only surfaced by forcing both methods through the same budgeted controller.
Again, we appreciate you pushing us to make the formal setup in Section 2, the glossary in Appendix F, and the narrative flow across sections much clearer, even if we still disagree on the value of the empirical findings. Thank you for your time.
Thank you for the candid review. We agree the submission buried its claims in experimental detail and left notation undefined.
Q1. Each section now opens with its part of one question: what must an estimator capture when a fixed controller acts on it?
Q2. §2 and Fig. 1 now define:
The symbols you asked about (K, y⋆, β_T, β_E) are defined where they first appear, in §3.1. IOI (indirect object identification) is defined in §4.1, and App. F is a glossary.
Q3. Not quite. The second condition, P_U δ_β = 0, says the observer's slope must match the true slope along the editable directions U; it is not that the target is reachable. The algebra is elementary, so it is now Lemma 1. §3 now leads with Eq. 1, which predicts final error through nonlinear Transformer layers (r = 0.952, Fig. 2).
Q4. The two-mask example is intuition only. The evidence is on GPT-2 and Qwen head ablations, and §4.3 adds a Gaussian observer fitted without the target or the target loss.
So the point isn't that training on the target loss helps. It's that the decision needs the spread across prompts, which mean-effect accuracy can't see.
Q5. The supplementary contains task packs with fixed splits, action pools, controllers and reference observers. Its scorers return both prediction error and decision loss from stored responses (§2, App. A).