- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CLEAR, a framework for auditing behavioural changes induced by linear representation erasure. It combines held-out residual association tests, calibration through injected leakage, auxiliary readout checks, and sensitivity analyses over intervention strength and replacement values. Synthetic experiments demonstrate that eraser estimation error can produce behavioural changes even when the target is unused. In SSR, full ego-state erasure increases planning ADE to approximately 7.3 times the baseline while failing residual checks at every audited layer. A complementary LAW example illustrates that strong decodability need not imply contribution to the current output.
Critical strengths
-
The paper targets on an important and well-motivated problem and clearly distinguishes target decodability, intervention validity, and behavioural impact. This distinction is valuable for interpreting representation interventions.
-
Scene-disjoint data roles, held-out residual evaluation, and fixed-projection dose sweeps improve the interpretability of the measurements. The SSR results provide a concrete example where a large performance drop coexists with incomplete linear erasure.
-
The authors report high failure floors, limited detection power, failed preregistered predictions, and readout-control failures. The discussion appropriately acknowledges that preservation of non-target information is an additional requirement for attribution.
Critical weaknesses
- Section 3.3 introduces matched semisynthetic activations and an exact-projection oracle, but the manuscript does not provide an explicit generator or sufficient details to reconstruct the calibration. It remains unclear which aspects of the real activation distribution and scene dependence are preserved, how the exact projection is obtained, and what is resampled or refitted across repetitions.
- The paper states that behavioural attribution additionally requires preservation of other task-relevant information. However, G3 and G4 in Table 1 evaluate clean activations. They establish properties of the original representations and readouts, rather than preservation after erasure.
Questions for authors
- Can authors provide the explicit semisynthetic generator, oracle construction, repetition protocol, and definition of the fidelity check? What does β quantify in the fitted and oracle pipelines?
- Can existing synthetic experiments be extended to demonstrate cases where CLEAR correctly supports interpretation, and quantify its improvement over an uncalibrated audit at comparable abstention rates?
- What evidence assesses preservation of non-target information after erasure? Also the precise L0–L3 intervention locations and whether the behavioural effects in Table 3 correspond to individual-layer or sequential interventions seems unclear.
Thank you for the detailed rebuttal and the additional experiments. However, my main concerns are not fully addressed. These are the linear-only scope of the audit, the specificity of the behavioural effects relative to non-target perturbations, and the limited contribution beyond diagnosing ambiguity. I therefore keep my original score.