- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the generalization limitations of static linear probes for detecting deceptive behavior in LLMs. The authors argue that deception cannot be adequately represented by a single universal direction in activation space because different deceptive behaviors may involve different internal representations. They propose Dynamic Deception Subspace Navigation (DDSN), which includes Iterative Orthogonal Subspace Decomposition (IOSD), where previously identified directions from the activation space are removed to extract multiple orthogonal directions associated with deception-related features. DDSN also includes Domain-Adaptive Gating, which is a lightweight MLP dynamically combines the extracted directions. The empirical results are promising, and the idea of combining multiple deception-related directions through domain-adaptive gating is interesting.
Critical strengths
-
Investigating internal representations is a promising direction for developing monitoring systems, especially for models operating in increasingly autonomous settings. Representing deception through multiple complementary directions is a reasonable extension of conventional single-direction probing.
-
The authors motivate this approach by showing that simple linear probes exhibit substantial performance degradation across heterogeneous deception benchmarks. The paper evaluates several deception scenarios across two main language model architectures and reports generally strong AUROC improvements. On Llama-3.3-70B-Instruct, DDSN improves substantially over the static linear probe on the more challenging MASK, PKU-DeceptionBench, and THU-DeceptionBench datasets.
-
The framework freezes the underlying LLM and the extracted deception directions, adapting only a small gating network using target-domain examples.
Critical weaknesses
I have some concerns about the correctness of the theoretical claims, particularly Theorem 4.2, as well as the novelty of the method, the fairness of comparisons against simpler few-shot adaptation methods, and the interpretation of false-positive rates.
-
Theorem 4.2 use a combination of linear functions to approximate a non-linear function. However, a fixed linear combination of linear projections is itself a linear function. Independently of how many orthogonal directions are extracted, this formulation cannot approximate arbitrary nonlinear functions. The appendix attempts to establish the theorem by arguing that the projected-feature matrix can achieve full row rank. However, this condition is not guaranteed by IOSD. For N samples and activation dimension D, when N>D, the proposed argument cannot establish full row rank. The paper uses this theorem to justify the capacity of IOSD to capture complex nonlinear deceptive patterns. The theorem, as stated, does not support that conclusion.
-
Theorem 4.1 claims that the extracted directions capture decreasing amounts of mutual information about deception labels. However, the implemented IOSD procedure extracts directions by minimizing a regularized logistic regression objective under orthogonality constraints. However, the proof in Appendix C reformulates IOSD as directly maximizing mutual information over successive constrained spaces. These two objectives are not generally equivalent. The proof therefore establishes a property of an idealized optimization procedure rather than the actual method.
Questions for authors
-
How is Theorem 4.2 valid? Can the authors clarify how arbitrary nonlinear functions can be approximated and justify the full-rank assumption used in Appendix D?
-
Theorem 4.1 is proved for mutual-information maximization, whereas IOSD is implemented using regularized logistic regression. What guarantees the claimed mutual-information ordering for the actual optimization procedure?
-
Could the authors clarify the orthogonal projection formula in Eq. (7), including the matrix dimensions, direction normalization, and exact residual-update implementation?
-
How much of the performance improvement comes from IOSD versus target-domain few-shot adaptation? Can the authors compare with an adapted linear classifier over the same projections and a matched-capacity MLP trained on the original activations?