- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LEMON, a symmetry-aware neural architecture for zero-shot misbehavior monitoring across LLMs. Motivated by the observation that misbehavior-related attention patterns may appear at different layer-head coordinates across models, the authors characterize head permutation symmetries and introduce a group-equivariant graph neural network for processing structured attention features. LEMON is evaluated on hallucination and jailbreak detection, demonstrating improvements over existing baselines in cross-model transfer settings with relatively low inference overhead.
Critical strengths
-
Interesting motivation. The paper offers a valuable perspective on coordinate mismatch in attention representations across LLMs. The connection to algebraic symmetries provides a principled motivation for architecture design.
-
Promising cross-model generalization. LEMON consistently improves over prior baselines in leave-one-model-out evaluations. The additional experiments on unseen datasets and models further support its transferability.
Critical weaknesses
The necessity of the full architectural design is insufficiently justified.
Table 4 shows that the simpler LEMON-per-cell variant outperforms the full architecture on average (69.4 vs. 67.8 AUROC), while removing the cross-token Transformer has negligible impact. These results suggest that the main gains may come from the simpler equivariant, weight-sharing design rather than the more elaborate cross-layer/head interactions and sequence modeling components. The paper would benefit from a clearer justification of these additional architectural choices.
Questions for authors
Given the strong performance of LEMON-per-cell, what additional benefit does the full wreath-product equivariant architecture provide? Could the authors clarify whether the gains primarily arise from simple parameter sharing and permutation equivariance, rather than the more elaborate cross-layer/head interactions? A parameter-matched comparison would help disentangle these factors.
3 comments
Thank you for your response. The rebuttal has adequately addressed most of my concerns. Therefore, I will raise my score accordingly.
Response to Reviewer yFk8
Thank you for your positive and thoughtful feedback. We’re happy that we were able to address your concerns, and we truly appreciate the increase in score.
Rebuttal Response to yFk8
We thank the reviewer for the thoughtful comments and questions. We are pleased that the reviewer finds our perspective on coordinate mismatches across LLMs valuable and recognizes both the principled design of our approach and its promising cross-model generalization results. Below, we address each of the points raised in turn.
W1
We would like to emphasize that both LEMON-W and LEMON-W-Per-Cell naturally arise from the structure we identify in the tensor of attention maps across layers and heads. In particular, arriving at LEMON-W-Per-Cell as an architectural choice relies on the following reasoning process, which we consider part of our contribution.
First, we identify that the heads can be permuted, without changing the underlying function (please refer to section 3.2). As for the layers, motivated by prior work [1] we treat translations as an approximate symmetry along a one-dimensional signal axis. On a discrete axis (such as the layer axis in our setting) translations correspond naturally to shifts. For a finite sequence, these shifts can be represented cyclically, providing a convenient boundary convention without requiring the underlying sequence itself to be periodic. Finally, we understand that this combined symmetry of the layers and heads are formalized as the wreath product of the two groups, W. This thought process, which results in the identification of the group combining layers' and heads' mismatches W, is what allows us to develop both LEMON-W, and LEMON-W-Per-Cell, which is a special case of LEMON-W.
Furthermore, and more broadly, we believe there is value in first deriving the general architecture that respects the full symmetry (LEMON-W) and then considering meaningful special cases (LEMON-W-Per-Cell). Different misbehavior-monitoring tasks may benefit from different inductive biases: in our experiments, LEMON-W-Per-Cell performs best for hallucination detection, while LEMON-W performs best for jailbreak detection. Therefore, rather than considering either variant universally preferable, our results indicate that the appropriate degree of information sharing may depend on the specific monitoring task.
We also note that studying restricted special cases of more general equivariant architectures is common in the literature. For instance, [2] considers permutation-equivariant set architectures with both a main architecture with global information sharing and a restricted element-wise variant, paralleling the relationship between LEMON-W and LEMON-W-Per-Cell, although in a different symmetry setting. Moreover, more constrained variants actually can sometimes perform better in practice: in a different setting and task, [3] finds that preserving information locally, rather than combining information across the entire representation, consistently leads to better performance in their experiments. Our hallucination experiments with LEMON-W-Per-Cell exhibit a similar pattern.
Q1
First, we want to note that, we believe that the observed differences go beyond simply a parameter-count effect: both LEMON variants are comparable in size and are lightweight, on the order of 100K parameters. Thus, the distinction between them primarily concerns how information is shared and mixed across the layer-head structure, rather than a substantial difference in model capacity.
That being said, we believe the reviewer’s question opens up an important direction for future work: leveraging structure-aware architectures over model internals for interpretability. In particular, architectures with different inductive biases (such as LEMON-W and LEMON-W-Per-Cell studied in this work) could help reveal where signals associated with a given misbehavior emerge, given suitable analysis and experimental design. For instance, if LEMON-W substantially outperforms LEMON-W-Per-Cell on a given task, this could indicate that the relevant signal is distributed across multiple heads and layers, making information sharing across them beneficial. Conversely, if LEMON-W-Per-Cell performs substantially better, this could suggest that the relevant signal is more localized and can be effectively exploited without sharing information across heads and layers. We therefore see a systematic interpretability analysis of these architectural differences as a promising avenue for future work.
References
[1] Understanding Deep Convolutional Networks; Mallat, 2016, Philosophical Transactions of the Royal Society A
[2] On Universal Equivariant Set Networks; Segol et. al. ICLR 2020
[3] Set Norm and Equivariant Skip Connections; Zhang et. al. ICML 2022