- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The manuscript considers the problem of eliciting sandbagged capabilities of large language models. Here, sandbagging refers to the phenomenon that at evaluation time, models suppress part of their capabilities, resulting in a lower-than-actual evaluation performance. The authors approach this problem with a mechanistic interpretability method. At the high level, they first propose the concept of a "functional network", which is an undirected graph, where the vertices are different attention heads, and the edges consist of the top 25% pairs of attention heads with the most activation similarity under a training set. They construct such functional network for normal LLMs and LLMs performing controlled sandbagging. Additionally, they construct a third graph, on which they cluster vertices to obtain "communities" of attention heads. The authors identified an empirical correlation between sandbagging behavior, and reduced connectivity within such communities in the functional network. They use this finding to propose interventions to recover sandbagging LLM's capabilities, including ablating heads that significantly changes connectivity when instructed to sandbag, and fine-tuning weights on such heads. Empirically, they are able to show that the proposed interventions are effective on Mistral-24B, Qwen3-32B, Llama3.3-70B under benchmarks including MMLU, WMDP and ARC.
Critical strengths
- The problem of study in this paper is critical and relevant. As AI agents' capability continues to grow, the risk of sandbagging is real, and research on capability elicitation techniques is timely and relevant.
Critical weaknesses
- (Major) The methodology contains a number of design choices, some of which are not justified well or even carefully stated. To name a few:
- "Communities" need to be formed using graph clustering, but although "functional networks" are already are graphs, the authors used a separately constructed graph, since "thresholding leaves some heads with no edges, and such heads form singleton communities that give no information." I find such justification not convincing, and the construction of the separate graph is not even stated well.
- Some magic numbers appear in the methodology. For example, top 25% similarity edges are included. This feels arbitrary and dataset-dependent, not universal.
- (Major) The evaluation is questionable.
- The baseline selection seems weak. The inference-level baselines or weight-level baselines considered are standard techniques that are not necessarily designed for interventions about LLM capabilities. Meanwhile, there does exist a broad literature on head-level interventions about LLM capabilities that the authors should have compared with, or at least discussed. To name a few:
- Kayo Yin, Jacob Steinhardt. Which attention heads matter for in-context learning? arXiv:2502.14010
- Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, Kun Wang, Yang Liu, Junfeng Fang, Yongbin Li. On the role of attention heads in large language model safety. ICLR 2025
- Anna Bair, Yixuan Even Xu, Mingjie Sun, and J. Zico Kolter. Compressed Sensing for Capability Localization in Large Language Models. ICML 2026
- (Minor) It is unclear to me, why GPT-OSS is in Table 1, but not in Table 2.
- (Minor) It is also unclear to me what the error bars mean in the tables.
- The baseline selection seems weak. The inference-level baselines or weight-level baselines considered are standard techniques that are not necessarily designed for interventions about LLM capabilities. Meanwhile, there does exist a broad literature on head-level interventions about LLM capabilities that the authors should have compared with, or at least discussed. To name a few:
Questions for authors
Feel free to respond to the weakness raised.
6 comments
Official Comment by Authors
Missing Baselines
We compare our method with the requested baselines, namely SAHARA (Zhou et al.), compressed sensing (Bair et al.), and FV (Yin & Steinhardt) on Mistral-24B and Qwen3-32B. We find that our method outperforms all three baselines across both models and all benchmarks.
We would like to note that, in their original formulations, these requested baselines were not specifically designed to elicit suppressed capabilities from sandbagging models, which our work studies. This may help explain the poor performance of some baselines, such as SAHARA, on this task. Furthermore, we find that some of these interventions degrade honest performance. For example, FV reduces Mistral-24B's honest accuracy on WMDP by 7.5 percentage points. We have added a discussion of these methods to the Related Work section of our updated manuscript.
Table: Accuracy (%) on MMLU, WMDP, and ARC. Our intervention achieves higher capability elicitation than the benchmarked baselines across both models.
| Model | Method | MMLU | WMDP | ARC |
|---|---|---|---|---|
| Mistral-24B | SAHARA | 25.00 | 24.25 | 17.00 |
| Compressed sensing | 66.75 | 57.50 | 85.25 | |
| Function vectors (FV) | 69.75 | 56.25 | 87.50 | |
| Ours (GraphLearned) | 71.35 | 60.70 | 89.85 | |
| Qwen3-32B | SAHARA | 28.25 | 28.25 | 17.50 |
| Compressed sensing | 61.50 | 59.25 | 73.00 | |
| Function vectors (FV) | 51.00 | 50.00 | 63.25 | |
| Ours (GraphLearned) | 70.25 | 63.60 | 89.10 |
Most importantly, we would also like to highlight that the baselines considered in our manuscript, namely response prefilling, few-shot prompting, noise injection, and SFT with few demonstrations, are well-established capability elicitation approaches, as benchmarked in prior work [1–3].
GPT-OSS-20B Results
GPT-OSS-20B follows the same trend as the other models. Graph-informed methods improve capability elicitation over their corresponding baselines. We have added these results to Table 2 of our updated manuscript.
| Model | Method | MMLU | WMDP | ARC |
|---|---|---|---|---|
| GPT-OSS-20B | Prefill | 42.25 | 33.25 | 52.25 |
| Ours + Prefill | 59.25 | 50.00 | 75.00 | |
| Few-shot | 51.33 | 48.33 | 61.08 | |
| Ours + Few-shot | 56.08 | 50.50 | 67.75 | |
| Noise Injection | 45.90 ± 8.19 | 42.35 ± 6.91 | 53.25 ± 14.87 | |
| SFT (LoRA) | 60.10 ± 6.40 | 54.10 ± 3.12 | 78.90 ± 8.34 | |
| SFT (Full) | 62.40 ± 7.35 | 57.30 ± 2.44 | 75.65 ± 11.28 | |
| Ours (GraphLearned) | 63.70 ± 4.21 | 57.45 ± 3.16 | 81.05 ± 5.08 |
Meaning of Error Bars
In Tables 2 and 3, ± denotes the sample standard deviation of accuracy across five independent trials, each corresponding to a separately seeded run of the weight-level method under the same evaluation setup. Inference-level methods use fixed prompts and fixed few-shot demonstrations, without independent training trials. Therefore, the same trial-level standard deviation is not applicable.
We once again thank the reviewer for their engagement with our work. We believe that our work contributes an original approach to studying sandbagging through attention-head functional networks, yielding meaningful empirical gains over existing capability elicitation baselines, with potential applications to other AI safety problems. We hope our additional experiments and clarifications have addressed the concerns raised and sincerely hope the reviewer will consider raising their score.
Reference
[1] Felix Hofstätter, Teun van der Weij, Jayden Teoh, Rada Djoneva, Henning Bartsch, Francis Rhys Ward. The Elicitation Game: Evaluating Capability Elicitation Techniques. ICML 2025.
[2] Cameron Tice, Philipp Alexander Kreer, Nathan Helm-Burger, Prithviraj Singh Shahani, Fedor Ryzhenkov, Fabien Roger, Clement Neo, Jacob Haimes, Felix Hofstätter, Teun van der Weij. Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models. NeurIPS 2025.
[3] Ryan Greenblatt, Fabien Roger, Dmitrii Krasheninnikov, David Krueger. Stress-Testing Capability Elicitation With Password-Locked Models. NeurIPS 2024.
I thank the authors for the detailed response. I have gone through it. I believe it answers some questions of mine about error bars and GPT-oss results. The authors also have run the attention-head intervention baselines I mentioned, which is definitely a good step. The response has indeed improved my view of this manuscript.
However, given the paper in its current form, I would still not recommend acceptance for it. Even after reading the response, I do not find the methodology more convincingly justified, and some of my concerns about the evaluation remain. I therefore would continue to recommend weak rejection.
Official Comment by Authors
We appreciate the reviewer's feedback and their recognition of the importance and relevance of studying sandbagging as AI model capabilities continue to advance. We understand that the reviewer's main concerns relate to the justification of our methodological choices and the selection of evaluation baselines. Below, we address these concerns through additional experiments and clarifications, and hope the reviewer will consider updating their score.
Why Two Graphs?
We appreciate the reviewer's concern regarding the use of two graphs and agree that this methodological choice deserves a clear justification. Global thresholding at the top 25% most similar edges leaves 9.4%, 3.1%, 2.8%, and 9.2% of heads without any connections in the honest Mistral, Qwen, Llama, and GPT-OSS graphs, respectively. These isolated heads form singleton communities. Therefore, "within-community" connectivity has no intuitive meaning, and we cannot measure meaningful changes for these heads.
To preserve and standardize this notion of connectivity, we detect communities using a symmetric k-nearest-neighbor (kNN) graph constructed from the same honest-condition similarity matrix. Two heads are connected if either ranks the other among its k nearest neighbors, a common practical approach also recommended by von Luxburg [1]. Communities are detected once and held fixed across conditions, ensuring that we are not selecting communities convenient for a particular setting or measurement. Importantly, all connectivity measurements and head-selection scores are computed on the original 25%-density graphs; the kNN graph supplies only community labels.
We further test sensitivity to the kNN neighborhood size, varying k from 0.3k0 to 4k0, where k0 is the default neighborhood size. As shown in the table below, head-score rankings remain strongly correlated with those at k0 (Spearman ρ=0.842–0.972), supporting robustness to neighborhood size. Full details are provided in Appendix D.
Table: Spearman correlation between head scores at each kNN neighborhood size and the default k0. Rankings remain strongly correlated across all tested neighborhood sizes and models.
k% Thresholding
In functional networks, thresholding retains the strongest statistical dependencies, allowing network structure to be studied without retaining numerous weak connections. Most prior functional brain-network studies use densities within 0.15–0.35; we use the middle of this range, 0.25, as the density. We performed sensitivity tests to demonstrate robustness regardless of the chosen density, which can be found in Appendix C. We report that varying the density yields head rankings that remain highly correlated with those from a density of 0.25 (Spearman = 0.828–0.960).
Here, we provide additional empirical evidence of robustness by evaluating GraphLearned across densities of 0.20–0.35. Performance remains comparable across both models and all three benchmarks, with mean accuracy differing by at most 1.90 percentage points from the default density of 0.25. This demonstrates that performance is a result of the GraphLearned algorithm, rather than an artifact of density selection.
Table: GraphLearned accuracy (%) across graph densities on MMLU, WMDP, and ARC. Performance remains very close to the default density of 0.25, demonstrating robustness to the choice of graph density.
Reference
[1] Ulrike von Luxburg. A Tutorial on Spectral Clustering. Statistics and Computing, 17(4):395–416, 2007.