acceptodds
Withdrawn from ICLR 2027

Selecting The Most Informative Tokens in Natural Language Autoencoders

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Natural language autoencoders translate a language model's internal activations into readable explanations. Explaining every token position is costly. Which positions should an auditor inspect to understand a potential threat? We study this question across 4.7 million explanations on prompt injection and concealment. We compare signals from model computation with a ranker trained only on chat structure. Chat structure usually selects more relevant explanations than the computational signals, without requiring a model forward pass for position selection. On three of four datasets, explaining just 5% of positions retains nearly all of the success rate from explaining every position, where success means obtaining an explanation about the threat. The benefit varies with the audit task. We also show that pretrained verbalizers recover words that models have learned to conceal through fine-tuning, without additional verbalizer training. These results identify where auditors can concentrate explanation generation and show that useful explanations can extend beyond the model a verbalizer was trained to describe.

Primary area:
interpretability and explainable AI
Keywords:
natural language autoencoders, mechanistic interpretability, best token selection, activation verbalizers, large language models, token selection, alignment auditing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6MiROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the efficiency of auditing large language models with natural language autoencoders. Because verbalizing every token position is costly, the authors study how well a wide range of signals predicts which positions are worth inspecting to find a threat. Their procedure verbalizes every position of each transcript and passes each verbalization to an LLM judge, which labels the position as 'on task' (relevant to the inspection) versus 'off-task' (irrelevant). These labels are then used to evaluate the signals. Across four datasets covering prompt injection and concealment they find that a simple structural baseline almost always outperforms signals derived from the internal representations of the model, and they conclude by recommending it as the practical recipe.

Critical strengths

The work provides a large and systematic evaluation study that is useful to guide future research on finding optimal predictive signals for efficient auditing.

The procedures used in the paper are well written and documented.

Critical weaknesses

The paper uses a large portion of its length to the analysis of the proposed signals. Although these signals carry some predictive information, they are consistently outperformed by the structural baseline and their behavior varies considerably across datasets and models, so they offer limited guidance toward a practical auditing recipe. While the systematic evaluation is valuable, it occupies a lot of space for a largely negative result. By contrast, the structural baseline that the authors ultimately recommend as the practical recipe receives much less analysis.

The paper aims to find position-selection strategies that improve on the heuristics used in prior works, yet it offers little to no comparison with those methods.

For these reasons I have concerns about the impact of the findings of the paper and I believe that the work needs some refinement, better addressing the task of providing a practical recipe for position-selection and I am inclined to decide weak rejection.

Questions for authors

At the same budget, does the structural baseline actually outperform the position-selection heuristic used in prior works, such as the final token before the answer or eight evenly spaced positions?

Does the structural baseline consistently select the same positions or chat roles, and does this selection generalize across datasets? Could a hardcoded rule, similar to those used in prior work, be extracted from the structural baseline?

The cost of the labeling procedure is unclear. Both the structural baseline and the signal directions must be fitted on labels, which requires verbalizing and then judging a portion of the dataset with an LLM. The experiments use four fifths of the transcripts for this. how many labeled transcripts are needed in practice, and how does this one-time cost compare with the savings at audit time.

Reviewer_oHPjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper is about deciding which positions in an natural language autoencoder a verbalizer should choose to verbalize, and how these positions can be chosen effectively (given a fixed budget). The authors evaluate on four datasets (2 prompt injection ones and 2 concealment ones). The paper evaluates primarily on whether the explanations that are generated are about the treat.

Critical strengths

the paper addresses a question that may have some practical relevance in explainability research, and which has not yet been studied systematically. the paper is careful with respect to its evaluation statistics. it is also (LLM style)-honest about negative results

Critical weaknesses

A serious issue of the paper is that it is a pain to read. The motivation, use case and problem setting is not explained sufficiently. What is the assumed audit workflow? Why this method as opposed to cheaper information from a linear probe or a text classifier? The whole paper appears to be full of LLM generated text. It uses lots of undefined terminology (bridge, cell, shard, organism, boundary ordinal, candidate, Box B...). Also, the sentences in the paper are not well connected, which makes it very hard to follow and incoherent. Throughout, the paper states results without ever explaining their relevance in the argument. Additionally, the paper doesn't interpret and discuss what its many results mean. The results in fact have different possible interpretations (e.g., label-trained chat-structure ranker beating signals from model internals could be a benchmark artifact, or the signals may carry redundant information, or the explanations may reflect mostly local context).

Finally, I am missing a demonstration that there is a usefulness of this method to human auditors in a downstream task. Are the explanations generated based on the tokens actually more useful to an auditor than what they would get from alternative methods? How feasible is it to auditors to even read the generated text from these chosen positions? Or is this just going to go into other LLMs?

Questions for authors

In which of your four settings would an auditor without NLAs fail to detect the threat, please explain why.

How do the final-position and even-grid conventions from prior work compare under your budgeted precision and success metrics?

Did you check the judge's labels against human labels on a sample, and what did you find?

Reviewer_3UrwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies which token positions should be explained by a natural language autoencoder when an auditor has a limited explanation budget. Across 4.7 million explanations, four models, and four datasets, it compares computational signals and their combinations against position and chat-structure baselines. Chat structure generally performs best, and explaining 5% of positions preserves most of the exhaustive explanation-relevance success rate on three datasets. The paper also shows that unchanged verbalizers can recover words concealed by fine-tuned models. The presentation is generally clear, and the systematic empirical comparison is a useful contribution. Its originality lies primarily in the evaluation rather than a new interpretability method. However, the relevance labels and limited evidence of generalization weaken the practical auditing conclusions.

Critical strengths
  1. Generating explanations for every token position across 4.7 million examples provides a thorough foundation for comparing position selectors without bias toward preferred locations.

  2. The paper shows that inexpensive chat metadata (structure) often selects relevant explanations better than complex computational signals. On OpenPromptInjection and taboo organisms, a 5% token budget retains 99.5% and 95.8% of exhaustive success.

  3. The distinction between pooled AUROC, case-macro AUROC, and budgeted precision is insightful. It successfully highlights that a high overall AUROC can hide zero precision at the top-ranked position.

Critical weaknesses
  1. Appendix G shows that benign system instructions in OpenPromptInjection receive on-task labels 28–47% of the time, and Tensor Trust shows inverse trends where explanations are less on-task during successful attacks. Since selectors optimize against these labels, the authors should validate a stratified sample with human annotators using full transcript context, separating legitimate instructions from actual compromises.

  2. Training and test folds share dataset layouts where threats and instructions appear in predictable places. The structural advantage is not tested on held-out attack or template families with varying conversation structures. Testing cross-layout transfer would strengthen the recommendation for chat structure.

  3. Audit success currently requires only a single on-task explanation, making the 95.8% retention metric sensitive to definition. Requiring three explanations drops retention sharply to 78.5% and 56.8%. It is better to report absolute success rates, check uncertainty, and verify whether selected explanations actually preserve correct downstream audit decisions.

Questions for authors

Do the structural baseline’s advantage and the 5% budget result hold after human validation of labels

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.