acceptodds
Under review as a conference paper at ICLR 2027

Encoding Prompts into Single Vectors via Linear Activation Aggregation

Abstract

Large language models process prompts by propagating a sequence of activations through dozens of layers before generating a response. How that sequence should be encoded into a single representation that keeps the relevant information of the prompt is largely unresolved. Existing approaches sit at two extremes: heavy learned compression methods for efficiency, and simple activation-engineering heuristics like the last-token activation, chosen for convenience rather than fidelity. Motivated by the advances in mechanistic interpretability, we show that a prompt can be represented as a linear combination of its own activations. We instantiate this approach with a lightweight MLP that predicts a weight for each token. This design encodes a prompt into a single activation vector whose information can be recovered by the frozen LLM when reinjected at an early layer. Despite its simplicity, the weighted-sum representation preserves prompt information more faithfully than end-to-end compression and provides an interpretable construction. Beyond its practical implications, our analysis reveals structure in the representation space of LLMs: (i) mid-layer representations transfer meaningfully to early layers, suggesting a degree of cross-layer compatibility in how information is encoded; (ii) a single activation vector encodes a quantifiable and recoverable amount of semantic information; (iii) a weighted sum of activations provides an effective and interpretable mechanism for single-vector prompt encoding.

Primary area:
interpretability and explainable AI
Keywords:
LLM, Mechanistic Interpretability, Explainable AI, Efficient Inference, Activation Engineering, Activation Patching, Representation Learning, Superposition, Linearity, Prompt Compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Jp9rOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether a natural-language prompt can be encoded into a single activation vector while preserving task-relevant information. The proposed approach extracts contextualized token activations from an intermediate layer of a frozen LLM and aggregates them into a single vector using token-wise scalar weights predicted by a lightweight MLP. The resulting vector is then injected into an early layer of the same LLM, allowing the model to answer without access to the original prompt.

Critical strengths
  • Simple and interpretable formulation : I found it interesting that a prompt can be compressed into a single vector using a relatively simple weighted sum. In particular, the observation that the constrained W-MLP generalizes better than the more expressive Transformer Compressor suggests that existing LLM activations may already contain useful structure for constructing compact prompt representations. The analysis of extraction/injection layers and token-level weights also provides useful empirical observations about how such representations behave.
Critical weaknesses
  • Insufficient baselines : The main contribution relies on learning token-wise weights with an additional MLP, but the empirical comparisons do not sufficiently establish whether learning this MLP is necessary. In particular, mean pooling and last-token representations are introduced as natural alternatives in the paper, but their results are omitted from the main results. I believe reporting these results is still important for understanding the benefit of the proposed aggregation method. More importantly, an attention-based aggregation appears to be a particularly relevant training-free baseline. For example, token importance derived from the frozen LLM’s attention could be used as weights to construct a vector from the same token activations. This seems especially relevant because Section 4 itself relates the learned weighting patterns, including the large weights assigned to punctuation tokens, to attention sinks. Without stronger training-free aggregation baselines, it is difficult to determine how much of the reported benefit comes specifically from learning the W-MLP rather than from appropriately aggregating representations that already exist in the frozen LLM.
Questions for authors
  • Could you report the numerical results and evaluation details for the mean-pooling and last-token baselines mentioned in Section 3.1, and additionally evaluate a training-free attention-based weighting scheme using the frozen LLM?
  • Is there a reason for using the raw scalar outputs of the W-MLP directly in the weighted sum, rather than normalizing the weights? I am particularly curious about robustness to changes in prompt length. Since the magnitude of v depend on both the learned weights and the number of tokens, could a distribution shift in prompt length substantially change the norm of the resulting patch vector? Have the authors evaluated whether normalization affects performance or length generalization?
4 comments

We thank the reviewer for recognizing the interest of our findings and for their constructive suggestions. It helped us strengthen the empirical evaluation and better characterize the method’s limitations.

(Q1) We thank the reviewer for highlighting these missing comparisons. While re-evaluating the baselines, we identified and corrected an implementation bug affecting the results on training free baselines. We also added the suggested training-free attention-based aggregation, taking as weight in the sum the attention to the last token at the chosen layer.

The natural training free baselines archieve a maximum of 60% accuracy on ARC-easy compared to 83% for the trained W-MLP. We will include the corrected results, the additional baselines, and their implementation details in the revised manuscript.

RepresentationInjection at layer 2Injection at layer 12
W-MLP, trained with injection at layer 283.4%73.5%
Last token53.5%60.1%
Max pooling53.3%58.9%
Mean pooling51.8%58.8%
Attention weighting39.2%40.7%

(Q2) We agree that the absence of normalization raises an important question about sensitivity to prompt length and may contribute to the lower accuracy on longer prompts reported in Figure 7 of the appendix. We trained W-MLP variants using length-normalized aggregation and softmax-normalized weights. Under the same training protocol, neither variant learned effective prompt representations, achieving performance close to the masked-prompt baseline.

MethodTest Accuracy
W-MLP as raw sum83.4
W-MLP sum divided by N44.0
W-MLP sum divided by log N44.1
W-MLP softmax of weights41.8

We also measured patch-vector norms as a function of prompt length and observed an overall increasing trend:

Prompt lengthNorm value
11–1585.82
16–20103.74
21–25106.70
26–30114.60
31–35114.36
36–40120.70

Taken together, these results suggest that longer prompts may require larger patch-vector norms for the LLM to recover their information. However, increasing the norm may also disrupt the model’s subsequent computation, potentially contributing to the lower accuracy on longer prompts. This points to a limitation of the current aggregation mechanism when extending to longer contexts. We will report these results and discuss this limitation explicitly in the revised manuscript.

Thank you for the additional experiments and clarifications. The new baselines address my concern about the benefit of learned aggregation. However, the robustness of raw aggregation to changes in prompt length remains insufficiently evaluated, and the normalization experiments do not directly resolve this concern.

Thank you for confirming that the additional baselines address the concern about learned aggregation. We agree that the normalization experiments do not directly evaluate robustness to changes in prompt length, they rather test alternative aggregation formulations and suggest representation scale as a possible factor in the degradation on longer prompts.

To clarify the scope, the paper does not claim robustness to long prompts. Its central finding is that a constrained weighted aggregation can preserve prompt-dependent behavior in the evaluated short-prompt setting. The degradation on longer prompts, reported in Figure 7, is an acknowledged limitation.

Could you clarify what additional evaluation you consider necessary beyond Figure 7? This would help us understand which aspects of the length-related limitation require further characterization within the scope of the paper.

Thank you for the clarification. I had overlooked the prompt-length analysis in Figure 7 and appreciate you pointing it out. I have no further questions on this point.

Reviewer_7EjfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method for compressing the activations of an LLM prompt (at an intermediate layer), such that when it is injected to an earlier layer largely reproduces the output of the original LLM prompt. Two compression schemes are proposed: (learned) linear combination of the hidden states and a transformer-based compression of the activations. The linear combination generally works better than transformer-based. While it is interesting to see that injecting this compressed representation in an earlier layer work, it is not clear what the purpose for doing so is. Is seems purely educational.

Critical strengths

The paper shows that compressed activations of an initial prompt can be taken from an intermediate layer of an LLM and be fed to an earlier layer, replicating the result of that prompt. It's interesting that a (learned) linear combination of the hidden representation (Eq. 1) can represent a prompt fairly well.

The paper reads very well and does a good job evaluating the results.

Critical weaknesses

The core question is how this insight could be useful. The paper does not really mention a use case. It is mainly a paper providing some interesting insights. I appreciate work that discovers new insights, I'm just not convinced the insights here are important, as I didn't see downstream tasks that might benefit.

Questions for authors

I would love to see how passing uncompressed activations from layer m passed to layer e perform in comparison to the compressed versions.

1 comment

We thank the reviewer for their positive assessment of the paper, and for the opportunity to clarify the significance and potential applications of our findings.

(W1) We agree that the manuscript should make the potential uses of the aggregated representation more concrete. We evaluate the representations on binary sentiment classification using texts from “Sp1786/multiclass-sentiment-analysis-dataset”. Specifically, we compare W-MLP patch vectors with commonly used last-token and mean-pooled token representations. We evaluate both a W-MLP trained on sentiment and one trained on the unrelated ARC-Easy tasks. Although the improvement with sentiment-trained W-MLP is modest, this provides a concrete example of using the aggregated representation beyond prompt replacement. As a reference the base accuracy of Ministral3-8B on the sentiment classification in natural language on this dataset is 86.8%. We will include this experiment in the revised manuscript.

ProbeMLP (512)LDA
W-MLP (trained on sentiment)92.292.5
Last token probe90.391.5
Mean token probe91.491.4
W-MLP (trained on ARC-easy)0.850.86

Beyond this probing example, our findings suggest two further research directions. For interpretability, the explicit aggregation weights provide a way to investigate how contextualized token representations contribute to a usable prompt representation. For prompt compression, the results motivate future works using existing activations rather than learning an unrestricted compressed representation. Developing this into a practical compression method remains an open challenge, particularly for longer and more complex prompts.

(Q1) Following the reviewer’s suggestion, we transferred the full, uncompressed sequence of activations from extraction layer m to injection layer e, using one patch per source-token position. ARC-Easy test accuracies are reported below. For reference, single-vector W-MLP achieves approximately 83%, compared with 96% for the original full prompt. Retaining the full activation sequence improves performance over single-vector aggregation for the same extraction–injection pair, but does not recover full-prompt performance. This remaining gap may reflect disruption of the model’s original computation and the use of placeholder tokens. We will report the layer-transfer matrix in the appendix and discuss this comparison in the main text.

Injection layer \ Extraction layerL2L12
L195.089.5
L295.187.7
L1242.283.3
L1841.546.0
Reviewer_BGQgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a method motivated by mechanistic interpretability that compresses a prompt into a single token, via a small neural network.

The compressor acts on the hidden states of the prompt as it is passed through a frozen large language model (LLM), and the compressed output is used to replace a placeholder token at test time when decompressing the output.

The method is evaluated on a "toy task" and ARC-Easy, and is compared to a masked-prompt baseline, and ablations are performed on the specific extraction and injection layers.

Finally, a qualitative analysis of what the learned patch vector represents is performed, where the authors show that low-information-density words such as "is", "the", and "this" are assigned less weight compared to others and terminal punctuation in short prompts.

Critical strengths

The approach is somewhat novel: rather than focusing on compressing a prompt from the final hidden states of a prompt, the work focuses on any single intermediate layer's hidden states

Critical weaknesses

The evaluation of the method is incomplete: no comparison is made with established prompt compression methods.

Questions for authors
  • How does this method differ from other prompt compressors with single memory tokens, besides that the compressor is not the frozen language model and the extracted and injected latent is derived from specific layers of the network instead of the last (encoding) or first (decoding)?
  • How does this method compare against mechanistic interpretability-based approaches that require a similar amount of compute, such as task or function vector approaches?
  • How is the performance when compressing hidden states from multiple layers into one or more hidden states or layers?
  • Since the prompts in Table 3 are very simple, can you reproduce the visualization in Table 3 on longer prompts with higher complexity, such as a multiple-step math problem instead of (b)?
1 comment

The review primarily discusses the work as a proposal for practical prompt compression. However, the paper’s central contribution concerns the representation itself: a simple W-MLP can aggregate contextualized token activations into a single vector while preserving the prompt behavior in a frozen LLM. The significance of this result lies in the success of this restricted aggregation mechanism, rather than in a claim of superior compression performance or efficiency. We expect more expressive, extensively trained compression methods to achieve higher task performance.

To assess the benefit of learned aggregation, we additionally evaluated last-token and mean-pooled representations as described in the response to reviewer 3 (Q1). We will include these comparisons in the revised manuscript.

(Q1) W-MLP learns only the scalar aggregation weights, rather than learning to construct a valid activation representation from scratch.

(Q2) Task and function vectors share our interest in understanding and reusing LLM representations. As discussed in Section 6, our method addresses a broader prompt setup: it operates directly on user instructions without requiring structured in-context demonstrations of a task.

(Q3) Jointly aggregating multiple layers or using multiple patch vectors is an interesting extension, already discussed in our future-work section. Here, we deliberately restrict the setup to a single source layer and a single patch vector to investigate whether this simple aggregation is sufficient. More expressive variants could improve performance, but are not necessary to establish this finding.

(Q4) The appendix already provides additional prompt-weight visualizations, including a longer text example in Figure 10 . Since the W-MLP is trained on short, simple user prompts, as motivated in Section 3.1, a multi-step mathematical problem would test out-of-distribution generalization. Nevertheless, we will include such a visualization in the revised manuscript to better illustrate the method’s behavior and limitations beyond its training setting.

Comments on the paper

1 comment

Thank you to the reviewers for their constructive suggestions. We have uploaded a revised version of the paper, with all changes highlighted in blue.

  • Weight visualization: Appendix J now includes an example showing W-MLP’s token weights on a math reasoning problem.
  • Additional baselines: We added baseline comparisons to the main results, with extended results for full-sequence activation patching and training-free alternatives in the appendix.
  • Activation norms: We added a discussion of norm-related limitations and a dedicated analysis in the appendix.
  • Downstream applications: We added a section showing that W-MLP representations improve sentiment probing accuracy over last-token and mean-pooled representations, with further details in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.