- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether a natural-language prompt can be encoded into a single activation vector while preserving task-relevant information. The proposed approach extracts contextualized token activations from an intermediate layer of a frozen LLM and aggregates them into a single vector using token-wise scalar weights predicted by a lightweight MLP. The resulting vector is then injected into an early layer of the same LLM, allowing the model to answer without access to the original prompt.
Critical strengths
- Simple and interpretable formulation : I found it interesting that a prompt can be compressed into a single vector using a relatively simple weighted sum. In particular, the observation that the constrained W-MLP generalizes better than the more expressive Transformer Compressor suggests that existing LLM activations may already contain useful structure for constructing compact prompt representations. The analysis of extraction/injection layers and token-level weights also provides useful empirical observations about how such representations behave.
Critical weaknesses
- Insufficient baselines : The main contribution relies on learning token-wise weights with an additional MLP, but the empirical comparisons do not sufficiently establish whether learning this MLP is necessary. In particular, mean pooling and last-token representations are introduced as natural alternatives in the paper, but their results are omitted from the main results. I believe reporting these results is still important for understanding the benefit of the proposed aggregation method. More importantly, an attention-based aggregation appears to be a particularly relevant training-free baseline. For example, token importance derived from the frozen LLM’s attention could be used as weights to construct a vector from the same token activations. This seems especially relevant because Section 4 itself relates the learned weighting patterns, including the large weights assigned to punctuation tokens, to attention sinks. Without stronger training-free aggregation baselines, it is difficult to determine how much of the reported benefit comes specifically from learning the W-MLP rather than from appropriately aggregating representations that already exist in the frozen LLM.
Questions for authors
- Could you report the numerical results and evaluation details for the mean-pooling and last-token baselines mentioned in Section 3.1, and additionally evaluate a training-free attention-based weighting scheme using the frozen LLM?
- Is there a reason for using the raw scalar outputs of the W-MLP directly in the weighted sum, rather than normalizing the weights? I am particularly curious about robustness to changes in prompt length. Since the magnitude of v depend on both the learned weights and the number of tokens, could a distribution shift in prompt length substantially change the norm of the resulting patch vector? Have the authors evaluated whether normalization affects performance or length generalization?
4 comments
Thank you for the additional experiments and clarifications. The new baselines address my concern about the benefit of learned aggregation. However, the robustness of raw aggregation to changes in prompt length remains insufficiently evaluated, and the normalization experiments do not directly resolve this concern.
Thank you for confirming that the additional baselines address the concern about learned aggregation. We agree that the normalization experiments do not directly evaluate robustness to changes in prompt length, they rather test alternative aggregation formulations and suggest representation scale as a possible factor in the degradation on longer prompts.
To clarify the scope, the paper does not claim robustness to long prompts. Its central finding is that a constrained weighted aggregation can preserve prompt-dependent behavior in the evaluated short-prompt setting. The degradation on longer prompts, reported in Figure 7, is an acknowledged limitation.
Could you clarify what additional evaluation you consider necessary beyond Figure 7? This would help us understand which aspects of the length-related limitation require further characterization within the scope of the paper.
Thank you for the clarification. I had overlooked the prompt-length analysis in Figure 7 and appreciate you pointing it out. I have no further questions on this point.
We thank the reviewer for recognizing the interest of our findings and for their constructive suggestions. It helped us strengthen the empirical evaluation and better characterize the method’s limitations.
(Q1) We thank the reviewer for highlighting these missing comparisons. While re-evaluating the baselines, we identified and corrected an implementation bug affecting the results on training free baselines. We also added the suggested training-free attention-based aggregation, taking as weight in the sum the attention to the last token at the chosen layer.
The natural training free baselines archieve a maximum of 60% accuracy on ARC-easy compared to 83% for the trained W-MLP. We will include the corrected results, the additional baselines, and their implementation details in the revised manuscript.
(Q2) We agree that the absence of normalization raises an important question about sensitivity to prompt length and may contribute to the lower accuracy on longer prompts reported in Figure 7 of the appendix. We trained W-MLP variants using length-normalized aggregation and softmax-normalized weights. Under the same training protocol, neither variant learned effective prompt representations, achieving performance close to the masked-prompt baseline.
We also measured patch-vector norms as a function of prompt length and observed an overall increasing trend:
Taken together, these results suggest that longer prompts may require larger patch-vector norms for the LLM to recover their information. However, increasing the norm may also disrupt the model’s subsequent computation, potentially contributing to the lower accuracy on longer prompts. This points to a limitation of the current aggregation mechanism when extending to longer contexts. We will report these results and discuss this limitation explicitly in the revised manuscript.