acceptodds
Withdrawn from ICLR 2027

Test-Time Adaptation under Resource-Constrained Deployment via Attention-Head Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Distribution shifts can degrade the response quality of large language models (LLMs), while resource-constrained deployments may lack relevant corpora or the memory needed to host additional pretrained evaluators. Many existing adaptation methods also assume access to multiple test samples from a coherent target distribution to stabilize updates, an assumption that may not hold for queries from different users or domains. We propose per-sample test-time adaptation through attention-head alignment. Brief prefix exploration exposes higher-quality candidates, while prompt–candidate attention-head activation differences provide transferable signals for selecting them. A lightweight scorer calibrated offline on a small set of generic prompt–reference pairs uses these signals to select a response for each query. Both the LLM and scorer remain fixed at deployment, enabling forward-only adaptation without retrieval, additional pretrained evaluators, or information retained from previous queries. Across three LLMs and six benchmarks, our method achieves higher mean ROUGE-LSUM than greedy decoding and the evaluated episodic update baselines for the evaluated backbones.

Primary area:
generative models
Keywords:
Test-time adaptation, large language models, attention-head alignment, resource-constrained deployment, candidate selection, forward-only inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CmUzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a response selector for independent queries to a frozen LLM. It explores the top three tokens at each of the first four decoding positions, then greedily completes the 81 prefixes. An SVR scores candidates using changes in attention-head activation norms between the prompt and the full prompt-response sequence. The scorer is calibrated offline on 64 generic prompt-reference pairs per backbone; neither it nor the LLM is updated at deployment. Evaluation covers three backbones and six datasets, with candidate-budget ablations, controlled-pool comparisons, semantic metrics, and post-release benchmark subsets.

Critical strengths
  1. Appendix I is the most convincing evidence for the head features. It compares selectors on identical candidate pools, with the same calibration data, regression family, and prompt-grouped cross-validation. Table 10 reports mean ROUGE-LSUM of 0.2399 for head alignment, versus 0.2344 for the likelihood-and-length SVR and 0.2279 for expected random selection. Table 11 also reports higher average BERTScore and BLEURT. These controls help separate the value of the selection signal from the benefit of generating more candidates.

  2. The method suits queries that must be processed independently. It uses no target-domain calibration examples, separate pretrained evaluator, or state retained from earlier queries. The empirical coverage is useful: Table 2 improves over greedy decoding in 17 of 18 combinations of backbone and dataset, and Table 3 reports similar aggregate gains with three generic calibration sources.

Critical weaknesses
  1. I could not determine the actual resource requirements from the runtime results. On Medicine with a 3B model on an A100, Appendix E reports 1.32x runtime for 27 candidates and 2.10x for 81, compared with 1.49x for TENT. These results are encouraging, but absolute end-to-end latency, peak memory, candidate batch sizes, and generated-token counts are missing. The text allows batched feature extraction when memory permits, so it is unclear how the timing changes under a memory limit. Table 7 also suggests a cheaper operating point: 3 and 9 candidates score 0.1688 and 0.1706, close to 0.1725 for 27 and 0.1723 for 81. Measuring these budgets under an explicit memory cap would help establish the claimed quality-cost tradeoff.

  2. The qualitative examples make me concerned about what the similarity gains measure. In J.1, the first listed top candidate denies that mulching can control couch grass, contradicting the supplied reference, yet scores 0.3272 in ROUGE-LSUM. In J.2, all three candidates largely repeat the request without naming a technical tool; the first scores 0.4444. A few examples cannot establish an overall failure rate, but these cases call for a check of correctness and instruction fulfillment. BERTScore and BLEURT are useful additions, and the paper acknowledges that correctness is not guaranteed. Still, the HealthBench and FinDER subsets are also assessed by reference similarity instead of their full benchmark criteria. The head-feature advantage is modest and uneven in the controlled comparison, with the likelihood-and-length SVR doing better on Medicine and Agriculture. I therefore lean toward weak rejection until the paper establishes a benefit in answer quality and under the intended resource constraints.

Questions for authors
  1. Can you report absolute end-to-end latency, peak memory, candidate batch sizes, and total generated tokens for 3, 9, 27, and 81 candidates under a stated memory cap? Does the advantage over episodic adaptation persist when all methods use that cap and timing includes candidate generation and feature extraction?

  2. Does "Candidate 1" in Appendix J denote the response returned by the scorer? How are the top three ordered? On a fixed subset of the controlled pools, can you compare head alignment, the likelihood-and-length SVR, and greedy decoding for correctness and instruction fulfillment? This would help determine whether the cases in J.1 and J.2 reflect a broader problem with the selection objective.

  3. Please specify the chat templates, assistant-generation prefixes, stopping rules, and generation limits for each backbone, and whether they were shared across methods. The prompt-like continuations and nonanswers in Appendix J make these details relevant to interpreting the results. This is a request to clarify the existing setup; I have not established an implementation error.

Reviewer_orp2OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. The paper addresses per-query response improvement under distribution shifts, targeting deployments without retrieval corpora, auxiliary pretrained evaluators, or a coherent stream of test inputs.

  2. The proposed framework generates candidates through short-prefix search and ranks them using prompt–response attention-head activation differences. A lightweight regressor is calibrated offline on reference-labeled examples; both the LLM and regressor remain frozen during deployment.

  3. Experiments across three LLMs and six benchmarks report higher mean ROUGE-LSUM than greedy decoding and the evaluated episodic parameter-update baselines, alongside analyses of candidate generation, scoring, and computational overhead.

Critical strengths
  1. A practically motivated deployment setting. Independent-query processing without persistent adaptation state accommodates heterogeneous requests. Avoiding backpropagation and an additional pretrained evaluator reduces deployment requirements, although candidate generation still incurs computational overhead.

  2. A lightweight use of model-internal signals. The method reuses attention-head activations from the deployed LLM to estimate candidate quality. Its offline-calibrated scorer offers a compact alternative to serving a separate large evaluator.

  3. Useful component-level empirical analysis. Beyond end-to-end comparisons, the paper examines candidate-generation strategies, regressor choices, calibration-set size, and fixed-pool selection baselines. These analyses help distinguish candidate exploration from scoring and characterize the method’s operating trade-offs.

Critical weaknesses

1. Misalignment between the claimed test-time adaptation setting and the actual methodology

The paper presents its approach as Test-Time Adaptation (TTA). However, both the LLM and the candidate-quality predictor remain frozen during deployment, with no parameter updates or online learning. The proposed framework primarily generates multiple candidate responses through prefix search and selects the final output using an offline-trained regressor. This process is more closely related to Test-Time Scaling (TTS), inference-time search, and reranking than to conventional model adaptation. While TTA does not necessarily require parameter updates, the authors do not sufficiently clarify what constitutes adaptation in their framework or how it fundamentally differs from existing inference-time selection methods.

Furthermore, the paper formulates its objective as minimizing the distance between an unknown test distribution and candidate- induced distributions. In practice, however, the method only predicts reference-based quality scores using an offline-trained regressor. The theoretical connection between distributional alignment and the actual optimization procedure remains insufficiently justified.

2. Limited evidence for the independent effectiveness of Attention-Head Alignment and potential confounding by response

length

Attention-Head Alignment is presented as the central technical contribution, yet Appendix Table 10 shows that, under identical candidate pools and calibration supervision, the proposed attention-head regressor improves average ROUGE-LSUM only from 0.2344 to 0.2399 compared with an SVR using likelihood and length features. It even performs worse on Medicine and Agriculture. This relatively small improvement raises concerns about whether attention-head alignment is truly responsible for the overall performance gains.

More importantly, the proposed alignment feature may be inherently confounded by response length. According to Eq. (8), the activation statistic for each attention head is computed by averaging the norms of its outputs across token positions:

aℓ,h(z)=1∣z∣∑t=1∣z∣∥Hℓ,h(z,t)∥2.a_{\ell,h}(z) = \frac{1}{|z|} \sum_{t=1}^{|z|} \|H_{\ell,h}(z,t)\|_2.

Eq. (9) then uses the following alignment feature for candidate-quality prediction:

Δ(x,y)=a(x⊕y)−a(x).\Delta(x,y) = a(x \oplus y) - a(x).

Let m=∣x∣m = |x| and n=∣y∣n = |y| denote the prompt and response lengths, respectively, and let aˉℓ,hy(x,y)\bar{a}_{\ell,h}^{y}(x,y) denote the average attention-head output norm over the response tokens, conditioned on the prompt. For a standard causal Transformer, appending response tokens does not alter the activations of the preceding prompt tokens. Therefore,

aℓ,h(x⊕y)=maℓ,h(x)+naˉℓ,hy(x,y)m+n.a_{\ell,h}(x \oplus y) = \frac{m a_{\ell,h}(x) + n \bar{a}_{\ell,h}^{y}(x,y)}{m+n}.

Consequently, the proposed alignment feature can be expressed as

Δℓ,h(x,y)=nm+n(aˉℓ,hy(x,y)−aℓ,h(x)).\Delta_{\ell,h}(x,y) = \frac{n}{m+n} \left( \bar{a}_{\ell,h}^{y}(x,y) - a_{\ell,h}(x) \right). This derivation shows that the alignment feature explicitly depends on the response-to-total-length ratio n/(m+n)n/(m+n). As a result, differences in the proposed features may arise partly from response length rather than genuine prompt-response compatibility. Moreover, the feature captures changes in average activation magnitude rather than semantic alignment directly.

Without sufficient controls that isolate this length-dependent factor and other activation statistics, it remains unclear whether the reported gains originate from meaningful attention-head alignment or simpler statistical properties of the generated responses.

3. Insufficient comparisons with strong baselines that match the actual methodological setting

The main comparisons focus on Tent, EATA, COME, and TLM, which primarily rely on parameter updates for test-time adaptation. However, the proposed method is fundamentally a candidate-generation and reranking framework. More direct competitors would include Best-of-N sampling, self-consistency where applicable, learned candidate rerankers, and other test-time scaling approaches.

Although Appendix I provides comparisons with random selection, minimum perplexity, and an SVR using likelihood and length features, these experiments are limited to fixed candidate pools and do not establish superiority over stronger inference-time search and selection methods. Furthermore, the proposed scorer benefits from reference-labeled offline calibration data, whereas the principal TTA baselines rely on unlabeled test inputs. The comparison should therefore also account for differences in supervision resources and computational budgets.

Overall, the current experiments demonstrate advantages over several single-sample parameter-update methods, but do not sufficiently establish the effectiveness of the proposed framework relative to its most direct methodological competitors.

4. Heavy reliance on ROUGE-LSUM and the absence of LLM-as-a-Judge evaluation

The paper uses ROUGE-LSUM both as the primary supervision signal for training the candidate-quality predictor and as the main evaluation metric. However, improvements in lexical overlap do not necessarily correspond to improvements in factual correctness, instruction following, or overall response quality.

This concern is particularly evident in Appendix J:

  • Agriculture: A candidate incorrectly states that mulching cannot be used to control couch grass, yet receives a relatively high ROUGE-LSUM score.
  • Finance: Several candidates largely paraphrase the input question without identifying the requested analytical tool, while still achieving relatively high scores.

These examples illustrate the limitations of relying primarily on reference-based lexical overlap.

Although the authors additionally report BERTScore and BLEURT, these metrics primarily measure semantic similarity and do not directly establish factual correctness or task completion. LLM-as-a-Judge is an important evaluation protocol for open-ended generation, particularly for domain-specific question answering and instruction-following tasks. The absence of such an evaluation is a significant limitation.

The authors should consider evaluating representative subsets using LLM-as-a-Judge, ideally with an independent judge from a different model family or a blinded human assessment, to determine whether the reported improvements translate into genuinely better responses rather than higher reference similarity.

Questions for authors

1. Why should the first four generated tokens be sufficiently informative across different models and tasks?

The paper primarily adopts prefix search with k=4k=4 and conducts its search-budget analysis mainly on the Medicine dataset with Llama-3.2-3B. However, the importance of early generation tokens can vary substantially across tasks and model architectures. For instance, the first few tokens of an open-ended response may consist primarily of generic introductory expressions, while critical decisions in mathematical reasoning or multi-step question answering may occur much later.

Could the authors investigate whether the effectiveness of early-token branching generalizes across different models and task categories? In particular, comparisons between branching at the beginning and at later generation positions, under matched candidate budgets, would help establish whether the observed improvement is specifically attributable to early-token decisions. Additional evaluations on standard question-answering or reasoning benchmarks, such as NQ-Open or MMLU, would further clarify the generality of this assumption.

2. How reliably does the calibrated predictor rank candidates within individual queries?

The quality predictor is calibrated using only 64 prompt-reference pairs. While Figure 5 reports positive cross-domain Pearson correlations, correlations aggregated over candidates from different prompts do not necessarily demonstrate effective ranking within the candidate set of an individual query.

Since the actual inference procedure selects the highest-scoring candidate separately for each query, the relevant question is whether the predictor reliably distinguishes better and worse candidates for the same input.

Could the authors report query-level ranking statistics, such as within-query Spearman correlation, pairwise ranking accuracy, Top-1 oracle agreement, or selection regret relative to the reference oracle? Such analyses would provide more direct evidence that the learned predictor supports the intended per-query selection mechanism, rather than merely capturing quality differences across prompts or domains.

3. Does the proposed method introduce systematic per-example regressions or undesirable generation behaviors?

The current experiments primarily report dataset-level average scores, while Appendix J provides only a limited number of qualitative examples. However, an improvement in average performance does not necessarily imply that most individual queries benefit from the proposed selection procedure.

Could the authors report the proportions of examples that improve, remain unchanged, or deteriorate relative to greedy decoding? It would also be useful to investigate severe regressions and determine whether they are associated with particular task categories, response lengths, or generation patterns.

For example, does the proposed selector disproportionately favor responses that copy the input, produce repetitive text, generate overly short answers, or drift away from the requested content? A systematic failure-mode analysis would provide valuable insight into the practical reliability and limitations of the approach.

4. Are the claimed resource-efficiency advantages sufficiently supported by realistic deployment experiments?

Resource-constrained deployment is a central motivation of the paper, yet the runtime analysis is primarily conducted on an NVIDIA A100 GPU, with representative results reported for the Medicine dataset using Llama-3.2-3B.

According to Appendix Table 6, the default 81-candidate configuration requires approximately 2.10 times the greedy decoding runtime, whereas the 27-candidate configuration requires only 1.32 times the runtime while achieving slightly better performance. This raises questions about the practical efficiency of the default configuration.

Could the authors provide a more comprehensive quality-efficiency analysis across different models and datasets, including end-to-end latency, total generated tokens, peak GPU memory usage, and throughput? In particular, experiments on memory-constrained GPUs or edge devices would substantially strengthen the deployment claims.

The authors should also clarify whether prefix caching, shared computation, or candidate batching is employed during candidate generation and attention-head feature extraction, as these implementation choices may significantly affect the actual computational and memory requirements.

Reviewer_tv25OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the problem of improving LLM's response to a single query, in a setup without using any retrieval, auxiliar evaluator or parameter updates. It proposes a new method which at its core by picking candidates generation based on attention-head activations. Specifically, it involves in a few steps: 1/ generates candidates by branching over the top-n tokens for the first few decoding steps followed by completing each branch greedily; 2/ a small regressor, calibrated offline on a few generic prompt reference pairs, then selects one candidate from the change in attention head activations between the prompt and the prompt plus candidate. The authors perform experiments with three 3-8B instruction tuned LLMs over four domain specific instruction following datasets, against greedy decoding and other a few baselines (Tent, EATA, COME and TLM), using ROUGE-LSUM as the only metric. The results show gains over greedy decoding in 17 out of 18 settings and larger main gains than other baselines.

Critical strengths

The paper studies a practical problem setup from an interesting angle and proposes a novel and simple approach to it. The paper targets per query adaptation of a frozen LLM when there is no retrieval corpus, no extra judge model and no other relevant queries. It proposes the idea to explore a few early decoding branches and pick one using the model's own attention head activations, with the scorer calibrated once on a small generic datasets. Using internal head signals to select response is a fresh direction that can be further extended on any tasks (if it works well). I think this is the main value I see in the paper.

Critical weaknesses
  1. The evaluation is too thin to show that the method actually produces better responses. All tasks are instruction following datasets and scored against a single ground truth with ROUGE-LSUM. I will be very cautious on using ROUGE to really measure the output quality of the LLM responses, especially we saw the absolute score (in Table 2 and 3) is pretty low. With this low score, and gains are in ballpark of 0.02-0.03, it is very difficult to tell whether these gains are meaningful or just noise, especially no statistical analysis is done there. Also, the task setup is too thin, why not also include a broader tasks like math, code etc. which is easier to verify answers?

  2. The baselines and ablation studies do not show where the gains come from, or how the method compares with other simpler ways to choose among the sample generations. The core idea of the paper is to generate multiple candidates during test time and pick one from them, yet it is compared only with greedy decoding and gradient based test time training. Other simple methods such as self-consistency or tree of thoughts also need only the frozen model, and they are not compared. I know they are not easy to compare with instruction following task, but I think we should test with broader set of tasks and compare with them, due to their simplicity and effectiveness. Also, the ablation on the core attention head signal is thin, and I am not sure if the real gain comes from this signal or simply generating more candidates (just for an example).

Nevertheless, I think the main issue is in the experiments, I am wiling to increase my rating if the experiments can be enhanced, and more analysis can be done.

Questions for authors
  1. I am curious how much the attention-head scorer adds over simpler ways of choosing among the same candidates. Do you have numbers for a consensus method, such as self-consistency style voting, on some appropriate tasks?
  2. Have you compare the model generation quality between baselines with other metrics (like a reward model) in addition to ROUGE? Using a reward model (or LLM as a Judge) should be pretty straightforward (and more relevant IMO) to measure the LLM response quality.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.