acceptodds
Under review as a conference paper at ICLR 2027

Loss-Aware Test-Time Subspace Adaptation for Reduced-Rank LLMs

Abstract

Low-rank compression reduces deployment memory but fixes the model subspace before the request is known. We ask whether a deployed reduced-rank language model can adapt its stored factors at inference time without reconstructing or retaining dense weights. We introduce request-local loss-aware test-time adaptation (TTA) in the deployed low-rank subspace: observed prompt tokens define a masked causal loss, request gradients choose update support, and a hard Kullback–Leibler (KL) trust region limits functional drift. Across Qwen3 models, test-time adaptation consistently improves held-out WikiText likelihood and improves GSM8K in selected gauge/support settings, while ARC-Challenge chat remains statistically unchanged despite substantial prompt-loss reduction. Matched-KL ablations further show that gauge, factor side, and block support can change downstream behavior even when functional displacement is nearly fixed. Together, these results identify update geometry and surrogate–task alignment as central design variables for adapting compressed models at test time.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Decomposition, Compression, Test-time adaptation, Subspace tracking

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_q9mHOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies request-local adaptation of language models deployed as low-rank factors. Observed prompt tokens define a causal language-model loss. The method takes one balanced factor-gradient step, optionally selects transformer blocks by gradient norm, and scales the update under a predictive-KL constraint before generation and checkpoint restoration. Experiments compare block-identity gauges and factor-update choices, select B/B + factor-A (BB+A), and evaluate the frozen configuration on continuation likelihood and reasoning tasks. The motivation is to recover performance after compression while controlling adaptation cost.

Critical strengths

The experiments document a useful implementation choice. From the same BB checkpoint, Tables 5–6 report 34.27% GSM8K accuracy with A-only updates in 1.332 seconds per problem, versus 34.72% in 6.428 seconds with joint updates. The cost difference is large, and the cheaper factor side reverses between AA and BB as the construction predicts. The comparison between AA+B and BB+A also shows that avoiding regauging does not by itself identify the better-performing choice.

The frozen evaluation reports both recovery and its limits. Tables 12–13 show all-block perplexity reductions of 7.0–17.1% at all twelve combinations of two larger models and six storage fractions; top-k also improves every point. On ARC, top-k reduces prompt cross-entropy by 0.3412 while accuracy changes from 66.21% to 66.13%, with beneficial and harmful answer changes nearly cancelling. These results distinguish successful prompt optimization from improved answers and provide useful information about this particular adaptation procedure.

Critical weaknesses

The residual method is a direct application of existing mechanisms. LatentLLM [1] already supplies block-identity storage, LoRA Done RITE [2] analyzes factor-coordinate dependence, and Revisiting Dynamic Evaluation [3] studies observed-token adaptation, restricted blocks, and online MLP LoRA. Adapting the compressed factors changes the parameters being optimized, but ordinary differentiation already permits those updates without the original dense weights. Under B-side gauge, holding B fixed automatically preserves its identity block, so BB+A avoids regauging by construction. Scalar balancing, KL scaling, and gradient-norm support selection organize this update but introduce little additional methodological insight. The paper's main new decision is to select BB+A empirically from the gauge/factor combinations. The discussion also omits [2] and [3].

The empirical findings do not establish a substantial advance beyond that configuration choice. The approximately fivefold adaptation-time difference compares a single-factor path that avoids regauging with joint updates that require it. Its magnitude and accuracy tradeoff are useful measurements, but the source of the cost saving is already explicit in the representation. The larger-model experiments establish recovery against static checkpoints, with limited comparative evidence for the selected update rule. Likewise, the geometry and ARC results instantiate the distinctions between update direction and magnitude, and between prompt loss and answer quality. The study measures these effects in compressed models but does not develop them into a new adaptation decision or a sufficiently distinctive empirical finding. Taken together, the evidence supports an effective configuration more strongly than a substantive methodological or scientific advance.

References

[1] Toshiaki Koike-Akino, Xiangyu Chen, Jing Liu, Ye Wang, Pu (Perry) Wang, and Matthew Brand, "LatentLLM: Activation-Aware Transform to Multi-Head Latent Attention," AAAI, 2026. Paper

[2] Jui-Nan Yen, Si Si, Zhao Meng, Felix Yu, Sai Surya Duvvuri, Inderjit S. Dhillon, Cho-Jui Hsieh, and Sanjiv Kumar, "LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization," ICLR, 2025. Paper

[3] Amal Rannen-Triki, Jörg Bornschein, Razvan Pascanu, Marcus Hutter, András György, Alexandre Galashov, Yee Whye Teh, and Michalis K. Titsias, "Revisiting Dynamic Evaluation: Online Adaptation for Large Language Models," arXiv:2403.01518, 2024. Paper

Questions for authors
  1. Clarification: request-dependent support. Table 3 changes both the block count, from 28 to 14, and which blocks are updated. What evidence from the completed experiments or analyses supports attributing the benefit to request-dependent gradient ranking, beyond restricting the support size? Proposition 2 motivates ranking for local prompt-loss reduction under common curvature, while Table 3 reports slightly greater prompt-loss reduction with all blocks. Clarifying the evidence for the ranking itself would help assess the methodological contribution of request-dependent selection.

  2. Clarification: matched predictive KL. For Tables 3 and 5, which token positions define the KL constraint in Appendix B.7, and how are the per-token divergences aggregated within and across requests for the reported KL? Are these definitions shared across the compared policies within each request? This would clarify whether the near-matched values represent the same measure of predictive change, which is central to interpreting the comparisons of update directions.

1 comment

Thank you for the careful novelty analysis and the specific references. We will discuss LoRA Done RITE (Yen et al., ICLR 2025) and Revisiting Dynamic Evaluation (Rannen-Triki et al., 2024) alongside LatentLLM. LoRA Done RITE studies transformation-invariant optimization of additive low-rank adapters; dynamic evaluation studies online language-model updates from observed tokens.

Our experiments address the deployment setting in which selected pretrained operators have already been replaced by reduced-rank factors, the original dense weights are absent, and each request begins from—and ultimately restores—the same compressed checkpoint. The study asks which stored coordinates and update supports give useful behavior under a bounded predictive change.

Factor-coordinate contribution. The measured AA/AB/BA/BB × A/B/A+B matrix tests 12 deployment-coordinate choices using the same model and hard KL budget. At closely matched realized KL (0.02035–0.02284), GSM8K accuracy changes by as much as −2.81 to +2.65 percentage points relative to gauge-matched fixed baselines.

Under B/B gauge, A-only adaptation reaches 34.27% in 1.332 s/request, compared with 34.72% in 6.428 s for joint A+B updates. Under A/A gauge, the low-cost no-regauge side reverses to B-only (1.208 s, versus 6.481 s for A-only). The gauge-preserving fast path follows directly from the identity-block construction; our contribution is the controlled measurement of its predictive consequences and the resulting deployment-coordinate tradeoff. We will foreground this distinction and position balancing and KL bisection as practical components of the procedure.

Support selection and paired evidence. Proposition 2 motivates gradient-norm ranking for a fixed-cardinality support under a common-curvature, local prompt-loss approximation. The empirical question is held-out behavior.

From the stored full-test prediction comparisons underlying Table 3, top-k versus all-block yields 80 wrong-to-correct and 63 correct-to-wrong transitions, giving +1.29 points with exact McNemar p = 0.1807.

We further performed a precommitted SmolLM2 control over 48 WikiText-validation requests with 12 compressed blocks and an equal six-block support for gradient top-k, fixed halves, and five random seeds. The measured held-out NLLs were 3.65091 (top-k), 3.65099 (fixed first-half), 3.65383 (five-seed random mean), and 3.64931 (all blocks). Top-k minus random was −0.00292 NLL, with descriptive paired interval [−0.00617, +0.00028].

These results support selective updating as an efficiency choice; evidence specific to gradient ranking remains modest. We will state Proposition 2's local assumptions and interpret the ranking as a low-cost heuristic.

Exact KL definition. For each request, the implementation selects the positions outside the supervised causal-loss mask in its up-to-512-token prompt window, excluding the first token. If there are more than 64 eligible positions, it selects up to 64 approximately evenly spaced indices.

The reported table value is the arithmetic mean of these per-request KL values. The masks, index-selection rule, and evaluation contexts are identical across compared policies for each given request in Tables 3 and 5. The line search targets 95% of the 0.025 hard budget, subject to direct nonlinear KL verification.

The newer SmolLM2 adaptation-source control holds the KL index set fixed across different adaptation masks and reports that change in protocol explicitly.

Empirical takeaway. Beyond likelihood recovery, the experiments show that matched scalar KL and improved observed-token CE leave consequential freedom in downstream update direction. The frozen C4→WikiText sweep finds reduced perplexity at all 12 compressed model/budget points, while ARC-Challenge's beneficial and harmful answer flips nearly cancel.

The additional SmolLM2 source control also favors adaptation to the current query (3.54029 NLL) over a reused shared-prefix update (3.56498) and unrelated-query update (3.55501). We will emphasize these measurable representation, coordinate, and surrogate-alignment effects as the paper's substantive findings.

Reviewer_pCvuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether a compressed language model can adapt itself to each request it receives, even after its original weights have been discarded. Selected MLP projections are stored as pairs of thin low-rank matrices, and because the same matrix can be split into such a pair in many equivalent ways, the authors fix one convenient form in which part of one factor is always the identity, so those entries never need to be stored. When a request arrives, the model computes a language-modeling loss on the prompt text it can see, picks the transformer blocks with the largest gradients, and takes one small gradient step on the stored factors. A KL budget limits how much the model's predictions can change, and the compressed checkpoint is restored before the next request. The authors argue that the way the factors are written down, which factor gets updated, and which blocks are chosen all change the direction of this update, while the KL budget only controls its size. Across Qwen3 models, the method reliably improves perplexity on WikiText and gives small but significant gains on GSM8K in a few settings, while ARC-Challenge does not improve, showing that fitting the prompt better does not always translate into better answers.

Critical strengths
  1. Well-controlled experimental design. The paper uses paired exact McNemar tests, deterministic decoding, and a dense parity check against the LM Evaluation Harness. The matched-KL setup is a sensible way to separate the direction of an update from its size.

  2. A clear and relevant observation. The paper shows that lower prompt loss and similar KL do not reliably predict downstream accuracy, and that equivalent factorizations can lead to different updates. This is a useful caution for anyone doing test-time adaptation on compressed models.

Critical weaknesses
  1. Figure 1 is AI-generated: The main overview figure is disclosed as an AI-generated illustration, and its caption distances it from the paper by stating that it is "for explanatory purposes only." This suggests the figure reflects an AI model's interpretation of the method rather than the authors' own description of it. That is not ideal for a teaser figure. The content shows attention projections being decomposed, while the text only describes compressing MLP projections, and it is not clear whether attention is compressed. It also shows a different gauge (A⊤A=IA^\top A = I) from the one actually used. The authors should replace it with a figure they have produced and verified against the implemented method.

  2. The compression setting is unclear: The paper does not clearly state which modules are compressed or what the 0.95 stored-parameter fraction is measured over. If only the MLPs are compressed, the total memory saving is below 5%. It is also unclear whether the SVD, ASVD, and SVD-LLM baselines compress the same modules as LatentLLM.

  3. The baseline comparison is not fair: The paper itself notes (Sec. 3, Eq. 3) that gauge fixing lets LatentLLM retain a larger rank than the baselines under the same memory budget. The large gap in Table 1 may therefore come from the extra rank rather than from a better decomposition. An SVD-LLM + gauge baseline is needed to separate the two.

  4. The evaluation is limited and the gains are small: Only Qwen3 models up to 4B and three tasks are tested. The best GSM8K gain is about 2.7 points, ARC-Challenge does not improve, and only one of twelve GSM8K compression settings shows a significant improvement. Simple baselines are missing, such as applying the same adaptation to the dense model or adapting once on the few-shot demonstrations.

  5. The cost is underreported: Adaptation adds about 1 to 6 seconds per request, which is large for models of this size. End-to-end latency, generation time, and peak memory are not reported, even though the method is motivated by deployment efficiency.

Questions for authors
  1. Compression scope: Which modules are compressed for each method, including the SVD, ASVD, and SVD-LLM baselines? Does the 0.95 stored-parameter fraction refer to the MLP projections or the whole model, and what is the resulting whole-model memory saving?

  2. Retained rank: What rank does each method retain at the 0.95 budget? Could the authors add an SVD-LLM + gauge baseline to show how much of the Table 1 gap comes from the extra rank rather than the decomposition itself?

  3. Missing baselines: How does the method compare with applying the same adaptation to the dense model, and with adapting once on the few-shot demonstrations and then freezing the model? Since ARC-Challenge shows a large drop in prompt loss without any accuracy gain, could the authors also report how the likelihood of the correct answers changes after adaptation?

  4. Cost: What is the end-to-end latency per request, including generation, compared with the fixed compressed model and the dense model? What is the peak memory during the per-request backward pass?

1 comment

Thank you for the detailed assessment of compression fairness, methodological clarity, and deployment cost. The additional SmolLM2 experiments directly separate offline rank/storage effects, online request conditioning, and dense-model adaptation.

Method diagram and compression scope. We will replace Figure 1 with an author-verified vector schematic showing only the compressed MLP gate_proj, up_proj, and down_proj modules used in the principal experiments. It will show the B-side block-identity gauge, updates of its fully stored A factor, masked observed-token loss, hard predictive-KL bisection, generation, and checkpoint restoration.

Every method in Table 1 compresses the same selected MLP projections. The stored-parameter fraction is a budget over those target weight matrices, with unchanged embedding, attention, normalization, and other weights counted in whole-model accounting.

The original code's integer rank-budget formulas provide the following logical floating-point coefficient accounting at a 0.95 targeted-projection budget (excluding discrete pivot metadata):

Original modelFull-model parametersTarget MLP parametersNative rankGauge rankWhole-model coefficient saving
Qwen3-0.6B596,049,920264,241,1527299492.24%
Qwen3-1.7B1,720,574,9761,056,964,6081,4591,8993.09%

The ranks apply to each targeted gate/up/down matrix, which has the same pair of dimensions within each model. Native SVD, ASVD, and SVD-LLM use unrestricted factor-pair storage; the LatentLLM deployment uses the block-identity chart. The larger retained rank is a substantial source of its fixed-model advantage at the same target budget.

Matched SVD-LLM whitening/gauge control. We evaluated the whitening/truncation component of SVD-LLM on SmolLM2-135M, using 4,096 frozen C4-validation calibration tokens and 12 compressed MLP blocks. The additional gauge representation changes the feasible rank under the same coefficient budget. Results on the same 24 WikiText test windows were:

Target projection budgetNative rank / NLL ↓B-gauge rank / NLL ↓
0.80335 / 4.2063417 / 3.6161
0.95397 / 3.7506531 / 3.2221

At equal rank, native and gauged factors agree to approximately 10⁻⁵ NLL per window. The equal-budget gains therefore arise from the additional rank supported by gauge storage.

Under the repository's per-projection objective and matched covariance/ridge, the whitening/truncation operator coincides with the activation-aware rank-constrained solution. This ablation isolates the storage mechanism; SVD-LLM's separate post-truncation compensation stage remains a distinct component of the full published method. We will revise the interpretation of Table 1 accordingly.

Additional adaptation baselines. On eight matched SmolLM2 requests, a one-step dense MLP update lowered NLL from 2.87284 to 2.86058 (8/8 accepted, mean KL 0.02384). In a 12-request compressed-model source ablation, adapting once to a shared prompt prefix gave 3.56498 NLL, while adaptation to the current query achieved 3.54029; adaptation to an unrelated query gave 3.55501. The current-query comparison provides evidence for request-specific information beyond a reusable update.

Task behavior and cost. The submitted C4→WikiText sweep improves held-out perplexity at all 12 model/budget combinations, while ARC exact-match remains close to its compressed baseline (66.21% fixed; 66.13% top-k). The ARC paired flips are 19 wrong-to-correct and 20 correct-to-wrong (p = 1.0), illustrating the mismatch between prompt fitting and answer correctness. Correct-choice likelihood and answer-margin diagnostics would complement these exact-match results.

The SmolLM2 48-request experiment measures 2.53 s/request for top-6 versus 2.93 s/request for all 12 blocks on CPU, and the original A40 Table 3 measures 2.73 versus 3.72 s/request. These are adaptation costs. We will report them explicitly as incremental overhead and keep checkpoint storage, temporary factor state, full peak backward memory, and complete generation latency as distinct quantities. The present measurements do not provide an A40 peak-memory or full serving-latency number.

The central empirical claim is accordingly sharpened: the deployed low-rank factors support controlled request-local adaptation, with its benefits governed by the offline representation, update coordinates, and alignment between the observed-token loss and downstream metric.

Reviewer_ZChsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies one-step adaptation of already compressed language-model factors using observed prompt tokens, gradient-based block selection, and a KL-constrained update. Its contribution is an adaptation procedure and investigation of update coordinates, building on LatentLLM’s existing block-identity representation. Across Qwen3 models, the authors report consistent WikiText likelihood improvements, gains in selected GSM8K settings, and no significant ARC-Challenge improvement.

Critical strengths
  • The likelihood results provide consistent evidence that compressed factors remain usefully adaptable. With a frozen configuration and C4 calibration, adaptation improves held-out WikiText perplexity at all twelve model/compression settings in Table 12, by 7.0–17.1% for all-block updates. This is the strongest evidence for the method’s basic premise. The ARC results appropriately delimit that finding: reduced prompt loss does not reliably improve answer accuracy.
  • The coordinate analysis yields a concrete implementation insight. Equations 32–34 explain how factor choice changes the operator update. Table 6 also demonstrates the predicted runtime reversal: updating the fully stored factor avoids regauging under the corresponding identity-block representation. This connects the mathematical parameterization to engineering choices.
Critical weaknesses
  • The benefit of request-specific selection is insufficiently isolated. Table 3 compares top-k with all-block adaptation, without a matched fixed or random half-block control. Its 1.29-point accuracy advantage also lacks a paired statistical comparison. Furthermore, GSM8K adaptation uses demonstration answers alongside query tokens (Eq. 9), without a demonstration-only control. Consequently, the evidence does not establish how much improvement requires request-dependent support or learning from the current query. These controls matter because they distinguish the proposed adaptive choices from simpler ways of updating the same compressed model.
  • The deployment benefit remains incompletely measured. Stored-factor counts omit activations, gradients, and the state needed to restore the checkpoint. No peak adaptation-memory measurements are reported, and the denominator of “stored-parameter fraction” needs a precise whole-model accounting. Table 3’s approximately 27% timing reduction concerns adaptation alone relative to all-block TTA; top-k still adds 2.73 seconds per request. Without peak memory and total request latency, I cannot assess whether the demonstrated gains justify the cost in the memory-constrained setting motivating the paper.
Questions for authors
  • Does the stored-parameter fraction cover the entire model or only compressed projections? What are the corresponding parameter counts and, if measured, peak adaptation memory at fraction 0.95?
  • Can you provide the paired all-block/top-k prediction counts underlying Table 3 and specify the GSM8K demonstrations and KL token mask?
1 comment

Thank you for identifying the cross-budget WikiText recovery and the gauge/factor runtime reversal as the central findings. We investigated the two questions you raised: the incremental value of request-dependent selection, and the information used by request-local adaptation.

Matched support and paired predictions. We recovered the paired prediction counts behind Table 3. With all-block adaptation as the reference, gradient top-k changes 80 wrong predictions to correct and 63 correct predictions to wrong on the 1,319 GSM8K problems. The net gain is 17 predictions (+1.29 percentage points), with exact paired McNemar p = 0.1807. We will describe this difference as descriptive, and retain the fixed-versus-top-k significance results separately.

We also ran a precommitted SmolLM2-135M control with 12 C4-calibrated compressed MLP blocks, a hard KL limit of 0.025, and 48 stratified WikiText-validation requests (3,072 held-out continuation tokens). Each request was evaluated with the same six-block budget under gradient selection, fixed first/last halves, and five independently seeded random selections. All 432 updates were accepted; realized KL ranged from 0.02342 to 0.02385.

SmolLM2 policyHeld-out NLL ↓TTA s/request
Fixed compressed3.66517—
All 12 blocks3.649312.93
Gradient top-63.650912.53
Fixed first-63.650992.59
Fixed last-63.665312.37
Random-6, mean of five seeds3.65383~2.42

The predeclared top-k-minus-random paired NLL difference is −0.00292 (descriptive 95% bootstrap interval [−0.00617, +0.00028]; 29/48 requests favor top-k). Gradient selection tracks all-block likelihood with fewer active updates, while its advantage over equal-size alternatives remains inconclusive. We will present top-k as a practical update-allocation rule and separate its cost effect from the effect of gradient ranking.

Current-query information. The GSM8K protocol uses eight train-set demonstrations sampled with seed 0, shared across requests. The adaptation mask includes demonstration-answer tokens and the observed query tokens; the held-out answer never enters the loss. To isolate request conditioning at low cost, we evaluated a separate SmolLM2 causal-LM control on 12 paired WikiText test requests. Current-query-only top-6 adaptation reached 3.54029 NLL, versus 3.56498 for a shared-prefix update learned once and 3.55501 for an unrelated validation-query update. Current-query adaptation was better on 11/12 and 9/12 requests, respectively. This language-modeling analogue supports the value of adapting to the current observation.

KL and deployment accounting. The manuscript implementation measures forward predictive KL from the pre-update compressed model on up to 64 approximately evenly spaced causal target positions outside the request-loss mask, taken from the selected prompt window. KL is averaged over those positions within each request and then over requests. The compared Table 3 policies use the same prompt mask and KL positions per request.

The 0.95 stored fraction applies to compressed gate/up/down MLP projections, with all other model weights retained. Architecture-based whole-model floating-point savings at that budget are approximately 2.24% for Qwen3-0.6B and 3.09% for Qwen3-1.7B, before pivot metadata. In the SmolLM2 0.80-budget follow-up, we measured a 4.71% theoretical BF16 whole-checkpoint reduction including 32-bit pivot indices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.