acceptodds
Under review as a conference paper at ICLR 2027

Multi-Query Associative Correction: Enriching the Readout Interface for History-Conditioned Low-Rank Adaptation

Abstract

Standard LoRA learns a fixed low-rank correction map, while results from explicitly partitioned adapters suggest that how a fixed adaptation budget is organized can matter. We ask whether such differentiation can instead be learned from a common training stream, without predefined subset identities. We study this through a readout-centric view of low-rank adaptation and identify an allocation–expression gap: allocating multiple or wider readouts does not by itself produce differentiated corrections. Such differentiation requires data-dependent variation to occupy the exposed coordinates and remain distinct after decoding. This analysis highlights the joint importance of data-dependent evidence exposure and differentiated decoding. Under a compact recurrent-state design, these motivate multiple conditioned reads from a shared state and group-specific decoding, while previous-state retrieval additionally motivates a direct path for current evidence. MQ-ACM instantiates this design with one shared associative state, while joint task-gradient initialization provides a coordinated starting geometry for the paired input–output coordinates. On our main Llama-3.1-8B generation setting, the default , configuration improves rank-8 LoRA by on MT-Bench, points on GSM8K, and points on HumanEval. A complementary commonsense evaluation under a distinct adaptation setting provides additional validation. Effective-rank and alignment diagnostics characterize realized correction structure; interventions support functional use of associative and query-specific decoded terms. These results support readout organization as a distinct design dimension for history-conditioned low-rank adaptation, with additional computation despite fixed state storage.

Primary area:
foundation or frontier models, including LLMs
Keywords:
parameter-efficient fine-tuning, low-rank adaptation, large language models, associative memory, multi-query readout, gradient-based initialization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QaFUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MQ-ACM, a low-rank adaptation method in which each adapted attention block keeps a recurrent r×rr \times r associative state (scalar-gated delta rule) and reads it through KK query-specific decoder blocks, widening the readout to R=KrR = Kr while state storage stays fixed at the r×rr \times r state. §3 develops an "allocation–expression" vocabulary (effective correction rank) and Proposition 1 shows tied decoders collapse the attainable span to rank ≤ r. Experiments fine-tune Llama-3.1-8B on math/code/instruction data and Llama-3-8B on commonsense: at r=8, K=4 improves rank-8 LoRA by 0.50 MT-Bench, 5.96 GSM8K, 6.30 HumanEval; inference is 3.04× slower than LoRA.

Critical strengths
  1. Unusually reproducible specification — executable pseudocode, pinned normalization/state-update/mixed-precision decisions, closed-form storage and FLOP accounting, and honest reporting of the unfavourable 3.04× latency.
  2. The paper reports its inconvenient results. The best K is 1, 8, 4 per benchmark; retraining ablations favour Neither/No-JI on two of three benchmarks; a depressive baseline run is retained and noted. The candour is worth crediting even where it damages the narrative.
  3. Real diagnostic effort with the right controls — the fixed-R factorization sweep (E.2) holds R = 32 while varying (K, r), and intervention protocols are specified down to token masks and verification that λ = 1 reproduces unmodified logits.
Critical weaknesses
  1. The mechanism the paper is named after is not the source of the reported gains. The best MT-Bench score is at K=1 (6.75 vs 6.58 at the advertised default K=4) — a configuration with no multi-query at all; on GSM8K the K=4-over-K=1 margin (0.50) is far inside seed std. Only HumanEval shows a multi-read effect, and it is non-monotonic. Two of three headline numbers are carried by the recurrent memory + direct path + joint initialization, not by the readout organization the title claims.
  2. The headline GSM8K/HumanEval gains rest on baselines the authors did not run — LoRA and LoRA-GA numbers are transposed from other papers (§5.1, C.3), exactly the two columns carrying the +5.96/+6.30 claims, with no significance testing anywhere.
  3. The closest structural precursor (δ-mem) is cited as such but never compared against, so the paper cannot partition credit between multiple reads, the recurrent-correction route, and joint initialization. The one baseline that would settle it is missing.
  4. The paper's own theory makes a sharp falsifiable prediction and the paper declines to test it. Proposition 1 predicts tied decoders forfeit the benefit; the clean tied-decoder retrain at matched parameters is deferred in E.5 — a substantive gap, not a scope note, since decoder sharing is the thesis.
Questions for authors
  1. The mechanism in the title is not the source of the gains. Please report a decomposition table at matched parameter count and identical data/recipe: (a) rank-8 LoRA → (b) the δ-mem-style recurrent correction alone (single read, no direct path, no joint init) → (c) + direct path → (d) + joint init → (e) + multiple reads (K=4). This is the experiment that determines whether the title is earned — §5.2's K-sweep cannot answer it, because K=1 removes the multi-read but retains everything else. Please also explain mechanistically why multiple reads help code synthesis but not math or conversation, and whether the non-monotonic HumanEval curve is a real interface effect or seed noise.
  2. The headline gains rest on baselines you did not run. Please re-run LoRA, LoRA-GA (and GoRA) in your own environment for GSM8K and HumanEval, with paired significance testing (paired bootstrap over seeds, or more than three seeds) for the +5.96/+6.30 claims. These two columns carry the central quantitative claim and are exactly the ones transposed from other papers; if the transposed numbers cannot be reproduced, please state the gap explicitly.
  3. The closest structural precursor is never compared. δ-mem is cited in your own words as "the closest structural precursor," yet appears in none of Tables 1, 2, 3. Please add the δ-mem baseline; without it, none of the gains can be attributed to multiple reads versus the δ-mem route versus joint initialization.
  4. Your own theory makes a falsifiable prediction you declined to test. Proposition 1 predicts that tying Dh,1=⋯=Dh,KD_{h,1}=\cdots=D_{h,K} collapses the attainable span to Col(Dh⋆S)\mathrm{Col}(D_h^\star S) with dimension ≤ r. Please run the tied-decoder retrain at K=4 with matched parameters and report it — a null or small result there would be highly informative. Deferring it in E.5 is a substantive gap, not a scope note, since decoder sharing is the paper's thesis.
1 comment
Reviewer_QaFUOpenReview ↗

Response to author

I thank the authors for the detailed and candid response, and for the substantial new ablations and significance analyses. While I acknowledge that MQ-ACM outperforms static LoRA at comparable parameter budgets and most published low-rank variants under the paper's own reported numbers, the requested in-house reruns of LoRA, LoRA-GA, and GoRA with paired statistics were not delivered, so the headline math and code advantages over external baselines — which rest on cross-recipe comparisons — remain unestablished (W2/Q2). The decomposition ladder still lacks the rank-8 LoRA reference row and the single-read MT-Bench entries, and the authors' own tied-decoder retrain indicates that the titled mechanism pays only on code, so I think the current evidence remains insufficient to support the abstract's broad claim and I maintain my rating。

Reviewer_6u5GOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies readout organization in low-rank adaptation. It distinguishes local width r, aggregate width R=Kr, and the effective rank of decoded corrections, arguing that allocating additional coordinates does not necessarily produce differentiated outputs. MQ-ACM maintains one shared r-by-r associative state within each adapted attention block, reads history through K input-conditioned queries, combines these reads with a direct path from the current input, and produces attention corrections through group- and projection-specific decoders. The method also uses joint task-gradient initialization and query-diversity regularization. Experiments cover dialogue, mathematics, and code generation with Llama-3.1-8B, and commonsense reasoning with Llama-3-8B, with effective-rank, gradient-coverage, and intervention analyses of model behavior.

Critical strengths

The method offers a viable way to retain a rich readout interface over a smaller associative state. Multiple queries share one state and produce corrections through distinct decoders, allowing state storage and readout width to be configured separately. In Appendix E.2, at fixed R=32, (K,r)=(4,8) uses one-sixteenth the recurrent-state storage of (1,32), while achieving GSM8K/HumanEval means of 73.74/49.39 versus 72.91/47.76. This makes a smaller state with multiple reads a worthwhile architectural option. These results support feasibility; the storage benefit concerns the additional recurrent state, while the full model retains its KV cache and the default configuration increases generation latency.

The common-differential decomposition clarifies the interaction between reads and decoders and makes the mechanism directly testable. Equation (8) separates the associative correction into a common term and a query-specific differential term. A nonzero differential term requires differences in both reads and decoders, although cancellation can still eliminate it. This identifies a concrete quantity that can be scaled to test its functional role. In Table E.5, restoring the differential coefficient from 0 to 1 reduces GSM8K/HumanEval NLL from 0.8815/0.3400 to 0.8529/0.3336. Although the algebra is elementary, turning differentiation into a computable and intervenable quantity is a useful analytical contribution.

Critical weaknesses

The trade-off between performance gains, parameter count, and inference cost remains unclear. Default MQ-ACM uses 21.10M adapter parameters, approximately 3.1 times the 6.82M of rank-8 LoRA. Table 4 also reports an increase in generation latency from 32.52 to 98.89 ms/token, approximately threefold, on an H100 with batch size 1 and prefill excluded. However, the main generation tasks lack a strong static baseline at a comparable parameter budget, making it difficult to establish the benefit of multi-query organization over simply increasing static adaptation capacity. The rank-32 comparison on commonsense tasks in Table 2 is useful complementary evidence, but its different backbone and training setup limit its relevance to the main generation tasks. Comparing task performance, parameter count, and latency under the same measurement conditions would clarify the practical gains and deployment costs of the design.

The independent benefit of multiple reads and differentiated decoding is not sufficiently established. Existing checkpoint interventions show that trained models use the query-specific differential term, but do not establish whether this structure outperforms simpler alternatives trained from scratch. The paper lacks retrained direct-only and common-read controls that preserve direct-path capacity. Furthermore, the differential-term intervention reports NLL without corresponding free-generation task scores. Targeted controls would help establish the actual contribution of multi-query readout to downstream task performance.

Questions for authors

Table 1 fixes local width r, while the default model has approximately three times the adapter parameters and generation latency of rank-8 LoRA, according to Table 4. Could you clarify the performance-cost conclusions supported by these comparisons? Are any results against a static baseline at a comparable parameter budget already available for the main generation tasks? If not, please explain which benefits the current evidence distinguishes and which may still reflect increased capacity. Please also specify whether LoRA was merged into the backbone for the latency comparison, whether the methods used the same inference backend, and what is currently known about the main sources of overhead.

Table E.5 shows that removing the query-specific differential term worsens NLL, supporting its functional use in trained models. Are corresponding free-generation scores, or retrained common-read/direct-only controls preserving direct-path capacity, already available? If not, could you clarify what the existing evidence establishes about use of the differential term in a trained model versus superiority of the multi-query architecture over simpler alternatives, and specify the intended scope of the central conclusion?

1 comment

Two issues remain:

  1. GSM8K shows only a 0.20 gap, MT-Bench was not run for common-read, and K=1 scores 6.75 against 6.58 at the default K=4, so the general multi-query claim is supported on one benchmark only.
  2. Without the retrained direct-only control, the gain still cannot be separated from static capacity plus the direct path.
Reviewer_2WBrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MQ-ACM, an adapter method that augments the fixed-size recurrent memory of Lei et al., 2026 with multiple queries which are decoded separately. This architecture is designed to address the allocation-expression gap, where LoRA is allocated with rank R but only learns corrections that vary across less than R directions.

Critical strengths
  1. The method is a simple, low-storage way to add history-dependence to an adapter.

  2. The evaluation is broad (covering multiple tasks) and transparent (showing error bars, timing, and task-dependent gains).

Critical weaknesses
  1. The central conceptual claim is not adequately supported. The allocation-expression gap largely restates the known distinction between allocated and effective rank (raised by e.g. AdaLoRA, GoRA/RaLoRA papers). In Table E.2, effective rank is uncorrelated or anti-correlated with task score, and is not calculated for LoRA or other baselines. It is also not a meaningful descriptor of a nonlinear, history-dependent adapter. It is computed from the correction covariance pooled over all tokens and sequences, which mixes variation due to the current input with variation due to the history. A static adapter can achieve a high effective rank purely from input variation, with no history dependence. In MQ-ACM, the measured correction also includes the direct path, which is itself a static rank-R map. Effective rank therefore does not capture the history-dependence the method is designed to add. Furthermore, it's not clear why LoRA is inadequate to handle history dependence -- LoRA applied to any transformer model already leverages history dependence through adapting multiple self-attention layers.

  2. The experiments don’t isolate what the method contributes. The gains over rank-8 LoRA use about 3x the parameters while the parameter-matched LoRA comparison is within noise. δ\delta-mem, which provides the memory MQ-ACM builds on, is never compared against, so the value of multiple queries and separate decoders is untested. Furthermore, decoding is about 3× slower than LoRA.

Questions for authors

In which settings should the memory path help more?

What causes the 3× decoding slowdown, given that the extra compute should be negligible compared to the backbone?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.