acceptodds
Under review as a conference paper at ICLR 2027

The Cost of Eigenspace Restriction in Shared Attention Control

Abstract

Soft prompts and prefixes control a frozen Transformer through learned records: a record's key decides when attention retrieves it, and its value decides what it returns. Confining records to a low-dimensional subspace cuts their parameters, and the value map's highest-gain eigenspace is the natural spectral choice. We show that this choice can require exponentially more records. In a two-head scalar attention interface with fixed projections and readout, we count the equal-weight records needed to approximate a growing-amplitude target m tanh q, equivalently a fixed target under growing attention gain. At every fixed relative tolerance we determine the optimal exponential count rate exactly, both for unrestricted records and for the eigenspace restriction, and the restricted rate is strictly larger below an explicit tolerance. The converse is an entropy-area bound: equal-weight records pay for how far attention moves from uniform. Because the restriction ties every value to its key, attention must keep moving across the query interval, whereas unrestricted records can saturate retrieval early and let a palette of values supply the rest; counted constructions attain both rates. The loss persists for thin full-dimensional strips around the eigenspace, nearby fixed-key value maps and, under a disk-margin condition, an RMSNorm realization, and computer-certified explicit libraries exhibit it at moderate amplitude in four parameter settings. In frozen RoBERTa-base, one unrestricted shared key/value record gives lower SST-2 held-out loss than up to 256 subspace-confined records, and the strongest confined arms stay above it with a fivefold training budget; on AG News, at the same block, the differences are at most 0.021 nats. In the scalar interface, finite optimization favors unrestricted records at every tested count under uniform queries, with reversals under some query distributions.

Primary area:
learning theory
Keywords:
attention, shared key-value representations, approximation rates, expressivity, RMSNorm, resource complexity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_Vc6UOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how restricting shared attention records to the highest-gain eigenspace of a fixed value map affects the number of records needed to approximate a target response. In a two-head scalar attention model with equal-weight records, the authors derive exact asymptotic record-count rates for approximating mtanh⁡qm\tanh q. They show that, below an explicit error threshold, the eigenspace restriction incurs an exponentially larger record count than unrestricted records. An entropy–area argument explains the gap: restricted records tightly couple retrieval keys and returned values, while unrestricted records can vary values among records with similar keys. The paper also establishes robustness extensions and certified finite-amplitude separations. Experiments with frozen RoBERTa provide task-dependent empirical parallels, but do not directly validate the theoretical exponential rates.

Critical strengths
  • An interpretable mechanism. The entropy–area argument connects record complexity to the evolution of attention’s retrieved keys. It explains why maximizing value gain can be insufficient when the restriction couples retrieval and returned values.

  • Robustness beyond exact confinement. The analysis extends to thin full-dimensional strips, nearby fixed-key value maps, and an RMSNorm realization under an explicit condition. These results strengthen the argument that the separation is not solely an artifact of exact one-dimensional support.

Critical weaknesses
  • Limited motivation for the chosen restriction. The highest-gain value eigenspace is mathematically natural, but the paper does not sufficiently establish its relevance to existing prompt-compression methods. This makes the practical significance of the precise separation difficult to assess.

  • Insufficient connection between theory and experiments. The RoBERTa comparisons change both subspace geometry and trainable dimensionality, while operating outside the theorem’s bias-free setting. They demonstrate performance differences but do not isolate the proposed key–value coupling mechanism or validate the exponential rates.

  • Difficult presentation. Numerous technical conventions and qualifications appear before the central mechanism becomes clear. A simple worked example showing how eigenspace confinement obstructs value adjustment at nearly fixed keys would make the contribution substantially easier to understand.

  • Limited actionable guidance. The paper suggests evaluating subspaces through their freedom to vary values at similar keys, but does not provide a practical metric or selection procedure. Consequently, the implications for designing better prompt spaces remain largely qualitative.

Questions for authors

see weaknesses.

Reviewer_zMCxOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how many equal-weight records are needed to control an attention response, comparing unrestricted records with records confined to the value map’s top eigenspace. It derives the optimal exponential rates for both families at every fixed tolerance δ > 0 (Theorems 1–3) and shows that the restricted rate is strictly larger whenever δ < tanh(Q)/2. This separation is proved by an entropy–area lower bound that holds for any library, matched by explicit constructions, and is confirmed at finite amplitude by computer-verified libraries. The gap persists under thin strips, small attention-weighted deviations, perturbed value matrices, anchor removal, and RMSNorm, and it is examined empirically in frozen RoBERTa-base and in finite-budget optimization of the scalar model, with all unfavorable results reported.

Critical strengths

The paper gives an exact characterization. For both families and every fixed tolerance, the paper proves matching lower and upper bounds on the growth rate of the record count, and the entropy–area argument explains why tying values to keys raises this rate.

Critical weaknesses
  • The theory attributes the loss to tying values to keys, but the RoBERTa restricted arms only reduce the dimension and do not impose this tie.
  • The practical relevance of the separation is unclear. The result depends on a stylized, bias-free interface and a restricted family that is not an existing method.
Questions for authors

The same as weakness

Reviewer_PzewOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies a minimal two-head, scalar-query attention interface with fixed maps and equal-weight records. It derives exact asymptotic rates for the number of records needed to approximate a target response m⋅tanh⁡qm\cdot\tanh q to relative tolerance δ>0\delta > 0. Unrestricted records attain the rate J!([t−(A+1)δA]+)J!\left(\left[\frac{t-(A+1)\delta}{A}\right]+\right), whereas records restricted to the principal eigenspace of the value map (y=−xy=-x) require the strictly larger rate 1AJ!([t−2δ]+)\frac{1}{A}J!\left([t-2\delta]+\right).

The gap is explained by an entropy-area identity: the restriction forces a longer path in key space and therefore higher relative entropy. Matching constructions, robustness results (thin strips, nearby maps, RMSNorm), computer-certified finite libraries, and limited experiments on frozen RoBERTa-base and a scalar optimizer are provided.

Critical strengths
  1. Exact matching upper and lower rates at every positive tolerance, together with a clean geometric mechanism (entropy-area budget). This is rare and consequential: most resource analyses of attention give only one-sided or asymptotic bounds.

  2. Unusually thorough validation pipeline for a theory paper: computer-certified finite libraries that exhibit concrete separation at moderate amplitudes, plus candid reporting of adverse empirical cells.

  3. Transparent and disciplined scope. The authors consistently emphasize the idealized nature of the interface, maintain a detailed assumption ledger, and openly document cases where the empirical gap shrinks or reverses. This intellectual honesty strengthens the credibility of the theoretical claims.

Critical weaknesses
  1. The exponential separation appears to be an artifact of the bias-free, equal-weight interface. Once per-record logit biases (or free integer masses) are allowed, the restricted family can also realize the target with only O(m)O(m) atoms, collapsing the gap. Real soft prompts and PEFT methods routinely use such biases, so the practical relevance of the result is unclear.

  2. Extreme narrowness of the model and weak empirical link. The interface removes almost every degree of freedom present in real attention (multi-head interactions, low-rank key maps, free biases, residual connections, multi-token prompts). The RoBERTa experiments under-train the restricted arms, share a single learning rate, and show mixed or vanishing gaps on accuracy; they do not isolate the geometric cost claimed by the theory.

  3. Overstated robustness and limited finite-scale impact. The continuity results (thin strips, nearby maps) hold only for very small perturbation radii, and the certified libraries that exhibit separation require extremely large record counts (up to 101110^{11}) or still show only modest rate ratios at moderate amplitudes. This leaves open whether the identified cost is practically meaningful even inside the idealized model.

Questions for authors
  1. Does the weighted-record (or free-bias) version of the restricted family also achieve only O(m)O(m) atoms? If so, in what regime is the exponential separation relevant to soft prompts or PEFT?

  2. Can the RoBERTa advantage be shown with per-arm tuned learning rates, fully converged training, and accuracy (rather than cross-entropy) as the primary metric?

  3. What rates do other one-dimensional restrictions (e.g., y=xy=x or rotated lines) produce in the toy model? Is the top eigenspace special? Can a multi-head or low-rank-key variant of the theorem be stated, or a concrete real-model setting exhibited where the same key–value coupling is present and costly?

  4. How sensitive are the asymptotic rates and the finite certified separations to the precise choice of palette offsets and the number of sign/exchange orbits? Is there a systematic way to keep the orbit count small while still matching the unrestricted rate? Is there a critical transverse width at which a thin full-dimensional strip recovers the unrestricted rate, or does any positive width still incur a strict asymptotic penalty?

  5. How sensitive are the asymptotic rates and the finite certified separations to the precise choice of palette offsets and the number of sign/exchange orbits? Is there a systematic way to keep the orbit count small while still matching the unrestricted rate?

  6. Is there a critical transverse width at which a thin full-dimensional strip recovers the unrestricted rate, or does any positive width still incur a strict asymptotic penalty?

Reviewer_PESKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the expressivity cost of restricting learned attention records to the highest-gain eigenspace of a value map. In a simplified two-head scalar attention setting, it establishes conditions under which this restriction requires exponentially more records than an unrestricted construction to achieve the same approximation tolerance. The theoretical analysis is supplemented by synthetic experiments and evaluations with frozen RoBERTa-base. However, the motivation for the particular spectral restriction, the presentation of the central argument, and the broader relevance of the analysis are insufficiently established.

Critical strengths

I did not identify substantial strengths in the current presentation.

Critical weaknesses
  1. The abstract is overly dense and does not communicate a clear central message.

The abstract introduces numerous specialized concepts, technical qualifications, extensions, and numerical findings without first establishing the practical question that motivates the work. The discussion moves from exact exponential count rates to entropy-area bounds, robustness conditions, normalization, certified constructions, and downstream evaluations. Consequently, it is difficult to distinguish the main contribution from supporting technical results.

The abstract should first explain which design decision is being examined and why it matters, then state the principal finding and its scope. Technical details should support that message rather than obscure it. The current presentation leaves readers uncertain about whether the primary contribution is a theoretical counterexample, a critique of a particular spectral heuristic, or a practical recommendation for prompt and prefix design.

  1. The claim that the highest-gain eigenspace is a “natural spectral choice” is insufficiently motivated.

This premise is central to the paper, but it is not self-evident. The authors should identify prior methods that actually adopt this restriction, or provide a clear objective under which it is a justified choice. Without such motivation, it is unclear whether the paper examines an established design practice or a particular constraint introduced primarily to construct the separation.

More fundamentally, maximizing spectral gain or preserving Euclidean magnitude is not automatically equivalent to preserving task-relevant information. The paper needs to explain why this criterion should be expected to identify an effective subspace for attention records, rather than assuming that a high-gain direction is necessarily useful for adaptation.

The scope of the conclusion should also be explicit: demonstrating a limitation of this particular eigenspace restriction does not, by itself, establish a limitation of low-dimensional records more generally. Comparisons with alternative or task-learned subspaces would help clarify what aspect of the restriction causes the disadvantage.

  1. The introduction does not develop a coherent research narrative.

The introduction reads more like a sequence of reported findings than an argument leading from a meaningful research question to a clearly defined contribution. The relationship among soft prompts, prefix records, spectral confinement, and record-count complexity needs to be established before the paper presents its collection of theoretical and empirical results.

In particular, the reader should understand why the proposed restriction is worth studying, what existing understanding the paper challenges, and what conclusion follows from the separation. The introduction should also distinguish what is proved in the scalar interface from what is supported by the frozen-Transformer experiments. Without this structure, the individual results are difficult to interpret as parts of a unified contribution.

  1. The derivations and synthetic experiments before Section 5 do not sufficiently establish their explanatory value.

I found the analysis before Section 5 insufficiently informative about the broader problem the paper aims to address. The central result concerns a specific interface with equal-weight records, fixed projections and readout, and a growing-amplitude target or equivalent growing-gain formulation. These choices may enable a sharp mathematical characterization, but the paper needs to explain which assumptions drive the separation and why the resulting mechanism is relevant to the intended applications.

A simplified counterexample can be valuable, but its significance depends on what it teaches beyond the constructed setting. The paper should make clearer what practical expectation is overturned, which design choices should change as a result, and whether the synthetic experiments test predictions beyond illustrating the theoretical construction. My concern is not that the derivations are necessarily incorrect, but that their conceptual and practical implications remain underdeveloped.

Questions for authors
  1. What motivates the specific eigenspace restriction?

Which existing methods or clearly defined optimization objectives support treating the value map’s highest-gain eigenspace as a natural choice? How should readers distinguish the limitation of this particular spectral criterion from the limitations of subspace-constrained adaptation more generally?

  1. What is the principal insight from the analysis before Section 5?

Which assumptions are essential to the exponential separation, and what concrete lesson should a practitioner or attention researcher draw from it? Can the authors demonstrate that the mechanism remains relevant under alternative, reasonably motivated subspace choices?

  1. How should the theoretical result be reconciled with the setting-dependent empirical findings?

The abstract reports a substantial separation on SST-2, much smaller differences on AG News, and reversals under some query distributions. What determines when the restriction is harmful, and what scope of practical recommendation is justified by these results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.