acceptodds
Withdrawn from ICLR 2027

When Can Prefixes Compile LoRA? Exact Resource-Capped Tests for Frozen Attention

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Can a fixed continuous prefix replace a given low-rank adapter while the attention head stays frozen? In this research, we show that the answer depends on the adapter's target through three conditions. First, observability: at one causal readout, every independent key–value prefix sees the content only through the query, attention partition, and value numerator, so a target that differs on two inputs with equal summaries incurs an error floor at every prefix length; norm caps extend this floor to nearly equal summaries. Second, realizability: at a common query, any prefix reduces exactly to two aggregate variables, and the norm-capped optimum is an attained second-order-cone program, also after a fixed output projection; it places two equal-norm rank-one value updates on opposite sides of compilability. Third, implementation: under affine query exposure, signed slots approximate a rank- value update, but their values grow as , and the construction passes all 400 tolerance checks in float64 yet only 38 in bfloat16. A first-layer GPT-2 readout with fixed token and position meets the common-query condition without clamping activations; at three such heads, the capped optimum leaves 18.4% to 74.2% of the projected adapter effect uncompiled, with a head-dependent value–query ordering. All claims concern local approximation at one head, not whole-network equivalence.

Primary area:
learning theory
Keywords:
query-projection LoRA, parameter-efficient fine-tuning, large language models, learning theory

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_QtMROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether a fixed attention prefix can reproduce the effect of a LoRA adapter in a frozen attention head. The authors propose conditions for when such a replacement is possible, derive theoretical bounds on the approximation error, and present numerical evaluations of their results.

Critical strengths

please see above.

Critical weaknesses

The paper is extremely unclear and poorly written, to the point that it reads like an LLM-generated manuscript without sufficient human revision. The problem is not properly motivated or clearly defined at the beginning, and the abstract and introduction are filled with unexplained terminology such as "signed slots," "tolerance checks," "realizability," "common-query condition," "near-equal summaries," and "clamping activations." The significance of the (\epsilon^{-3/2}) scaling is also not adequately explained. Even the central notion of "compilation" is not introduced clearly. The related work is similarly difficult to follow and does not adequately explain the connection to existing results.

More concerningly, Appendix J describes the GPT-2 numerical results as assumed scenarios rather than executed measurements, despite their presentation as empirical findings in the main text. Given these substantial issues with clarity and the presentation of evidence, I recommend rejecting the paper in its current form and encourage the authors to substantially revise it.

Questions for authors

please see above.

Reviewer_8nmgOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I might be an outlier, but I had an extremely hard time reading this paper. Each sentence was a separate pain. Consequently, I don't think I actually understand what this paper is trying to do. As far as I can see, the authors are asking whether one can replace LoRA with a fixed prefix given an input domain, and are coming up with some conditions when this is possible.

Critical strengths

I don't understand the theoretical contributions of this paper, but it is nice to see that there is an empirical evaluation section.

Critical weaknesses

The paper is generally very badly written, therefore impossible to understand. The introduction is especially bad. Take even the first page: Ignoring the text that doesn't really follow a straightforward logic, at the end of the page we have the first contribution. This comes with an equation \Sigma = (q, Z, N). None of these variables are defined yet. What should a reader take away from this? The second contribution is titled "Exact capped realizability." Pretty mysterious naming. Let's see if the text helps. The next sentence is: "At a common query, every prefix length reduces to two aggregate variables, and norm-capped feasibility is an SOCP with an exact reconstruction, also after a fixed output projection; a proved slack covers unequal queries." First, I don't know what SOCP stands for, it hasn't been defined. I can't make sense of what this sentence is trying to say. The whole paper goes on and on with sentences like this.

To be fair, it does get slightly better after the introduction, but not to a level that one would expect from a publishable research paper.

I think this is a critical weakness that warrants rejection.

Questions for authors
  1. What is the point of Section 4? Can you give a high level summary?
  2. What is the point of Section 5? Can you give a high level summary?
  3. What is the point of Section 6? Can you give a high level summary?
Reviewer_jFdiOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors study the question: can a low-rank adapter be replaced by a fixed continuous key-value prefix at a single attention head, with content states and positions held fixed? The authors analyze “observability,” “exact capped realizability,” and “construction and precision.” They provide theoretical obstructions to compilation and an exact characterization under norm caps when inputs share a common query. They also provide an explicit construction that uses 2r slots to approximate a rank-r value update under additional assumptions, including affine query exposure and a constant self-attention score for one-token inputs.

Critical strengths
  1. The theory appears to be correct (though I did not check carefully). I appreciate the explicit construction, and the bound on parameter magnitude as a function of approximation error tolerance.
  2. I think the question of whether lora can be replaced by prefix is an interesting one.
Critical weaknesses

I have a serious concern about the factuality of the reported GPT-2 results. This is in the territory of possible academic fraud, and is by far a bigger concern than any other weaknesses I have about the strength of results (which I list separately under questions for authors).

I initially became suspicious due to a strange phrasing of the reproducibility statement. I then asked GPT-6 to look through the paper for other similar statements, and it found a rather long list of very suspicious statements throughout the appendix.

I think the GPT-6 reponse is pretty clear and concise; for full transparency, I am pasting the un-edited (but truncated) reponse at the end (the un-truncated version is in "AI Documentation For Reviewing"). I have manually verified that none of the listed items are hallucinated.

Due to all these suspicious parts, I am recommending strong reject unless the authors can provide code that one can independently run and verify that the claimed GPT-2 results are reproducible.


my prompt:

Attached is a paper that I am reviewing. I suspect that the authors fabricated their data, but some of the llm-written sections seem to let-slip this fact (e.g. ". The accompanying numerical scenario file fixes every assumed adapter effect and approximation error and checks their normalizations and ordering. These inputs are not extracted pretrained measurements")

I need you to carefully review the text and data and list the evidence for and against my suspicion.


GPT-6-Astra-max response

Your concern is strongly supported for the claimed GPT-2 experiments. The paper explicitly describes those numbers as assumed and the evaluation as unexecuted, while presenting the same numbers as empirical findings in the abstract, introduction, figures, and conclusions. That is a substantial reporting problem.

The narrower conclusion is well supported: constructed numerical scenarios are presented as measured GPT-2 results. The PDF alone does not establish that every experiment was fabricated or establish the authors’ intent.

I reviewed all 26 pages and independently checked several numerical relationships in 26965_When_Can_Prefixes_Compil(1).pdf. References below use the paper’s printed page and line numbers. I did not have the accompanying code, scenario file, checkpoints, or raw outputs.

The evidence supporting your concern is as follows.

1. There is an even more explicit admission than the sentence you identified: the GPT-2 evaluation is described as “unexecuted.”

Appendix F.5, p. 19, lines 998–1000, discusses the evaluation in Appendix J and ends:

“Its unexecuted status is distinct from the controlled results in Section 7.”

Appendix J then begins, p. 23, lines 1207–1208:

“Adapter effects, conic brackets, optimizer errors, and transfer errors are assumed; their normalizations and order relationships are calculated.”

This is unusually decisive evidence. It does not merely say that the experiment uses synthetic inputs or chosen hyperparameters. It says that the quantities being reported as experimental outcomes—including optimization and transfer errors—are assumed.

Your quotation from the reproducibility statement, p. 10, lines 490–492, independently reinforces this:

“The accompanying numerical scenario file fixes every assumed adapter effect and approximation error and checks their normalizations and ordering. These inputs are not extracted pretrained measurements.”

Together, these passages strongly support the conclusion that Appendix J reports hypothetical outcomes rather than results extracted from executed GPT-2 experiments.

2. The main text repeatedly presents these same hypothetical numbers as empirical findings.

LocationPresentation in the paperWhy it matters
Abstract, p. 1, lines 25–28At three GPT-2 heads, the capped optimum leaves 18.4%–74.2% of the adapter effect uncompiled.Presented as a finding about actual pretrained heads, without a hypothetical qualification.
Introduction, p. 2, lines 79–81The test leaves that fraction of “trained adapter effects” uncompiled.Implies that trained adapters were obtained and evaluated.
Figure 1, p. 2The GPT-2 numbers appear under “Measured”; the caption says “Measured values are from Section 7.”Explicitly labels the numbers as measurements.
§7.4, p. 9, lines 452–461“Finally, we apply the capped test…” followed by specific residuals, ten-restart comparisons, and held-out transfer errors.Describes completed experimental procedures and their outcomes.
Conclusion, p. 9, lines 479–483Value–query ordering changes across pretrained heads; the test separates resource limits from optimizer error.Draws substantive empirical conclusions from the assumed numbers.

This is therefore not an isolated ambiguous sentence. The contradiction affects a highlighted contribution and its supporting evidence.

3. The appendix says that a central empirical pattern was deliberately built into the scenarios.

Appendix J.2, p. 23, lines 1239–1241, states:

“These scenarios deliberately include both orderings of value and query targets.”

It gives the same head-0 and head-4 examples that §7.4 presents as observations, then says:

“Such a pattern would be compatible with a target-specific boundary.”

The distinction is consequential:

  • A scenario deliberately chosen to exhibit both orderings illustrates what the theory permits.
  • An experiment finding both orderings would provide evidence that this happens in pretrained models.

The main text treats the former as the latter. The claimed head-dependent ordering is therefore not independently demonstrated by these tables.

4. The numerical tables contain repeated arithmetic patterns consistent with constructed scenarios.

I recalculated the following differences from the printed values:

DataRepeating pattern
Table 8, p. 25: normalized best-learned error minus conic upper endpoint0.025, 0.030, 0.035, 0.040, repeated for each of the three heads, to displayed precision.
Tables 7 and 9: evaluation-oracle ratio minus fitting-domain conic ratio0.025, 0.028, 0.031, repeated four times in printed row order.
Table 9: conic-transfer ratio minus evaluation-oracle ratio0.065, 0.066, 0.067, repeated four times.
Table 9: learned-transfer ratio minus conic-transfer ratio0.047, 0.049, 0.051, repeated four times.
Table 10, p. 26: learned error minus exact-count optimum0.029, 0.034, 0.058 for slot counts 1, 4, 16, repeated across all six target/cap combinations.

For example, all 12 rows of Table 9 can be reconstructed from the fitting-domain ratios by adding three short, repeating sequences.

Such regularity alone would not prove fabrication. Here, however, it corroborates the explicit description of a scenario file that fixes errors and checks their ordering. In particular, the small optimization and transfer gaps highlighted in §7.4 appear to be features of the assumed scenarios, not established properties of actual runs.

5. The apparent solver certificates are explicitly disclaimed.

Table 7, p. 24, lines 1243–1245, presents lower and upper conic endpoints separated by at most 10−610^{-6}. This gives the appearance of tightly resolved numerical optima.

But Appendix J.5, p. 25, lines 1331–1333, says that a lower endpoint requires a checked infeasibility witness and then states:

“The intervals in Table 7 are not such witnesses.”

Appendix J.2, p. 23, lines 1236–1238, also says:

“A physical reconstruction attaining the upper endpoint would be obtained from Equation (15), but its pretrained attention evaluation is not asserted by the assumed table.”

Thus, the paper does not establish either that these brackets certify the actual GPT-2 optimization problems or that the corresponding prefixes were reconstructed and evaluated.

The mathematical theorem can justify a procedure for computing an optimum. It cannot establish that an assumed number is the optimum for a particular pretrained model.

6. Important provenance for the supposed trained-adapter experiment is missing from the PDF.

Appendix J gives experimental-looking details: heads, ranks, training seeds, context lengths, ten restarts, 2,000 optimization steps, and a learning rate. However, I could not find:

  • An identifiable trained-adapter checkpoint or the adapter training objective and dataset.
  • The actual fitting/evaluation contexts or their source.
  • An explanation of how the three stated training seeds produce the single reported row per head/target/rank.
  • Actual extracted summaries, solver residuals, or reconstructed-prefix outputs.

These omissions alone would be a reproducibility weakness, not proof of fabrication. In combination with “unexecuted” and “assumed,” they reinforce the reading that this is a proposed experimental design populated with illustrative outcomes.

There is also a minor cross-table inconsistency. For head 0, value rank 4, Table 8’s learned error normalized by Table 7’s adapter effect is approximately 0.437, whereas Table 10 reports 0.441 at the matching caps and four slots. For head 4, query rank 4, the corresponding values are 0.359 and 0.353. Separate optimization runs could explain this, so I would treat it as a clarification issue rather than central evidence.



my follow-up prompt to clarify the patterns in tables 7 and 9:

can you double check your claim about numerical tables. I cannot verify the four-time repetition in tables 7 and 9


GPT-6-Astra-max response

I rechecked it: the arithmetic is correct, but I should have made the comparison clearer. The repetition is in the differences, taken in printed row order. The three-row cycle crosses the boundaries between heads.

Specifically, subtract:

  • AA: Table 7’s last column, Ehi/DGE_{\mathrm{hi}}/D^G, on p. 24.
  • BB: Table 9’s “Evaluation oracle” column, on p. 25.

Here is every subtraction from the PDF: 26965_When_Can_Prefixes_Compil(1).pdf

RowHeadTarget / rankTable 7: AATable 9: BBB−AB-A
10Value / 10.1840.2090.025
20Value / 40.4070.4350.028
30Query / 10.6310.6620.031
40Query / 40.5420.5670.025
54Value / 10.6830.7110.028
64Value / 40.7420.7730.031
74Query / 10.2670.2920.025
84Query / 40.3190.3470.028
98Value / 10.3880.4190.031
108Value / 40.5660.5910.025
118Query / 10.4510.4790.028
128Query / 40.4870.5180.031

Thus, rows 1–3, 4–6, 7–9, and 10–12 each give the sequence (0.025, 0.028, 0.031). It does not restart at each head.

The other two patterns occur entirely within Table 9:

RowsConic transfer − Evaluation oracleLearned transfer − Conic transfer
1, 4, 7, 100.0650.047
2, 5, 8, 110.0660.049
3, 6, 9, 120.0670.051

For example, its first row gives 0.274−0.209=0.0650.274-0.209=0.065 and 0.321−0.274=0.0470.321-0.274=0.047; its fourth row gives 0.632−0.567=0.0650.632-0.567=0.065 and 0.679−0.632=0.0470.679-0.632=0.047.

Two qualifications matter. These equalities hold at the displayed three-decimal precision. Also, Table 7 and Table 9 normalize by their respective fitting- and evaluation-domain adapter effects, so the first comparison concerns normalized ratios, not an additive increase in raw error. The repeating pattern is real, but the explicit statements that the outcomes are “assumed” and the evaluation “unexecuted” remain much stronger evidence about their provenance.

Questions for authors

I list below lesser weaknesses/questions. Please prioritize addressing my concern about the GPT-2 results.

  1. What is the advantage of using a key-value prefix, vs using lora? Is there a practical advantage?
  2. The authors state that “Throughout, content states and their positions stay fixed. This interface contains every key-value pair that coupled soft tokens can realize at the head, but it excludes prefix-induced changes to earlier layers and positional shifts.” In an actual transformer, prefixes in earlier layers generally change the content representations entering later heads. These interactions are unaccounted for in their analysis.
  3. The analysis focuses on local approximation at a single attention head, which is quite far from an actual transformer (the authors do clearly acknowledge this).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.