acceptodds
Under review as a conference paper at ICLR 2027

Beyond Rotations: AuroOFT for Expressive Quantized Orthogonal Fine-Tuning

Abstract

Quantized orthogonal fine-tuning (QOFT) enables parameter-efficient adaptation of low-bit language models by learning structured activation rotations before frozen quantized weights. However, its task-specific updates remain constrained to linear orthogonal transformations, limiting input-dependent nonlinear corrections. We introduce AuroOFT, which keeps QOFT as a stable quantization-compatible branch while attaching a zero-start gated low-rank nonlinear residual to each adapted linear layer. AuroOFT maps activations into an RMS-normalized compact latent space and uses adaptive nonlinear bases with bounded or token-dependent gating. The zero-initialized up projection makes AuroOFT functionally identical to QOFT at initialization, while orthogonality remains a branch-level stability property rather than a property of the combined nonlinear layer. Under matched data, optimization, decoding, and parser protocols, AuroOFT improves Macro-6 over matched QOFT by 1.30–2.70% on the 1.5B/3B Qwen2.5 settings, exceeds QLoRA by 6.52-10.62%, and saves 32.3-44.7% trainable parameters relative to QLoRA in representative scales. The small exam-style multiple-choice math set is treated only as a protocol-sensitivity diagnostic. Our code is available at the anonymous repository: https://anonymous.4open.science/r/AuroOFT-F3FD.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Supervised Fine-Tuning, Low-Rank Adaptation, Large Language Models, Knowledge Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4RWxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes AuroOFT, a nonlinear residual augmentation method for quantized orthogonal fine-tuning. AuroOFT adds a zero-start, gated nonlinear low-rank residual branch alongside the QOFT branch and uses an adaptive nonlinear layer (ANL) to enhance the adapter's expressive capacity. Experiments show that AuroOFT outperforms QOFT on Qwen2.5 models at the 1.5B and 3B scales.

Critical strengths
  1. The paper proposes a framework that combines QOFT's orthogonal adaptation path with a nonlinear low-rank residual. It enables additional input-dependent corrections while preserving the frozen quantized weights and the original orthogonal branch.

  2. The paper evaluates AuroOFT on mathematical reasoning benchmarks using Qwen2.5 models of different sizes. It also includes ablation studies of different ANL variants, comparisons with parameter-matched linear residuals, and multi-seed experiments in selected settings, which help assess the contribution of the nonlinear residual.

Critical weaknesses
  1. The paper extends QOFT with the nonlinear adaptation idea from AuroRA to address the proposed "orthogonal expressivity bottleneck." However, it does not provide sufficient evidence that QOFT's orthogonal linear constraints actually constitute an expressivity bottleneck in mathematical reasoning tasks.

  2. Although AuroOFT outperforms QOFT with a parameter-matched linear residual on Qwen2.5-3B Base, the linear-residual variant achieves comparable or higher accuracy than AuroOFT on on the 1.5B-it and 3B-it settings.

Questions for authors
  1. Please provide a more detailed analysis of specific mathematical reasoning problems that QOFT fails to solve. For these problems, compare the reasoning outputs of QOFT, QOFT with a linear residual, and AuroOFT to identify which errors are corrected by the nonlinear residual and explain how these findings support the proposed "orthogonal expressivity bottleneck" hypothesis.

  2. Please further analyze the conditions under which nonlinear residuals provide additional performance gains.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 4RWx — Overview and index

Response to Reviewer 4RWx

We are very grateful for your careful reading and for the two weaknesses and two questions you raised; they go to the heart of what our evidence can and cannot show. Your comments are valuable because they demand mechanism-level accounting — what changes in the three models' reasoning, and under what conditions the nonlinear residual actually helps — rather than aggregate scores. Upon receiving the reviews we organized the rerun program around exactly those demands; it is now running simultaneously on three GPU servers, and we will report updated numbers in this discussion each day until all results are complete. As of November 16 (Beijing time), three six-task evaluations have completed with fully inspectable per-item counts, seven further trainings have finished with evaluations running or queued, and below we answer each of your points with the completed results, the pending markers, and the boundaries stated plainly.

All accuracies below are percentages; differences are percentage points. Macro-6 is the equal-weight mean of the six non-SAT benchmarks. Submitted-paper values, recovered pre-submission predictions, and new rebuttal evaluations are labeled as such; new numbers trace to run manifests, seeds, and stored per-item predictions. Quoted manuscript text below is the wording we commit to implement in the revision; bracketed Pending markers identify evidence still in production, and we will resolve each one before the final submission.


Reading guide

The response is organized by the original review items. No substantive content is omitted from the complete response. Continuations preserve the evidence, tables, derivations, and proposed revision quotations in order. Proposed manuscript changes are not represented as already implemented.

  • W1 — The proposed bottleneck is not established as a failure mechanism (1 comment).
  • W2 — Linear controls sometimes match or outperform AuroOFT (1 comment).
  • Q1 — What changes in the three models' reasoning outputs? (1 comment).
  • Q2 — Under what conditions does the nonlinear residual help? (1 comment).

Please read each thread in ascending [part/total] order.

AuthorsOpenReview ↗

W1. The proposed bottleneck is not established as a failure mechanism [1/1]

W1. The proposed bottleneck is not established as a failure mechanism

Reviewer's concern. The paper does not demonstrate that QOFT's orthogonal update caused the observed reasoning errors; the expressivity-bottleneck narrative is presented without mechanism-level evidence.

Thank you for insisting on the distinction between a hypothesis and a demonstrated mechanism; we accept the criticism and will restructure the claims accordingly. The submitted evidence shows that adding a residual changes scores in various settings; it does not show that orthogonality caused specific reasoning errors. What we can support today is (i) item-level evidence that scored outcomes change in both directions, and (ii) a controlled linear-side comparison now partially complete under a matched protocol. The bottleneck phrase will be repositioned as a design hypothesis that organizes controls, and wording that presents it as a proven failure mechanism will be removed.

In terms of direct evidence, the recovered item-level records quantify outcome changes without overclaiming their nature. For all 2,226 non-SAT items we recovered paired predictions of QOFT and the old 3B-Instruct Lite model: 140 scored corrections of QOFT errors, 102 newly wrong answers, 940 items correct for both, and 1,044 incorrect for both. The effect is task-dependent (GaoKao: 30 corrections versus 14 regressions; AMC23: 2 versus 6). These counts establish that the branch both fixes and breaks individual answers; they are not Enhanced results, include no third system, and cannot distinguish better mathematics from formatting or evaluation effects.

As completed rebuttal evidence, the linear side of the mechanism question now has a full number, with its partners in flight. The ANL-linearized branch on Qwen2.5-3B-Instruct — the same residual with its nonlinear basis removed while keeping the internal projections, mixture, RMSNorm, and gate — completed training and its full six-task evaluation: 41.40 Macro-6 (AMC23 11/40 = 27.50, AQUA 99/254 = 39.00, CMATH 496/600 = 82.70, GaoKao 204/385 = 53.00, Minerva 58/272 = 21.30, OlympiadBench 168/675 = 24.90; one empty prediction and one timeout flag retained as diagnostics). Under the identical protocol, the QOFT partner's evaluation began November 15 and the Enhanced partner is at step 1,380 of 1,563 [Pending P08: both six-task scores]. In addition, the stronger form of the linear control you implicitly ask for — a plain BAx residual with no normalization and a fixed unit gate — has been trained as a separate run and its evaluation is now running [Pending P04]. Together these give, for the first time, a same-protocol reading of how much of the branch's effect survives linearization.

As a theoretical clarification, the mechanism claim must remain layer-local and conditional. The full Transformer is nonlinear; the restricted object is the task-specific update at a QOFT-adapted layer — a structured input-side rotation. A nonlinear residual can express activation-dependent corrections outside that family, but capacity is not causation: at zero initialization the residual is exactly zero, and a trained model need not exploit the added flexibility beneficially. Neither function-family inclusion nor a changed score identifies why a particular solution changed.

The boundary we will state. Until the three-way, same-item analysis is complete — full outputs, independently checked answers, fixed case-selection rules, with both corrected and newly wrong cases — the paper will not present any example or aggregate as a causal account of QOFT's errors. [Pending P03: the complete three-way item-level records; no invented examples.]

Planned revision. In the revised manuscript, we will update Sections 1 and 6 and Appendix E ("Evidence Policy"). The revised text will read:

"Orthogonal expressivity bottleneck" is used as a layer-local design hypothesis, not a demonstrated failure mechanism. Recovered paired records show 140 scored corrections and 102 new errors across 2,226 items. A completed ANL-linearized Instruct control obtains 41.40 Macro-6 with full counts under the evaluation protocol of Appendix B; the matched QOFT and Enhanced partners and a plain-BAx control are evaluated under the identical protocol. Case studies describe observed behavior of the tested systems, with both corrected and newly wrong cases represented, and do not attribute causes.

AuthorsOpenReview ↗

W2. Linear controls sometimes match or outperform AuroOFT [1/1]

W2. Linear controls sometimes match or outperform AuroOFT

Reviewer's concern. Prior linear controls are reported to match or exceed AuroOFT in several settings, so the nonlinear residual's benefit is unclear.

We thank you for this point because it is empirically correct on the submitted record, and the correct response is to test it under a matched protocol rather than to argue from aggregates. Submitted Table 17 reports identity-ANL at 37.4 versus AuroOFT 37.35 on 1.5B-Instruct and 44.2 versus 43.73 on 3B-Instruct; within the submitted 3B Base table, identity-ANL exceeds Enhanced on AQUA (35.1 vs 34.2) and CMATH (88.2 vs 87.6). These are binding observations, and the new program was designed to answer them head-on.

In terms of direct evidence, the matched linear-versus-nonlinear comparison has begun producing complete rows. The ANL-linearized control on 3B-Instruct now provides one fully counted row: 41.40 Macro-6 with all 2,226 items scored. Its same-model partners — QOFT (evaluation started November 15) and Enhanced (step 1,380 of 1,563) — will complete the same-model triple [Pending P08]. On 3B Base, the ANL-linearized run at seed 43 has completed training and entered evaluation, and its QOFT and Enhanced same-seed partners are respectively in evaluation [Pending P01] and complete (38.83 with full counts). The plain low-rank row whose task cells were missing from Table 3 has been retrained from scratch, with training completed and evaluation running, and the plain-BAx control's evaluation is also running [Pending P04]. If any linear control matches or exceeds Enhanced under this matched protocol, we will state that outcome and withdraw a basis-specific advantage claim for that setting; if Enhanced leads, the instruct counterexamples you cite remain beside the finding as boundary conditions.

As complementary evidence, capacity and seed checks address the residual explanations. The submitted rank-2-to-rank-4 step (16.74M→20.49M parameters; 45.25→45.58 Macro-6) shows a modest capacity return that does not isolate the basis; a second training seed (44) for both QOFT and Enhanced is in progress — the Enhanced side has completed training and entered evaluation, the QOFT side in training — so that a paired difference can be compared against seed variability rather than one draw [Pending P06]. This is precisely the check that aggregated submitted rows cannot provide.

As a methodological clarification, the two linear controls answer different questions and both are being run. The ANL-linearized branch changes only the basis computation, retaining RMSNorm, gating, the internal projections, and initialization — so it isolates nonlinearity while keeping the branch's other machinery. A plain BAx branch removes normalization and fixes the gate at unit initialization, asking whether bare linear capacity suffices but changing more components at once. Reporting both, under one protocol with full counts, is the honest way to answer your concern; conflating them would overstate either.

The boundary we accept. No new claim of nonlinear advantage will be made from incomplete pairs; training progress is not accuracy evidence; and the instruct counterexamples stay in the paper regardless of how the matched comparisons turn out. [Pending P01, P04, P06, P08.]

Planned revision. In the revised manuscript, we will revise Section 5.3, Table 17 in Appendix D, and Section 7. The revised text will read:

Under matched same-seed protocols, the ANL-linearized branch is compared against the complete Enhanced branch and the QOFT carrier, with a plain-BAx control and a second training seed reported alongside. Submitted settings in which linear controls matched or exceeded AuroOFT are retained as boundary conditions, and any basis-specific statement is restricted to validated contrasts with complete task-level counts.

Reviewer_QPvtOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose a modified version of quantized orthogonal fine-tuning of LLMs by adding a tiny neural network that learns more flexible corrections. The evaluation is done on 1.5-7B Qwen2.5 models on 7 finetuning tasks. It reports modest improvements on smaller models (1.5-3B), but worse results on larger ones (7B).

Critical strengths
  • AuroOFT augments QOFT with a parallel, gated nonlinear residual while keeping the quantized backbone frozen. This clean separation allows additional adaptation capacity without altering the underlying rotation parameterization.
  • Focused on Qwen2.5-3B Base, the reported three-seed experiments and unified-script rerun support the consistency of the improvement over QOFT.
Critical weaknesses
  • The zero-start and containment properties are correct but trivial.
  • Novelty is limited, and the broader motivation is unproven. QOFT provides quantized orthogonal adaptation, while AuroRA already places adaptive nonlinear transformations between low-rank projections. AuroOFT’s contribution is their integration, with normalization, gating, and variants. However this work does not provide a convincing empirical value or a clear new insight, the gains are inconsistent across scales.
Questions for authors

N/

6 comments
AuthorsOpenReview ↗

W1. Zero-start and containment are elementary properties [1/1]

Response to Reviewer QPvt

We sincerely thank you for a review that engages directly with the structural content of our method and with the consistency of its results. Both of your points are valuable to the improvement of this work: the first asks us to state precisely which properties are elementary consequences of the construction and which require evidence, and the second asks us to reconcile gains that vary across model scales. After receiving the reviews, we began controlled reruns immediately on three GPU servers running in parallel; we will update the experimental data in this discussion daily as evaluations complete. As of November 16 (Beijing time), three six-task evaluations have completed with fully inspectable counts, seven further trainings have completed with evaluations running or queued, and we report below how each result — favorable or not — bears on your concerns.

All accuracies below are percentages; differences are percentage points. Macro-6 is the equal-weight mean of the six non-SAT benchmarks. Submitted-paper values, recovered pre-submission predictions, and new rebuttal evaluations are labeled as such; every new number traces to a run manifest, seed, and stored per-item predictions. Quoted manuscript text below is the wording we commit to implement in the revision; bracketed Pending markers identify evidence still in production, and we will resolve each one before the final submission.


Reading guide

The response is organized by the original review items. No substantive content is omitted from the complete response. Continuations preserve the evidence, tables, derivations, and proposed revision quotations in order. Proposed manuscript changes are not represented as already implemented.

  • W1 — Zero-start and containment are elementary properties (1 comment).
  • W2 — Limited novelty and inconsistent gains across scales (2 comments).

Please read each thread in ascending [part/total] order.

AuthorsOpenReview ↗

W1. Zero-start and containment are elementary properties [1/1]

W1. Zero-start and containment are elementary properties

Reviewer's concern. Zero initialization of the up projection and the function-family containment of a nonlinear residual over a linear one are elementary properties, not contributions.

Thank you for this precise formulation; we accept it and will reposition these properties accordingly. Zero-start and containment are indeed elementary consequences of the construction, and presenting them as contributions overstated them. Their correct role in the paper is architectural: zero-start guarantees that training begins from the unmodified carrier function, and containment organizes which controls are meaningful. Neither property carries empirical weight by itself, and the revision will state them as design facts rather than claimed novelty.

In terms of direct evidence, what deserves the contribution slot instead is the controlled comparison program. The submitted Table 1 reports 42.55 (QOFT) versus 45.25 (AuroOFT) Macro-6 on Qwen2.5-3B Base with all six task scores improving — a useful but confounded comparison, since the branch adds approximately 3.74M parameters. The new rebuttal program replaces reliance on such confounded rows with matched contrasts: the seed-43 QOFT baseline has completed training and entered evaluation on November 13 [Pending P01: its six-task score completes the first same-seed paired comparison against the completed seed-43 Enhanced evaluation of 38.83]; the ANL-linearized control on 3B-Instruct completed evaluation at 41.40 Macro-6 (11/40, 99/254, 496/600, 204/385, 58/272, 168/675 correct; all 2,226 items scored); a second training seed (44) is in progress for both QOFT and Enhanced — the Enhanced side has completed training and entered evaluation, the QOFT side in training — so that paired differences can be checked against seed variability. These are the comparisons that can support claims; the elementary properties cannot, and the revision will say so.

As a logical point that your review makes concrete, both the favorable and the negative submitted outcomes share the same zero-start construction and containment property. The +2.70-point increment on 3B Base and the −5.63-point decrement on 7B-Instruct use the identical structural setup. Their opposite directions demonstrate directly that these properties cannot predict whether finite training will improve accuracy. This is the clearest reason to reposition them as design facts rather than as contributions: they are present in every result we report, favorable or unfavorable, and therefore cannot distinguish success from failure.

As a structural clarification, containment also has a specific, limited use. The full Transformer is nonlinear; the relevant object is the task-specific update at a QOFT-adapted layer, which is a structured input-side rotation. A nonlinear residual can represent activation-dependent corrections outside the linear-update family, but representational capacity is not performance — a trained model need not use it beneficially, and at initialization the residual is exactly zero. We will therefore use containment only to explain why nonlinear and linearized branches form a natural controlled pair, never as evidence that the nonlinear branch caused any improvement.

The boundary we will state. No structural property of the construction — zero-start, containment, or the parallel placement — will be offered as a substitute for the matched empirical contrasts above; where those contrasts remain incomplete, the revision will mark the question open rather than implied by theory. [Pending P04: the plain-BAx control's completed outputs; Pending P08: the Instruct QOFT/Enhanced partners.]

Planned revision. In the revised manuscript, we will update the Introduction contribution list and Section 4.6 ("Structural Properties"). The revised text will read:

Zero initialization and function-family containment are elementary design facts of the construction: the first guarantees an unchanged initial function, and the second motivates pairing nonlinear and linearized branches as controls. Neither is claimed as a contribution. The empirical content of the method is established by matched same-seed comparisons among the carrier, the complete residual, and its linearized variants, reported with complete task-level counts.

AuthorsOpenReview ↗

W2. Limited novelty and inconsistent gains across scales [1/2]

W2. Limited novelty and inconsistent gains across scales

Reviewer's concern. The work combines existing techniques, and reported gains are inconsistent across model scales, with 7B results below the baseline.

We appreciate that this comment holds the method to the correct standard — gains that survive across scales — and we will meet it with matched evidence rather than averages of heterogeneous records. We agree that the components are prior work (the carrier from OFTv2/QOFT; nonlinear low-rank residuals from LoRAN/AuroRA), that the submitted gains are not uniform across scales, and that the 7B results are negative. The revision positions the contribution as the construction plus a controlled characterization of when the residual helps, not as a universally improving method.

In terms of direct evidence, the submitted scale pattern is reported in full, including its unfavorable side. Submitted Macro-6 numbers are: 1.5B-Instruct 36.05→37.35 (+1.30); 1.5B Base 31.53→33.37 (+1.83); 3B-Instruct 43.35→43.73 (+0.38); 3B Base 42.55→45.25 (+2.70); 7B-Instruct 58.10→52.47 (−5.63); 7B Base 50.77→50.43 (−0.33). The task-level details strengthen your concern further: on 7B-Instruct, AQUA falls from 70.9 to 46.8, a 24.1-point decline; on 7B Base, AQUA falls from 59.4 to 44.3 even though gains on other tasks reduce the aggregate decline. On 3B-Instruct, AMC23 decreases from 45.0 to 35.0 despite the small positive Macro-6 difference. We have not established whether the 7B AQUA decline is due to changes in mathematical reasoning, output format, truncation, or another evaluation-related factor, and we will not assign a cause without the corresponding checked traces. The nonlinear branch also cannot generally be merged into the frozen weights, so fewer trainable parameters do not establish faster or cheaper inference. We will not interpret the small 3B-Instruct increment as a reliable gain, and we will present the 7B decrement as a genuine negative result discussed alongside hypotheses (capacity saturation of the frozen carrier, residual budget, and evaluation sensitivity) that our current evidence cannot adjudicate. The strength of the favorable cases must be established under matched protocols rather than assumed from the submitted rows.

As completed rebuttal runs now show, the matched program addresses exactly this consistency question. Three of its strands bear directly on scale and stability: (i) the seed-43 QOFT-versus-Enhanced Base pair — the Enhanced side is complete with fully inspectable counts (15 / 40, 119 / 254, 357 / 600, 193 / 385, 40 / 272, and 164 / 675 scored correct on the six benchmarks, 38.83 Macro-6 over 2,226 items), and the QOFT side's evaluation is running [Pending P01]; (ii) the 3B-Instruct triple — the ANL-linearized control is complete at 41.40 (11 / 40, 99 / 254, 496 / 600, 204 / 385, 58 / 272, 168 / 675), the Enhanced partner is at step 1,380 of 1,563, and the QOFT partner's evaluation began November 15 [Pending P08]; and (iii) a second training seed (44) for QOFT and Enhanced on Base — the Enhanced side has completed training and entered evaluation, the QOFT side is in training — so that whatever paired difference emerges can be checked against seed variability rather than a single draw [Pending P06]. If the paired gains shrink, vanish, or reverse at any scale under the matched protocol, the revision will report that outcome and restrict the method's claimed scope accordingly; we will not average away heterogeneity across scales, records, or seeds.

Reviewer_HJD3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes AuroOFT, which augments quantized orthogonal fine-tuning (QOFT) with a gated, low-rank nonlinear residual branch. By initializing the residual branch’s output projection to zero, AuroOFT starts from the same function as QOFT and learns input-dependent nonlinear corrections during fine-tuning. Experiments on mathematical reasoning show improvements over QOFT in the evaluated 1.5B and 3B settings, but performance declines at 7B.

Critical strengths
  1. The paper is clearly written and easy to follow. The architecture, intended contributions, and scope are clearly articulated. The authors also explicitly acknowledge important limitations, including the absence of global orthogonality guarantees for the combined layer.
Critical weaknesses
  1. Limited methodological novelty. The main contribution appears to be the integration of a QOFT backbone with an AuroRA-style nonlinear residual. The paper does not sufficiently establish what new scientific insight follows from this combination beyond architectural integration and a search over residual configurations. A clearer technical distinction from existing approaches, or a stronger analysis of why the combination works, would strengthen the contribution.

  2. Insufficient justification for retaining the orthogonal backbone. The QOFT branch retains its approximately orthogonal structure, but the combined nonlinear mapping no longer inherits the preservation properties that motivate orthogonal fine-tuning. Although the paper acknowledges this boundary, it should more directly establish why an orthogonal backbone remains beneficial after introducing the nonlinear residual. Comparisons with nonorthogonal alternatives under matched parameter and training budgets would help clarify this point.

  3. Limited direct evidence for the proposed bottleneck. The authors appropriately present the “orthogonal expressivity bottleneck” as a design hypothesis. However, the experiments primarily demonstrate that adding a nonlinear residual can improve performance in some settings; they do not directly establish that the restriction to orthogonal updates is responsible for the original reasoning failures. Diagnostics identifying which errors the nonlinear residual resolves, and whether matched linear corrections can resolve them, would strengthen the motivation.

  4. Incomplete attribution of the performance gains. Because the methods use different numbers of trainable parameters, Tables 1 and 2 alone cannot distinguish improvements due to increased capacity from those due to the proposed correction mechanism. The parameter-matched controls in Tables 3 and 17 partially address this concern, but the conclusions are setting-dependent: the linear controls match or exceed AuroOFT on the instruct models. The paper should more clearly explain when nonlinearity provides an advantage and how its contribution is separated from normalization, gating, and architectural depth.

  5. Incomplete experimental reporting. Table 3 reports the aggregate Macro-6 score for the plain low-rank control but leaves its per-benchmark results unfilled. These missing results limit a detailed comparison of the methods. The authors should complete the relevant measurements and provide the configurations and execution records needed to reproduce the central comparisons.

Minor Weaknesses

  1. Figure and table sizing should be more consistent. In particular, Figures 1 and 2 would benefit from larger, more legible labels, while Tables 6 and 10 could be more compact. Please standardize typography across figures and tables.

  2. Please check table-caption placement against the venue’s formatting guidelines. Placing captions above tables would be preferable if permitted by the required style.

Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer HJD3 — Overview and index

Response to Reviewer HJD3

We are sincerely grateful that you devoted your time to examining both the motivation of AuroOFT and the evidence required to support it. Your comments are valuable to us precisely because they separate three questions that our presentation conflated: whether adding a residual branch helps at all, whether the branch's nonlinear basis matters beyond its other components, and whether the quantized orthogonal carrier remains useful after augmentation. Upon receiving the reviews, we began the first batch of controlled reruns immediately. The complete rerun program is now executing in parallel across three GPU servers, and we will update the results in this discussion daily as six-task evaluations complete. As of November 16 (Beijing time), that program has produced three completed six-task evaluations with fully inspectable counts, and a second batch of seven completed trainings whose evaluations are running or queued. We report these results below, including the unfavorable ones, together with the comparisons that remain incomplete.

All accuracies below are percentages; differences are percentage points. Macro-6 is the equal-weight mean of the six non-SAT benchmarks. We distinguish three record types throughout: submitted-paper values, recovered pre-submission predictions, and new rebuttal evaluations. New evaluations were checked through their stored per-item predictions and integer counts; every number below can be traced to a run manifest, a seed, and an evaluation protocol. Quoted manuscript text below is the wording we commit to implement in the revision; bracketed Pending markers identify evidence still being produced, and we will resolve each one before the final submission.


Reading guide

The response is organized by the original review items. No substantive content is omitted from the complete response. Continuations preserve the evidence, tables, derivations, and proposed revision quotations in order. Proposed manuscript changes are not represented as already implemented.

  • W1 — Methodological novelty and scientific insight (2 comments).
  • W2 — Why retain the orthogonal carrier? (1 comment).
  • W3 — Evidence for the proposed expressivity bottleneck (1 comment).
  • W4 — Capacity and component attribution (1 comment).
  • W5 — Missing task results and execution records (2 comments).
  • M1 — Figure and table legibility (1 comment).
  • M2 — Table-caption placement (1 comment).

Please read each thread in ascending [part/total] order.

Comments on the paper

9 comments
AuthorsOpenReview ↗

W1. Methodological novelty and scientific insight [2/2]

W1. Methodological novelty and scientific insight — continued [2/2]

As complementary evidence, the submitted tables already bound the claim. Table 2 reports 43.42 (Lite), 43.78 (SplineNorm), 45.25 (Enhanced-r2), and 45.58 (Enhanced-r4), so the residual design matters empirically in that setting, while also changing capacity and computation. Table 17 reports identity-ANL at 37.4 versus AuroOFT at 37.35 on 1.5B-Instruct and 44.2 versus 43.73 on 3B-Instruct; those counterexamples prevent any general claim that the nonlinear basis is necessary or consistently beneficial. The new rebuttal runs are designed exactly because these submitted comparisons cannot carry the attribution question alone.

As a theoretical clarification, the methodological distinction concerns the adaptation object. OFTv2/QOFT develops input-side rotations compatible with frozen quantized weights; LoRAN and AuroRA provide prior nonlinear low-rank constructions. AuroOFT keeps the QOFT computation as one path and places a compact, zero-initialized correction in a separate path, so the carrier interface is preserved, the initial function is unchanged, and the residual family can be varied without replacing the carrier. These are design requirements the construction satisfies; they do not by themselves establish a performance advantage, which is why the paired reruns above are the evidence that matters.

The boundary we will state explicitly. If the completed matched comparison shows the nonlinear basis does not account for the increment in a given setting, we will remove the basis-specific reading of that increment; the instruct counterexamples above are already treated as binding constraints rather than exceptions.

Planned revision. In the revised manuscript, we will rewrite the relevant parts of Sections 1–2 and 6 and adjust the interpretation of Tables 1–3 and 17 accordingly. The revised text will read:

AuroOFT investigates a nonlinear correction beside a quantized orthogonal adaptation path, building on OFTv2/QOFT for the carrier and prior nonlinear low-rank adaptation for the residual idea. The submitted 3B Base comparison reports 42.55 Macro-6 for QOFT and 45.25 for AuroOFT while adding approximately 3.74M parameters, so it does not isolate nonlinear bases. Instruct controls and 7B results preclude a universal advantage claim. Matched same-seed comparisons with controlled branch variants are reported separately, and the contribution is evaluated through those conditional comparisons rather than by treating architectural integration or zero initialization as a new general principle.

AuthorsOpenReview ↗

W2. Why retain the orthogonal carrier? [1/1]

W2. Why retain the orthogonal carrier?

Reviewer's concern. The complete AuroOFT layer is not orthogonal. Why retain QOFT instead of using a nonorthogonal carrier with a comparable parameter budget?

We appreciate this question because it cleanly separates a motivated architectural choice from an experimentally established one, and we will answer it in both registers. We retain QOFT because we augment an existing quantized adaptation path whose submitted performance is already strong; we do not claim the combined layer inherits orthogonality, nor that current evidence proves QOFT is the best carrier. Your requested comparison — a similarly sized nonorthogonal carrier carrying the same branch — is exactly the right experiment, and it is now running.

In terms of direct evidence, the submitted decomposition explains the starting point. On Qwen2.5-3B Base, Table 1 reports Macro-6 of 34.63 for QLoRA, 42.55 for QOFT, and 45.25 for AuroOFT. Thus 7.92 points of the displayed 10.62-point AuroOFT-versus-QLoRA margin are already present in the carrier; the added residual accounts for the remaining displayed 2.70 points. This is why we study a correction around the strong QOFT reference — and also why we must not attribute the whole QLoRA margin to our residual. The reported trainable budgets are 24.73M, 13.00M, and 16.74M respectively; the original QLoRA row is not the requested LoRA-plus-Enhanced system and cannot settle the carrier question.

As completed rebuttal runs now show, the carrier contrast is underway. Following your review we trained the requested nonorthogonal-carrier system: rank-7 LoRA carrying the same Enhanced-r2 branch on Qwen2.5-3B Base at seed 42, with matched effective batch, steps, data order, and injection configuration. That system's full six-task evaluation is now complete: 37.11 Macro-6 (AMC23 10/40 = 25.00, AQUA 70/254 = 27.56, CMATH 448/600 = 74.67, GaoKao2023en 196/385 = 50.91, Minerva 51/272 = 18.75, OlympiadBench 174/675 = 25.78; all 2,226 items scored, with 13 empty predictions and 3 timeout flags retained as diagnostics; full execution record in W5). The QOFT-plus-Enhanced partner at the same seed is at step 1,390 of 1,563 and completes within hours, with evaluation to follow immediately. [Pending P02: the QOFT-carrier six-task score with counts, plus the near-budget-matched parameter accounting.] Until that partner is scored, the LoRA-carrier row is a standalone measurement and we do not read it as a carrier comparison; we will report the paired difference in whichever direction it falls, together with its per-task pattern.

At the layer level, the structural statement is deliberately narrow. QOFT applies an approximately orthogonal input-side Cayley-Neumann rotation before a frozen NF4 weight; AuroOFT retains that computation and adds a residual at its output. Zero-initializing the up projection recovers the corresponding QOFT function exactly; after learning, the full mapping is generally nonorthogonal. The motivation is continuity with the QOFT computation and a separately controllable correction — not a guarantee of angle preservation, norm preservation, or forgetting resistance. The nonlinear branch adds inference computation and is not generally mergeable into a static weight.

The boundary of the current evidence. A single completed LoRA-versus-QOFT carrier comparison would still not identify orthogonality itself as the cause of any difference, because carriers differ in more than orthogonality; estimating a carrier-by-basis interaction additionally requires each carrier paired with its own linearized branch, which the current queue retains as an optional extension. Section 4 will restrict all orthogonality statements to the carrier's rotation module, and Section 5 will list which contrasts are available versus pending. [Pending P07: Enhanced-specific runtime measurements; the old Lite-profiled numbers cannot be substituted.]

Planned revision. In the revised manuscript, we will update Sections 4.6, 5, and 6. The revised text will read:

Orthogonality is a property of QOFT's approximate rotation module, not of the full AuroOFT layer after adding a learned residual. The submitted 3B Base scores of 34.63/42.55/45.25 (QLoRA/QOFT/AuroOFT) show that most of the displayed AuroOFT–QLoRA margin belongs to the carrier. A matched LoRA-carrier contrast carrying the same branch is reported separately; until it is validated, retaining QOFT is a motivated architectural choice rather than an established optimal carrier. The nonlinear branch is generally non-mergeable and adds inference work, which we quantify for the Enhanced configuration.

AuthorsOpenReview ↗

W3. Evidence for the proposed expressivity bottleneck [1/1]

W3. Evidence for the proposed expressivity bottleneck

Reviewer's concern. Improved accuracy after adding a residual does not directly demonstrate that QOFT's orthogonal update caused its reasoning errors. Problem-level evidence and an appropriate linear comparison are missing.

Thank you for distinguishing a motivating hypothesis from an established explanation; this distinction will be enforced throughout the revision. We agree that the reported improvement cannot identify orthogonality as the cause of an incorrect solution. The defensible statement is that the residual expands the corrections available at an adapted layer, and that submitted experiments report benefits in some settings; establishing which errors change requires full outputs, and explaining why they change requires more than accuracy or a selected example.

In terms of direct evidence, the recovered item-level records are informative but deliberately narrow. We recovered paired QOFT and old 3B-Instruct Lite predictions for all 2,226 non-SAT items: 140 scored corrections of QOFT errors, 102 newly wrong Lite answers, 940 items correct for both systems, and 1,044 incorrect for both. The effect varies by task (GaoKao: 30 corrections vs 14 new errors; AMC23: 2 vs 6). These counts establish that scored outcomes change in both directions; they are not Enhanced results, contain no third system, and do not reveal whether a changed score reflects better mathematics, formatting, or evaluation behavior.

As completed rebuttal runs now show, the linear side of your requested comparison has produced its first full number. The ANL-linearized control on Qwen2.5-3B-Instruct — the branch with its nonlinear basis removed while retaining the internal projections, mixture, RMSNorm, and gate — has completed training and its full six-task evaluation. It obtains 27.50/39.00/82.70/53.00/21.30/24.90 on AMC23/AQUA/CMATH/GaoKao2023en/Minerva/OlympiadBench, equal-weight Macro-6 41.40, with all 2,226 items scored and per-item predictions retained (settings in W5). The matched QOFT and Enhanced partners on 3B-Instruct are training (Enhanced at step 1,380 of 1,563 as of November 16; the QOFT partner completed training and entered evaluation on November 15). [Pending P08: their six-task scores.] A plain BAx residual without normalization and with a fixed unit gate — the stronger form of your requested linear control — has also been trained as a separate run; its evaluation is now running, and we will report it against the ANL-linearized and nonlinear versions under the identical protocol. [Pending P04.]

As a theoretical clarification, the interpretation must remain layer-local. The full Transformer is already nonlinear; the restricted object is the task-specific update at a QOFT-adapted layer, namely a structured input-side rotation. A nonlinear residual can express activation-dependent corrections outside that family, but a trained model need not use that capacity beneficially, and at zero initialization the residual is identically zero — representational possibility is not an active correction. Neither function-family inclusion nor a changed prediction supplies a causal account of a particular failure.

The boundary we accept. Until the three-way item-level analysis on identical problems is complete, "orthogonal expressivity bottleneck" remains a design hypothesis that organizes controls, not a demonstrated failure mechanism; we will remove any wording in the Introduction or Discussion that presents the case analysis as proof of cause. [Pending P03: same-item three-way outputs with parser results, independently checked answers, and counts of corrections and regressions; no invented examples.]

Planned revision. In the revised manuscript, we will update Sections 1 and 6 and Appendix E ("Evidence Policy"). The revised text will read:

We use "orthogonal expressivity bottleneck" as a layer-local design hypothesis. The full Transformer is nonlinear, and a correctness change after adding a branch does not identify orthogonality as the cause of an earlier failure. Recovered 3B-Instruct Lite records show 140 scored corrections and 102 new errors across 2,226 paired items. A completed ANL-linearized Instruct control obtains 41.40 Macro-6 under the full protocol; its QOFT and Enhanced partners and a plain-BAx control are evaluated under the identical protocol and reported with per-task counts. Any case study will describe observed behavior of the tested systems with both corrected and newly wrong cases represented.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.