- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes AuroOFT, a nonlinear residual augmentation method for quantized orthogonal fine-tuning. AuroOFT adds a zero-start, gated nonlinear low-rank residual branch alongside the QOFT branch and uses an adaptive nonlinear layer (ANL) to enhance the adapter's expressive capacity. Experiments show that AuroOFT outperforms QOFT on Qwen2.5 models at the 1.5B and 3B scales.
Critical strengths
-
The paper proposes a framework that combines QOFT's orthogonal adaptation path with a nonlinear low-rank residual. It enables additional input-dependent corrections while preserving the frozen quantized weights and the original orthogonal branch.
-
The paper evaluates AuroOFT on mathematical reasoning benchmarks using Qwen2.5 models of different sizes. It also includes ablation studies of different ANL variants, comparisons with parameter-matched linear residuals, and multi-seed experiments in selected settings, which help assess the contribution of the nonlinear residual.
Critical weaknesses
-
The paper extends QOFT with the nonlinear adaptation idea from AuroRA to address the proposed "orthogonal expressivity bottleneck." However, it does not provide sufficient evidence that QOFT's orthogonal linear constraints actually constitute an expressivity bottleneck in mathematical reasoning tasks.
-
Although AuroOFT outperforms QOFT with a parameter-matched linear residual on Qwen2.5-3B Base, the linear-residual variant achieves comparable or higher accuracy than AuroOFT on on the 1.5B-it and 3B-it settings.
Questions for authors
-
Please provide a more detailed analysis of specific mathematical reasoning problems that QOFT fails to solve. For these problems, compare the reasoning outputs of QOFT, QOFT with a linear residual, and AuroOFT to identify which errors are corrected by the nonlinear residual and explain how these findings support the proposed "orthogonal expressivity bottleneck" hypothesis.
-
Please further analyze the conditions under which nonlinear residuals provide additional performance gains.
5 comments
W1. The proposed bottleneck is not established as a failure mechanism [1/1]
W1. The proposed bottleneck is not established as a failure mechanism
Reviewer's concern. The paper does not demonstrate that QOFT's orthogonal update caused the observed reasoning errors; the expressivity-bottleneck narrative is presented without mechanism-level evidence.
Thank you for insisting on the distinction between a hypothesis and a demonstrated mechanism; we accept the criticism and will restructure the claims accordingly. The submitted evidence shows that adding a residual changes scores in various settings; it does not show that orthogonality caused specific reasoning errors. What we can support today is (i) item-level evidence that scored outcomes change in both directions, and (ii) a controlled linear-side comparison now partially complete under a matched protocol. The bottleneck phrase will be repositioned as a design hypothesis that organizes controls, and wording that presents it as a proven failure mechanism will be removed.
In terms of direct evidence, the recovered item-level records quantify outcome changes without overclaiming their nature. For all 2,226 non-SAT items we recovered paired predictions of QOFT and the old 3B-Instruct Lite model: 140 scored corrections of QOFT errors, 102 newly wrong answers, 940 items correct for both, and 1,044 incorrect for both. The effect is task-dependent (GaoKao: 30 corrections versus 14 regressions; AMC23: 2 versus 6). These counts establish that the branch both fixes and breaks individual answers; they are not Enhanced results, include no third system, and cannot distinguish better mathematics from formatting or evaluation effects.
As completed rebuttal evidence, the linear side of the mechanism question now has a full number, with its partners in flight. The ANL-linearized branch on Qwen2.5-3B-Instruct — the same residual with its nonlinear basis removed while keeping the internal projections, mixture, RMSNorm, and gate — completed training and its full six-task evaluation: 41.40 Macro-6 (AMC23 11/40 = 27.50, AQUA 99/254 = 39.00, CMATH 496/600 = 82.70, GaoKao 204/385 = 53.00, Minerva 58/272 = 21.30, OlympiadBench 168/675 = 24.90; one empty prediction and one timeout flag retained as diagnostics). Under the identical protocol, the QOFT partner's evaluation began November 15 and the Enhanced partner is at step 1,380 of 1,563 [Pending P08: both six-task scores]. In addition, the stronger form of the linear control you implicitly ask for — a plain BAx residual with no normalization and a fixed unit gate — has been trained as a separate run and its evaluation is now running [Pending P04]. Together these give, for the first time, a same-protocol reading of how much of the branch's effect survives linearization.
As a theoretical clarification, the mechanism claim must remain layer-local and conditional. The full Transformer is nonlinear; the restricted object is the task-specific update at a QOFT-adapted layer — a structured input-side rotation. A nonlinear residual can express activation-dependent corrections outside that family, but capacity is not causation: at zero initialization the residual is exactly zero, and a trained model need not exploit the added flexibility beneficially. Neither function-family inclusion nor a changed score identifies why a particular solution changed.
The boundary we will state. Until the three-way, same-item analysis is complete — full outputs, independently checked answers, fixed case-selection rules, with both corrected and newly wrong cases — the paper will not present any example or aggregate as a causal account of QOFT's errors. [Pending P03: the complete three-way item-level records; no invented examples.]
Planned revision. In the revised manuscript, we will update Sections 1 and 6 and Appendix E ("Evidence Policy"). The revised text will read:
"Orthogonal expressivity bottleneck" is used as a layer-local design hypothesis, not a demonstrated failure mechanism. Recovered paired records show 140 scored corrections and 102 new errors across 2,226 items. A completed ANL-linearized Instruct control obtains 41.40 Macro-6 with full counts under the evaluation protocol of Appendix B; the matched QOFT and Enhanced partners and a plain-BAx control are evaluated under the identical protocol. Case studies describe observed behavior of the tested systems, with both corrected and newly wrong cases represented, and do not attribute causes.
W2. Linear controls sometimes match or outperform AuroOFT [1/1]
W2. Linear controls sometimes match or outperform AuroOFT
Reviewer's concern. Prior linear controls are reported to match or exceed AuroOFT in several settings, so the nonlinear residual's benefit is unclear.
We thank you for this point because it is empirically correct on the submitted record, and the correct response is to test it under a matched protocol rather than to argue from aggregates. Submitted Table 17 reports identity-ANL at 37.4 versus AuroOFT 37.35 on 1.5B-Instruct and 44.2 versus 43.73 on 3B-Instruct; within the submitted 3B Base table, identity-ANL exceeds Enhanced on AQUA (35.1 vs 34.2) and CMATH (88.2 vs 87.6). These are binding observations, and the new program was designed to answer them head-on.
In terms of direct evidence, the matched linear-versus-nonlinear comparison has begun producing complete rows. The ANL-linearized control on 3B-Instruct now provides one fully counted row: 41.40 Macro-6 with all 2,226 items scored. Its same-model partners — QOFT (evaluation started November 15) and Enhanced (step 1,380 of 1,563) — will complete the same-model triple [Pending P08]. On 3B Base, the ANL-linearized run at seed 43 has completed training and entered evaluation, and its QOFT and Enhanced same-seed partners are respectively in evaluation [Pending P01] and complete (38.83 with full counts). The plain low-rank row whose task cells were missing from Table 3 has been retrained from scratch, with training completed and evaluation running, and the plain-BAx control's evaluation is also running [Pending P04]. If any linear control matches or exceeds Enhanced under this matched protocol, we will state that outcome and withdraw a basis-specific advantage claim for that setting; if Enhanced leads, the instruct counterexamples you cite remain beside the finding as boundary conditions.
As complementary evidence, capacity and seed checks address the residual explanations. The submitted rank-2-to-rank-4 step (16.74M→20.49M parameters; 45.25→45.58 Macro-6) shows a modest capacity return that does not isolate the basis; a second training seed (44) for both QOFT and Enhanced is in progress — the Enhanced side has completed training and entered evaluation, the QOFT side in training — so that a paired difference can be compared against seed variability rather than one draw [Pending P06]. This is precisely the check that aggregated submitted rows cannot provide.
As a methodological clarification, the two linear controls answer different questions and both are being run. The ANL-linearized branch changes only the basis computation, retaining RMSNorm, gating, the internal projections, and initialization — so it isolates nonlinearity while keeping the branch's other machinery. A plain BAx branch removes normalization and fixes the gate at unit initialization, asking whether bare linear capacity suffices but changing more components at once. Reporting both, under one protocol with full counts, is the honest way to answer your concern; conflating them would overstate either.
The boundary we accept. No new claim of nonlinear advantage will be made from incomplete pairs; training progress is not accuracy evidence; and the instruct counterexamples stay in the paper regardless of how the matched comparisons turn out. [Pending P01, P04, P06, P08.]
Planned revision. In the revised manuscript, we will revise Section 5.3, Table 17 in Appendix D, and Section 7. The revised text will read:
Under matched same-seed protocols, the ANL-linearized branch is compared against the complete Enhanced branch and the QOFT carrier, with a plain-BAx control and a second training seed reported alongside. Submitted settings in which linear controls matched or exceeded AuroOFT are retained as boundary conditions, and any basis-specific statement is restricted to validated contrasts with complete task-level counts.
Response to Reviewer 4RWx — Overview and index
Response to Reviewer 4RWx
We are very grateful for your careful reading and for the two weaknesses and two questions you raised; they go to the heart of what our evidence can and cannot show. Your comments are valuable because they demand mechanism-level accounting — what changes in the three models' reasoning, and under what conditions the nonlinear residual actually helps — rather than aggregate scores. Upon receiving the reviews we organized the rerun program around exactly those demands; it is now running simultaneously on three GPU servers, and we will report updated numbers in this discussion each day until all results are complete. As of November 16 (Beijing time), three six-task evaluations have completed with fully inspectable per-item counts, seven further trainings have finished with evaluations running or queued, and below we answer each of your points with the completed results, the pending markers, and the boundaries stated plainly.
All accuracies below are percentages; differences are percentage points. Macro-6 is the equal-weight mean of the six non-SAT benchmarks. Submitted-paper values, recovered pre-submission predictions, and new rebuttal evaluations are labeled as such; new numbers trace to run manifests, seeds, and stored per-item predictions. Quoted manuscript text below is the wording we commit to implement in the revision; bracketed Pending markers identify evidence still in production, and we will resolve each one before the final submission.
Reading guide
The response is organized by the original review items. No substantive content is omitted from the complete response. Continuations preserve the evidence, tables, derivations, and proposed revision quotations in order. Proposed manuscript changes are not represented as already implemented.
Please read each thread in ascending [part/total] order.