- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes OPSA, a nonlinear low-rank adapter designed to improve fine-tuning on long-tailed data. It introduces an orthogonal phase representation and momentum-aware clipping to reduce spectral concentration in adapter updates. The paper provides theoretical analyses and evaluates the method on long-tailed image and text classification tasks, with baseline comparisons and ablation studies.
Critical strengths
-
The paper raises an interesting question about whether long-tailed data causes low-rank adapters to concentrate their updates in a few dominant directions. Using spectral imbalance to study this problem is a useful idea. OPSA also introduces a different adapter design that tries to address this issue without increasing the number of trainable parameters.
-
OPSA reports lower spectral imbalance than the baselines and better macro F1 on both image and text classification. The improvement in tail F1 on 20 Newsgroups is particularly encouraging. Although the current experiments are not enough to establish the proposed mechanism, these results suggest that the nonlinear adapter design may be worth investigating.
Critical weaknesses
- The main theoretical explanation is flawed and does not match the experiments.
Theorem 3 in Appendix A.11 contains a serious problem. The proof uses the equality of two vector norms to argue that corresponding coordinates saturate together, which does not follow mathematically. A simple 2D counterexample, , shows that a coordinate can reach the phase horizon while its derivative remains positive. This directly challenges the claimed gradient reversal mechanism.
More importantly, Table 2 shows that 99.7% of latent units are outside the theoretical basin by the end of training, with a maximum amplitude of 5.06 compared with the boundary of 1.57. Enforcing the basin in OPSA-v2 also leads to worse performance. Theorem 2 relies on gain-ordering assumptions that are not verified for the actual nonlinear adapter. As a result, it is unclear whether the theory explains the observed spectral improvements or the performance gains at all.
The claim of zero additional inference cost after merging is also unsupported. Unlike LoRA, OPSA contains input-dependent normalization and cosine nonlinearities, so its update cannot generally be merged exactly into a fixed weight matrix. No post-merging accuracy or inference-time results are provided.
- The experimental evidence is not reliable enough, and the code release does not allow full reproduction.
Appendix A.3 admits that the three-seed and key ablation experiments use a validation set overlapping with training for checkpoint selection. Even with the same imbalance ratio and seed, Tables 1 and 2 report macro F1 scores of 0.569 and 0.617. The difference is explained by different experimental protocols, but it is much larger than the main reported improvement over LoRA. This makes the ablation conclusions, especially the claimed importance of MAPC, difficult to trust without results under a consistent protocol.
Reproducibility is another major concern. The released repository does not contain the complete OPSA implementation, MAPC optimizer, training pipeline, or main checkpoints. The MAPC update rules are also inconsistent between the main text and appendix. The authors additionally state that the text checkpoints were not saved, so the central spectral metric cannot be recomputed for that benchmark.
These issues are especially concerning because the theoretical explanation is already inconsistent with the observed training behavior. Without reliable experiments and complete implementation details, it is difficult to independently verify the paper's main claims.
Questions for authors
-
How do the authors justify the gradient reversal mechanism in Theorem 3? The equality does not imply coordinate-wise saturation. For example, with , the first coordinate reaches the phase horizon, but . How does Theorem 3 account for this case? Also, given that nearly all latent units leave the theoretical basin during training, what evidence shows that the claimed gradient reversal and spectral contraction mechanisms actually explain the experimental results?
-
Can the authors provide the key results under a consistent experimental protocol? In particular, does removing MAPC still cause a similar performance drop when checkpoint selection uses a validation set fully disjoint from training? How much of the difference between Tables 1 and 2 comes from the different splits and checkpoint-selection procedures? The complete OPSA and MAPC implementations would also be necessary to verify these results, especially given the inconsistent MAPC definitions in the paper.
-
How exactly is OPSA merged for inference? Since the adapter includes input normalization and cosine nonlinearities, its output is not generally equivalent to a fixed linear weight update. Is the proposed merging exact or approximate? If it is approximate, what are the changes in prediction performance and inference latency after merging? This is important for evaluating the paper's efficiency claims.
We re-ran every method under one clean protocol and added ViT-L/16, HAM10000 and OL3I Heart CT. OPSA has the lowest spectral imbalance of eight adapters and is clearly ahead at the most severe imbalance ratio (150). We corrected Theorems 2 and 3, added results that explain where the contraction comes from, and withdrew the claims that did not hold. The camera-ready version will add multi-seed runs for every method and dataset, the full text ablations, and the code release.
We thank reviewers w2Rs, PUKB and Sb6X. Below: what we ran, corrected and will add (I: spectral imbalance, lower is better).
1. New experiments
2. Theory corrections
3. Points addressed
4. Camera-ready commitments