- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces SPAR, a two-stage framework for adapting low-rank adapter structure. Adaptive Rank (AR) dynamically adds module-specific rank-one components during fine-tuning using gradient signals and their history. Adaptive Sparsity (AS) then compresses the learned updates into sparse factors through input-weighted reconstruction, with task-quality checks determining whether replacements are accepted. The stages address training capacity and final representation separately. Experiments cover DeBERTaV3-base on GLUE, Qwen3-4B-Base on mathematical reasoning, and applications to other adapters. The authors report that AS reduces mean nonzero A/B parameters on GLUE from 0.916M to 0.610M, with mean task scores of 89.99 before and 90.01 after reconstruction.
Critical strengths
-
The problem is important and the approach is clear. Choosing adapter capacity is a practical challenge. Separating training-time capacity growth from post-training compression provides a coherent and potentially useful approach.
-
Paired experiments directly support AS’s compression capability. Reconstruction of the same source adapters reduces mean nonzero A/B parameters by 33.4% while essentially preserving the mean GLUE score. This directly supports quality-preserving structural reduction under the evaluated settings.
-
The stages show some composability. Appendix F combines AR with LoRA+ and applies AS to the authors’ trained AdaLoRA and IncreLoRA adapters, supporting the framework’s potential applicability to other methods.
Critical weaknesses
-
The main comparison does not adequately control differences in experimental protocols. GLUE baselines retain their original training and reporting protocols, leaving training configurations, tuning resources, and model selection as confounds. For example, the same method, AdaLoRA, achieves CoLA 71.45 in its original paper but 70.86 under the reproduction setup used in the SoRA paper. This illustrates meaningful variation for the same method across protocols and implementations, complicating interpretation of SPAR’s modest leads. Appendix F’s local baselines differ from the main-table references and do not resolve the common-protocol comparison. The main table therefore does not reliably establish SPAR’s relative advantage.
-
Missing closest baselines weaken the evidence for superiority. GoRA is cited and discussed but not evaluated. AROMA is included in mathematics but lacks a GLUE comparison. These methods are directly relevant to the rank-selection and allocation problem addressed by SPAR. Without their results under comparable conditions, the experiments do not adequately establish SPAR’s advantage over the closest alternatives.
-
Capacity differences limit conclusions about performance and efficiency. Full SPAR’s lower mean final parameter count than the cited ordinary LoRA reference is positive evidence. However, AR uses greater capacity on QNLI, final capacities are unreported for mathematics, and SPAR retains 0.610M nonzero parameters versus PARA’s 0.197M in the compression comparison. Without comparisons at similar capacity or quality, a superior performance–capacity tradeoff is not established. Missing AR/AS stage results and costs also make the complete pipeline’s efficiency evidence on generation preliminary.
-
The released code does not enable verification of the main experiments. The inspected release’s REPRODUCIBILITY.md lists missing GLUE/Qwen main-experiment entry points, exact configurations, and checkpoint/table mappings. Partial examples do not reproduce the main tables, making training protocols, capacities, and model selection difficult to verify externally and compounding the preceding uncertainties.
My primary concern is that the existing comparisons do not adequately establish SPAR’s advantage over key alternatives. The paired AS experiments remain meaningful, but do not resolve the protocol, baseline, and capacity concerns above.
Questions for authors
-
Are there results for key baselines under common data, evaluation, and model-selection protocols, with comparable training and tuning resources? How do Appendix F’s local baseline configurations relate to the published main-table references?
-
Can the authors provide common-protocol GLUE comparisons with GoRA and AROMA to assess SPAR’s advantage over the closest alternatives?
-
What are the training and final capacities for each method, especially in mathematics? Are there comparisons at similar capacity or quality, and stage-wise AR/AS results and costs for mathematics?
-
Can the key table rows be linked to executable entry points, exact configurations, checkpoints or run records, and aggregation rules?
2 comments
Response to Reviewer BydR (2/2)
Regarding W3 and Q3: the training and final capacities of each method, especially in mathematics, whether there are comparisons at similar capacity or quality, and the stage-wise AR and AS results and costs for mathematics.
Thank you for bringing up this important point. The capacities of every method on Qwen3-4B-Base are in Table R12 of our response to W1 and Q1, and the corresponding GLUE means are 0.916M at the end of AR training and 0.610M after AS (Table 3). On QNLI, AR allocates 1.534M parameters against 1.327M for LoRA r = 8, and less than LoRA on each of the other seven tasks, averaging 0.916M (Table 6).
Two comparisons are available at similar adapter sizes. SPAR w/o AS at 5.900M reaches 73.00 on MATH500 and 17.00 on AIME2024, against 70.60 and 15.67 for LoRA r = 16 at 5.898M; full SPAR at 3.106M reaches 75.53 and 16.33, against 74.60 and 14.00 for GoRA at 3.306M, which is larger. On GLUE, FlexLoRA given the per-task component budgets selected by AR reaches 89.49 at 0.909M against 89.99 at 0.916M for AR (Table R1 in our response to Reviewer oeuU).
Table 3 compares the reported operating points of the compression baselines, which have different retained parameter counts. Under the Table 3 setting, PARA retained 0.197M parameters on average, where its mean score is 86.77 against 89.99 for the adapters it compresses. Compressing the same adapters to the size AS selects, PARA changes the mean score by −0.11 and Post-Squeeze by −0.10, while AS changes it by +0.02 (Table R8 in our response to Reviewer uBjx, W2/Q1). That setting therefore costs 3.22 points of mean score, while at the size AS selects the score is retained.
The two stages are reported separately for the mathematical benchmarks in Table R12. AS reduced the nonzero parameter count by 47% and took about 23 additional hours, compared with 69 hours for AR training. The reported MATH500 score rose by 2.53 points and AIME2024 fell by 0.67 points.
Regarding W4 and Q4: whether the key table rows can be linked to executable entry points, exact configurations, checkpoints or run records, and aggregation rules.
Thank you for raising this important question. We have updated the anonymous release (https://anonymous.4open.science/r/SPAR-6F72/) with more detail, including (1) a step-by-step training guide in the README: environment, data, and the commands for AR and AS on the eight GLUE tasks and on Qwen3-4B-Base, with example configurations and how to tune them; (2) the trained checkpoints after AR and after AS, with scripts that reproduce the score of each checkpoint; and (3) a table map and the aggregation rules for the reported numbers.
Response to Reviewer BydR (1/2)
Regarding W1 and Q1: how SPAR compares with the key baselines under common data, evaluation, and model-selection protocols with comparable training and tuning resources, and how the local baselines in Appendix F relate to Table 1.
Thank you for your thoughtful question. Reusing published scores is standard in this line of work when methods are evaluated on the same datasets under the same task setups. The GLUE references in Table 1 use the DeBERTaV3-base task setups of AdaLoRA and IncreLoRA, which SPAR also follows. SPAR has the highest mean score among the published results that cover all eight tasks, 90.01 (Table 1).
To further assess SPAR's relative advantage, we evaluated SPAR, GoRA, AROMA, and fixed-rank LoRA on Qwen3-4B-Base using the same training data and scoring protocol (Table R12). Full SPAR outperforms the strongest baseline on each benchmark, scoring 75.53 on MATH500 against 74.60 for GoRA, and 16.33 on AIME2024 against 15.67 for LoRA r = 16. We mislabeled the SPAR w/o AS scores (73.00 and 17.00) as full SPAR in Table 2 and will report both stages separately in the revision.
In our own implementation on GLUE, SPAR averages 90.01 against 89.26 for FlexLoRA, the strongest baseline (Table R7 in our response to Reviewer uBjx, W2/Q1). All four methods share the data, the validation metrics, and the periodic evaluation intervals, and each selects its best validation checkpoint (Table R9 in the same response).
We apologize for leaving the purpose and parameter budgets of the Appendix F runs unclear. We trained AdaLoRA and IncreLoRA adapters in our own implementation to test whether AS also applies to adapters trained by other methods, so Table 7 compares each adapter before and after AS. AdaLoRA reports GLUE results at two budgets, 0.32M and 1.27M. For your CoLA example, our implementation used the smaller setting, 0.332M factor parameters, and scored 68.33 before AS, whereas Table 1 cites 71.45 from the published 1.27M setting.
Table R12. Qwen3-4B-Base: the two stages reported separately, with the added GoRA comparison. For LoRA, GoRA, and AROMA, the final counts are also their end-of-training adapter counts. SPAR ends AR training with 5.900M factor parameters and retains 3.106M nonzero parameters after AS.
Regarding W2 and Q2: common-protocol GLUE comparisons with GoRA and AROMA, to assess SPAR's advantage over these methods.
Thank you for your insightful question. We evaluated GoRA and AROMA on all eight GLUE tasks in our own implementation, with the same DeBERTaV3-base backbone, data, validation metrics, and periodic evaluation intervals as SPAR. SPAR scores higher than both methods on every task, averaging 90.01 against 88.27 for GoRA and 87.88 for AROMA, and it retains fewer nonzero adapter parameters on average, 0.610M against 1.207M for GoRA and 0.801M for AROMA (Table R7 in our response to Reviewer uBjx, W2/Q1).