acceptodds
Under review as a conference paper at ICLR 2027

Low-Rank Adapter Structure Should Be Read Off, Not Written In

Abstract

Low-Rank Adaptation (LoRA) efficiently represents task-specific updates, but an important question remains: how much structure does a task actually need? Standard LoRA fixes the number of rank-one components in advance and uses dense factors. We propose SPAR (SParsity And Rank, adaptively read), a two-stage framework that uses task information to determine component counts and factor sparsity. During fine-tuning, Adaptive Rank compares each module's task-loss gradient signal with its historical peak to guide component growth and stopping. After fine-tuning, Adaptive Sparsity refits each learned adapter update with sparse factors to obtain a compact representation. Component counts and coordinate supports are selected through input-aware reconstruction and model-level task-quality checks, without a prescribed coordinate keep ratio. Thus, SPAR uses task information to determine how much capacity to build during learning and how much structure to retain afterward. On eight GLUE tasks with DeBERTaV3-Base, Adaptive Rank achieves a higher average score than existing LoRA baselines with fewer factor parameters. Adaptive Sparsity further reduces the mean number of nonzero factor parameters by 33.4% while maintaining task performance. We also demonstrate that SPAR is effective with Qwen3-4B-Base on the mathematical reasoning benchmarks MATH500 and AIME2024. The two stages also apply to other adapter methods, as illustrated by adaptive capacity allocation in LoRA and LoRA+ and sparse refitting of AdaLoRA and IncreLoRA updates. These results support our central view: adapter structure can be read off from task-dependent signals rather than written in as a prescribed final rank and coordinate keep ratio. Source code is available at https://anonymous.4open.science/r/SPAR-6F72/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parameter-efficient fine-tuning, Low-rank adaptation, Adaptive rank, Sparse factorization, Adapter compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BydROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SPAR, a two-stage framework for adapting low-rank adapter structure. Adaptive Rank (AR) dynamically adds module-specific rank-one components during fine-tuning using gradient signals and their history. Adaptive Sparsity (AS) then compresses the learned updates into sparse factors through input-weighted reconstruction, with task-quality checks determining whether replacements are accepted. The stages address training capacity and final representation separately. Experiments cover DeBERTaV3-base on GLUE, Qwen3-4B-Base on mathematical reasoning, and applications to other adapters. The authors report that AS reduces mean nonzero A/B parameters on GLUE from 0.916M to 0.610M, with mean task scores of 89.99 before and 90.01 after reconstruction.

Critical strengths
  1. The problem is important and the approach is clear. Choosing adapter capacity is a practical challenge. Separating training-time capacity growth from post-training compression provides a coherent and potentially useful approach.

  2. Paired experiments directly support AS’s compression capability. Reconstruction of the same source adapters reduces mean nonzero A/B parameters by 33.4% while essentially preserving the mean GLUE score. This directly supports quality-preserving structural reduction under the evaluated settings.

  3. The stages show some composability. Appendix F combines AR with LoRA+ and applies AS to the authors’ trained AdaLoRA and IncreLoRA adapters, supporting the framework’s potential applicability to other methods.

Critical weaknesses
  1. The main comparison does not adequately control differences in experimental protocols. GLUE baselines retain their original training and reporting protocols, leaving training configurations, tuning resources, and model selection as confounds. For example, the same method, AdaLoRA, achieves CoLA 71.45 in its original paper but 70.86 under the reproduction setup used in the SoRA paper. This illustrates meaningful variation for the same method across protocols and implementations, complicating interpretation of SPAR’s modest leads. Appendix F’s local baselines differ from the main-table references and do not resolve the common-protocol comparison. The main table therefore does not reliably establish SPAR’s relative advantage.

  2. Missing closest baselines weaken the evidence for superiority. GoRA is cited and discussed but not evaluated. AROMA is included in mathematics but lacks a GLUE comparison. These methods are directly relevant to the rank-selection and allocation problem addressed by SPAR. Without their results under comparable conditions, the experiments do not adequately establish SPAR’s advantage over the closest alternatives.

  3. Capacity differences limit conclusions about performance and efficiency. Full SPAR’s lower mean final parameter count than the cited ordinary LoRA reference is positive evidence. However, AR uses greater capacity on QNLI, final capacities are unreported for mathematics, and SPAR retains 0.610M nonzero parameters versus PARA’s 0.197M in the compression comparison. Without comparisons at similar capacity or quality, a superior performance–capacity tradeoff is not established. Missing AR/AS stage results and costs also make the complete pipeline’s efficiency evidence on generation preliminary.

  4. The released code does not enable verification of the main experiments. The inspected release’s REPRODUCIBILITY.md lists missing GLUE/Qwen main-experiment entry points, exact configurations, and checkpoint/table mappings. Partial examples do not reproduce the main tables, making training protocols, capacities, and model selection difficult to verify externally and compounding the preceding uncertainties.

My primary concern is that the existing comparisons do not adequately establish SPAR’s advantage over key alternatives. The paired AS experiments remain meaningful, but do not resolve the protocol, baseline, and capacity concerns above.

Questions for authors
  1. Are there results for key baselines under common data, evaluation, and model-selection protocols, with comparable training and tuning resources? How do Appendix F’s local baseline configurations relate to the published main-table references?

  2. Can the authors provide common-protocol GLUE comparisons with GoRA and AROMA to assess SPAR’s advantage over the closest alternatives?

  3. What are the training and final capacities for each method, especially in mathematics? Are there comparisons at similar capacity or quality, and stage-wise AR/AS results and costs for mathematics?

  4. Can the key table rows be linked to executable entry points, exact configurations, checkpoints or run records, and aggregation rules?

2 comments
AuthorsOpenReview ↗

Response to Reviewer BydR (1/2)

Regarding W1 and Q1: how SPAR compares with the key baselines under common data, evaluation, and model-selection protocols with comparable training and tuning resources, and how the local baselines in Appendix F relate to Table 1.

Thank you for your thoughtful question. Reusing published scores is standard in this line of work when methods are evaluated on the same datasets under the same task setups. The GLUE references in Table 1 use the DeBERTaV3-base task setups of AdaLoRA and IncreLoRA, which SPAR also follows. SPAR has the highest mean score among the published results that cover all eight tasks, 90.01 (Table 1).

To further assess SPAR's relative advantage, we evaluated SPAR, GoRA, AROMA, and fixed-rank LoRA on Qwen3-4B-Base using the same training data and scoring protocol (Table R12). Full SPAR outperforms the strongest baseline on each benchmark, scoring 75.53 on MATH500 against 74.60 for GoRA, and 16.33 on AIME2024 against 15.67 for LoRA r = 16. We mislabeled the SPAR w/o AS scores (73.00 and 17.00) as full SPAR in Table 2 and will report both stages separately in the revision.

In our own implementation on GLUE, SPAR averages 90.01 against 89.26 for FlexLoRA, the strongest baseline (Table R7 in our response to Reviewer uBjx, W2/Q1). All four methods share the data, the validation metrics, and the periodic evaluation intervals, and each selects its best validation checkpoint (Table R9 in the same response).

We apologize for leaving the purpose and parameter budgets of the Appendix F runs unclear. We trained AdaLoRA and IncreLoRA adapters in our own implementation to test whether AS also applies to adapters trained by other methods, so Table 7 compares each adapter before and after AS. AdaLoRA reports GLUE results at two budgets, 0.32M and 1.27M. For your CoLA example, our implementation used the smaller setting, 0.332M factor parameters, and scored 68.33 before AS, whereas Table 1 cites 71.45 from the published 1.27M setting.

Table R12. Qwen3-4B-Base: the two stages reported separately, with the added GoRA comparison. For LoRA, GoRA, and AROMA, the final counts are also their end-of-training adapter counts. SPAR ends AR training with 5.900M factor parameters and retains 3.106M nonzero parameters after AS.

MethodFinal adapter params (M)MATH500 (avg@3)AIME2024 (avg@10)
LoRA (r = 8)2.94970.1313.67
LoRA (r = 16)5.89870.6015.67
AROMA8.83770.8014.44
GoRA (r_ref = 8)3.30674.6014.00
SPAR w/o AS5.90073.0017.00
SPAR3.10675.5316.33

Regarding W2 and Q2: common-protocol GLUE comparisons with GoRA and AROMA, to assess SPAR's advantage over these methods.

Thank you for your insightful question. We evaluated GoRA and AROMA on all eight GLUE tasks in our own implementation, with the same DeBERTaV3-base backbone, data, validation metrics, and periodic evaluation intervals as SPAR. SPAR scores higher than both methods on every task, averaging 90.01 against 88.27 for GoRA and 87.88 for AROMA, and it retains fewer nonzero adapter parameters on average, 0.610M against 1.207M for GoRA and 0.801M for AROMA (Table R7 in our response to Reviewer uBjx, W2/Q1).

AuthorsOpenReview ↗

Response to Reviewer BydR (2/2)

Regarding W3 and Q3: the training and final capacities of each method, especially in mathematics, whether there are comparisons at similar capacity or quality, and the stage-wise AR and AS results and costs for mathematics.

Thank you for bringing up this important point. The capacities of every method on Qwen3-4B-Base are in Table R12 of our response to W1 and Q1, and the corresponding GLUE means are 0.916M at the end of AR training and 0.610M after AS (Table 3). On QNLI, AR allocates 1.534M parameters against 1.327M for LoRA r = 8, and less than LoRA on each of the other seven tasks, averaging 0.916M (Table 6).

Two comparisons are available at similar adapter sizes. SPAR w/o AS at 5.900M reaches 73.00 on MATH500 and 17.00 on AIME2024, against 70.60 and 15.67 for LoRA r = 16 at 5.898M; full SPAR at 3.106M reaches 75.53 and 16.33, against 74.60 and 14.00 for GoRA at 3.306M, which is larger. On GLUE, FlexLoRA given the per-task component budgets selected by AR reaches 89.49 at 0.909M against 89.99 at 0.916M for AR (Table R1 in our response to Reviewer oeuU).

Table 3 compares the reported operating points of the compression baselines, which have different retained parameter counts. Under the Table 3 setting, PARA retained 0.197M parameters on average, where its mean score is 86.77 against 89.99 for the adapters it compresses. Compressing the same adapters to the size AS selects, PARA changes the mean score by −0.11 and Post-Squeeze by −0.10, while AS changes it by +0.02 (Table R8 in our response to Reviewer uBjx, W2/Q1). That setting therefore costs 3.22 points of mean score, while at the size AS selects the score is retained.

The two stages are reported separately for the mathematical benchmarks in Table R12. AS reduced the nonzero parameter count by 47% and took about 23 additional hours, compared with 69 hours for AR training. The reported MATH500 score rose by 2.53 points and AIME2024 fell by 0.67 points.

Regarding W4 and Q4: whether the key table rows can be linked to executable entry points, exact configurations, checkpoints or run records, and aggregation rules.

Thank you for raising this important question. We have updated the anonymous release (https://anonymous.4open.science/r/SPAR-6F72/) with more detail, including (1) a step-by-step training guide in the README: environment, data, and the commands for AR and AS on the eight GLUE tasks and on Qwen3-4B-Base, with example configurations and how to tune them; (2) the trained checkpoints after AR and after AS, with scripts that reproduce the score of each checkpoint; and (3) a table map and the aggregation rules for the reported numbers.

Reviewer_uBjxOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SPAR, which is a two-stage method. The LoRA rank is adjusted during the training process, and the learned adapters are sparse after the training. The results indicate that using fewer adapter parameters can improve task performance.

Critical strengths
  1. The paper addresses a practical problem: how to choose the right rank and sparsity for different tasks.
  2. The experiments cover both GLUE and mathematical reasoning tasks, with generally positive results.
Critical weaknesses
  1. Limited novelty. Adaptive rank selection and sparse refitting are already well-studied. The main contribution seems to be their combination with new selection rules.
  2. Unfair comparisons. Some GLUE baselines use results from previous papers, while the compression baselines in Table 3 have different parameter counts.
  3. Missing efficiency results. The paper reports parameter savings but does not show actual memory usage, inference speed, or the extra computational cost of SPAR.
Questions for authors
  1. Can the authors compare SPAR with other methods under the same training settings and parameter budgets?
  2. How sensitive is the rank selection to the stopping threshold and noisy gradient signals?
3 comments
AuthorsOpenReview ↗

Response to Reviewer uBjx (1/3)

Regarding W1: the novelty, since adaptive rank selection and sparse refitting are well-studied.

Thank you for raising this important question. SPAR uses task-dependent evidence to determine both the training-time component count and the final sparse representation: AR stops a module from growing when its task-loss gradient signal falls below τ times that module's historical peak, and AS accepts a sparse refit only when the resulting model's task performance holds. These criteria are not new selection rules added on top of existing methods: they replace the external controls that those methods require, so SPAR needs neither a prescribed total rank nor a prescribed coordinate keep ratio.

For the component count, AdaLoRA, IncreLoRA, FlexLoRA, and GoRA select ranks adaptively, but still under a prescribed total budget or reference rank. In Adaptive Rank (AR) the total is an output of training: each module grows until its task-loss gradient signal falls below τ times its historical peak, so under the same τ = 0.2, RTE ends with 235 components and QNLI with 663 (Table 5).

For the final representation, these methods keep dense factors once the ranks are set. Adaptive Sparsity (AS) refits the learned update on the module's inputs with new sparse factors, removing 33.4% of the nonzero parameters while the mean GLUE score is maintained (89.99 to 90.01). Existing compression methods need a prescribed size or threshold, and the score drops at their preset sizes, whereas AS selects the size itself (Table 3).

The two stages are moreover general. In our implementation, AR raises the GLUE mean of LoRA r = 8 from 88.82 to 89.99 (Table R2 in our response to Reviewer oeuU) and of LoRA+ from 87.87 to 88.48, with 31% and 24% fewer factor parameters (Table 6). AS compresses trained AdaLoRA and IncreLoRA adapters with no decrease in any task's mean score (88.50 to 88.58 and 88.39 to 88.51; Table 7). Each stage therefore contributes on its own, not only in combination.

AuthorsOpenReview ↗

Response to Reviewer uBjx (2/3)

Regarding W2 and Q1: the comparison under the same training settings and parameter budgets.

Thank you for your insightful question. Regarding the training settings, reusing published scores is standard in this line of work because the results are obtained on the same datasets under the same task setups. The GLUE baselines in Table 1 follow the DeBERTaV3-base task setups used by AdaLoRA and IncreLoRA, and SPAR follows those same setups. Under these shared GLUE task settings, SPAR achieves the highest mean score among the published results, 90.01 (Table 1).

We also ran three closely related methods highlighted by the reviewers for their adaptive rank selection or allocation: FlexLoRA, GoRA, and AROMA. Within each GLUE task, all four methods used the same data, validation metric, and evaluation interval, with method-specific training settings (Table R9). SPAR achieved the highest mean score (90.01 versus 89.26 for FlexLoRA, the strongest baseline) and retained the fewest adapter parameters on average (0.610M nonzero factor entries; Table R7).

Table R7. GLUE validation results using shared data, validation metrics, and periodic evaluation intervals. Parameter counts average over the eight tasks and exclude the classification head. The full SPAR row reports nonzero factor entries.

MethodParams (M)MNLISST-2CoLAQQPQNLIRTEMRPCSTS-BAvg
AROMA0.80188.5695.8070.2391.2694.0483.1588.4091.5887.88
GoRA1.20789.8594.9968.2691.1193.3286.0491.0991.5188.27
FlexLoRA1.27090.1096.3771.2389.8494.2189.1791.4291.7389.26
SPAR w/o AS0.91690.6096.4172.1892.0394.6090.2591.9191.9689.99
SPAR0.61090.6296.4172.1892.0394.6490.2591.9991.9790.01

Regarding the parameter budgets, Table 3 reports the compression baselines at their prescribed sizes, while AS selects its size itself. To compare at approximately matched sizes, we gave PARA, Post-Squeeze, and ComPEFT the sizes selected by AS and compressed the same 24 pre-AS adapters (Table R8). AS is competitive at these sizes and preserves or improves the score on 21 of the 24 adapters. We also gave FlexLoRA the per-task component budgets selected by AR: FlexLoRA reaches 89.49 with 0.909M parameters, compared with 89.99 and 0.916M for AR (Table R1 in our response to Reviewer oeuU), so AR is also better at a comparable parameter count.

Table R8. Compressing the same pre-AS adapters on GLUE to approximately the nonzero parameter counts selected by AS (24 adapters: eight tasks, three seeds). Score change is relative to SPAR w/o AS. e_X is the relative update-action error.

MethodNonzero (M)Score changee_XNo decrease
AS0.610+0.020.04221 / 24
PARA0.609−0.110.01317 / 24
Post-Squeeze0.609−0.100.08013 / 24
ComPEFT0.610−2.871.0400 / 24

Table R9. Per-task GLUE training settings. Adapter learning rates range from 1e-4 to 2.2e-3 across methods and tasks. Within each task every method uses the same data, the same validation metric and the same evaluation interval, and selects the checkpoint with the best validation score.

TaskEpochsEvaluation interval (steps)
RTE50100
MRPC30100
CoLA20–25100
STS-B20–25100
SST-28–20500
QNLI51,000
QQP51,000
MNLI5–121,000
AuthorsOpenReview ↗

Response to Reviewer uBjx (3/3)

Regarding W3: the memory usage, inference speed, and extra computational cost.

Thank you for this valuable question. We conducted the corresponding measurements. For the inference speed, the unmerged inference time per batch on RTE, MRPC, and CoLA is within about 3% of LoRA r = 8 (Table R10). For the memory usage, the peak memory at inference differs from LoRA r = 8 by less than 1% (Table R10), and during training, SPAR uses 0.20–0.33 GB more; AS stays below 4 GB of GPU memory and uses about 12 GB of CPU memory. For the extra computational cost, AR takes 1.1–1.7× the training time of LoRA r = 8, and AS takes 1–2.6 hours per task after training (Table R2 in our response to Reviewer oeuU); on Qwen3-4B-Base, AR adds about 2% to the training time, and AS takes about one third of it.

Table R10. Unmerged inference: time per batch (ms) / peak GPU memory (MiB), on one RTX A5000.

ModelRTEMRPCCoLA
Base model (no adapter)125.4 / 1236.667.6 / 882.437.4 / 815.7
LoRA r = 8140.4 / 1243.975.7 / 889.740.0 / 823.0
SPAR w/o AS142.5 / 1240.875.6 / 885.840.2 / 822.0
SPAR140.6 / 1239.873.4 / 885.340.0 / 819.4
SPAR / LoRA (time)1.0010.9701.000

Regarding Q2: the sensitivity of the rank selection to the stopping threshold and noisy gradient signals.

We appreciate your thoughtful question. The selected component count is sensitive to both. At τ = 0.1, AR selects 2.60× as many components on MNLI and 3.17× as many on QQP as at τ = 0.5. Across GLUE, the largest score deviation from each task's selected τ is 0.75 points (Table R3 in our response to Reviewer oeuU).

With τ fixed, we used reduced-batch gradient estimates as a proxy for noisier rank-decision signals, computing the gradient for rank decisions from 16 or 8 of the 32 examples per batch. Compared with using all 32 examples, the selected counts decreased by 1–31% with 16 examples and by 6–52% with 8 examples. Scores changed by at most 0.39 points (Table R11).

Table R11. Rank decisions from a noisier gradient signal on GLUE, relative to decisions from all 32 examples: change in score / change in component count.

Task16 of 32 examples8 of 32 examples
MNLI+0.03 / −7%+0.10 / −18%
SST-2+0.23 / −19%−0.11 / −37%
CoLA−0.39 / −22%−0.09 / −34%
QQP+0.14 / −6%+0.37 / −6%
QNLI+0.20 / −31%+0.27 / −52%
RTE0.00 / −15%+0.12 / −21%
MRPC+0.08 / −15%+0.25 / −26%
STS-B−0.01 / −1%−0.09 / −7%
Reviewer_oeuUOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SPAR (Sparsity And Rank, adaptively read), a two-stage framework for adapting low-rank adapter structures using task-dependent evidence. Its first stage, Adaptive Rank, uses the largest singular value of the task-loss gradient matrix and a historical-peak stopping rule to determine component growth during fine-tuning. Its second stage, Adaptive Sparsity, refits learned adapter updates with sparse factors and selects compact representations using input-aware reconstruction and model-level task-quality checks. The paper evaluates SPAR on eight GLUE tasks with DeBERTaV3-Base and on MATH500 and AIME2024 with Qwen3-4B. It also evaluates the two stages independently with LoRA, LoRA+, AdaLoRA, and IncLoRA. The reported results show improved average GLUE performance, a 33.4% reduction in mean nonzero factor parameters while maintaining task performance, and improved mathematical reasoning results over the reported baselines. My main concern is the limited methodological novelty. Adaptive rank allocation is already well studied, including in closely related work such as FlexLoRA. SPAR primarily introduces different evidence and decision rules for rank adaptation, while the distinct contribution of Adaptive Sparsity beyond existing reconstruction and compression techniques is not sufficiently established. More direct comparisons and targeted ablations are needed to demonstrate that these design choices provide substantial advantages over existing methods.

Critical strengths
  1. SPAR separates adaptive component growth during fine-tuning (Adaptive Rank) from post-training sparse refitting (Adaptive Sparsity). The use of a gradient-based signal for rank growth and input-aware reconstruction for compression provides a reasonably clear rationale for the two stages, although the novelty of these mechanisms remains limited.

  2. The experiments cover eight GLUE tasks, mathematical reasoning benchmarks (MATH500 and AIME2024), and extensions to LoRA, LoRA+, AdaLoRA, and IncLoRA. The reported results provide some evidence of the framework's practical utility in maintaining task performance while reducing adapter parameters.

Critical weaknesses
  1. Adaptive rank allocation has been extensively studied, and closely related work such as FlexLoRA already explores evidence-guided rank expansion and pruning. SPAR mainly differs in its gradient-based evidence and historical-peak stopping rule, while the novelty of its Adaptive Sparsity stage is also insufficiently established beyond existing reconstruction and compression techniques. The paper does not convincingly demonstrate that these differences constitute a substantial methodological advance rather than an incremental combination of existing ideas.

  2. Although the reported results are promising, the experiments do not fully isolate the contributions of the individual mechanisms or establish their advantages over the most closely related methods under controlled settings. In addition, the computational overhead of gradient-based rank adaptation and post-training sparse refitting is not sufficiently characterized. These limitations make it difficult to determine whether the added complexity is justified by substantial and reproducible benefits.

Questions for authors

How does SPAR provide a substantive advance over existing adaptive-rank methods, particularly FlexLoRA, which also uses evidence to guide rank expansion and pruning with function-preserving initialization? Beyond using the largest singular value of the task-loss gradient matrix and a historical-peak stopping rule, what distinct capability or advantage does Adaptive Rank provide?

  1. How much does each component of SPAR contribute to the final performance and compression results, and what is the additional wall-clock time and memory overhead of Adaptive Rank and Adaptive Sparsity compared with the relevant baselines?

  2. How sensitive are the final component counts and task performance to the growth threshold (\tau), the historical-peak stopping rule, and the examination schedule?

  3. Adaptive Sparsity accepts compressed candidates based on task metrics evaluated on a subset of training data. How well do these acceptance decisions predict performance on independent validation and test data?

  4. What fundamental methodological advance does SPAR offer beyond existing adaptive-rank methods such as FlexLoRA? Since adaptive rank allocation is already well studied, its gradient-based evidence and post-training sparse refitting appear largely incremental. The paper does not convincingly establish sufficient methodological novelty, which substantially limits its contribution and overall significance.

2 comments
AuthorsOpenReview ↗

Response to Reviewer oeuU (2/3)

Regarding Q3: the sensitivity of the component counts and the task performance to τ, the historical-peak stopping rule, and the examination schedule.

Thank you for your thoughtful question. The growth threshold (τ) and the historical-peak stopping rule clearly change the final component counts, while the task score stays close to its value at the settings used in the paper. The examination schedule changes both only slightly on average; we tested it on RTE and CoLA.

For the growth threshold (τ): τ controls how much capacity is built, since a module stops growing when its task-loss gradient signal falls below τ times its historical peak. We varied τ from 0.1 to 0.5 in steps of 0.1. On every task, the score stayed within 0.75 points of the score obtained with that task's original τ (Table R3). The component counts generally decrease as τ increases. τ = 0.2 is used for 5 of the 8 tasks in the paper, and using it for all eight leaves the mean score almost unchanged, 89.95 against 89.99, though it raises the component counts on MNLI and QQP.

For the historical-peak stopping rule: the final component counts are sensitive to it. Replacing it with one fixed threshold for all tasks changes the score by only −0.36 to +0.46 points, while the component counts change by −15% to +137% (Table R4). That one shared threshold of 0.109 was itself calibrated from completed runs, and the same absolute value gives very different budgets across tasks because the signals it is compared against differ in scale; the historical peak instead expresses the threshold relative to each module's own signal.

For the examination schedule: all tasks share one schedule in the paper. Halving or doubling each of its three settings on RTE and CoLA (Table R5) changes the score by 0.34 points on RTE and 0.41 on CoLA on average, which is comparable to the seed standard deviation of these two tasks (0.36 and 0.41 in Table 1), and changes the component counts by at most 16%.

Table R3. Changing τ on GLUE: change in score / change in component count, relative to the τ used in the paper for each task, whose cell shows the score in the paper (mean 89.99).

Task (selected τ)τ = 0.1τ = 0.2τ = 0.3τ = 0.4τ = 0.5
RTE (0.2)−0.60 / +1%90.25−0.60 / −1%+0.72 / −4%−0.12 / −10%
MRPC (0.2)−0.33 / +6%91.91−0.08 / −6%0.00 / −8%−0.25 / −7%
CoLA (0.2)−0.75 / +10%72.18−0.54 / −17%−0.54 / −22%−0.24 / −25%
STS-B (0.3)−0.25 / +27%−0.14 / +17%91.96−0.12 / −19%−0.06 / −34%
QNLI (0.2)−0.14 / +38%94.60−0.02 / −24%+0.09 / −42%−0.05 / −55%
SST-2 (0.2)−0.11 / +24%96.410.00 / −15%−0.23 / −30%0.00 / −39%
MNLI (0.5)−0.11 / +160%−0.04 / +134%−0.11 / +94%+0.20 / +34%90.60
QQP (0.5)−0.40 / +217%−0.12 / +163%−0.13 / +101%−0.26 / +39%92.03

Table R4. Replacing the historical-peak stopping rule with one fixed threshold (0.109) for all tasks: change in score / change in component count.

TaskFixed threshold
MNLI−0.10 / +109%
SST-2+0.46 / −13%
CoLA−0.16 / −15%
QQP+0.33 / +137%
QNLI−0.04 / −8%
RTE−0.36 / −2%
MRPC−0.25 / −9%
STS-B+0.30 / +22%

Table R5. Halving (×0.5) or doubling (×2) each setting of the examination schedule on RTE and CoLA: change in score / change in component count.

Setting changedRTE ×0.5RTE ×2CoLA ×0.5CoLA ×2
Sensitivity window+0.12 / +4%−0.60 / −7%+0.04 / +2%+0.86 / −7%
Minimum interval−0.84 / +9%0.00 / −16%+0.23 / +6%−0.19 / −14%
Maximum interval−0.36 / +2%−0.12 / −4%−0.59 / +2%−0.54 / −5%
AuthorsOpenReview ↗

Response to Reviewer oeuU (3/3)

Regarding Q4: Whether the acceptance decisions of AS predict performance on independent data.

We appreciate your thoughtful question. The decisions made on the training data hold up on data not used for them. After all accepted replacements, the three-seed mean validation score changes by 0.00 to +0.08 points on the eight GLUE tasks, and no task decreases (Table 1). On the independent test sets of Qwen3-4B-Base, MATH500 improves from 73.00 to 75.53 and AIME2024 decreases from 17.00 to 16.33. We mislabeled the SPAR w/o AS results (73.00 on MATH500 and 17.00 on AIME2024) as full SPAR in Table 2. We will report both stages separately in the revision.

We also conducted an additional experiment that changes the size of this subset of training data from one quarter to four times the default (Table R6). On the five tasks tested, the validation score changes by at most 0.19 points relative to the default subset size, so the final result is stable across these sizes.

Table R6. Changing the size of the subset of training data used by the task-quality checks of AS: change in validation score.

Task×0.25×0.5×2×4
RTE0.000.000.000.00
MRPC0.000.00−0.160.00
CoLA0.000.00−0.06+0.19
STS-B−0.01−0.01−0.01−0.01
QNLI−0.02−0.01+0.02−0.06

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.