acceptodds
Withdrawn from ICLR 2027

From Nested Solving to Predictive Tracking: Learning Lower-Level Dynamics for Bi-level Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Bi-level optimization couples an upper-level objective with a parameter-dependent lower-level problem, requiring repeated lower-level solves as the upper-level variable evolves. Repeatedly solving the lower-level problem is costly, but inaccurate responses distort the upper-level update. We introduce SUPRO, a first-order method that tracks a persistent lower-level response across outer iterations. SUPRO models the joint state of the lower-level iterate and an auxiliary value response to predict response motion, and adapts the predictor online using feedback from subsequent corrections. Fixed-budget gradient corrections and objective-based candidate selection control prediction errors. We establish a non-asymptotic stationarity guarantee for the associated fixed value-function penalty, together with a response-tracking bound that improves as predictions become more accurate. SUPRO uses only single-loop computation and requires neither Hessian-vector products nor implicit linear solves. Experiments on nonconvex problems, representative learning tasks, and large-model post-training show that SUPRO realize remarkable performances while maintaining stable optimization.

Primary area:
learning theory
Keywords:
Bilevel Optimization, Learning to Optimize, Predictive Response Tracking, Nonconvex Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RqHwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a first-order bilevel method named SUPRO that learns to track an auxiliary lower-level response. A joint representation of the explicit decision and auxiliary response predicts a displacement, and fixed-budget gradient corrections produce reactive and predictive candidates and objective comparisons safeguard their selection. A learned proposal and bounded response mixing further modify the updates. Under a uniform PL condition the authors prove a tracking bound that scales with the smaller of the response-path variation and the prediction error in theorem 4.2 and O(1/T)O(1/T) rate of the fixed λ\lambda penalty in theorem 4.4. Experiments on a synthetic nonconvex problem, few-shot learning, data hyper-cleaning, and RoBERTa-base adaptation on SST-2 show faster convergence and comparable final accuracy against other baselines.

Critical strengths
  • The paper explicitly distinguishes fixed-penalty stationarity from original bilevel stationarity, and includes a counterexample in Appendix A.5.2.

  • Table 1 reports shorter time to 90% accuracy on Omniglot settings, Table 2 reports synthetic improvements, and Tables 3-4 explore both algorithmic components and architectural choices.

Critical weaknesses
  • Figure 2 and Tables 3-4 compare outer iterations, but SUPRO constructs multiple candidates, evaluates their objectives and gradients and trains a predictor. The paper acknowledged these costs in Appendices A. Table 3's 90% threshold improves from 350 iterations for two-step GD/linear extrapolation/no learned prediction to 320 for SUPRO, this small reduction may be offset by per-iteration overhead, so wall-clock time evaluation is needed on SST-2.

  • In Table 3 removing the reactive candidate even yields a higher peak accuracy though threshold attainment is slower. Two pairs of variants report same numbers of all columns (linear extrapolation & no learned prediction, no mixing & two-step GD), which suggests the components distinguishing them are inactive and removing the reactive fallback gives the highest peak.

  • Theorem 4.2 conditions the improvement on prediction error ETE_T, which is not established for the online trained predictor (i.e., it does not establish that online losses reduce that error). No tracking error connects this mechanism to the observed gains. Also the rate established in Theorem 4.4 is obtained from the reactive branch.

Questions for authors
  • Please report wall-clock time comparison and numbers of gradient/function evaluations to reach the target for all methods on BiDoRA and hyper-cleaning. Besides, do improvements in tables 3, 4 hold across different random seeds?

  • Can you provide diagnostics for prediction/tracking errors and candidate selection on the synthetic problem where exact lower-level responses are available?

  • Along the trajectories, is ccET<cbVTc_cE_T<c_bV_T in practice, and how often are learned proposals accepted?

  • Why is w/o learned network (89.22%) which keeps selection and mixing worse than the two-step GD reference (91.17%)?

9 comments
AuthorsOpenReview ↗

Q1 Wall-clock efficiency and computational cost

1. Wall-clock efficiency and computational cost

We thank the reviewer for the careful assessment and for recognizing the prediction-dependent analysis and the distinction between fixed-penalty and original bilevel stationarity. We appreciate the concern that candidate construction and online training introduce costs beyond an outer iteration. Our practical objective is to improve performance within a given training budget.

We evaluated four methods at adapter ranks 4, 8, and 16, using three paired seeds per configuration: 36 completed BiDoRA runs. Within each rank and seed, the methods share initial task parameters, data splits, and upper-/lower-level minibatch sequences. All runs use FP32, batch size 32, and 500 outer steps.

Timing includes prediction, correction, candidate evaluation, online network training, data handling, and subsequent validation. It starts after initial validation and excludes startup, model loading, initial warmup/validation, and final saving.

RankMethodFinal accuracy (%)Time (min)Peak memory (GiB)
4MEHA88.42 ± 1.2029.49 ± 0.0517.16
4BOME87.50 ± 1.0929.55 ± 0.0717.15
4Two-step GD89.11 ± 1.0229.24 ± 0.0117.06
4SUPRO89.14 ± 1.4926.68 ± 0.0418.87
8MEHA89.53 ± 1.0429.53 ± 0.0417.17
8BOME89.22 ± 1.1129.55 ± 0.0117.15
8Two-step GD90.33 ± 0.6929.26 ± 0.0217.07
8SUPRO90.75 ± 0.7026.93 ± 0.1319.25
16MEHA88.19 ± 1.3229.53 ± 0.0817.18
16BOME88.03 ± 1.2229.65 ± 0.0117.16
16Two-step GD89.41 ± 1.6829.31 ± 0.1017.11
16SUPRO90.48 ± 0.9027.44 ± 0.0320.01

Accuracy and time are mean ± sample standard deviation; memory entries are means across three seeds.

SUPRO achieves the highest mean final accuracy and shortest training time at each rank under these configurations. Its 500-step time reduction ranges from 6.38% to 9.70%, including online learning and candidate-evaluation costs. At rank 16, mean final accuracy exceeds MEHA, BOME, and two-step GD by 2.29, 2.45, and 1.07 percentage points, respectively. The additional peak memory is approximately 1.7–2.9 GiB.

For 500 steps, SUPRO uses 2,385 task-gradient requests and 74 response-network backward passes. MEHA/BOME use 3,000 task-gradient requests, and two-step GD uses 2,500. This corresponds to 20.5% and 4.6% fewer task-gradient requests. Network backward passes are reported separately because these operations have different costs; objective-evaluation costs are included in timing.

For reproducibility, this additional SUPRO configuration predicts at every step and performs full correction and selection during the first 30 steps and every 10 steps thereafter: 77 selection events and 423 prediction-only steps. BOME uses three inner steps.

At ranks 4/8, the main upper- and lower-variable learning rates warm up to 2.5×10−62.5\times10^{-6} over 100 steps. SUPRO increases them to 5×10−65\times10^{-6} over steps 101–200; the baselines retain 2.5×10−62.5\times10^{-6}. The corresponding main learning rates are halved at rank 16. These results assess the complete practical configurations under the declared schedules; the original component controls are discussed separately below.

AuthorsOpenReview ↗

Q2 Accuracy under a fixed time budget and cost to reach the target

2. Accuracy under a fixed time budget and cost to reach the target

Thank you for suggesting comparisons based on actual training time. For the 36-run BiDoRA evaluation described above, we report accuracy at a fixed 25-minute budget and the time and task-gradient cost to reach 90% accuracy.

The fixed-budget result is the last completed validation at or before 25 minutes, without interpolation.

RankMEHA accuracy (%)BOME accuracy (%)Two-step GD accuracy (%)SUPRO accuracy (%)
477.75 ± 9.5952.14 ± 2.1287.16 ± 0.8088.69 ± 1.46
888.65 ± 0.4087.88 ± 0.9989.11 ± 0.8090.75 ± 1.15
1687.58 ± 2.2587.16 ± 2.2988.30 ± 1.9990.37 ± 0.53

SUPRO achieves the highest mean accuracy within the same time budget at all three ranks. Its gains over two-step GD are 1.53, 1.64, and 2.06 percentage points, calculated before rounding. At rank 8, the gains over MEHA and BOME are 2.10 and 2.87 percentage points. By the last validation within this budget, SUPRO completes 450–460 steps, compared with 420 for the baselines.

We also report the first validation reaching 90% accuracy, together with success counts:

RankMethodSuccessful seedsTime to 90% (min)Task-gradient requests to 90%
4MEHA0/3NR—
4BOME0/3NR—
4Two-step GD1/327.472,350
4SUPRO2/324.18 ± 2.162,160 ± 191
8MEHA1/321.842,220
8BOME1/326.582,700
8Two-step GD2/323.39 ± 3.322,000 ± 283
8SUPRO3/320.31 ± 3.111,800 ± 274
16MEHA1/325.462,580
16BOME1/327.882,820
16Two-step GD2/323.44 ± 3.212,000 ± 283
16SUPRO2/319.65 ± 3.661,710 ± 318

NR means the target was not reached within 500 steps. Means and sample standard deviations are calculated over successful runs; single-success entries have no standard deviation. These conditional statistics should be interpreted together with their success counts.

Target gradient counts are calculated from the recorded first-hit steps and fixed evaluation schedule. SUPRO uses nine task-gradient requests at each full selection event and four at each prediction-only step; MEHA/BOME use six per step, and two-step GD uses five. These counts exclude initialization and response-network backward passes. Timing includes online training and candidate evaluations.

Across the nine rank–seed settings, SUPRO reaches 90% in seven runs, compared with five for two-step GD and two each for MEHA and BOME. At rank 8, SUPRO reaches the target in all three seeds. The fixed-budget and target-based measurements therefore provide complementary evidence of practical efficiency.

AuthorsOpenReview ↗

Q3 Reactive-candidate ablation and matching table entries

3. Reactive-candidate ablation and matching table entries

We appreciate the reviewer’s close examination of Table 3. The reactive candidate provides a prediction-independent reference for response selection. Its role is to control the response update when learned predictions are less informative; the highest validation accuracy attained by an individual run measures a different aspect of performance.

Removing the reactive candidate yields a peak accuracy of 91.743%, compared with 91.514% for full SUPRO. The full method, however, reaches 90% at 320 iterations, compared with 390 iterations without the reactive candidate. This experiment therefore associates the reactive reference with earlier target attainment, while the variant without it obtains a slightly higher later peak.

Regarding the matching entries, Table 3 summarizes each trajectory through peak accuracy, the first iteration attaining that peak, and the first iteration reaching 90%. Accuracy is evaluated on 872 examples, and evaluations occur every 10 iterations (Table 5). Matching values of these discrete summaries do not imply identical updates or trajectories.

The prediction controls provide a concrete example. Under the original single-seed control protocol, retaining C/U, candidate selection, and mixing while changing only the prediction residual gives:

Prediction configurationAccuracy at step 500 (%)Peak accuracy (%)
Full learned prediction90.825791.5138
Without the learned P residual90.825791.2844
Linear extrapolation89.908391.2844

Although linear extrapolation and removal of the learned prediction residual attain the same peak accuracy, their step-500 accuracies differ by 0.9174 percentage points. The complete method attains the highest peak and matches the higher final accuracy. These controls use the original ablation protocol and are presented separately from the additional periodic-correction experiments.

The no-mixing and two-step-GD variants also have different update definitions. The former retains learned response construction and candidate selection, whereas the latter directly applies two gradient-based response updates. Figure 3 provides their trajectories, including a small difference around iteration 270 despite their matching summaries. We will make the relationship between these trajectory-level observations and Table 3 more explicit.

The intended interpretation is therefore based on both target attainment and attained accuracy. In the original component study, full SUPRO reaches 90% earlier than each ablated configuration, while the supplementary wall-clock results evaluate the complete practical configuration with its additional computational costs included.

Reviewer_ZKXNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SUPRO, a novel framework for bilevel optimization. The authors establish convergence guarantees under the Polyak–Łojasiewicz (PL) condition on the lower-level objective and demonstrate the effectiveness of the proposed method through empirical evaluations.

Critical strengths
  1. SUPRO exploits temporal information across successive lower-level problems, offering a useful approach to reducing repeated lower-level optimization.

  2. Under the stated assumptions, the analysis establishes stationarity guarantees for the fixed-penalty objective and quantifies how accurate predictions improve the tracking bound.

  3. Experiments cover numerical problems, few-shot learning, data hyper-cleaning, and parameter-efficient fine-tuning. The results demonstrate improved time or fewer iterations to target performance, while ablation studies examine the contributions of individual components.

Critical weaknesses
  1. The main analysis relies on a uniform lower-level PL condition, which excludes nonoptimal stationary points and is substantially stronger than general nonconvexity. Its validity is not established for the neural-network objectives considered in the experiments.

  2. The theoretical guarantees rely on exact objective comparisons and prescribed gradient updates, whereas the learning experiments use minibatch evaluations and, in the RoBERTa setting, Adam. Evaluating candidates on the same minibatch with matched dropout randomness reduces comparison noise, but does not establish the required safeguard for the full objective. The manuscript does not derive the corresponding error bounds for its actual sampling procedure. Similarly, the gradient-descent analysis does not directly cover Adam’s momentum and adaptive preconditioning. Consequently, the convergence guarantees cannot be directly transferred to the full experimental implementation.

  3. The large-model evaluation is limited to parameter-efficient adaptation of RoBERTa-base on SST-2. This provides a useful test case, but does not establish scalability to substantially larger generative models or to settings with many more trainable lower-level parameters.

  4. The method introduces considerable architectural and optimization complexity without sufficient sensitivity analysis.

  5. The presentation is difficult to follow, as many essential methodological details and theoretical qualifications are deferred to the appendix. This makes it challenging to understand the complete algorithm and assess the scope of its guarantees from the main text alone.

  6. The paper should report accuracy-versus-wall-clock-time curves to better assess practical efficiency. Since the proposed method introduces additional computational overhead, comparisons based solely on iteration counts may overstate its convergence advantage. It remains unclear whether the reported gains translate into faster convergence than competing methods once all computational costs are included.

Questions for authors

See weakness

7 comments

We thank the reviewer for recognizing the value of exploiting temporal information in bilevel optimization and for the careful comments on the theoretical analysis and experimental evaluation. We understand the concerns about the assumptions, practical implementation, and computational overhead. Our work aims to establish a principled response-tracking framework and examine its effectiveness under practical training conditions. Below, we clarify the scope of the analysis and provide additional evidence from 36 BiDoRA comparison runs, 15 sensitivity runs, and supplementary synthetic experiments.

1. Scope and role of the lower-level PL condition.

We appreciate the reviewer’s concern about the applicability of uniform PL geometry to neural-network objectives. We agree that this condition is stronger than general nonconvexity and excludes nonoptimal stationary points. Its role in our analysis is to quantify how a fixed correction budget controls response errors while allowing nonconvex lower-level objectives and nonunique minimizers. A uniform lower-level PL condition is also used in V-PBGD (Shen and Chen, ICML 2023, Assumption 2), providing a closely related analytical setting for value-function-based bilevel optimization.

Within this setting, our analysis addresses how prediction interacts with correction and candidate selection. Theorem 4.2 bounds the cumulative tracking error through the smaller of the response-path variation and the prediction-error terms. Corollary A.6 connects this tracking bound to fixed-penalty stationarity. Theorem 4.4 further establishes a joint convergence bound using the reactive safeguard, without requiring accurate predictions.

The manuscript also provides local-PL extensions in Appendix A.5.4. These results require PL geometry only within an invariant response region containing the relevant prediction and correction trajectories, together with the stated conditions for transporting responses between successive upper-level iterates. We will bring these conditions and their interpretation into the main discussion.

Our neural-network experiments primarily assess practical performance in representative bilevel learning tasks. We will state more explicitly that these empirical results do not establish PL geometry for the experimental objectives, and that the stationarity guarantees concern the fixed-penalty formulation under the stated assumptions.

2. Minibatch evaluations, Adam, and the practical implementation.

We appreciate the reviewer’s concern regarding the gap between the theoretical analysis and the practical implementation. We agree that the scope of the theoretical guarantees should be clearly distinguished from the empirical evaluation. Our experiments primarily aim to assess SUPRO’s effectiveness and computational efficiency under practical neural-network training conditions.

To this end, we adopt minibatch training and Adam, following implementation practices used in existing bilevel optimization methods. Specifically, minibatch or task-sampled training is used in BVFSM and IAPTT/AIT, while Adam is employed for upper-level optimization in BVFSM and MEHA. These choices allow us to evaluate SUPRO in practically relevant settings.

For candidate comparisons, using the same minibatch and matched dropout randomness makes the evaluations more comparable. We agree that this alone does not establish the safeguard for the full objective. Appendix A.5.3 makes the required error control explicit: if all candidate evaluations satisfy a simultaneous error bound ∣f^t(ζ)−f(xt,ζ)∣≤dt|\widehat f_t(\zeta)-f(x_t,\zeta)|\le d_t, then the accepted response satisfies

\le \min\{f(x_t,\xi_t^R),f(x_t,\xi_t^P)\} +\epsilon_t+2d_t.$$ This result specifies how evaluation errors affect the lower-level safeguard; applying it to a sampling procedure requires the corresponding error bounds. We will clarify that the deterministic analysis covers the prescribed gradient updates, while Adam and the periodic correction schedule used in the additional experiments are practical variants evaluated empirically. We also examined whether learned proposals contribute to the actual training trajectory. Across the three seeds at each rank, their acceptance rates are: | Adapter rank | Learned-proposal acceptance rate | Accepted / evaluated proposals | |---|---:|---:| | 4 | 38.96 ± 3.90% | 90 / 231 | | 8 | 41.99 ± 10.50% | 97 / 231 | | 16 | 41.99 ± 3.00% | 97 / 231 | Each run contains 77 candidate-selection events. The remaining prediction-only steps are excluded from this denominator. Every accepted proposal in these runs strictly improves both logged minibatch objectives relative to the safeguard reference. These observations show that the learned proposals are selected and contribute to the updates under the practical evaluation rule.

3. Evaluation with increasing numbers of trainable adapter parameters.

Thank you for pointing out the importance of evaluating the method beyond a single adapter configuration. Our experimental focus is on whether predictive response tracking remains useful as the trainable lower-level problem grows within parameter-efficient adaptation. We therefore extended the BiDoRA evaluation from rank 4 to ranks 8 and 16 on RoBERTa-base/SST-2.

We evaluated SUPRO, MEHA, BOME with three inner steps, and the two-step GD reference using three paired seeds at each rank, giving 36 completed runs. Within each rank and seed, the methods share the initial task parameters, data split, and sequence of upper- and lower-level minibatches. All runs use FP32, batch size 32, 500 outer steps, and validation every 10 steps.

The table reports mean ± sample standard deviation across three seeds; peak memory is reported as its mean.

RankMethodFinal accuracy (%)Peak accuracy (%)Training time (min)Peak memory (GiB)
4MEHA88.42 ± 1.2088.49 ± 1.1629.49 ± 0.0517.16
4BOME87.50 ± 1.0987.61 ± 1.2029.55 ± 0.0717.15
4Two-step GD89.11 ± 1.0289.49 ± 0.5229.24 ± 0.0117.06
4SUPRO89.14 ± 1.4989.95 ± 1.4826.68 ± 0.0418.87
8MEHA89.53 ± 1.0489.75 ± 1.0729.53 ± 0.0417.17
8BOME89.22 ± 1.1189.83 ± 0.9629.55 ± 0.0117.15
8Two-step GD90.33 ± 0.6990.37 ± 0.7529.26 ± 0.0217.07
8SUPRO90.75 ± 0.7091.06 ± 0.8026.93 ± 0.1319.25
16MEHA88.19 ± 1.3289.22 ± 1.8029.53 ± 0.0817.18
16BOME88.03 ± 1.2289.07 ± 1.6429.65 ± 0.0117.16
16Two-step GD89.41 ± 1.6889.60 ± 1.6729.31 ± 0.1017.11
16SUPRO90.48 ± 0.9090.71 ± 0.8027.44 ± 0.0320.01

SUPRO achieves the highest mean final and peak accuracy and the shortest training time at all three ranks under these configurations. At rank 4, it reaches a similar final accuracy to two-step GD while reducing training time by 8.74%. At rank 16, its mean final accuracy exceeds MEHA, BOME, and two-step GD by 2.29, 2.45, and 1.07 percentage points, respectively.

For reproducibility, these additional SUPRO runs perform prediction at every step, with the full correction and selection sequence executed during the first 30 steps and every 10 steps thereafter. For ranks 4 and 8, the main upper- and lower-variable learning rates warm up to 2.5×10−62.5\times10^{-6} over 100 steps. SUPRO then increases them to 5×10−65\times10^{-6} over steps 101–200, while the baselines retain 2.5×10−62.5\times10^{-6}. The corresponding main learning rates are halved at rank 16. These comparisons assess the complete configurations under the declared schedules; they do not isolate the contribution of the prediction head.

The memory measurements also quantify the cost of maintaining the additional response state. SUPRO’s peak memory increases from 18.87 to 20.01 GiB as the adapter rank increases from 4 to 16, while its time advantage is retained. We will describe these results specifically as evidence for scaling the trainable adapter size within RoBERTa-base.

4. Architectural complexity and sensitivity to implementation choices.

We agree that the additional architecture and online training should be justified through their measured performance and cost. To examine the main practical choices, we conducted 15 runs covering five configurations with three seeds each.

These sensitivity experiments use a common rank-4 protocol: full correction and selection during the first 100 steps, followed by one event every 10 steps. Each run therefore contains 140 correction/selection events and 2,700 task-gradient evaluations. The comparisons below are made within this common protocol.

First, we varied the coordinate-GRU hidden dimension described in Appendix A.3.2.

Coordinate-GRU hidden dimensionResponse-network parametersFinal accuracy (%)Training time (min)Peak memory (GiB)
431,82389.83 ± 0.9330.05 ± 0.2518.19
8, default33,53990.18 ± 0.5730.05 ± 0.0218.88
1638,50790.48 ± 0.5030.14 ± 0.1020.24

The default hidden dimension of 8 provides a useful accuracy–memory tradeoff. Increasing it to 16 improves mean final accuracy by 0.30 percentage points and increases peak memory by 1.36 GiB, with little change in training time. This also illustrates why both parameter count and state-storage cost should be reported.

Second, we varied the update intervals of the lightweight heads and recurrent/token-mixing backbone introduced in Appendix A.3.3. Intervals are measured in full correction/selection events.

Head / backbone update intervalFinal accuracy (%)Training time (min)Network backward passes
1 / 189.87 ± 0.7530.52 ± 0.38137
1 / 2, default90.18 ± 0.5730.05 ± 0.02137
2 / 489.98 ± 0.2929.52 ± 0.1069

Reducing the online update frequency decreases the number of network backward passes from 137 to 69, while mean final accuracy changes from 90.18% to 89.98%. The corresponding training time decreases from 30.05 to 29.52 minutes. More frequent backbone updates provide no improvement in mean final accuracy in this comparison.

These results complement the architectural ablations in Table 4 and provide direct evidence about the cost–performance tradeoffs around the default configuration.

Reviewer_ZGhUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SUPRO, a first-order bilevel optimization method that learns to track an auxiliary lower-level response within a value-function penalty framework. It combines an online response predictor with reactive and predictor-corrected candidates, objective-based safeguards, and response-guided mixing into the explicit lower-level iterate. The paper provides convergence analysis for the fixed penalty objective and evaluates the method on synthetic optimization, few-shot learning, data hyper-cleaning, and parameter-efficient language-model adaptation.

Critical strengths
  1. The motivation and safeguarded design are appealing. Exploiting temporal information across successive lower-level problems is a reasonable way to improve response tracking. Comparing a predictor-corrected candidate with a reactive candidate provides a clear mechanism for retaining useful predictions while limiting dependence on their quality.

  2. The experiments provide encouraging evidence. The paper includes several task types and component ablations. Tables 1 and 2 report time-to-target improvements, while the BiDoRA experiments suggest earlier attainment of high validation accuracy. The authors also acknowledge that shared predictor parameters do not eliminate coordinate-state storage or candidate-evaluation costs.

Critical weaknesses
  1. The end-to-end computational and memory benefits are insufficiently established. Efficiency is a central motivation, but the BiDoRA comparison in Figure 2 and the ablations in Tables 3–4 primarily measure progress by outer iterations. SUPRO adds recurrent and Transformer computations, online predictor training, phase-specific gradients, and multiple candidate evaluations. Although predictor weights are shared across coordinates and frozen model weights can be reused, coordinate-wise hidden states, the auxiliary response, temporary candidates, and associated gradients still consume memory. The paper does not report predictor parameter counts, peak memory, or a breakdown of these costs for the language-model experiment. The timing results on smaller tasks are useful, but they do not establish whether the iteration savings translate into a favorable runtime–memory trade-off in the setting where expensive lower-level computation is emphasized.

  2. The incremental benefit of the learned mechanism is not yet convincing relative to its complexity. In Table 3, SUPRO reaches 90% accuracy at iteration 320, compared with 350 for both linear extrapolation and the two-step GD reference, an approximately 8.6% reduction in iterations. Its peak accuracy exceeds the variant without the learned prediction residual by only 0.230 percentage points. These results come from a single fixed seed. Removing the entire learned network produces a larger degradation, but simultaneously removes the prediction, correction, and upper-guidance heads, so it does not isolate the benefit of predictive tracking. Matched wall-clock comparisons and repeated runs are needed to determine whether the additional architecture and online training are worthwhile. Moreover, the language-model evaluation is limited to RoBERTa-base on SST-2 with rank-4 adapters, leaving the broader scalability claim insufficiently tested.

Questions for authors
  1. Complete cost accounting: Can the authors report the predictor parameter count, peak GPU memory, forward/backward evaluation counts, and runtime breakdown for BiDoRA, including candidate construction, objective comparisons, and online predictor training? Please compare these measurements with the strongest first-order baseline and the two-step GD reference under the same hardware and precision.

  2. Benefit under matched budgets: Can the authors compare SUPRO, linear extrapolation, the variant without the learned prediction residual, and the two-step GD reference using accuracy-versus-wall-clock curves and multiple random seeds? Reporting both time to a target accuracy and accuracy at a fixed time budget would clarify whether the learned mechanism provides a net practical advantage.

  3. Robustness and scaling: How sensitive are performance and overhead to the predictor's hidden-state size and online update frequency? A larger trainable-parameter setting, or a controlled scaling study, would help establish whether the additional state storage remains manageable. Predictor-corrected candidate win rates would also help connect the observed gains to the proposed mechanism.

7 comments
AuthorsOpenReview ↗

Response to Reviewer ZGhU (Q1): Practical Efficiency, Memory Cost, and Scalability of SUPRO

We thank the reviewer for the constructive comments. Our additional experiments show that SUPRO achieves higher accuracy within a fixed time budget and retains a runtime advantage after including online predictor training. We report the associated memory cost and sensitivity results below.

Q1. Computational cost and peak memory.

We extended the BiDoRA experiment in Section 5 to adapter ranks 4, 8, and 16, with three paired seeds per method. The 36 runs use matched initial task parameters, data splits, and minibatch sequences, with batch size 32 and FP32.

The timing includes prediction, gradient correction, candidate evaluation, online predictor training, and subsequent validation. Model loading, initial warm-up/evaluation, and final saving are excluded. Peak memory is the process-lifetime maximum allocated GPU memory. Results are means ± standard deviations over three seeds.

RankMethodTime for 500 steps (min)Accuracy at 25 min (%)Peak memory (GiB)
4MEHA29.49 ± 0.0577.75 ± 9.5917.16
4BOME29.55 ± 0.0752.14 ± 2.1217.15
4Two-step GD29.24 ± 0.0187.16 ± 0.8017.06
4SUPRO26.68 ± 0.0488.69 ± 1.4618.87
8MEHA29.53 ± 0.0488.65 ± 0.4017.17
8BOME29.55 ± 0.0187.88 ± 0.9917.15
8Two-step GD29.26 ± 0.0289.11 ± 0.8017.07
8SUPRO26.93 ± 0.1390.75 ± 1.1519.25
16MEHA29.53 ± 0.0887.58 ± 2.2517.18
16BOME29.65 ± 0.0187.16 ± 2.2917.16
16Two-step GD29.31 ± 0.1088.30 ± 1.9917.11
16SUPRO27.44 ± 0.0390.37 ± 0.5320.01

SUPRO reduces the 500-step training time by 6.38%–9.70% relative to the compared methods. Its additional peak memory is approximately 1.7–2.9 GiB, accounting for the auxiliary response, candidates, and coordinate-wise states discussed in Appendix A.3.4.

The response network described in Appendix A.3.2 contains 33,539 parameters with a coordinate-GRU hidden dimension of 8. Over 500 steps, SUPRO makes 2,385 task-gradient requests, compared with 3,000 for MEHA/BOME and 2,500 for two-step GD. Its 74 predictor-backward passes are counted separately and included in the measured runtime.

For these additional experiments, prediction runs at every step; full correction and selection run during the first 30 steps and every 10 steps thereafter, giving 77 selection events. The measurements above evaluate this periodic-correction implementation.

AuthorsOpenReview ↗

Response to Reviewer ZGhU (Q2): Practical Efficiency, Memory Cost, and Scalability of SUPRO

Q2. Accuracy under fixed time budgets and repeated runs.

At a fixed budget of 25 minutes, SUPRO improves mean accuracy over two-step GD by 1.53, 1.64, and 2.06 percentage points at ranks 4, 8, and 16, respectively. Accuracy is taken from the last completed validation within the budget. The advantage therefore persists when the cost of the learned response optimizer is included.

At rank 8, SUPRO reaches 90% accuracy in all three runs, taking 20.31 ± 3.11 minutes. MEHA, BOME, and two-step GD reach this threshold in 1/3, 1/3, and 2/3 runs, respectively. We report success counts alongside target times to account for runs that do not reach the threshold.

These comparisons use the declared training settings: SUPRO doubles its main learning rate after warm-up, while the baselines retain their post-warm-up rates. They measure the performance of the complete configurations.

For the prediction component, we also report the existing controls corresponding to the variant definitions in Table 3. Both alternatives retain the learned correction and upper-guidance heads, candidate selection, and response mixing.

Prediction configurationFinal accuracy (%)Peak accuracy (%)
Full SUPRO90.82691.514
Without learned prediction residual90.82691.284
Linear extrapolation89.90891.284

In this single-seed comparison, SUPRO achieves the highest peak accuracy. The two alternatives have different final accuracies despite sharing the same peak, so the matching peak values in Table 3 do not imply identical trajectories. The multiple-seed fixed-time results above provide the broader evidence for the practical benefit of the complete method.

AuthorsOpenReview ↗

Response to Reviewer ZGhU (Q3): Practical Efficiency, Memory Cost, and Scalability of SUPRO

Q3. Sensitivity and scaling.

We varied the coordinate-GRU hidden dimension introduced in Appendix A.3.2 and the online update frequency described in Appendix A.3.3. These experiments use three seeds per configuration at rank 4. All sensitivity runs perform correction and selection at every step for the first 100 steps and every 10 steps thereafter.

Coordinate-GRU hidden dimensionFinal accuracy (%)Time (min)Peak memory (GiB)
489.83 ± 0.9330.05 ± 0.2518.19
890.18 ± 0.5730.05 ± 0.0218.88
1690.48 ± 0.5030.14 ± 0.1020.24

Runtime changes little across these hidden dimensions, while the memory measurements show the cost of larger coordinate states. The default dimension of 8 achieves 90.18% mean accuracy with 1.36 GiB less peak memory than dimension 16.

We also reduced the head and backbone update frequencies from every 1 and 2 correction events to every 2 and 4 events. This reduces predictor-backward passes from 137 to 69 and training time from 30.05 to 29.52 minutes, with mean final accuracy changing from 90.18% to 89.98%. More frequent backbone updates increase training time to 30.52 minutes without improving mean accuracy.

The rank-4/8/16 experiments extend the original rank-4 evaluation to larger adapter parameterizations. SUPRO retains its fixed-time accuracy advantage at all three ranks, with its peak memory increasing from 18.87 to 20.01 GiB.

Finally, we measured acceptance of the full learned proposal in Eq. (14) under the selection rule in Eq. (15). Acceptance rates are 38.96 ± 3.90%, 41.99 ± 10.50%, and 41.99 ± 3.00% at ranks 4, 8, and 16. These rates use the 77 selection events per run as the denominator. Every accepted proposal strictly improves both evaluated minibatch objectives relative to the safeguarded reference in these runs, showing that the learned correction and upper-guided proposal contribute to the selected responses.

We welcome further discussion and would be happy to address any remaining questions or concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.