acceptodds
Under review as a conference paper at ICLR 2027

The Path Matters: Retention-Budget Continuation for Low-Rank LLM Post-Training

Abstract

Low-rank adaptation (LoRA) enables efficient specialization of large language models, but low rank alone does not ensure capability retention. How the optimization path shapes specialization under a fixed behavioral allowance and finite computation remains unclear. We introduce Retention-Budget Continuation for LoRA (RBC-LoRA), a homotopy method that constrains forward sequence Kullback-Leibler divergence from the original model on reference prompts. A temporary quadratic factor anchor supplies a unique starting solution; continuation then strengthens the task objective, removes the anchor, and expands the retention allowance. First-order primal-dual correction and empirical acceptance checks guide progression toward the prescribed constrained endpoint while preserving adapter rank. Under local regularity and correction assumptions, we characterize solution-path sensitivity and tracking error; a divergence bound also controls changes in bounded expected scores on the reference distribution. Experiments show that RBC-LoRA improves target-task scores over independently tuned direct constrained training on math and code tasks. These gains accompany less measured degradation of pretrained behaviour, reflected in lower audit divergence and smaller retained-benchmark losses. These findings establish the optimization path as a practical design choice for improving the specialization–retention trade-off without increasing adapter capacity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Low-Rank Adaptation, Parameter-Efficient Fine-Tuning, LLM Post-Training, Catastrophic Forgetting, Homotopy Continuation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RTcgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RBC-LoRA, a framework for LoRA fine-tuning that aims to improve the trade-off between target-task specialization and retention of pretrained capabilities. Instead of directly solving the final KL-constrained optimization problem, RBC-LoRA gradually strengthens task adaptation while relaxing the retention constraint. Experiments on Qwen2.5-3B and Llama-3.2-3B using GSM8K and MBPP show that RBC-LoRA achieves higher target-task performance than direct constrained training while better preserving the model’s original capabilities.

Critical strengths
  1. The paper studies an interesting and meaningful problem. Rather than introducing another retention regularizer, it asks whether the optimization path to the same final constrained problem can improve the specialization–retention trade-off. I find this perspective novel and practically relevant.

  2. The experimental protocol is relatively careful. The paper separates training, controller, and final-audit reference sets and provides targeted ablations for the retention-budget schedule, anchor, task ramp, and stage progression. The results also show a consistent improvement over direct constrained training across the tested tasks and backbones.

Critical weaknesses
  1. The experimental scope is limited. The experiments are restricted to two 3B-scale models and primarily focus on math and code specialization. It is therefore unclear whether the proposed method generalizes to larger models or to a broader range of adaptation objectives. In addition, retention is evaluated only on MMLU, HellaSwag, and IFEval, which provides limited evidence that the original capabilities are broadly preserved.

  2. It is still unclear how important the specific continuation path is. Although the paper compares adaptive progression with fixed increments, both follow a similar increasing trajectory toward the same final constrained problem. It would be useful to know whether the proposed path itself is important, or whether other trajectories that reach the same final problem would perform similarly.

Questions for authors
  1. Does RBC-LoRA show similar gains on larger models, more diverse target tasks, and a broader set of retention benchmarks?

  2. Can the paper provide more direct evidence that following the proposed continuation path is beneficial? For example, comparing different λ\lambda trajectories that reach the same final constrained problem, including reverse or non-monotonic schedules, would help clarify whether the specific path itself is important.

2 comments
AuthorsOpenReview ↗

Response to Reviewer RTcg (1/2)

W1 and Q1: To broaden the experimental scope, we added instruction-following adaptation on Qwen2.5-7B and Gemma-2-9B-it, extending our original 3B GSM8K and MBPP settings. Using 16 A100 GPUs, we compared RBC-LoRA, LoRA, and direct constrained training on a 20k-example UltraChat subset across ranks 8, 16, and 32 with three seeds per configuration (54 trained adapters in total). Both constrained methods use forward sequence KL from the original policy to the adapted policy under the same final allowance of 2.0 nats per response (see revised paper lines 1298–1311, page 25).

This setup cleanly separates the target adaptation objective from retained capabilities: Evol-Instruct and UltraFeedback evaluate instruction following, while GSM8K, MATH, HumanEval, and MBPP measure retained math and code performance (complementing MMLU, HellaSwag, and IFEval in the 3B suite). On both IF benchmarks, each adapted model's responses are compared against GPT-3.5-turbo responses using DeepSeek-v4.1-flash as the judge (Xu et al., 2026).

The table below reports rank-16 results (mean ± sample SD over three seeds), where IF columns show win rates (WR, %) and Δ\Delta columns show percentage-point changes from the pretrained model (see revised paper Tables 7 and 8, pages 25–26).

ModelMethodEvol-Instruct WR (%) ↑UltraFeedback WR (%) ↑ΔGSM8K (pp)ΔMATH (pp)ΔHumanEval (pp)ΔMBPP (pp)
Qwen2.5-7BLoRA73.67 ± 0.2575.40 ± 0.20-1.83 ± 0.12-3.33 ± 0.06-1.20 ± 0.00-4.80 ± 0.20
Qwen2.5-7BDirect constrained73.07 ± 0.2974.80 ± 0.20-0.73 ± 0.15-1.30 ± 0.10-0.40 ± 0.35-2.00 ± 0.20
Qwen2.5-7BRBC-LoRA74.47 ± 0.2976.30 ± 0.20-0.37 ± 0.15-0.60 ± 0.10+0.20 ± 0.35-0.80 ± 0.20
Gemma-2-9B-itLoRA85.10 ± 0.2083.97 ± 0.12-1.43 ± 0.15-1.53 ± 0.06-2.80 ± 0.35-2.33 ± 0.12
Gemma-2-9B-itDirect constrained84.40 ± 0.5083.60 ± 0.20-0.57 ± 0.15-0.60 ± 0.10-0.80 ± 0.35-1.00 ± 0.20
Gemma-2-9B-itRBC-LoRA85.93 ± 0.2384.50 ± 0.20-0.27 ± 0.15-0.27 ± 0.15-0.20 ± 0.35-0.40 ± 0.20

At rank 16, RBC-LoRA improves mean IF win rates over direct constrained training by 1.40/1.50 pp on Qwen and 1.53/0.90 pp on Gemma, while simultaneously reducing forgetting across all four retained math and code benchmarks. For example, Qwen's MBPP drop decreases from 2.00 to 0.80 pp, and Gemma's HumanEval drop decreases from 0.80 to 0.20 pp.

Across all three ranks on both backbones, RBC-LoRA outperforms both baselines on both IF evaluations and matches or exceeds direct constrained training on every retained benchmark, with a single tie on Gemma rank-8 HumanEval (see revised paper Tables 7 and 8, pages 25–26). Together with the 3B results, these findings demonstrate consistent specialization and retention gains across 3B–9B models and across math, code, and instruction-following tasks.

AuthorsOpenReview ↗

Response to Reviewer RTcg (2/2)

W2 and Q2: Our component controls directly evaluate alternative trajectories that share the exact same λ=1\lambda=1 constrained endpoint, demonstrating that removing any element of the continuation path degrades target accuracy by 0.71–1.52 pp. Specifically, our ablations separate which intermediate problem family is traversed from how stage progression is paced. The fixed-increment control retains RBC-LoRA's full objective and allowance schedules, isolating the effect of stage pacing, whereas the constant-budget, no-anchor, and budget-only controls alter the intermediate optimization trajectory while preserving the final objective and allowance at λ=1\lambda=1 (see revised paper lines 430–450, pages 8–9).

With LL denoting the task loss and Q=∥θ−θ0∥2/2Q=\|\theta-\theta_0\|^2/2, RBC-LoRA optimizes λL+γ(1−λ)Q\lambda L+\gamma(1-\lambda)Q under an expanding allowance ϵλ\epsilon_\lambda. Constant-budget anchor annealing retains this objective deformation but fixes the allowance at ϵf\epsilon_f throughout; the no-anchor control optimizes λL\lambda L under ϵλ\epsilon_\lambda; and budget-only training optimizes LL under ϵλ\epsilon_\lambda. Together, these variants isolate budget expansion, the anchored start, the task ramp, and adaptive discretization along the continuation path (see revised paper lines 1084–1092, page 21).

All controls were evaluated on Qwen2.5-3B/GSM8K at rank 8 and ϵf=2.0\epsilon_f=2.0 under the shared reference sets and 24 charged GPU-hour cap. The table below summarizes the results over the original seeds 1–3, reporting target gains relative to the pretrained model (mean ± sample SD) and three-seed mean audit KL in nats per response:

VariantChange relative to the full continuation familyGSM8K gain (pp) ↑Mean audit KL ↓
RBC-LoRAFull objective and allowance schedules; adaptive pacing6.36 ± 0.221.78
Direct constrainedFinal objective and allowance throughout5.42 ± 0.351.86
Constant-budget anchorFinal allowance throughout5.53 ± 0.461.90
No-anchor task rampRemove the factor anchor5.65 ± 0.231.90
Budget-onlyRemove the anchor and task ramp4.84 ± 0.582.07
Fixed incrementsFull family at six prescribed levels5.68 ± 0.551.73
Direct + warmupDirect training with 500-step learning-rate warmup5.45 ± 0.481.90

RBC-LoRA improves target gain by 0.83 pp over constant-budget anchor annealing and by 0.71 pp over the no-anchor control. Comparing the no-anchor task ramp with budget-only training further isolates the task ramp, showing a 0.81 pp advantage. For stage pacing, fixed increments achieve a slightly lower mean audit KL than adaptive RBC-LoRA (1.73 versus 1.78), whereas adaptive progression yields 0.69 pp higher target gain and a smaller maximum retained-benchmark drop (0.90 versus 1.16 pp). Thus, pacing exhibits a clear specialization–retention trade-off, while each modification to the intermediate problem family degrades both target accuracy and audit KL (see revised paper Table 3, page 9).

By coupling increasing task pressure with anchor decay and nested feasible-set expansion toward the shared λ=1\lambda=1 endpoint, the full continuation path achieves superior endpoint quality compared with each tested trajectory and pacing alternative. We have updated Section 5.3 to explicitly separate the intermediate problem family from stage pacing and tie our path claims directly to these controlled comparisons (see revised paper lines 430–450, pages 8–9).

Reviewer_tzdKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RBC-LoRA, a retention-aware low-rank fine-tuning method that aims to balance downstream task specialization and pretrained capability retention. By introducing a homotopy continuation framework, the method progressively increases task optimization pressure and relaxes the KL-divergence constraint, while employing primal-dual correction and adaptive progression to guide training. The paper provides theoretical analyses of solution-path sensitivity and tracking behavior. Experiments on mathematical reasoning and code generation tasks demonstrate improved target-task performance and reduced capability degradation compared with direct constrained fine-tuning.

Critical strengths
  1. The paper addresses the trade-off between downstream task adaptation and pretrained capability retention during LoRA fine-tuning. Its focus on optimization trajectories offers a relatively novel and practically relevant perspective.

  2. The paper integrates homotopy continuation, KKT conditions, and path-tracking analysis to provide theoretical insights into the proposed progressive training strategy. The overall methodological framework is well structured.

  3. RBC-LoRA outperforms direct constrained training on both GSM8K and MBPP, while achieving better results on multiple capability-retention metrics. The ablation studies further support the effectiveness of its key components.

Critical weaknesses
  1. The path-tracking analysis relies on relatively strong assumptions, including local regularity and corrector contraction. However, these conditions have not been verified for the actual stochastic optimization process, leaving a gap between the theoretical analysis and practical implementation.

  2. The experiments primarily focus on two 3B models and two target tasks, GSM8K and MBPP, with only three random seeds per setting. This limits the evidence for the method's generalizability to larger models and more diverse downstream tasks.

  3. Some direct constrained training runs fail to reach the final constrained endpoint, which may affect the fairness of the comparisons. Additionally, Appendix C.4 reports inconsistencies between certain execution records and the described algorithm, which require further clarification.

Questions for authors
  1. Could the authors further investigate whether the practical optimizer satisfies the path-tracking and contraction assumptions required by the theoretical analysis, thereby strengthening the connection between the theoretical results and actual training behavior?

  2. Could the authors provide additional experiments on larger models and more diverse downstream tasks, such as HumanEval, with more random seeds to further evaluate the generalizability and stability of the proposed method?

  3. Could the authors provide a more controlled comparison that accounts for baseline runs failing to reach the final constrained endpoint? Additionally, could they clarify the inconsistencies between the execution records and the described algorithm reported in Appendix C.4?

4 comments
AuthorsOpenReview ↗

Response to Reviewer tzdK (1/3)

W1 and Q1: Because the exact solution branch z∗(λ)z^*(\lambda) is intractable in neural network fine-tuning, the theoretical contraction factor qq cannot be measured directly; instead, we assess the practical optimizer at each stage through its finite-sample feasibility and KKT residuals. Specifically, Theorem 3 assumes local distance reduction toward an ideal solution branch:

∣∣Ck(z)−z∗(λk+1)∣∣||\mathcal C_k(z)-z^*(\lambda_{k+1}) ||

≤q∥∣z−z∗(λk+1)∥∣+ξk,q<1.\leq q\||z-z^*(\lambda_{k+1})\||+\xi_k,\qquad q<1.

Here the corrector Ck\mathcal C_k includes the optimizer history encountered during training. This solution-referenced condition—distinct from contraction between arbitrary pairs of Adam states—combines with local branch regularity to yield the tracking recurrence and explain how increment size, predictor error, and correction accuracy interact (see revised paper lines 276–279, page 6).

In practice, we monitor the implemented optimizer through two empirical KL estimates and the normalized stationarity and complementarity residuals. The final diagnostic values from the original primary RBC-LoRA traces are summarized below:

SeedTraining-diagnostic KLController KLNormalized stationarityNormalized complementarity
11.7011.7020.00770.0144
21.8061.8090.00420.0183
31.8031.8040.00360.0102

Here KL is reported in nats per response. At λ=1\lambda=1, the diagnostic feasibility threshold is 1.05×2.0=2.101.05\times2.0=2.10, and the stationarity/complementarity tolerances are 0.01/0.02. Every recorded final tuple satisfies these thresholds. These diagnostics verify empirical residual control at the returned states, without estimating distance to the ideal branch or the theoretical contraction factor qq (see revised paper lines 500–508, page 10).

The controller's acceptance rule bridges theory and implementation: it checks feasibility and KKT residuals before advancing λ\lambda and halves the proposed increment upon rejection. Thus, Theorem 3 characterizes tracking under ideal branch and contraction assumptions, whereas the recorded diagnostics evaluate the finite-tolerance optimizer. We also distinguish Proposition 4—an asymptotic endpoint-stationarity result requiring vanishing raw residuals on a fixed problem—from the finite tolerances used during training (see revised paper lines 281–287, page 6).

AuthorsOpenReview ↗

Response to Reviewer tzdK (2/3)

W2 and Q2: Following your suggestion, we expanded the experiments in two directions: a five-seed evaluation of the three core methods in Table 1 and instruction-following adaptation on two larger models (7B and 9B) across multiple ranks, including HumanEval retention.

In the primary Table 1 setting (Qwen2.5-3B-Instruct on GSM8K at rank 8), we added seeds 4 and 5 for RBC-LoRA, LoRA, and direct constrained training. The table below summarizes all five runs per method (mean ± sample SD), with audit KL in nats per response and retained-benchmark changes (Δ\Delta) as adapted minus original scores in percentage points (see revised paper Table 1, page 7).

MethodGSM8K (%) ↑Audit KL ↓ΔMMLU (pp)ΔHellaSwag (pp)ΔIFEval (pp)
LoRA77.12 ± 0.456.42 ± 0.46-2.86 ± 0.67-2.40 ± 0.59-5.44 ± 1.72
Direct constrained76.84 ± 0.392.05 ± 0.30-1.06 ± 0.33-1.07 ± 0.47-1.64 ± 0.52
RBC-LoRA77.78 ± 0.341.85 ± 0.10-0.61 ± 0.26-0.56 ± 0.21-1.12 ± 0.34

RBC-LoRA outperforms direct constrained training in all five paired seeds, achieving a paired GSM8K advantage of 0.94 ± 0.13 pp, and exceeds ordinary LoRA by 0.66 ± 0.17 pp (also positive in every seed) while incurring smaller mean losses on every retained benchmark. Expanding the primary comparison to five seeds therefore confirms both the target-task and retention gains (see revised paper lines 370–375, page 7).

To evaluate larger models and broader tasks, we trained RBC-LoRA, LoRA, and direct constrained training on Qwen2.5-7B and Gemma-2-9B-it using a 20k-example UltraChat subset across 16 A100 GPUs (see revised paper lines 1298–1306, page 25). We evaluated ranks 8, 16, and 32 with three seeds per configuration, producing 54 trained adapters. Both RBC-LoRA and direct constrained training use the forward sequence KL from the original policy to the adapted policy, with a shared final allowance of 2.0 nats per response (see revised paper lines 1298–1311, page 25).

We evaluate instruction following on Evol-Instruct and UltraFeedback by comparing each adapted model's responses against GPT-3.5-turbo responses using DeepSeek-v4.1-flash as the judge (Xu et al., 2026). The table below reports rank-16 win rates (mean ± sample SD over three seeds; see revised paper Table 2, page 9).

ModelMethodEvol-Instruct win rate (%) ↑UltraFeedback win rate (%) ↑
Qwen2.5-7BLoRA73.67 ± 0.2575.40 ± 0.20
Qwen2.5-7BDirect constrained73.07 ± 0.2974.80 ± 0.20
Qwen2.5-7BRBC-LoRA74.47 ± 0.2976.30 ± 0.20
Gemma-2-9B-itLoRA85.10 ± 0.2083.97 ± 0.12
Gemma-2-9B-itDirect constrained84.40 ± 0.5083.60 ± 0.20
Gemma-2-9B-itRBC-LoRA85.93 ± 0.2384.50 ± 0.20

The same ranking holds at ranks 8 and 32: RBC-LoRA outperforms both baselines on both IF benchmarks across every model/rank configuration and paired seed. We also include HumanEval to measure retained code capability after instruction-following adaptation, alongside GSM8K, MATH, and MBPP. At rank 16, for example, HumanEval score changes are +0.20 pp for RBC-LoRA versus −0.40 pp for direct constrained training on Qwen, and −0.20 pp versus −0.80 pp on Gemma (see revised paper Tables 7 and 8, pages 25–26).

AuthorsOpenReview ↗

Response to Reviewer tzdK (3/3)

W3 and Q3: In our protocol, direct constrained training fixes λ=1\lambda=1 and ϵf\epsilon_f throughout training. Its failed endpoint status records that the final checkpoint did not pass the controller's correction/acceptance criteria, even though λ\lambda is already fixed at 11. Our primary comparison evaluates all returned adapters under the shared 24 charged GPU-hour cap, including unsuccessful returns and their consumed compute. The three failed direct runs—tight-budget Qwen/GSM8K, Llama/GSM8K, and Qwen/MBPP, each at seed 3—all consume the full 24-hour budget (see revised paper lines 377–399, pages 7–8).

To ensure a controlled comparison that accounts for these failed runs, we computed a paired sensitivity analysis from the original run records. Within each setting, we restrict both methods to the seeds where direct training completes. The resulting paired target-task differences (where positive values favor RBC-LoRA) are summarized below:

SettingPaired target advantage, all original seeds (pp)Paired target advantage, direct-completed subset (pp)Seeds in subset
Qwen2.5-3B / GSM8K, ϵf=0.8\epsilon_f=0.8+0.71+0.361–2
Qwen2.5-3B / GSM8K, ϵf=2.0\epsilon_f=2.0+0.94+0.941–3
Llama-3.2-3B / GSM8K, ϵf=2.0\epsilon_f=2.0+0.98+0.881–2
Qwen2.5-3B / MBPP, ϵf=2.0\epsilon_f=2.0+9.07+9.001–2

RBC-LoRA retains a positive target advantage in every completed-seed subset. Notably, in the primary Qwen/GSM8K setting at ϵf=2.0\epsilon_f=2.0, all three direct runs completed, leaving the +0.94 pp advantage unchanged. We retain the all-return comparison as our primary finite-compute result and report these completed-seed subsets as a descriptive sensitivity check (see revised paper lines 399–403, page 8).

The expanded five-seed comparison on Qwen/GSM8K (ϵf=2.0\epsilon_f=2.0) likewise preserves a 0.94 ± 0.13 pp paired advantage over direct training, with mean audit KL of 1.85 for RBC-LoRA versus 2.05 for direct training. Because controller completion (checked with tolerance τfeas\tau_{\mathrm{feas}} on the controller set) and held-out audit feasibility are distinct—for instance, direct seed 3 completes on the controller but records an audit KL of 2.052 against the nominal 2.0 allowance—we report controller status, held-out audit KL, and target scores separately (see revised paper Table 1, page 7, and lines 1130–1133, page 21).

Regarding the execution-record discrepancies in Appendix C.4, we traced both issues to a bug in our trajectory-logging script, fixed the export pipeline, and recovered the complete check-by-check diagnostic records. The revised Figure 5 now plots both the training-diagnostic and controller KL estimates alongside the normalized stationarity and complementarity residuals, confirming that every accepted stage and final correction across the three primary RBC-LoRA runs passes two consecutive checks within all numerical thresholds (see revised paper Figure 5, page 24, and lines 1286–1289, page 24). The recovered logs also confirm that stage progression strictly follows Algorithm 1: in seed 2, after the proposal to λ=5/6\lambda=5/6 is rejected (training and controller KL of 1.867 and 1.850 versus the 1.785 threshold), training rolls back to λ=2/3\lambda=2/3 and halves the increment from 1/61/6 to 1/121/12; the next two stages (λ=3/4\lambda=3/4 and 5/65/6) each converge in 75 updates (≤C/2=100\le C/2 = 100), triggering the cheap-acceptance rule that doubles the increment back to 1/61/6 before stepping to λ=1\lambda=1 (see revised paper lines 1290–1298, pages 24–25). Throughout our analysis, we continue to report controller completion, independent audit KL, and downstream benchmark scores as distinct evaluation metrics.

Thank you for the authors’ thoughtful response and detailed explanations. Your answers have largely addressed my concerns.

Reviewer_bt2ROpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how the training path affects the tradeoff between task specialization and capability retention in LoRA fine-tuning. The proposed RBC-LoRA gradually increases the task objective and KL allowance while removing a temporary regularization term that keeps the LoRA parameters close to their initialization. The experiments on math and code tasks show improved performance and less degradation on retained benchmarks compared with direct constrained training under comparable compute budgets.

Critical strengths
  1. The motivation is clear. The paper investigates whether a better training path can improve task performance and reduce forgetting without increasing the LoRA rank or changing the final objective.

  2. The ablation studies are reasonably thorough, examining the effects of the KL budget schedule, parameter regularization, and adaptive progression, with a learning-rate warmup baseline for comparison.

  3. The experiments consider both task performance and capability retention on two backbones. The results over three seeds and comparisons under a common compute budget help support the findings.

Critical weaknesses
  1. The theoretical analysis relies on assumptions that have not been verified for the actual training procedure. Also, the retention bound does not provide a meaningful guarantee at the main KL budget of 2.0.

  2. The experiments are limited to two 3B models on math and code tasks. Results on larger models and different LoRA ranks would help establish the generality of the method.

  3. The appendix mentions missing KL diagnostics in some recorded checks and an inconsistency in the continuation schedule. These issues need clarification, since the checks and schedule are important parts of the proposed method.

Questions for authors

see the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer bt2R (1/2)

W1: We have updated Section 4 to explicitly separate the conditional path analysis from the empirical retention measurements. The anchored starting problem has a unique solution by construction: at λ=0\lambda=0, the objective is γ∥θ−θ0∥2/2\gamma\|\theta-\theta_0\|^2/2, its Hessian is γI\gamma I, and c(θ0)=0<ϵ0c(\theta_0)=0<\epsilon_0. The path-sensitivity result applies locally under the stated KKT regularity conditions, and the tracking result additionally assumes that correction reduces distance to the solution branch. These conditions define the scope of the theory, while the practical acceptance tests assess empirical feasibility and normalized optimality residuals (see revised paper lines 218–221, page 5).

The tracking bound explains the continuation design under these assumptions:

ek+1≤q[(1+hkLv)ek+hkδk+M2hk2]+ξk.e_{k+1}\leq q\left[(1+h_kL_v)e_k+h_k\delta_k+\frac{M}{2}h_k^2\right]+\xi_k.

Here hkh_k is the continuation increment, δk\delta_k is predictor error, q<1q<1 is the assumed local correction factor, and ξk\xi_k is the additive correction error. Smaller increments reduce the predictor-error and curvature terms, while empirical correction checks determine whether a proposed stage is accepted. We therefore report empirical diagnostics separately from the ideal solution-path analysis (see revised paper lines 276–279, page 6).

We have also made the numerical scope of Proposition 5 explicit. At ϵf=2.0\epsilon_f=2.0, ϵf/2=1\sqrt{\epsilon_f/2}=1 spans the full [0,1][0,1] score range, so the bound provides no nontrivial quantitative retention guarantee at this operating point. The bound applies to the reference prompt distribution and the response policies used to define KL, whereas retention on MMLU, HellaSwag, and IFEval is measured directly through benchmark score changes (see revised paper lines 296–298, page 6).

In the expanded five-seed primary comparison, mean audit KL is 1.85 for RBC-LoRA and 2.05 for direct constrained training. The corresponding mean score changes on MMLU, HellaSwag, and IFEval are −0.61/−0.56/−1.12 pp for RBC-LoRA and −1.06/−1.07/−1.64 pp for direct training. These measurements support the empirical retention claim at the main operating point (see revised paper Table 1, page 7).

W2: Following your suggestion, we added instruction-following experiments on Qwen2.5-7B and Gemma-2-9B-it, each at ranks 8, 16, and 32, to assess generality across model sizes and adapter ranks. Using 16 A100 GPUs, we compared RBC-LoRA, LoRA, and direct constrained training over three seeds per configuration, producing 54 trained adapters.

Within each model/rank comparison, all methods share the same target-data split, adapter modules, and three-epoch schedule. Both constrained methods use forward sequence KL from the original policy to the adapted policy, with a shared final allowance of 2.0 nats per response. We hold rank fixed throughout each run (see revised paper lines 1298–1311, page 25).

We adapt the models on a 20k-example UltraChat subset and evaluate instruction following on Evol-Instruct and UltraFeedback by comparing each adapted model's responses against GPT-3.5-turbo responses using DeepSeek-v4.1-flash as the judge (Xu et al., 2026). The table below summarizes the resulting win rates (WR, %) as mean ± sample SD over seeds 1–3, and we evaluate retained capabilities on GSM8K, MATH, HumanEval, and MBPP after adaptation (see revised paper Tables 7 and 8, pages 25–26).

ModelRankDirect: Evol-Instruct WR (%)RBC: Evol-Instruct WR (%)Direct: UltraFeedback WR (%)RBC: UltraFeedback WR (%)
Qwen2.5-7B871.10 ± 0.5072.03 ± 0.4572.30 ± 0.2073.20 ± 0.20
Qwen2.5-7B1673.07 ± 0.2974.47 ± 0.2974.80 ± 0.2076.30 ± 0.20
Qwen2.5-7B3273.40 ± 0.5074.77 ± 0.4575.10 ± 0.2076.70 ± 0.20
Gemma-2-9B-it882.13 ± 0.4583.20 ± 0.5282.10 ± 0.2082.70 ± 0.20
Gemma-2-9B-it1684.40 ± 0.5085.93 ± 0.2383.60 ± 0.2084.50 ± 0.20
Gemma-2-9B-it3284.87 ± 0.4586.37 ± 0.2983.70 ± 0.2084.70 ± 0.20

Across all six model/rank configurations, RBC-LoRA exceeds direct constrained training by 0.93–1.53 pp on mean Evol-Instruct win rate and 0.60–1.60 pp on mean UltraFeedback win rate. It also outperforms ordinary LoRA on both IF evaluations in every configuration, by 0.47–1.03 pp and 0.40–1.00 pp, respectively, with consistent improvements in every paired seed. On all four retained benchmarks, RBC-LoRA matches or exceeds direct constrained training in every model/rank configuration, with a tie on HumanEval for Gemma at rank 8 (see revised paper Tables 7 and 8, pages 25–26).

These experiments extend our empirical evidence to 7B–9B models and instruction-following adaptation across multiple LoRA ranks.

AuthorsOpenReview ↗

Response to Reviewer bt2R (2/2)

W3: Thank you for flagging the missing KL fields and schedule discrepancy in Appendix C.4. Upon inspecting our trajectory-export pipeline, we identified and fixed a logging bug that had omitted intermediate KL entries and misrecorded stage transitions. We have recovered the complete check-by-check records and updated both the trajectory plot and the acceptance-and-rollback analysis. The revised Figure 5 now displays both the training-diagnostic and controller KL estimates alongside the normalized stationarity and complementarity residuals across all scheduled checks (see revised paper Figure 5, page 24, and lines 1286–1298, pages 24–25).

The complete logs confirm that every accepted continuation stage and final endpoint correction across the three primary RBC-LoRA runs is preceded by two consecutive 25-update checks satisfying all acceptance criteria: both KL estimates remain within 1.05ϵ(λ)1.05\epsilon(\lambda), and the normalized stationarity and complementarity residuals remain within 0.01 and 0.02, respectively (see revised paper lines 1286–1289, page 24).

The recovered records also confirm exact adherence to Algorithm 1's rollback and step-size rules. In seed 2, a proposed step from λ=2/3\lambda=2/3 to 5/65/6 fails when training-diagnostic and controller KL reach 1.867 and 1.850 (exceeding the threshold 1.05×1.700=1.7851.05 \times 1.700 = 1.785). The controller rolls back to λ=2/3\lambda=2/3 and halves the increment from 1/61/6 to 1/121/12. Because the next two stages (λ=3/4\lambda=3/4 and 5/65/6) each pass in 75 updates (≤C/2=100\le C/2 = 100), they satisfy the two-stage cheap-acceptance rule and double the increment back to 1/61/6 for the final step to λ=1\lambda=1, fully resolving the discrepancy noted in the original submission (see revised paper Figure 5, page 24, and lines 1290–1298, pages 24–25).

Thanks for the response, it addressed my concerns. I like to maintain my positive rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.