- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes AuroSFT, a parameter-efficient multi-task SFT framework that reduces the overhead of full-model rollback. Building on MSFT, it replaces full-model checkpoints with lightweight adapter states and introduces nonlinear transformations to adapter weights. The proposed framework demonstrates improvement across a wide range of task settings.
Critical strengths
- Broad coverage of domains and tasks. The experiments cover multiple tasks across different domains, providing a relatively comprehensive empirical evaluation of the proposed method.
- Practical significance. Introducing PEFT into the multi-task fine-tuning rollback framework substantially reduces the number of parameters that need to be stored and restored, improving storage efficiency.
Critical weaknesses
- Limited novelty. The proposed method primarily replaces full-parameter fine-tuning in MSFT with AuroRA-based parameter-efficient fine-tuning, with the main modification being the application of the nonlinear ANL to adapter weights rather than projected sample representations.
- Marginal improvements. AuroSFT achieves only 0.08% improvement over the straightforward combination of LoRA and MSFT. Moreover, its improvements over MSFT become marginal for models larger than 1.5B parameters.
- Additional ablation studies are needed to evaluate the benefits of modification on the nonlinear ANL in AuroRA.
Questions for authors
- Clarify the contributions and novelty.
12 comments
W1. Limited novelty beyond mSFT and AuroRA-based parameter-efficient fine-tuning [1/2]
W1. Limited novelty beyond mSFT and AuroRA-based parameter-efficient fine-tuning [1/2]
Concern. The method primarily replaces full-parameter fine-tuning in mSFT with AuroRA-based parameter-efficient fine-tuning, and its main modification appears to be applying ANL to adapter weights rather than projected sample representations.
We appreciate this criticism because it helps us identify an overbroad presentation of novelty in the original manuscript. Our direct response is that the contribution is the formulation, diagnosis, and repair of scheduler-carried adapter state. We study the adapter as a time-dependent object under eviction and rollback, measure post-exclusion erosion, introduce group-protected state transitions, and evaluate structure-matched and negative controls.
The inherited components are not counted as new contributions. mSFT supplies the task scheduler and AuroRA-S already provides a closely related static weight-side adapter. We do not claim the scheduler, the static adapter equation, input linearity, rank boundedness, or exact mergeability as independent new contributions. The research question begins with the state transitions that those inherited properties do not address.
A direct comparison with prior work clarifies the resulting research focus:
| Work | Dynamic task-wise scheduling | Compact carried state | Task-peak rollback | Post-eviction state-erosion analysis | Group-protected rollback |
|---|---|---|---|---|---|
| mSFT | Yes | No; full-model state | Yes | Not its main research object | No |
| Standard LoRA | No | Yes | Not its main research object | Not its main research object | No |
| AuroRA-S | No | Yes | Not its main research object | Not its main research object | No |
| Our revised focus | Inherited from mSFT | Adapter state | Adapter-only, stage-dependent | Yes | Yes, through F2 |
The research challenge begins after the components are combined. A static adapter under a fixed schedule only needs to represent the current model update. An adapter inside mSFT must instead represent a sequence of stage-dependent states. The scheduler evicts tasks according to validation peaks, selects a checkpoint associated with one task, and continues training on the surviving mixture. This creates difficulties that cannot be solved by storage compression alone:
- Peak interpretation: an epoch-0 peak followed by degradation can indicate cross-task interference rather than genuine overfitting.
- Zero-gradient erosion: an excluded task receives no further gradient, but surviving tasks continue modifying the shared low-rank factors on which its predictions depend.
- Rollback mismatch: a checkpoint selected for one task may not be the best state for another task.
- Unbounded stagnation: later stages can consume substantial stage budget without improving the global-best checkpoint.
At the theoretical level, the submitted propositions establish state sufficiency and an accounting framework. The state-sufficiency argument shows that a frozen backbone makes the adapter state and metadata sufficient to reconstruct a stage projection. The peak-to-final decomposition then separates task-peak gain from subsequent stage-transfer change, providing a way to measure whether the peak gain survives. Neither proposition by itself solves the interaction failures above.
Direct mechanistic analysis motivates the new state-aware repairs. F2 introduces block-diagonal task-group adapters, freezes a group after all its tasks have been evicted, and rolls back only active groups. F3 addresses stagnation by stopping after two consecutive stages without global-best improvement or after two whole-stage rollbacks to epoch 0. F1 changes the interpretation of epoch-0 peaks, although our attribution control shows that its measured benefit should not be attributed specifically to correct victim identification.
Direct empirical evidence supports a nuanced contribution rather than a universal new adapter claim. F2 improves the fresh Qwen2.5-3B base by 0.60 overall points and 1.25 mathematics points. F2+F3 improves the OLMo2-1B base by 4.10 overall points and 4.25 mathematics points. Conversely, repairs underperform the base in several smaller-backbone configurations. This combination of positive and negative findings helps identify when protected state is useful and when rank splitting or early termination costs accuracy.
W1. Limited novelty beyond mSFT and AuroRA-based parameter-efficient fine-tuning [2/2]
W1. Limited novelty beyond mSFT and AuroRA-based parameter-efficient fine-tuning — continued [2/2]
Complementary auxiliary evidence comes from the submitted mixture-scale and MedMCQA experiments. AuroSFT improves over fixed-mixture SFT by 11.9, 0.7, and 0.6 points at mixture sizes 5, 10, and 15, respectively. The MedMCQA analysis reports a 1.25-point overall improvement over SFT; in its 11-group display, AuroSFT is no worse than reproduced mSFT on 10 groups and strictly better on 8. The reported overall improvement follows the original aggregation rather than an unweighted mean of these 11 displayed bars. These results support the broader value of adapter-state continuation, but we do not use them to claim that every component is uniformly beneficial.
We separate inherited properties from the proposed investigation. mSFT supplies the task-wise schedule; LoRA supplies compact low-rank adaptation; AuroRA-S already supplies the closely related static weight-side nonlinear form. Input linearity, rank boundedness, and mergeability are not counted as new mathematical contributions. The revised method explanation uses these inherited properties to make stage-state reconstruction precise, then studies subsequent transitions that those properties do not protect.
The proposed structure is not an unconditional no-forgetting guarantee. F2 separately parameterizes task groups, freezes a group after its last task is evicted, and later rollback restores only the active groups. This protects the saved factors themselves; changing active blocks elsewhere in the same forward network can still alter an excluded task's predictions. Similarly, an epoch-0 peak is a diagnostic signature compatible with interference rather than proof of its unique cause. The revised contribution is an empirical and structural investigation of this interaction, with these qualifications stated explicitly.
Group separation does not mean task-conditional computation. The implemented forward pass sums all group updates for every input, so an active task can update every unfrozen group. The block-diagonal description concerns the latent parameter-state organization, not separate input/output coordinates or per-task routing. Explicit freezing and selective rollback protect a fully evicted group's factors; no structural claim of zero cross-task influence is made for the complete network.
Changes in the revised manuscript. Section 1, “Introduction”; Section 2, “Related Work”; Section 4.5, “Diagnosis: Where the Shared-Adapter Schedule Loses Mathematical Tasks”; Section 5, “Discussion and Limitations”; Appendix “Rebuttal Evidence and Reporting Corrections,” subsection “State Protection and Audited Replication.”
Revised manuscript text.
The static weight-side adapter used here has essentially the same functional form as AuroRA-S. Input-linearity, rank boundedness, and exact mergeability are inherited properties of that static family, rather than independent novelty claims. Our dynamic setting asks when its state should be saved, which blocks should be restored, and how retained-task accuracy changes after exclusion.
Compact state does not guarantee retained knowledge. A task with an epoch-0 peak can be excluded even when its immediate decline reflects interference rather than ordinary overfitting. Once excluded, it receives no gradient, but surviving tasks keep changing the shared adapter. A rollback chosen for one task can also discard useful progress for another. Our contribution is to measure these interactions and evaluate state-aware repairs, with controls that separate state restoration, protected structure, and nonlinear optimization.
Group factors are separately parameterized; every input uses their sum. All unfrozen blocks can receive gradients from active tasks. A group is frozen once all its assigned tasks are excluded, and rollback restores only still-active groups.
Response to Reviewer 6XMw — Overview and index
Response to Reviewer 6XMw
We thank the reviewer for critically assessing the novelty and empirical scope of our work. The comments prompt us to clarify which components are inherited, which research questions arise only when those components interact, and where our method does or does not provide measurable benefits. They also encourage us to strengthen the ablation evidence and to report the limits of scalability with the same prominence as the positive results.
Throughout this response, accuracies are percentages and differences are percentage points. Overall accuracy is the unweighted mean of the ten benchmark accuracies; mathematics is the mean of GSM8K and AQUA. Submitted results, fresh experiments, structure-matched adapter controls, and trajectory analyses are identified separately. New results and reporting corrections are incorporated into the revised manuscript; section and table references below refer to that revision.
The revised interpretation separates three objects: adapter-only rollback determines which state is restored; F2 determines which task-group factors remain mutable; ANL determines how the update is parameterized and optimized. F1 is the eviction-rule intervention, F1ctl its fixed non-victim protection control, F2linear the linear version of the protected structure, and F3 the stagnation-based termination rule. None of these names implies a general guarantee of improved accuracy.
The evidence freeze excludes the budget-matched full-model mSFT control and additional fresh F2/F2linear training-seed lines that have not yet passed completion and provenance review. No intermediate value from these lines is used below. A result will be added only after its actual training seed, stage chain, final checkpoint, and same-checkpoint evaluation records agree; if a line fails, stops, or remains unverifiable, we will report that status rather than leave an unfulfilled experimental claim.
Reading guide
This response is divided into threaded comments to comply with the 5,000-character limit. No substantive response content is omitted. Each W/Q thread begins with the direct answer; continuation comments retain the manuscript locations, complete tables, derivations, and revised-manuscript excerpts in their original order.
There are 4 W/Q threads and 9 W/Q comments in total:
Continuation titles use the form
[part/total]. Please read each numbered thread in ascending part order.