- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes Descent-Anchored Projected Regularization (DAPR), a single-pass optimizer intervention for retaining capabilities during LLM post-training. DAPR preserves Adam’s moment estimates and adds a state-dependent correction selected by a moving loss center: a preconditioned descent correction above the center, or a projected ascent correction with initialization anchoring at or below it. On 4B and 8B models, DAPR substantially reduces capability loss during extended SFT and approaches SAM’s retention at near-AdamW runtime. Results under DPO and on larger models are mixed. The paper also introduces LA-DAPR, which uses language-specific centers for multilingual tuning.
Critical strengths
The main SFT experiment result is strong, and the study also tests multiple model scales, which are comprehensive and attractive.
Critical weaknesses
For the detailed experiment, it does not establish that DAPR’s distinctive mechanism drives its gains. On Gemma-3-4B, a simpler anchoring plus weight-decay control comes close to DAPR (61.56 vs. 62.35 mean), and this comparison is not tested across other settings. The evidence therefore supports DAPR’s effectiveness as a package more than its proposed mechanism.
Questions for authors
Can the authors explain why DAPR lowers the top Hessian eigenvalue but raises the Hessian trace substantially on the curvature results on Gemma-3-4B? Is the proposed mechanism intended to suppress only the leading curvature direction, and how does that change relate to capability retention?
2 comments
Dear Reviewer HMrY,
Thank you for the time you spent reviewing our paper and for recognizing the SFT results and the evaluation across model scales. We also appreciate your questions about what drives DAPR's gains and how to interpret the curvature measurements. In the revised manuscript and our response, we add anchor-free DAPR and match anchoring controls to examine the role of the projected correction. We also re-evaluate directional curvature in float32 and explain why a lower curvature along the active gradient direction can coexist with a higher Hessian trace.
As the discussion period is drawing to a close, could you let us know whether our response addresses your concerns, or whether there is another point you would like us to clarify? We would be glad to respond while there is still time.
Thank you again for your time and efforts.
On behalf of all authors
Thanks for actionable suggestions
We thank Reviewer HMrY for recognizing DAPR as a single-pass internal Adam intervention. Below we address W1 and Q1 (Tables 1–2, 6, 8, 13; App. B.2.4).
Weakness 1: Isolating Projected Crossings from
Anchoring + Weight Decay&L2-SPAnchoring + weight decayretains DAPR's projected crossing update while adding λwd=0.01—and direct controls confirm that anchor-free DAPR (λbase=0) achieves 5.6×–15.6× curvature suppression on its own, whereas standaloneAdamW + L2-SPfails under Stress DPO.Gemma-3-4B) tests DAPR with and without λbase/λwd, while Tables 2 & 6 directly compareAdamW, always-on and ct-gatedAdamW + L2-SP(λbase=0.01), andDAPR w/o anchor(λbase=0) across SFT and β=0.01 Stress DPO.Without anchoring(λbase=0,λwd=0) still beats AdamW (61.00%vs.60.88%3-Mean). On SFT (Table 6),DAPR w/o anchorslashes pure-float32κdir by 15.6× (56.31→3.61) on 8B (61.66%4-Mean, matchingAdamW + L2-SP's61.74%and gated62.14%vs.62.24%AdamW and61.20%default DAPR) and 5.6× (7.22→1.29) on 4B (+3.72pp GSM8K over default DAPR). Under Stress DPO (Table 2), standaloneAdamW + L2-SPdrops 8B GSM8K to45.34%(-5.46pp below DAPR's50.80%), while on 4B DAPR beats AdamW by+4.63pp GSM8K (49.36%vs.44.73%;+10.24pp over Flooding's39.12%).Question 1: Directional Curvature Suppression vs. Background Subspace Trace
float32Pearlmutter HVPs (Table 1) verify consistent single-pass κdir reductions across every scale:7.22→1.78(-75%,1.29at λbase=0) on 4B,56.31→12.51(-78%,3.61at λbase=0) on 8B,10.76→4.65(-57%) on 12B, and3.79→0.77(-80%) on 14B at 1.07×–1.23× AdamW runtime.