- Rating
- 1(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper proposes a method for fine-tuning LLMs on one specific domain without affecting its behaviour on other domains.
Critical strengths
(S1) The topic seems of practical interest for the fine-tuning of LLMs. I am not expert on the topic, but the proposed method seems to follow an established line of methods with some technical novelties and improved performance.
(S2) Thorough experiments across 5 backbones and 2 retained domains (math/commonsense). Detailed analysis and comparisons with a number of competing methods.
Critical weaknesses
(W1) The paper is difficult to follow. I am not expert on this particular topic but I am broadly knowledgeable about ML and LLMs so I would expect to understand more of (at least) the high-level ideas of the method. The text is actually grammatically fine on the surface, but the underlying structure, organization of ideas, and choice of technical words is really messy. I don't know if this is the reason, but in my experience this is exactly what happens in LLM-generated text (rather than LLM-edited text).
(W2) The motivation for the new method is motivated with technical arguments that differentiate it with existing methods, such as:
- "These studies motivate geometric constraints within compact task updates."
- "The methods specify reference geometry for shared weights or factors, without selecting a local retained reference for each token." However besides these technicalities, it's not clear what (if anything) is an issue with these existing methods. Is the argument for the new one that it just works better in the end? As a reader at the beginning of the paper I'm really not convinced why I should keep reading unless I'm deeply interested in these technical details.
Questions for authors
I don't have particular questions. I was not expert or particularly interested in the exact topic of this paper and I am still not. The writing is poor (albeit it appears superficially fine), and the contributions seem (to my non-expert eye) fairly technical and with limited reach. The poor writing does not help in making it appealing, such that I I don't think this paper will be particularly meaningful beyond a niche audience.
Maybe I'm wrong thouguh, in which case other more-qualified reviewers might recommend acceptance.
Response to Reviewer LBc1
Thank you for this candid feedback from the perspective of a broader ML reader. You are right that our original Introduction jumped into method-by-method technical distinctions before explaining the practical problem and the intuition behind the atlas. Below, we address your two Critical Weaknesses (W1 and W2) in order and summarize the corresponding revisions across the manuscript.
1. Presentation: High-level idea of ATLAS, experimental verification, and manuscript revisions
The goal of ATLAS is to teach a language model a new skill (such as code generation) while keeping its answers on existing skills (such as mathematics and commonsense reasoning) stable at comparable learning performance.
To make the method clear without requiring background in continual-learning subspaces, we now introduce ATLAS through the role of each component in a single shared low-rank branch (Figures 1 and 2):
Our experiments directly verify each part of this design and its practical benefit:
To make the paper accessible and self-contained, we rewrote the Abstract and Introduction (Section 1) to lead with the practical problem and high-level intuition, updated Related Work and Method (Sections 2–3) to explain the role of centers and directions in plain terms before presenting the equations, and brought the global-mean control, GLM accuracy preservation, and paired-seed intervals directly into Sections 5–7 and Appendices A–H.
2. Motivation: What is the issue with existing methods?
Even when overall benchmark accuracy changes little after fine-tuning, updates to shared weights can flip individual answers that downstream applications rely on for version-to-version consistency. Code-training examples specify the new behavior to learn, while calibration examples from an existing skill show the internal activation states associated with behavior to preserve.
Existing parameter-efficient methods use this information or modulate updates in two main ways:
The issue with these approaches is how they treat preserved-domain activations. Hidden states from a skill to preserve do not form a single uniform cloud; they occupy distinct local neighborhoods with different centers and different directions of variation. A single global projection applies the same static reference everywhere, while target-routed and token-loss methods adjust updates per token without referencing where the current token lies relative to preserved activations. This motivates routing each token to its nearby retained neighborhoods so that updates are conditioned on local centers and directions.