acceptodds
Under review as a conference paper at ICLR 2027

Learn Here, Move Less Elsewhere: Input-Conditioned Plasticity from Retained-Domain Activation Atlases

Abstract

Task-specific fine-tuning can rewrite a language model's answers beyond the training task, complicating updates that must preserve existing behavior. We introduce ATLAS, which turns retained-domain representations into an input-dependent rule for task adaptation. An activation atlas supplies local reference centers and directional filters to a shared low-rank residual. Target supervision learns the residual, while retained geometry shapes its action throughout training and inference. On Qwen3-8B, ATLAS achieves lower mean retained-output Kullback–Leibler (KL) divergence than all seven published baselines at shared coding-performance requirements, with consistent advantages across multiple training seeds. Structural comparisons identify the contributions of retained reference states and directional conditioning, and answer-level analyses show fewer rewritten mathematical answers and more stable commonsense choices. Experiments spanning five backbones and two retained domains further demonstrate coding gains with reduced retained-output movement. With compact storage and modest decoding overhead, ATLAS provides a practical mechanism for acquiring specialized skills while maintaining continuity in existing responses.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
parameter-efficient fine-tuning, retention, activation geometry, input-conditioned adaptation, language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LBc1OpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper proposes a method for fine-tuning LLMs on one specific domain without affecting its behaviour on other domains.

Critical strengths

(S1) The topic seems of practical interest for the fine-tuning of LLMs. I am not expert on the topic, but the proposed method seems to follow an established line of methods with some technical novelties and improved performance.

(S2) Thorough experiments across 5 backbones and 2 retained domains (math/commonsense). Detailed analysis and comparisons with a number of competing methods.

Critical weaknesses

(W1) The paper is difficult to follow. I am not expert on this particular topic but I am broadly knowledgeable about ML and LLMs so I would expect to understand more of (at least) the high-level ideas of the method. The text is actually grammatically fine on the surface, but the underlying structure, organization of ideas, and choice of technical words is really messy. I don't know if this is the reason, but in my experience this is exactly what happens in LLM-generated text (rather than LLM-edited text).

(W2) The motivation for the new method is motivated with technical arguments that differentiate it with existing methods, such as:

  • "These studies motivate geometric constraints within compact task updates."
  • "The methods specify reference geometry for shared weights or factors, without selecting a local retained reference for each token." However besides these technicalities, it's not clear what (if anything) is an issue with these existing methods. Is the argument for the new one that it just works better in the end? As a reader at the beginning of the paper I'm really not convinced why I should keep reading unless I'm deeply interested in these technical details.
Questions for authors

I don't have particular questions. I was not expert or particularly interested in the exact topic of this paper and I am still not. The writing is poor (albeit it appears superficially fine), and the contributions seem (to my non-expert eye) fairly technical and with limited reach. The poor writing does not help in making it appealing, such that I I don't think this paper will be particularly meaningful beyond a niche audience.

Maybe I'm wrong thouguh, in which case other more-qualified reviewers might recommend acceptance.

1 comment
AuthorsOpenReview ↗

Response to Reviewer LBc1

Thank you for this candid feedback from the perspective of a broader ML reader. You are right that our original Introduction jumped into method-by-method technical distinctions before explaining the practical problem and the intuition behind the atlas. Below, we address your two Critical Weaknesses (W1 and W2) in order and summarize the corresponding revisions across the manuscript.

1. Presentation: High-level idea of ATLAS, experimental verification, and manuscript revisions

The goal of ATLAS is to teach a language model a new skill (such as code generation) while keeping its answers on existing skills (such as mathematics and commonsense reasoning) stable at comparable learning performance.

To make the method clear without requiring background in continual-learning subspaces, we now introduce ATLAS through the role of each component in a single shared low-rank branch (Figures 1 and 2):

  • Offline neighborhood summary (charts): Before fine-tuning, we sample hidden states from calibration examples of a skill to preserve (such as GSM8K math problems) and group them into KK local neighborhoods, termed charts. Each chart stores its centroid and its top principal directions of variation.
  • Reference centers: During training and inference, each token finds its nearest retained charts and centers its state relative to their centroids before entering the adapter. The adapter thus computes updates relative to nearby retained states.
  • Directional filtering and scaling: Using the local variation directions of those same charts, the branch attenuates components along directions where retained activations vary and scales the update amplitude from the filtered state. All charts share the same trainable low-rank matrices, so adding charts adds no trainable parameters.

Our experiments directly verify each part of this design and its practical benefit:

  • Role of local centers and directions (Section 6, Table 2, Appendix D.4): On Qwen3-8B at matched coding scores, centering each token at its routed local reference reduces distributional drift (retained KL) by 38.44% relative to an uncentered residual and by 11.01% relative to subtracting the single global mean of the calibration activations. Adding directional filtering and amplitude scaling in full ATLAS reduces KL by a further 4.64% beyond routed centering.
  • Preserving accuracy when fine-tuning causes forgetting (Section 5, Appendix E.4, Table 20): On GLM-4-9B, where ordinary fine-tuning and ATLAS solve the exact same numbers of coding problems across three seeds (67.26% MBPP+), ordinary fine-tuning drops mathematical accuracy from 63.68% to 54.69%, whereas ATLAS retains 62.12%, preserving an additional 7.43 percentage points of accuracy.

To make the paper accessible and self-contained, we rewrote the Abstract and Introduction (Section 1) to lead with the practical problem and high-level intuition, updated Related Work and Method (Sections 2–3) to explain the role of centers and directions in plain terms before presenting the equations, and brought the global-mean control, GLM accuracy preservation, and paired-seed intervals directly into Sections 5–7 and Appendices A–H.

2. Motivation: What is the issue with existing methods?

Even when overall benchmark accuracy changes little after fine-tuning, updates to shared weights can flip individual answers that downstream applications rely on for version-to-version consistency. Code-training examples specify the new behavior to learn, while calibration examples from an existing skill show the internal activation states associated with behavior to preserve.

Existing parameter-efficient methods use this information or modulate updates in two main ways:

  • Subspace-oriented methods use a single global summary of activations or weights: CorDA uses activation covariance to orient weight decomposition and learns the smallest singular components in its knowledge-preserving mode, while LoRA-Null initializes adapters in activation null spaces.
  • Input-conditioned and token-level methods adjust adaptation per token: TopLoRA generates token-specific diagonal transformations within shared low-rank factors, SLIM routes tokens using distances to target-task clusters, and STM and TALR mask or reweight target-token losses.

The issue with these approaches is how they treat preserved-domain activations. Hidden states from a skill to preserve do not form a single uniform cloud; they occupy distinct local neighborhoods with different centers and different directions of variation. A single global projection applies the same static reference everywhere, while target-routed and token-loss methods adjust updates per token without referencing where the current token lies relative to preserved activations. This motivates routing each token to its nearby retained neighborhoods so that updates are conditioned on local centers and directions.

Reviewer_PfV1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ATLAS, which attempts to teach a language model a new skill while leaving other outputs on other domains as unchanged as possible. It fits K-Means centers and local PCA bases on retained-domain activations at one layer. A small trainable residual then acts on each token relative to its nearest region’s center, steering away from those directions so the update stays out of the way of existing behavior. The authors use 3 seeds and compare methods at matched coding scores. The paper reports 10–17% lower retained KL than seven published baselines on Qwen3-8B. Across five backbones it reports about 62% lower KL than internal controls, plus fewer changed math and commonsense answers. Ablations attribute most of the gain to centering rather than the local directions.

Critical strengths
  1. The biggest strength is the design and thoroughness of the experiments. The authors have done multiple seeded runs across 5 epoch trajectories. There is no cherry picking of arbitrary checkpoints. This stands out to other PEFT papers by not saying behavior is preserved by just not learning as much. Also makes the KL comparison s more trustworthy.

  2. The controls in this paper are fantastic. The ablations break down the full method into parts (centering, distance scaling, input and output filtering), and the effective-norm and random-basis experiments test whether the local geometry matters beyond just shrinking the update. I also greatly appreciate the item level churn reporting in Fig. 4.

Critical weaknesses
  1. In Table 11, every method scores within half a point on GSM8K (86.10-86.58%), even though their retained KL differs by up to 8x. In this setting, lower KL doesn’t translate into better-preserved math ability. Where answers do change, ATLAS also blocks improvements: it trails comparators by 1-2 points on Qwen3.5 GSM8K (F.3), nearly 10 points against Replay on GLM (Table 19), and 6.5 points on CSQA short generation (Table 21). On Qwen3-4B, it also learns the target task much less (about 15% vs. 36% MBPP+, Table 6). None of this appears in the main text.

  2. Table 2 shows that subtracting a routed retained center accounts for most of the KL reduction (38.4% vs. 41.3% for full ATLAS). The local PCA directions add only 0.5-4.6%, comparable to the roughly 0.8% seed-to-seed spread in Table 12, and the input-filter gain has one unfavorable seed. On GLM, the only non-Qwen model, a single global chart outperforms the local atlas by 3-4x (Table 8). The evidence supports a simpler mechanism, center-based attenuation of the update near retained data, than the geometric framing in the title and introduction.

  3. There is one target task (120 MBPP problems, one layer), and the atlas is fit and evaluated on the same retained distribution (Table 3), so protection of unseen domains is untested. The three seeds vary only training. The atlas comes from a single K-means run, and learning-rate selection, coefficient sweeps, and decoding sensitivity each use one seed. Against the published baselines, the only uncertainty estimates for answer changes (Table 20) cross zero for everything except within-error changes. Meanwhile, the 67.64% figure cited in Section 5 comes from a less controlled comparison than the matched-score analysis (C.2).

Can you show a setting where ordinary fine-tuning actually damages the retained skill, and ATLAS prevents that damage in accuracy, not just KL?

Questions for authors

What is the frozen Qwen3-8B model’s GSM8K accuracy under your evaluation protocol? Does any method in Table 11 fall meaningfully below it?

Since centering alone captures most of the gain, can you give cross-seed confidence intervals for the component ablations in Table 2? Did you sweep K, q, p, and tau?

On Qwen3-4B, validation-selected ATLAS reaches about 15% MBPP+ versus 36% for Plain. Does ATLAS consistently slow target learning?

Have you tried a setting where Plain fine-tuning measurably hurts retained accuracy, for example with more training data, more steps, or adaptation at several layers? Does ATLAS’s advantage hold, grow, or shrink there?

ATLAS suppresses wrong-to-correct transitions about as much as correct-to-wrong ones, which leaves it 1-2 points behind comparators on Qwen3.5 GSM8K and about 10 points behind Replay on GLM. Is that the intended behavior?

4 comments
AuthorsOpenReview ↗

Response to Reviewer PfV1 — Part 1 of 2

Thank you for recognizing the rigor of our trajectory-level evaluation and controls, and for pointing out that key accuracy and behavioral results were previously in the appendices while the main text over-emphasized directional filtering relative to centering. We have brought the accuracy-preservation results, the centering hierarchy, and the answer-transition trade-offs directly into Sections 5–7. Following your five questions in order, Part 1 addresses Q1–Q3, and Part 2 addresses Q4–Q5 and the remaining points from your Critical Weaknesses.

Q1. Frozen Qwen3-8B GSM8K accuracy and Table 11 methods

Under our evaluation protocol, the frozen Qwen3-8B model's GSM8K accuracy is 86.35% (1,139/1,319), now added to the top row of Table 11 (Appendix C.2). Across all methods in Table 11, GSM8K accuracy remains between 86.10% and 86.58%, close to the frozen baseline, even as retained KL varies by a factor of more than eight (1.2876×10−31.2876 \times 10^{-3} for ATLAS compared with 10.7280×10−310.7280 \times 10^{-3} for CorDA-KPM). Thus, the five-epoch single-layer budget on Qwen3-8B characterizes distributional stability when accuracy remains near its starting level. By contrast, on GLM-4-9B (see Q4 in Part 2), the same protocol causes Plain fine-tuning to lose 9.00 percentage points on GSM8K, and ATLAS recovers 7.43 percentage points of that loss at identical coding performance.

Q2. Cross-seed intervals for Table 2 ablations, centering, and hyperparameter sweeps (K,q,p,τK, q, p, \tau)

We computed paired effects within each seed over the ten requirements in Table 2 (151–160 solved MBPP+ problems) and formed descriptive 95% Student-tt intervals (df=2df = 2, normality assumption on log ratios, no multiplicity adjustment; Section 6 and Appendix D.5, Table 14):

Comparator in Table 2Mean KL (×10⁻³) ↓ATLAS KL reduction (%)Paired-seed 95% interval (%)
Residual Plain1.971841.30[30.30, 50.55]
Centered residual1.21384.64[3.41, 5.86]
Distance-scaled residual1.18492.31[0.65, 3.94]
PCA-derived scaling1.16560.70[-0.55, 1.93]
Input filter1.16360.52[-0.66, 1.69]
Output filter1.17381.39[-0.43, 3.18]
Both filters, no scaling1.17201.24[-0.36, 2.82]
Global-PCA (K = 1)1.19252.93[1.70, 4.15]
Native Weight-LoRA1.330212.98[8.72, 17.05]

Pairing accounts for variation shared by methods trained with the same seed, conditional on the fixed atlas and evaluation set. As you noted, routed centering provides the largest gain over Plain, and we revised Section 6 to foreground this hierarchy. To test what local routing adds beyond general centering, we ran a global-mean centering control, Btanh⁡(A(h−hˉ))B\tanh(A(h - \bar{h})), using the global mean hˉ\bar{h} of the same 4,096 calibration activations (Appendix D.4, Table 12): mean retained KL is 1.3640×10−31.3640 \times 10^{-3} for global-mean centering, 1.2138×10−31.2138 \times 10^{-3} for routed centering (11.01% lower, with seed reductions of 13.25%, 12.91%, and 6.71%), and 1.1575×10−31.1575 \times 10^{-3} for full ATLAS (15.14% lower than global mean). Lowering every routed-center coding score by one problem and raising every global-mean score by one retains a 9.27% KL reduction. Meanwhile, the close KL point estimates of PCA-derived scaling (1.1656×10−31.1656 \times 10^{-3}) and Input filter (1.1636×10−31.1636 \times 10^{-3}) show that these compact variants retain most of the full construction's KL reduction, while their narrow intervals describe the uncertainty range of the remaining incremental difference.

Regarding hyperparameter sweeps: Appendix F.2 evaluates nine ATLAS configurations across learning rates {10−4,2×10−4,4×10−4}\{10^{-4}, 2\times 10^{-4}, 4\times 10^{-4}\} and scales s∈{32,64,128}s \in \{32, 64, 128\} on seed 7302026 (Table 24 reports the equal-candidate comparison), holding K=32,q=4,p=16K = 32, q = 4, p = 16, and τ=64\tau = 64 fixed. Comparing K=32K = 32 with its K=1K = 1 special case (Global-PCA) across all four Qwen backbones on GSM8K and Qwen3-8B on CSQA shows that K=32K = 32 wins 14 of 15 seed-level comparisons, whereas GLM-4-9B favors K=1K = 1 (Table 8).

Q3. Target learning on Qwen3-4B and across backbones

Across the five backbones, independent selection by target-validation NLL gives the following MBPP+ pass@1 accuracies (Appendix B, Table 6): Qwen3-4B (Plain 35.71%, ATLAS 14.73%), Qwen3-8B (Plain 72.17%, ATLAS 71.43%), Qwen3.5-9B (Plain 63.54%, ATLAS 65.62%), Qwen3-14B (Plain 71.13%, ATLAS 72.62%), and GLM-4-9B (Plain 65.77%, ATLAS 70.54%). ATLAS reaches higher validation-selected coding scores on three backbones and a similar score on Qwen3-8B; the lower endpoint is specific to Qwen3-4B under the five-epoch, single-layer protocol.

AuthorsOpenReview ↗

Response to Reviewer PfV1 — Part 2 of 2

Q4. Setting where ordinary fine-tuning hurts retained accuracy and ATLAS prevents that damage

The GLM-4-9B experiments provide a direct setting where unconstrained Plain fine-tuning causes severe forgetting on GSM8K and ATLAS prevents most of that accuracy loss at identical coding performance. We have moved these results into Section 5 and expanded Appendix E.4 (Table 20). Across the three seeds, the target-matched Plain and ATLAS checkpoints solve the exact same numbers of MBPP+ problems: 156, 148, and 148 out of 224 (Appendix E.1, Table 17).

GLM-4-9B stateMean MBPP+ (%) ↑Mean GSM8K (%) ↑Harmful flips (F) ↓Corrections (G)
Frozen starting model36.6163.68——
Plain fine-tuning67.2654.69651295
ATLAS67.2662.12183121

At matched 67.26% coding accuracy, Plain fine-tuning reduces GSM8K accuracy by 9.00 percentage points (from 63.68% to 54.69%, with 651 correct-to-wrong flips against 295 corrections). In contrast, ATLAS limits the accuracy drop to 1.57 percentage points (retaining 62.12%) and reduces harmful flips by 71.9% (from 651 to 183). ATLAS thus preserves an additional 7.43 percentage points of mathematical accuracy relative to Plain (0.23, 11.90, and 10.16 percentage points across the three seeds).

Q5. Suppression of wrong-to-correct transitions and behavior on Qwen3.5, GLM Replay, and CSQA

Because ATLAS trains on code supervision without a retained-task supervised loss, constraining the adapted model to stay close to the starting distribution dampens unguided transitions in both directions (FF and GG). We discuss this trade-off in Section 7 and Appendix E:

  • Qwen3.5-9B GSM8K (Section 7, Table 19): Across the nine matched pairs, the comparators produce 740 harmful flips and 1,108 corrections; ATLAS produces 297 harmful flips (−59.86%-59.86\%) and 448 corrections (−59.57%-59.57\%), cutting total answer churn in half (from 22.43% to 10.18%). Numeric-normalized GSM8K accuracy rises from 67.93% (frozen) to 69.20% for ATLAS and 71.03% for the comparators, trading 1.83 points of unguided net flips for a 54.60% reduction in rewritten answers.
  • GLM-4-9B paired with Replay (Appendix E.4, Table 20): Replay adds supervised updates sampled from the 2,048-example GSM8K buffer during code adaptation. ATLAS produces fewer harmful flips, 183 compared with 313 (−41.5%-41.5\%), and fewer corrections, 121 compared with 632; the resulting GSM8K accuracy is 9.63 percentage points lower than Replay and 7.43 percentage points higher than Plain.
  • CSQA conditional choice and short generation (Appendix E.6, Tables 22–23): Conditional-choice accuracy over options A–E is 85.22% for ATLAS and 85.20% for the comparators (85.50% frozen), with choice churn reduced by 38.42% (0.85% compared with 1.39%). Under two-token short generation, the parser marks 24.49% of frozen Qwen3-8B outputs as invalid because they do not parse to a standalone letter A–E, yielding 65.11% parsed accuracy. After code fine-tuning, the comparators convert 11.70% of all questions from invalid to valid option-letter outputs (74.66% parsed accuracy); ATLAS converts 4.02% from invalid to valid format (68.16% parsed accuracy) while cutting raw-text generation churn from 12.80% to 5.38%.

Additional Clarifications on Critical Weaknesses (W3)

  • Calibration and evaluation splits: On GSM8K and CSQA, the atlas is fit on 2,048 training-split examples (using a 10-initialization K-means run on seed 7302026) and evaluated on the disjoint 1,319-question GSM8K test set and 1,221-question CSQA validation set (Table 3), testing unseen questions within the calibrated retained domain.
  • LoRA-Null and OPLoRA churn (Table 21): On Qwen3-8B, correctness-transition totals (F+GF+G) are close at 248 (ATLAS) and 251 (comparators), while changes between distinct incorrect answers (WW) fall from 123 to 96 (21.95% reduction, 95% bootstrap interval [3.03%,40.68%][3.03\%, 40.68\%]), supporting the within-error stability result in Section 7.
  • 67.64% in Section 5: Our primary baseline result is the 10.16%–17.60% lower KL at shared coding requirements across all seven baselines (Table 1). For independent validation selection (Table 11), Section 5 states first that ATLAS reduces retained KL by 15.47% to 88.00% across the eight evaluated configurations and achieves lower KL in all 24 seed-level comparisons; within the LoRA-Null and OPLoRA subset, the geometric mean KL reduction is 67.64%, with a mean ATLAS-minus-comparator MBPP+ difference of −0.10-0.10 percentage points.

Thank you for the detailed response and the added analyses. The GLM-4-9B result is the strongest new evidence: at matched coding scores, ATLAS limits the GSM8K drop to 1.57 points versus 9.00 for Plain, which is the kind of accuracy-level protection I asked for. However, several concerns remain. That effect comes from a single backbone with large seed variance (0.23 to 11.90 points), and it is the backbone where Global-PCA achieves 3-4x lower KL than the local atlas. Replay, using the same retained data, ends about 9.6 points higher on GSM8K. The new cross-seed intervals also confirm that the directional filters are not distinguishable from zero, so centering accounts for most of the effect. K, q, p and tau remain unswept, and protection of unseen retained domains is still untested. I appreciate the rigor of the evaluation, but because the evidence supports center-based attenuation and churn reduction more than the local-geometry framing in the title and method, I am keeping my score.

AuthorsOpenReview ↗

Replying to Response to Reviewer PfV1

Thank you for reading our rebuttal and explaining the concerns that still inform your rating. Since posting our initial response, we have continued running experiments to address those points. We have now completed the cross-domain evaluation, the accuracy comparison at matched coding performance, and the atlas sensitivity sweep. We also compared chart granularity on a separate retained validation set.

Transfer to an uncalibrated domain. We evaluated the Qwen3-8B checkpoints trained with the GSM8K atlas directly on all 1,221 CSQA validation questions, with no CSQA calibration or refitting. In each of three seeds, both ATLAS and Plain solved exactly 160/224 MBPP+ problems. ATLAS reduced CSQA KL by 69.08% and option changes from the frozen model from 1.42% to 1.09% (23.08% fewer). Conditional choice accuracy was 85.07% for ATLAS and 84.71% for Plain, with all three seeds favoring ATLAS on KL, option changes, and conditional accuracy. Short-generation accuracy was 73.90% for ATLAS, 79.42% for Plain, and 65.19% for the frozen model. The transfer result concerns conditional choice and output stability.

Accuracy at matched coding performance. On those same Qwen3-8B checkpoint pairs, GSM8K answer changes fell from 5.81% to 3.66% (36.96% fewer), and correct-to-wrong transitions fell from 82 to 55 across the three seed pairs. GSM8K accuracy was 86.10% for ATLAS, 86.38% for Plain, and 86.20% for the frozen model in this evaluation. On GLM, ATLAS reduced the GSM8K accuracy loss from 9.00 to 1.57 percentage points at matched coding performance. Across the three GLM pairs, ATLAS scored 61.87%, 62.47%, and 62.02%, while Plain scored 61.64%, 50.57%, and 51.86%. The spread in the paired gains largely reflects variation in Plain's accuracy across these seeds. Replay uses the same 2,048 retained examples as supervised training data. In the GLM Replay comparison, ATLAS produced 183 harmful flips and 121 corrections, compared with Replay's 313 flips and 632 corrections. These counts account for Replay's higher accuracy and ATLAS's greater answer continuity in this comparison.

What the geometric controls measure. Selecting a retained reference for each token is itself part of the local geometry. With PCA filtering and amplitude scaling disabled in both controls, routed centering lowers KL by 11.01% relative to global-mean centering, with reductions in all three seeds. The intervals crossing zero for full ATLAS relative to input-only filtering or PCA-derived scaling quantify the gain over variants that already use local PCA. In a separate control that keeps the routing structure and replaces the fitted directions with random directions, the fitted directions yield lower KL in all 11 comparisons, with a 5.53% aggregate reduction.

Atlas sensitivity and chart granularity. Varying one configuration parameter at a time around (K, q, p, tau) = (32, 4, 16, 64), with K in {16, 64}, q in {2, 8}, p in {8, 32}, and tau in {32, 128}, changed mean retained KL by at most 1.77% over the shared requirements of 145–160 solved MBPP+ problems on Qwen3-8B. This sweep fits each local basis using an equal number of nearest calibration vectors per chart at each K. Across three training seeds, varying K changed mean KL by at most 2.28% relative to K = 32 over each seed's shared requirements. The q, p, and tau comparisons use one training seed. In the separate structure study, a rule based on target validation NLL and held-out retained KL selected K = 32 for Qwen3-8B in all three seeds. Neither GLM variant met the prespecified target validation NLL cutoff. In a descriptive same-epoch comparison on the independent GSM8K validation set, K = 1 yielded lower KL in all 15 GLM pairs. These results support treating chart granularity as a backbone-dependent choice.

Thank you again for the time you have spent on our paper. We hope the new cross-domain results and parameter sweeps address the gaps you identified. If these results and the clarifications above address your remaining concerns, we would be grateful if you would consider raising your rating.

Reviewer_97toOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ATLAS conditions a shared low-rank nonlinear residual on retained-domain activation geometry. Fixed local centers and PCA bases determine token-dependent centering, directional filtering, and amplitude during training and inference. The paper evaluates code adaptation with mathematics and commonsense retention across five backbones and three training seeds. Its main Qwen3-8B comparison reports lower retained-output KL than seven published baselines at shared coding requirements, supported by structural ablations and answer-churn analyses.

Critical strengths
  • Comparing retained KL at shared coding requirements across three seeds is more informative than comparing retention alone.
  • Evals are comprehensive with compared benchmarks.
  • Presentation is clear.
Critical weaknesses
  • The incremental contribution of the full atlas is not sufficiently compelling. Most of the improvement over Plain comes from centering.
  • Global-PCA achieves substantially lower KL than local ATLAS on GLM (Table 8). Although several subspace methods and norm controls are included, retained-data gradient projection or train-time norm/trust-region control on the same nonlinear residual is not evaluated.
  • Appendix H does report decoding/prefill overhead, atlas storage, and PCA-fitting time. However, fitting excludes activation extraction and uses a different neighborhood-support construction from the main experiments. Comparable end-to-end preprocessing/training costs and runtime measurements for the seven main baselines and simpler centering/distance controls are absent. Given the small incremental gains over compact variants, a matched-quality cost comparison is important to justify keeping the full atlas active during inference.
Questions for authors

See my comments above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 97to

Thank you for examining the contribution of each geometric component and the computational costs. Following the order of your Critical Weaknesses (W1–W3), we report a global-mean centering control, paired-seed intervals, and optimization times across all 14 compared configurations, alongside the updates in Section 6 and Appendices D and H.

W1. Incremental contribution of the full atlas beyond centering

To separate general activation centering from input-dependent routing, we evaluated a global-mean centering control, Btanh⁡(A(h−hˉ))B\tanh(A(h - \bar{h})), where hˉ\bar{h} is the mean of the 4,096 calibration activations used to construct the atlas. Routed centering replaces hˉ\bar{h} with the token-dependent reference c(h)c(h) in Btanh⁡(A(h−c(h)))B\tanh(A(h - c(h))). Both share the training data, layer, rank, initialization, learning rate, and five-epoch budget of the Qwen3-8B study (Section 6, Appendix D.4).

Over the ten shared coding requirements in Table 2 (151–160 solved MBPP+ problems), the three-seed geometric mean retained KL values are:

Residual constructionMean retained KL (×10⁻³) ↓
Residual Plain1.9718
Global-mean centering1.3640
Routed centering1.2138
Distance scaling1.1849
Full ATLAS1.1575

Global-mean centering reduces KL by 30.82% relative to Plain. Routing each token to its local reference c(h)c(h) reduces KL by a further 11.01% relative to global-mean centering (13.25%, 12.91%, and 6.71% across seeds). Full ATLAS reduces KL by 15.14% relative to global-mean centering and by 4.64% relative to routed centering (95% paired-seed interval [3.41%,5.86%][3.41\%, 5.86\%], Table 14). Lowering every routed-center score by one problem and raising every global-mean score by one retains a 9.27% KL reduction (Appendix D.4). As your review notes, most of the gain over Plain comes from routed centering, and compact variants such as Input filter (1.1636×10−31.1636 \times 10^{-3}) recover nearly the full directional benefit; Section 6 now highlights these lightweight variants alongside full ATLAS.

W2. Global-PCA on GLM and directional / norm controls

Global-PCA is the single-chart (K=1K = 1) special case of ATLAS. Across the four Qwen backbones on GSM8K and Qwen3-8B on CSQA, multi-chart ATLAS (K=32K = 32) achieves lower aggregate retained KL than Global-PCA in 14 of 15 seed comparisons, including a 2.93% reduction on Qwen3-8B ([1.70%,4.15%][1.70\%, 4.15\%]). On GLM-4-9B, Global-PCA achieves lower KL (Table 8). Thus, preferred chart count KK varies by model family.

To test residual orientation at matched magnitude after training, the effective-norm control rescales the Plain residual at each token to the norm of its paired ATLAS checkpoint; ATLAS achieves lower KL in 11 of 12 comparisons across the four Qwen backbones (3.89% reduction, Table 16), complementing Output-KL (Figure 3, Table 6) and CLoRA (Table 1). Evaluating train-time norm constraints or retained-gradient projection on the same nonlinear branch is a natural extension.

W3. Preprocessing, training, and inference costs

Across the five-epoch, 40-step Qwen3-8B trajectories (three seeds per setting), mean optimization times covering batch preparation and parameter updates are:

MethodOptimization time (s)
Plain71.33 ± 0.08
Global-PCA69.94 ± 1.63
ATLAS71.41 ± 2.13
Routed centering71.53 ± 1.05
Distance scaling70.55 ± 1.65
Native LoRA71.04 ± 0.25
LoRA-Null68.83 ± 3.91
OPLoRA (16)70.48 ± 0.59
OPLoRA (128)71.41 ± 0.95
TopLoRA92.19 ± 0.58
CorDA-KPM93.06 ± 2.62
CLoRA155.40 ± 0.07
STM157.16 ± 0.23
TALR249.14 ± 0.39

Full ATLAS has nearly the same optimization time as routed centering, distance scaling, and Plain. At inference, active FP32 buffers occupy 8.50 MiB of GPU memory (15.85 MiB on disk in FP64 with metadata). Appendix H (Table 27) reports 2.52%–3.13% decoding overhead and 5.83%–12.26% prefill overhead for ATLAS relative to the frozen model across batch sizes 1, 8, and 24, alongside Frozen, Plain, Global-PCA, and Replay. Six linear weight baselines in Table 1 merge static updates into the weight matrix after training, adding no separate inference branch. In Appendix A.4, the 38.54 s fitting benchmark uses an equal-support neighborhood (nsupport=128n_{\text{support}}=128, excluding activation extraction) to fix SVD dimensions, yielding bases Uj∈Rd×pU_j \in \mathbb{R}^{d\times p} with the same storage and inference FLOPs as the cluster-member atlas. Benchmarking inference latency across all intermediate residual variants is a natural next step.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.