acceptodds
Under review as a conference paper at ICLR 2027

Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning

Abstract

Fine-tuning a pretrained language model on a curated dataset can produce spurious correlations between the fine-tuning task and unintended latent factors—such as misaligned personas or political slant—that the curation procedure has entangled with the task. The model can latch onto these spurious correlations, leading to bias and reduced out-of-distribution generalisation. We prove that under reasonable assumptions on task complexity and the spurious correlation, such latent factors can be identified, without supervision, from the weights of a naive LoRA fine-tune. Existing approaches to removing bias, such as activation steering, remove identified factors from residual-stream activations, either at inference or during training. We argue, however, that the goal should be to remove the spurious correlation, not the latent factor itself, as the pretrained model may rely on it for genuine task signal. To enable this, we propose GRASP, GRadient projection of Associated Spurious Patterns, which prevents the model from acquiring new reliance on the identified latent factor while preserving any pretrained content along it. We validate on three fine-tuning tasks. The first two involve emergent misalignment, where fine-tuning on a narrow task—in our case, writing insecure code and giving bad medical advice—leads to misaligned responses on unrelated topics. Here our method completely removes misalignment in the insecure code case and reduces them by  5x in the bad medical advice case, beating all baselines in the trade-off between misalignment-reduction and task-preservation. The last is a novel political-bias experiment, where fine-tuning on right-skewed Reddit financial-advice data causes political-lean drift on unrelated topics. Here our method reduces drift by more than half, while improving financial task performance, beating all baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Emergent Misalignment, fine-tuning, spurious correlation, causal inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_DkQzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The author proposes a new technique to identify spurious features associated with the dataset by analyzing the weights of LoRA updates, particularly the singular vectors associated with the first few epochs of training, and then uses the projected gradient ascent technique to update and improve the model.

Critical strengths
  • The authors have included rigorous proofs, addressed key concerns related to the training in the writing, and have given necessary theoretical justification for the approach.

  • The causal data generation and how it can influence the training process seem intuitive.

  • The influence of personal factors and their impact on the training process are well explained. These insights will be particularly useful for synthetic data generation, as such methods can introduce inherent biases due to the role-playing behavior of large teacher models.

Critical weaknesses

While I overall like the motivation and theoretical justification, I believe some key experiments are missing, and if included it will further strengthen the paper.

  1. In general, in many practical settings, datasets are often collected from different sources without a single curator's intent or persona influencing the overall data generation process. In fact, multiple personas may influence the data collection process. Considering this, how sensitive is the proposed approach to the presence of multiple independent personas in the dataset, and can the top-1 left and right singular vectors effectively capture such diversity? An experiment, perhaps using synthetic data generated by an LLM conditioned on different personas, would further strengthen the paper.

  2. Stage 3 of GRAP (Lines 277–280) retrains the model based on the knowledge gathered from Stage 1 (Algorithm 1). While the results presented by the authors show that only a few epochs (<3) are required, the efficiency of the method may largely depend on the strength of the spurious features in the training data. Hence, an experiment that directly examines this relationship, particularly the accuracy of Stage 1 in identifying spurious features, would be helpful.

  3. I am a bit concerned about the assumptions made in Lines 374–377. Will the model only achieve superior performance over the baselines when these assumptions are satisfied? How likely are these assumptions to hold in real-world data, as highlighted in Weakness 1?

Questions for authors
  • How does the author choose the value of ll in Algorithm 1, line 272 . Do they use weights across multiple epochs or only first few epochs?
4 comments
AuthorsOpenReview ↗

Response to reviewer

Thank you for your thoughtful review. We were glad that you found the theoretical justification rigorous and the causal account of how data is generated intuitive.

W1: Multiple independent personas. We agree this is a situation that can arise in practice, and our Domain 3 is an example of it. The Reddit corpus has no single curator: its 3,407 posts are drawn from 43 subreddits and written by many different authors, each with their own voice. GRASP still halves the political drift (+0.137 is reduced to +0.061) while improving content quality (0.77 is increased to 0.99).

Our theory explains why this works. A persona that appears in only a small fraction of examples behaves like per-example noise and averages out (the τ/√N term in Theorem 1). Only an attribute shared across the corpus, here the right-leaning stance, builds up into a dominant direction in the update. So the method targets attributes shared across the corpus and is, by design, insensitive to many independent personas. We think this is the right behaviour: individual author voices do not form a systematic shortcut, whereas a stance shared across the corpus does.

When several personas are each widely shared, they form a rank-k subspace rather than a single direction. Our theory covers this case (see Appendix E of our paper for the full treatment, including a way to estimate k), and GRASP then projects out the top k directions.

To test the multi-persona case directly, we fine-tuned on an equal mixture of the medical and right-leaning Reddit corpora. The resulting update's spectrum differs from the single-corpus runs: the second singular value rises to 0.62σ₁ (vs. ≈0.5σ₁ for either corpus alone), so the top two directions stand together above a slowly decaying tail (σ₂/σ₃ = 1.30, with subsequent ratios 1.21, 1.19, 1.12). This elbow after σ₂ indicates a rank-2 persona subspace. Applying GRASP with a rank-2 projection, using the rank-k generalisation developed in detail in Appendix E, reduced emergent misalignment from 2.3% (naive) to 0.0%. More generally, the rank-k theoretical results in Appendix E provide the principled approach for higher-dimensional personas, including corpora that induce several distinct personas.

W2: Spurious strength and Stage-1 accuracy. Theorem 1 predicts that identification becomes less accurate as the spurious signal weakens, and following your suggestion we tested this directly. In Domain 3, we varied the share of right-leaning examples from 70% (as in the paper) down to 50%, 30% and 10%, keeping the corpus size fixed so that only the signal strength changed. For each corpus we compared the direction found by Stage 1 with a political direction obtained from labelled examples, which Stage 1 never sees. Their alignment is 0.139, 0.124, 0.101 and 0.078 respectively, against 0.026 for a random direction. Accuracy therefore declines gradually as the signal weakens, as the theory predicts, and is still about three times chance when only 10% of examples carry the attribute. On efficiency, the cost of Stage 3 does not depend on this strength: it always runs the same number of steps as a standard fine-tune. We will add this experiment to the paper.

W3: Do the assumptions hold on real data? This is an important question. Our theorems only claim identification where the assumptions hold. But importantly, the assumptions can be checked before intervening, without any labels. A clear spectral gap after σ₁ is the signature of a dominant shared attribute, and Appendix E.4 in our paper outlines how to use it as a diagnostic: if there is no clear gap, the method should not be applied.

We also expect the assumptions to hold reasonably widely. A2, that the intended task is spread over many directions, describes typical fine-tuning, since instruction tuning and domain adaptation span many sub-tasks. A1, that a persona is encoded as a low-dimensional direction (we include a rank-k generalisation of the rank-1 result from the main body of the paper in the appendix), is supported by the interpretability work on persona directions that we cite. Domain 3 uses naturally occurring human data, and the leading gap appears in most weight matrices (70% in Domain 2 and 82% in Domain 3).

Q: Which ΔW_naive do we use? We use a single checkpoint: the final Stage-1 update (Algorithm 1, lines 273–276), not an aggregate over epochs. Stage 1 runs for about two epochs in Domain 1 and one epoch in Domains 2 and 3. The extracted directions are then kept fixed throughout Stage 3. We will state this explicitly in the revision.

Thank you again for the detailed engagement. We are happy to clarify any points that are not sufficiently addressed, and if there is specific further evidence that would help increase score, we would welcome the opportunity to provide it.

Dear Reviewer,

We just wanted to quickly check-in as the end of the rebuttal period draws near to say that we remain available to discuss our responses, and perform any additional analyses that might improve your assessment of our paper or raise your score. Thank you again for your time and for your comments on our paper.

Warmest wishes,

The Authors

Dear Authors,

Thank you for the rebuttal. Although I still have reservations about the generalizability of the proposed approach across different personas, as well as the practicality of its theoretical assumptions, I believe the paper has merit and addresses an important problem. Furthermore, the empirical evidence largely aligns with the main claims of the paper. Hence, I have decided to increase my score. I encourage the authors to clearly articulate the limitations of the proposed approach and discuss the concerns highlighted during the rebuttal process in the final version.

Thank you again for the time you have given our paper, we really appreciate it. We will make sure to update the paper to more clearly articulate the concerns raised during the review process.

Reviewer_FWo1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies unintended behavioral changes caused by fine-tuning, including emergent misalignment and political drift. It proposes a method that extracts the leading singular-vector pair from the LoRA update and then restarts fine-tuning while projecting gradients away from the identified directions. The central hypothesis is that a spurious attribute produces a coherent, dominant update, whereas the desired task signal spreads across many directions. Experiments cover multiple tasks. GRASP substantially reduces measured misalignment or political drift while generally retaining more in-domain performance.

Critical strengths

S1. The paper is clearly motivated and addresses an important problem in LLM fine-tuning. The framework is well structured, and the reasoning behind the proposed approach is easy to follow.

S2. The authors make a useful distinction between preserving existing representations and restricting new updates. Removing a concept from activations can damage useful pretrained capabilities, whereas preventing fine-tuning from strengthening a particular direction offers a sensible alternative.

S3. The method addresses a practical challenge by requiring no concept labels. It extracts directions from a preliminary LoRA checkpoint and applies a straightforward gradient projection, making it appealing when the unintended attribute is not known beforehand.

S4. The empirical results across multiple domains are encouraging and suggest that the method may apply to a broader range of tasks.

Critical weaknesses

W1. There is a gap between the derivation and the LoRA implementation. The paper states that, under vanilla SGD, “projecting gradients projects updates exactly” (Section 2.5). The algorithm projects gl:=∇Wlℓg_l:=\nabla_{W_l}\ell away from ulvl⊤u_lv_l^\top (Algorithm 1), intending to keep hl(θ)=ul⊤Wlvlh_l(\theta)=u_l^\top W_lv_l fixed (Eq. 5). However, this equivalence is not established for factorized LoRA updates. Writing ΔWl=BlAl\Delta W_l=B_lA_l, where Al,BlA_l,B_l are the LoRA factors, an SGD step with learning rate η\eta produces $ \delta(\Delta W_l) =-\eta\left(g_lA_l^\top A_l+B_lB_l^\top g_l\right)+O(\eta^2), $ omitting the fixed LoRA scaling constant. Thus, projecting glg_l does not automatically ensure ⟨δ(ΔWl),ulvl⊤⟩=0\langle\delta(\Delta W_l),u_lv_l^\top\rangle=0. The paper should clarify how the constraint is enforced through the LoRA factors and AdamW updates.

W2. GRASP requires a preliminary fine-tune followed by projected retraining. The paper discusses per-step projection overhead but does not report the total training cost, including both stages and direction extraction. End-to-end cost measurements are needed to assess its practical efficiency.

Questions for authors

Q1: The method assumes that the spurious gradient component has a consistent low-rank structure across training examples and dominates the update spectrum. While this assumption is reasonable, I am curious whether LoRA’s low-rank parameterization could amplify the observed dominance of the leading singular mode. Does this pattern persist with larger LoRA ranks or full fine-tuning?

3 comments
AuthorsOpenReview ↗

Response to reviewer

Thank you for taking the time to review our paper so carefully. We were glad to read that you found it clearly motivated and easy to follow, and that it addresses an important problem in LLM fine-tuning.

W1: How the constraint is enforced with LoRA and AdamW. Thank you for this careful point; you are right. With ΔW = BA, the SGD step is δ(ΔW) = −η(gAᵀA + BBᵀg) + O(η²), so projecting the dense gradient g does not by itself keep ⟨ΔW, uvᵀ⟩ fixed. The equivalence in Section 2.5 holds for dense weights only.

In our LoRA implementation, we enforce the constraint directly on the factors rather than on g, a choice we made for computational efficiency and ease of implementation; the paper should have stated this explicitly. Write a = uᵀB for the update's write along u, so that uᵀΔW = aA. At each site we project the gradient of B so that each step leaves ‖a‖² unchanged (Eq. 6–8 applied to the LoRA parameter B). Since Stage 3 starts from a fresh adapter with B = 0, ‖a‖ stays close to zero throughout training. Then for every input direction v, |uᵀΔWv| = |aAv| ≤ ‖a‖‖A‖‖v‖, so the leakage at the v of Eq. 5 is controlled, through both terms of your expansion.

This approach constrains the whole write along u, which is slightly stronger than our theory requires, and we chose it for two reasons. First, it is a simpler and cheaper way to enforce the constraint: the persona could be written along u through any input direction, and identifying and constraining each separately would be difficult, whereas uᵀΔW = 0 implies uᵀΔWv = 0 for every v. Second, under Assumption A1 the spurious attribute is a direction the model writes into its representation; v is only the input pattern the naive run happened to use. Constraining only uᵀΔWv lets the model keep writing along u through other inputs, and in our runs it does: the residual write along u stays near its naive level, whereas the factor-level constraint cuts it about four-fold.

We will update the paper to make this clear.

On AdamW: element-wise adaptive scaling means the constraint holds only approximately, as for any gradient projection under an adaptive optimiser. We therefore check it after training, as already reported in Appendix D: the leakage ρ = |uᵀΔWv|/‖ΔW‖_F falls from 0.718 for the naive update to 0.015 with GRASP (mean over all 196 sites, a 49× reduction), and stays below 0.07 at every site. We will make this clearer in the paper, updating Section 2.5 and Algorithm 1 to state the factor-level constraint and the derivation above, and discussing AdamW explicitly.

W2: End-to-end cost. We agree that the total cost should be reported, and we have now measured it for Domain 2 on a single A100. The naive fine-tune (Stage 1) took 33.6 minutes. Extracting the directions (Stage 2) is a thin SVD of the low-rank factors and took under a second. The projected fine-tune (Stage 3) runs for the same number of steps as Stage 1 and took 34.4 minutes, of which the projection itself accounted for only 1.7%. Peak GPU memory did not change. In total, GRASP costs about twice a single fine-tune, or 2 to 2.5 times in Domain 1, where Stage 1 used about two epochs and Stage 3 used two or three. For context, the supervised baselines (v-ref and CAFT) also need a fine-tuned model, plus a labelling step. We will add these measurements to the paper.

Q1: Could LoRA's low rank create the dominant mode? This is a good question: is the leading mode a property of the data, or a side effect of the low-rank parameterisation? To find out, we repeated the Domain 2 naive fine-tune at LoRA ranks 64 and 256, and with full fine-tuning. The largest spectral gap lies between the top singular value and the rest of the spectrum in 69% of weight matrices at ranks 16, 64 and 256, and in 62% under full fine-tuning. The dominant first mode therefore persists as rank grows and remains without LoRA at all, so it is not an artifact of LoRA.

We would also like to clarify that our theory does not depend on the spurious component being rank-1. The identification and ablation results extend to a rank-k spurious subspace, together with a way to estimate k from the spectrum (Appendix E). Rank-1 is simply the case that holds empirically in our settings, as the analysis above confirms. We will make this generality clearer in the main text and add the analysis to the appendix.

Thank you again for the detailed and constructive engagement. We are happy to clarify any points that are not sufficiently addressed, and if there is specific further evidence that would help strengthen your assessment and score, we would welcome the opportunity to provide it.

Thanks to the authors for their response. I hope they will add the experimental results for W2 to the paper and discuss Q1 more thoroughly in the next revision. I think this is an interesting paper, and I will maintain my positive score.

Thank you for your response and for your positive assessment. We will add the additional experimental results and discussion to the paper. We appreciate you time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.