- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The author proposes a new technique to identify spurious features associated with the dataset by analyzing the weights of LoRA updates, particularly the singular vectors associated with the first few epochs of training, and then uses the projected gradient ascent technique to update and improve the model.
Critical strengths
-
The authors have included rigorous proofs, addressed key concerns related to the training in the writing, and have given necessary theoretical justification for the approach.
-
The causal data generation and how it can influence the training process seem intuitive.
-
The influence of personal factors and their impact on the training process are well explained. These insights will be particularly useful for synthetic data generation, as such methods can introduce inherent biases due to the role-playing behavior of large teacher models.
Critical weaknesses
While I overall like the motivation and theoretical justification, I believe some key experiments are missing, and if included it will further strengthen the paper.
-
In general, in many practical settings, datasets are often collected from different sources without a single curator's intent or persona influencing the overall data generation process. In fact, multiple personas may influence the data collection process. Considering this, how sensitive is the proposed approach to the presence of multiple independent personas in the dataset, and can the top-1 left and right singular vectors effectively capture such diversity? An experiment, perhaps using synthetic data generated by an LLM conditioned on different personas, would further strengthen the paper.
-
Stage 3 of GRAP (Lines 277–280) retrains the model based on the knowledge gathered from Stage 1 (Algorithm 1). While the results presented by the authors show that only a few epochs (<3) are required, the efficiency of the method may largely depend on the strength of the spurious features in the training data. Hence, an experiment that directly examines this relationship, particularly the accuracy of Stage 1 in identifying spurious features, would be helpful.
-
I am a bit concerned about the assumptions made in Lines 374–377. Will the model only achieve superior performance over the baselines when these assumptions are satisfied? How likely are these assumptions to hold in real-world data, as highlighted in Weakness 1?
Questions for authors
- How does the author choose the value of in Algorithm 1, line 272 . Do they use weights across multiple epochs or only first few epochs?
4 comments
Check-in
Dear Reviewer,
We just wanted to quickly check-in as the end of the rebuttal period draws near to say that we remain available to discuss our responses, and perform any additional analyses that might improve your assessment of our paper or raise your score. Thank you again for your time and for your comments on our paper.
Warmest wishes,
The Authors
Dear Authors,
Thank you for the rebuttal. Although I still have reservations about the generalizability of the proposed approach across different personas, as well as the practicality of its theoretical assumptions, I believe the paper has merit and addresses an important problem. Furthermore, the empirical evidence largely aligns with the main claims of the paper. Hence, I have decided to increase my score. I encourage the authors to clearly articulate the limitations of the proposed approach and discuss the concerns highlighted during the rebuttal process in the final version.
Response
Thank you again for the time you have given our paper, we really appreciate it. We will make sure to update the paper to more clearly articulate the concerns raised during the review process.
Response to reviewer
Thank you for your thoughtful review. We were glad that you found the theoretical justification rigorous and the causal account of how data is generated intuitive.
W1: Multiple independent personas. We agree this is a situation that can arise in practice, and our Domain 3 is an example of it. The Reddit corpus has no single curator: its 3,407 posts are drawn from 43 subreddits and written by many different authors, each with their own voice. GRASP still halves the political drift (+0.137 is reduced to +0.061) while improving content quality (0.77 is increased to 0.99).
Our theory explains why this works. A persona that appears in only a small fraction of examples behaves like per-example noise and averages out (the τ/√N term in Theorem 1). Only an attribute shared across the corpus, here the right-leaning stance, builds up into a dominant direction in the update. So the method targets attributes shared across the corpus and is, by design, insensitive to many independent personas. We think this is the right behaviour: individual author voices do not form a systematic shortcut, whereas a stance shared across the corpus does.
When several personas are each widely shared, they form a rank-k subspace rather than a single direction. Our theory covers this case (see Appendix E of our paper for the full treatment, including a way to estimate k), and GRASP then projects out the top k directions.
To test the multi-persona case directly, we fine-tuned on an equal mixture of the medical and right-leaning Reddit corpora. The resulting update's spectrum differs from the single-corpus runs: the second singular value rises to 0.62σ₁ (vs. ≈0.5σ₁ for either corpus alone), so the top two directions stand together above a slowly decaying tail (σ₂/σ₃ = 1.30, with subsequent ratios 1.21, 1.19, 1.12). This elbow after σ₂ indicates a rank-2 persona subspace. Applying GRASP with a rank-2 projection, using the rank-k generalisation developed in detail in Appendix E, reduced emergent misalignment from 2.3% (naive) to 0.0%. More generally, the rank-k theoretical results in Appendix E provide the principled approach for higher-dimensional personas, including corpora that induce several distinct personas.
W2: Spurious strength and Stage-1 accuracy. Theorem 1 predicts that identification becomes less accurate as the spurious signal weakens, and following your suggestion we tested this directly. In Domain 3, we varied the share of right-leaning examples from 70% (as in the paper) down to 50%, 30% and 10%, keeping the corpus size fixed so that only the signal strength changed. For each corpus we compared the direction found by Stage 1 with a political direction obtained from labelled examples, which Stage 1 never sees. Their alignment is 0.139, 0.124, 0.101 and 0.078 respectively, against 0.026 for a random direction. Accuracy therefore declines gradually as the signal weakens, as the theory predicts, and is still about three times chance when only 10% of examples carry the attribute. On efficiency, the cost of Stage 3 does not depend on this strength: it always runs the same number of steps as a standard fine-tune. We will add this experiment to the paper.
W3: Do the assumptions hold on real data? This is an important question. Our theorems only claim identification where the assumptions hold. But importantly, the assumptions can be checked before intervening, without any labels. A clear spectral gap after σ₁ is the signature of a dominant shared attribute, and Appendix E.4 in our paper outlines how to use it as a diagnostic: if there is no clear gap, the method should not be applied.
We also expect the assumptions to hold reasonably widely. A2, that the intended task is spread over many directions, describes typical fine-tuning, since instruction tuning and domain adaptation span many sub-tasks. A1, that a persona is encoded as a low-dimensional direction (we include a rank-k generalisation of the rank-1 result from the main body of the paper in the appendix), is supported by the interpretability work on persona directions that we cite. Domain 3 uses naturally occurring human data, and the leading gap appears in most weight matrices (70% in Domain 2 and 82% in Domain 3).
Q: Which ΔW_naive do we use? We use a single checkpoint: the final Stage-1 update (Algorithm 1, lines 273–276), not an aggregate over epochs. Stage 1 runs for about two epochs in Domain 1 and one epoch in Domains 2 and 3. The extracted directions are then kept fixed throughout Stage 3. We will state this explicitly in the revision.
Thank you again for the detailed engagement. We are happy to clarify any points that are not sufficiently addressed, and if there is specific further evidence that would help increase score, we would welcome the opportunity to provide it.