acceptodds
Under review as a conference paper at ICLR 2027

Activation Diffusion, Revisited: Design Choices for Diffusion-Based Metamodels of LLM Activations

Abstract

Recently, diffusion-based metamodels of LLM residual-stream activations were proposed as priors for downstream tasks such as improved activation-based model control by projecting steered activations back to the data manifold, and unsupervised detection of human-interpretable concepts using the activation of the diffusion metamodel, similarly to sparse autoencoders. They also proposed a raw generative quality metric as the ability of the diffusion model to match the data mean and covariance, and showed initial evidence that all of these metrics tend to improve with scale. We take an in-depth look at design choices of training these diffusion models. In particular, (i) we find that informativeness of the chosen diffusion activation distribution matters greatly for the steering correction task, (ii) we propose a new diffusion parameterisation that results in particularly good generative quality scores on activation data, (iii) we provide a comprehensive analysis of the interplay between steering and diffusion projection strengths, and (iv)we further verify that the concept detection performance scales with increasing width and depth through an extensive hyperparameter sweep. Our results provide a basis towards practical real world usage of diffusion-based metamodels.

Primary area:
interpretability and explainable AI
Keywords:
activation steering, diffusion models, interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5DyWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies design choices for diffusion-based meta-models of LLM activations. It makes three main contributions: a velocity-to-x (v2x) parameterization that improves Fréchet distance, evidence that more task-relevant activation priors improve SDEdit-corrected steering, and a systematic characterization of the interaction between steering strength and diffusion noise level. The paper evaluates these choices primarily on Llama-3.2-1B activations and studies both generative quality and downstream activation steering.

Overall, I found the paper interesting and appreciated the breadth of the empirical study, particularly the systematic exploration of diffusion design choices and steering behavior. However, I have several concerns about how the main results should be interpreted.

Critical strengths

The paper provides a broad and systematic empirical investigation of diffusion-based activation meta-models. I particularly appreciate that the authors explore several design axes, including parameterization, loss weighting, training distribution, model scale, and SDEdit noise strength.

Another strength is the negative result that substantially better Fréchet distance does not translate into better downstream steering. If I am interpreting this result correctly, it is useful evidence that generative-quality metrics may not be reliable proxies for the downstream utility of activation diffusion models.

Critical weaknesses
  1. It is not clear to me that the trait-aligned prior result isolates "projection" from direct trait injection.
    My reading of the paper is that the trait-specific GLPs are trained on large corpora whose responses already strongly express the target trait, including activations extracted in the presence of the trait-eliciting system prompts. In contrast, the steering vectors are estimated from only 20 QA pairs. Unless I have misunderstood the setup, it therefore seems possible that the trait-specific GLP itself contains substantial information about the desired behavior and contributes additional steering, rather than merely projecting an off-manifold activation back toward a plausible activation distribution. I may have overlooked a control, but I could not identify an experiment that cleanly separates these possibilities. GLP-only (α=0\alpha=0), mismatched/cross-trait priors, or steering vectors constructed from the same large trait corpus would substantially help clarify this point.

  2. I am not fully convinced that the downstream comparison supports the conclusions drawn about v2x.
    Figure 15 appears to show no steering advantage for v2x, but, as far as I understand, both standard FM and v2x are trained at the same learning rate in this experiment, whereas Table 1 identifies substantially different best learning rates for the two methods. I would be hesitant to conclude that v2x provides no downstream benefit without a comparison in which each method is independently tuned, ideally across multiple training seeds. More broadly, the proposed v2x parameterization seems closely derived from EDM. it retains the EDM-style pre-conditioning and interpolation structure while replacing the low-noise endpoint with flow-matching velocity prediction and normalizing the target variance. I think the methodological novelty should therefore be positioned carefully, particularly given that its main demonstrated improvement is currently limited to the FD metric.

Questions for authors
  1. It would be helpful if the authors could clarify whether the trait-specific GLP changes trait expression when applied with α=0\alpha=0. Have you also tried mismatching the steering vector and the trait prior (e.g., a pessimistic steering vector with a deceptive GLP)? These controls seem important for distinguishing generic manifold correction from direct behavioral steering by the prior.

  2. I may be missing a technical reason for using a shared learning rate in Figure 15, but given the different optimal learning rates reported in Table 1, could the authors provide a downstream comparison where standard FM and v2x are independently tuned? Would the conclusion that their steering performance is essentially identical still hold?

  3. From my reading, some choices such as fine-tuning epoch count and the system-prompt-included extraction variant were selected based on downstream steering performance. Were these decisions made using a separate validation set from the 20 questions used for the reported evaluation? If not, how sensitive are the reported Pareto improvements to these choices?

  4. The current evaluation is primarily on one 1B model, one layer, 20 evaluation questions per trait, and one LLM judge. I understand that exhaustive replication is expensive, but it would help clarify which conclusions the authors expect to generalize beyond this particular setting, especially given the differences from the original GLP results on Llama-8B.

3 comments

We thank the reviewer for the constructive feedback. We have worked to address the issues noted, and list them below in order:

W1 / Q1. Does the trait-aligned prior project, or inject the trait?

We ran the three controls the reviewer proposes, on all five traits (App. J.6).

  • Prior alone (α = 0). Applying the trait-aligned prior with no steering vector leaves the trait score within about 3 points of the unsteered model for four traits (Fig. 29). Hallucinating is the exception, and not because of the trait prior: the score rises from 54 to 63–71 under every prior, most under FineWeb and least under the trait-aligned one.
  • Mismatched priors. We steered with each vector and corrected with the trait-aligned priors of the four other traits (Fig. 30). For evil, pessimistic and deceptive traits, the matching prior is ahead of the mismatched ones. For hallucinating, the trait-aligned prior is on the Pareto frontier but some other priors are competitive (potentially related to the result that any prior-alone correction is enough to increase the hallucinating score, making hallucinating a special case). For sycophantic, the deceptive trait prior is surprisingly even better than the sycophantic trait prior, potentially due to the deceptive trait causing sycophancy in a context that elicits it (e.g. questions of the form "Don't you agree that...").
  • Steering vector from the same corpus. A mean-difference vector built from the trait-aligned corpus steers about as well as the persona vector on four traits, and the trait-aligned prior improves it in the same way (Fig. 31). So the corpus, used for linear steering, does not replace the prior.

Our reading: the prior does not significantly steer by itself, but the reviewer is right that a matching prior carries information about the steered behaviour, which shows up only together with the steering vector. A prior closer to the steered distribution preserves more of the steering.

W2 / Q2. v2x against standard FM, independently tuned

We trained both methods at their own best learning rates from Table 1 (1.25e-5 for v2x, 2e-4 for standard flow matching) with three seeds each and evaluated all five traits (App. J.7, Figs. 32–33, replacing the single-seed figure of the submission). The conclusion holds: v2x has the lower FD (0.24 against 0.33) and no steering advantage. Averaged over traits and α at t = 0.5, standard flow matching is 2.9 points more coherent with a 1.5 points lower trait score, and it is ahead on both axes at high α for evil and pessimistic (α = 4: 46 / 47 against 38 / 41 for evil).

Novelty. We agree that v2x is not, by itself, a significant deviation from standard diffusion model practice. However, we are not aware of a paper that uses velocity prediction at low noise levels and x-prediction at high noise levels, and we were surprised that this is significantly more effective than EDM on activation data, when it comes to the FD metric. Similar relatively small changes in parameterisation are sometimes found to result in large improvements, which is why we are interested in it, not necessarily due to significant algorithmic innovation. We have now clarified and narrowed the positioning of v2x in the contributions section of the paper.

Q3. Were epoch count and trait-aligned activation extraction method chosen on a validation set?

No, in the submission they were chosen on the 20 evaluation questions. We redid this with separate validation and test questions (App. J.2). On validation, 10 against 20 epochs and extraction with against without the system prompt differ by about one point of our summary statistic (Table 6), and priors trained on unfiltered or differently filtered data give the same test curves (Fig. 26). The improvement of the trait-aligned prior over the Alpaca prior for evil, pessimistic and deceptive is also present on the test questions (Fig. 23).

Q4. Scope

  • 20 questions: the main comparison is now repeated on 100 new questions per trait (App. J.2).
  • One judge: the results hold under three different judges (Qwen and Gemma-based) and the judges are checked against 400 human labels (App. J.3).
  • One model, and the differences from Luo et al.: We repeated the results on Llama-3.1-8B-Instruct, where the priors also improve over standard steering and trait-aligned priors are generally the most helpful. The difference from their FineWeb result comes from subtle differences in the steering-vector extraction procedure used and general differences between 8B and 1B models (App. J.8; also see our reply to DGK9, W3).
  • What we expect to generalise: that the training distribution of the prior matters more than its parameterisation or its FD, and that a Gaussian prior is not enough. The size of the gain is trait-dependent already on this model: clear for evil, pessimistic and deceptive, absent for sycophantic with our steering vectors, and in favour of the Alpaca prior for hallucinating.

We would like to remind the reviewer that the discussion period is approaching an end, and we would be happy to address any further questions! We also hope that the responses and additional experiments provided address the main concerns.

To recap, regarding the weaknesses: We provided additional experiments isolating trait injection from projection, provided additional experiments w.r.t. v2x and clarified the contribution: v2x mainly improves the FD score, which also shows that the FD score does not directly translate to improved steering performance. We also provided additional experiments aimed at answering the additional questions the reviewer raised.

Thank you for your additional experiments and efforts. You addressed most of my concerns. I found this paper interesting and learned something from it, though I still have concerns about it's novelty level. Overall, I am positively inclined toward the work.

Reviewer_kqg5OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper is a close followup of the previous paper by Luo et al. It extends the previous method and observations and provide some new information. This includes a key conclusion that increasing the informativeness of the activation prior makes a large difference for the downstream performance, an improved diffusion implementation, and some other insights.

Critical strengths
  1. This paper does provide a lot of useful information. An interested reader could obtain much knowledge from the paper. This may look like a short sentence, but I'm actually happy to see such a submission that could really contribute some new knowledge.
Critical weaknesses
  1. This paper is loosely organized. Many observations are made but without a clear story connecting them. Therefore, a reader has to manually reorganize the information and take what's needed. Nevertheless, I acknowledge that this might be due to the nature of a quick followup study.
  2. The motivation behind the exact improved diffusion model design is unclear. For example, although we see velocity-to-x is better, why do the authors decide to make this design at the first place? Providing the motivation would help readers learn more.
Questions for authors
  1. Please explain the motivation behind the improved diffusion model design.
3 comments

We thank the reviewer for the positive and constructive feedback.

Paper structure Indeed, the nature of the paper is a follow-up study, providing additional information around the idea of Luo et al., and as such some of the results do not strictly conform to a single narrative. Our overall takeaway is: 1) We can significantly improve the FD metric as proposed by Luo et al., but turns out that the FD metric is mostly not predictive of steering performance, informing future research on better proxy metrics. 2) As such, it seems that improvements in the data have larger immediate returns for steering quality itself.

Motivation for the velocity-to-x design The overall motivation was based on two factors: 1) The results in the original Luo et al. paper implied that the FD metric might be a good proxy for other capabilities, such as steering performance, and as we initially wanted to explore whether we can improve on it 2) similar parameterisation changes have often led to significant improvements in the diffusion literature in the past (see, e.g., EDM). Instead of settling on an existing method, we wanted to explore whether we can design a new parameterisation that is particularly successful for activation data. We indeed partially succeeded in this, but also found out that FD is not necessarily a strong proxy for the steering correction capabilities per se. This may be due to the new parameterisation helping mainly in the high noise levels, and the SDEdit steering correction procedure being applied at the lower noise levels.

Thank the authors for the response. I see the other reviewers have concerns on method or analysis rigor. I agree it's an important factor of a good paper, but, seriously, this is the most informative submission in my review batch... My initial rating might be a bit biased by the disappointment from the other submissions, but I still lean towards a positive score, mainly from the informativeness side.

We thank the reviewer for the response! We agree that the reviewers had valid points regarding methodology, but we also believe that we mostly addressed these concerns directly through additional experiments.

Again, thank you for the time taken for the review.

Reviewer_DGK9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits the Generative Latent Prior (GLP) of Luo et al. (2026), a diffusion model over LLM residual-stream activations, by ablating its training recipe on layer-7 activations of Llama-3.2-1B. The prior's training data is found to be the dominant lever for SDEdit-based steering correction: chat and trait-aligned activations beat FineWeb, which here does not improve over plain steering (Fig. 5). A new velocity-to-x parameterisation improves Fréchet distance (0.514→0.419 at L=12) but not steering (Fig. 15), and an α×t sweep shows that stronger steering benefits from more aggressive SDEdit noise (Figs. 8–9).

Critical strengths
  1. They did a lot of ablation studies over a hundred denoisers with per-configuration learning-rate sweeps across preconditioner, weighting, π(t), data, depth and width (Tables 1, 6, 7). The experimental results are valuable and detailed. The data-informativeness result (Fig. 5) and the sweep (Figs. 8–9) give concrete guidance, and reporting is complete.
  2. Results are reported clearly, no matter for negative results or positive results. For example, they show that FD gains do not transfer to steering (Figs. 15–17), Tulu does not beat Alpaca, EDM's π(t) hurts (Table 4), and depth barely matters for steering (Fig. 23).
Critical weaknesses
  1. There are some evaluation leakage issues. Three parts of the pipeline are not isolated from the final evaluation. (a) The trait-aligned training activations are selected by GPT-4.1-mini using the same prompts (Appendix H) that later score the evaluation (trait ≥ 60 and coherence ≥ 60; Appendix I.4). The prior is therefore trained on exactly the activations this judge rewards, and any bias of the judge is inherited and amplified. (b) Several design choices were made by inspecting results on the same 20 evaluation questions per trait, with no validation/test split: the Alpaca/Tulu epoch counts ("a judgement call", Sec. 4.2), the system-prompt-included extraction variant (Appendix I.5), and the fixed t=0.5 in the headline figures.
  2. v2x is incremental and optimises a non-predictive metric. It only changes EDM's low-noise endpoint, its gain is confined to t≤0.3t\le0.3 (Fig. 17) where FD is determined, and it gives no steering benefit.
  3. Narrow scope and overstated secondary claims. One model, one layer, several results on the evil trait only; the FineWeb GLP does not help here unlike in Luo et al., with no diagnosis.
Questions for authors

Gaussian and linear prior baseline. What does SDEdit-style projection under a Gaussian prior fitted to the same activations (FineWeb, Alpaca, trait-aligned) do on the steering task of Fig. 5? This would show whether nonlinear structure is needed and would also explain why FD, a second-moment metric, fails to predict steering performance.

3 comments

We thank the reviewer for the constructive feedback. We have worked to address all the issues noted, and list them below in order:

W1. Evaluation leakage

(a) The trait-aligned training data is filtered by the judge that scores the evaluation. We trained trait-aligned priors on the unfiltered rollouts, which removes the judge from the training data, and on rollouts filtered by a different judge (Qwen2.5-32B-Instruct). On new test questions both give the same trade-off curves as the original prior for all five traits, within the error bars (App. J.4, Fig. 26).

We also re-scored the test generations with two open-weight judges (Qwen2.5-32B-Instruct, Gemma-3-27B-it). The comparison between priors is the same under all three judges (App. J.3, Fig. 24). Against 400 blinded human labels (labeled by a paper author), the three judges agree with the annotator to a similar degree (AUROC 0.67–0.79 for coherence, 0.77–0.81 for trait; Table 7), and no judge is detectably more lenient on GLP-corrected answers than on plainly steered ones (Fig. 25).

(b) Choices made on the 20 evaluation questions. We generated 100 new questions per trait, disjoint from the training prompts, and split them into 50 validation and 50 test questions (App. J.2). On validation, the settings of the paper are confirmed or not distinguishable from their alternatives (Table 6): t = 0.5 is best for all three priors, and t = 0.3 is clearly worse; 10 against 20 epochs and extraction with against without the system prompt differ by about one point. The test questions were then evaluated once. The trait-aligned prior again gives the best curve for evil, pessimistic and deceptive, and no prior helps significantly on sycophantic, as on the original questions; for hallucinating the Alpaca prior is further ahead than before (Fig. 23).

W2. Issues with v2x (learning rate tuning, and novelty)

Hyperparameter tuning. We repeated the steering comparison with each method at its own best learning rate, three seeds and all five traits (App. J.7, Figs. 32–33). Velocity-to-x has the lower FD on every seed (0.24 against 0.33, Table 9) and no steering advantage; where the two differ by more than the seed spread (Table 10), standard flow matching keeps slightly more coherence. We agree with the reviewer that FD does not predict steering here, and the paper now says so on this stronger evidence.

Novelty. We agree that v2x is not, by itself, a significant deviation from standard diffusion model practice. However, we are not aware of a paper that uses velocity prediction at low noise levels and x-prediction at high noise levels, and we were surprised that this is significantly more effective than EDM on activation data, when it comes to the FD metric. In the context of the steering usecase, it also helps drive the point that an improved FD does not necessarily improve steering performance, in contrast to the implied correlation in the original Luo et al. paper. This raises finding improved proxy metrics for steering quality as an important research problem. We have now clarified and narrowed the v2x contribution in the contributions.

W3. Narrow scope

  • One model. We repeated the main comparison on Llama-3.1-8B-Instruct at layer 15 (App. J.8, Fig. 35). The chat-data priors improve on standard steering there as well, and the trait-aligned priors slightly improve over chat-data priors.
  • Some results on the evil trait only. All new experiments cover the five traits, including the v2x comparison, which was evil-only in the submission.
  • The FineWeb GLP does not help, unlike in Luo et al. We traced this to 1) Luo et al. using Llama-3.1-8B-Instruct instead of the 1B model we use, and 2) the steering vectors in our paper and the original Luo et al. are slightly different. Our extraction is a simplified version of the procedure of Luo et al. and gives shorter vectors. With vectors from their exact procedure and 8B models, we get very similar results on the traits they evaluated on (Figs. 34–35 in the Appendix). Sec. 4.2 now describes the two procedures.

Q1. Gaussian prior baseline

We ran it (App. J.5). For a Gaussian the SDEdit map is closed form, so no training is needed. A full-covariance Gaussian fitted to the trait-aligned activations stays close to standard steering at its best noise level and is worse at t = 0.5 (Fig. 27); diagonal and isotropic Gaussians are clearly worse (Fig. 28). So second-moment structure alone does not reproduce the effect, which fits with FD, a second-moment metric, not predicting steering in the limit. This does not, however, make it obvious that FD could not be useful as a proxy metric when comparing different generative models directly: In image generation, FID is known to have the failure mode that reproducing the training data produces very low FID scores. Nevertheless, it in practice it is considered a useful metric of generative quality.

We hope that these address the concerns of the reviewer.

Thank you for the detailed response. If I understand correctly, the v2x gets a better FID but doesn't benefit the downstream task. So I feel this is a mathematical trick rather than a useful contribution for what the paper is actually trying to solve. Besides, I think predicting the x and v at the same time is a widely used method to improve the diffusion model. The idea is not a new thing.

Thank you for the engagement in the review! To reiterate, the goal of the paper is to provide a follow-up analysis on design choices and metrics used in the original Luo et al. paper. While v2x does not improve the performance on the downstream task for now, it does show that the FD score, as proposed in the original paper, is not necessarily a good predictor of downstream performance.

For the point about novelty, we are not aware of the v-to-x parameterisation being a widely used method in diffusion literature, and we think it would be helpful if the reviewer pointed out specific literature that has shown improvements due to it. v-prediction and EDM-prediction do interpolate effectively between epsilon-prediction and x-prediction, but combining x-prediction and velocity prediction is not common practice to our knowledge.

Further, we would like to hear the reviewers comments on the other aspects and contributions of the paper: In particular, we provided a significant amount of additional experiments regarding the steering results. Many of these considered the central claim/contribution that more specific diffusion priors help with the steering performance. To clarify, this was the central downstream task contribution that the paper was claiming in the first place, and this has now been clarified further in the introduction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.