acceptodds
Under review as a conference paper at ICLR 2027

Behavior Gradient Fields

Abstract

For a given model behavior, e.g., out-of-domain generalization or refusal of harmful requests, many models may exhibit this desired behavior, and there are thus many datasets which could have produced these models. On one hand, the multiplicity of permissible datasets is useful; understanding this space of datasets and their induced models can inform how robust a target behavior is, where it conflicts with other objectives, and whether alternative paths avoid those conflicts. However, the known subset of *inducing datasets* is typically small, with a few datasets either given or produced through simple data-mixing procedures. To this end, we propose **behavior gradient fields**, a framework for tracing model behavior and its geometric implications across data, weight, and behavior spaces; we introduce practical algorithms for exploring these fields via **data steering maps**. Within this framework, we investigate spurious correlations in image classification, as well as steering refusal and sycophancy in LLMs. We use interactions between behavior fields to understand when objectives align or conflict, explain unexpected side effects, and jointly steer behaviors. We also find that some fields are more concentrated, with implications for robustness, and that we can find paths around apparent disjointness in behavior fields. Our results emphasize the consequential properties of the geometry of inducing datasets.

Primary area:
interpretability and explainable AI
Keywords:
data-centric interpretability, dataset geometry, model multiplicity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pBuiOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces behavior gradient fileds, that links have a behavior objective propagates through the training data space and model weight space (Figure 1). Based on the set of inducing datasets, the authors propose data streeing maps, which are algorithms that are used to discover diverse inducing datasets. These data steering maps are used to find paths between different datasets and (possibly) avoid unwanted behavior such as spurious correlations.

Critical strengths
  • The motivation is clearly stated and supported throughout the manuscript, and the data level view of multiplicity is a novel perspective. Figures 1, 3, and 5 are well constructed and directly support the paper’s narrative. The two proposed methods MG-DSM and A-DSM are built on two recent important works: metagradient descent (Engstrom et al. 2025) and probe-based activation (Xiao et al. 2026). Hence, the cover two distinct data steering options.
Critical weaknesses
  • The only critical weaknesses that I see is the lack of baselines to compare against. E.g., in the vision experiments including spurious correlations, MG-DSM is only compared against the linear mixing between two datasets. While this helps in illustrating the dormant spurious correlation mechanism, it is a weak baselines for judging MG-DSM's performance. One can provide simple stronger baselines than convex linear mixing, such as mixing schedules beyond the straight line given between A and B, or a non-convex mixing ratio tuned on the same validation set as MG-DSM. Furthermore, the LLM experiments only compare the proposed MG-DSM and A-DSM against each other, providing simple baselines could help show how much the steering signal itself contributes.
Questions for authors
  • From my understanding, the paper presenets two data steering maps, MG-DSM (Algorithm 1) and A-DSM (Algorithm 2) as different ways to implement the proposed framework, where the first is based on the metagradient and the second on the activation. However, their roles in the paper seem unclear to me, as MG-DSM outperforms A-DSM in nearly every setting. A-DSM seems to serve two purposes, as a baseline comparison against MG-DSM that supports the claim of metagradient based data steering being more successful than activation based data steering. Furthermore, it might also be used to support the multiplicity argument that multiple/different steering paths may lead to same behavior through different data. Could the authors clarify what role does A-DSM specifically play in the paper? If it is provided specifically as a baseline, I suggest framing it as such other than a second proposed algorithm. If not, I suggest explaining in what scenario someone would prefer A-DSM over MG-DSM.
  • To support the comparison between A-DSM and MG-DSM, I suggest reporting the computational cost of A-DSM and MG-DSM.
2 comments

We appreciate the reviewer’s thoughtful feedback and recognition of data-level multiplicity as a “novel perspective.” In response, we have added baseline and computational cost comparisons, as well as clarified the role of A-DSM below.

Q1 (Role of A-DSM): The role of A-DSM is to show that the attribution scoring rule and its underlying assumptions can lead to consequentially different steering paths. A-DSM (activation- based) and MG-DSM (gradient-based) exemplify this difference: (1) for refusal, A-DSM strongly weights safety sources, while MG-DSM finds strong signal in scientific reasoning datasets and identifies more concentrated paths; (2) for sycophancy, A-DSM’s limited gains shows that an effective inference-time steering vector does not necessarily provide an effective data-steering signal. We note that attribution scoring rule is one axis of data multiplicity, and others like joint objectives and the training map are also consequential.

There is also a practical advantage to A-DSM, as activation-based scoring is significantly cheaper than differentiating through training, which we address below. It is therefore both a practical steering method and a contrasting approach for studying multiplicity rather than solely a baseline for MG-DSM.

W1: The only critical weakness that I see is the lack of baselines to compare against. For example, in the vision experiments including spurious correlations, MG-DSM is only compared against linear mixing between two datasets. While this helps in illustrating the dormant spurious correlation mechanism, it is a weak baseline for judging MG-DSM’s performance. One can provide simple stronger baselines than convex linear mixing, such as mixing schedules beyond the straight line given between A and B, or a non-convex mixing ratio tuned on the same validation set as MG-DSM. Furthermore, the LLM experiments only compare the proposed MG-DSM and A-DSM against each other. Providing simple baselines could help show how much the steering signal itself contributes.

Language Model Setting. To help practitioners assess these tradeoffs, we have added computational cost comparisons in Figure 6(c), alongside four additional gradient-based baselines. As with MG-DSM, these methods use gradient information to estimate how training examples affect the behavior proxy, but differ in how they account for the training trajectory, optimizer state, curvature, and gradient magnitude. We also compare the iterative reweighting procedure used here with simpler data filtering across all methods, with details in Appendix A.1.2.

Our new Section 4.3 (“Generality of Data Steering Maps”) additionally extends the evaluation to Llama-8B. Below, we report Llama-8B sycophancy performance and attribution costs; the other settings are shown in Figure 6.

MethodMG-DSMA-DSMTracInTrackStarEK-FAC (Influence Functions)Less
Llama 8B sycophancy reduction (pp)9.10 ± 2.510.42 ± 0.336.70 ± 0.801.69 ± 0.544.88 ± 1.173.57 ± 1.28
Attribution cost (GPU-hours)25.167.2728.058.0827.669.65

Table: Llama-8B sycophancy reduction after three rounds, averaged over four seeds, versus attribution cost (Figure 6). The corresponding refusal comparison, where A-DSM is more effective, is shown in Figure 9.

We find that MG-DSM achieves a larger sycophancy reduction at similar computational cost to influence functions (EK-FAC) and TracIn. Across the four model-behavior settings, it achieves the highest final-round mean in three of four. For other methods, we interestingly find performance can vary across settings: TrackStar achieves a 38.7 pp refusal increase on Llama, but reduces sycophancy by only 1.1 pp on Gemma and 1.7 pp on Llama; two hypotheses here are that preserving gradient magnitude or incorporating earlier checkpoints helps capture sycophancy steering signals. We believe that further understanding how these underlying assumptions affect steering signal is an interesting direction for future work.

Prior work [1, 2], which has focused on activation and likelihood-based attribution for post-training behaviors, has shown limited success with steering sycophancy. Our results show that gradient-based methods, particularly MG-DSM, can be more effective in this setting, while refusal responds to a broader range of methods. We have added clarifications in Sections 3 and 4.3 to address both Q1 and W1.

[1] Blank et al. Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimization. arXiv, 2026.

[2] Bergen et al. Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal. arXiv, 2026.

Image/Tabular Setting: For Section 4.1., our focus is on finding a path connecting two datasets while avoiding undesirable intermediate behavior, such as reliance on spurious correlations. Any continuous scalar mixing schedule connecting A to B must pass through every intermediate mixing ratio, including those corresponding to undesirable behavior.

Selecting a mixing ratio using validation loss addresses a related but different objective of finding a single high-performing mixture (rather than a path connecting A to B). We evaluate this baseline using the same validation objective as MG-DSM; note that validation loss can be nonconvex with respect to the mixing coefficient. In both settings, selection favors mixtures near an endpoint. This is consistent with the observed drop in performance of intermediate mixtures along the original mixing path. For our SpuCoBirds setting, it selects 99.5% A / 0.5% B across all five seeds. For tabular, every selected mixture assigns at least 92.5% weight to one source.

To similar effect, we also test a nonlinear sampling schedule (α=t2\alpha=t^2), using the same number of sampled points and identical endpoints, and compare our path-length ratio (ρ\rho), which measures how far the prediction trajectory deviates from the straight path between its endpoints.

SettingOriginal (α=t\alpha=t): detour (ρ\rho): ↓(α=t2\alpha=t^2): detour (ρ\rho) ↓
Class frequency dormancy (SpuCoBirds, ResNet-9)5.54 ± 0.195.38 ± 0.21
Measurement noise dormancy (tabular, MLP)1.81 ± 0.041.81 ± 0.04

Table: Prediction space detour under two sampling schedules for mixing path, for spurious correlations arising from measurement noise or class frequency differences. Mean ± SD across five seeds.

Along with this table, we have added accuracy graphs in Appendix A.3.1, which show that changing the schedule merely shifts the accuracy dip along the schedule parameter without removing it; in SpuCoBirds, the lowest mean accuracy is 53% under both schedules, but occurs at t=0.25t=0.25 for the original schedule and t=0.50t=0.50 under the squared.

Constructing a different path requires changing the relative weights of examples within each source, beyond adjusting the overall A/B ratio. Compton et al. [3], whose work inspired our class-frequency setting, propose source-label balancing, i.e., matching class proportions across sources, whether 50/50 or another shared ratio. By construction, this negates spurious associations induced by class frequency differences.

However, in our tabular setting, sources are already class-balanced, so this intervention is equivalent to linear mixing and therefore cannot mitigate spurious associations. We emphasize that part of our contribution is demonstrating that measurement noise can activate spurious correlations along a mixing path (despite matched class proportions across sources), which to our knowledge, has not been shown prior. This motivates methods such as MG-DSM, which uses gradients of the behavioral objective through training to determine how to reweight examples. We have clarified the above points in Appendix A.3.1.

[3] Compton et al. When More is Less: Incorporating Additional Datasets Can Hurt Performance By Introducing Spurious Correlations. Machine Learning for Healthcare, 2023.

Reviewer_t7fnOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks which training datasets produce a model with a given behavior, and argues that many different datasets do. It calls the gradient of a behavior score with respect to the training-data weights a "behavior gradient field". It then proposes "data steering maps", which follow this gradient to move from one dataset to another. There are two versions. MG-DSM differentiates a behavior loss through the whole training run (a metagradient) and shifts weight between examples. A-DSM ranks examples by how well their activations match a behavior direction. The experiments have two parts. On SpuCoAnimals and a small tabular problem, MG-DSM is compared to linear mixing of two sources. It avoids an accuracy dip in the middle. On LLM post-training, the authors reweight 10,240 Dolci DPO pairs over 5 rounds, with LoRA on Gemma-2 2B, Qwen2.5 1.5B and OLMo-2 1B, 4 seeds, to raise refusal and lower sycophancy.

Critical strengths
  1. The problem and question is very interesting. Differentiating through DPO objective directly to reweight data is a good demonstration. The sycophancy result on Gemma model is the most notable result of this approach.
  2. Measuring side effects of one domain on another is good practice. The asymmetry (sycophancy steering raising refusal, but not vice versa) holds up on all 4 seeds.
  3. The concentration ratios and the path length ratio are useful descriptive tools, even if they are not causal.
Critical weaknesses
  1. What the method returns is one score per training example. The first-round metagradient is exactly a data-attribution score, so the claim that attribution "does not explore the space" is thin. It is not clear what subsequent rounds add.
  2. My biggest concern is that there are no relevant baselines. Sample-level scores are never tested or compared. The only comparisons are linear mixing (synthetic) and A-DSM, which the authors adapted themselves. Plain metagradient descent is not run separately. Per-step validation-gradient reweighting, one-step gradient-similarity selection (similar to LESS), influence-style attribution, and dynamic data-mixing methods are all cited or standard, and none is compared.
  3. Round 1's scores should be compared to prior work on data attribution or data selection. You can design controlled experiments like: retaining the top-K% samples, or removing the K% worst samples, and compare against any of the attribution/selection methods you talk about.
  4. Nothing in the method is specific to "behavior". The score can be any differentiable number. With math, instruction following and code it would be a data-mixture method with its own standard baselines. The paper does not say why refusal and sycophancy, or what is special about these tasks.
Questions for authors
  1. Can you compare the sample-level scores with other attribution and selection methods (LESS, influence-style estimators, per-step reweighting, dynamic data mixing, plain metagradient descent)? For example: keep the top 5, 10 and 20% by score and re-train. Then compare with random selection? Do 5 rounds beat 1 round? This would genuinely help the results be much more convincing to the practitioners working in this field.
  2. Does the method work for non-safety targets, such as math, code and instruction following? If not, why is it specific to behaviors? What does the framework predict that you could not already see from the runs?
  3. Is the data order fixed across rounds? If you retrain the final weights with a different LoRA initialisation and data order, does the behavior hold?
7 comments

We appreciate the reviewer’s thoughtful feedback and recognition of the problem as “very interesting,” along with their interest in our sycophancy results, asymmetric side effects, and geometric tools. In our response, we have added attribution baselines and ablations, alongside clarifications of the framework’s contribution and the role of iterative reweighting, as detailed below.

Framework Clarifications

W1: What the method returns is one score per training example. The first-round metagradient is exactly a data-attribution score, so the claim that attribution “does not explore the space” is thin. It is not clear what subsequent rounds add.

Attribution and steering round clarifications: We agree that the first-round metagradient is a data attribution score. To clarify, our work shows how different ways of parameterizing data steering maps, like choice of attribution algorithm, allows us to explore the broader space of inducing datasets for a target behavior. We show that the choice of scoring rule (for sis_i) can indeed correspond to diverse solutions, as we demonstrate with MG-DSM (gradient-based scores) weighting scientific reasoning datasets while A-DSM (activation-based scores) strongly weights safety ones for refusal. We additionally show how incorporating secondary objectives (ψ\psi) produces diverse tradeoffs, as in Sections 4.1 and 4.2.3. Choices of the training map, e.g., initialization and data order as controlled by the seed, and starting point via the initial data pool are also consequential for this exploration.

The purpose of multiple reweighting rounds is to iteratively optimize the data weights. As in gradient descent, multiple smaller steps allow the update direction to adapt after each change. Additionally, reweighting explores a larger space of training distributions than filtering, which can only fully drop samples. In our response to W2, we compare filtering with reweighting and show that certain methods, including MG-DSM, benefit strongly from this iterative reweighting.

We have added Figure 7 in Appendix A.1 to provide visual intuition for round-by-round reweighting, and our existing results tables in Appendix B.1. show increasing gains from R1 to R5. We have also added clarifying comments for data steering maps in Section 3 and Section 4.3.

W4(a): Nothing in the method is specific to “behavior.” The score can be any differentiable number. With math, instruction following and code, it would be a data-mixture method with its own standard baselines. The paper does not say why refusal and sycophancy, or what is special about these tasks.

Q2: Does the method work for non-safety targets, such as math, code and instruction following? If not, why is it specific to behaviors?

Behavior specificity and priority: Refusal and sycophancy are chosen for three reasons:

  1. Both behaviors have been hypothesized to be mechanistically related [1]. Intuitively, learning broader resistance to user pressure could increase refusal and reduce sycophancy. Alternatively, a model could learn to refuse based on topic-specific cues (e.g., biosecurity) without changing its sycophancy, as discussed at the beginning of Section 4.2.1. This pair is therefore well suited to investigating whether multiple solutions arise in practice. Our finding of asymmetric side effects -- steering sycophancy affects refusal, while refusal steering has little effect on sycophancy -- is informative in this respect.

  2. They reveal interesting properties and limitations of different steering maps. Refusal provides a case where both A-DSM and MG-DSM work well, but can find very different solutions (r=−0.16r=-0.16 between their attribution vectors, where MG-DSM finds a more concentrated path and weights more scientific reasoning datasets). Sycophancy provides a contrasting case, where even if a behavior is steerable through inference-time activation interventions, the corresponding direction need not be informative for data interventions. We show that gradient-based methods like MG-DSM are more effective in this setting.

  3. Sycophancy can transfer through seemingly neutral preference pairs [2], making its training signal difficult to identify. In contrast, selecting math or code examples provides an intuitive starting point for practitioners improving performance in those domains. There is no similarly obvious selection rule for sycophancy within a general preference pool. Prior literature for this purpose, which primarily test activation-based attribution, have shown limited success [2, 3].

To clarify, we use behavior in the general sense of measurable model outputs or performance; this is consistent with usage in data attribution literature (e.g., Park et al., 2023 “Attributing Model Behavior at Scale”). To support this generality, we chose to apply our framework in a vastly different setting -- generalization in image classification -- to accompany our language model results.

[1] Li et al. Analysing the Safety Pitfalls of Steering Vectors. Findings of ACL 2026.

[2] Blank et al. Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimization. arXiv, 2026.

[3] Bergen et al. Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal. arXiv, 2026.

W4(b): What does the framework predict that you could not already see from the runs?

Framework role and predictiveness: There are several cases our tools can be used in a predictive manner that may be helpful for a practitioners:

Alignment between gradient fields. Along the sycophancy steering trajectory, we found that the sycophancy metagradients showed strong similarity with refusal (r=0.35r=0.35). In settings with similar gradient fields, we would expect there to be interesting cross-behavior interactions (as we end up finding, steering sycophancy strongly impacts refusal, and asymmetrically so), and that joint steering to work well given the compatibility of behaviors, which we find also find to be true (the joint path showed the largest increase of refusal, +17pp, with sycophancy -11pp).

Diagnosing emergent behaviors with ρ\rho. A large detour in prediction space, as measured by ρ\rho, is indicative of models along a path acquiring emergent behaviors, distinct from those learned by the base/endpoint models. These emergent behaviors include latching onto new spurious correlations, as our Section 4.1. examples show. We specifically show how comparatively large ρ\rho values provide a diagnostic for investigating (potentially undesirable) associations that arise during interpolation (Figure 10).

Additional Baselines & Ablations

Q1: Can you compare the sample-level scores with other attribution and selection methods: LESS, influence-style estimators, per-step reweighting, dynamic data mixing, and plain metagradient descent? For example, keep the top 5%, 10% and 20% by score and retrain. Then compare with random selection? Do five rounds beat one round? This would genuinely help the results be much more convincing to practitioners working in this field.

Additional baselines (also addressing W2 and W3): Following our response in W1, our primary focus is understanding data multiplicity, and we focus on A-DSM and MG-DSM as two ways to steer data using attribution algorithms with very different assumptions (activation vs. gradient-based) and important tradeoffs. We agree that further baselines and ablations are also important for understanding how other design choices affect data steering.

To further help practitioners assess these tradeoffs, we have added (i) four attribution baselines, (ii) comparisons of filtering versus iterative reweighting for each method, and (iii) experiments on Llama-8B refusal and sycophancy.

Data Steering Comparison Across Attribution Methods

We compare with four additional gradient-based methods: LESS, EK-FAC (influence functions), TracIn (which aggregates gradient inner products across checkpoints), and TrackStar (which uses cosine similarity between optimizer-adjusted gradients). Further details are in Appendix A.1.1.

We evaluate each method within our reweighting setup and compare against filtering in the next section. We find that all methods show larger gains at the final round than at R1, and MG-DSM achieves the highest average R1 gain in all four settings. For reference, the R1 update for Llama drops the bottom 15% of examples and doubles the weights of the top 15%.

Comparing final round scores, MG-DSM performs strongly across all four settings, achieving the highest average in three of the four. Interestingly, the strongest alternative differs across settings:

  • TrackStar performs very similarly to MG-DSM on Llama refusal (38.7 vs. 39.6 pp), but produces much smaller sycophancy reductions (1.1 vs. 12.4, and 1.7 vs. 9.1 pp). Two hypotheses for this are that gradient magnitude, which cosine normalization removes, contains useful sycophancy steering signal, or that final checkpoint scoring misses signals that were informative earlier in training.
  • TracIn produces the highest Gemma refusal gain (15.5 vs. 14.4 pp), but falls significantly behind MG-DSM on Llama refusal (+18.9 versus +39.6 pp).
  • EK-FAC (influence function, which does not use Adam’s optimizer state) achieves ~half MG-DSM’s sycophancy reduction on both models.

Our results show the diversity of attribution methods’ effectiveness across model and behaviors. We believe isolating the contributions of these design choices is an interesting direction for future work.

Llama-8B Refusal Increase (R1->R3)Llama-8B Sycophancy Reduction (R1->R3)Gemma-2B Refusal Increase (R1->R5)Gemma-2B Sycophancy Reduction (R1->R5)
MG-DSM26.89 → 39.584.77 → 9.104.40 → 14.441.58 → 12.35
A-DSM11.41 → 22.140.15 → 0.421.49 → 6.660.08 → 0.71
TracIn7.63 → 18.933.27 → 6.703.65 → 15.471.13 → 9.92
TrackStar16.99 → 38.730.59 → 1.691.61 → 10.570.19 → 1.12
EK-FAC (Influence Function)16.07 → 31.063.46 → 4.882.16 → 7.920.55 → 5.23
LESS7.00 → 20.110.46 → 3.571.83 → 9.730.51 → 3.77

Steering gains across reweighting rounds for six attribution methods (Figure 6, Section 4.3). We indicate with bold and italics the highest and second-highest round means in each setting. We additionally report attribution cost comparison curves in Figure 6(c).

Reviewer_2LSxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers the not-so-researched problem of mapping behaviors exhibited by LLMs back to their datasets. Currently, this is quite heuristic and is an art, but the authors propose the idea of behavior gradient fields which propagate behavior objectives through the training procedure to model weights and data spaces. In order to make behavior gradient fields practical, they propose two types of "data steering maps" (DSMs) - one that uses metagradients (proposed in Engstrom et al (2025)) to differentiate a proxy loss through the training map wrt the data weights (MG-DSM). The second approach (A-DSM) uses behavior directions found with probe-based attributions of behavior proxy pairs. It then identifies data weights that result in better alignment of the activation difference for the training pairs with the first behavior directions. The methodology for re-weighting is similar to that of MG-DSM.

The other interesting part of the paper is where they show spurious correlations may occur due to data mixing and how the MG-DSM method can avoid this by using appropriate weights. They also compute the behavior geodesic in the prediction space and show that MG-DSM paths in the prediction space can be closer to the behavior geodesic compared to the baseline linear mixing. Experiments with LLMs for steering one behavior alone or two behaviors together (refusal, sycophancy) are promising particularly for MG-DSM. These analyses also show some surprising effects such as Gemma upweighting non-safety datasets for steering refusal.

Critical strengths
  • The problem itself is quite exciting and new, and I would count this as one of the main strengths of the paper. As the authors point out, currently data mixing for model training is not well-studied and the attempt here (particularly after Engstrom et al's work that made it practical) counts as a definite strength.

  • The second thing that attracted me in the paper was the demonstration of spurious correlations via class frequencies and noise. Showing how this happens by tracing the path between models obtained by mixing datasets with linear and MG-DSM mixing is definitely interesting and a new insight.

Critical weaknesses
  • One of the main weaknesses of the paper is the clarity and the lack of explanation for certain choices made. The big one here is how the authors handle the high-dimensional weight space - for neural networks in particular, there is an equivalence class of weights and the metric chosen should be invariant to that. I am not sure if L2L_2 is the right metric for this. Similarly for data, the use of sliced W2W_2 must be explained - why is this a good metric. The least risky is the metric in the behavior space which is easy to justify. Along the same lines, I could not understand why the behavior geodesic is posed as the gold-standard that the interpolated models need to keep up with.

  • While the behavior gradient fields in figures 2 and 3 look good, it is hard for me to understand how accurate they are because all embedding methods lose information and particularly for high-dimensional data spaces, this will be quite tricky. The authors compute and mention κ^D→Φ\hat{\kappa}_{D \rightarrow \Phi} for MG-DSM and A-DSM, but this can also be done for linear mixing since that is the baseline. There should be some qualification associated to these down-projected gradient fields and may be some controls to ensure some level of accuracy.

Questions for authors

A. I would be glad to know some clarifications regarding the two main weaknesses listed above.

B. The authors can also consider answering the following to the extent possible.

  1. Experiments in section 4.1 focus on spuriousness introduced in image datasets. Is there some analog for this insight in language datasets?

  2. In Sec. 4.2.1, how did you choose the 128 prompts per behavior - why 128?

  3. In Sec. 4.2.1, how are the behavior proxy prompts different from the 10,240 pairs chosen for training? How are these two used (particularly for AG-DSM)?

  4. How does averaging two individual fields to get thenjoint contour make sense in Figure 5? Is there some justification for this?

  5. Minor: The optional objective Ψ\Psi in the algorithms is not defined till much later.

5 comments

We thank the reviewer for their thoughtful feedback and positive comments on the problem and our spurious correlation findings. We have clarified our metric and visualization choices and added analyses in response.

W1: One of the main weaknesses of the paper is the clarity and the lack of explanation for certain choices made. The big one here is how the authors handle the high-dimensional weight space: for neural networks in particular, there is an equivalence class of weights and the metric chosen should be invariant to that. I am not sure if L2L_2 is the right metric for this. Similarly for data, the use of sliced W2W_2 must be explained: why is this a good metric? The least risky is the metric in the behavior space, which is easy to justify.

Metric and embedding clarifications: To help clarify our metric and embedding space choices, we have added Appendix C, covering details including:

  • We use sliced W2W_2 metric for data space as it allows us to compare dataset distributions with disjoint supports while being cheaper to compute than a full optimal transport solve. In Appendix C, we added a (2×22\times2) ablation for refusal, crossing two common language embeddings -- MPNet (used in the main figure) and MiniLM -- with sliced W2W_2 and debiased Sinkhorn divergence, an alternative that can also compare distributions with disjoint supports by solving a regularized optimal transport problem.

    We find consistent qualitative conclusions, where concentration (κ^D→Φ\widehat{\kappa}_{D\to\Phi}) is higher for Gemma MG-DSM relative to OLMo and Qwen MG-DSM (which have similar values), as well as relative to Gemma A-DSM; this is especially important to note, given Gemma MG-DSM’s counterintuitive relationship between training data and refusal, including strong steering signals in scientific reasoning datasets.

MG-DSM

EmbeddingDistanceGemmaOLMoQwen
MPNetSliced W2W_20.9000.4010.454
MiniLMSliced W2W_20.6090.2850.317
MPNetSinkhorn divergence0.4940.2440.259
MiniLMSinkhorn divergence0.3870.1900.209

A-DSM

EmbeddingDistanceGemmaOLMoQwen
MPNetSliced W2W_20.3360.2530.329
MiniLMSliced W2W_20.2610.2090.258
MPNetSinkhorn divergence0.2850.2380.317
MiniLMSinkhorn divergence0.2230.1900.268

Tables: Refusal concentration κ^D→Φ\widehat{\kappa}_{D\to\Phi} across embeddings and distances (Appendix C). Bold shows the highest value within each method and row.

  • For images, we use DINOv2 features, motivated by evidence that DINOv2-based metrics have better alignment with human perception of image fidelity as compared to other reasonable alternatives (e.g., Inception-based features) [1]

  • In our weight space plots, distances are computed only within each initialization seed, so the models begin with the same parameter ordering. We note related work finds that models trained or fine-tuned from a shared initialization often lie in a single low error basin (motivating “model soups” [2]). For models trained from different initializations, permutation or OT alignment [3] could further account for these symmetries. We instead compute weight distances within each seed and overlay the separately embedded trajectories for visualization with MDS. Importantly, our main conclusions do not depend on weight distances, as ρ\rho is defined in prediction space, while κD→Φ\kappa_{D\to\Phi} measures behavioral change relative to data distance. This was a motivation of our emphasis on multiplicity in data space, to complement the extensive literature on model weight multiplicity, e.g., the Rashomon effect.

[1] Stein et al. Exposing Flaws of Generative Model Evaluation Metrics and Their Unfair Treatment of Diffusion Models. NeurIPS 2023.

[2] Wortsman et al. Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy Without Increasing Inference Time. ICML 2022.

[3] Ainsworth et al. Git Re-Basin: Merging Models Modulo Permutation Symmetries. ICLR 2023.

W2: While the behavior gradient fields in Figures 2 and 3 look good, it is hard for me to understand how accurate they are because all embedding methods lose information, and particularly for high-dimensional data spaces, this will be quite tricky. The authors compute and mention κ\kappa for MG-DSM and A-DSM, but this can also be done for linear mixing since that is the baseline. There should be some qualification associated with these down-projected gradient fields and maybe some controls to ensure some level of accuracy.

Role and interpretation of behavior gradient field figures: We would like to clarify that Figures 2 and 3 are illustrative. Their goal is to provide visual intuition that is consistent with our quantitative results, and that our conclusions are supported by but not derived from down-projected quantities. For example, κ^D→Φ\widehat{\kappa}_{D\to\Phi}, which visually correspond to tightness of contours, is calculated from the full, unprojected trajectory. The same holds for ρ\rho and any reported correlation and cosine similarity values.

Role and interpretation of our geometric tools: Regarding the application of κ\kappa and ρ\rho: these can be viewed as somewhat analogous quantities, where κ\kappa is contraction across spaces, while ρ\rho is that with respect to a reference path within the same space. We have organized our results in line with what we would believe to be most helpful for practitioners working in the respective settings: for safety-relevant behaviors in language models, small data changes causing large behavior changes (κ\kappa) is an important insight, while data mixing showing large prediction detours (ρ\rho) as models latch onto “emergent behaviors" is a helpful diagnostic.

In the spurious correlation setting, κ\kappa captures sensitivity to both increases and decreases in accuracy along a path connecting fixed endpoints. In the image setting, we find MG-DSM has higher κ\kappa than linear mixing, which reflects its sharp initial accuracy increase relative to the data change. In the tabular setting, MG-DSM has lower κ\kappa, as there it maintains accuracy while linear mixing shows a drop, which we have added a table and discussion for in Appendix A.3.1.

Q1: Experiments in Section 4.1 focus on spuriousness introduced in image datasets. Is there some analog for this insight in language datasets?

W1 (b): Along the same lines, I could not understand why the behavior geodesic is posed as the gold standard that the interpolated models need to keep up with.

Is there an analogue of spuriousness to language datasets?

Yes -- a close analogue comes from work that examines “emergent generalization” [4] in language models. An illustrative example is Qin et al. [5] (“Sometimes I am a Tree”), which shows how data composition shifts language models between learning shortcuts, such as matching a verb to its nearest noun, and hierarchical grammar rules that generalize much better. In particular, they show that intermediate data mixing settings can have very unstable generalization. We think this is an exciting direction, as well as helpful framing, and have added relevant discussion in Section 4.1.

[4] Zhao et al. Random Scaling of Emergent Capabilities. 2025.

[5] Qin, Saphra, and Alvarez-Melis. Data Drives Unstable Hierarchical Generalization in LMs (“Sometimes I am a Tree”). EMNLP 2025.

Behavior geodesic implication:

The behavior geodesic is not a gold standard or optimization target, but a reference path (ρ=1\rho=1) for measuring detours in prediction space. MG-DSM never optimizes directly for it, and the geodesic may be unreachable within the hypothesis class.

The corresponding intuition is: if there have been large changes in associations learned along a path of models (broadly, “emergent behaviors,”), we may expect there to be a geometric trace of this, especially within prediction space. Spurious correlations are an illustrative example of optimization “shortcuts” that are very distinct from generalizable predictive rules and can therefore produce large detours in prediction space. The behavior geodesic provides a reference for measuring these detours, which may reflect models learning uncharacteristic, emergent behaviors along the path.

Note that it is insufficient to look solely at scalar scores. For example, in our SpuCoBirds experiment, the two paths have nearly equal magnitudes of deviation (approximately 6–7 pp) from endpoint accuracy.

To complement our interpretation, we have added qualitative analysis in Appendix A.3, ranking examples by their contributions to linear mixing’s detour. We find that looking at the highest-ranked examples’ prediction trajectories helps characterize the underlying behavior (here, background over-reliance).

Further Clarifications

Q2: In Section 4.2.1, how did you choose the 128 prompts per behavior? Why 128? In Section 4.2.1, how are the behavior proxy prompts different from the 10,240 pairs chosen for training? How are these two used, particularly for AG-DSM?

Behavior proxy prompts vs. training: The set of behavior proxy prompts are disjoint from the training prompts. For sycophancy, for instance, the proxy prompts are pulled from Perez et al. (2023) sycophancy tasks (also disjoint from the evaluation set), whereas training prompts are from Ai2’s Dolci-DPO dataset. For the proxy, responses are generated by the model being steered. For MG-DSM, the proxy pairs define the outer objective differentiated through training. For A-DSM, these contrastive pairs define a behavior direction in model activation space, called a steering vector, which is used to score training pairs by their activation alignment. We have described our proxy-building procedure in more detail in Appendix A.1.

Our choice of 128 prompts was motivated by related work that calculates these steering vectors, which are typically used for changing model behavior at inference-time; the canonical paper of Arditi et al. [6] uses 128 harmful and harmless prompts for constructing refusal steering vectors. Similarly, Xiao and Aranguri [7], whose method we build on for A-DSM, construct behavior probes from 150 prompts. We have added an ablation supporting that averaging over more proxy pairs can improve steering up to a point: for Gemma 2B refusal, 128 and 256 pairs produce refusal gains of +10.17 and +10.39 pp, respectively, compared with +7.20 and +7.08 pp for 32 and 64 pairs. We have clarified the above points in Appendix A.1.

Proxy pairs3264128256
Gemma 2B refusal gain (pp)+7.20+7.08+10.17+10.39

Table 7: Ablating number of prompts used to construct behavior proxy, tested on Gemma 2B refusal (three rounds, four seeds).

[6] Arditi et al. Refusal in Language Models Is Mediated by a Single Direction. NeurIPS 2024.

[7] Xiao and Aranguri. Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Training. 2026.

Q3: How does averaging two individual fields to get the joint contour make sense in Figure 5? Is there some justification for this?

Averaging fields for the joint contour: We average the fields in Figure 5 in correspondence with how we define our joint objective, which is an average of the two individual behavior objectives (by linearity, its gradient is also the average of their gradients). There are more advanced techniques for multi-objective optimization that we believe would be interesting for future work. We find the simple averaging setup works for our setting, with the jointly steered trajectory having the best tradeoff of increased refusal / decreased sycophancy (+17 / -11pp). The purpose of Figure 5 (left) is to provide visual prior that the individual fields may be similar; the correlation of the sycophancy metagradients with refusal is r=0.35r=0.35, which is also quantitative evidence of this. From these observations, we may expect that there to be cross-behavior effects and for multi-objective steering to work well, which we find to be empirically true.

Q4: Minor: the optional objective Ψ\Psi in the algorithms is not defined until much later.

Placement: We initially had defined Ψ\Psi in the MG-DSM paragraph, and have moved the definition earlier within Section 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We thank the reviewers for their thoughtful and helpful feedback. We appreciate that reviewers found the problem “exciting and new” [2LSx] and “very interesting” [t7fn], and recognized the data-level view of multiplicity as a “novel perspective” [pBui].

We would like to emphasize that the primary focus of our work is a principled framework, behavior gradient fields, for understanding and eliciting the largely underexplored phenomena related to data multiplicity. Through data steering maps, we explore axes of data multiplicity and demonstrate the consequences of varying design choices, such as the attribution scoring method and secondary behavior objectives.

We chose settings that illustrate important aspects of data multiplicity. This also motivated a complementary empirical contribution of effective data attribution and steering for behaviors such as sycophancy, where prior interventions have shown limited success (Blank et al.; Bergen et al.). While these attribution results are important, our core contribution is understanding the diversity and geometry of inducing datasets, rather than proposing a universally superior attribution method. We believe our empirical attribution results and comparisons across algorithms are nevertheless valuable for practitioners seeking to understand their tradeoffs, as the choice of attribution scoring method is one axis along which we explore data multiplicity. We have therefore added further baselines and analyses in Section 4.3 and the appendix.

Following the reviewers’ recommendations, we have made the following main additions and clarifications:

  • Section 4.3 (“Generality of Data Steering Maps”) compares additional attribution scoring rules and their tradeoffs between behavioral gains and compute (Figure 6), covering Gemma-2B refusal and sycophancy and extending our evaluation to Llama-8B.
  • Appendix A.1.2 shows additional ablations comparing filtering with reweighting, plain metagradient descent with MG-DSM, transfer across initialization and data order, and behavior proxy sizes.
  • Appendix A.3 adds qualitative analysis of examples contributing most to prediction-space detours, further connecting our geometric tool (ρ\rho) to emergent behaviors in image classification.
  • Appendix C clarifies our metric and visualization choices and includes ablations across data embeddings and distances.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.