acceptodds
Under review as a conference paper at ICLR 2027

Scalable Attribution and Control of Model Behavior During Training

Abstract

Attributing and controlling model behavior during training requires identifying each example's contribution quickly enough to act before the next update. However, examples in the same training batch can produce similar behavioral changes, making their individual contributions difficult to distinguish. We address this ambiguity through mutual information, accounting for interference within the batch by quantifying how much the combined behavioral change reveals about each example's contribution. We show that this mutual information is a logarithmic function of Behavioral Gradient Uniqueness (BGU). BGU gives the information measure its geometric interpretation. Our Batch-Space Ghost (BS-Ghost) algorithm makes these scores practical inside the training loop through shared computation in batch space, without storing model-sized example gradients. On a complete 1,000-example Qwen2.5-7B-Instruct workload, our BS-Ghost implementation adds 27 seconds (8.0%) to 5.5 minutes of ordinary training. Removal and retraining demonstrate that BGU identifies data that causally shapes final behavior. At each training step, signed information identifies which examples strengthen or weaken the target behavior, explaining how behavior develops during training. Signed information also enables cheap intervention during training: it predicts how changing example weights will affect behavior in the next update. We then use these predictions to choose weights that steer behavior toward a desired target. This makes our framework a practical foundation for scalable oversight and verification of training pipelines and processes, helping evaluators assess model alignment, understand how it develops during training, and guide interventions that shape ongoing learning.

Primary area:
interpretability and explainable AI
Keywords:
behavioral attribution, training-time control, mutual information, Behavioral Gradient Uniqueness, efficient gradient computation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_89QgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to quantify how much each training data point in the current training batch affects model behavior through one-step optimization. When the data in the batch have similar influence, it is challenging to distinguish each data point’s effect. This paper proposes BGU, which uses mutual information to reflect this in-batch relation and distinguish the effect of each training data point. The paper proposes an efficient computation algorithm, BS-Ghost, which uses ghost inner products to avoid direct gradient computation. The experiments on Qwen2.5-7B-Instruct show that data removal using the proposed method reduces the evil score, and that the method requires almost 8% additional training cost compared to vanilla training.

Critical strengths

I agree with the motivation of the paper that it is challenging to distinguish which data in a batch actually affects the model when data in the batch affect it similarly.

Critical weaknesses
  • The proposed method is substantially similar to [1]. [1] already proposes a score that has almost the same structure as BGU, and it already proposes BS-Ghost, which shares the core algorithm and even has the exact same name as the one proposed in this paper. The paper already cites [1]; however, it does not explain what the contribution of the paper is compared to [1].
  • The performance of the proposed method is not better than that of the baselines. The experiments lack essential baselines, random removal and GREATS [2]. In particular, GREATS has a similar motivation to this paper, which is quantifying in-batch data relations, so a comparison with it is essential.
  • The 44 times speedup is not evaluated under the same setup. The paper should provide a fair comparison under the same setup, including the same GPU.

[1] Abdelghafar, Sleem, Maryam Aliakbarpour, and Chris Jermaine. "Auditing Information Disclosure During LLM-Scale Gradient Descent Using Gradient Uniqueness" arXiv preprint arXiv:2510.10902, 2026.

[2] Wang, Jiachen T., et al. "Greats: Online selection of high-quality data for llm training in every iteration." NeurIPS 2024

Questions for authors
  • What is the paper’s contribution compared to [1], which is discussed in the weaknesses?
  • Is the computation time comparable to that of other attribution methods, including GREATS, under the same setup?
4 comments

We thank the reviewer for their review. We also flag this review to the Area Chair as it contains factual errors. For example, the reviewer states that our experiments lack a comparison with GREATS, although GREATS is already evaluated in Figure 1, with implementation details in Appendix D.3. Another example, claiming that the paper is substantially similar to another paper is false and indicates either a very superficial reading for the work or an adversarial reading for the work. We show all of this below.

W1: “The proposed method is substantially similar to [1]. [1] already proposes a score that has almost the same structure as BGU, and it already proposes BS-Ghost, which shares the core algorithm and even has the exact same name as the one proposed in this paper. The paper already cites [1]; however, it does not explain what the contribution of the paper is compared to [1].”

Q1: “What is the paper’s contribution compared to [1], which is discussed in the weaknesses?”

The distinction is substantive: our paper develops a behavioral predictor and uses it to control training. The cited work does not provide this predictor or control system. Our algorithm also introduces new computational techniques that make this control both accurate and efficient during training. The contributions enabling this are:

  • Our central contribution is control of model behavior during training through model-predictive control, instead of the expensive RL-based methods which require many rollouts. For this, we developed our novel feedback control system.

  • A key technical contribution enabling this control is an attribution-based predictive model. We are the first to operationalize data attribution techniques in this way. The controller uses that model to choose example weights as training evolves, evaluating candidate changes without generating candidate rollouts and estimating their rewards.

  • Control requires predicting the direction and size of a behavioral change. Equation (2) defines the response through the actual optimizer which gradient uniqueness does not do. Equations (3)-(5) then account for overlapping behavioral contributions. The quadratic form and ghost kernels in gradient uniqueness provide foundations, which we explicitly credit in Appendix B.2; they do not supply this behavioral predictor or the control system.

  • Our algorithm has multiple new innovations to tackle specific cost and accuracy problems:

    • Follow the optimizer. We differentiate through AdamW's changing moments and adaptive denominator, and account for global clipping. Appendix B.1 derives the clipping correction across all trainable blocks. Appendix C shows why: in the small-perturbation test, including the optimizer reduces median AdamW prediction error from about 56% to below 0.001%.
    • Derive the required behavioral contractions. We perform the optimizer calculation once per behavioral measurement and share the result across examples. Equation (8) and Appendix B.2 derive how to combine this transformed behavioral derivative with training activations and backward errors. These contractions compute behavioral responses, rather than the training-gradient Gram matrix used in gradient uniqueness, without a separate backward pass or model-sized gradient per training example.
    • Efficient engineering of large-scale computations We choose the multiplication order to form smaller intermediate tensors, group compatible blocks, and reuse small adapter gradients to avoid repeated work. Processing behavioral coordinates in chunks accumulates the quantities needed for scores and directions without keeping the full response matrix. Appendix C checks the contractions, including LoRA blocks, biases, and cross-position terms, with FP32 relative error below 0.004%. Table 2 further shows score-preserving acceleration of three existing methods: all nine top-30 selections are unchanged.
    • Reduce repeated behavioral backward passes. Across updates, only behavioral derivatives are reused; example gradients and the optimizer calculation remain current. Refreshing every four updates retains rank correlation 0.97 with fresh derivatives on 816 development examples (Appendix C). Appendix A.6 bounds the reuse error, separately from the error of predicting finite weight changes.
  • Tracing and explaining how model behaviors develops during training (Figure 3) is a key contribution that enables BGU to be used as an efficient in-run monitoring and interpretability tool.

Some shared foundations do not make the contributions identical. Influence-function methods build on one another; Adam-aware Shapley extends in-run Shapley; GREATS and in-run Shapley share a lot of technical similarities and originates from same line of research. We likewise cite gradient uniqueness and identify what we reuse, which is little as demonstrated above.

W2: “The performance of the proposed method is not better than that of the baselines. The experiments lack essential baselines, random removal and GREATS [2]. In particular, GREATS has a similar motivation to this paper, which is quantifying in-batch data relations, so a comparison with it is essential.”

We did include GREATS in Figure 1 and matched random reweighting in Figure 5. The comparisons answer two different questions:

  • Figure 1 tests the attribution signals available for control. We need to predict how changing example weights will move a chosen behavior through the current optimizer, and distinguish contributions that overlap within an update. Figure 1 tests whether the scores identify individual contributions once their effects combine. BGU obtains Spearman correlation 0.784, versus 0.588 for our optimizer-aware response, 0.575 for EK-FAC, and 0.194 for GREATS. Appendix D.3 checks GREATS against its released selection function. In particular, the comparison with our own response isolates the value of accounting for overlapping contributions. This is why we investigated these methods: attribution is the tool used to build our behavioral predictor.

  • Figure 5 tests whether targeted weights control behavior. This is our main task. The attribution methods compared in the paper do not provide the feedback loop that predicts changes, chooses bounded example weights, and repeats as training evolves. Our matched random controller permutes the same weights across examples, preserving their values and total batch weight. It therefore tests whether the effect comes from assigning weights to the right examples, rather than simply changing the distribution of training weights. At step 211, harmful answers are 2/200 with feedback, 71/200 with random weights, and 84/200 with ordinary training. This is the random control for our online intervention; it is not a random-removal result for Figure 2.

Once responses are computed, the controller can evaluate candidate weights without generating candidate rollouts and estimating their rewards. This is the practical reason for building an attribution-based predictor: it enables behavioral control within fine-tuning without a separate rollout-based RL stage.

W3: “The 44 times speedup is not evaluated under the same setup. The paper should provide a fair comparison under the same setup, including the same GPU.”

Q2: “Is the computation time comparable to that of other attribution methods, including GREATS, under the same setup?”

Table 2 isolates the implementation gain under the same V100 setup and the same score definitions. Grad-Dot/Grad-Sim run 1.20-1.30x faster and LESS 2.29-2.32x faster, with all nine top-30 selections unchanged. Thus, these gains do not come from different hardware or changing what a method computes. Separately, ordinary training takes 331.90 s and training with every example scored takes 358.51 s: 26.61 s added, or 8.02%.

We included Concept Influence because it is the closest comparison for attribution using internal behavioral representations. Persona Vector projections are one example of the high-dimensional behavioral measurements our framework supports. Omitting it would omit the most relevant connection.

The full configuration and timing hardware needed to reproduce its reported experiments were not provided, and no public implementation was available to us. Our reproduction had lower fidelity than BGU and was substantially slower than BS-Ghost on V100. We used the authors' published results and timings so that these reproduction difficulties would not penalize their method. We also preserved their exclusion of inverse-Hessian preparation. The approximately 44x figure compares our 26.61 s added scoring cost with the published EK-FAC/Concept Influence costs of 1,161/1,170 s. Table 1 and Appendix F state this comparison's scope; Table 2 supplies the separate matched-hardware evidence.

The published DATE-LM results use A6000, while ours use the older V100. A6000 has greater peak FP32 throughput and memory capacity, so the hardware difference does not give our implementation that advantage. Our LESS implementation takes 2,032-2,062 s on V100 versus approximately 5,400 s published on A6000; the paired V100 measurements above establish the gain without relying on a hardware comparison.

So, GREATS is included in the recoverability test in Figure 1; the matched timings in Table 2 cover Grad-Dot, Grad-Sim, and LESS.

Thanks to the authors for the detailed response. I will maintain my score, and the reasons are given in the comments below.

W1

I still think the paper is substantially similar to [1]. The paper should clarify the difference from [1] and what components are its actual contribution beyond it.

The paper claims its contribution as proposing BGU and BS-Ghost. However, the following components are the same as or substantially similar to [1].

  • BGU and GNQ [1] share the overall structure. The difference is that BGU substitutes the gradient of GNQ with the change in the concept measurement reflecting the optimizer effect.
  • The derivation of BGU in Equations 3 to 5 is very similar to the derivation of GNQ. The two derivations consist of the normal distribution entropy and the matrix determinant lemma. This is the reason for sharing the overall structure mentioned above.
  • Using ghost kernels to calculate BGU in batch space without storing the per-example gradients is almost the same as [1]'s BS-Ghost GNQ, and the name of the method is the same.

I acknowledge that the paper contributes beyond [1]; however, for the reasons above, I think BGU and BS-Ghost are highly grounded in [1].

Nevertheless, the current manuscript almost does not provide a comparison with [1]. In the authors' response, they say they already explicitly credit the quadratic form and ghost kernel in Appendix B.2. However, their actual citation is only one sentence: "We extend the ghost kernels of Abdelghafar et al. (2026) to compute optimizer-aware behavioral responses without materializing full per-example gradients." It explains the relation with [1] too briefly and does not compare the similarity of the quadratic form. Furthermore, in the main text, the paper does not cite [1] even once. Regarding the relation and similarity with [1], the paper should importantly address the comparison and the remaining contribution beyond [1] in the main text.

I agree that the shared foundation does not mean an identical contribution, and I don't argue that the shared foundation is a problem of the paper. However, insufficient comparison with the related work that provides the foundation of the paper is a significant problem, leaving room for the contribution of the paper to be overestimated.

W2

First, I would like to clarify that W2 is a comment on Figure 2 of Section 4.1. I'm sorry that I didn't explicitly mention it in the review, causing a misreading for the authors. In my view, Figure 1 is the minimal controlled experiment that shows how the proposed method works, since it uses the small-scale model GPT-2 and synthetic data. This type of experiment gives its own meaning; however, an experiment in a realistic setting is needed to show whether the method can be used in those settings. In the paper, Section 4's experiments show the results in realistic settings, and especially Figure 2 is the most important experiment that shows whether the proposed method could change the model behavior beyond the baselines. However, BGU does not perform better than the baselines on two of the three datasets the paper tested. Further, the random removal baseline and GREATS are essential baselines for this experiment, so they should be compared.

W3

I am convinced by the authors' response that a time comparison to Concept Influence is challenging. However, if so, the paper should not argue in the introduction that the method is 44 times faster than EK-FAC and Concept Influence, because it is not a fair comparison regardless of whether it is favorable or unfavorable to the paper, since the GPU highly affects the time comparison. Additionally, the authors seem to have run EK-FAC on their own, so measuring the time on the V100 and reporting it should make the comparison fair.

[1] Abdelghafar, Sleem, Maryam Aliakbarpour, and Chris Jermaine. "Auditing Information Disclosure During LLM-Scale Gradient Descent Using Gradient Uniqueness" arXiv preprint arXiv:2510.10902, 2026.

This reply is another proof that your reading for the work is completely adversarial, and we still flag your review to the area chair. You are just reiterating stuff we addressed so we will focus on showing how your reading for the work is not aiming to improve the work rather you are just aiming to create false reasons to reject the work.

Regarding W1. You are just reiterating things we already fully addressed in Part 1/2. So, If stating these differences in the work explicitly is what's remaining, we are happy to add this to the paper.

Regarding W2.

  • First in Figure 2, we already achieve better results than the "baselines" unlike your claim which is again factually incorrect. In GSM8K Mistakes dataset, BGU is better than all baselines. In Misaligned Opinions dataset, BGU is better than Vector Filter and Influence Function. In all datasets, BGU is better than Influence Functions, comparable to all the other methods.
  • Second, there is NO baselines to our work None of the methods control model behavior during training and other contributions we explained in detail in our Part 1/2 answer to you. And, yet again, you are claiming false claims saying that "Figure 2 is the most important experiment". That's false, we never claimed that nor it is true. The point of this experiment as we stated in the paper is to show that BGU IDENTIFIES DATA THAT CAUSALLY SHAPES FINAL BEHAVIOR which is the title of the section.

Regarding W3. Yet, another proof that your reading for the work is adversarial and cheery picking, when you referred to the importance of the GPU, while you will never acknowledge:

  • our results in Table 2 which we already put it for you in Part 2/2 which clearly demonstrate the key algorithmic innovations we came up with. since you love to reiterate we will reiterate this for you The published DATE-LM results use A6000, while ours use the older V100. A6000 has greater peak FP32 throughput and memory capacity, so the hardware difference does not give our implementation that advantage. Our LESS implementation takes 2,032-2,062 s on V100 versus approximately 5,400 s published on A6000; the paired V100 measurements above establish the gain without relying on a hardware comparison.

  • Even in the experiment you are referring to, the time we are reporting is completely sufficient even without comparing with other methods. 8% overhead is objectively a low overhead!

Lastly, you want to convince us that this one sentence that we used to refer to the cited work [1] which you built your rejection upon, was magical that let you go do all these investigations of how this work is different than the other work. Really!

We do not care about your score at this moment, and that's why we flagged you to the area chair. Unfortunately, reviewers like you do all of this because they know their identity will never be in public.

Reviewer_oF9vOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper proposes a framework for attributing and controlling model behavior during training. It first computes how changing each example’s weight would affect a behavioral measurement through the current optimizer update. It then measures how distinguishable each example’s response is from the other responses in its batch. They are then used to reweight training examples and suppress unwanted behavior.

Critical strengths
  1. The problem is useful and well motivated. It is also a unique problem compared to many previous TDA works.

  2. The results on removing data are strong.

  3. The speedup results on DATE-LM are also strong.

Critical weaknesses
  1. The writing for the method section and the experiments looks sloppy and not well written.

  2. The response-only controller achieves similar suppression of the internal persona score as the signed-information controller in the appendix. This makes it unclear what the information term adds?

  3. The relationship to Gradient Uniqueness needs to be clearer. The earlier work [1] you already cite introduces the same quadratic form and batch-space computation. The paper should distinguish these existing components from your work. Currently, I think the novelty is low.

  4. Figure 2 seems to take the results for the four baselines from Concept Influence. But Appendix D.1 states that the evaluation conditions are unmatched. What are the differences? It seems very hard to trust the results if the settings are unmatched in some ways. This is the main reason for a strong reject.

  5. The utility evaluation is performed at the final checkpoint, whereas the main improvement in harmfulness occurs during training. Measuring utility at the same checkpoints where feedback suppresses harmful behavior would be useful.

  6. The authors do not do any of the popularly used evaluation/ metrics in TDA papers like rank correlation between corpus scores and the changes in final behavior under leave-one-out retraining in the GPT-2. Computing LDS would also be helpful.


[1] Abdelghafar, Sleem, Maryam Aliakbarpour, and Chris Jermaine. "Quantifying Information Disclosure During Gradient Descent Using Gradient Uniqueness." (2025).

Questions for authors
  1. The recoverability experiment compares a one-step intervention with baselines that aggregate attribution across multiple training steps or checkpoints. Since these scores describe different parts of training, their recovery performance is not directly comparable?

  2. Appendix D.1 scales λ\lambda with each batch’s response magnitude, so scaling all responses in a batch leaves its information scores unchanged. How does this affect the corpus ranking compared with using the same λ\lambda throughout training, particularly when some updates change behavior more strongly than others?

  3. Figure 4 tests downweighting selected examples, whereas the controller in Figure 5 also upweights other examples. Can you report prediction error for the full weight vectors actually applied during feedback?

  4. Table 1 reports the cost of computing attribution scores. What is the overhead of the complete feedback procedure in Figure 5, including evaluating behavior after the ordinary update and solving the weight optimization?

  5. Why does the controller in Appendix E.3 prohibit upweighting examples with positive projected responses? When all responses are positive, keeping total batch weight forces every weight to remain unchanged. But moving weight from examples with larger positive responses to those with smaller positive responses could still reduce the predicted behavioral increase. I don’t understand the intuition here.

3 comments

Thank you for recognizing that our problem is “unique” compared to previous TDA work, and for recognizing the strong removal and speedup results.

W3: “The relationship to Gradient Uniqueness needs to be clearer. The earlier work [1] you already cite introduces the same quadratic form and batch-space computation. The paper should distinguish these existing components from your work. Currently, I think the novelty is low.”

Please check our Part 1/2 answer to Reviwer 89Qg.

W2: “The response-only controller achieves similar suppression of the internal persona score as the signed-information controller in the appendix. This makes it unclear what the information term adds?”

The response and information terms have different jobs. The response predicts whether the weights will reach the behavioral target; the information term in Equation (10) guides which weight changes are less costly. Different allocations can reach the same target, so comparable suppression does not test whether individual contributions are distinguishable.

Figure 1 tests that additional question. When examples produce overlapping effects, response magnitude alone does not describe how identifiable each contribution is. BGU achieves recovery correlation 0.784 versus 0.588 for our optimizer-aware response, using the same responses. We reported both controllers because control succeeds with both allocation choices on this dataset.

Q1: “The recoverability experiment compares a one-step intervention with baselines that aggregate attribution across multiple training steps or checkpoints. Since these scores describe different parts of training, their recovery performance is not directly comparable?”

Our controller needs to decide what will help at the next update. An example can strengthen a behavior at one training stage and weaken it at another. A score accumulated over a trajectory can therefore correctly describe the history without identifying the example's contribution now. Figure 1 tests whether existing signals answer this local question; it does not redefine their original attribution tasks.

The decoder uses executed behavioral changes, not attribution scores, and all methods use the same reference questions. We also compare BGU with our optimizer-aware response using the same local derivatives. That comparison cannot be explained by different checkpoints or aggregation: it directly tests the benefit of accounting for overlapping contributions.

W6: “The authors do not do any of the popularly used evaluation/ metrics in TDA papers like rank correlation between corpus scores and the changes in final behavior under leave-one-out retraining in the GPT-2. Computing LDS would also be helpful.”

The two tests predict different consequences of changing the data. LDS and leave-one-out retraining ask what happens to the final model after modifying the training dataset. Our predictor asks what happens at the next optimizer update after changing the current batch's weights. Retraining also changes later gradients and model states, so the two questions need not give the same ranking.

For control, we test the quantity we actually use: execute ordinary and reweighted updates from the same state and compare the measured correction with our prediction (Figure 4). We then test repeated feedback because accurate local predictions must remain useful as training evolves (Figure 5). Removal and retraining separately test our corpus-selection application. A final-model attribution metric cannot replace these tests of the predictor and controller.

Q2: “Appendix D.1 scales λ with each batch’s response magnitude, so scaling all responses in a batch leaves its information scores unchanged. How does this affect the corpus ranking compared with using the same λ throughout training, particularly when some updates change behavior more strongly than others?”

With a fixed resolution, batches with the same pattern of relative effects can receive different scores simply because one update's responses are larger. Our scaling removes that preference: the corpus ranking asks how distinctive an example is relative to the effects present when it was trained. A fixed resolution can therefore change rankings within and across batches. The ranking measures relative distinctiveness rather than absolute behavioral movement.

Absolute movement is still needed for control. The predictor used in Equation (10), defined in Equation (25), retains the signed response magnitudes, so a larger predicted correction remains larger even when the information score is scale-invariant.

W1: “The writing for the method section and the experiments looks sloppy and not well written.”

We will organize the introduction and method around the control problem: explain what we want to control, how we build the predictor, and how we use the predictions to choose example weights. We will then connect each experiment directly to the question it answers.

W4: “Figure 2 seems to take the results for the four baselines from Concept Influence. But Appendix D.1 states that the evaluation conditions are unmatched. What are the differences? It seems very hard to trust the results if the settings are unmatched in some ways. This is the main reason for a strong reject.”

Concept Influence is the closest comparison for attribution using internal behavioral representations. Persona Vector projections are one example of the high-dimensional behavioral measurements we support. Excluding Concept Influence would omit the most relevant related approach.

The obstacle was reproducing its reported experiments exactly: the full configuration was not provided, and no public implementation was available to us. Our reproduction had lower fidelity than BGU and was substantially slower than BS-Ghost on V100. Using those results could confuse implementation differences with the quality of their method. We therefore used the authors' published results and timings, explicitly identified as external references in Figure 2 and Appendix D.1. The comparison combines their reported runs with our local runs. We also preserve their exclusion of inverse-Hessian preparation.

Our causal removal claim comes from changing the training data and retraining from matched initializations. Across three such pairs, with 200 held-out answers per run, removing 10% of GSM8K Mistakes reduces mean evil from 11.89 to 2.38. Table 3 reports the individual runs and utility. This tests the effect of our selected removals independently of the external curves. We will explain why we chose Concept Influence and how its published results are used beside Figure 2.

Q3: “Figure 4 tests downweighting selected examples, whereas the controller in Figure 5 also upweights other examples. Can you report prediction error for the full weight vectors actually applied during feedback?”

The controller in Equation (10) uses a predictor that accounts jointly for increased and decreased example weights, as explicitly defined in Equation (25). Appendix A.6 analyzes the combined intervention for arbitrary positive weights, so applying the predictor to the controller does not require extending a downweight-only formula.

The empirical error measurement has a narrower role: Figure 4 tests 339 selected-downweighting interventions, with mean vector cosine 0.84 and projected RMSE 0.042. Joint upweighting and downweighting can produce additions or cancellations that differ from those interventions. Figure 5 tests whether repeated feedback works, but does not report a separate prediction-error statistic for the complete weight vectors applied by the controller.

Q5: “Why does the controller in Appendix E.3 prohibit upweighting examples with positive projected responses? When all responses are positive, keeping total batch weight forces every weight to remain unchanged. But moving weight from examples with larger positive responses to those with smaller positive responses could still reduce the predicted behavioral increase. I don’t understand the intuition here.”

The rule makes every changed example push in the intended direction under the local predictor. Downweighting a positive response and upweighting a negative response both predict a reduction, so control need not rely on one helpful change compensating for another harmful change. This is a conservative policy choice, not a restriction of the predictor.

You are right about the cost: with only positive responses, preserving total weight leaves every weight at one. Transferring weight from a larger positive response to a smaller one can reduce the predicted increase, but permits an individually adverse upweighting compensated by downweighting elsewhere. That is a valid, less restrictive controller choice. We will make this tradeoff explicit in Appendix E.3.

W5: “The utility evaluation is performed at the final checkpoint, whereas the main improvement in harmfulness occurs during training. Measuring utility at the same checkpoints where feedback suppresses harmful behavior would be useful.”

Our utility evaluation asks whether behavioral control preserves the capability of the resulting model after the same number of training updates. We therefore evaluate MMLU after both training runs finish: 73.81% for ordinary training and 73.79% with feedback.

Preserving MMLU at every intermediate checkpoint is not a meaningful success criterion for this comparison. Controlled and ordinary training can follow different learning trajectories while reaching comparable final capability within the same training length. Requiring intermediate capability to remain unchanged would impose an additional restriction on the learning process without establishing whether the resulting model is better or worse. The relevant criterion is the capability achieved after the same number of updates, which our evaluation directly measures.

Q4: “Table 1 reports the cost of computing attribution scores. What is the overhead of the complete feedback procedure in Figure 5, including evaluating behavior after the ordinary update and solving the weight optimization?”

The weight optimization adds nearly no overhead in our experiments. It adjusts one weight per example in the batch (Equation (10)) using the behavioral responses already computed, without additional model forward/backward passes or generated rollouts for each candidate.

We will clarify the timing scope and controller choices in the paper. We hope this addresses your concerns.

Reviewer_4afUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method to attribute and control model behavior to specific training examples at training time. The proposed Behavioral gradient uniqueness (BGU) method computes both the contribution and directionality of each training example with minimal overhead to the training cost. On a post-training run with Qwen-2.5-7B model, BGU method adds just 8% latency overhead.

Experiments cover two aspects: attribution and control. With three baseline methods, Grad-Sim, Grad-Dot and LESS, BGU shows significant gains in latency of computing attribution scores. Section 4 also includes a discussion of steering behavior using BGU.

I think the main weakness of this paper is a lack of comparison of recent attribution or data selection methods (see my comments below). Table 1 includes a latency comparison with influence functions but there is no comparison of the attribution quality with these methods from Table 1. Some of the recent works in influence functions also target latency with projected gradient based methods.

Critical strengths
  • Paper tackles a hard problem of data attribution at (post-)training time for large scale models (here, 7B). This is an important space and as the authors show it has useful applications in steering model alignment or identifying low-quality examples quickly. Latency results in Table 1 shows the minimal overhead imposed by the proposed BGU method.
Critical weaknesses
  • Some of the attribution methods used in the work (LESS, Grad-Dot, Grad-Sim in Table 2) are relatively old, and doesn't include any new methods introduced in the last two years. I want to note that there is a comparison to recent methods in Figure 1(c) but the setup is not clear to me and a bit confused why these were not included in Table 2.
  • For behavioral control experiments, there is no comparison to influence function based data selection methods. The only baselines in Figure 5 are default training and matched random.
Questions for authors
  • minor editing comment: define BGU when you first refer it in line 48 of the introduction.
  • another editing comment: if figure 5 from page 9 is important enough to reference in first paragraph of introduction, I'd recommend moving it.
  • how does this method work for full-parameter finetuning?
1 comment

Thank you for recognizing the importance of controlling model behavior during training, its applications to steering alignment, and the low measured overhead.

W1: “Some of the attribution methods used in the work (LESS, Grad-Dot, Grad-Sim in Table 2) are relatively old, and doesn't include any new methods introduced in the last two years. I want to note that there is a comparison to recent methods in Figure 1(c) but the setup is not clear to me and a bit confused why these were not included in Table 2.”

Figure 1 compares attribution signals; Table 2 isolates the contribution of our computation.

  • Figure 1 tests a difficulty that a large influence value alone cannot resolve. An example can have a large behavioral effect yet be hard to identify because other examples produce similar effects. We change example weights from the same training state and ask a decoder to recover those changes from the measured behavior. The decoder receives no attribution scores, giving an independent test of which contributions remain identifiable. Recent methods, including MAGIC, LinFAC, Adam Shapley, and GREATS, are included. Using the same optimizer-aware responses, BGU obtains correlation 0.784 versus 0.588 for our optimizer-aware response. This isolates the value of accounting for overlap.

  • Table 2 holds the attribution method fixed and changes how we compute it. If both changed, a speedup could simply come from computing a different quantity. Here, all nine top-30 selections are unchanged. On the same V100, Grad-Dot/Grad-Sim run 1.20-1.30x faster and LESS 2.29-2.32x faster. This demonstrates an algorithmic contribution even when we use another method's score. The three methods in Table 2 are the ones for which we provide this paired implementation comparison; they are not the full set of quality baselines in Figure 1.

W2: “For behavioral control experiments, there is no comparison to influence function based data selection methods. The only baselines in Figure 5 are default training and matched random.”

An attribution method becomes a control baseline only after specifying how its scores choose an intervention. Our controller must decide how much to change each weight, predict the resulting behavioral correction through the current optimizer, and respect bounds and total batch weight. Equations (2) and (10) connect those decisions. An influence-based controller would also need a weight-allocation rule and a policy for updating its information as training evolves. These choices determine the intervention being compared; an attribution formula alone does not specify them.

Our controls separate the value of targeting from the amount of reweighting. Matched random receives the same weights at the same updates, shuffled across examples. Thus it holds the weight values and timing fixed. The difference between 2/200 harmful answers with feedback and 71/200 with random weights at step 211 supports assigning weights according to behavioral effects. Appendix E.3 also compares response-only feedback: the predictor, constraints, and feedback policy stay fixed while the information penalty is removed. That isolates the additional information term within our control system.

The predictive model lets us evaluate candidate weight assignments without generating rollouts and estimating rewards for each candidate. This is how we use attribution to make control practical during training.

Q3: “how does this method work for full-parameter finetuning?”

Equation (8) applies to a fully trainable weight matrix. It uses the factorization of an affine layer's gradient into activations and backward errors; it does not require a low-rank weight matrix. The final score calculation also remains in batch space, so full-parameter training does not introduce a model-sized matrix inverse.

What changes is the memory strategy. LoRA lets us cache small adapter gradients. With full parameters, we retain, offload, or recompute the activation/error factors until the complete batch gradient is available for the optimizer and clipping calculation (Appendix B.3). Appendix C checks the calculation on models without LoRA. The measured 8.02% overhead specifically concerns rank-32 RS-LoRA.

Q1: “minor editing comment: define BGU when you first refer it in line 48 of the introduction.”

Q2: “another editing comment: if figure 5 from page 9 is important enough to reference in first paragraph of introduction, I'd recommend moving it.”

We will define Behavioral Gradient Uniqueness at first occurrence and move the Figure 5 system overview to the introduction. We will also explain the different purposes of Figure 1 and Table 2 beside those comparisons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.