- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper aims to quantify how much each training data point in the current training batch affects model behavior through one-step optimization. When the data in the batch have similar influence, it is challenging to distinguish each data point’s effect. This paper proposes BGU, which uses mutual information to reflect this in-batch relation and distinguish the effect of each training data point. The paper proposes an efficient computation algorithm, BS-Ghost, which uses ghost inner products to avoid direct gradient computation. The experiments on Qwen2.5-7B-Instruct show that data removal using the proposed method reduces the evil score, and that the method requires almost 8% additional training cost compared to vanilla training.
Critical strengths
I agree with the motivation of the paper that it is challenging to distinguish which data in a batch actually affects the model when data in the batch affect it similarly.
Critical weaknesses
- The proposed method is substantially similar to [1]. [1] already proposes a score that has almost the same structure as BGU, and it already proposes BS-Ghost, which shares the core algorithm and even has the exact same name as the one proposed in this paper. The paper already cites [1]; however, it does not explain what the contribution of the paper is compared to [1].
- The performance of the proposed method is not better than that of the baselines. The experiments lack essential baselines, random removal and GREATS [2]. In particular, GREATS has a similar motivation to this paper, which is quantifying in-batch data relations, so a comparison with it is essential.
- The 44 times speedup is not evaluated under the same setup. The paper should provide a fair comparison under the same setup, including the same GPU.
[1] Abdelghafar, Sleem, Maryam Aliakbarpour, and Chris Jermaine. "Auditing Information Disclosure During LLM-Scale Gradient Descent Using Gradient Uniqueness" arXiv preprint arXiv:2510.10902, 2026.
[2] Wang, Jiachen T., et al. "Greats: Online selection of high-quality data for llm training in every iteration." NeurIPS 2024
Questions for authors
- What is the paper’s contribution compared to [1], which is discussed in the weaknesses?
- Is the computation time comparable to that of other attribution methods, including GREATS, under the same setup?
4 comments
Part 2/2
W2: “The performance of the proposed method is not better than that of the baselines. The experiments lack essential baselines, random removal and GREATS [2]. In particular, GREATS has a similar motivation to this paper, which is quantifying in-batch data relations, so a comparison with it is essential.”
We did include GREATS in Figure 1 and matched random reweighting in Figure 5. The comparisons answer two different questions:
-
Figure 1 tests the attribution signals available for control. We need to predict how changing example weights will move a chosen behavior through the current optimizer, and distinguish contributions that overlap within an update. Figure 1 tests whether the scores identify individual contributions once their effects combine. BGU obtains Spearman correlation 0.784, versus 0.588 for our optimizer-aware response, 0.575 for EK-FAC, and 0.194 for GREATS. Appendix D.3 checks GREATS against its released selection function. In particular, the comparison with our own response isolates the value of accounting for overlapping contributions. This is why we investigated these methods: attribution is the tool used to build our behavioral predictor.
-
Figure 5 tests whether targeted weights control behavior. This is our main task. The attribution methods compared in the paper do not provide the feedback loop that predicts changes, chooses bounded example weights, and repeats as training evolves. Our matched random controller permutes the same weights across examples, preserving their values and total batch weight. It therefore tests whether the effect comes from assigning weights to the right examples, rather than simply changing the distribution of training weights. At step 211, harmful answers are 2/200 with feedback, 71/200 with random weights, and 84/200 with ordinary training. This is the random control for our online intervention; it is not a random-removal result for Figure 2.
Once responses are computed, the controller can evaluate candidate weights without generating candidate rollouts and estimating their rewards. This is the practical reason for building an attribution-based predictor: it enables behavioral control within fine-tuning without a separate rollout-based RL stage.
W3: “The 44 times speedup is not evaluated under the same setup. The paper should provide a fair comparison under the same setup, including the same GPU.”
Q2: “Is the computation time comparable to that of other attribution methods, including GREATS, under the same setup?”
Table 2 isolates the implementation gain under the same V100 setup and the same score definitions. Grad-Dot/Grad-Sim run 1.20-1.30x faster and LESS 2.29-2.32x faster, with all nine top-30 selections unchanged. Thus, these gains do not come from different hardware or changing what a method computes. Separately, ordinary training takes 331.90 s and training with every example scored takes 358.51 s: 26.61 s added, or 8.02%.
We included Concept Influence because it is the closest comparison for attribution using internal behavioral representations. Persona Vector projections are one example of the high-dimensional behavioral measurements our framework supports. Omitting it would omit the most relevant connection.
The full configuration and timing hardware needed to reproduce its reported experiments were not provided, and no public implementation was available to us. Our reproduction had lower fidelity than BGU and was substantially slower than BS-Ghost on V100. We used the authors' published results and timings so that these reproduction difficulties would not penalize their method. We also preserved their exclusion of inverse-Hessian preparation. The approximately 44x figure compares our 26.61 s added scoring cost with the published EK-FAC/Concept Influence costs of 1,161/1,170 s. Table 1 and Appendix F state this comparison's scope; Table 2 supplies the separate matched-hardware evidence.
The published DATE-LM results use A6000, while ours use the older V100. A6000 has greater peak FP32 throughput and memory capacity, so the hardware difference does not give our implementation that advantage. Our LESS implementation takes 2,032-2,062 s on V100 versus approximately 5,400 s published on A6000; the paired V100 measurements above establish the gain without relying on a hardware comparison.
So, GREATS is included in the recoverability test in Figure 1; the matched timings in Table 2 cover Grad-Dot, Grad-Sim, and LESS.
Thanks to the authors for the detailed response. I will maintain my score, and the reasons are given in the comments below.
W1
I still think the paper is substantially similar to [1]. The paper should clarify the difference from [1] and what components are its actual contribution beyond it.
The paper claims its contribution as proposing BGU and BS-Ghost. However, the following components are the same as or substantially similar to [1].
- BGU and GNQ [1] share the overall structure. The difference is that BGU substitutes the gradient of GNQ with the change in the concept measurement reflecting the optimizer effect.
- The derivation of BGU in Equations 3 to 5 is very similar to the derivation of GNQ. The two derivations consist of the normal distribution entropy and the matrix determinant lemma. This is the reason for sharing the overall structure mentioned above.
- Using ghost kernels to calculate BGU in batch space without storing the per-example gradients is almost the same as [1]'s BS-Ghost GNQ, and the name of the method is the same.
I acknowledge that the paper contributes beyond [1]; however, for the reasons above, I think BGU and BS-Ghost are highly grounded in [1].
Nevertheless, the current manuscript almost does not provide a comparison with [1]. In the authors' response, they say they already explicitly credit the quadratic form and ghost kernel in Appendix B.2. However, their actual citation is only one sentence: "We extend the ghost kernels of Abdelghafar et al. (2026) to compute optimizer-aware behavioral responses without materializing full per-example gradients." It explains the relation with [1] too briefly and does not compare the similarity of the quadratic form. Furthermore, in the main text, the paper does not cite [1] even once. Regarding the relation and similarity with [1], the paper should importantly address the comparison and the remaining contribution beyond [1] in the main text.
I agree that the shared foundation does not mean an identical contribution, and I don't argue that the shared foundation is a problem of the paper. However, insufficient comparison with the related work that provides the foundation of the paper is a significant problem, leaving room for the contribution of the paper to be overestimated.
W2
First, I would like to clarify that W2 is a comment on Figure 2 of Section 4.1. I'm sorry that I didn't explicitly mention it in the review, causing a misreading for the authors. In my view, Figure 1 is the minimal controlled experiment that shows how the proposed method works, since it uses the small-scale model GPT-2 and synthetic data. This type of experiment gives its own meaning; however, an experiment in a realistic setting is needed to show whether the method can be used in those settings. In the paper, Section 4's experiments show the results in realistic settings, and especially Figure 2 is the most important experiment that shows whether the proposed method could change the model behavior beyond the baselines. However, BGU does not perform better than the baselines on two of the three datasets the paper tested. Further, the random removal baseline and GREATS are essential baselines for this experiment, so they should be compared.
W3
I am convinced by the authors' response that a time comparison to Concept Influence is challenging. However, if so, the paper should not argue in the introduction that the method is 44 times faster than EK-FAC and Concept Influence, because it is not a fair comparison regardless of whether it is favorable or unfavorable to the paper, since the GPU highly affects the time comparison. Additionally, the authors seem to have run EK-FAC on their own, so measuring the time on the V100 and reporting it should make the comparison fair.
[1] Abdelghafar, Sleem, Maryam Aliakbarpour, and Chris Jermaine. "Auditing Information Disclosure During LLM-Scale Gradient Descent Using Gradient Uniqueness" arXiv preprint arXiv:2510.10902, 2026.
This reply is another proof that your reading for the work is completely adversarial, and we still flag your review to the area chair. You are just reiterating stuff we addressed so we will focus on showing how your reading for the work is not aiming to improve the work rather you are just aiming to create false reasons to reject the work.
Regarding W1. You are just reiterating things we already fully addressed in Part 1/2. So, If stating these differences in the work explicitly is what's remaining, we are happy to add this to the paper.
Regarding W2.
- First in Figure 2, we already achieve better results than the "baselines" unlike your claim which is again factually incorrect. In GSM8K Mistakes dataset, BGU is better than all baselines. In Misaligned Opinions dataset, BGU is better than Vector Filter and Influence Function. In all datasets, BGU is better than Influence Functions, comparable to all the other methods.
- Second, there is NO baselines to our work None of the methods control model behavior during training and other contributions we explained in detail in our Part 1/2 answer to you. And, yet again, you are claiming false claims saying that "Figure 2 is the most important experiment". That's false, we never claimed that nor it is true. The point of this experiment as we stated in the paper is to show that BGU IDENTIFIES DATA THAT CAUSALLY SHAPES FINAL BEHAVIOR which is the title of the section.
Regarding W3. Yet, another proof that your reading for the work is adversarial and cheery picking, when you referred to the importance of the GPU, while you will never acknowledge:
-
our results in Table 2 which we already put it for you in Part 2/2 which clearly demonstrate the key algorithmic innovations we came up with. since you love to reiterate we will reiterate this for you The published DATE-LM results use A6000, while ours use the older V100. A6000 has greater peak FP32 throughput and memory capacity, so the hardware difference does not give our implementation that advantage. Our LESS implementation takes 2,032-2,062 s on V100 versus approximately 5,400 s published on A6000; the paired V100 measurements above establish the gain without relying on a hardware comparison.
-
Even in the experiment you are referring to, the time we are reporting is completely sufficient even without comparing with other methods. 8% overhead is objectively a low overhead!
Lastly, you want to convince us that this one sentence that we used to refer to the cited work [1] which you built your rejection upon, was magical that let you go do all these investigations of how this work is different than the other work. Really!
We do not care about your score at this moment, and that's why we flagged you to the area chair. Unfortunately, reviewers like you do all of this because they know their identity will never be in public.
Part 1/2
We thank the reviewer for their review. We also flag this review to the Area Chair as it contains factual errors. For example, the reviewer states that our experiments lack a comparison with GREATS, although GREATS is already evaluated in Figure 1, with implementation details in Appendix D.3. Another example, claiming that the paper is substantially similar to another paper is false and indicates either a very superficial reading for the work or an adversarial reading for the work. We show all of this below.
The distinction is substantive: our paper develops a behavioral predictor and uses it to control training. The cited work does not provide this predictor or control system. Our algorithm also introduces new computational techniques that make this control both accurate and efficient during training. The contributions enabling this are:
Our central contribution is control of model behavior during training through model-predictive control, instead of the expensive RL-based methods which require many rollouts. For this, we developed our novel feedback control system.
A key technical contribution enabling this control is an attribution-based predictive model. We are the first to operationalize data attribution techniques in this way. The controller uses that model to choose example weights as training evolves, evaluating candidate changes without generating candidate rollouts and estimating their rewards.
Control requires predicting the direction and size of a behavioral change. Equation (2) defines the response through the actual optimizer which gradient uniqueness does not do. Equations (3)-(5) then account for overlapping behavioral contributions. The quadratic form and ghost kernels in gradient uniqueness provide foundations, which we explicitly credit in Appendix B.2; they do not supply this behavioral predictor or the control system.
Our algorithm has multiple new innovations to tackle specific cost and accuracy problems:
Tracing and explaining how model behaviors develops during training (Figure 3) is a key contribution that enables BGU to be used as an efficient in-run monitoring and interpretability tool.
Some shared foundations do not make the contributions identical. Influence-function methods build on one another; Adam-aware Shapley extends in-run Shapley; GREATS and in-run Shapley share a lot of technical similarities and originates from same line of research. We likewise cite gradient uniqueness and identify what we reuse, which is little as demonstrated above.