acceptodds
Under review as a conference paper at ICLR 2027

Representer Point Selection via Tracking Regularized Gradient Trajectory

Abstract

Understanding the individual contribution of training samples to model predictions is essential for enhancing model transparency and facilitating data-driven debugging. While recent advancements, such as Representer Point Selection and Neural Tangent Kernels, offer efficient frameworks for sample-based attribution, they face significant practical hurdles. Specifically, existing methods often necessitate performance-altering regularization that deviates from the original model, rely on computationally prohibitive Hessian-Vector Products, or fail to faithfully reconstruct the original parameters. Consequently, they explain a surrogate approximation rather than the actual model's predictions. To address these challenges, we propose an alternative derivation for Representer Point Selection via Tracking Regularized Gradient Trajectory (RPS-TrReg), which identifies sample importance by tracing a regularized optimization path toward the original model's parameter state. Unlike previous methods that optimize a surrogate model in isolation, RPS-TrReg incorporates a regularization term to ensure the tracking phase remains strictly anchored to the target model's weights. This enables a high-fidelity decomposition of the target model into a linear combination of training samples without the need for expensive second-order derivatives. Empirically, our results demonstrate that RPS-TrReg consistently outperforms state-of-the-art explanation methods in data debugging tasks across vision and language classification benchmarks.

Primary area:
interpretability and explainable AI
Keywords:
Interpretable AI, Representer Point Selection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_S9aYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RPS-TrReg, a representer point method that explains the predictions of a trained classifier in terms of its training samples. The authors argue that RPS-l2 explains an l2-regularized last layer rather than the trained model, and that RPS-LJE and influence functions need the inverse Hessian. RPS-TrReg freezes the feature extractor and re-trains the last layer on the training loss plus a penalty that keeps it close to the original last layer. In the derivation this re-training starts from zero. By unrolling the gradient descent steps and using the stationarity condition at the last step, the authors then write the original last layer as a linear combination of the training features. Each training point gets a global importance, computed from its loss gradients along the fine-tuning path. Its local importance is the inner product between its features and those of the test point. The method is evaluated on a binary horse vs. car subset of CIFAR-10 with a ResNet-20 and on IMDB sentiment classification with a Bi-LSTM, against influence functions, RPS-l2, RPS-LJE, TracIn and NTK-Trac. In the main experiment, 30% of the training labels are flipped, training points are inspected in order of their global importance and the model is retrained after each 5% is corrected. RPS-TrReg finds more flipped labels than the baselines on CIFAR-10 and slightly more on IMDB, and gives the highest accuracy after correction at most inspection levels. The authors also report the L1 distance between the reconstructed and the original last-layer weights, where RPS-TrReg is lowest. In runtime, RPS-TrReg is slightly slower than RPS-l2 and NTK-Trac but faster than influence functions and RPS-LJE.

Critical strengths

The main strength is that the paper addresses a real problem, since RPS-l2 explains an l2-regularized last layer rather than the model that was actually trained. Anchoring the penalty to the original weights instead of to zero is a simple fix, and the derivation is short and easy to follow. The algebra in Eqs. 9-14 is correct when the last layer starts at zero. The anchoring idea could also be useful for other representer methods. Also the cost is low. The method only needs per-sample gradients of the last layer along one fine-tuning run and no Hessian. In Table 1 it is between 3% and 21% slower than RPS-l2 and NTK-Trac and between about 2.4 and 8 times faster than influence functions and RPS-LJE, so it would be cheap to use in practice.

Critical weaknesses

The first weakness is that the experiments do not test whether RPS-TrReg explains the original model better. The only direct measure of fidelity is Figure 2, the L1 distance between the reconstructed and the original last-layer weights. From Eqs. 7, 10 and 13, with a zero start and full-batch steps, this difference is exactly the gradient of the fine-tuning objective at the last step divided by lambda k^T. For RPS-TrReg the figure therefore measures how well the fine-tuning converged, and for RPS-l2 it mostly measures how far its weights drifted from the original ones. Further, with many more training points than features, infinitely many weightings reconstruct the original weights exactly, so a good reconstruction does not show that the attributions are correct. The data debugging experiment ranks training points by the global importance alone. This score does not depend on the test point and is built only from each point's own loss gradients, so it may largely track the training loss. Koh & Liang (2017) use the training loss as a baseline for this task, but the paper does not include it. Since fidelity is the central claim of the paper, a counterfactual evaluation, such as retraining without the top-ranked points or the linear datamodeling score of TRAK, is needed. The paper also does not report the learning rate, number of steps or batch size, and gives no error bars in Figure 1, where the IMDB differences are small.

Also I have a concern about the limited novelty and the small setting. The discounted gradient sum in Eq. 10 has the same form as the fast approximation in HyDRA, which keeps the L2 regularization and drops the Hessian. It is also closely related to the tracking representers of Tsai et al. (2023a), which they evaluate with a last-layer kernel. The new parts are the anchor to the original weights and the use of the stationarity condition to solve for them. However, the paper does not show which of these matters, for example by dropping the anchor or the terminal term separately. MAGIC is described as global and orthogonal to this work, but it attributes individual predictions and I think it is the closest strong baseline. The paper is motivated by the cost of Hessian-based methods on large networks, but all runtimes in Table 1 are below 1.6 seconds. The speed-up over influence functions and RPS-LJE is also smaller on IMDB (about 4 and 2.4 times) than on CIFAR-10 (about 6 and 8 times), which does not support the claim that the advantage grows with dataset size.

Questions for authors
  1. How was the last layer initialized in the experiments, and was the fine-tuning full-batch as in Eq. 8, even though the text says SGD?
  2. Are training points in Figure 1 ranked by the signed or the absolute global importance, and how does RPS-TrReg compare with ranking by training loss?
  3. Is the "total influence" used for influence functions and TracIn in Figure 1 the self-influence that Koh & Liang (2017) and Pruthi et al. (2020) use for this task? TracIn ranks below random selection on CIFAR-10, which would be unexpected with self-influence.
6 comments

Weakness 1.1 & Question 2: Lack of Training Loss Experiments. We thank you for this crucial observation. In Fig. 1, we rank training samples by global importance. Following the reviewer’s recommendation, we conducted leave-one-out (LOO) experiments. For each of ten randomly selected test samples, we ranked the training samples by signed importance. For RPS methods, signed importance is defined as global importance×local importance\text{global importance} \times \text{local importance}. We removed each training sample in turn, retrained the model, and measured the change in the test sample’s loss.

For each test sample, we calculated the Pearson correlation rr between the method’s signed importance scores for the training samples and the observed test-loss changes after removing each training sample and retraining. We report the mean and standard deviation of these correlations across the ten test samples.

As shown in the tables below, RPS-TrReg achieves the highest mean Pearson correlation on both datasets and outperforms Influence Functions without requiring Hessian computation. MAGIC and HyDRA also perform strongly. Their trajectory-based attributions capture how training examples affect parameter updates across optimization steps, while Hessian-related information captures local curvature. Together, these may help explain their stronger correlations with the observed loss changes.

Bi-LSTM on IMDB Results:

MethodPearson’s r
RPS-l2l_285.34 ± 1.35
RPS-LJE82.30 ± 4.43
Influence Function87.79 ± 2.17
MAGIC90.58 ± 0.16
HyDRA89.46 ± 2.67
RPS-TrReg (Ours)91.62 ± 1.62

ResNet-20 on CIFAR-10 Results:

MethodPearson’s r
RPS-l2l_282.45 ± 1.18
RPS-LJE78.47 ± 5.45
Influence Function89.45 ± 1.44
MAGIC96.30 ± 0.95
HyDRA90.98 ± 0.99
RPS-TrReg (Ours)96.46 ± 1.76

Weakness 1.2 & Question 1: Optimization details & full-batch vs. SGD. Thank you for raising this question and helping us improve the clarity of our experimental setup. In all experiments, the tracking procedure for the last linear layer was conducted using full-batch gradient descent, exactly matching Eq. (8). The mention of "SGD" was intended in the broader sense of first-order gradient methods, though we acknowledge it is imprecise and will update Section 3.2 to specify full-batch gradient descent.

Additionally, RPS-TrReg and RPS-l2l_2 use backtracking line search, which applies a dynamic step size for each update with λ=0.01\lambda = 0.01. TracIn and NTK-Trac use a learning rate of 5×10−35\times 10^{-3}, and RPS-LJE uses 1×10−51\times 10^{-5} following the original paper. All tracking procedures are conducted for 30 steps. We omitted error bars from the figure to avoid visual clutter given the density of the plotted curves. However, as shown in the table in our response to Weakness 2.1, our results are stable with low variance. We will include a detailed table with full standard deviations in the Appendix of the revised version.

Weakness 2.1 & Weakness 2.3: Difference to HyDRA and Consider MAGIC as a Baseline. Unlike RPS-TrReg, the L2L_2 regularization in HyDRA is not designed to anchor reconstructed weights to the original target parameters Θ†\Theta^\dagger, nor does HyDRA perform explicit weight reconstruction. Instead, HyDRA includes L2L_2 regularization in its training objective, this term contributes to the gradient trajectory and appears in the assumptions used to bound the approximation error when replacing the inverse-Hessian term with a Hessian-free hypergradient approximation. In contrast, RPS-TrReg uses trajectory regularization specifically to keep the optimization path anchored to the target model's parameters without needing to approximate the Hessian at all.

Following your suggestion, we added both HyDRA and MAGIC as baselines to our original data debugging benchmarks (Bi-LSTM on IMDB and ResNet-20 on CIFAR-10) and conducted additional experiments using GPT-2 on SST-2 and ViT-B/16 on MNIST (binary classification of even vs. odd digits). For a fair comparison, MAGIC and HyDRA use the same backtracking line search protocol as RPS-TrReg. As shown in the tables below, which report the percentage (%) of mislabeled data identified at 10%, 20%, and 30% data inspection, while MAGIC achieves the highest score on CIFAR-10, RPS-TrReg consistently outperforms both baselines across the remaining three benchmarks. We will include these comparisons and the discussion of HyDRA in the revised manuscript.

Bi-LSTM on IMDB Results:

Method10% Checked20% Checked30% Checked
HyDRA13.29 ± 1.0125.56 ± 1.4437.05 ± 1.69
MAGIC13.91 ± 0.8026.19 ± 1.0937.46 ± 1.40
RPS-TrReg14.88 ± 0.7227.46 ± 0.8838.53 ± 0.95

ResNet-20 on CIFAR-10 Results:

Method10% Checked20% Checked30% Checked
HyDRA19.33 ± 2.3032.65 ± 2.7543.09 ± 3.74
MAGIC25.00 ± 2.3040.08 ± 4.7052.50 ± 6.70
RPS-TrReg21.89 ± 3.0635.80 ± 4.0547.61 ± 4.09

ViT-B/16 on MNIST Results:

Method10% Checked20% Checked30% Checked
HyDRA32.94 ± 0.2056.06 ± 0.9865.06 ± 1.19
MAGIC33.13 ± 0.0665.30 ± 0.3588.21 ± 1.97
RPS-TrReg33.20 ± 0.0366.12 ± 0.1094.32 ± 0.39

GPT-2 on SST-2 Results:

Method10% Checked20% Checked30% Checked
HyDRA17.18 ± 1.5129.03 ± 2.2337.32 ± 2.84
MAGIC23.26 ± 1.0441.02 ± 1.4655.40 ± 1.61
RPS-TrReg30.19 ± 0.1054.90 ± 0.1872.55 ± 0.32

Weakness 2.2: Experiment of Dropping the Anchor. Thank you for this insightful question. In our experiments, our implementation of the NTK-Trac baseline is mathematically equivalent to RPS-TrReg without the proximity anchor term (12λ∥Θ−Θ†∥2\frac{1}{2}\lambda\|\Theta - \Theta^\dagger\|_2). As shown in Fig. 1 and Fig. 2, removing this anchor term leads to a noticeable drop in data debugging performance on both CIFAR-10 and IMDB, as well as a substantially larger L1L_1 weight alignment gap, because the trajectory is no longer bound to Θ†\Theta^\dagger and fails to faithfully approximate the original model weights. The following tables present the data debugging comparison between our method with and without the anchor term included:

Bi-LSTM on IMDB Results:

Method10% Checked20% Checked30% Checked
NTK-Trac (dropping anchor)14.19 ± 0.9625.76 ± 0.8436.21 ± 0.82
RPS-TrReg14.88 ± 0.7227.46 ± 0.8838.53 ± 0.95

ResNet-20 on CIFAR-10 Results:

Method10% Checked20% Checked30% Checked
NTK-Trac (dropping anchor)18.62 ± 2.2233.12 ± 3.3945.77 ± 4.49
RPS-TrReg21.89 ± 3.0635.80 ± 4.0547.61 ± 4.09

Weakness 2.4: Comprasion to the cost of Hessian-based methods. We thank the reviewer for pointing this out. We agree that the runtimes in Tab.1 do not show that our speed advantage grows with dataset size. These experiments use binary classification with a frozen feature extractor, so the influence-based baselines compute only the final-layer Hessian: H=1n∑i=1npi(1−pi)ϕiϕi⊤+εIH=\frac{1}{n}\sum_{i=1}^{n}p_i(1-p_i)\phi_i\phi_i^\top+\varepsilon I. Here, n is the number of training examples, ϕi∈Rd\phi_i\in\mathbb{R}^d is example i's feature vector, pip_i is its predicted positive-class probability, I is the d×dd\times d identity matrix, and ε=10−8\varepsilon=10^{-8} is numerical damping, d is the feature dimension. This is the exact Hessian for the binary logistic loss with respect to the frozen linear head. In our experiments it is 768×768768\times768.

For a CC-class head, a dense Hessian over all head parameters expands to Cd×CdCd \times Cd, making exact inversion substantially more expensive. Relative to the binary-head case, its storage scales by C2C^2 and dense inversion by C3C^3 (about 100×100\times and 1,000×1{,}000\times, respectively, for C=10C = 10). Since we have not measured those multi-class settings, we will qualify our scalability claim in Section 4.4 to state that Tab. 1 demonstrates efficiency for frozen-feature binary classification, and we will remove the claim that the speed advantage scales with dataset size.

Question 3: Total influence definition and TracIn performance below random. Thank you for pointing this out. In Fig.1, the Influence Function baseline ranks examples by the magnitude of their self-influence, si=1ngi⊤H−1gis_i=\frac{1}{n}g_i^\top H^{-1}g_i, where gig_i is the per-example gradient and H is the Hessian. The TracIn score is the self-influence specialization ∑tηt∣∇θℓi(θt)∣2\sum_t \eta_t|\nabla_\theta \ell_i(\theta_t)|^2, accumulated over 30 updates.

The possible explaination of TracIn ranks below random is that early erratic gradients caused by fitting contradictory flipped labels disproportionately inflate scores for clean but boundary examples, while late-stage memorization drives down the gradients of already-fitted mislabeled samples. Unlike Influence Functions (which normalize curvature via H−1H^{-1}) or RPS-TrReg (which anchors the trajectory to the final model state Θ†\Theta^\dagger), unregularized squared gradient norms become dominated by optimization noise.

Reviewer_oUYYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes training-example attribution by freezing a trained network's features and tracking an auxiliary regularized optimization trajectory of its final layer. It uses the resulting coefficients to reconstruct model predictions and identify influential examples.

Critical strengths

Avoiding Hessian computations is attractive, and the reported label-debugging results on binary provide some evidence of practical usefulness.

Critical weaknesses

The reconstruction derivation assumes zero initialization, whereas the described procedure starts from the trained weights; the paper does not explain these choices or control the potentially amplified final residual with zero initialization. The experiments show debugging gains but do not establish a major advantage over existing attribution methods.

Questions for authors

Section 3.1 states that parameters are initialized with the trained weights, whereas Section 3.2 assumes zero initialization. Please explain this difference and clarify which initialization is used in the experiments. Please also report the step size, number of iterations, and final gradient norm, and explain the amplified residual in Equation (13).

2 comments

Weakness 1.1 & Question 1.1: Contradiction in Initialization (Θ0\Theta_0) between Section 3.1 and Section 3.2. We apologize for the confusion caused by our description in Section 3.1. We intended to state that the feature-extractor backbone is initialized with the trained model’s weights and held frozen, while the last linear layer is zero-initialized and optimized.

In all derivations (Eqs. 9–10) and empirical experiments, the tracking optimization of the last layer is initialized at Θ0=0\Theta_0=0. We wil revise Section 3.1 to make this distinction clear.

Weakness 1.2 & Question 1.2: Experimental Hyperparameters and Amplified Residual. Thank you for raising this question. RPS-TrReg and RPS-l2l_2 use backtracking line search, which applies a dynamic step size for each update with λ=0.01\lambda = 0.01. TracIn and NTK-Trac use a learning rate of 5×10−35\times 10^{-3}, and RPS-LJE uses 1×10−51\times 10^{-5} following the original paper. All tracking procedures are conducted for 30 steps with full-batch gradient descent. At the final RPS-TrReg iterate, the mean absolute gradient is 8×10−58\times10^{-5} on IMDB and 1.4×10−41.4\times10^{-4} on CIFAR-10.

Regarding the amplified residual, if you are referring to the kTk^T factor in Eq. (13), k=1−ηλk = 1 - \eta\lambda, where η\eta is the learning rate and λ\lambda is the regularization coefficient. With a fixed learning rate, kTk^T gives the contraction factor on the residual after TT steps. For η=λ=0.01\eta = \lambda = 0.01, k30≈0.9970k^{30} \approx 0.9970 and k10000≈0.3679k^{10000} \approx 0.3679. In our experiments, backtracking line search adjusts ηt\eta_t across steps, so the corresponding factor is ∏t=1T(1−ηtλ)\prod_{t=1}^{T}(1 - \eta_t\lambda). We use the kTk^T notation to present the global and local importance terms compactly. We will clarify this in the revised version.

Weakness 1.3: Advantage over Existing Attribution Methods. Thank you for raising this point. While the debugging margin on IMDB is indeed modest, data debugging only shows one aspect of our method's contribution. Unlike competing baselines, RPS-TrReg combines strong debugging accuracy with faithful reconstruction of the original target weights Θ†\Theta^\dagger and full avoidance of expensive Hessian computations.

To further evaluate generalization across architectures and modalities, we conducted additional data debugging experiments on SST-2 with GPT-2 and MNIST with ViT-B/16. As shown in the tables below, RPS-TrReg achieves the highest mislabeled data detection rate across all four benchmark settings, whereas baselines such as Influence Functions and RPS-l2l_2 perform unevenly across datasets. Across all four benchmarks, our method achieves an average rank of 1.00, compared to 2.50 for RPS-l2l_2 and 3.25 for Influence Functions. We will add these results to the revised manuscript.

GPT-2 on SST-2 Results:

Method10% Checked20% Checked30% Checked
RPS-l2l_229.63 ± 0.1154.06 ± 0.1472.53 ± 0.21
RPS-LJE15.90 ± 0.8030.16 ± 1.8843.47 ± 2.98
Influence Function20.87 ± 0.6739.53 ± 1.4555.75 ± 2.25
RPS-TrReg (Ours)30.19 ± 0.1054.90 ± 0.1872.55 ± 0.32

ViT-B/16 on MNIST Results:

Method10% Checked20% Checked30% Checked
RPS-l2l_233.03 ± 0.0165.47 ± 0.0494.16 ± 0.20
RPS-LJE23.04 ± 0.1843.72 ± 0.2762.07 ± 0.32
Influence Function29.45 ± 0.0957.02 ± 0.2380.99 ± 0.24
RPS-TrReg (Ours)33.20 ± 0.0366.12 ± 0.1094.32 ± 0.39

Dear Reviewer oUYY,

Thank you again for your constructive feedback on our submission. In our rebuttal above, we clarified the zero-initialization of the linear head (Θ0=0\Theta_0 = 0) versus the frozen pretrained backbone, detailed the backtracking line search hyperparameters and residual contraction factor ∏t=1T(1−ηtλ)\prod_{t=1}^{T}(1-\eta_t\lambda), and provided new results on GPT-2 (SST-2) and ViT-B/16 (MNIST).

Could you please let us know if our responses and additional experiments have adequately resolved your concerns? If any points remain unclear or if you have further questions, we would be very glad to address them before the discussion period concludes.

Reviewer_mSSPOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a new method, RPS-TrReg, for representer point selection based on a new way to regularize the the connected optimization problem. The results indicate that the approach is quite efficient in smaller settings (an LSTM trained on an IMDB corpus and a ResNet-20 for CIFAR-10 classification). Further, it outperforms the chosen baselines in two data debugging benchmarks based on these settings. They also provide qualitative results that further illustrate the usefulness of the approach.

Critical strengths

(S1) Interesting problem and innovative approach to it

(S2) In the benchmarks run, the method outperforms the chosen baselines and further, it is more efficient than comparable approaches while staying close to the original model

Critical weaknesses

(W1) The benchmark experiments only include two small-scale settings. This leaves the following points/questions open

(W1.1) Does the approach scale to realistic settings? You claim "modern" architectures (line 135), which seems not to be the case. Besides, given that one of the claims is efficiency, it would be nice to see whether RPS-TrReg could be used to study larger models, e.g. a small LLM like GPT-2 or a vision transformer.

(W1.2) Does RPS-TrReg generally outperform across comparable smaller settings (e.g. MNIST, ImageNet, GLUE)?

(W2) No code submission. The work of this community is enabled by an open-source culture which allows us to quickly build on and extend previous findings. This includes this very paper you're submitting for review. Besides, some of your claims would be even stronger if the reviewers had access to your code to check how your have implemented the experiments you describe in the paper. To enable future endeavors using your method, transparency and reproducibility, please promise to provide your code with the final revision.

Minor weaknesses:

(MW3) The reconstruction of the representer weights and model weights (Figure 2) is measured in terms of L1L_1 distance. Your regularization minimizes the L2L_2 distance of the same weights, which makes this evaluation seem more like a sanity check to me. Would it make sense to also measure cosine or infinity norm? The graph has enough space and it would, imo, strengthen your claims.

(MW4) Some kind of downstream evaluation beyond showing a small set of examples would help position this method. For example, does it help human users to identify outliers better or predict influential training examples? Of course, this is a big ask and not something a not feasible within a single review cycle. I would like to encourage the authors to think of such applications and explore this, maybe in future work. The community would greatly benefit if it had a better sense of what methods like yours can help with in practice.

Questions for authors

See above.

5 comments

Weakness 1.1 & 1.2: Modern Architectures and Additional Benchmarks. Thank you for your constructive feedback and suggestions. Following your suggestion, we expanded our data debugging experiments to include both a pre-trained Vision Transformer (ViT-B/16) on MNIST (formulated as binary classification of even vs. odd digits for computational efficiency compared to ImageNet) and a pre-trained language model (GPT-2) on SST-2 (from the GLUE benchmark).

As shown in the tables below, which report the percentage (%) of mislabeled data identified at 10%, 20%, and 30% data inspection, RPS-TrReg consistently outperforms all competing baselines across every inspection threshold on both datasets. We will add these experiments and results to the revised manuscript.

GPT-2 on SST-2 Results:

Method10% Checked20% Checked30% Checked
RPS-l2l_229.63 ± 0.1154.06 ± 0.1472.53 ± 0.21
RPS-LJE15.90 ± 0.8030.16 ± 1.8843.47 ± 2.98
Influence Function20.87 ± 0.6739.53 ± 1.4555.75 ± 2.25
Ours30.19 ± 0.1054.90 ± 0.1872.55 ± 0.32

ViT-B/16 on MNIST Results:

Method10% Checked20% Checked30% Checked
RPS-l2l_233.03 ± 0.0165.47 ± 0.0494.16 ± 0.20
RPS-LJE23.04 ± 0.1843.72 ± 0.2762.07 ± 0.32
Influence Function29.45 ± 0.0957.02 ± 0.2380.99 ± 0.24
Ours33.20 ± 0.0366.12 ± 0.1094.32 ± 0.39

Weakness 2: Code availability and reproducibility. Thank you for emphasizing the importance of reproducibility and open-source transparency. To facilitate verification of our implementations during the review process, we have prepared an anonymous repository containing the complete codebase and configuration scripts: https://anonymous.4open.science/r/RPS-TrReg-DE72/.

We will also release the public repository and include the link in the final camera-ready revision.

Minor Weakness 3: Evaluation metrics beyond L1L_1 distance in Fig. 2. Thank you for this constructive suggestion. Following your recommendation, we evaluated the alignment using both Cosine Similarity (Cosine) and L∞L_\infty norm (IN) across 10 independent runs.

As shown in the table below, RPS-TrReg consistently achieves the highest directional alignment (Cosine Similarity) and the lowest worst-case coordinate error (L∞L_\infty norm) across both CIFAR-10 and IMDB, confirming that our weight reconstruction advantage holds beyond the L1L_1 metric. We will include these additional metrics in Fig. 2 of the revised manuscript.

MethodCIFAR-10 Cosine (↑)CIFAR-10 IN (↓)IMDB Cosine (↑)IMDB IN (↓)
RPS-l2l_295.7 ± 0.30.309 ± 0.0274.4 ± 3.50.296 ± 0.06
RPS-LJE52.6 ± 1.781.75 ± 0.540.5 ± 1.229.15 ± 6.35
NTK-Trac95.6 ± 0.10.554 ± 0.0773.5 ± 5.00.329 ± 0.05
RPS-TrReg (Ours)96.8 ± 0.10.210 ± 0.0176.2 ± 2.90.281 ± 0.06

Minor Weakness 4: Downstream Applications. Thank you for this valuable perspective and encouragement. In future work, we plan to explore two practical downstream applications:

  • Validation-Guided Data Pruning: By ranking training instances according to their influence on a given clean validation set, we can select the most informative subset of training data. Training on this high-value subset can substantially reduce training time and computational cost compared to training on the entire dataset.
  • Out-of-Distribution (OOD) Detection: Identifying hard OOD samples or anomalous inputs that diverge from in-distribution (ID) training representations can be naturally formulated as a sample-level data debugging task. Leveraging RPS-TrReg's prediction decomposition can help flag atypical or out-of-distribution test cases.

We will add a brief discussion of these directions in the revised manuscript.

Thank you for your reply. Your new experiments and explanations have convinced me of the soundness of your paper. I assume you will in detail explain experiment settings etc. in the next revision of the paper. Please elaborate on those settings below, perhaps by drafting your new Experiment Settings section.

I have studied the other reviewer's comments and your rebuttals and did not find any new arguments or critiques that would affect my assessment.

In sum, I have decided to increase my store as I think this is interesting work and will likely be impactful to our community. My scoring might still be affected if I find your description of your experiment settings insufficient.

Thank you very much for your positive feedback, for increasing your score, and for recognizing the potential impact of our work. We will certainly include these detailed experimental settings in the revised version of the paper. Below is the drafted Experimental Setups section for your review:

Experimental Setups

Baselines. We compare RPS-TrReg with data-attribution baselines in two categories: (1) Hessian-based methods: Influence Functions (IF) and RPS-LJE, which use inverse-Hessian information to account for local loss curvature; and (2) gradient- and training-trajectory-based methods: RPS-ℓ2\ell_2, TracIn, NTK-Trac, MAGIC, and HyDRA. RPS-ℓ2\ell_2 uses a representer-point decomposition, while TracIn and NTK-Trac aggregate gradient-based scores across training checkpoints. Under our last-layer formulation, NTK-Trac corresponds to RPS-TrReg without the proximity-anchor term. MAGIC and HyDRA estimate data effects through the training process: MAGIC differentiates through the optimization trajectory, whereas HyDRA uses a Hessian-free approximation with a bounded error to reduce computation. We also include a random-ranking baseline as a non-informative reference.

Datasets and Models. We evaluate our methods and baselines on four binary classification benchmarks. Following the protocol of RPS-LJE, we use ResNet-20 on the "horses vs. cars" subset of CIFAR-10 and a Bi-LSTM on IMDB. To evaluate the methods with Transformer architectures, we also use GPT-2 on SST-2 and ViT-B/16 on MNIST, mapping the MNIST labels to even versus odd digits. For GPT-2, we represent each sentence using the hidden state of its final non-padding token and truncate sequences to 128 tokens. For ViT-B/16, we replicate each grayscale MNIST image across three channels, resize it to 224×224224\times224, and normalize it using ImageNet statistics. In both Transformer experiments, we freeze the pretrained backbone and analyze only the binary classification head.

Optimization and Hyperparameters. For Bi-LSTM on IMDB and ResNet-20 on CIFAR-10, methods that require optimization (RPS-ℓ2\ell_2, TracIn, NTK-Trac, MAGIC, HyDRA, and RPS-TrReg) use full-batch updates for 30 steps. RPS-ℓ2\ell_2, MAGIC, HyDRA, and RPS-TrReg use backtracking line search at each update. TracIn and NTK-Trac use a fixed learning rate of 5×10−35\times10^{-3}. RPS-LJE uses a learning rate of 1×10−51\times10^{-5}, following its original paper. For methods with a regularization term, RPS-ℓ2\ell_2 and RPS-TrReg use λ=1×10−2\lambda=1\times10^{-2}, while HyDRA uses λ=2×10−5\lambda=2\times10^{-5}, following its original paper. For GPT-2 on SST-2 and ViT-B/16 on MNIST, methods that require optimization use full-batch updates for 200 steps. RPS-ℓ2\ell_2 and RPS-TrReg use λ=5×10−2\lambda=5\times10^{-2}, while the other method-specific hyperparameters remain unchanged. All experiments are conducted on a single NVIDIA GeForce RTX 4090 GPU with 24 GB of memory.

Data Debugging. In each run, we intentionally corrupt the training datasets by randomly flipping 30% of the training labels, which significantly degrades model performance. The primary objective is to identify and rectify these harmful samples as efficiently as possible. In our experimental protocol, we inspect training samples in increments: at each iteration i=0,1,…,8i = 0, 1, \dots, 8, we select a batch of ki=0.05i×nk_i = 0.05i\times n samples (where nn represents the total number of training points) that exhibit the highest global importance value α\alpha. For Influence Functions, TracIn, MAGIC, and HyDRA, which do not permit the decomposition of influence into separate local and global components, we utilize their total influence values for this selection process. Following each 5% increment of data correction, the model is retrained, and the resulting test accuracy is recorded. We report the mean accuracy improvement across 10 random trials.

Weight Alignment. To evaluate the fidelity of the candidate methods, we measure the alignment between the reconstructed parameters ∑ᵢ αᵢ ϕ(xᵢ)ᵀ and the target model state Θ†. A high-fidelity attribution method should yield reconstructed weights that closely approximate the actual parameters of the model being explained. We quantify this alignment by calculating the average L₁ distance, L∞ distance, and cosine similarity across 10 random runs. Note that Influence Functions, TracIn, MAGIC, and HyDRA are excluded from this comparison, as they are not designed to explicitly reconstruct target model parameters.

Please let us know if you have any further questions or if additional details would be helpful, we are always happy to answer. Thanks again!

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for the constructive feedback. During the rebuttal phase, we strengthened our manuscript with the following updates:

  • New Architectures & Baselines: We expanded our benchmarks to GPT-2 (SST-2) and ViT-B/16 (MNIST), and added comparisons against MAGIC and HyDRA across all four datasets.
  • Counterfactual & Alignment Evaluations: We conducted single-example Leave-One-Out (LOO) retraining experiments on CIFAR-10 and IMDB (achieving the highest Pearson correlation with actual test-loss changes), and added Cosine Similarity and L∞L_\infty norm metrics for weight reconstruction.
  • Clarifications & Code: We clarified the zero-initialization (Θ0=0\Theta_0 = 0) of the linear head, provided full optimization hyperparameters and experimental setup details (including the LOO protocol), and released an anonymous code repository.

Following our responses, Reviewer mSSP (Reviewer 1) confirmed that all concerns were resolved and raised the score to Clear Accept (4). Reviewer S9aY (Reviewer 3) stated in the comment that our LOO tests and new experiments addressed the main concerns and expressed willingness to raise the score to Weak Accept (though the rating field has not yet been updated). For Reviewer oUYY (Reviewer 2), we provided explanations and additional experiments regarding initialization, residual contraction, and broader architectural benchmarks, though we have not yet received a follow-up response.

We thank the Area Chair and Reviewers for the time and support, and we will incorporate all new results and clarifications into the final revision.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.