acceptodds
Under review as a conference paper at ICLR 2027

Woodbury Influence: Effective Evaluation of Unseen Sample Familiarity Before Training

Abstract

Influence Functions (IFs) provide a powerful framework for quantifying the impact of individual training samples on the trained model using gradient and Hessian information. Most IFs primarily focus on the effect of removing samples rather than adding novel ones, but this unseen-sample regime arises in many real-world settings that must determine how familiar a new sample is to the current optimal model. Extending IFs to this regime presents two fundamental challenges: (1) the prohibitive computational cost of estimating influence based on Hessian, and (2) the absence of validated labels in IF frameworks for unseen data. To overcome these challenges, we propose Woodbury Influence (WIF) as a measure of an unseen sample's familiarity to the entire model parameters before training on it, leveraging pseudo-labeling and the Woodbury matrix identity. Empirical analyses reveal the underlying mechanisms driving the effective separation of unfamiliar samples from familiar ones, accurate approximation of the true optimal model, and competitive computational cost. Furthermore, we propose practical methods for applying WIFs to representative tasks in this regime, including Out-of-Distribution (OOD) detection, Open-Set Recognition (OSR), and Continual Learning (CL) : offline class-order selection. WIF achieves superior performance across all three benchmarks, establishing a robust, influence-based framework for these settings.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Influence Function, Data-centric, Woodbury Matrix identity, Out-of-Distribution (OOD) detection, Open-Set Recognition (OSR)

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Dye2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Woodbury Influence (WIF), a post-hoc score for evaluating how much an unlabeled query sample would perturb a trained model’s parameters if added to the training set. WIF combines hard pseudo-labeling with a damped, layer-wise empirical-Fisher approximation, using the Woodbury identity to compute the inverse-Hessian action in closed form. The authors validate the score against retraining on a small MNIST setup and against LiSSA on ResNet-18, then evaluate it on OOD detection, open-set recognition, and class-order selection for continual learning. The underlying question—whether influence can be useful for assessing unseen samples—is interesting. However, the paper’s empirical evidence and efficiency claims do not yet establish that WIF is a reliable or broadly practical OOD method. The reported gains are uneven, the most striking OSR result is not accompanied by a corresponding AUROC gain, and important details about the estimator’s computational cost and experimental protocols are missing.

Critical strengths
  1. The sample-addition perspective is practically relevant and less explored than training-sample influence.

  2. The Woodbury derivation is clear under the paper’s layer-wise empirical-Fisher approximation.

  3. The paper includes useful ablations of pseudo-labeling and curvature.

Critical weaknesses
  1. Direct validation against retraining is limited to a small MNIST model. The ResNet experiment compares against LiSSA, not retraining.

  2. The efficiency analysis does not clearly include the cost of constructing and inverting the Gram matrix or report end-to-end memory and latency.

  3. OOD results are mixed: WIF improves FPR95, but does not consistently lead in AUROC; some gains are small.

  4. OSR shows a large FPR95 gain but nearly identical AUROC to the best baseline. The explanation for this gap needs stronger evidence.

  5. The continual-learning evaluation is small and underspecified, making it difficult to assess whether the ordering gains generalize.

Questions for authors
  1. In the OOD experiments, is WIF computed using only the final-layer block as in Eq. 10? If so, which experiments use the full layer-wise estimator in Eq. 9?

  2. Does the reported timing include construction of GG, formation and inversion of the Gram matrix, and amortization across queries? What are the concrete values of nn, dd, and query batch size?

  3. How sensitive are OOD and OSR results to the choice of damping λ\lambda, and was it selected using any OOD validation data?

  4. Can the authors provide paired uncertainty estimates for the AUROC and FPR95 differences, particularly for OSR and the smaller OOD gains?

  5. For continual learning, what data are available when the class order is selected, and how many class splits or permutations are used?

4 comments

We thank the reviewer Dye2 for the careful reading. We look forward to a constructive discussion.

W1

We believe this point refers to Figure 1. We validate the approximation quality of WIF in the sample-incremental regime through two complementary experiments: one direct (retraining with small dataset and architecture) and one indirect (without retraining with larger dataset and architecture).

The left panel of Fig. 1 shows the direct validation. Following the protocol of [1], we validate against exact influence obtained via retraining, but only on a small model and dataset, since retraining once per query sample is prohibitively expensive for larger datasets and architectures. For example, if a single training run takes 10 minutes, evaluating 10,000 data points would require 100,000 minutes. This is impractical, so we adopt the same evaluation protocol as the original IF [1].

The right panel of Fig. 1 shows the indirect validation. The direct approach becomes infeasible for larger datasets and architectures because retraining is prohibitively expensive. To assess approximation quality at these scales, we instead measure the correlation between WIF and LiSSA. LiSSA is substantially cheaper than actual retraining and has been shown to approximate it closely [1].

[1] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 1885–1894, 2017.



W2

We have updated the complexity analysis in Table 3. Table 3 reports the complexities of Gram matrix construction and inverse matrix computation. Complexities retain only the dominant term, and lower-order terms are omitted, since all terms of end-to-end would be too long. For OOD detection, n is the number of training samples in the dataset (100,000). d is the dimension of the last layer (512 for ResNet18 and 1024 for ViT). Q is the number of incoming samples to be scored (Q=10,000). Batch size appears in this analysis because of the reference gradient. The closed form is constructed from all n gradients (full-batch).

Table 3. Time and Space Complexity of Influence Function methods. WIF always has complexity equal to or better than the classical methods Exact IF and LiSSA. Compared with DataInf, WIF is at a disadvantage in the Gram matrix and inverse matrix construction and in memory complexity. However, WIF gains in per-query computation, so its overall speed is faster. We denote the number of reference samples by nn and the parameter dimension by dd. QQ denotes the number of incoming query samples. Complexities retain only the dominant term. Lower-order terms are omitted.

MethodHessian inversionTime complexity (construction)Time complexity (per query)Memory complexityTotal time / sample (s)
Exact IFMatrix inversionO(nd2+d3)\mathcal{O}(nd^2 + d^3)O(Qnd)\mathcal{O}(Qnd)O(d2)\mathcal{O}(d^2)Prohibitively costly
LiSSA w/ pseudoIterativeO(nd2)\mathcal{O}(nd^2)O(Qnd)\mathcal{O}(Qnd)O(d2)\mathcal{O}(d^2)0.108
DataInf w/ pseudoClosed-form (additional approximation)O(nd)\mathcal{O}(nd)O(Qnd)\mathcal{O}(Qnd)O(d)\mathcal{O}(d)0.020
WIF (ours)Closed-form & switching formO(max⁡(n,d)×min⁡(n,d)2)\mathcal{O}(\max(n,d) \times \min(n,d)^2)O(Q×min⁡(n,d)2)\mathcal{O}(Q \times \min(n,d)^2)O(min⁡(n,d)2)\mathcal{O}(\min(n,d)^2)0.018

W3 & W4

Your understanding is correct: the AUROC is relatively similar across methods, but the gain in FPR95 increases dramatically. We explain this in three points.

1) OSR, a generally harder task

OSR can be regarded as a near-OOD task, which is the more challenging setting in OOD detection. On far-OOD, a comparatively easy setting, existing methods already perform well, achieving results on par with or better than WIF. On harder settings such as near-OOD and OSR, however, existing methods struggle considerably. We believe this is how the AUROC results should be interpreted.

2) Favorable distribution shape

Sec 5.5 analyzes why WIF achieves strong performance on FPR95. FPR95 sets the threshold at the 95% point of the ID data and measures how much the OOD distribution overlaps with the ID distribution beyond it. The favorable distribution shape for minimizing this overlap concentrates most of the data at the head and becomes sparser toward the tail (the 95% point), decaying gradually. Among the compared methods, WIF comes closest to this ideal shape.

3) Amplified effect of PL depending on model performance

Table. 2 shows that WIF exhibits a notably larger FPR95 gap relative to the baselines on TinyImageNet. In all other OOD detection and OSR benchmarks, we use models that achieve about 95% accuracy on the ID test data. On TinyImageNet, however, ResNet18 cannot reach this level of accuracy, so we use a model with accuracy in the 80% range. This model has a considerably high error rate on the ID test data, which makes it more susceptible to the effect of pseudo-labeling. We therefore conjecture that the large FPR95 gap on this dataset stems from the amplified influence of pseudo-labeling in Sec 5.4

We additionally conducted experiments on the classical benchmarks CIFAR-10/CIFAR-100 and SVHN. OSR results Table has been revised as follows. The results remain consistent with the original trends.

Table 2. Open-Set Recognition results on CIFAR10/CIFAR100, SVHN, and Tiny ImageNet with ResNet-18. Our method outperforms all the baselines considered. Higher AUROC and lower FPR95 indicate better performance. All results are reported as mean ± standard deviation across 5 runs with different known/unknown class splits. Best results in bold.

Method spaceMethodCIFAR10/100 AUROC ↑CIFAR10/100 FPR95 ↓SVHN AUROC ↑SVHN FPR95 ↓Tiny ImageNet AUROC ↑Tiny ImageNet FPR95 ↓
No GradientMSP0.770 ± 0.0120.702 ± 0.00710.917 ± 0.0080.517 ± 0.0210.776 ± 0.0110.826 ± 0.081
No GradientEnergy0.782 ± 0.0170.658 ± 0.0680.925 ± 0.0050.472 ± 0.0170.779 ± 0.0200.834 ± 0.071
No GradientODIN0.755 ± 0.0210.693 ± 0.0720.921 ± 0.0040.455 ± 0.0150.782 ± 0.0130.815 ± 0.070
No GradientASH0.789 ± 0.0130.650 ± 0.0690.927 ± 0.0070.410 ± 0.0210.778 ± 0.0100.853 ± 0.078
No GradientSCALE0.859 ± 0.0110.650 ± 0.0590.920 ± 0.0080.423 ± 0.0230.779 ± 0.0120.841 ± 0.061
GradientGradNorm0.763 ± 0.0180.679 ± 0.0610.923 ± 0.0070.426 ± 0.0190.773 ± 0.0160.853 ± 0.082
Gradient + HessianWIF (ours)0.875 ± 0.0210.619 ± 0.0530.935 ± 0.0040.399 ± 0.0160.784 ± 0.0120.570 ± 0.030

W5

We would first like to clarify the scope of our contributions. We propose WIF, which enables label-free influence estimation in the sample-incremental regime, and we justify it theoretically and empirically. We then instantiate it for OSR, OOD detection, and order sensitivity in continual learning (CL), aiming to verify its potential and extend its applicability rather than to chase state-of-the-art numbers. Finally, we identify two mechanisms behind its separation of familiar and unfamiliar samples.

We did not study continual learning (CL) in its entirety. Instead, we restricted our scope to presenting a concrete use case, order sensitivity in CL, together with a method for addressing it. Within this scope, we used EWC and GEM, which are widely used representatives of regularization-based and replay-based approaches, respectively. However, we recognize that these two methods alone may not be sufficient. We therefore tried to add one more representative method for each category, LwF (regularization-based) and ER (replay-based), and updated the table accordingly.

We agree that claiming general effectiveness across CL would be an overgeneralization. For this reason, we deliberately narrowed our experiments to this specific problem and report results only within that scope. Broader applicability and robust methods that can compete with state-of-the-art (SOTA) approaches were mentioned only as possible directions for future work in the Appendix limitation section.

Table 4. Class-incremental CL on Tiny ImageNet with ResNet-18. The WIF-based Ben-first ordering achieves the best performance. Average Accuracy (AA, ↑) is better when higher, and Forgetting Measure (FM, ↓) is better when lower. Results are averaged over 5 seeds.

OrderEWC AA ↑EWC FM ↓GEM AA ↑GEM FM ↓LwF AA ↑LwF FM ↓ER AA ↑ER FM ↓
Random0.40 ± 1.190.60 ± 1.310.54 ± 0.870.46 ± 0.510.37 ± 0.300.63 ± 0.330.66 ± 0.140.33 ± 0.41
Des-first w/ WIF0.37 ± 1.150.58 ± 1.370.62 ± 0.770.44 ± 0.450.33 ± 0.310.67 ± 0.440.65 ± 0.130.32 ± 0.47
Ben-first w/ Logit0.41 ± 1.580.56 ± 1.360.72 ± 0.680.32 ± 0.460.37 ± 0.270.63 ± 0.380.75 ± 0.190.28 ± 0.28
Ben-first w/ WIF (ours)0.42 ± 1.210.54 ± 1.660.74 ± 0.620.25 ± 0.470.40 ± 0.310.53 ± 0.380.78 ± 0.140.22 ± 0.40

Q1

This is already described in Section 4.2. Motivated by the experimental findings of GradNorm [3], we design WIF. Using only the last layer achieves better OOD (out-of-distribution) detection and OSR performance and even reduces computation time.

[3] Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.



Q2

Addressed in W2.



Q3

The effect of λ is large enough to alter the performance ranking. Each dataset requires its own optimal λ to achieve the best performance. We heuristically varied λ from 0.1 to 0.00001 until the best performance was reached. Since this procedure is highly heuristic, we discuss it in the Limitations section.



Q4

Addressed in W3 & W4.



Q5

This is presented in Sec 4.2 (formally) and Sec. 5.7 (detailed). We evaluate on Tiny ImageNet with ResNet-18. The model is trained on 5 base classes and then incrementally learns 5 more, one class per step.

Reviewer_H52BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Woodbury Influence to explore which data are familiar to the current model and which are not with Influence Functions (IF). Specifically, the authors propose to use Woodbury matrix identity to enable efficient computation and pseudo labels to enable the IF to work. Multiple tasks, including OOD detection, OSR, and continual learning, have been used to evaluate the proposed method.

Critical strengths
  1. The paper extends IF from sample removal to unseen sample addition. I think this is the first paper that focuses on this topic.

  2. The design of pseudo-labelling and the use of Woodbury matrix are reasonable and easy to understand.

Critical weaknesses
  1. Even though the method has been evaluated for the tasks, the compared methods are quite old methods and the proposed methods apparently cannot perform well in the general benchmarks of the three tasks if some recent methods are included. In addition, for CL benchmark, some commonly used methods are not compared, such as LwF.

  2. The paper proposes a research question: which data are familiar to the current model, but I do not find a clear answer to this question.

  3. I do not understand the reason for using IFs to answer this research question. Actually, many techniques can be used for all three tasks, such as Mahalanobis distance.

  4. Even though the Woodbury matrix identity can improve efficiency, the cost of the proposed method can still be very high if the scale of the data increases. I think it would be better if the authors could include a comparison with other methods on the computational cost to see the efficiency of the method.

Questions for authors

Why does larger influence mean unfamiliar? Is there any theoretical or empirical proof?

3 comments

We thank the reviewer H52B for the careful reading. We look forward to a constructive discussion.

W1

We would first like to clarify the scope of our contributions. We propose WIF, which enables label-free influence estimation in the sample-incremental regime, and we justify it theoretically and empirically. We then instantiate it for OSR, OOD detection, and order sensitivity in continual learning (CL), aiming to verify its potential and extend its applicability rather than to chase state-of-the-art numbers. Finally, we identify two mechanisms behind its separation of familiar and unfamiliar samples.

1) OOD detection

For OOD detection, we consider our evaluation sufficient and not old. The benchmark used in our OOD detection experiments remains in active use in OpenOOD. Our baselines include MSP and recent methods such as GradNorm (2021), ASH (2023), and SCALE (2024). Especially, MSP, SCALE, and GradNorm rank among the top methods on the OpenOOD leaderboard [1].

2) OSR

For OSR, we evaluate on the still challenging ImageNet-based benchmark, where even the current state-of-the-art methods achieve only about 0.7 to 0.8 AUROC and thus fall short of fully solving the problem [1][2]. We therefore consider experiments on this benchmark alone to be sufficient. However, we understand that evaluation on this single dataset may be considered insufficient. We therefore additionally conducted experiments on the classical benchmarks CIFAR-10/CIFAR-100 and SVHN. OSR results Table has been revised as follows. The results remain consistent with the original trends.

Table 2. Open-Set Recognition results on CIFAR10/CIFAR100, SVHN, and Tiny ImageNet with ResNet-18. Our method outperforms all the baselines considered. Higher AUROC and lower FPR95 indicate better performance. All results are reported as mean ± standard deviation across 5 runs with different known/unknown class splits. Best results in bold.

Method spaceMethodCIFAR10/100 AUROC ↑CIFAR10/100 FPR95 ↓SVHN AUROC ↑SVHN FPR95 ↓Tiny ImageNet AUROC ↑Tiny ImageNet FPR95 ↓
No GradientMSP0.770 ± 0.0120.702 ± 0.00710.917 ± 0.0080.517 ± 0.0210.776 ± 0.0110.826 ± 0.081
No GradientEnergy0.782 ± 0.0170.658 ± 0.0680.925 ± 0.0050.472 ± 0.0170.779 ± 0.0200.834 ± 0.071
No GradientODIN0.755 ± 0.0210.693 ± 0.0720.921 ± 0.0040.455 ± 0.0150.782 ± 0.0130.815 ± 0.070
No GradientASH0.789 ± 0.0130.650 ± 0.0690.927 ± 0.0070.410 ± 0.0210.778 ± 0.0100.853 ± 0.078
No GradientSCALE0.859 ± 0.0110.650 ± 0.0590.920 ± 0.0080.423 ± 0.0230.779 ± 0.0120.841 ± 0.061
GradientGradNorm0.763 ± 0.0180.679 ± 0.0610.923 ± 0.0070.426 ± 0.0190.773 ± 0.0160.853 ± 0.082
Gradient + HessianWIF (ours)0.875 ± 0.0210.619 ± 0.0530.935 ± 0.0040.399 ± 0.0160.784 ± 0.0120.570 ± 0.030

[1] J. Zhang, J. Yang, P. Wang, H. Wang, Y. Lin, H. Zhang, Y. Sun, X. Du, Y. Li, Z. Liu, Y. Chen, and H. Li. OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection. arXiv preprint arXiv:2306.09301, 2023.

[2] H. Wang, S. Vaze, and K. Han. Dissecting Out-of-Distribution Detection and Open-Set Recognition: A Critical Analysis of Methods and Benchmarks. International Journal of Computer Vision (IJCV), 133(3):1326-1351, 2025. doi:10.1007/s11263-024-02222-4.

3) CL

We did not study continual learning (CL) in its entirety. Instead, we restricted our scope to presenting a concrete use case, order sensitivity in CL, together with a method for addressing it. Within this scope, we used EWC and GEM, which are widely used representatives of regularization-based and replay-based approaches, respectively. However, we recognize that these two methods alone may not be sufficient. We therefore tried to add one more representative method for each category, LwF (regularization-based) and ER (replay-based), and updated the table accordingly.

We agree that claiming general effectiveness across CL would be an overgeneralization. For this reason, we deliberately narrowed our experiments to this specific problem and report results only within that scope. Broader applicability and robust methods that can compete with state-of-the-art (SOTA) approaches were mentioned only as possible directions for future work in the Appendix limitation section.

Table 4. Class-incremental CL on Tiny ImageNet with ResNet-18. The WIF-based Ben-first ordering achieves the best performance. Average Accuracy (AA, ↑) is better when higher, and Forgetting Measure (FM, ↓) is better when lower. Results are averaged over 5 seeds.

OrderEWC AA ↑EWC FM ↓GEM AA ↑GEM FM ↓LwF AA ↑LwF FM ↓ER AA ↑ER FM ↓
Random0.40 ± 1.190.60 ± 1.310.54 ± 0.870.46 ± 0.510.37 ± 0.300.63 ± 0.330.66 ± 0.140.33 ± 0.41
Des-first w/ WIF0.37 ± 1.150.58 ± 1.370.62 ± 0.770.44 ± 0.450.33 ± 0.310.67 ± 0.440.65 ± 0.130.32 ± 0.47
Ben-first w/ Logit0.41 ± 1.580.56 ± 1.360.72 ± 0.680.32 ± 0.460.37 ± 0.270.63 ± 0.380.75 ± 0.190.28 ± 0.28
Ben-first w/ WIF (ours)0.42 ± 1.210.54 ± 1.660.74 ± 0.620.25 ± 0.470.40 ± 0.310.53 ± 0.380.78 ± 0.140.22 ± 0.40


W2

This research question is already addressed in Section 4.1. We define familiarity using the Influence Function (IF). As shown in Eq. 1, the IF approximates how much the model parameters change when a sample is added. In other words, the IF quantifies the parameter shift induced by a single sample. If a sample is already familiar from training, adding it barely moves the optimum of the model, which yields a low IF value. In contrast, an unfamiliar sample that the model has never encountered shifts the optimum substantially, which yields a high IF value. In general, such a shift arises in two situations: (1) when the labels of many inputs are entirely wrong, and (2) when the input information is sparse. OOD samples involve both.

Based on this rationale, we design familiarity using the IF. Compared to the evaluated baselines, the resulting measure performs strongly on OOD detection and OSR, which require distinguishing familiar samples from unfamiliar ones. The same holds for order sensitivity, where samples are ranked by separating parameter-destructive ones from benign ones. We have revised this part to emphasize it more and to elaborate on it in greater detail.

W3

We would like to reiterate the scope of our contribution. We then instantiate it for OSR, OOD detection, and order sensitivity in continual learning (CL), aiming to verify its potential and extend its applicability rather than to chase state-of-the-art numbers.

1) Positioning IF as a Distinct Lens on Familiarity.

Familiarity can be expressed in many ways. However, when the quantity of interest is an estimate of the optimal change, namely how much the model would have to change under the complex conditions of the trained model, rather than a difference between current states, the Influence Function (IF) is the formulation that addresses it. Methods that capture this property reveal what other approaches cannot, so they form a distinct line of work that deserves separate consideration in this field. At the same time, IF is a general-purpose estimator of parameter change. Its use for OOD detection is one application among many, and it should not be perceived as a dominant solution to OOD detection.

2) Performance on downstream applications should be regarded as only one part of our contribution.

We show, through theoretical justification and empirical experiments, that IF remains valid when extended to the sample-incremental regime. We also demonstrate strong performance on application tasks. In addition, we analyze why IF performs well in these settings.

3) The two methods measure fundamentally different quantities.

Mahalanobis distance (MD) measures density in the feature space (representation-level impact). This is precisely where it differs from IF. IF measures how much the parameters move when a sample is added (model-level impact). For the question “Is the model familiar with this sample?”, we believe the parameter-based IF provides the more suitable answer.

4) In practice, MD has not been shown to perform well on all three tasks.

MD is known to perform strongly on OOD detection. However, it is not an established method for measuring order sensitivity in CL. Forgetting is a problem of parameter interference, and its correspondence with density-based distance has never been verified. Likewise, we are not aware of any evidence that MD achieves high performance on OSR. If the reviewer has supporting references, we would appreciate being pointed to them.



W4

We already provide an implementation that reduces the computation to a d-dimensional space to handle large n (see Table 3 & Appendix B). By leveraging the push-through identity trick, WIF performs the computation in a dimension d that is much smaller than n, even when n is very large. Furthermore, the runtime of WIF is nearly identical to that of the other methods in Appendix B Table 6.

Table 6. Total time per sample for each method in OOD detection. WIF does not incur a prohibitive computational cost compared to other methods. All computations were conducted in the OOD detection setting.

MethodTotal time / sample (s)
MSP0.005
Energy0.005
ODIN0.015
ASH0.019
SCALE0.019
GradNorm0.005
WIF (ours)0.018

We would like to reiterate that our aim is to verify the potential of WIF and extend its applicability, rather than to chase state-of-the-art numbers.



Q

Addressed in W2&W3. As shown in Eq. 1, the IF formally approximates how much the model parameters change when a sample is added. A larger influence means that the sample changes the model parameters more, and a larger parameter change indicates that the model is unfamiliar with the sample and requires substantial updates to accommodate it.

Reviewer_J2WjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Woodbury Influence (WIF), the norm of the parameter-space influence vector ‖H⁻¹∇θℓ(z⋆, θ̂)‖. It is computed for unlabeled query samples, and the score is read as how “unfamiliar” a sample is to the trained model. The method is evaluated on OOD detection, OSR, and continual learning.

Critical strengths
  1. The paper is well-written and easy to follow.

  2. The question is well motivated: a principled, model-level notion of familiarity that uses curvature as well as gradient information.

  3. Limitations are disclosed fairly openly (retraining validation only on MNIST, no sensitivity analysis of λ).

Critical weaknesses
  1. Incomplete testbench for OOD detection, OSR, and continual learning. OSR is evaluated only on TinyImageNet, and there are only two evaluated methods for continual learning, which are far from sufficient. The work looks not completed yet.

  2. The “GT-label” variant scores ID samples with ground-truth labels, but OOD samples can only be scored with pseudo-labels. Comparing a mismatched pipeline against a matched one and reporting a 41.6% improvement is not a meaningful finding about pseudo-labeling. In the GT case, OOD-missed is 5,621 of 5,640, i.e., essentially everything.

  3. The claim that a smoother ID score-distribution tail explains better FPR95 is not generally valid. FPR95 depends on the relative positions of the ID quantile and OOD distribution, rather than smoothness alone.

Questions for authors

See 2 and 3 in the Weaknesses. But the completeness of the experiments is low.

3 comments

We thank the reviewer J2Wj for the careful reading. We look forward to a constructive discussion.

W1

We would first like to clarify the scope of our contributions. We propose WIF, which enables label-free influence estimation in the sample-incremental regime, and we justify it theoretically and empirically. We then instantiate it for OSR, OOD detection, and order sensitivity in continual learning (CL), aiming to verify its potential and extend its applicability rather than to chase state-of-the-art numbers. Finally, we identify two mechanisms behind its separation of familiar and unfamiliar samples.

1) OOD detection

For OOD detection, we consider our evaluation sufficient. The benchmark used in our OOD detection experiments remains in active use in OpenOOD. Our baselines include MSP and recent methods such as GradNorm (2021), ASH (2023), and SCALE (2024). Especially, MSP, SCALE, and GradNorm rank among the top methods on the OpenOOD leaderboard [1].

2) OSR

For OSR, we evaluate on the still challenging ImageNet-based benchmark, where even the current state-of-the-art methods achieve only about 0.7 to 0.8 AUROC and thus fall short of fully solving the problem [1][2]. We therefore consider experiments on this benchmark alone to be sufficient. However, we understand that evaluation on this single dataset may be considered insufficient. We therefore additionally conducted experiments on the classical benchmarks CIFAR-10/CIFAR-100 and SVHN. OSR results Table has been revised as follows. The results remain consistent with the original trends.

Table 2. Open-Set Recognition results on CIFAR10/CIFAR100, SVHN, and Tiny ImageNet with ResNet-18. Our method outperforms all the baselines considered. Higher AUROC and lower FPR95 indicate better performance. All results are reported as mean ± standard deviation across 5 runs with different known/unknown class splits. Best results in bold.

Method spaceMethodCIFAR10/100 AUROC ↑CIFAR10/100 FPR95 ↓SVHN AUROC ↑SVHN FPR95 ↓Tiny ImageNet AUROC ↑Tiny ImageNet FPR95 ↓
No GradientMSP0.770 ± 0.0120.702 ± 0.00710.917 ± 0.0080.517 ± 0.0210.776 ± 0.0110.826 ± 0.081
No GradientEnergy0.782 ± 0.0170.658 ± 0.0680.925 ± 0.0050.472 ± 0.0170.779 ± 0.0200.834 ± 0.071
No GradientODIN0.755 ± 0.0210.693 ± 0.0720.921 ± 0.0040.455 ± 0.0150.782 ± 0.0130.815 ± 0.070
No GradientASH0.789 ± 0.0130.650 ± 0.0690.927 ± 0.0070.410 ± 0.0210.778 ± 0.0100.853 ± 0.078
No GradientSCALE0.859 ± 0.0110.650 ± 0.0590.920 ± 0.0080.423 ± 0.0230.779 ± 0.0120.841 ± 0.061
GradientGradNorm0.763 ± 0.0180.679 ± 0.0610.923 ± 0.0070.426 ± 0.0190.773 ± 0.0160.853 ± 0.082
Gradient + HessianWIF (ours)0.875 ± 0.0210.619 ± 0.0530.935 ± 0.0040.399 ± 0.0160.784 ± 0.0120.570 ± 0.030

[1] J. Zhang, J. Yang, P. Wang, H. Wang, Y. Lin, H. Zhang, Y. Sun, X. Du, Y. Li, Z. Liu, Y. Chen, and H. Li. OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection. arXiv preprint arXiv:2306.09301, 2023.

[2] H. Wang, S. Vaze, and K. Han. Dissecting Out-of-Distribution Detection and Open-Set Recognition: A Critical Analysis of Methods and Benchmarks. International Journal of Computer Vision (IJCV), 133(3):1326-1351, 2025. doi:10.1007/s11263-024-02222-4.

3) CL

We did not study continual learning (CL) in its entirety. Instead, we restricted our scope to presenting a concrete use case, order sensitivity in CL, together with a method for addressing it. Within this scope, we used EWC and GEM, which are widely used representatives of regularization-based and replay-based approaches, respectively. However, we recognize that these two methods alone may not be sufficient. We therefore tried to add one more representative method for each category, LwF (regularization-based) and ER (replay-based), and updated the table accordingly.

We agree that claiming general effectiveness across CL would be an overgeneralization. For this reason, we deliberately narrowed our experiments to this specific problem and report results only within that scope. Broader applicability and robust methods that can compete with state-of-the-art (SOTA) approaches were mentioned only as possible directions for future work in the Appendix limitation section.

Table 4. Class-incremental CL on Tiny ImageNet with ResNet-18. The WIF-based Ben-first ordering achieves the best performance. Average Accuracy (AA, ↑) is better when higher, and Forgetting Measure (FM, ↓) is better when lower. Results are averaged over 5 seeds.

OrderEWC AA ↑EWC FM ↓GEM AA ↑GEM FM ↓LwF AA ↑LwF FM ↓ER AA ↑ER FM ↓
Random0.40 ± 1.190.60 ± 1.310.54 ± 0.870.46 ± 0.510.37 ± 0.300.63 ± 0.330.66 ± 0.140.33 ± 0.41
Des-first w/ WIF0.37 ± 1.150.58 ± 1.370.62 ± 0.770.44 ± 0.450.33 ± 0.310.67 ± 0.440.65 ± 0.130.32 ± 0.47
Ben-first w/ Logit0.41 ± 1.580.56 ± 1.360.72 ± 0.680.32 ± 0.460.37 ± 0.270.63 ± 0.380.75 ± 0.190.28 ± 0.28
Ben-first w/ WIF (ours)0.42 ± 1.210.54 ± 1.660.74 ± 0.620.25 ± 0.470.40 ± 0.310.53 ± 0.380.78 ± 0.140.22 ± 0.40


W2

1) The reason pseudo-labeling (PL) is the only option for OOD samples

WIF’s formulation involves gradients, which require labels to compute. Since computing the gradient with the ground-truth (GT) labels of OOD samples is mathematically infeasible, we compute the score with pseudo-labels (PL) instead. In other words, PL is the only option for OOD data, so its effect cannot be isolated through an ablation study on OOD data. We therefore examine the role of PL on the ID dataset, where GT labels are mathematically available.

2) Experimental goal: What role does PL play on the ID dataset?

Since the effect of PL (pseudo-labels) on OOD samples cannot be verified directly, we designed an ablation study to examine the role of PL on ID data. GT labels must not be used at the OOD detection test stage. We assume GT access only in this ablation, solely to isolate the role of PL. Our model achieves 94.73% accuracy on the ID test set. Using GT therefore builds the ID score distribution with 5.27% label error (noise). If the model were 100% accurate, GT and PL would yield exactly the same score distribution. This noise is a factor that degrades OOD detection performance. We therefore build the score distribution from PL only. This removes the noise. We then test whether it better separates samples that are familiar to the model from those that are unfamiliar. The left panel of Fig. 2 shows that this works for the gradient-based method. The right panel of Fig. 2 shows that PL also works with WIF.

W3

1) Fixed 95% quantile and Robustness across quantiles

FPR95 is a widely used metric in OOD detection benchmarks, and we adopt it for compatibility with them. Fixing the position of the ID quantile at 95% and analyzing other factors is a well-justified direction. Nevertheless, following the reviewer’s comment, we varied the percentile and found WIF to be empirically robust (see Appendix Fig. 4). Regardless of the percentile, WIF still retains an small overlapping region compared with the other methods, which demonstrates its strong performance.

FPR95 is determined by the 95-th percentile of the ID scores. Under this metric, a method achieves a low FPR95 simply by minimizing the overlap between the ID and OOD score distributions. No existing method separates the two distributions perfectly, so every method retains some overlap, and this overlap always begins at the tail of the ID distribution. The decisive factor is therefore the shape of the ID distribution. We conjecture that the ID distribution is head-skewed, and its smoothly decaying tail reduces the overlap.

2) Nearly identical OOD score shapes across methods

Fig. 3 shows that the OOD score distributions of all methods share a nearly identical shape. Most samples concentrate in the head region. The density becomes sparse toward the tail and decays gently. Since the OOD data distributions have almost the same form, the differences between methods arise from the ID score distributions. This suggests that OOD detection performance is determined by the ID score distribution.

3) Favorable ID distribution shape

An favorable ID distribution is dense at the head, where most samples concentrate, and becomes progressively sparser toward the tail with a gradual decay.

Strictly speaking, then, the analysis should consider not only the smooth tail near the 95% point but also the dense head together with the smooth, sparse tail. Among the methods in Fig. 3, WIF is the closest one whose ID distribution exhibits this ideal shape.

We revised the script to state these points. Thank you for the constructive feedback.



Q

Addressed in W1, W2, and W3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.