- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper propose Normal Log-Density Taylor Signatures (NLTS) for comparing a reference distribution with another distribution . Each observation is represented through the reference score by selected directions of the negative-log-density Hessian and their associated curvature values and finally the remaining score energy. The resulting are compared using MMD. The advantage is sensitivity to changes in normal geometry, together with a weaker response to redistribution along the data manifold. The paper provides a local theoretical interpretation and experiments on synthetic distributions, CIFAR-10-C.
Critical strengths
The distinction between different types of distributional change is very intesting, intuitive then well motivated! Using geometry to determine which changes should matter is a useful direction for distribution comparison particularly in high dimensional : measure concentration is at my knowledge not enough considered in litterature and I appreciate this concern. The construction is clear, understandable, and the Taylor expansion provides a good interpretation when the reference score and Hessian are exact. The synthetic evaluation includes both target changes and tangential controls, which is natural for the stated objective. The author are also honest about several important limitations. It distinguishes pointwise conditional invariance from invariance of the overall signature distribution, discusses repeated eigenvalues, and presents the real-data experiments conclusions as indirect. The CIFAR-10-C results show very promising sensitivity to corruptions.
Critical weaknesses
- It is unclear why two (or any arbitrary) directions are needed, or what they add compared with simpler calculations Indeed, if the data spread equally in every normal direction, can looking at one normal direction already detect this change ? The paper compares two directions with four, but may report the results with one direction. More importantly, the method could be compared with a test using only the squared length of the score, . This quantity already measures how strongly the reference log density changes near an observation. The full method contains this information because
Comparing the full method with this simple quantity, and with alone, would show whether calculating the two directions provides a improvement and strengthen the contribution.
-
The two selected directions may change substantially without a meaningful change in the data. In the affine Gaussian example, all the normal directions have the same eigenvalue. There is therefore no uniquely pair of directions which can be chosen. Moreover, different initializations or small numerical errors can produce different choices. Squaring the projections removes dependence on the sign of a direction is good but it does not remove dependence on which directions are selected. But I believe that this does not automatically mean that the final test is unstable. but the authours should explicit that : whether changing the initialization or the number of iterations changes the calculated numbers and the final decisions. I also recognize that the authors discuss the ambiguity in Appendix A.4.3, but its practical effect remains unclear to me.
-
The calculation used on CIFAR differs from the calculation studied in the theory. The theory uses a Hessian, which is a symmetric matrix of second derivatives. On CIFAR, the authors instead approximate derivatives so the resulting operator is not necessarily symmetric. Appendix F.2 explains that the algorithm starts with random directions and updates them through eight iterations. The final directions are therefore not simply random but author should show that these eight iterations are sufficient or that the results are stable across initializations.
-
The results using only eigenvalues in Table 10 need more clarification : for the affine Gaussian example described in Appendix D.1, the Hessian is the same at every observation so its eigenvalues are therefore also the same. A method using only these eigenvalues should have no information for distinguishing the two distributions. But Table 10 reports detection power of 0.98 for Local artifact and 0.78 for Single-dimension expansion using only eigenvalues. If these experiments use the stated affine Gaussian, the results are difficult to understand. Could the authors specify the distributions and numerical calculations used for these rows ?
-
The experiments do not establish how reliably the method distinguishes the intended types of change. If we suppose that the reference data are concentrated near a curve due to measure concentration with a thin spread around one part and a thicker spread around another then increasing the proportion of observations in the thicker region changes the distribution of the calculated numbers, even if the spread at every location remains unchanged. Any experiment of this kind would help determine when the method responds to redistribution along the curve. The image experiments demonstrate detection of corruptions, but provide poor evidence about changes the method should ignore. and the paper should explain more precisely why this redistribution is considered a change to detect.
If the author carefully clarify all my weaknesses, I'm willing to raise my score as it is an very interesting idea.
Minor : the font of the title/sections seems different. I'm unsure about its importance but anyway, could the authors correct that in order to avoid any desk reject because of the ICLR guideline?
Questions for authors
See weaknesses.
7 comments
Q3. Nonsymmetric CIFAR operator and numerical stability
CIFAR's finite-difference (FD) operator is not guaranteed to be symmetric. We evaluated test stability across power iterations () and three initialization seeds on CIFAR-10-C.
| Iterations | Gaussian noise | JPEG compression | Brightness |
|---|---|---|---|
| 4 | 100% (100%) | 97.2% (94.4%) | 27.8% (77.8%) |
| 8 (default) | 100% (100%) | 100% (100%) | 36.1% (77.8%) |
| 16 | 100% (100%) | 100% (100%) | 30.6% (77.8%) |
| 32 | 100% (100%) | 100% (100%) | 27.8% (77.8%) |
Detection rate (decision agreement across initializations); severity 1, , 12 tests/seed, .
At the default , Gaussian noise and JPEG achieved 100% detection and initialization agreement, maintained at . Brightness remained challenging (36.1% detection; 19.4% null rejection).
Importantly, in practice, FD achieved substantially faster NLTS feature extraction (approximately 4.5$\times$ under our benchmark settings) without loss in detection performance relative to the symmetrized operator for Gaussian noise and JPEG compression (100% for both). See our response to Reviewer kbGz, Q1 for the detailed comparison.
Together, these findings demonstrate stable test decisions for the strongly detected shifts without requiring a symmetric operator, while not implying exact eigendirection recovery or universal stability. We will clarify these empirical findings and the scope of the theoretical guarantees in the revision.
Q4. Inconsistent eigenvalue-only results in Table 10
The reviewer is correct: the affine Gaussian Hessian is constant, so its eigenvalues cannot distinguish distributions.
Our investigation identified an assembly error: results from a different method explored in our subsequent research were inadvertently included in Table 10. We found four affected entries, including two beyond those identified by the reviewer. We sincerely apologize for this error, which overstated eigenvalue-only performance.
The corrected results (Cohen's / empirical power) are:
| Scenario | Reported | Corrected |
|---|---|---|
| Dimension-mixture shift | 1.02 / 0.48 | 0.00 / 0.000 |
| Local artifact | 2.36 / 0.98 | 0.02 / 0.06 |
| Curved local artifact | 3.29 / 0.95 | 0.48 / 0.10 |
| Single-dimension expansion | 1.28 / 0.78 | 0.01 / 0.03 |
We recomputed the eigenvalue-only test using MMD on the top two absolute eigenvalues of the fixed-reference Hessian and verified their constancy in the affine Gaussian setting. An audit of the remaining reported numerical results found no further discrepancies.
These corrections do not affect the NLTS results or the paper's main conclusions; rather, they clarify the limitations of eigenvalue magnitudes alone. We will correct Table 10 accordingly.
Q5. What changes should NLTS detect, and when should it ignore tangential redistribution?
We thank the reviewer for the concrete example, which helps clarify the intended scope of NLTS.
Motivation and detection target. Our goal is to detect changes in off-manifold variation across the population. Such changes can arise in two ways:
- Case 1: The conditional distribution of normal displacements changes at a manifold location, .
- Case 2: The conditional distributions remain unchanged, but the sampling frequencies of manifold locations change, .
The reviewer's example is Case 2. When more samples come from regions with greater normal thickness, the population contains more samples with large off-manifold deviations, even though local normal variation remains unchanged. For applications monitoring departures from the reference manifold, this change in the population is important to detect.
Conversely, NLTS aims to suppress responses to tangential redistribution that leaves population-level normal variation unchanged, as examined by the six tangential controls in Fig. 3(b) and Appendix D.4.
Already established in the paper. Proposition 3 shows that tangential reweighting preserves the conditional normal distribution at each location. Importantly, we already state immediately after Proposition 3 that this does not imply invariance of the population-level signature distribution (Case 2). Eq. (18) gives the precise condition: NLTS has a positive population discrepancy when the reference-defined signature distributions differ.
Additional experiment setting. Following the reviewer's suggestion, we constructed a curved manifold with location-dependent normal thickness. We varied the sampling distribution along the manifold while keeping the conditional normal distributions unchanged. For manifold location , we used and Normal thickness was greatest near , so increasing increased the proportion of samples from thicker regions. We also tested two tangential controls: uniform normal thickness, and heterogeneous thickness with reweighting proportional to , which preserves the population distribution of normal thickness by symmetry.
Two diagnostic checks were included: a conditional-normal oracle using known manifold coordinates and normalized normal displacements, and a tangent-location control using . The former checks responses to local normal changes; the latter verifies redistribution along the manifold. Neither is a competing baseline. We used analytical scores and Hessians (, , ; 300 trials; permutation tests at ). The main reweighting experiments used .
| Distribution change | NLTS | Conditional-normal oracle | Tangent-location control |
|---|---|---|---|
| No change ( vs. ) | 4.7% | 4.0% | 5.7% |
| Reweighting, uniform thickness | 6.0% | 5.7% | 99.3% |
| Reweighting, heterogeneous but symmetry-matched thickness | 5.3% | 5.0% | 99.3% |
| Reweighting toward thicker regions | 99.0% | 6.0% | 99.3% |
| Increased normal thickness (1.8$\times$) | 99.3% | 100% | 5.3% |
Empirical rejection rates; significance level 0.05.
Results: NLTS detects the intended population-level change. NLTS remains near the nominal rejection level for uniform-thickness and symmetry-matched reweighting (6.0% and 5.3%), but detects the reviewer's proposed redistribution toward thicker regions (99.0%). The conditional-normal oracle remains near nominal (6.0%), consistent with the unchanged local conditional distributions. NLTS also detects actual normal-thickness changes (99.3%).
We further varied the redistribution strength . NLTS rejection increased from 5.5% to 14%, 50%, 91%, and 100%, respectively (200 trials per setting), while the conditional-normal oracle remained near nominal (3.5%–8.5%).
Revision to the Introduction. We will make the detection target explicit by adding the following explanation:
Our goal is to detect changes in off-manifold variation across the population. Such changes can arise either from changes in local conditional normal distributions or from changes in the frequency of sampling regions with different normal thickness. In the latter case, the population-level prevalence of off-manifold deviations can change even when local conditional distributions remain unchanged. NLTS aims to capture both types of change while limiting sensitivity to tangential redistribution that leaves population-level off-manifold variation unchanged.
Q1. Why two directions, and what do they add beyond simpler score-based statistics?
We agree that k=1 may suffice for isotropic changes. However, real-world shifts can be anisotropic, motivating k=2 as the smallest multi-directional signature. To test this choice, we conducted two additional ablations.
(1) Directional ablation (controlled synthetic). We used Gaussian distributions with 8 tangent dimensions and 10 or 66 normal dimensions (D=18 or 74), with the first two normal directions given by the exact reference Hessian. We applied controlled variance-scaling shifts (not dimension expansion) and compared k=1 and k=2. The table reports empirical power averaged over n∈{16,32,64,128} and both dimensional settings (200 trials per setting, α=0.05).
While both perform comparably on uniform normal and first-direction shifts, k=2 improves second-direction detection by 36.44 percentage points, demonstrating the benefit of isolating directional information otherwise pooled in the residual.
(2) Score norm and residual-only comparison (CIFAR-10-C). As the reviewer notes, $ |s_P(x)|^2=a_1(x)^2+a_2(x)^2+r_2(x). $ NLTS retains this score energy while additionally representing its Hessian-aligned directional decomposition and curvature. We compared these representations across 342 matched settings (19 corruptions × 3 severities × 6 sample sizes):
Both NLTS variants outperform the scalar baselines by approximately 11 percentage points, demonstrating the benefit of the structured score-curvature signature.
Together, these results support the full signature over simpler score-based statistics and k=2 for anisotropic sensitivity, without claiming that two directions are universally necessary. We will incorporate both ablations and clarify the design choice.
Q2. Does eigenvector ambiguity compromise test stability?
We agree that squared projections do not resolve basis ambiguity, but as the reviewer suggests, this need not imply test instability.
To assess practical stability, we used the affine Gaussian synthetic model (D=256, 8 tangent and 248 isotropic normal dimensions; normal standard deviation σn=0.05), with analytical scores and Hessians. We varied eigensolver initialization (6 random starts) and iteration count (T∈4,8,16) across 30 independent datasets (k=2).
Test decisions were identical across all iteration counts, while initialization agreement reached 100% for the clear isotropic shift. Although weaker shifts showed some initialization sensitivity, the results confirm that eigenvector ambiguity does not necessarily compromise test-level stability. We will clarify this distinction and report the new results.