- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how a nonlinear maximum a posteriori estimate changes when the centre of its Gaussian prior is shifted. It contrasts the Fisher/Gauss–Newton prediction with the exact local sensitivity. The main argument is that, when the Fisher matrix has one weakly constrained direction, the omitted curvature along that direction can substantially alter prior sensitivity. The authors propose a rank-one correction that shifts the corresponding Fisher eigenvalue using a directional curvature estimate.
Critical strengths
-
A concrete and practically relevant diagnostic. Perturbing the prior centre and refitting directly tests the quantity being predicted. This provides a clear empirical comparison between Fisher-based sensitivity, exact curvature, and observed displacement.
-
Substantial empirical evidence in the main application. The clinical cohort is large, and the paper reports effect sizes, patient-resampled confidence intervals, exam-level consistency, and robustness across instruments and prior settings. The improvement in T1 rank correlation from 0.419 to 0.915 is especially compelling: the discrepancy affects ordering as well as magnitude.
-
A potentially useful inexpensive correction. Estimating curvature along a single weak direction is attractive when that direction dominates sensitivity. Recovering 92–99% of the full-Hessian correction in the clinical setting provides meaningful evidence for this approach.
-
Breadth and disclosure of limitations. The public-data experiments strengthen the case beyond one private cohort. The authors also report settings where the rank-one correction fails, acknowledge that it does not improve influence functions, and disclose that it is slower than the full approach at four parameters. These qualifications improve the credibility of the empirical presentation.
Critical weaknesses
-
The central approximation requires additional assumptions. Equation 6 gives the sensitivity ratio as approximately . This requires small directional curvature relative to , in addition to the stated spectral-gap assumptions. Yet the reported is 0.655 for clinical MRI and exceeds 1 in several other experiments, making the first-order approximation inaccurate or even negative. Appendix C acknowledges this and supplies the more appropriate rational expression, but the abstract and main theoretical presentation still foreground the first-order result. More fundamentally, the gap alone does not justify neglecting the other blocks of . In the Fisher eigenbasis, the effective weak-direction curvature includes the Schur-complement term . Bounds on cross-direction coupling and the constrained block are needed to justify dropping this term. The rank-one approximation is empirically successful in some settings, but its stated theoretical conditions are incomplete.
-
The controls do not isolate misspecification from fitting and regularization. The residual and curvature are evaluated at a fitted parameter that depends on the observations and prior. Even with a correctly specified model and zero-mean observation noise, the fitted residual need not be independent of curvature or have zero mean under regularization. The proposed nulls also require stronger assumptions than stated. Zero mean does not imply sign symmetry or a 50% probability of either sign. Voxel permutation requires exchangeability assumptions and may disrupt anatomical structure and spatial dependence. Consequently, these tests demonstrate structured residual–curvature alignment more directly than they establish its cause. A controlled synthetic experiment should vary model misspecification while holding acquisition, noise, prior, and optimization fixed. Matched correctly specified simulations would be particularly valuable for supporting the title’s “not noise” claim.
-
Several claims about neural-network curvature are incorrect. Section 4.4 states that a ReLU network has almost everywhere because it is piecewise linear. ReLU networks are piecewise linear in their inputs, but generally not jointly piecewise linear in all trainable weights. Mixed second derivatives across layers can remain nonzero within a fixed activation region. The assertion that near interpolation necessarily produces indefinite posterior curvature is also problematic. At exact interpolation, the residual curvature term vanishes, so , and is positive definite for . Indefinite curvature at returned solutions should therefore prompt investigation of convergence and optimization, rather than be attributed generally to interpolation.
-
The network admission criteria risk selecting favourable cases. Appendix D admits runs partly based on observed displacement being aligned with , positive curvature, and measured sensitivity lying in . These criteria condition the evaluation on properties related to the proposed explanation. Moreover, the relevant minimum condition is , rather than necessarily . A positive-definite posterior Hessian can coexist with negative likelihood curvature and prior sensitivity exceeding one. The manuscript should clarify its notation, justify exclusions independently of measured outcomes, and report results for all runs.
-
Finite shifts are conflated with local derivatives and prior attribution. The exact formula predicts an infinitesimal derivative, whereas the main clinical experiment uses a 25% prior-centre shift. The appendix reports changes of up to approximately 10% with shift size, comparable to or larger than the remaining discrepancy after exact correction. A sequence of shrinking, preferably central, perturbations would distinguish nonlinear path effects from numerical or theoretical error. In addition, local sensitivity to the prior centre is not generally the fraction of a fitted value “coming from the prior.” That interpretation is justified in the linearized decomposition, but needs qualification for nonlinear fits without known ground truth.
-
The broader machine-learning claims exceed the demonstrated scope. The network experiments concern small, single-hidden-layer regressors fitted with a specialized optimizer. The rank-one correction does not help influence functions, and the Laplace experiment does not establish improved calibration. Likewise, testing a diagonal Fisher matrix on prior sensitivity does not establish the effect on continual-learning performance. The distinction between Fisher information and Hessian curvature is already established in the literature, including work cited by the authors. The contribution would be clearer if framed around the empirical sensitivity diagnostic and its conditional correction, with more precise positioning against existing curvature analyses.
Questions for authors
Please see the weakness part.
3 comments
Response to Reviewer ysT7 (2/2): Matched Control Built and It Changed Our Title; Admission Criteria Fixed, Central Shifts Run, Claims Narrowed to What We Measured
W2. The controls do not isolate misspecification from fitting and regularization... A controlled synthetic experiment should vary model misspecification while holding acquisition, noise, prior, and optimization fixed. Matched correctly specified simulations would be particularly valuable for supporting the title's "not noise" claim.
We built exactly the experiment you specify, and it changed our title.
-
The effect tracks alignment, not cause. Varying misspecification with acquisition, noise, prior and optimisation fixed, the residual grows by five orders of magnitude while the Fisher error shrinks (table in our reply to 22am). The structural reason is that at a weak-prior MAP the normal equations push injected misfit where the curvature direction has no component.
-
The "not noise" half survives. A correct-model noiseless fit over-predicts by 32%, parameter-independent to 0.2%, with no noise anywhere in it.
-
The title now claims only what the tests establish. It reads curvature-correlated residuals, not noise. As you put it, the tests demonstrate structured residual-curvature alignment more directly than they establish its cause, so the title now names the alignment.
W4. Appendix D admits runs partly based on observed displacement being aligned with v, positive curvature, and measured sensitivity lying in (0, 1]. These criteria condition the evaluation on properties related to the proposed explanation.
- The sensitivity bound is removed from admission, and the posterior test replaced the data-Hessian test. You are right that the alignment test still conditions on a measured outcome, so we report both sets: the median exact ratio is 1.0003 with the filter and 1.0003 without, over all 77 runs, with only the tail moving. That test catches refits that left the stationary point the implicit function theorem describes, visible as a 1/delta divergence.
W5. The exact formula predicts an infinitesimal derivative, whereas the main clinical experiment uses a 25% prior-centre shift... In addition, local sensitivity to the prior centre is not generally the fraction of a fitted value "coming from the prior."
-
You identified the dominant error term. Differencing centrally at the same 25% shift moves measured/Hessian from 1.061 / 0.963 / 1.018 to 1.011 / 1.012 / 1.001. About 1% survives, flat across a 16-fold range.
-
Your wording point stands too. Section 3 now calls equation 2 a linearisation and defines the phrase to that order, and the appendix states that the measured object is the derivative of the estimate with respect to the prior centre, defined for any twice-differentiable fit at a non-degenerate stationary point.
W6. The broader machine-learning claims exceed the demonstrated scope.
-
We have adopted your framing. The contributions had claimed the failure and its repair reproduce on influence functions, where the repair in fact recovers -0.02%. That overreach is gone. The Discussion now takes each case separately: influence functions reorder (0.628 to 0.881) and the repair does not transfer; Laplace is a measured null; for elastic weight consolidation we claim only that a diagonal Fisher cannot see a mixed near-null direction.
-
On your point that the Fisher-versus-Hessian distinction is already established, you are right that we cited none of the local prior-sensitivity literature. Related work now cites Giordano et al. (2018) on linear response for variational covariances and local robustness, and Immer et al. (2021) on the Gauss-Newton substitution as a local linearisation. Against that, what we add is the sign of the dropped term, its measurement at clinical scale, and a scalar that corrects it without the Hessian.
In the revised PDF (changes in blue): the title, Section 2's positioning, Section 3's wording on the linearisation, Section 4.1's central-difference results, and the rewritten Discussion.
Your review changed what this paper claims, and we are grateful for that. We hope the narrowed claims and the new controls address your concerns, and if so we would be grateful if you would consider raising your score.
Follow-up to Reviewer ysT7: Checking Whether Anything Remains Open Before the Discussion Period Closes
Dear Reviewer ysT7,
With the discussion period closing on 19 November, we wanted to check whether anything remains open on your side.
Acting on your review, we reordered the equation ladder so the exact scalar leads and the first-order form carries its condition, removed both incorrect network claims, replaced the data-Hessian admission test with the posterior test, built the controlled misspecification experiment you specified, and changed the title: that experiment showed the effect follows the residual's alignment with curvature rather than its cause, so "model misspecification" was not the right word. Your Schur point also led us to a second diagnostic, , which narrowed one of our claims rather than supporting it.
We genuinely appreciate the care that review took, and we would like to use the remaining time well. If any of W1 to W6 still feels unresolved, or if something we changed has raised a new question, please tell us and we will do our best to address it before the deadline.
Response to Reviewer ysT7 (1/2): You Were Right on All Three Theory Points; Conditions Now Stated, Coupling Measured, and One Claim Narrowed
Thank you for this review. Your three theoretical objections were all correct, and acting on them changed the paper's claims rather than just its wording. We also appreciate your noting that the paper "reports settings where the rank-one correction fails" and that these qualifications "improve the credibility of the empirical presentation".
You are right that we never stated the conditions, and right again that the gap does not bound what we dropped. Both are fixed.
The equation ladder is reordered. The exact rational form is now equation 7, the first-order form is equation 8 with its condition printed in the display, and the abstract quotes the exact one. The violation is not marginal: at v⊤Cv/λ≈+6 on photovoltaics, the first-order form predicts to 62% where the rational form predicts to 1.0%.
We measured the coupling instead of bounding it (full table in our reply to 22am). It is 0.0000 of the kept curvature where the gap holds, and up to 168% where it does not.
Your Schur point is sharp and it narrows one of our claims. The gap bounds the stiff Fisher block but says nothing about the stiff curvature block, which equation 6 also assumes is unmoved. We added a second diagnostic, equation 24, η⊥=∥(F⊥+λI)−1/2C⊥⊥(F⊥+λI)−1/2∥2, the stiff block's curvature against its own posterior curvature. It is 0.0247 median on the photovoltaic fits against 4.50 on the networks, separating the arms the same way the gap does.
The claim this narrows. Our consequence (ii), that a parameter with no weight on the weak direction (vj=0) is untouched, assumes that diagnostic is negligible. In a constructed case with a gap of 10−4 and the weak-direction curvature exactly zero, so the rank-one repair predicts 1 and makes no change, the exact ratio still falls from 0.9998 to 0.1730 as a parameter's loading on the weak direction falls. Consequence (i) and the scalar are unaffected. The appendix now states consequence (ii) with this diagnostic as its condition.
Both claims were wrong and both are gone.
ReLU curvature is the same order as tanh, not zero. A ReLU network is linear in each layer's weights separately but not jointly, so only the blocks containing the activation's second derivative vanish. On the Airfoil fit the curvature norm is 3261.5 for ReLU against 5100.5 for tanh, with the first-layer block exactly zero and the cross-layer block carrying 2306.2. We use tanh because the second derivative must exist everywhere.
You are right about interpolation. At exact interpolation the residual term vanishes and the data Hessian is positive semidefinite. The Yacht fits hold negative data curvature and a positive-definite posterior at the same time, which is ordinary at a MAP solution; 22am independently drove the gradient norm to 1.6×10−13 with the signs unchanged. Section 4.6 now says this instead of attributing the exclusions to interpolation.
In the revised PDF (changes in blue): the equation ladder in Section 3.1, the new Appendix C diagnostic, and the corrected Section 4.4 and 4.6. Our second reply covers your remaining three points. We hope this answers W1 and W3, and we would welcome your view on whether the narrowed claims are now stated correctly.