- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes "LENS", a post-hoc uncertainty quantification method that exploits the nonlinear geometry of the empirical loss landscape in parameter space. LENS constructs an ensemble of models with comparable empirical losses by exploring approximate loss-level sets through a gradient-based predictor-corrector procedure. The predictive disagreement among the collected models is then used to estimate uncertainty. The authors evaluate LENS on regression, image classification, and LoRA-adapted language model benchmarks, reporting improvements in uncertainty calibration and likelihood-based metrics while maintaining competitive predictive performance.
Critical strengths
None.
Critical weaknesses
1. Insufficient Theoretical and Conceptual Justification
The paper lacks a sufficiently rigorous justification for connecting nonlinear loss-level-set exploration to reliable uncertainty quantification. In particular, Definition 3.1 introduces a globally defined relative level-set tube that may contain disconnected components, whereas the proposed algorithm primarily explores local regions around a single trained solution. The authors do not adequately explain how incomplete exploration affects the representativeness of the resulting ensemble or its uncertainty estimates. Furthermore, Proposition 3.3 is a direct restatement of the definition, while Proposition 3.4 only establishes local first-order loss preservation under smoothness assumptions, without providing guarantees for the finite-step sampling procedure or uncertainty estimation.
2. Insufficient Empirical Validation and Methodological Reliability
The experimental evaluation does not sufficiently establish whether the reported improvements arise from the proposed nonlinear level-set geometry or from task-specific sampling choices. LENS introduces numerous hyperparameters that directly influence ensemble construction, yet their selection criteria and sensitivity are not adequately investigated. Moreover, the rationale for baseline selection is unclear, and several experimental observations remain insufficiently explained. Finally, the lack of quantitative computational cost comparisons makes it difficult to assess the practical efficiency of the proposed method.
Questions for authors
Definition 3.1 defines a global relative level-set tube that may contain multiple disconnected components, whereas LENS explores only a potentially limited subset of this tube through initialization-dependent local sampling. Given that the resulting ensemble may not adequately represent the set of plausible low-loss models, what justifies interpreting its predictive disagreement as a reliable measure of epistemic uncertainty? In particular, how do the authors ensure that unexplored low-loss regions do not substantially affect the resulting uncertainty estimates?
4 comments
Q1: Unexplored Regions
LENS’s disagreement measures predictive variation among configurations with comparable loss that are constructed from the available checkpoint. Loss comparability constrains the ensemble; it does not, by itself, guarantee calibrated uncertainty. The justification for its usefulness comes from held-out predictive and calibration evaluations, including distribution-shift experiments.
Establishing that unexplored low-loss regions cannot alter the uncertainty estimates would require global coverage or additional assumptions beyond the scope of this study. Our contribution is an initialization-dependent ensemble construction with demonstrated empirical utility, rather than an exhaustive characterization of all plausible models. Initialization dependence and incomplete exploration are acknowledged limitations in Section 5.
Ensemble-LENS additionally investigates aggregation across independently trained checkpoints and demonstrates improvements in several shifted settings. This supports the value of combining within-checkpoint exploration with across-checkpoint diversity, without claiming that the resulting ensemble exhaustively covers the global feasible set.
Thank you for authors’ thoughtful response to my concerns. Reading comments has helped resolve several of my points, including some that stemmed from my own misunderstandings specifically regarding the hyperparameter and cost analysis for W2, as well as the exploration space mentioned in Q1. However, as my concerns regarding the theoretical justification and sensitivity analysis remain, I would like to ask a few follow-up questions to clarify these aspects.
To follow up on the theoretical justification, Proposition 3.4 assumes the function is twice continuously differentiable. Could you clarify whether this condition can hold in a structure that incorporates activation functions, and perhaps comment on how easily or with what degree of difficulty this assumption is satisfied in such a context?
Regarding sensitivity, it is significant that the hyperparameters proposed can be applied across various environments. However, this seems quite different from the sensitivity I mentioned. Realistically, it is not possible to conduct experiments across every model and setting. However, since the foundation presented in the paper is based on empirical results, there is a need for guidance on how to make adjustments when the proposed setting does not perform as expected. For instance, if the provided hyperparameters do not yield good results, how should users go about adjusting them? Most users cannot easily predict the impact of these hyperparameters Therefore, I requested that practical guidelines be provided based on the interactions or behaviors of the parameters. This is all the more important given that the response to W1 mentions that uncertainty is evaluated empirically.
Thank you for the follow-up and for clarifying the remaining concerns. We address the smoothness assumption and practical hyperparameter adjustment separately below.
Smoothness assumption: Proposition 3.4 requires local smoothness of the construction loss near the current parameters, not global smoothness throughout parameter space. Smooth activations such as SiLU are compatible with this assumption. For ReLU-based networks, the justification is local. Away from activation-switching boundaries, the activation pattern remains unchanged within a sufficiently small neighborhood, where each ReLU acts as either the identity or zero. Thus, ReLU does not itself prevent local smoothness. The potential restriction, however, is that this neighborhood can become small near switching boundaries; thus we do not claim that the proposition applies at those boundaries. We will further clarify this in the paper. Finally, we want to clarify that for proposals, loss admissibility is checked separately through explicit loss evaluation and acceptance testing, without assuming that every proposal remains within a smooth neighborhood.
Practical hyperparameter guidance: As acknowledged in limitations in Section 5, in this work we do not provide an automatic hyperparameter-selection rule. However, based on the roles and interactions of the construction parameters the following practical guidance for hyperparameter adjustment can be considered.
-
Tangent step size controls the extent of exploration. Very small steps can preserve predictions close to the deterministic reference but provide insufficient predictive diversity to improve calibration. Increasing the step range can expose additional disagreement. Excessively large steps, however, can increase loss drift, reduce acceptance, and require more correction. We therefore recommend increasing the step range gradually when predictions remain nearly identical, and reducing it when collection becomes difficult.
-
Loss tolerance controls the permitted departure from the reference construction loss. A tighter tolerance enforces closer empirical fit but can make collection difficult through frequent rejection. A looser tolerance facilitates collection but admits greater loss deviations, potentially compromising predictive performance. It should therefore balance empirical-fit preservation and collection feasibility, rather than being relaxed solely to increase acceptance. We do not prescribe a fixed acceptance-rate threshold. Instead, a short pilot can assess whether the requested ensemble can be collected within the available computational budget. For a target of configurations and a budget of proposals, provides a basic collection-feasibility benchmark, not a calibration-optimal target. Acceptance should therefore guide collection feasibility with respect to the task at hand alongside validation performance, rather than be maximized in isolation.
-
Correction depth determines how much correction work each proposal receives. Too few iterations can reject proposals that additional correction could recover. Increasing the budget is useful when residuals are decreasing but beyond a sufficient correction depth, further increases would not provide additional benefits.
To examine these behaviors empirically, we evaluated 48 Yacht configurations over three seeds, varying these three parameters while holding the remaining settings fixed.
With and , increasing from 0.1 to 5 reduces CalErr from 0.267 to 0.178, although NRMSE increases from 0.087 to 0.103. Increasing further to 10 worsens CalErr to 0.223 and lowers proposal acceptance.
With and , relaxing from through to increases acceptance from approximately 2% to 62%, while CalErr first decreases from 0.212 to 0.178, then slightly increases to 0.190. Finally, at and , CalErr improves from 0.204 at to 0.178 at , then remains approximately stable.
These results provide a focused sensitivity analysis supporting the suggested adjustment guidelines. In practice, construction diagnostics should be considered alongside validation calibration, NLL, and predictive error when choosing adjustments. We will include these results in the appendix.
We thank the reviewer for the assessment. Below, we address the conceptual and empirical concerns and clarify the intended contribution, supporting evidence, and limitations of LENS.
W1: Theoretical Justification
Definition 3.1 specifies the admissible parameter set, whereas the algorithm specifies how configurations are constructed within that set. The tube may contain disconnected components, but defining a global feasible set does not imply exhaustive exploration. LENS starts from a trained checkpoint and explores reachable, approximately loss-preserving configurations. These roles are distinct and consistent.
Proposition 3.3 makes the empirical-fit constraint explicit where admissible configurations have loss within the prescribed relative tolerance of the reference model. This is a feasibility statement, not a calibration or posterior-recovery theorem. Its purpose is to distinguish disagreement among predictors with comparable loss from disagreement introduced through unconstrained deterioration of empirical fit. The methodological contribution is the geometry-guided ensemble construction, rather than the algebraic bound itself.
As already stated in Sections 3.3–3.4, Proposition 3.4 intentionally establishes local first-order loss preservation. Finite-step drift is handled separately through bounded correction and explicit relative-loss acceptance testing. Configurations that fail the prescribed criterion are excluded. Thus, the local proposition motivates the proposal geometry, while the acceptance procedure verifies admissibility.
The paper explicitly states that LENS is not formulated as MCMC or an approximation to a prescribed Bayesian posterior. Its theoretical statements concern construction geometry and empirical-loss admissibility; its predictive uncertainty is evaluated empirically.
W2: Empirical Validation
As discussed in our responses to Reviewers GcTq and YaSc, the experimental settings involve substantial hyperparameter sharing rather than independently optimized sampling configurations for every dataset. The damping factor and gradient tolerance are shared across regression and classification, and the correction step cap is shared between CIFAR-10 and QA. All QA tasks use the same sampling settings, while the UCI tasks share settings apart from the smaller loss tolerance on Energy.
Our additional WG-M experiment, reported in the regularization table in our response to Reviewer GcTq’s first concern, directly examines transfer to a modified construction objective by adding regularization while retaining the original sampler settings. All regularized runs collected the requested ensemble without sampler-hyperparameter re-tuning, with broadly comparable predictive performance. These results support practical reuse under the tested objective changes.
The baseline selection follows the methodological scope. SWAG and AS provide closely related parameter-space comparisons; DE evaluates diversity across independent training runs; MCD provides an established stochastic-prediction baseline; and TFB adds a recent Bayesian baseline applicable to low-rank adapters.
Importantly, the empirical evidence demonstrates predictive benefits across different architectures and data modalities. Single-checkpoint LENS improves calibration and likelihood relative to DE under CIFAR-10 corruption shift, and provides the strongest reported calibration across the shifted QA tasks. The additional regression-recalibration experiments retain clear calibration advantages on Facebook and Yacht. The Ensemble-LENS results provide further evidence that within-checkpoint exploration can complement diversity across independently trained models, supporting the empirical utility of the proposed construction.
Quantitative computational comparisons are already provided in Appendix E and Figure 4. These include construction-time comparisons against SWAG and AS on Facebook and Wine-red, alongside computational and storage complexity analysis. The empirical timings concern the regression models, rather than comprehensive timing across every benchmark.