acceptodds
Under review as a conference paper at ICLR 2027

LENS: Level-set ENSemble for post-hoc uncertainty quantification

Abstract

To exploit the nonlinear geometry of the empirical loss landscape in parameter space, we introduce Level-set ENSemble (LENS), an ensemble-based post-hoc uncertainty quantification method. Rather than imposing a parametric approximation to the posterior or restricting uncertainty exploration to a linear subspace, LENS constructs an ensemble of parameter configurations constrained to lie near a fixed loss level set. This set contains models with empirical loss comparable to the trained model but potentially different predictions. The resulting model ensemble yields uncertainty estimates derived from structured, non-linear variation in parameter space. The proposed method is architecture-agnostic and does not require retraining. Across diverse prediction tasks spanning text, image, and tabular data, LENS improves uncertainty estimation and calibration over strong post-hoc methods, while maintaining predictive accuracy. Our results establish loss level sets as a principled and practical route to exploring non-linear uncertainty structure in deep learning models.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Uncertainty Quantification, Bayesian Learning, Post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7DDHOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes "LENS", a post-hoc uncertainty quantification method that exploits the nonlinear geometry of the empirical loss landscape in parameter space. LENS constructs an ensemble of models with comparable empirical losses by exploring approximate loss-level sets through a gradient-based predictor-corrector procedure. The predictive disagreement among the collected models is then used to estimate uncertainty. The authors evaluate LENS on regression, image classification, and LoRA-adapted language model benchmarks, reporting improvements in uncertainty calibration and likelihood-based metrics while maintaining competitive predictive performance.

Critical strengths

None.

Critical weaknesses

1. Insufficient Theoretical and Conceptual Justification

The paper lacks a sufficiently rigorous justification for connecting nonlinear loss-level-set exploration to reliable uncertainty quantification. In particular, Definition 3.1 introduces a globally defined relative level-set tube that may contain disconnected components, whereas the proposed algorithm primarily explores local regions around a single trained solution. The authors do not adequately explain how incomplete exploration affects the representativeness of the resulting ensemble or its uncertainty estimates. Furthermore, Proposition 3.3 is a direct restatement of the definition, while Proposition 3.4 only establishes local first-order loss preservation under smoothness assumptions, without providing guarantees for the finite-step sampling procedure or uncertainty estimation.

2. Insufficient Empirical Validation and Methodological Reliability

The experimental evaluation does not sufficiently establish whether the reported improvements arise from the proposed nonlinear level-set geometry or from task-specific sampling choices. LENS introduces numerous hyperparameters that directly influence ensemble construction, yet their selection criteria and sensitivity are not adequately investigated. Moreover, the rationale for baseline selection is unclear, and several experimental observations remain insufficiently explained. Finally, the lack of quantitative computational cost comparisons makes it difficult to assess the practical efficiency of the proposed method.

Questions for authors

Definition 3.1 defines a global relative level-set tube that may contain multiple disconnected components, whereas LENS explores only a potentially limited subset of this tube through initialization-dependent local sampling. Given that the resulting ensemble may not adequately represent the set of plausible low-loss models, what justifies interpreting its predictive disagreement as a reliable measure of epistemic uncertainty? In particular, how do the authors ensure that unexplored low-loss regions do not substantially affect the resulting uncertainty estimates?

4 comments

We thank the reviewer for the assessment. Below, we address the conceptual and empirical concerns and clarify the intended contribution, supporting evidence, and limitations of LENS.

W1: Theoretical Justification

Definition 3.1 specifies the admissible parameter set, whereas the algorithm specifies how configurations are constructed within that set. The tube may contain disconnected components, but defining a global feasible set does not imply exhaustive exploration. LENS starts from a trained checkpoint and explores reachable, approximately loss-preserving configurations. These roles are distinct and consistent.

Proposition 3.3 makes the empirical-fit constraint explicit where admissible configurations have loss within the prescribed relative tolerance of the reference model. This is a feasibility statement, not a calibration or posterior-recovery theorem. Its purpose is to distinguish disagreement among predictors with comparable loss from disagreement introduced through unconstrained deterioration of empirical fit. The methodological contribution is the geometry-guided ensemble construction, rather than the algebraic bound itself.

As already stated in Sections 3.3–3.4, Proposition 3.4 intentionally establishes local first-order loss preservation. Finite-step drift is handled separately through bounded correction and explicit relative-loss acceptance testing. Configurations that fail the prescribed criterion are excluded. Thus, the local proposition motivates the proposal geometry, while the acceptance procedure verifies admissibility.

The paper explicitly states that LENS is not formulated as MCMC or an approximation to a prescribed Bayesian posterior. Its theoretical statements concern construction geometry and empirical-loss admissibility; its predictive uncertainty is evaluated empirically.

W2: Empirical Validation

As discussed in our responses to Reviewers GcTq and YaSc, the experimental settings involve substantial hyperparameter sharing rather than independently optimized sampling configurations for every dataset. The damping factor and gradient tolerance are shared across regression and classification, and the correction step cap is shared between CIFAR-10 and QA. All QA tasks use the same sampling settings, while the UCI tasks share settings apart from the smaller loss tolerance on Energy.

Our additional WG-M experiment, reported in the regularization table in our response to Reviewer GcTq’s first concern, directly examines transfer to a modified construction objective by adding regularization while retaining the original sampler settings. All regularized runs collected the requested ensemble without sampler-hyperparameter re-tuning, with broadly comparable predictive performance. These results support practical reuse under the tested objective changes.

The baseline selection follows the methodological scope. SWAG and AS provide closely related parameter-space comparisons; DE evaluates diversity across independent training runs; MCD provides an established stochastic-prediction baseline; and TFB adds a recent Bayesian baseline applicable to low-rank adapters.

Importantly, the empirical evidence demonstrates predictive benefits across different architectures and data modalities. Single-checkpoint LENS improves calibration and likelihood relative to DE under CIFAR-10 corruption shift, and provides the strongest reported calibration across the shifted QA tasks. The additional regression-recalibration experiments retain clear calibration advantages on Facebook and Yacht. The Ensemble-LENS results provide further evidence that within-checkpoint exploration can complement diversity across independently trained models, supporting the empirical utility of the proposed construction.

Quantitative computational comparisons are already provided in Appendix E and Figure 4. These include construction-time comparisons against SWAG and AS on Facebook and Wine-red, alongside computational and storage complexity analysis. The empirical timings concern the regression models, rather than comprehensive timing across every benchmark.

Q1: Unexplored Regions

LENS’s disagreement measures predictive variation among configurations with comparable loss that are constructed from the available checkpoint. Loss comparability constrains the ensemble; it does not, by itself, guarantee calibrated uncertainty. The justification for its usefulness comes from held-out predictive and calibration evaluations, including distribution-shift experiments.

Establishing that unexplored low-loss regions cannot alter the uncertainty estimates would require global coverage or additional assumptions beyond the scope of this study. Our contribution is an initialization-dependent ensemble construction with demonstrated empirical utility, rather than an exhaustive characterization of all plausible models. Initialization dependence and incomplete exploration are acknowledged limitations in Section 5.

Ensemble-LENS additionally investigates aggregation across independently trained checkpoints and demonstrates improvements in several shifted settings. This supports the value of combining within-checkpoint exploration with across-checkpoint diversity, without claiming that the resulting ensemble exhaustively covers the global feasible set.

Thank you for authors’ thoughtful response to my concerns. Reading comments has helped resolve several of my points, including some that stemmed from my own misunderstandings specifically regarding the hyperparameter and cost analysis for W2, as well as the exploration space mentioned in Q1. However, as my concerns regarding the theoretical justification and sensitivity analysis remain, I would like to ask a few follow-up questions to clarify these aspects.

To follow up on the theoretical justification, Proposition 3.4 assumes the function is twice continuously differentiable. Could you clarify whether this condition can hold in a structure that incorporates activation functions, and perhaps comment on how easily or with what degree of difficulty this assumption is satisfied in such a context?

Regarding sensitivity, it is significant that the hyperparameters proposed can be applied across various environments. However, this seems quite different from the sensitivity I mentioned. Realistically, it is not possible to conduct experiments across every model and setting. However, since the foundation presented in the paper is based on empirical results, there is a need for guidance on how to make adjustments when the proposed setting does not perform as expected. For instance, if the provided hyperparameters do not yield good results, how should users go about adjusting them? Most users cannot easily predict the impact of these hyperparameters Therefore, I requested that practical guidelines be provided based on the interactions or behaviors of the parameters. This is all the more important given that the response to W1 mentions that uncertainty is evaluated empirically.

Thank you for the follow-up and for clarifying the remaining concerns. We address the smoothness assumption and practical hyperparameter adjustment separately below.

Smoothness assumption: Proposition 3.4 requires local smoothness of the construction loss near the current parameters, not global smoothness throughout parameter space. Smooth activations such as SiLU are compatible with this assumption. For ReLU-based networks, the justification is local. Away from activation-switching boundaries, the activation pattern remains unchanged within a sufficiently small neighborhood, where each ReLU acts as either the identity or zero. Thus, ReLU does not itself prevent local smoothness. The potential restriction, however, is that this neighborhood can become small near switching boundaries; thus we do not claim that the proposition applies at those boundaries. We will further clarify this in the paper. Finally, we want to clarify that for proposals, loss admissibility is checked separately through explicit loss evaluation and acceptance testing, without assuming that every proposal remains within a smooth neighborhood.

Practical hyperparameter guidance: As acknowledged in limitations in Section 5, in this work we do not provide an automatic hyperparameter-selection rule. However, based on the roles and interactions of the construction parameters the following practical guidance for hyperparameter adjustment can be considered.

  • Tangent step size αmax⁡\alpha_{\max} controls the extent of exploration. Very small steps can preserve predictions close to the deterministic reference but provide insufficient predictive diversity to improve calibration. Increasing the step range can expose additional disagreement. Excessively large steps, however, can increase loss drift, reduce acceptance, and require more correction. We therefore recommend increasing the step range gradually when predictions remain nearly identical, and reducing it when collection becomes difficult.

  • Loss tolerance ε\varepsilon controls the permitted departure from the reference construction loss. A tighter tolerance enforces closer empirical fit but can make collection difficult through frequent rejection. A looser tolerance facilitates collection but admits greater loss deviations, potentially compromising predictive performance. It should therefore balance empirical-fit preservation and collection feasibility, rather than being relaxed solely to increase acceptance. We do not prescribe a fixed acceptance-rate threshold. Instead, a short pilot can assess whether the requested ensemble can be collected within the available computational budget. For a target of KK configurations and a budget of BB proposals, K/BK/B provides a basic collection-feasibility benchmark, not a calibration-optimal target. Acceptance should therefore guide collection feasibility with respect to the task at hand alongside validation performance, rather than be maximized in isolation.

  • Correction depth MM determines how much correction work each proposal receives. Too few iterations can reject proposals that additional correction could recover. Increasing the budget is useful when residuals are decreasing but beyond a sufficient correction depth, further increases would not provide additional benefits.

To examine these behaviors empirically, we evaluated 48 Yacht configurations over three seeds, varying these three parameters while holding the remaining settings fixed.

With ε=10−3\varepsilon=10^{-3} and M=30M=30, increasing αmax⁡\alpha_{\max} from 0.1 to 5 reduces CalErr from 0.267 to 0.178, although NRMSE increases from 0.087 to 0.103. Increasing αmax⁡\alpha_{\max} further to 10 worsens CalErr to 0.223 and lowers proposal acceptance.

With αmax⁡=5\alpha_{\max}=5 and M=30M=30, relaxing ε\varepsilon from 10−410^{-4} through 10−310^{-3} to 10−210^{-2} increases acceptance from approximately 2% to 62%, while CalErr first decreases from 0.212 to 0.178, then slightly increases to 0.190. Finally, at ε=10−3\varepsilon=10^{-3} and αmax⁡=5\alpha_{\max}=5, CalErr improves from 0.204 at M=10M=10 to 0.178 at M=30M=30, then remains approximately stable.

These results provide a focused sensitivity analysis supporting the suggested adjustment guidelines. In practice, construction diagnostics should be considered alongside validation calibration, NLL, and predictive error when choosing adjustments. We will include these results in the appendix.

Reviewer_YaScOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Level-set ENSemble (LENS), which constructs ensembles by constructing diverse parameters on approximately the same loss-level set. The idea could be interesting. However, the empirical evaluation does not meet the bar of acceptance for ICLR. The problems in the evaluation might be fixable during the rebuttal.

The paper empirically only compares different raw ensembling techniques without temperature scaling. The considered evaluation metrics are NLL, ECE, Brier, and Accuracy. Accuracy is not the focus of this, and LENS does not perform particularly strongly in terms of accuracy. NLL, ECE, and Brier are highly sensitive to the temperature of the predictive distribution. Therefore, it is very common, both in practice and in the literature, to tune a global calibration constant on a validation dataset to globally rescale the confidence. For classification, this is called temperature-scaling (TS) [e.g., Guo et al., 2017] or pool-then-calibrate (PTC), specifically for ensembles. For regression, rescaling the uncertainty is also very common [Levi et al. (2022), Kuleshov et al., 2018]. However, no form of calibration is ever applied in this paper. I cannot vote to accept this paper without a comparison of these methods where calibration is applied to each of the competitors (including LENS). I explain this in more detail in the Weakness section.

Critical strengths

The idea sounds natural, plausible, simple, and interesting, maybe promising. Is this idea completely new, or have other people tried similar things?

It seems plausible to me that this method has significant advantages compared to SWAG or AS. However, I don’t see any intuitive argument or any convincing empirical evidence that LENS has significant advantages over deep ensembles (DE).

Critical weaknesses

W1: The lack of TS: The most critical weakness of this paper is the lack of calibration techniques such as temperature-scaling (TS) for classification [e.g., Guo et al., 2017] or rescaling of the intervals for regression [Levi et al. (2022), Kuleshov et al., 2018]. For example, Fig 3(b) shows a consistent one-sided deviation from the diagonal. In your training regime, Figure 3(b) tells us that all the methods that you compare are consistently under-confident (since the empirical coverage is consistently above the predicted coverage, if I’m reading Fig 3(b) correctly). The problem of one-sided deviations from the diagonal is basically solved in the literature via calibration techniques around the idea of rescaling: If predictions of a model are on average too under-confident, you can simply scale down the confidence by a global factor per dataset, and in the other case, if predictions of a model are on average too over-confident, you can simply scale down the confidence by a global factor per dataset. Fig 3(b) suggests that rescaling would substantially improve all methods in metrics such as ECE, NLL, and Brier score. This makes it impossible to guess if the ranking among methods would persist after rescaling. In general, rescaling-based calibration methods are well established, and some form of it should be part of basically any uncertainty quantification (UQ) pipeline. Also, explicitly for ensembles, it is well known that posthoch-calibration techniques such as TS are crucial, pool-then-calibrate (PTC) [e.g., https://arxiv.org/abs/2007.08792].

W2: The paper uses different hyperparameters for LENS depending on the dataset? How were these HP selected? Were HPs selected based on their performance on the test set? If only the HP that performed best on the test set is reported, there would be a selection bias. These HPs should also be selected on the validation dataset.

W3: I think the sentence “Unlike independently trained ensembles, LENS does not require training multiple independent models from scratch.” (line 470-471) is less strong than it might seem. In practice, people cannot directly guess the best hyperparameters (HP) for their problem, so they have to train an ensemble of different models with different HP anyways. Therefore, such mixed-HP ensembles can be obtained at basically zero additional training cost. At inference time, LENS requires 5-20 forward passes, while your implementation of DE (“Ens.”) only uses 3 ensemble members. Therefore, DE has much lower computational inference costs than LENS. Hence, I wouldn’t argue that LENS is computationally cheaper than DE. For example, greedy ensembles [e.g., https://dl.acm.org/doi/10.1145/1015330.1015432 and https://ieeexplore.ieee.org/document/4053111] are very established papers from 20 years ago on weighted mixed-HP ensembles (they can select the same ensemble member multiple times, effectively leading to weighted ensembles). My guess is that such methods are simpler, easier to implement, computationally cheaper, and perform better after rescaling the magnitude of the uncertainty on the validation dataset.

Questions for authors

What exactly are you plotting in Figure 3(b)? Are x-axis and y-axis coverage or interval width? How do you bin the data? How do you define the calibration error in Fig 3(a) for regression? In the appendix, you only define ECE for classification.

Do you define somewhere what you mean by “post-hoc uncertainty”? I see your point that LENS is to some extent “post-hoc”, as it can be applied to an already trained model, without requiring training it from scratch; however, one could also argue that it is not fully post-hoc, as it requires access to the full training dataset to run LENS, and the LENS training also sounds computationally quite expensive.

Do you have any intuition why LENS should be better than DE? For DE, you also have an ensemble of modes of approximately similar loss, often more diverse than LENS, which could hardly move from one local loss basin to another. DE is easier to implement and extremely modular (e.g., you can combine models with different HP and different architectures).

In Fig 2: DE is missing. In Figure 2(b), it would be interesting to see a version of this figure (in the appendix) where TS is applied to each of these methods (i.e., PTC in the case of DE).

For regression, each individual ensemble member outputs (μ(x),σ(x))(\mu(x),\sigma(x)) of a Gaussian trained via NLL (as in [Nix & Weigend, 1994]) to also include aleatoric uncertainty?

For each method in Figure 3 you should apply some form of rescaling-based calibration and report these results either instead of or additionally to the existing results.

In Table 1&2, you should TS to each method (i.e., PTC in the case of DE) and report these results either instead of or additionally to the existing results.

Appendix H and Appendix I are almost identical.

Figure 5 in Appendix H/I suggests that the epistemic uncertainty estimation is quite unstable in the number of samples. Also, the results in the main paper suggest that it is a priori not clear for which dataset which number of samples should be selected. Maybe a method like JUCAL [https://arxiv.org/abs/2602.20153] applied on top of your method could stabilize this.

How do you train LEns_20 vs Lens 5. Do you stop LENS_5 after you obtain 5 parameter vectors in T\varepislon\mathcal{T}_{\varepislon} or do you train an LENS 20 and select the 5 most diverse parameter vectors within the 20?

Why do you report AUROC only in Table 4? AUROC (and Accuracy) are the only metrics in your paper that wouldn’t be affected by TS. Table 4, give a little bit of hope that Ens.-LENS could improve over LENS, but this one experiment is far from significant. In Table 4, Ens outperforms Ens.-LENS in terms of AUROC.

If LENS outperforms its competitors after you apply some form of post-hoc calibration on top of each method (such as TS), I can change my vote to accept. Without a very strong response to W1, I cannot accept this paper. If there is no response or only a poor response to W1, I also consider lowering my score.

10 comments

We thank the reviewer for the detailed feedback and for recognizing the simplicity and potential of LENS. We appreciate the suggestions for strengthening the evaluation and provide additional experimental results and methodological clarifications below.

General concern/question 1: related similar ideas
As discussed in Section 2, prior work has studied level sets of network outputs for analysis, loss-level sets for optimization, and low-loss connectivity between trained solutions. LENS’s contribution is to use parameter-space loss-level sets as a geometric basis for post-hoc UQ, constructing predictive ensembles that expose disagreement among comparable-loss configurations reachable from a trained checkpoint. To our knowledge, LENS introduces a novel use of parameter-space loss-level sets for post-hoc ensemble-based UQ.

General concern/question 2: advantages relative to Deep ensembles

LENS and deep ensembles obtain predictive diversity in complementary ways. Deep ensembles (DE) exploit variation across independent training runs, whereas LENS explores alternative parameter configurations with comparable empirical loss starting from a single trained checkpoint. Its practical benefit is therefore the ability to construct a predictive ensemble from an existing model without additional independent training or fine-tuning runs. At a conceptual level, LENS shares the motivation of loss-informed importance sampling by using empirical fit to guide parameter-space exploration. Its loss constraint focuses exploration on alternative predictors with comparable fit, whose disagreement on unseen inputs can expose predictive variability that the empirical objective leaves unresolved. Such exploration may complement variation across independently trained DE members. DE can provide a strong source of predictive diversity, and we do not claim universal superiority over it. Rather, we evaluate whether LENS can provide competitive or improved uncertainty estimates from a single checkpoint. The empirical results below support its benefits in the studied settings.

Empirically, this single-checkpoint based ensemble construction can improve uncertainty quality relative to deep ensembles. For example, although DE performs better on in-distribution CIFAR-10, on CIFAR-10-C, LENS reduces NLL from 1.69 to 1.58 and ECE from 19.94% to 16.00%. Similar gains appear in QA evaluations, where LENS improves calibration and often NLL relative to DE in both ID and OOD settings. Furthermore, combining level-set exploration with independently trained models yields further improvements in uncertainty metrics as shown by Ensemble-LENS in the appendix. These results support the utility of single-checkpoint exploration for ensemble generation and suggest that it can complement diversity across training runs.

W1- The lack of TS.

Following your suggestion, we have provided temperature-scaled classification comparisons for Det., DE, MCD, SWAG, AS, TFB, and LENS in Appendix Table 9. Temperature scaling (TS) recalibrates an existing predictive distribution, whereas LENS constructs loss-comparable predictors from a single fine-tuned checkpoint. Importantly, recalibration does not eliminate LENS’s calibration advantages. LENS achieves the lowest ECE in 7 of 10 QA settings and is best or second-best in 9 of 10. It is also best or second-best in NLL in 7 of 10. DE achieves the strongest calibrated NLL, but requires three independent fine-tuning runs, whereas LENS uses one.

The CIFAR-10-C results illustrate that ID-fitted recalibration does not uniformly improve performance under distribution shift. Native LENS achieves 16.03% ECE and 1.580 NLL, compared with 19.16% and 1.654 for SWAG after TS. Recalibrating LENS gives 19.13% ECE and 1.680 NLL. Thus, LENS remains competitive after independent calibration, while its native predictions outperform every evaluated temperature-scaled baseline on CIFAR-10-C in ECE, NLL, and Brier score. This is consistent with Ovadia et al. [1], who observed that calibration on the validation distribution does not guarantee calibration under distribution shift.

Recalibration is also not uniformly beneficial in regression (please see the response to Q6). On Wine-red, rescaling worsens LENS’s already well-calibrated native predictions. Furthermore, native LENS has lower mean interval-calibration error on Facebook, Wine-red, and Yacht than every competing baseline after rescaling.

Finally, standard TS optimizes validation NLL rather than ECE [2]. Accordingly, the fitted adjustment does not necessarily improve test calibration error or remain beneficial under distribution shift. Overall, recalibration can help, but does not remove the calibration advantages of LENS’s native uncertainty estimates.

W2 - Hyper-parameter selection
As noted in our response to Reviewer GcTq, most LENS hyperparameters are shared across all datasets within each of our three experimental settings, and some are shared across settings (e.g., the damping factor and gradient tolerance across regression and classification, and the correction step cap across CIFAR-10 and QA). All LENS hyperparameters were selected on the validation split; the test set was never used for selection. Among the candidates, we chose the configuration that performed well jointly on validation accuracy, NLL and ECE. To further limit selection bias, we report all QA sample budgets rather than only the best one. Moreover, the experiment added for Reviewer GcTq shows that the same hyperparameters transfer to a modified (regularized) loss, whose geometry differs from the one used for selection, without re-tuning, while LENS performance remains comparable. We clarify these details in the Appendix J.

W3 - Construction/inference cost, and Ensemble selection

We distinguish ensemble-construction cost from inference cost. You are correct that our three-member DE baseline requires fewer forward evaluations than LENS with 20 members. Our practical advantage is in constructing additional predictors from an existing checkpoint without independently training additional models, not reducing the inference cost irrespective of the ensemble size.
The cited greedy ensemble-selection methods address a different starting setting where they select and weight predictors from an existing model library. Such a library can be re-used when available, but its availability is not guaranteed in the single-checkpoint post-hoc setting considered here. Repeatedly selecting an existing model changes its ensemble weight; it does not create additional distinct predictors. LENS instead constructs alternative, comparable-loss parameter configurations from the available checkpoint.
Particularly, LENS requires a trained checkpoint and task relevant labeled data to construct the ensemble. The empirical loss on these data defines the exploration geometry. Its practical contribution is therefore ensemble construction from an existing checkpoint, rather than selection from a pre-existing collection of trained models.

Q1 - Regression calibration metrics

Figure 3(b) plots nominal coverage on the x-axis and empirical coverage on the y-axis, rather than interval width. Coverage-based calibration plots are established diagnostics for probabilistic regression [3]. Specifically, for each nominal level (c∈C={0.1,0.2,…,0.9}c\in \mathcal{C} = \lbrace0.1,0.2,\ldots ,0.9\rbrace), we construct the central Gaussian predictive interval

\qquad z_c=\Phi^{-1}\left(\frac{1+c}{2}\right).$$ where $\mu_i$ and $\sigma_i^2$ denote the predictive mean and variance, and $\Phi$ is the standard normal CDF. The corresponding empirical coverage is $\widehat C(c)=\frac{1}{n}\sum_{i=1}^{n}\mathbf{1}\lbrace y_i\in I_i(c)\rbrace$. Thus, each plotted point compares the nominal coverage with the fraction of test targets contained in their corresponding intervals. No binning of examples is involved. The regression calibration error reported in Figure 3(a) is the mean absolute coverage discrepancy: $\mathrm{CalErr}=\frac{1}{|\mathcal C|}\sum_{c\in\mathcal C}|\widehat C(c)-c|$. This follows the interval-calibration formulation of \[4\], evaluated using our specified nine-level grid. The metric is computed separately for each run and then summarized across runs. Unlike classification ECE, it measures predictive-interval coverage discrepancies rather than confidence-bin accuracy. We put these in the Appendix F. **Q2 \- post-hoc definition** By “post-hoc,” we mean that LENS is applied to an already-trained checkpoint and constructs an ensemble without training additional independent models. Its predictor–corrector updates explore alternative parameter configurations near a prescribed reference loss, rather than seeking a new loss minimum. LENS requires task-relevant labeled data to construct the ensemble. Construction involves loss and gradient evaluations, whose computational cost is discussed in the Appendix E. Similarly, SWAG, used here as a post-hoc approach, constructs its weight-distribution approximation using a continued SGD trajectory initialized from pretrained weights. Consistent with such approaches in previous literature, “post-hoc” here describes application to an already-trained model, not an absence of additional computation.

Q3 - DE versus LENS

LENS targets predictive variation among comparable-loss configurations reachable from an existing trained checkpoint. Such configurations can produce different predictions without requiring exploration of distinct loss basins. As discussed in our response to your earlier concern (please see the response to General concern/question 2), this loss-guided exploration may complement variation across independent training runs, providing a possible explanation for the observed calibration improvements. Our Ensemble-LENS results support this interpretation where applying level-set exploration to independently trained checkpoints further improves uncertainty metrics on CIFAR-10-C and the reported OOD QA benchmarks.

Deep ensembles are a straightforward baseline. LENS addresses the practical setting in which a single trained checkpoint is available and additional ensemble members are generated through comparable loss-level set exploration. In contrast, constructing a DE requires additional independent training or fine-tuning runs. LENS thus provides an alternative way for constructing an ensemble from an existing checkpoint.

Our results also highlight settings where a single checkpoint based LENS improves uncertainty estimates and calibration over the three-member DE. These findings support LENS as a useful alternative in those settings, rather than general superiority over DE.

Q4 - Fig 2 visualization

Figure 2 is intended as a controlled illustration of how different single-checkpoint ensemble constructions explore parameter space and induce predictive disagreement. LENS, SWAG, and AS all start from the same trained checkpoint, allowing their respective exploration geometries to be examined. In contrast, DE introduces variation across independently trained models and therefore falls outside this particular comparison; its predictive performance is evaluated in our quantitative experiments.

Temperature scaling addresses a complementary stage of the uncertainty-estimation pipeline. It recalibrates predictive probabilities rather than constructing alternative predictors or changing the sampled parameter locations. In particular, Figure 2(b) visualizes disagreement among individual predictors, whereas pool-then-calibrate operates on their averaged predictive distribution. Following the reviewer’s suggestion, we assess the effect of recalibration through the additional TS comparisons, while retaining Figure 2 as an illustration of ensemble-construction geometry.

Q5 - Regression output

Our regression networks output scalar predictions and are trained using MSE. For uncertainty evaluation, we construct a Gaussian predictive distribution whose variance combines ensemble prediction variance with a constant observation-noise term consistent with the predictive variance decomposition in [5]. In our implementation, the constant term is estimated from the deterministic reference model’s validation residual MSE and shared across its associated post-hoc methods, rather than learned through an input-dependent variance head. We put these details in the Appendix F.

Q6 - Recalibrated regression results

Following your suggestion, we applied validation-NLL-fitted variance scaling to every method in Figure 3, preserving predictive means while rescaling total predictive variance. The recalibrated calibration errors are:

MethodFacebookWine-redEnergyYacht
Det.0.4460.0160.0470.268
MCD0.4340.0200.0420.260
Ens.0.4450.0150.0360.262
SWAG0.4480.0170.0450.267
AS0.4460.0160.0780.261
LENS0.3900.0210.0370.082

LENS retains clear calibration advantages after recalibration, particularly on Facebook and Yacht. On Yacht, its calibration error is 0.082 versus approximately 0.26 for the competing methods. On Energy, LENS remains comparable to Ens.

On Wine-red, LENS is already well calibrated before rescaling; recalibration increases its calibration error from 0.014 to 0.021, illustrating that validation-fitted rescaling does not necessarily improve test calibration. Notably, before rescaling, LENS already has lower mean calibration error on Facebook, Wine-red, and Yacht than every competing baseline after rescaling. Consistent with our discussion in the paper, NLL remains broadly comparable across methods, although Ens. achieves lower NLL on three datasets.

These results show that variance rescaling improves calibration for many methods, but does not eliminate LENS’s calibration advantages.

Reviewer_GcTqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a loss-geometry-guided uncertainty quantification framework that characterizes model uncertainty by exploring parameter variations along a level set of the empirical loss landscape. Starting from a trained model with parameters θ\theta and a target loss value, the proposed method generates parameter perturbations through gradient-based updates in directions tangent to the loss level set, thereby inducing model diversity while approximately preserving the empirical loss. The proposed framework is evaluated across classification, regression, and large language model (LLM) question-answering tasks using appropriate metrics to assess predictive performance, uncertainty quantification, and calibration. The experimental results demonstrate the effectiveness of the proposed approach across diverse tasks.

Critical strengths

The paper is well-motivated, clearly written, and methodologically rigorous. It introduces a novel approach to uncertainty quantification by leveraging a Taylor expansion of the loss function and further develops a practical strategy to exploit the resulting loss geometry to induce model variations. The extensive experimental evaluation, incorporating both calibration metrics and out-of-distribution (OOD) detection, provides strong empirical support for the effectiveness of the proposed method and reinforces the paper's central claims.

Critical weaknesses

The proposed method relies heavily on several hyperparameters, including τg\tau_g, the step cap, and the damping factor. The authors also acknowledge that the step cap and damping factor require task- or domain-specific tuning. Since the proposed uncertainty quantification (UQ) framework explicitly depends on the geometry of the loss landscape, my primary concern is whether these hyperparameters must be retuned whenever a new loss function or additional loss component is introduced. In contrast, established UQ approaches such as Deep Ensembles and MC Dropout generally require less task-specific tuning of the uncertainty estimation procedure itself. If adapting the proposed method to a new task or loss function requires repeated hyperparameter optimization, this could substantially limit its practicality and generalizability. I encourage the authors to investigate the sensitivity of the proposed method to these hyperparameters and clarify whether a single configuration can generalize across different loss functions, datasets, and domains.

Questions for authors

Would introducing an additional regularization term or a contrastive loss component substantially alter the optimal hyperparameter settings of the proposed UQ method? Given its dependence on the loss landscape, it is unclear whether such modifications would necessitate extensive hyperparameter retuning for each domain-specific application.

2 comments

We thank the reviewer for the thoughtful review and positive assessment of our motivation, methodology, and empirical evaluation. We appreciate the questions regarding hyperparameter reuse across objectives and address them below through clarifications and an additional regularization experiment.

W1: Hyperparameter Reuse Across Tasks and Objectives

LENS uses task-specific loss to define its exploration geometry, but our experiments did not require independently tuning every hyperparameter for each dataset or objective. The damping factor and gradient tolerance were held fixed across MSE-based regression and cross-entropy-based classification, and CIFAR-10 and QA use the same correction step cap. All six QA benchmarks used the same sampling settings. Similarly, the UCI benchmarks shared one configuration, except for a smaller loss tolerance on Energy. Thus, the reported results were obtained with substantial parameter sharing across datasets, architectures, and objectives, rather than individually optimized sampling configurations for every benchmark.
Overall, these results support the practical reuse of core LENS settings across different datasets, architectures, and objectives, without independently tuning every hyperparameter for each benchmark.

To directly examine whether adding a loss component requires re-tuning, we evaluated LENS on WG-M using

Lλ(θ)=L(θ)+λ∣∣θ∣∣22L_\lambda(\theta)=L(\theta)+\lambda||\theta||_2^2,

where θ\theta denotes the trainable LoRA parameters. For each checkpoint, λ\lambda was fixed so that the initial penalty equaled either 1% or 10% of the construction loss. We retained the same checkpoints, construction data, and LENS hyperparameters, including the damping factor, gradient tolerance, correction step cap, and sample budget K=20K=20. The target level was recomputed using the regularized objective. Results below report mean ± standard deviation over three seeds:

Construction objectiveAccuracy (%)ECE (%)NLL
LENS79.01 ± 0.398.47 ± 0.080.54 ± 0.02
LENS + 1% penalty78.90 ± 0.808.79 ± 1.560.533 ± 0.016
LENS + 10% penalty78.90 ± 0.459.82 ± 1.040.554 ± 0.037

All regularized runs collected the requested 20 configurations within the unchanged proposal budget. Although stronger regularization increased correction work, no sampler-hyperparameter retuning was required. These results support practical reuse of the original settings under the tested objective changes, without implying that those settings remain optimal for arbitrary losses.

Q1: Effect of Additional Regularization
To address this, in the answer to the previous concern, we evaluated LENS with quadratic regularization, which has a standard Gaussian-prior interpretation in likelihood-based models. Here, we use this penalty as a controlled modification of the construction objective. LENS incorporates the resulting geometry through the modified loss and gradients while retaining its original predictor–corrector settings. As shown above, both tested regularization strengths allowed collection of the requested ensembles without re-tuning. These results further support practical reuse of the construction settings.

Thank you again for your thoughtful review. We would appreciate your feedback on whether our clarifications and additional WG-M regularization experiment address your concerns about hyperparameter reuse under changes to the construction objective. In particular, both tested regularization strengths allowed collection of the requested ensembles without retuning the sampler settings. Please let us know if any remaining points would benefit from further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.