acceptodds
Under review as a conference paper at ICLR 2027

HIFI: A Highly Efficient Framework for Statistical Inference with Foundation Models

Abstract

Foundation models provide powerful representations for unstructured data, such as text and images, and parameter-efficient fine-tuning (PEFT) makes their adaptation to downstream tasks increasingly practical. For neural estimators that lack a tractable sampling distribution, resampling-based procedures offer a flexible route to statistical inference, but they often require hundreds of model fits, making them computationally prohibitive at foundation-model scale. We introduce HIFI, a two-stage fine-tuning framework for statistical inference on parameters such as conditional means. Stage 1 adapts a foundation model once on a larger sample, capturing the complex structure of the data and controlling estimation bias through a single fit. Stage 2 applies a resampling inference procedure to a smaller independent sample, concurrently fitting multiple lightweight PEFT estimators to the residual to quantify estimation variability without relearning the full task. HIFI accommodates a broad class of resampling procedures, including ensemble subsampling and the nonparametric bootstrap. Across vision and text experiments, HIFI scales to pretrained models ranging from tens of millions to more than ten billion parameters while providing accurate point estimates and well-calibrated confidence intervals across diverse architectures, and reduces training compute by up to 91.9% relative to direct resampling. An application to gene-expression inference illustrates the practical value of the framework for uncertainty-aware scientific inference.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
statistical inference, resampling, parameter-efficient fine-tuning, foundation models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9ZhjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to address the high computational cost of conventional resampling methods, which require training a model hundreds of times to estimate statistical confidence intervals in foundation models.

To this end, the authors propose a two-stage framework called HIFI. In Stage 1, a base model is trained once on a large dataset to reduce estimation bias. In Stage 2, resampling is applied to a smaller dataset to train multiple lightweight LoRA models that fit residuals and estimate uncertainty.

In experiments across various vision and text models, HIFI achieved lower estimation bias and coverage closer to target confidence levels compared to existing methods, while significantly reducing training compute relative to direct resampling. Furthermore, the authors demonstrate that HIFI can provide confidence intervals while preserving predictive performance in a real-world gene expression prediction task.

Critical strengths
  1. Demonstrates high computational efficiency and stable statistical inference performance. While reducing training compute by 79.5–91.9% compared to conventional direct resampling, it achieves low estimation bias and coverage close to target confidence levels across most experiments. This is significant as it enhances the practical applicability of statistical inference in large-scale foundation models.
  2. Validates applicability across diverse models and real-world tasks. The method was evaluated on six pretrained vision and text models, demonstrating the ability to provide confidence intervals while maintaining predictive performance comparable to existing baselines in a real-world gene expression prediction task.
Critical weaknesses
  1. The independent contribution of residual learning is unclear. The paper does not compare against a baseline that shares the same Stage 1 backbone but directly predicts the ground truth instead of the residual in Stage 2. Consequently, it is difficult to distinguish whether performance improvements stem from residual learning itself or simply from leveraging a sufficiently trained Stage 1 backbone.
  2. Insufficient validation of Stage 2's residual correction capability. HIFI relies on the assumption that a small PEFT model can adequately correct Stage 1 estimation errors. However, the authors do not analyze whether confidence interval coverage is maintained when Stage 1 bias is large or when Stage 2 expressiveness is limited. This issue is directly tied to the reliability of statistical inference.
Questions for authors
  1. What results are observed if direct prediction is performed instead of residual learning in Stage 2? Comparing MAB, MAE, Coverage, and CI Length under the same Stage 1 backbone and PEFT configuration would help clarify the actual contribution of the proposed residual learning approach.
  2. What is the experimental evidence supporting that resampling only Stage 2 can sufficiently correct the estimation bias of Stage 1? Specifically, I would like to know how the residual estimation error and final coverage vary when Stage 1 has a large estimation error or when Stage 2 model capacity is constrained.
6 comments

We thank the reviewer for this suggestion, which helped strengthen the manuscript. In our original experiments, we did not extensively tune the hyperparameters, as HIFI appeared relatively robust to these choices. We now provide additional ablations below.

1. Residual PEFT vs directly learning the target.

We initially tried fitting (Y) directly in Stage 2 using the adapted Stage-1 backbone, but found that it produced conservative coverage and wider confidence intervals, motivating the residual formulation.

We therefore added a Direct-Y ablation using the same settings as in the paper. Across all settings, HIFI achieves lower MAB, shorter CIs, and coverage closer to the nominal 95% level.

BackboneMethodMAB (HIFI/Direct-Y)MAE (HIFI/Direct-Y)95% coverage (HIFI/Direct-Y)95% CI length (HIFI/Direct-Y)
ViT-B/32ESM0.018/0.0230.041/0.0480.937/0.9990.282/0.492
ViT-B/32Bootstrap0.017/0.0240.047/0.0460.938/0.9820.231/0.283
DINOv2-B/14ESM0.011/0.0170.044/0.0430.955/1.0000.319/0.906
DINOv2-B/14Bootstrap0.011/0.0160.046/0.0420.953/1.0000.243/0.541
Swin-BESM0.013/0.0180.035/0.0390.951/1.0000.236/0.442
Swin-BBootstrap0.013/0.0200.042/0.0380.943/0.9930.215/0.268
Qwen3-Embedding-8BESM0.014/0.0230.060/0.0610.928/1.0000.374/0.605
Qwen3-Embedding-8BBootstrap0.015/0.0210.064/0.0660.937/0.9880.316/0.403

2. Stage-1 estimation quality vs Stage-2 model capacity

We use the paper's ViT-B/32 HIFI-Bootstrap setup and evaluate each configuration over 100 replications. For Stage 1, we consider two checkpoints: a one-epoch checkpoint and the best checkpoint selected by the lowest Stage-1 validation MSE. We examine two Stage-2 design choices: the LoRA rank and the number of LoRA-adapted blocks, while keeping all other settings fixed within each ablation.

(a) Robustness to LoRA rank.

We vary the LoRA rank from 8 to 128, with adapters applied to the last two Transformer blocks.

  • Stage 2 corrects Stage-1 estimation error. With the one-epoch Stage-1 checkpoint, MAB decreases from 0.03264 to 0.02570–0.02591 after Stage 2.
  • Better Stage-1 estimation improves statistical inference. Across all LoRA ranks, using the best Stage-1 checkpoint yields lower MAB and shorter confidence intervals, with coverage closer to the nominal 95% level than the one-epoch checkpoint.
  • Robustness to Stage-2 capacity. Across both Stage-1 checkpoints, coverage remains stable as the LoRA rank increases from 8 to 128, while interval lengths change only slightly.
  • Computation did not increase with rank in this experiment. Higher ranks selected earlier checkpoints on average, with the validation-selected epoch decreasing from 12.21 at rank 8 to 6.69 at rank 128.
MetricStage 1 checkpoint (MAB)Rank 8 (α = 16)Rank 16 (α = 32)Rank 32 (α = 64)Rank 128 (α = 256)
MABEpoch 1 (0.03264)0.025700.025750.025820.02591
Best (0.01796)0.017560.017580.017600.01762
Cov.95 (%)Epoch 192.7992.9893.1694.00
Best93.5593.7793.9694.75
CI lengthEpoch 10.25110.25180.25170.2518
Best0.23250.23230.23230.2309
Wall Time (min/rep)Epoch 115.914.413.212.2
Best14.813.312.111.4

(b) Robustness to the number of LoRA-adapted blocks.

We vary Stage-2 capacity from head-only fitting to LoRA adaptation in the last 2 or 6 Transformer blocks, with rank fixed at 8 and alpha 16. Head-only fitting trains a newly initialized two-layer MLP head with a 64-unit hidden layer and scalar output.

  • Stage 2 corrects Stage-1 estimation error. With the one-epoch Stage-1 model, MAB decreases from 0.03264 to 0.02561.
  • Better Stage-1 estimation improves statistical inference. Even with head-only fitting, MAB decreases from 0.02615 to 0.01792 and coverage increases from 90.75% to 92.52%.
  • Robustness to Stage-2 capacity. HIFI remains effective even with head-only fitting. Additional capacity improves coverage from 90.75% to 93.24% and from 92.52% to 93.81% for the one-epoch and best checkpoints, respectively.
  • Computation–calibration trade-off. Increasing Stage-2 capacity improves coverage by 1.3–2.5 percentage points, but increases wall time from approximately 4 minutes to 25–32 minutes.
MetricStage 1 CheckpointHead OnlyHead + Last 2 LoRAHead + Last 6 LoRA
MABEpoch 1 (0.03264)0.026150.025700.02561
Best (0.01796)0.017920.017560.01736
95% Coverage (%)Epoch 190.7592.7993.24
Best92.5293.5593.81
95% CI LengthEpoch 10.22220.25110.2559
Best0.21320.23250.2310
Wall Time (min/rep)Epoch 14.2815.8925.26
Best4.2714.7931.95

Robustness of HIFI. For comparison, coverage across backbones ranges from 88.5%–96.5% for HulC, 59.8%–99.8% for Bayesian-PEFT, and 82.4%–100.0% for Laplace-LoRA. Detailed Laplace-LoRA results are provided in our response to Reviewer EktL.

AuthorsOpenReview ↗

Manuscript Update

Thank you for your comments on residual fitting and the roles of Stage-1 estimation quality and Stage-2 model capacity. We have incorporated the corresponding analyses into the revised manuscript:

  1. Residual PEFT vs directly learning the target: Added the Direct-Y comparison to Ablation Study 2 and Figure 3, comparing HIFI’s residual formulation with directly fitting the response in Stage 2 using the Stage-1-adapted backbone.
  2. Stage-1 quality and Stage-2 capacity: Added Ablation Study 3, examining Stage-1 checkpoint quality, LoRA rank, and the number of adapted blocks.

We hope these revisions address your questions. We would be grateful if you could reconsider our contribution in light of our previous clarifications and the revised manuscript. We would be happy to address any remaining concerns.

Thank you for the additional experiments and detailed response. The Direct-Y comparison clearly confirms the necessity of residual learning, so my first concern has been adequately resolved.

However, what I aimed to verify in my second question is not simply whether Stage 2 can reduce the average estimation error, but whether residual correction in Stage 2 alone can maintain the statistical reliability of the confidence intervals even when Stage 1's estimation error is large.

While the newly added checkpoint and LoRA capacity ablations are useful, I would like to examine the following aspects more directly:

  1. Analysis stratified by the magnitude of Stage 1 estimation error: Across your existing repeated runs, could you separate cases where Stage 1's estimation error is small versus large, and analyze how Stage 2's residual estimation error and final coverage differ between them?
  2. Relationship between Stage 2's correction capability and coverage: In the additional experiments, coverage tended to decrease when Stage 2's capacity was constrained. It would be helpful to analyze whether this drop stems from insufficient residual correction or from differences in confidence interval length.

Such an analysis would be important for clarifying the conditions under which a design that resamples only Stage 2 without resampling Stage 1 can be considered reliable.

Reviewer_EktLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper combines a relatively large-scale task adaptation with multiple lightweight residual resampling steps, reducing the computational cost of repeated direct adaptation. It also achieves good interval results in simulation experiments.

Critical strengths
  1. Broad model coverage. The paper evaluates HIFI on three vision backbones and three text backbones with different architectures, with the text models ranging from approximately 7.6B to 10.7B parameters. This provides relatively sufficient experimental evidence for the applicability of the method across different model architectures.

  2. Computational efficiency is quantitatively validated. In addition to interval coverage and length, the paper reports training EFLOPs and actual runtime. Under comparisons with the same total data size, HIFI reduces training computation by 79.5%–91.9% compared to direct resampling, which provides good support for its efficiency claim.

Critical weaknesses

I think the main issue with this paper is that the novelty is not strong enough in my view. The idea of first adapting a large language model and then performing UQ at a low cost is not uncommon (e.g., Laplace-LoRA, ICLR 2024; Multi-head committees enable direct uncertainty prediction for atomistic foundation models). Apart from combining PEFT with ESM bootstrap, what is the new contribution?

Also, the real-world data provides limited support for the quality of the prediction intervals. The Xenium experiments report point prediction metrics such as RMSE, SSIM, and PCC, but the confidence intervals are mainly presented through visualizations. The quantitative evidence showing coverage close to the nominal level comes from synthetic tasks, which makes me question the effectiveness of the method.

Another issue is that the paper uses residual PEFT in the second stage to learn the residuals rather than directly learning the target itself. This module lacks an effective ablation study. I noticed that in Section 5.4, the authors compare the proposed method with single-stage direct resampling. However, I think a more reasonable comparison would be to have two groups starting from the same first-stage model, using the same PEFT module, with one predicting the residuals and the other directly fitting Y.

Questions for authors

See Weakness

4 comments

We thank the reviewer for raising these concerns. We clarify these points and provide an additional ablation study below.

1. Novelty and contributions

We agree that adapting a foundation model and then performing low-cost UQ is not uncommon. However, our goal is to address an important gap in deep-learning UQ literature by bringing frequentist inference for population quantities to foundation models.

(a) Frequentist inference for population estimands, rather than predictive inference for YY.

HIFI targets the population conditional mean μ0(x)=E(Y∣X=x)\mu_0(x)=\mathbb{E}(Y\mid X=x). Unlike much of deep-learning UQ, which targets predictive uncertainty for YY, prediction intervals for noisy outcomes also include the irreducible variability of Y∣X=xY\mid X=x and therefore do not generally require estimation bias to be negligible relative to sampling variability. In contrast, valid frequentist inference for μ0(x)\mu_0(x) generally does. This motivates HIFI's design: Stage 1 reduces systematic estimation error, while Stage 2 quantifies sampling variability. To our knowledge, prior work has not provided a framework for frequentist confidence intervals for population conditional means estimated using fine-tuned large pretrained foundation models.

(b) Scalable frequentist inference at foundation-model scale.

Existing frequentist inference for neural networks has largely been developed for conventional fully connected DNNs, in part because available approaches either rely on specialized analytical structure or require repeated model fitting. The former is difficult to extend to modern foundation models, while the latter becomes prohibitively expensive at foundation-model scale. HIFI is designed specifically to overcome this computational barrier.

(c) Scientific applications of frequentist inference.

HIFI performs frequentist inference on the conditional mean function, relevant to scientific questions such as quantifying uncertainty in mean gene expression and conditional treatment effects in causal inference.

(d) Different inferential targets and guarantees of the cited methods.

Multi-head committees quantify prediction uncertainty through head disagreement but do not provide frequentist confidence intervals for μ0(x)\mu_0(x).

Laplace-LoRA can provide credible intervals for μ0(x)\mu_0(x), but because its inferential framework is Bayesian rather than frequentist, these intervals are not designed to provide frequentist coverage guarantees.

To empirically assess this distinction, our paper already includes Bayesian-PEFT, whose 95% coverage varies substantially across backbones (59.8%--99.8%). We additionally included Laplace-LoRA, where coverage is highly sensitive to prior-precision tuning. For a controlled comparison, both variants use the same fine-tuned model, differing only in prior-precision tuning via model evidence or validation NLL, as proposed by the authors. The resulting coverage does not consistently achieve the nominal level. Only ViT-B/32 and Swin-B achieve coverage comparable to HIFI.

MethodBackboneMAB95% Coverage95% CI LengthWall Time (min/rep)
HIFI-ESMQwen0.01492.8%0.37481
Laplace EvidenceQwen0.155100.0%1.00059.5
Laplace Validation NLLQwen0.01990.7%0.37262.4
HIFI-ESMXLM-R0.01492.6%0.361120
Laplace EvidenceXLM-R0.119100.0%0.97965.8
Laplace Validation NLLXLM-R0.01883.4%0.20667.7
HIFI-ESMNV-Embed0.01394.5%0.52255
Laplace EvidenceNV-Embed0.11499.9%0.97379.8
Laplace Validation NLLNV-Embed0.01382.4%0.21382.1
HIFI-ESMViT0.01893.7%0.28215
Laplace EvidenceViT0.016100.0%2.27710.8
Laplace Validation NLLViT0.01695.0%0.17611.4
HIFI-ESMDINOv20.01195.5%0.31937
Laplace EvidenceDINOv20.012100.0%16.8558.7
Laplace Validation NLLDINOv20.01288.3%0.2229.0
HIFI-ESMSwin0.01395.1%0.23624
Laplace EvidenceSwin0.012100.0%4.67226.3
Laplace Validation NLLSwin0.01296.3%0.19127.6

2. Why coverage is evaluated on synthetic data

We thank the reviewer for raising this important question regarding coverage evaluation on real-world data. A key difficulty is that the true population conditional mean μ0(x)\mu_0(x) is unobserved. In datasets such as Xenium, we observe only noisy realizations YY, so for most real-world datasets, nominal coverage for μ0(x)\mu_0(x) cannot be directly evaluated. This is why our primary coverage evaluation uses synthetic experiments, where μ0(x)\mu_0(x) is known.

Nevertheless, we sought to provide the strongest real-data validation possible. The Xenium dataset contains 427,532 fine-grid locations, allowing us to construct an empirical finite-population benchmark. We treat our previously estimated fine-grid prediction surface as a fixed reference quantity. We randomly sample 4,455 locations to mimic the limited sample available in practice and apply HIFI using their observed gene-expression values as noisy training responses. We evaluate whether the resulting confidence intervals cover the reference values at 500 fixed held-out locations across 100 replications.

We evaluated all 313 genes. For most genes, both methods achieved coverage close to the nominal 95% level. We provide the MAB, empirical coverage, and CI length for all 313 genes at this link (https://anonymous.4open.science/r/Additional-Real-Data-Result-4B6E/). MAB and CI length are reported on the count scale.

3. Residual PEFT vs. directly learning the target

We initially tried fitting (Y) directly in Stage 2 using the adapted Stage-1 backbone, but found that it produced conservative coverage and wider confidence intervals, motivating the residual formulation.

We therefore added a Direct-Y ablation using the same settings as in the paper. Across all settings, HIFI achieves lower MAB, shorter CIs, and coverage closer to the nominal 95% level.

BackboneMethodMAB (HIFI/Direct-Y)MAE (HIFI/Direct-Y)95% coverage (HIFI/Direct-Y)95% CI length (HIFI/Direct-Y)
ViT-B/32ESM0.018/0.0230.041/0.0480.937/0.9990.282/0.492
ViT-B/32Bootstrap0.017/0.0240.047/0.0460.938/0.9820.231/0.283
DINOv2-B/14ESM0.011/0.0170.044/0.0430.955/1.0000.319/0.906
DINOv2-B/14Bootstrap0.011/0.0160.046/0.0420.953/1.0000.243/0.541
Swin-BESM0.013/0.0180.035/0.0390.951/1.0000.236/0.442
Swin-BBootstrap0.013/0.0200.042/0.0380.943/0.9930.215/0.268
Qwen3-Embedding-8BESM0.014/0.0230.060/0.0610.928/1.0000.374/0.605
Qwen3-Embedding-8BBootstrap0.015/0.0210.064/0.0660.937/0.9880.316/0.403
AuthorsOpenReview ↗

Manuscript Update

We appreciate your comments on our contribution and empirical validation. Following our earlier response, we have updated the manuscript:

  1. Baselines: Added Laplace-LoRA to Tables 1 and 2, with additional prior-precision sensitivity results in the appendix.
  2. Real-data evaluation: Expanded the Real Data Analysis section with a finite-population benchmark across all 313 Xenium genes, evaluating confidence intervals against a fixed estimated reference surface. We have also extended the analysis presented in our earlier response by comparing HIFI with three additional baselines.
  3. Residual PEFT vs directly learning the target: Added the Direct-Y comparison to Ablation Study 2 and Figure 3, comparing HIFI’s residual formulation with directly fitting the response in Stage 2 using the Stage-1-adapted backbone.

We hope our earlier clarifications and these additional results address your concerns about HIFI’s contribution and empirical validation. We would appreciate your taking these updates into account when reassessing our work, and we welcome any further feedback.

Based on the ablation studies, the two-stage correction approach is indeed a valuable design choice. Regarding experiments on real-world data, my primary interest lies in whether the 95% confidence intervals provided by HIFI achieve their nominal coverage rate for the true values. However, I recognize that the true expected estimates—required for such verification—are generally unavailable in the authors' experimental setup, making it difficult to empirically falsify the results; thus, I find the visualizations provided by the authors acceptable.

I acknowledge these empirical improvements. My reservations, however, persist regarding the paper's level of innovation, as it largely resembles a stacking of existing modules. In their response, the authors emphasize the specific objective of "frequentist inference for conditional means." Yet, from a methodological standpoint, the core components—fitting a base model followed by a lightweight module for uncertainty quantification, employing two-stage sample splitting with residual correction, and utilizing off-the-shelf resampling-based inference mechanisms (such as ESM and Bootstrap)—all have well-established precedents. While applying these techniques to frequentist inference for the conditional means of base models offers clear practical utility, I remain skeptical about the degree of methodological innovation in this framework. I would not object if the paper were accepted, but I am inclined to maintain my current score.

Reviewer_BjpROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose HIFI, a two-stage framework for estimating the conditional mean and confidence interval when finetuning foundation models. They show that HIFI has comparable error rates and asymptotic coverage, and also demonstrate that HIFI outperforms HulC and Bayesian-PEFT across synthetic language and vision tasks.

Critical strengths
  • The problem is well-motivated: being able to significantly update large models to downstream datasets while maintaining uncertainty could be very useful, and the paper provides a practical way to do that even for models with billions of parameters. The method is also intuitive and easy to understand, and I like that the method could be adjusted depending on the amount of compute available.
  • The experiments cover a wide range of model and model sizes across both vision and text domains. The backbones are diverse, and the coverage of HIFI is impressive.
Critical weaknesses
  • There are lots of decisions which need to be made for this method to work effectively, and these decisions seem under-explored in this work. For example, the division of data between Stage 1 and Stage 2 seem crucial to the success of the method, but there are no ablations that vary their split. Furthermore, it would be useful to see how different design decision impact the effectiveness of Stage 2, such as the number of models, the LoRA rank, the size of the MLP, etc. The paper indicates that Stage 2 corrects the bias from Stage 1 and quantifies the uncertainty, and it would be useful to see how well it performs on either of these tasks across the test settings.
  • The paper proposes HIFI because it is compute-efficient, but we see very little demonstration of this. The paper does not compare the compute for HIFI against their baselines (HulC and Bayesian PEFT) . The authors also do not show cheaper uncertainty baselines that are common in deep learning, such as Laplace-LoRA or seed ensembles. Furthermore, the compute benefits are dependent on the split between D1 and D2, which is not discussed.
Questions for authors

I would appreciate it if you could address my questions from the weaknesses above.

7 comments

We thank the reviewer for these helpful suggestions. In our original experiments, we did not extensively tune the Stage-2 capacity hyperparameters or optimize the data split for each resampling procedure. To address these concerns, we have added several new ablation studies and computational comparisons, summarized below.

1. Data allocation between Stage 1 and Stage 2

HIFI is a general framework for scalable frequentist inference that can accommodate different resampling-based inference procedures, which may favor different data allocations. Using the paper’s ViT-B/32 setup, we vary only the Stage 1–Stage 2 split.

  • Bootstrap and ESM respond differently to the split. Bootstrap maintains near-nominal coverage across allocations, whereas shifting more data to Stage 2 makes ESM increasingly conservative with wider intervals.
  • Larger Stage 2 increases computation. For both procedures, allocating more data to Stage 2 substantially increases the computational cost.
MethodStage 1 / Stage 2MAB95% Coverage95% CI LengthWall Time (min/rep)
HIFI-BootstrapPaper: 87.6% / 12.4%0.01793.8%0.23132
HIFI-Bootstrap60% / 40%0.018093.5%0.177673.9
HIFI-Bootstrap30% / 70%0.018694.5%0.1827112.5
HIFI-ESMPaper: 87.6% / 12.4%0.01893.7%0.28215
HIFI-ESM60% / 40%0.018799.3%0.361533.7
HIFI-ESM30% / 70%0.018299.99%0.656168.9

2. Stage-2 design choices and effectiveness

We use the paper's ViT-B/32 HIFI-Bootstrap setup and evaluate each configuration over 100 replications. For Stage 1, we consider two checkpoints: a one-epoch checkpoint and the best checkpoint selected by the lowest Stage-1 validation MSE. We examine two Stage-2 design choices: the LoRA rank and the number of LoRA-adapted blocks, while keeping all other settings fixed within each ablation.

(a) Robustness to LoRA rank.

We vary the LoRA rank from 8 to 128, with adapters applied to the last two Transformer blocks.

  • Stage 2 corrects Stage-1 estimation error. With the one-epoch Stage-1 checkpoint, MAB decreases from 0.03264 to 0.02570–0.02591 after Stage 2.
  • Better Stage-1 estimation improves statistical inference. Across all LoRA ranks, using the best Stage-1 checkpoint yields lower MAB and shorter confidence intervals, with coverage closer to the nominal 95% level than the one-epoch checkpoint.
  • Robustness to Stage-2 capacity. Across both Stage-1 checkpoints, coverage remains stable as the LoRA rank increases from 8 to 128, while interval lengths change only slightly.
  • Computation did not increase with rank in this experiment. Higher ranks selected earlier checkpoints on average, with the validation-selected epoch decreasing from 12.21 at rank 8 to 6.69 at rank 128.
MetricStage 1 checkpoint (MAB)Rank 8 (α = 16)Rank 16 (α = 32)Rank 32 (α = 64)Rank 128 (α = 256)
MABEpoch 1 (0.03264)0.025700.025750.025820.02591
Best (0.01796)0.017560.017580.017600.01762
Cov.95 (%)Epoch 192.7992.9893.1694.00
Best93.5593.7793.9694.75
CI lengthEpoch 10.25110.25180.25170.2518
Best0.23250.23230.23230.2309
Wall Time (min/rep)Epoch 115.914.413.212.2
Best14.813.312.111.4

(b) Robustness to the number of LoRA-adapted blocks.

We vary Stage-2 capacity from head-only fitting to LoRA adaptation in the last 2 or 6 Transformer blocks, with rank fixed at 8 and alpha 16. Head-only fitting trains a newly initialized two-layer MLP head with a 64-unit hidden layer and scalar output.

  • Stage 2 corrects Stage-1 estimation error. With the one-epoch Stage-1 model, MAB decreases from 0.03264 to 0.02561.
  • Better Stage-1 estimation improves statistical inference. Even with head-only fitting, MAB decreases from 0.02615 to 0.01792 and coverage increases from 90.75% to 92.52%.
  • Robustness to Stage-2 capacity. HIFI remains effective even with head-only fitting. Additional capacity improves coverage from 90.75% to 93.24% and from 92.52% to 93.81% for the one-epoch and best checkpoints, respectively.
  • Computation–calibration trade-off. Increasing Stage-2 capacity improves coverage by 1.3–2.5 percentage points, but increases wall time from approximately 4 minutes to 25–32 minutes.
MetricStage 1 CheckpointHead OnlyHead + Last 2 LoRAHead + Last 6 LoRA
MABEpoch 1 (0.03264)0.026150.025700.02561
Best (0.01796)0.017920.017560.01736
95% Coverage (%)Epoch 190.7592.7993.24
Best92.5293.5593.81
95% CI LengthEpoch 10.22220.25110.2559
Best0.21320.23250.2310
Wall Time (min/rep)Epoch 14.2815.8925.26
Best4.2714.7931.95

3. Additional baselines and Computational efficiency

We thank the reviewer for raising this point. Our original computational claim focused on making frequentist inference feasible at foundation-model scale. To our knowledge, existing methods have not provided frequentist confidence intervals for population conditional means at foundation-model scale.

(a) Uncertainty baselines in deep learning. Most existing deep-learning UQ methods focus on predictive uncertainty, whereas HIFI provides the frequentist inference for population conditional mean. Seed ensembles are one example of predictive uncertainty.

(b) Comparison with Bayesian baselines. Bayesian methods can also quantify uncertainty for population quantities, but they are not designed to provide frequentist coverage guarantees. Empirically, Bayesian-PEFT in the paper has 95% coverage ranging from 59.8% to 99.8% across settings. We also added Laplace-LoRA as a computationally efficient Bayesian baseline and found that, although it can reduce computation on some backbones, its coverage is highly sensitive to prior-precision tuning and does not consistently attain the nominal level. Additional results are provided in our response to Reviewer EktL.

(c) Comparison with HulC. HulC is the closest external frequentist comparator. Although HulC is computationally cheaper for Qwen (20.86 vs. 81 min/rep) and has similar cost on ViT-B/32 (13.31 vs. 15 min/rep), it has lower coverage and higher MAB than HIFI-ESM in both cases. Importantly, HIFI can also operate with a much lighter Stage 2: on ViT-B/32, head-only HIFI-Bootstrap requires only 4.27 min/rep, while achieving 92.52% coverage and 0.01792 MAB, compared with 13.31 min/rep, 91.4% coverage, and 0.027 MAB for HulC.

BackboneMethodMAB95% Coverage95% Interval LengthWall Time (min/rep)
ViT-B/32HIFI-ESM0.01893.7%0.28215
ViT-B/32HulC0.02791.4%0.20613.31
ViT-B/32Bayesian-PEFT0.05476.4%0.18314.7
ViT-B/32Laplace Evidence0.016100.0%2.27710.8
ViT-B/32Laplace Validation NLL0.01695.0%0.17611.4
QwenHIFI-ESM0.01492.8%0.37481
QwenHulC0.07588.9%0.41820.86
QwenBayesian-PEFT0.08589.0%0.426116.6
QwenLaplace Evidence0.155100.0%1.00059.5
QwenLaplace Validation NLL0.01990.7%0.37262.4
AuthorsOpenReview ↗

Manuscript Update

Thank you again for your thoughtful comments and constructive suggestions, which have helped us strengthen the manuscript. We have incorporated the additional analyses into the revised manuscript:

  1. Data allocation: Added Ablation Study 1 in the main text, examining Stage-1–Stage-2 allocation for both bootstrap and ESM under a fixed total sample size.
  2. Stage-1 quality and Stage-2 capacity: Added Ablation Study 3 in the main text, examining Stage-1 checkpoint quality, LoRA rank, and the number of adapted blocks.
  3. Baselines and computation: Added Laplace-LoRA and wall-time comparisons to the vision and language results in Tables 1 and 2, with additional prior-precision sensitivity results in the appendix.

We would be grateful if you could reconsider our contribution in light of the clarifications provided in our previous response and the revisions incorporated into the manuscript. We would also be happy to address any remaining questions or concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Manuscript Update

Dear Area Chair and Reviewers,

We appreciate your time and effort in evaluating our work and are grateful for the reviewers’ constructive feedback.

We have revised the manuscript in response to the questions and suggestions raised by all three reviewers. The main updates are summarized below:

  1. Data allocation (Ablation Study 1; Section 5.3, Figure 2): Added an ablation examining Stage-1–Stage-2 data allocation for both bootstrap and ESM under a fixed total sample size. The original Stage-1 sample-size ablation has been moved to Appendix E.5.

  2. Two-stage learning and residual fitting (Ablation Study 2; Section 5.4, Figure 3): Added a Direct-Y baseline to the existing comparison between HIFI and single-stage resampling. Direct-Y retains the Stage-1-adapted backbone and fits the response directly in Stage 2 using the same Stage-2 data and resampling settings as HIFI.

  3. Stage-1 quality and Stage-2 capacity (Ablation Study 3; Section 5.5, Figure 4): Added an ablation examining Stage-1 checkpoint quality, LoRA rank, and the number of adapted blocks, with both statistical performance and wall time reported.

  4. Baselines and computation (Tables 1 and 2): Added Laplace-LoRA and wall-time comparisons, together with prior-precision sensitivity analyses in Appendix E.

  5. Real-data evaluation (Section 6, Figure 5): Added a finite-population benchmark across all 313 Xenium genes, evaluating intervals against a fixed estimated reference surface and comparing HIFI with HuLC, Bayesian-PEFT, and Laplace-LoRA. Benchmark details are provided in Appendix F.5.

We hope these revisions address the reviewers’ concerns and strengthen the manuscript. We would appreciate your consideration of the revised manuscript and additional evidence, and we welcome any further feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.