- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper aims to address the high computational cost of conventional resampling methods, which require training a model hundreds of times to estimate statistical confidence intervals in foundation models.
To this end, the authors propose a two-stage framework called HIFI. In Stage 1, a base model is trained once on a large dataset to reduce estimation bias. In Stage 2, resampling is applied to a smaller dataset to train multiple lightweight LoRA models that fit residuals and estimate uncertainty.
In experiments across various vision and text models, HIFI achieved lower estimation bias and coverage closer to target confidence levels compared to existing methods, while significantly reducing training compute relative to direct resampling. Furthermore, the authors demonstrate that HIFI can provide confidence intervals while preserving predictive performance in a real-world gene expression prediction task.
Critical strengths
- Demonstrates high computational efficiency and stable statistical inference performance. While reducing training compute by 79.5–91.9% compared to conventional direct resampling, it achieves low estimation bias and coverage close to target confidence levels across most experiments. This is significant as it enhances the practical applicability of statistical inference in large-scale foundation models.
- Validates applicability across diverse models and real-world tasks. The method was evaluated on six pretrained vision and text models, demonstrating the ability to provide confidence intervals while maintaining predictive performance comparable to existing baselines in a real-world gene expression prediction task.
Critical weaknesses
- The independent contribution of residual learning is unclear. The paper does not compare against a baseline that shares the same Stage 1 backbone but directly predicts the ground truth instead of the residual in Stage 2. Consequently, it is difficult to distinguish whether performance improvements stem from residual learning itself or simply from leveraging a sufficiently trained Stage 1 backbone.
- Insufficient validation of Stage 2's residual correction capability. HIFI relies on the assumption that a small PEFT model can adequately correct Stage 1 estimation errors. However, the authors do not analyze whether confidence interval coverage is maintained when Stage 1 bias is large or when Stage 2 expressiveness is limited. This issue is directly tied to the reliability of statistical inference.
Questions for authors
- What results are observed if direct prediction is performed instead of residual learning in Stage 2? Comparing MAB, MAE, Coverage, and CI Length under the same Stage 1 backbone and PEFT configuration would help clarify the actual contribution of the proposed residual learning approach.
- What is the experimental evidence supporting that resampling only Stage 2 can sufficiently correct the estimation bias of Stage 1? Specifically, I would like to know how the residual estimation error and final coverage vary when Stage 1 has a large estimation error or when Stage 2 model capacity is constrained.
6 comments
Manuscript Update
Thank you for your comments on residual fitting and the roles of Stage-1 estimation quality and Stage-2 model capacity. We have incorporated the corresponding analyses into the revised manuscript:
- Residual PEFT vs directly learning the target: Added the Direct-Y comparison to Ablation Study 2 and Figure 3, comparing HIFI’s residual formulation with directly fitting the response in Stage 2 using the Stage-1-adapted backbone.
- Stage-1 quality and Stage-2 capacity: Added Ablation Study 3, examining Stage-1 checkpoint quality, LoRA rank, and the number of adapted blocks.
We hope these revisions address your questions. We would be grateful if you could reconsider our contribution in light of our previous clarifications and the revised manuscript. We would be happy to address any remaining concerns.
Thank you for the additional experiments and detailed response. The Direct-Y comparison clearly confirms the necessity of residual learning, so my first concern has been adequately resolved.
However, what I aimed to verify in my second question is not simply whether Stage 2 can reduce the average estimation error, but whether residual correction in Stage 2 alone can maintain the statistical reliability of the confidence intervals even when Stage 1's estimation error is large.
While the newly added checkpoint and LoRA capacity ablations are useful, I would like to examine the following aspects more directly:
- Analysis stratified by the magnitude of Stage 1 estimation error: Across your existing repeated runs, could you separate cases where Stage 1's estimation error is small versus large, and analyze how Stage 2's residual estimation error and final coverage differ between them?
- Relationship between Stage 2's correction capability and coverage: In the additional experiments, coverage tended to decrease when Stage 2's capacity was constrained. It would be helpful to analyze whether this drop stems from insufficient residual correction or from differences in confidence interval length.
Such an analysis would be important for clarifying the conditions under which a design that resamples only Stage 2 without resampling Stage 1 can be considered reliable.
We thank the reviewer for this suggestion, which helped strengthen the manuscript. In our original experiments, we did not extensively tune the hyperparameters, as HIFI appeared relatively robust to these choices. We now provide additional ablations below.
1. Residual PEFT vs directly learning the target.
We initially tried fitting (Y) directly in Stage 2 using the adapted Stage-1 backbone, but found that it produced conservative coverage and wider confidence intervals, motivating the residual formulation.
We therefore added a Direct-Y ablation using the same settings as in the paper. Across all settings, HIFI achieves lower MAB, shorter CIs, and coverage closer to the nominal 95% level.
2. Stage-1 estimation quality vs Stage-2 model capacity
We use the paper's ViT-B/32 HIFI-Bootstrap setup and evaluate each configuration over 100 replications. For Stage 1, we consider two checkpoints: a one-epoch checkpoint and the best checkpoint selected by the lowest Stage-1 validation MSE. We examine two Stage-2 design choices: the LoRA rank and the number of LoRA-adapted blocks, while keeping all other settings fixed within each ablation.
(a) Robustness to LoRA rank.
We vary the LoRA rank from 8 to 128, with adapters applied to the last two Transformer blocks.
(b) Robustness to the number of LoRA-adapted blocks.
We vary Stage-2 capacity from head-only fitting to LoRA adaptation in the last 2 or 6 Transformer blocks, with rank fixed at 8 and alpha 16. Head-only fitting trains a newly initialized two-layer MLP head with a 64-unit hidden layer and scalar output.
Robustness of HIFI. For comparison, coverage across backbones ranges from 88.5%–96.5% for HulC, 59.8%–99.8% for Bayesian-PEFT, and 82.4%–100.0% for Laplace-LoRA. Detailed Laplace-LoRA results are provided in our response to Reviewer EktL.