- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces doubly propagated calibration (DPC), a doubly robust conformal prediction algorithm mitigating the finite-sample errors in nuisance fitting. The DPC method appears to be built on the doubly robust prediction (DRP). The proposed method generally attains the nominal coverage (while slightly conservative but has great improvement compared to existing benchmarks) in simulation settings compared.
Critical strengths
-
The paper's formulation is excellent: from reviewing existing approaches briefly, pointing out key limitations and gaps, to introducing the proposed approaches gradually and clearly.
-
Presentation is helpful, e.g., Figure 1 at the beginning is convincing and helpful for understanding the question and scope. Figure 2 is intuitive and highly summarized. Algorithm 1 is helpful for my understanding.
-
Theoretical investigations are deep. Experiments are comprehensive and informative.
Critical weaknesses
Major comment
How does the method compare to some intermediate version of DRP to DPC. For example, using cross-fitted estimator in DRP instead of plug-in. Is their any optimality guarantee (especially in terms of efficiency) by the proposed DPC.
Minor issues
-
Please do not let any formulas, tables, figures or text out the specified margin limit. This cannot pass the final camera-ready check. Equations (2), (24), (42), (98), (118) etc. require revisions. Please check all lines in the paper. Enabling "grid view" in pdf viewer tool could be helpful.
-
"Existing methods often fail to balance robustness and efficiency" is a relatively strong and absolute opinion. They "can" but may not be said as "often". WCP may fail under severe covariate shifts, but sufficient overlap is an underlying assumption of this method, so that it is not so fair to claim this method often fails.
-
Could the authors highlight the novelty beyond DRP (Yang et al; 2024) in more depth? For example, Step 2 might be something their work cannot handle, if I understand correctly. These novel algorithms/steps are not naturally possible choices under Yang et al.'s framework, but require their own developments. In addition, you may highlight one advantage of adopting Yang et al. (2024)'s equation, besides doubly robust, is it attains the semiparametric efficiency bound under correctly specified nuisance models. Your algorithm further increases the flexibility of the nuisance fitting, so it may help improving efficiency towards the efficiency bound.
-
(Optional). I know this suggestion might be a little personal preference, but I feel some simulation data generation details are not necessary for the main text readability, which could be moved to appendix, and just summarize in words what does each scenario mean.
-
Theorem 3.1 is the key of this paper. It requires some remarks in the paper about its practical implications.
Questions for authors
-
You mentioned in the end of Section 2, DRP removes nuisance-induced bias, but not sampling variability. Did you mean their original algorithm didn't use sample-splitting? Is this related to using plugging-in estimator or some other reasons? If nuisances are trained in a training set, why that is still a plug-in?
-
So is your final estimator still using the influence function (or empirical moment equation) in (2), found by Yang et al. (2024)? The main change is the implementation details?
-
Is the slightly conservative performance in simulation expected rather than by chance? Is that an asymptotic behavior or just finite sample?