- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies online conformal inference under distribution shifts. The authors propose an -insensitive hybrid quantile loss (-HQ) that combines the standard quantile loss with a bounded quadratic component to provide more informative gradient feedback. Building on this loss, they develop a dual-calibration framework that adaptively rescales residuals, adjusts the insensitive zone, and uses historical gradient directions for an additional threshold correction. The paper presents dynamic regret and coverage-error bounds, along with experiments on financial time series and medical image classification datasets.
Critical strengths
-
The problem is relevant and well motivated. Improving the coverage–efficiency trade-off under non-stationarity is an important problem in online conformal inference. The limitations of binary quantile-loss feedback provide a reasonable motivation for exploring alternative update rules.
-
The hybrid loss and modular design are interesting. Combining a bounded quadratic component with the standard quantile loss offers a way to incorporate residual magnitude without losing non-vanishing gradients for large deviations. The paper also makes a substantial effort to analyze the proposed loss and connect the resulting online updates to regret and coverage guarantees.
-
The experimental evaluation is reasonably comprehensive. The authors compare against several recent baselines across different datasets and prediction models. The ablation and sensitivity analyses are also useful for understanding how the individual components affect performance.
Critical weaknesses
-
The asymptotic coverage argument appears to have a gap. Theorem 5.2 establishes a dynamic regret bound of order , but Corollary 5.4 assumes only when concluding that the normalized regret vanishes. This would generally require . Moreover, is defined through instantaneous loss minimizers, which may fluctuate even under stationary score distributions. It is therefore unclear when the regret bound leads to a meaningful asymptotic coverage guarantee.
-
Some parts of the theoretical analysis need clarification. The expression for in Eq. (5) appears inconsistent with the preceding equal-peak condition, which also affects the constants in Eq. (6). In addition, Appendix B.1 and the coverage proof in Appendix B.4 treat the residual mapping as a single affine transformation, whereas Eq. (7) defines different mappings for positive and negative residuals. The arguments relying on a common slope do not seem to follow directly when the residuals have opposite signs.
-
The interpretation of historical gradient directions as a drift signal is not fully convincing. The momentum correction uses an average of normalized gradient directions. However, these directions are inherently asymmetric for quantile estimation. For example, with a continuous score distribution, , and a correctly calibrated threshold, their expected average is , which equals when , even without distribution shift. Thus, a persistent directional signal does not necessarily indicate drift, raising questions about how the additional correction affects calibration.
-
The empirical advantage of the full framework remains somewhat limited. In Table 1, with LightGBM, the simpler -HQ-OGD achieves better MIS than -HQ-DCOGD on three of the four financial datasets, suggesting that the momentum correction does not consistently improve performance. The full method also falls below the nominal 90% coverage on DERM12345. More importantly, the experiments mainly report aggregate metrics, making it difficult to assess whether the method actually adapts faster to distribution shifts. The class-ordered medical image streams provide only a limited test of this claim.
Questions for authors
-
Could the authors clarify the conditions under which the normalized dynamic regret vanishes in Corollary 5.4? In particular, how should behave under stationary or slowly varying score distributions, given that it is defined using instantaneous loss minimizers?
-
Could the authors verify Eq. (5) and clarify how the proofs in Appendices B.1 and B.4 account for the sign-dependent residual mapping? Do the stated regret and coverage bounds apply directly to the implemented algorithm?
-
How does the momentum mechanism distinguish genuine distribution drift from the inherent asymmetry of quantile-loss gradient directions? Could the authors provide rolling coverage and recovery-time comparisons with -HQ-OGD to demonstrate when the additional calibration is beneficial?
5 comments
Response 3: Gradient direction bias and the EMA-centered mechanism (Weakness 3). We thank the reviewer for highlighting the systematic bias in momentum direction statistics. For correctly calibrated continuous scores, the standard quantile gradient direction satisfies ( for ). Using the raw mean conflated this inherent asymmetry with actual drift.
To resolve this, we introduce an EMA-centered mechanism. With window mean , the EMA baseline updates as (initialized at ), and the centered momentum is . This measures recent variations relative to the historical baseline, avoiding static bias. From the EMA update, . Appendix B.3 utilizes this difference structure and Abel summation to establish , strictly preserving the regret order.
Furthermore, the centered signal concurrently drives both calibrations: its sign dictates the momentum correction , while its magnitude dynamically adjusts the insensitive-zone width . As directional deviation increases, the insensitive zone shrinks to enhance drift sensitivity; as the baseline adapts, both corrections diminish (see Section 4.2-4.3, Eqs. (9)-(10), (13)-(15); Theorem 5.2; Appendix B.3; Appendix C.1, Algorithm 1).
Response 4: Incremental benefits and dynamic adaptability evaluation (Weakness 4). We acknowledge that the original aggregate metrics obscure transient recovery; conventional datasets are insufficient to evaluate adaptability to abrupt shifts. We add four synthetic drift scenarios and introduce Recovery Time (RT) and rolling coverage analyses.
(1) MIS increments. -HQ-DCOGD achieves lower MIS than -HQ-OGD in Variance Changepoint (1.356 vs. 1.362), Changepoint (1.203 vs. 1.236), and Extreme Drift (1.208 vs. 1.214), and comparable values under Smooth Drift (1.073). Incremental benefits primarily occur in structural shifts (see Section 6.3, Table 3; Appendix D.11, Table 19).
(2) Transient recovery. In Changepoint, the full model shortens average recovery times by 9.0 and 4.8 steps (; ), with lower RT standard deviation. MIS decreases from 1.236 to 1.203, and interval width from 0.920 to 0.915. Figure 2 illustrates the rolling coverage trajectory. Notably, ECI achieves a lower MIS (1.196) but at only 86.880% coverage vs. our 88.608%, reflecting a distinct coverage-efficiency trade-off (see Section 6.3; Appendix D.11, Tables 19-20, Figure 2).
(3) Medical trade-off. On DERM12345 and ResNet-50, -HQ-DCOGD yields Coverage 87.551%, PI-Size 1.981, MDSC 0.673, vs. -HQ-OGD's 89.238%, 2.089, and 0.671. We report this sub-nominal coverage, distinguishing empirical coverage, set efficiency, and composite scores without equating smaller sets to coverage improvement (see Section 6.2, Table 2; Appendix D.6-D.7).
Validated across 4 financial, 2 medical, and 4 synthetic datasets, the evidence substantiates the transient adaptation value in abrupt settings, while delineating that incremental benefits are not universally established (see Section 6.1-6.3; Appendices D.5-D.7, D.11).
Responses to Specific Questions
Question 1: Since relies on realized scores, random fluctuations may cause even under stationarity. The bound implies is sufficient for vanishing normalized regret; Corollary 5.4 explicitly retains (see Response 1).
Question 2: Eq. (5) is corrected for the factor and Eq. (6) verified. B.1 defines the scaled loss on the positive-slope branch to align gradients and handles opposite-sign residuals via the loss upper bound. B.4 uses independent branch slopes and the triangle inequality, supplemented by conservative clipping mass control (see Response 2).
Question 3: The EMA baseline isolates long-term bias via recent directional variations. This signal concurrently drives both the momentum term and the dynamic insensitive zone. Changepoint experiments show the full model shortens average recovery times by 9.0 and 4.8 steps vs. -HQ-OGD (see Response 3 and Response 4).
On the Rating and Confidence. We understand your concerns regarding mathematical rigorousness, inherent direction bias, and dynamic adaptation evidence. We hope these clarifications—spanning sign-dependent mapping proofs, EMA-centered momentum, condition-dependent regret bounds, and changepoint experiments—strengthen your confidence in our work.
Supplementary Update on the Clipping/Fallback Issue (W2)
We would like to provide a brief supplementary update regarding the clipping/fallback issue raised in W2.
In our previous response, we acknowledged that the affine-branch analysis did not establish global gradient consistency for the hard clipping/fallback branch. We have since performed a further theoretical revision to address this limitation.
For the integral surrogate loss defined in Appendix B.1, we have
Importantly, this identity does not involve , and therefore remains consistent with the implemented gradient on both the affine and clipping/fallback branches.
We have also revised Assumption 5.1 to distinguish the positive-slope affine condition from the clipping-branch endpoint condition, introducing a unified radial lower bound:
Using this condition together with the sign-preserving property, Appendix B.4 re-establishes the residual-to-loss lower bound underlying Eqs.(63)–(64), including clipping/fallback cases. The finite-time contribution arising from the warm-up phase is also explicitly accounted for.
Consequently, under the stated regularity and endpoint conditions, the revised regret and coverage analyses apply to the implemented Algorithm 1, rather than being restricted to the non-saturated affine branch.
We appreciate the reviewer's earlier observation, which motivated us to further strengthen the consistency between the implemented algorithm and its theoretical guarantees. We hope this fully addresses the raised concern.
Revisions made in: Sections 4.2–4.3, Assumption 5.1, Theorem 5.3, Corollary 5.4, Appendices B.1, B.4 and B.5.
We thank the reviewer for the positive evaluation of our research problem, loss function design, and theoretical/empirical analyses. Addressing your concerns regarding theoretical rigorousness, mapping consistency, and empirical evidence, our detailed responses follow.
Response 1: Theoretical conditions for the asymptotic coverage corollary (Weakness 1). We acknowledge that deducing the vanishing of normalized dynamic regret solely from VT=O(T) is invalid. Since the instantaneous comparator relies on the realized score, its path length may grow linearly even under a stationary distribution. To address this, we explicitly select the common instantaneous minimizer qtopt=st for both the original and scaled losses, and define VT=∑t=1T−1∣st+1−st∣. Under the specified regularity conditions, Theorem 5.2 provides the dynamic regret upper bound:
R~Tdyn≤μCjointT(4R2+3RVT)=O(T(1+VT)).
Consequently, the normalized regret is bounded by O((1+VT)/T). Only when VT=o(T) does this bound guarantee convergence to zero; for VT=Θ(T), this cannot be deduced. Accordingly, the revised Corollary 5.4 retains the long-term regret term:
limsupT→∞∣CT−(1−α)∣≤μCR∞+amin2Cε∞+μCΔ∞,
where R∞=limsupT→∞TRTdyn, ε∞=limsupT→∞T1∑t=1Tεt, and Δ∞=2amin2CMZabs2+amin1−τ+2aminMZabs. This bound decomposes the asymptotic coverage error into a potentially non-vanishing normalized regret, the average insensitive-zone width, and the residual loss bias, while conservatively incorporating the clipping-induced boundary probability mass. The corollary is now rigorously formulated as an asymptotic coverage error upper bound under the specified conditions (see Section 5.1–5.2, Theorems 5.1–5.3, Corollary 5.4; Appendix B.2–B.5).
Response 2: Gain-balancing and sign-dependent residual mapping (Weakness 2). We acknowledge that Eq. (5) was inconsistent with the equal-peak condition, and that the common-slope arguments in Appendices B.1 and B.4 were insufficient. We address this from three aspects:
(1) Gain-balancing formula. Solving the equal-peak constraint τA=(1−τ)8γ33λB under γ=1 yields λ(τ)=31(3B8A1−ττ)2. The theoretical constants in Eq. (6) are synchronously verified as M=2τA, μ=τmin(α,1−α), and K=τA(1+334) (see Section 4.1, Eqs. (5)–(6); Appendix A.1–A.2).
(2) Piecewise mapping and gradient consistency. We distinguish mapping branches. In the positive-slope piecewise-affine branch ϕt(z)=ats(z)z, we define a scaled loss ℓ~t(q)=ats(z)1Lε-HQ(ϕt(st−q)) to mitigate slope-factor interference. Within this branch, the chain rule yields −Lε-HQ′(ϕt(st−q))=gt, aligning with the actual update signal in Algorithm 1 (see Section 4.2–4.3, Eqs. (7), (11)–(15); Appendix B.1).
(3) Branch handling in proofs. B.1 selects the common minimizer qtopt=st and leverages Lε−HQ(u)≤K(∣u∣−εt)+ for opposite-sign residuals. In B.4, threshold deviations are established using the triangle inequality ∣st−qt∣+∣st−qtqt∣≥∣qt−qtqt∣ with independent slopes at,1,at,2∈[amin,amax]. B.4 also independently estimates the clipping boundary mass $E [|u| \mid \mathcal F _{t-1}] \le \frac{C Z _{\mathrm{abs}} ^ 2}{a _ {\min}} + Z _{\mathrm{abs}} $ (see Section 5.2, Theorem 5.3; Appendix B.1, Eq. (40); Appendix B.4).