acceptodds
Withdrawn from ICLR 2027

Online Asynchronous Federated Conformal Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We propose OAFCP, a novel online asynchronous federated conformal prediction framework for uncertainty quantification across heterogeneous clients with streaming data. Specifically, our method jointly updates model parameters and conformal quantiles as new observations arrive, producing adaptive prediction intervals for regression and prediction sets for classification without sharing raw data. Clients process their data streams at different speeds, while the server aggregates available updates at fixed intervals without waiting for all clients. The aggregation weights compensate for differences in computation speed to balance client contributions and ensure long-term aggregation fairness. On the theoretical side, we establish long-run coverage guarantees under asynchronous aggregation and, under score-distribution stability conditions, show that client-weighted instantaneous coverage converges to the nominal level. Experiments on synthetic data and six real-world datasets demonstrate the finite-sample performance of OAFCP.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Conformal Prediction, Federated Learning, Streaming data, Asynchronous Optimization, Data Heterogeneity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_jMkyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Online Asynchronous Federated Conformal Prediction (OAFCP), a framework for uncertainty quantification in federated learning with streaming data and heterogeneous client computation speeds. The method jointly updates predictive models and conformal thresholds locally, while asynchronously aggregating client updates at fixed server intervals using participation-frequency-aware weights. The authors establish theoretical results concerning long-run coverage and asymptotic calibration under certain assumptions. Experiments on synthetic data and six real-world datasets suggest that OAFCP can achieve competitive coverage and prediction efficiency compared with a synchronous baseline, particularly under distribution shifts.

Critical strengths
  1. The paper addresses the intersection of online conformal prediction and asynchronous federated learning, which is an important yet relatively underexplored setting.
  2. OAFCP combines online model updates with adaptive conformal calibration and fixed-interval asynchronous aggregation. The client-server communication protocol is clearly motivated, and the proposed framework does not require exchanging raw observations.
  3. The paper goes beyond proposing an algorithm by investigating both long-run empirical coverage and instantaneous conditional coverage.
Critical weaknesses
  1. The learning-rate definitions used in the algorithm and theoretical analysis seems to be inconsistent. Section 3.2 specifies local calibration updates with learning rates indexed by each client's observation count. However, Assumption 3 introduces a task-level learning-rate convention based on the initialization round. Furthermore, Theorem 1 defines a reference learning rate involving the server-round index, aggregation interval, and local task size.
  2. The experimental evaluation compares OAFCP only against the online synchronous federated conformal prediction baseline (OSFCP). It would be important to compare against additional approaches, such as asynchronous federated learning combined with local online conformal calibration, and alternative aggregation-weighting strategies.
  3. The paper lacks ablation studies examining the individual contributions of asynchronous aggregation, client-frequency compensation, and joint model-threshold updates.
Questions for authors

Can the authors provide stronger baselines and ablation studies? How does OAFCP compare with an asynchronous federated learning method equipped with independent local online conformal calibration? What happens when the participation-frequency correction is removed or replaced with uniform aggregation weights?

Reviewer_6ZHiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes an Online Asynchronous Federated Conformal Prediction (OAFCP) framework designed for heterogeneous data streams by integrating online conformal prediction with asynchronous federated learning. Addressing the challenges of online model learning and calibration threshold updates across heterogeneous clients possessing streaming data, the framework utilizes fixed-interval weighted aggregation at the server to achieve uncertainty quantification, offering significant research value. Under conditions involving bounded scores, controlled delays, appropriate step sizes, and additional distributional stability, the paper presents theoretical results concerning finite-time error bounds for weighted long-term empirical coverage, almost sure convergence of global and local calibration thresholds, and asymptotic validity of client-side weighted instantaneous conditional coverage, thereby providing theoretical guarantees. Furthermore, simulations and experiments on six real-world datasets demonstrate that the method outperforms synchronous baselines in terms of calibration and interval efficiency in scenarios involving partial regression distribution shifts, although its advantages in classification tasks are relatively limited. However, issues remain regarding the consistency between the calibration updates implemented in the article and the assumptions used in the proofs, as well as the extent to which theoretical guarantees support the stated primary coverage objective.

Critical strengths
  1. This paper combines online learning and conformal prediction, allowing asynchronous communication. The research problem has practical significance and, to some extent, fills the research gap between prediction uncertainty quantification and asynchronous federated learning.

  2. The paper does not forcibly treat model update and threshold update as the same optimization objective, but proposes a joint update mechanism for model learning and conformal calibration with clear structure and explicit mathematical meaning.

  3. The formula for the frequency compensation weight is concise and the physical explanation is clear. By weighting client updates according to their participation periods, the framework preserves the prescribed client importance weights on average over complete scheduling cycles under the fixed-duration model.

  4. The experimental design of the article has a wide coverage, including synthetic simulated data and six real data sets, covering regression, classification, distribution variation and hardware heterogeneity.

Critical weaknesses
  1. Section 3.2 defines the local calibration step size according to the client's observation index, namely ηt\eta_t. At the same time, the article mentions “the kth observation in this task is denoted by $ (x_{t}^{i},y_{t}^{i})$, where t=ni(tρi(n))+k,k=1,…,Kt=n_{i}(t^{\rho_{i}(n)})+k,k=1,\ldots,K”. However, Assumption 3 additionally specifies that the task-level step sizes satisfy ηi,kρi(n)=ηρi(n)\eta_{i,k}^{\rho_i(n)}=\eta_{\rho_i(n)}. These definitions are generally different.

  2. There is a discrepancy between the stated primary coverage objective and the theoretical guarantees. Section 2 explicitly identifies pooled coverage over all completed predictions as the primary objective. However, Theorem 1 establishes finite-horizon bounds for aggregation-weighted empirical coverage rather than the original criterion. Although the authors acknowledge this distinction in Section 4, the manuscript does not establish sufficient conditions under which the weighted coverage guarantees imply the stated pooled-coverage objective. Corollary 1 considers client-importance-weighted instantaneous conditional coverage. Its weighting structure is compatible with that of Theorem 1 under periodic scheduling, but it does not independently establish the pooled-coverage objective.

  3. There are some inconsistencies between the assumptions and the scope of application of the convergence guarantee. For example, Theorem 2 explicitly lists Assumptions 1, 2, 4, 5 and 6, but its proof uses the polynomial step size condition in Assumption 3 (including 1/2<a+b<11/2 < a + b < 1 and its related summability property). These conditions should be explicitly included in the statement of the theorem.

  4. The article lacks a concluding summary, a discussion of limitations, and a discussion of future research directions.

Questions for authors

Questions

  1. Does the implemented OAFCP algorithm use a distinct step size for every local observation, as specified in Section 3.2, or a constant task-level step size based on the server initialization round, as specified in Assumption 3? Could the authors provide the precise relationship between the local observation index, server round, and local calibration step size?

  2. Is the principal statistical target unweighted coverage across completed prediction events or client-importance-weighted coverage? Which of the theoretical coverage quantities is used to calculate the experimental “Long-run Coverage” and “Average Coverage” results?

  3. How is Equation (23) in Appendix D.3 derived? Given the reference sequence ηˉn=η0(1+nΔt+K)−a\bar\eta_n=\eta_0(1+n\Delta_t+K)^{-a}, how does the stated ratio involving (n+1)/(1+ρi(n))(n+1)/(1+\rho_i(n)) follow? Can the authors provide a corrected derivation and establish the required summability of the perturbation under the actual algorithm?

Suggestions

  1. Use consistent time indices throughout Sections 3–4 and Appendix D. Revisit the step-size comparisons, perturbation bounds, and limiting drift after this correction.

  2. The authors should clarify the intended primary coverage target, derive an appropriate relationship between the weighted and unweighted criteria, or revise the problem formulation to match the established guarantees. The exact weighting and normalization used for the empirical coverage metrics should also be specified.

  3. The authors should rederive the step-size ratios from the actual update rule, establish a valid perturbation bound, and explicitly verify the limiting drift and mixture weights.

  4. The author should provide complete theorem assumptions, specify the required convergence form, and distinguish the cases covered by the theoretical results.

  5. The article should further refine and expand upon the discussion regarding limitations and empirical results.

Reviewer_9VcaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes conformal prediction asynchronous federated learning using subgradient on pinball loss for each client to learn a standard online nonconformity threshold. They extend coverage bounds for importance-weighted mixture of client score distributions reusing techniques from FedFix and online conformal prediction. Small scale experiments are conducted with linear models on standard datasets such as MNIST and CIFAR with 10 clients report expected coverage on both regression and classification tasks.

Critical strengths

The problem of conformal prediction for streaming, heterogenous clients is under-explored. The proposed algorithm is lightweight and scalar valued threshold compatible with both regression and classification settings. The paper provides both worst case and asymptotic guarantees.

Critical weaknesses

The paper neglects prior work in federated conformal prediction (Lu 2023, Humbert 2023, Plassier 2023) and does not compare against proper baselines. The experiments are small scale and only use linear models and short time horizons. The asynchrony is simulated with fixed delays and missing experimental details for reproducibility. The step size indexing seems inconsistent mixing observation indices and round index steps. The design does not compare against independent client quantiles, which already achieve coverage while sharing the threshold only guarantees client weighted average coverage.

Questions for authors
  • which step size indexing is used for experiments? Does threshold converge empirically?
  • Can rerun experiments with random delays/dropoputs and larger models and compare against each client using its own online update for conformal thresholds?
  • Report top 1 accuracy, fraction of empty sets, and set size distribution for all datasets. How does dirichlet label skew affect reported accuracy.
  • how does your setting differ from pre existing federated work in conformal prediction? I suggest implementing and comparing more baselines and settings
Reviewer_DMZ1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes OAFCP for uncertainty quantification over asynchronous federated data streams. Clients update predictive models and conformal thresholds locally, while a server periodically aggregates stale increments using weights that compensate for unequal participation. The paper claims long-run coverage and, assuming stabilizing score distributions, convergence to a common quantile with nominal client-weighted instantaneous coverage. Experiments compare OAFCP with a synchronous counterpart on synthetic shifts and six real datasets, reporting regression improvements and mixed classification results.

Critical strengths
  1. The explicit task protocol provides a concrete formulation of streaming calibration under asynchronous federated updates.
  2. Table 1’s abrupt-drift F90 results report lower loss and narrower intervals, providing preliminary evidence of practical value for the complete pipeline.
Critical weaknesses
  1. The convergence claim fails for the stated recursion, and correcting it does not establish the pooled objective.
  • Equation (2) and Appendix D.2 index ηt\eta_t by client observation count, whereas Assumption 3 and Appendix D.3, Eq. (23), use a task/server-round clock. With ci=⌈τi/Δt⌉c_i=\lceil\tau_i/\Delta t\rceil, local counts grow as Kn/ciKn/c_i. Thus di=cipid_i=c_ip_i cancels participation frequency but leaves a learning-rate factor ciac_i^a. For fixed score distributions, the averaged drift uses p~i∝picia\widetilde p_i\propto p_ic_i^a, rather than pip_i. Bounded staleness does not remove this discrepancy.
  • Take K=Δt=γ0=1K=\Delta t=\gamma_0=1, c=(1,4)c=(1,4), p=(1/2,1/2)p=(1/2,1/2), a=3/4a=3/4, b=0b=0, α=0.1\alpha=0.1, q0=1q^0=1, and independent score draws S1∼U[0,2]S^1\sim U[0,2], S2∼U[0,1]S^2\sim U[0,1]. The intended quantile is q∗=1.6q^*=1.6. Convergence of the Section 3/D.2 recursion to this value would imply conditional drift in four-round cycle vv of 0.2η0(4−3/4−1)v−3/4+o(v−3/4)0.2\eta_0(4^{-3/4}-1)v^{-3/4}+o(v^{-3/4}). This negative drift is nonsummable, while the martingale noise in observation-revelation order has summable conditional variances. Hence this recursion cannot have the claimed almost-sure limit.
  • Independently, Section 2 specifies unweighted pooled coverage for arbitrary prescribed pip_i, while Section 4 analyzes compensated coverage. Even a corrected method reaching q∗=1.6q^*=1.6 in this example has pooled coverage 0.8(0.8)+0.2(1)=0.840.8(0.8)+0.2(1)=0.84. Section 4 acknowledges the distinction but does not establish the original objective. Any revised objective must also match the reported evaluation metric.
  1. The statistical contribution of threshold sharing is not established beyond existing components.
  • FedFix [1] supplies the aggregation mechanism and exact frequency weights. The local calibration update and its analysis with evolving predictors are established in [2]. Both are cited. Local/server quantile optimization in [3] is omitted, although its calibration setting differs. These precedents leave the asynchronous interaction as the contribution requiring separate justification.
  • Under Assumption 1, persistent local calibrators from [2, Theorem 1] already guarantee pooled long-run coverage for fixed MM, though not a common threshold. They can use identical asynchronous predictor trajectories because qq does not enter model updates. The OSFCP comparison changes predictor learning and calibration simultaneously. Lower predictive loss therefore cannot establish a threshold-sharing benefit, and narrower sets may follow from improved predictors. An additional guarantee or efficiency benefit from sharing thresholds remains to be established.
  • Appendix A.1 shares the regression function and noise distribution across clients. With correctly fitted absolute-residual predictors, their score distributions coincide, making the limiting quantile insensitive to normalized mixture weights. The stationary regression results therefore do not verify frequency compensation under persistent score heterogeneity.

[1] Fraboni, Vidal, Kameni, and Lorenzi. A General Theory for Federated Optimization with Asynchronous and Heterogeneous Clients Updates. JMLR, 2023.

[2] Angelopoulos, Barber, and Bates. Online Conformal Prediction with Decaying Step Sizes. ICML, 2024.

[3] Plassier, Makni, Rubashevskii, Moulines, and Panov. Conformal Prediction for Federated Uncertainty Quantification Under Label Shift. ICML, 2023.

Questions for authors
  1. What calibration step-size expression was implemented, and what limiting client weights does it induce? Please reconcile it with Equation (2), Assumption 3, Appendix D.2, and Eq. (23).
  2. Which coverage quantities do Tables 1–2 and Figure 4 evaluate? Specify the weights, denominators, instantaneous-coverage estimator, and the claim each metric tests. How is the pooled-coverage counterexample addressed?
  3. What additional guarantee or efficiency benefit does threshold sharing provide beyond persistent local calibration on identical predictor trajectories? Which theorem or controlled comparison establishes that contribution?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.