acceptodds
Under review as a conference paper at ICLR 2027

D-ICL: Decentralized In-Context Learning with Tabular Foundation Models

Abstract

Privacy-constrained tabular learning presents a fundamental challenge: data are distributed across silos that cannot share raw records, while each silo alone lacks sufficient coverage for accurate modeling. We introduce Decentralized In-Context Learning (D-ICL), a framework that enables effective collaboration by leveraging the in-context learning capability of frozen tabular foundation models, without sharing either data or model parameters. Each of agents maintains a private context and exchanges only predictive distributions over a shared query set ( is the number of agents). These predictions are aggregated to construct high-confidence pseudo-labels, which iteratively augment local contexts and drive collective improvement. We provide three theoretical guarantees: (i) a -fold variance reduction via consensus aggregation, (ii) monotonic performance improvement under a calibrated high-confidence pseudo-labeling scheme, and (iii) an convergence rate of the consensus loss over iterations. Empirically, across diverse tabular benchmarks under both IID and non-IID partitions, D-ICL consistently narrows the gap between isolated agents and a centralized oracle, often approaching centralized performance while strictly preserving data locality. The code base is available here.

Primary area:
optimization
Keywords:
Decentralized learning, in-context learning, tabular foundation model

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_9oUkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies decentralized learning with tabular foundation models(TFMs), where clients cannot share their raw data. The core of the proposed framework is to let different clients make predictions on a shared unlabeled dataset and aggregate these predictions to generate pseudo-labels, which are used to enrich their local contexts. The paper also gives theoretical results on variance reduction and convergence, and evaluates the method on several datasets under IID and non-IID settings.

Critical strengths
  1. The empirical evaluation is fairly broad, covering both classification and regression tasks, IID and non-IID settings and different communication topologies.
Critical weaknesses
  1. The main idea of D-ICL is closely related to existing consensus methods in FL [1].
  2. The baseline comparison is limited. Since D-ICL mainly relies on prediction sharing, more direct comparisons with federated distillation or other prediction-sharing methods are needed to show its advantage[2,3,4,5].

Refs:

[1] FedHarmony: Harmonizing Heterogeneous Label Correlations in Federated Multi-Label Learning

[2] FedMD: Heterogenous Federated Learning via Model Distillation

[3] Ensemble Distillation for Robust Model Fusion in Federated Learning

[4] Federated Model Distillation with Noise-Free Differential Privacy

[5] IMFLKD: an incentive mechanism for decentralized federated learning based on knowledge distillation

Questions for authors
  1. How is the shared unlabeled query pool obtained in practice?
  2. How much of the gain comes from iterative context enrichment rather than the initial consensus? D-ICL@0 already benefits from prediction averaging and the added value of repeated pseudo-labeling is not fully clear.
  3. How does D-ICL compare with simply labeling a small part of the shared pool under a similar budget?
Reviewer_C5CDOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new protocol for decentralized learning (DL) under a privacy-preserving setting. Each agent has a secret initial data set. The goal of DL in this paper is for all agents to have the agreed-upon label for each held-out sample in X∙X_\bullet, while keeping the initial dataset Ck\mathcal{C}_k secret. This setting is a decentralized prediction setting with a shared query pool.

The main novelty of the protocol lies in a context enrichment mechanism, where each agent select "good" samples from a shared query pool Q\mathcal{Q} to grow the Ck\mathcal{C}_k every iteration. The expanded local data set Ckt\mathcal{C}_k^t at the tt-th round is utilized to produce an improved prediction.

The authors conducted theoretical analysis to show that the federated predictive variance across the agent can be lowered by this enrichment process that selectively takes in easy-to-predict samples.

The authors also claim that this mechanism fits well the operational protocol of tabular foundation models (TFM).

Critical strengths
  • Proposed a DL protocol that fits in-context learning and hence, TFMs.
  • Provides a theoretical analysis showing that the proposed context enrichment procedure can reduce predictive variance.
Critical weaknesses

Problem setting and its positioning

The paper assumes that all clients are interested in predicting labels for a shared query pool. While this assumption enables consensus-based pseudo-labeling, it appears more restrictive than many decentralized learning scenarios where each client has its own downstream prediction set, e.g., https://ieeexplore.ieee.org/abstract/document/9592389 .

However, the paper does not provide a sufficiently precise characterization of the problem setting and its assumptions, which negatively affects readability and reproducibility.

Distributional Heterogeneity

The notion of a high-confidence sample appears to be defined through inter-client agreement. Does the method assume a shared conditional distribution P(y∣x)P(y|x) across clients? Under significant distribution shift, agreement does not necessarily imply correctness, and the generated pseudo-labels may not correspond to any client-specific target distribution. If the conditional distribution varies across clients, it is unclear why consensus pseudo-labels should remain meaningful.

Furthermore, the role of the stratified sampling mechanism introduced in Section 4 remains unclear. While it may improve the representativeness of the local context, it does not appear to address the underlying client-level distribution shift induced by heterogeneous data partitions. If the method implicitly assumes balanced local distributions or a shared conditional distribution across clients, these assumptions should be stated explicitly.

Aggregation weight

The paper does not clearly explain how the aggregation weights are determined. Since the final consensus prediction depends directly on these weights, additional discussion is necessary. It is also unclear whether the weighting scheme is fixed, data-dependent, or performance-dependent.

Relevance to TFM

It is unclear to what extent the observed gains originate from the proposed TFM-based formulation versus the iterative pseudo-labeling and context expansion mechanism. A comparison against decentralized self-training based on conventional supervised models would be necessary to isolate the contribution of the proposed TFM-based formulation. More specifically, an ablation comparing (1) local ICL, (2) prediction aggregation only, and (3) aggregation with context augmentation would help isolate the contribution of the proposed method.

Questions for authors

Please comment on the points rained in the weakness section. In particular, it would be helpful to discuss how the proposed framework extends to heterogeneous test distributions or multi-task settings in which clients do not share a common prediction target.

Reviewer_wDBYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes D-ICL, a collaborative learning framework based on decentralized predictive models. A key feature of the proposed method is that it does not require the sharing of data, gradients, or model parameters; instead, agents exchange predictive distributions and aggregate their predictions.

In the experiments, the authors demonstrate that D-ICL improves both predictive accuracy and RMSE across a variety of datasets, while also reducing prediction variance as theoretically expected. The study further investigates the effective ranges of the hyperparameter τ and the number of agents K.

Based on the experimental results, the authors claim that, by transmitting predictive distributions rather than gradients, D-ICL can reduce communication costs by several orders of magnitude compared with gradient-based learning approaches. In addition, the proposed method maintains a fully static context and therefore requires relatively little memory, making it well suited for deployment on resource-constrained devices.

Critical strengths

Unlike conventional Federated Learning, D-ICL does not transmit gradients or model update information; instead, it communicates only predictive distributions, thereby substantially reducing communication costs.

Furthermore, the proposed framework is supported by several theoretical guarantees, including variance reduction through distributed consensus, monotonic performance improvement enabled by high-confidence pseudo-labels, and a convergence rate of O(1/T). These properties provide a strong theoretical foundation for the effectiveness and stability of the method.

Critical weaknesses

While D-ICL substantially reduces privacy risks by avoiding the sharing of raw data and model parameters, it does not provide any formal privacy guarantees, such as differential privacy or cryptographic protection. The paper itself explicitly acknowledges this limitation, stating that although D-ICL enhances privacy preservation, it offers no theoretical privacy guarantees and therefore requires additional security measures for deployment in real-world applications.

Although D-ICL is theoretically considered to possess a certain degree of robustness to non-IID data, both the theoretical analysis and experimental results indicate potential limitations when data distributions differ substantially across agents or when severe class imbalance is present. Under such conditions, incorrectly generated pseudo-labels may propagate through the network and become amplified over time, potentially degrading overall model performance.

Questions for authors

I believe that the two most critical limitations of D-ICL are its lack of formal privacy guarantees and its vulnerability to non-IID data. Rather than relying on future developments such as enhanced privacy-preserving mechanisms or adaptive aggregation and pseudo-labeling strategies, I would like to focus on the proposed method itself. Specifically, to what extent can these two limitations be practically overlooked in the current D-ICL framework? Furthermore, even in the absence of formal privacy guarantees and additional safeguards against non-IID data, can the proposed method be considered sufficiently reliable and safe for real-world deployment?

Reviewer_4ctNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Decentralized In-Context Learning, a framework that enables collaborative learning across distributed data silos using frozen tabular foundation models without sharing raw data, gradients, or model parameters. The key idea is to exchange predictive distributions over a shared unlabeled query set, aggregate them through decentralized consensus, and iteratively enrich local contexts with high-confidence pseudo-labels. The authors provide theoretical analyses of variance reduction, monotonic improvement under calibration assumptions, and an O(1/T)O(1/T) convergence rate up to a heterogeneity-dependent error term. Experiments using TabICL and TabPFN across multiple classification and regression benchmarks under IID and non-IID settings demonstrate improvements over isolated agents, competitive performance relative to centralized learning, and substantially reduced communication costs compared to gradient-based approaches.

Critical strengths
  1. The paper explores an interesting and practical setting where multiple agents collaborate using frozen tabular foundation models without exchanging data or model parameters. This offers a lightweight alternative to conventional federated learning.

  2. The experiments cover multiple datasets, backbones, and data partition settings. The results demonstrate the benefits of prediction aggregation and suggest that consensus-based context enrichment can further improve performance.

Critical weaknesses
  1. The additional benefit of iterative context enrichment is not consistently demonstrated. In several settings, D-ICL@T performs similarly to or worse than D-ICL@0, suggesting that much of the improvement comes from simple prediction ensembling rather than the proposed iterative mechanism.

  2. The theoretical results rely on strong assumptions about pseudo-label quality and per-round loss reduction. It is unclear how well these assumptions hold in practice, particularly under non-IID settings. The lack of stronger federated and semi-supervised baselines also makes it difficult to assess the method's practical advantage.

Questions for authors
  1. Why does iterative context enrichment sometimes degrade performance compared to D-ICL@0? Can the authors provide more analysis of pseudo-label accuracy across rounds?

  2. Is there empirical evidence supporting the calibration and constant-contraction assumptions used in Propositions 2 and 3?

  3. How does D-ICL compare with stronger semi-supervised or federated learning baselines under comparable computational and communication budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.