D-ICL: Decentralized In-Context Learning with Tabular Foundation Models

Privacy-constrained tabular learning presents a fundamental challenge: data are distributed across silos that cannot share raw records, while each silo alone lacks sufficient coverage for accurate modeling. We introduce Decentralized In-Context Learning (D-ICL), a framework that enables effective collaboration by leveraging the in-context learning capability of frozen tabular foundation models, without sharing either data or model parameters. Each of $K$ agents maintains a private context and exchanges only predictive distributions over a shared query set ($K$ is the number of agents). These predictions are aggregated to construct high-confidence pseudo-labels, which iteratively augment local contexts and drive collective improvement. We provide three theoretical guarantees: (i) a $K$-fold variance reduction via consensus aggregation, (ii) monotonic performance improvement under a calibrated high-confidence pseudo-labeling scheme, and (iii) an $\mathcal{O}(1/T)$ convergence rate of the consensus loss over $T$ iterations. Empirically, across diverse tabular benchmarks under both IID and non-IID partitions, D-ICL consistently narrows the gap between isolated agents and a centralized oracle, often approaching centralized performance while strictly preserving data locality. The code base is available here.