acceptodds
Under review as a conference paper at ICLR 2027

FedTau: Correcting Observation-Induced Target Mismatch in Federated Learning

Abstract

Federated learning (FL) trains a shared predictor from heterogeneous client data, but existing methods mainly address heterogeneity after local objectives have already been defined. We identify an earlier source of discrepancy: clients may observe the same underlying prediction task through different sampling or recording mechanisms, causing their local supervision to encode different prediction targets. We refer to this phenomenon as observation-induced target mismatch. To address this, we formulate FL as learning a shared reference response distribution through client-specific observation mechanisms and propose FedTau, a unified -correction that corrects local supervision where the mismatch first occurs. For each observed record, FedTau evaluates the shared predictor through the corresponding observation mechanism and defines as the difference between the observed-data negative log-likelihood and the conventional supervised loss. Theoretically, we show that the corrected client objectives can achieve a common predictive optimum and analyze the effect of observation-mechanism estimation errors on local updates and aggregation. Experiments on nine datasets against twenty-three distinct baselines demonstrate consistent improvements across all three tasks. In particular, FedTau improves CIFAR-10 accuracy by over the second-best under severe class-prior heterogeneity, reduces RMSE by – across three right-censored regression datasets, and improves both token likelihood and token accuracy on all three sequence-prediction datasets.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Learning, Supervision Correction, Heterogeneous Observation, Local Objective Correction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kmkJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FedTau, an observation-aware framework for federated learning that attributes certain forms of local target discrepancy to client-specific observation mechanisms. The paper introduces a shared reference response distribution p∗p^*, models each client’s observed data as a transformation of this reference distribution through an observation mechanism TkT_k, and derives a τ\tau-correction that converts the conventional supervised loss into the corresponding observed-data negative log-likelihood. The framework is instantiated for class-selective sampling in classification, right-censored responses in regression, and truncated targets in sequence prediction.

Critical strengths
  1. Unified formulation across heterogeneous learning tasks. The idea of expressing local supervision through an observation-aware likelihood provides a coherent mathematical formulation that connects class-selective sampling, censoring, and sequence truncation. The resulting common-optimum analysis is conceptually interesting.

  2. Simple integration with federated optimization. FedTau modifies the local objective while retaining standard model architectures and FedAvg-style aggregation, making the framework potentially compatible with existing FL algorithms without requiring a new aggregation procedure.

Critical weaknesses
  1. The central concept and modeling assumption are insufficiently justified. The proposed “observation-induced target mismatch” appears to unify several established problems, including label distribution shift, censoring/capping, truncation, and measurement or label corruption. The paper does not yet establish what is fundamentally new about treating these phenomena as a single class of problems, nor whether there exists a sufficiently general observation model with a clear statistical interpretation that meaningfully unifies them. More importantly, the assumption that all client observations arise from a shared reference response distribution through client-specific observation mechanisms may be too strong for realistic FL, where clients can have genuinely different populations or conditional relationships.

  2. The practical availability and estimation of the observation mechanism and reference distribution are unclear. FedTau requires the client-specific observation mechanism to construct the τ\tau-correction, while the theoretical guarantee relies on the true TkT_k. However, clients may only have access to the final observed datasets and may not know how the data were generated or transformed. It is therefore unclear when TkT_k is identifiable, how T^k\hat T_k is obtained in practice, and how errors in T^k\hat T_k affect the claimed common optimum. The shared reference distribution p∗p^* also appears difficult to determine when the deployment distribution is unavailable. These assumptions substantially affect the practical validity of the proposed method.

Questions for authors
  1. What is the precise statistical definition of “observation-induced target mismatch”? How does it differ from existing notions such as label shift, covariate shift, concept shift, censoring, truncation, and label/measurement corruption? What property makes these seemingly different problems instances of the same phenomenon?

  2. What is the source of the shared reference distribution p∗p^* in practice? Is it assumed to be known, externally specified, or estimated from client data? If it is estimated, how is it distinguished from the heterogeneous client distributions? What information is assumed to be available for estimating T^k\hat T_k? Can TkT_k be identified from the observed client data alone, or does FedTau require additional metadata or knowledge of the data-collection process?

  3. The theoretical result is derived using TkT_k, whereas the practical method uses T^k\hat T_k. What conditions or quantitative guarantees ensure that the common-optimum property remains approximately valid when T^k≠Tk\hat T_k\neq T_k?

  4. In the classification experiments, Dirichlet label-skew partitioning produces conventional differences in local class distributions but does not explicitly construct an observation mechanism that transforms a shared reference target. How exactly does this setting instantiate the proposed observation-induced target mismatch? Are the observation mechanisms used in all experiments treated as oracle information? If so, how robust is FedTau when these mechanisms are estimated or misspecified?

5 comments
AuthorsOpenReview ↗

Response by Authors (Part 1)

We value the reviewer's positive assessment of FedTau's unified mathematical formulation, common-optimum analysis, and compatibility with existing FL methods. We will further clarify our statistical assumptions, practical applicability, and theoretical guarantees under mechanism-estimation errors.

W1 & Q1: Novelty and statistical definition

The main novelty is that we are the first to address the client-drift issue (caused by non-i.i.d. data) from where it actually first occurs. We redefine and correct the local learning objectives/observation for each client and align them to obtain a global predictor, whereas existing methods optimize training from a given specified local objective/target, but cannot correct the target that was already defined from a distorted observation (discrepancy). FedTau addresses this fundamental issue by correcting how local supervision is defined, rather than merely improving how predefined local objectives are optimized. This is the core gap that existing methods cannot jump over.

More precisely, observation-induced target mismatch occurs when directly fitting locally recorded observations leads to a population-optimal predictor different from the predictor required by the shared deployment task. This mismatch persists even with unlimited data and perfect local optimization. For example, as shown in Eq. (2) in the main paper, directly fitting U=min⁡(Y,ck)U=\min(Y,c_k) with squared error learns E[U∣X]\mathbb E[U\mid X] rather than E[Y∣X]\mathbb E[Y\mid X], potentially producing different prediction targets across clients with different thresholds.

FedTau provides a unified statistical formulation for this problem, rather than merely grouping existing task-specific corrections. Eq. (1) distinguishes the shared reference distribution p⋆p^\star from client observations, while Eq. (3) constructs local supervision from the observation likelihood. Appendix Eq. (11) further defines a general observation model:

\frac{\int R_k(A\mid y,x)s_k(y,x)q(dy\mid x)} {\int s_k(y,x)q(dy\mid x)},$$ where $s_k$ describes response-dependent sampling and $R_k$ describes how selected responses are recorded. This formulation covers class-selective sampling, right-censoring, sequence truncation, and other specified observation processes. These phenomena have different statistical forms, but share the same underlying issue: the observation mechanism changes the supervision received by the model relative to the intended reference prediction. FedTau derives the corresponding local objectives from the same observation-likelihood principle. Proposition 1 establishes that, under correctly specified observation mechanisms, we have $\mathbb E_{X\sim p_k^X}D_{\mathrm{KL}}\left(\mathcal T_k[p^\star](\cdot\mid X)\Vert\mathcal T_k[q_\theta](\cdot\mid X)\right)\geq0.$ Therefore, although clients have different input distributions $p_k^X$ and different observation mechanisms, our proposed $\tau$-correction can still minimize every client's local objective, referred to as a common predictive target. **W2 & Q2**: Practical availability and estimation FedTau does not require the true reference response distribution $p^\star(y\mid x)$ to be known before training. It is the underlying prediction distribution that the shared model $q_\theta$ aims to learn. What must be specified is the intended deployment task and sufficient information about how each client records its observations. In practice and also in our experiments, the required information has explicit sources and is used locally without transmitting raw observations. - Classification: Local class priors are estimated from client training-label counts. The reference prior is constructed from aggregated class counts using Eq. (9), $\widetilde\pi^\star(c)=(N_c+1)/(N+C)$, through federated aggregation procedures, or can be externally specified for a particular deployment task. - Right-censored regression: The censoring threshold $c_k$ and indicator $\delta$ come from the recording protocol. A censored record provides the event $Y\geq c_k$, even when the exact response is unavailable. - Sequence prediction: The observed prefix and continuation indicator are available from the truncation procedure. When a record is known to continue, the corresponding boundary probability is $1-p_{\mathrm{EOS}}$. We acknowledge that arbitrary observation mechanisms cannot always be identified from final recorded datasets alone. FedTau assumes that the relevant recording rules are available from the collection protocol, retained metadata, or estimable statistics. This is an explicit applicability condition (necessary).
AuthorsOpenReview ↗

Response by Authors (Part 2)

Q3: Tk\mathcal T_k vs. T^k\widehat{\mathcal T}_k

There exists an underlying observation mechanism Tk\mathcal T_k, but it may not be directly accessible. Therefore, our practical objective is to obtain an estimated mechanism T^k\widehat{\mathcal T}_k that approximates Tk\mathcal T_k as closely as possible. Using estimated quantities rather than their exact counterparts is common in optimization and statistical learning. The important question is how estimation errors affect the theoretical properties and practical performance of FedTau.

In fact, Appendix A.5.4 and Eq. (20) already provide quantitative answers to this question. We define the residual gradient at the reference optimum as εk=∥∇Gk(θ⋆)∥2\varepsilon_k=\|\nabla G_k(\theta^\star)\|_2, which measures the effect of mechanism-estimation errors on local optimization. Eq. (26) shows that the distance between a client's updated model and the reference optimum is upper-bounded by

\leq r^{H_k}\|\theta_t-\theta^\star\|_2 + (1-r^{H_k})\frac{\varepsilon_k}{m},$$ where $r=1-\eta m$. Equation (27) extends this bound to federated aggregation. Thus, when $\varepsilon_k\leq\bar\varepsilon$ for all participating clients, the residual error is controlled by $\bar\varepsilon/m$. Furthermore, Eq. (20) directly upper-bounds the residual gradient by the estimation error in the classification logit offset, i.e., $\varepsilon_k\leq\frac{M}{2}\|b_k-b_k^\star\|_2$. These results establish that **smaller mechanism-estimation errors lead to tighter bounds on the deviation from the shared reference optimum**, under the specified theoretical conditions. Our mechanism-mismatch experiments in Tables 4 and 12 provide complementary empirical evidence. These results are consistent with our theoretical analysis, demonstrating that inaccurate observation mechanisms can introduce additional optimization errors and degrade prediction performance. **Q4**: Observation mechanism in Dirichlet label-skew partitioning The Dirichlet label-skew partitioning itself is an observation mechanism that FedTau can learn. The observation mechanism can change which kind of samples are seen by the client. Under the label-shift assumption, clients share the same class-conditional feature distribution, \$ p_k(x\mid y)=p^\star(x\mid y), \$ but have different class priors, i.e., $\pi_k(y)\neq\pi^\star(y)$. According to Bayes' rule, \$ p_k(y\mid x)\propto p^\star(y\mid x)\frac{\pi_k(y)}{\pi^\star(y)}. \$ This relationship directly corresponds to the observation mechanism defined in FedTau's classification formulation (Eq. (4)). In our experiments, samples are randomly partitioned within each class according to class-specific Dirichlet proportions. Thus, the probability that a sample is assigned to a particular client depends on its label, which simulates different client-specific sampling mechanisms. Conventional local cross-entropy learns client-specific posteriors, whereas FedTau accounts for the prior differences to align local supervision with the shared reference target. Regarding oracle information, FedTau does not use the underlying Dirichlet allocation probabilities as oracle information. The local and reference class priors are estimated from the observed training-label counts. For right-censored regression, the censoring thresholds and indicators are obtained from the recording protocol, while sequence prediction uses the observed prefixes and continuation indicators. These mechanisms are therefore constructed from available training statistics or recording metadata, without requiring access to unobserved true responses. In Proposition 1, we prove that clients can train on their own observed data while sharing a common reference predictive optimum, even when their observation mechanisms differ.

Thank you to the authors for the detailed responses and additional clarifications. The revised explanations, particularly the estimation-error analysis in Q3 and the discussion of observation mechanisms in Q4, have addressed several of my initial concerns.

However, some questions remain. In particular, I am still not entirely clear about how the reference distribution can be reliably estimated from heterogeneous client data and whether the aggregated class priors necessarily represent the intended deployment distribution. I also believe the distinction between FedTau's unified formulation and existing approaches to label shift, censoring, and selection bias could be further clarified to better establish its novelty. Furthermore, the applicability of FedTau appears to rely on access to observation-related metadata or prior knowledge of the data-collection process, which is an important assumption worth emphasizing.

Overall, I appreciate the authors' efforts and the additional theoretical and empirical clarifications. While the responses have improved my understanding of the method, some questions regarding its practical applicability and novelty remain. Therefore, I have decided to maintain my original score.

Reviewer_mBP2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedTau, a federated learning framework that addresses observation-induced target mismatch arising when different clients observe the same underlying prediction task through different sampling or recording mechanisms. The key idea is to distinguish the reference response distribution from client-specific observed-data distributions and construct local supervision using the corresponding observation likelihood.

The proposed framework introduces a unified τ-correction, defined as the difference between the observed-data negative log-likelihood and the conventional supervised loss. The authors instantiate this principle for three settings: class-prior correction in classification, likelihood-based supervision for right-censored regression, and continuation-aware supervision for truncated sequence prediction.

The theoretical analysis establishes a common predictive optimum under correctly specified observation mechanisms and realizability assumptions, and provides bounds on the effects of mechanism-estimation errors. Experiments across nine datasets demonstrate improvements over a broad set of federated optimization baselines.

Critical strengths
  1. A useful formulation of observation-induced target mismatch in federated learning.

The paper highlights an important distinction between heterogeneous optimization objectives and heterogeneous observation mechanisms. In particular, it correctly emphasizes that improving local optimization or aggregation cannot necessarily recover the intended prediction target when the supervision itself reflects an incomplete or biased observation. Formulating this issue through a shared reference distribution and client-specific observation mechanisms provides a coherent perspective applicable to multiple prediction tasks.

  1. Broad empirical evaluation demonstrating the practical importance of observation-aware supervision.

The paper evaluates the proposed approach across classification, censored regression, and sequence prediction, using nine datasets and a substantial collection of federated baselines. The reported improvements are particularly notable under severe class-prior heterogeneity and heterogeneous right censoring. The additional ablations on correction components, mechanism matching, and correction placement provide useful evidence that the observation model matters during local training.

Critical weaknesses
  1. Limited methodological novelty and missing direct baselines.

The three task-specific corrections largely build on established objectives: Balanced Softmax for classification, censored Gaussian likelihood for regression, and continuation likelihood for truncated sequences. Proposition 1 also follows directly from the KL divergence formulation. While the unified perspective is useful, the methodological novelty remains limited. Moreover, comparisons with more directly relevant baselines, such as FedLC and simple EOS masking, are missing, making it difficult to isolate the contribution beyond using appropriate supervision.

  1. Concerns about experimental evaluation and reporting.

The main results report per-seed best test-set performance across training rounds rather than validation-selected checkpoints, potentially introducing optimistic selection bias. Although single-run diagnostics are provided, consistent multi-seed validation-based results would strengthen the claims.

Questions for authors

Q1. Methodological novelty and comparison with existing approaches.

Given that the classification objective is equivalent to Balanced Softmax under a uniform reference prior, and that the regression and sequence objectives follow established likelihood principles, what is the specific methodological contribution beyond incorporating these losses into local federated training? Could the authors provide comparisons with directly relevant label-calibration approaches, such as FedLC, and discuss which aspects cannot be obtained through existing observation-aware supervision?

Q2. Sequence prediction: comparison with EOS masking.

The current sequence baselines appear to treat truncated prefixes as if they terminate with EOS. A natural alternative is to train on the observed prefix while simply masking the artificial EOS loss, without introducing the additional continuation-event term. How much of FedTau's improvement comes from removing incorrect EOS supervision, and how much comes from explicitly optimizing the non-EOS continuation probability? This comparison would materially strengthen the empirical contribution.

Q3. Checkpoint selection and experimental robustness.

Could the authors report the primary results across all five seeds using validation-selected checkpoints, rather than selecting the best test score independently for each seed? In particular, do the relative rankings and reported margins remain consistent for classification and sequence prediction under this protocol?

5 comments
AuthorsOpenReview ↗

Response by Authors (Part 1)

We are pleased that the reviewer acknowledges the importance of observation-aware supervision and the comprehensive experiments and ablation studies. We will further clarify our contributions, add EOS-masking comparisons, and explain our evaluation protocol.

W1 & Q1:

Methodological novelty. The main novelty is that we are the first to address the client-drift issue (caused by non-i.i.d. data) from where it actually first occurs. We redefine and correct the local learning objectives/observation for each client and align them to obtain a global predictor, whereas existing methods optimize training from a given specified local objective/target, but cannot correct the target that was already defined from a distorted observation (discrepancy). FedTau addresses this fundamental issue by correcting how local supervision is defined, rather than merely improving how predefined local objectives are optimized. This is the core gap that existing methods cannot jump over.

We cannot say that FedTau simply unifies existing methods, but it provides a general framework for correcting observation-induced target mismatch at its source, from which task-specific corrections can be systematically derived. FedTau analyzes the local observation process itself and investigates why a deployment task, after being sampled and observed by different clients, generates supervision to different objectives. After that, we theoretically prove that FedTau ensures that local supervision derived from different client observation mechanisms remains aligned with the same deployment target (a consistent predictive optimum). Mathematically, the τ\tau-correction is represented by the relations between p∗p^* shared reference response distribution, qθq_\theta predictive model, and Tk\mathcal T_k each client’s unique observation mechanism, as the difference between the observed-data negative log-likelihood and the conventional supervised loss, defined in Eq. (3) in the main paper.

FedTau does not claim that the individual likelihood corrections are fundamentally new. Instead, its main contribution is to identify observation-induced target mismatch as a federated learning problem and provide a unified framework for constructing observation-aware local objectives aligned with a shared deployment target. This framework applies to different prediction tasks with identifiable observation mechanisms, as demonstrated through classification, regression, and sequence prediction in this paper.

Methodological differences between FedTau and existing methods (e.g., FedLC). FedLC primarily addresses label-distribution heterogeneity in classification through class-dependent logit calibration. In contrast, FedTau starts from client-specific observation mechanisms and derives local supervision from the observed-data likelihood, aiming to align heterogeneous local objectives with a shared deployment target.

FedLC constructs class-dependent calibration margins from local class counts, whereas FedTau derives its classification correction from the relationship between local and reference class priors, i.e., bk,c=log⁡π~k(c)−log⁡π~⋆(c)b_{k,c}=\log\widetilde{\pi}_k(c)-\log\widetilde{\pi}^{\star}(c), and incorporates this offset into local supervision. When the reference prior is uniform, the objective recovers the client-side Balanced Softmax. When the local and reference priors are identical (bk=0b_k=0), the correction vanishes. Thus, Balanced Softmax can be understood as a special case of FedTau's classification formulation, demonstrating its consistency with established class-prior corrections.

FedLC is only applicable in classification-based calibration, while FedTau is applicable in any ML tasks as long as the observation processes exist. Therefore, the key methodological difference is that FedTau provides a unified observation-aware approach to constructing local learning objectives, rather than applying task-specific calibration to predefined supervision.

KL divergence of Proposition 1. The important thing of Proposition 1 is which distributions appear in the KL divergence and why they lead heterogeneous clients toward a shared predictive optimum. The KL divergence is just the math tool that helps understand the theory, not the contribution itself.

Each client's excess observation risk can be expressed as EX∼pkXDKL(Tk[p⋆](⋅∣X),∥,Tk[qθ](⋅∣X))≥0.\mathbb E_{X\sim p_k^X}D_{\mathrm{KL}}\left(\mathcal T_k[p^\star](\cdot\mid X),\Vert,\mathcal T_k[q_\theta](\cdot\mid X)\right)\geq 0. Therefore, although clients have different input distributions pkXp_k^X and different observation mechanisms, our proposed τ\tau-correction can still minimize every client's local objective, referred to as a common predictive target. The proof is mathematically concise because Eqs. (1) and (3) in the main paper explicitly establish the shared learning target. The contribution is not a new property of KL divergence, but the construction and analysis of observation-aware federated objectives.

AuthorsOpenReview ↗

Response by Authors (Part 2)

Q2: Comparison with EOS masking

We thank the reviewer for suggesting this important comparison. We conduct an additional ablation study:

  • A — Artificial EOS: The truncated prefix is incorrectly treated as a complete sequence ending with EOS.
  • B — EOS masking: The artificial EOS loss is removed, without adding a continuation-event loss.
  • C — Continuation-aware supervision: The artificial EOS loss is replaced with the observed continuation-event loss, as in FedTau.

We have the following results (lower is better).

DatasetMethodA · Artificial EOSB · EOS maskingC · ContinuationA−B · gain from removing artificial EOSB−C · gain from continuation
XSumFedDC2.4409 ± 0.00052.4230 ± 0.00042.4070 ± 0.0004+0.0179+0.0160
XSumFedAdam2.4617 ± 0.00032.4370 ± 0.00052.4180 ± 0.0004+0.0247+0.0190
XSumFedDyn2.4418 ± 0.00142.4212 ± 0.00152.4050 ± 0.0012+0.0206+0.0162
XSumFedTau2.4551 ± 0.00062.4062 ± 0.00082.3836 ± 0.0008+0.0489+0.0226
SAMSumFedDC1.7350 ± 0.00061.7281 ± 0.00051.7233 ± 0.0006+0.0069+0.0048
SAMSumFedAdam1.7296 ± 0.00101.7222 ± 0.00091.7178 ± 0.0011+0.0074+0.0044
SAMSumFedDyn1.7348 ± 0.00181.7301 ± 0.00131.7260 ± 0.0016+0.0047+0.0041
SAMSumFedTau1.7301 ± 0.00251.7198 ± 0.00051.7105 ± 0.0014+0.0103+0.0093
CNN/DailyMailFedDC2.0531 ± 0.00062.0451 ± 0.00042.0401 ± 0.0005+0.0080+0.0050
CNN/DailyMailFedAdam2.0255 ± 0.00062.0212 ± 0.00052.0179 ± 0.0005+0.0043+0.0033
CNN/DailyMailFedDyn2.0359 ± 0.00052.0303 ± 0.00042.0308 ± 0.0004+0.0056−0.0005
CNN/DailyMailFedTau2.0187 ± 0.00122.0142 ± 0.00092.0109 ± 0.0010+0.0045+0.0033

These results show that FedTau's improvements do not arise solely from masking incorrect EOS targets (A-B). Explicit continuation supervision provides an additional reduction in test NLL on all three datasets (B-C).

W2 & Q3: Validation-selected checkpoint

Our main results are obtained from five independent random seeds. For each seed, we select the best test performance across all communication rounds under the same fixed training budget. We then calculate the mean and std across the five seeds. This evaluation metric is widely used in the existing literature in federated learning.

The validation-selected checkpoint is another evaluation metric to help test performance. We agree, and we are happy to provide the results under this metric. In fact, we have already included related evaluations in our appendix. Tables 16 and 19 show results using validation-selected checkpoints. Tables 17 and 18 show results at a fixed communication round. Although these experiments are based on a single seed, they provide additional evidence that FedTau's improvements are not limited to selecting the best test performance.

We are happy to provide five-seed validation-selected results. However, our experiments involve 23 distinct baselines across 9 datasets and three prediction tasks. Due to a limited rebuttal period, in this rebuttal, we only show the experimental results for NLP tasks. We will provide full results when they become available.

From the table, we observe that the relative ranking between these two metrics is almost the same. This is because the data distribution for the validation dataset and test dataset is the same.

MethodXSum NLL ↓XSum Acc. ↑SAMSum NLL ↓SAMSum Acc. ↑CNN/DailyMail NLL ↓CNN/DailyMail Acc. ↑
FedAvg2.4554 ± 0.000549.00 ± 0.031.7309 ± 0.002659.75 ± 0.092.0192 ± 0.001560.17 ± 0.08
FedProx2.4556 ± 0.000649.01 ± 0.041.7319 ± 0.002959.77 ± 0.072.0189 ± 0.000860.20 ± 0.05
FedAvgM2.4536 ± 0.000349.05 ± 0.021.7310 ± 0.001259.72 ± 0.042.0541 ± 0.000760.17 ± 0.03
FedAdam2.4619 ± 0.000349.02 ± 0.021.7301 ± 0.001359.81 ± 0.052.0257 ± 0.000759.99 ± 0.05
FedDyn2.4424 ± 0.001250.46 ± 0.041.7356 ± 0.002259.55 ± 0.072.0361 ± 0.000560.09 ± 0.02
FedDC2.4412 ± 0.000550.51 ± 0.011.7353 ± 0.000859.58 ± 0.052.0533 ± 0.000660.11 ± 0.04
FedExP2.4416 ± 0.000750.47 ± 0.031.7353 ± 0.001059.53 ± 0.082.0190 ± 0.000859.86 ± 0.03
FedSpeed2.4413 ± 0.000650.60 ± 0.051.7353 ± 0.000859.67 ± 0.072.2276 ± 0.000654.83 ± 0.05
FedLESAM2.4417 ± 0.001050.53 ± 0.051.7351 ± 0.000559.66 ± 0.042.0211 ± 0.000859.88 ± 0.04
FedTau2.3840 ± 0.000951.42 ± 0.021.7112 ± 0.001560.38 ± 0.052.0113 ± 0.001060.28 ± 0.05
Reviewer_mBP2OpenReview ↗

Partially addressed questions, but maintain my rating of 2. (I can give 2.4 if I could, but not a 3)

I thank the authors for the additional experiments.

Q2 is addressed. The A/B/C ablation is exactly the decomposition I asked for, and the B−C column shows the continuation term contributes beyond simply removing the artificial EOS target. One caveat: on CNN/DailyMail the margin (+0.0033) is close to the seed variation and FedDyn reverses sign (−0.0005), so I would report this unevenness rather than describe the gain as uniform.

Q1 is not addressed. The conceptual distinction from FedLC is argued clearly, but my question was empirical: no FedLC or Balanced Softmax numbers are given. This matters more after the response to Reviewer DqMv, where adding FedTau lifts ten structurally different methods by a uniform +10 to +18 points on CIFAR-10. Since the classification instantiation reduces to client-side Balanced Softmax under a uniform reference prior, a single baseline row under the same partition would separate the general framework from an established class-prior correction.

Q3 is partially addressed. I appreciate the five-seed validation-selected results on the sequence datasets. But the paper's headline claim is the classification margin, and classification and regression still rely on per-seed best-test selection; Tables 16 and 19 are single-seed. That protocol is most optimistic precisely where seed variance is largest, which is where the reported margin lives.

I therefore maintain my rating of 2.

Reviewer_DqMvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies an observation-induced target mismatch,which means client-specific sampling or recording mechanisms can cause different local prediction targets before optimization. To address this issue, it proposes FedTau, modeling client-specific observation mechanisms and introducing a unified τ\tau-correction to local supervision. Specifically, the τ\tau-correction transforms the conventional supervised loss applied directly to observed data into an observation-aware negative log-likelihood, with the goal of aligning heterogeneous client objectives toward a shared reference predictive target. The proposed framework is instantiated for class-selective sampling in classification, right-censored responses in regression, and truncated targets in sequence prediction. The paper provides theoretical analysis of the common predictive target and the effect of observation-mechanism estimation errors, and evaluate FedTau on 9 datasets against 23 baselines across the 3 tasks, reporting consistent improvements under the considered heterogeneous observation mechanisms.

Critical strengths

The authors provide a relatively novel perspective on data heterogeneity in federated learning by formulating client-specific observation mechanisms as a source of local target mismatch. Instead of only correcting client drift during local optimization or model aggregation, FedTau directly corrects the local supervision according to the observation mechanism.

The paper provides extensive experiments across a wide range of datasets and baselines. The consistent performance improvements across classification, censored regression, and sequence prediction demonstrate the effectiveness and general applicability of the proposed framework.

Critical weaknesses

Although the formulation is interesting, the three task-specific instantiations of FedTau are closely related to existing techniques. For example, the correction for classification is related to prior/logit adjustment and Balanced Softmax. Therefore, the main novelty appears to lie in unifying these existing observation-aware corrections under a federated learning framework, rather than introducing fundamentally new correction mechanisms.

Questions for authors

Could the authors further clarify the methodological differences between FedTau and existing loss-correction methods?

Can FedTau be combined with existing federated optimization methods that address client drift or local optimization inconsistency, such as FedProx, SCAFFOLD, or FedDyn? It would be interesting to investigate whether applying these federated optimization methods on top of the FedTau could provide further performance improvements.

4 comments
AuthorsOpenReview ↗

Response by Authors (Part 1)

We appreciate the reviewer's recognition of FedTau's novel perspective on FL heterogeneity and extensive experimental validation. We will further clarify our methodological novelty and provide additional experiments on integrating FedTau with existing FL methods.

W1: The main novelty is that we are the first to address the client-drift issue (caused by non-i.i.d. data) from where it actually first occurs. We redefine and correct the local learning objectives/observation for each client and align them to obtain a global predictor, whereas existing methods optimize training from a given specified local objective/target, but cannot correct the target that was already defined from a distorted observation (discrepancy). FedTau addresses this fundamental issue by correcting how local supervision is defined, rather than merely improving how predefined local objectives are optimized. This is the core gap that existing methods cannot jump over.

We cannot say that FedTau simply unifies existing methods, but it provides a general framework for correcting observation-induced target mismatch at its source, from which task-specific corrections can be systematically derived. FedTau analyzes the local observation process itself and investigates why a deployment task, after being sampled and observed by different clients, generates supervision to different objectives. After that, we theoretically prove that FedTau ensures that local supervision derived from different client observation mechanisms remains aligned with the same deployment target (a consistent predictive optimum). Mathematically, the τ\tau-correction is represented by the relations between p∗p^* shared reference response distribution, qθq_\theta predictive model, and Tk\mathcal T_k each client’s unique observation mechanism, as the difference between the observed-data negative log-likelihood and the conventional supervised loss, defined in Eq. (3) in the main paper.

Technically speaking, FedTau provides a unified principle applicable to different prediction tasks with identifiable observation mechanisms. In this paper, we demonstrate its applicability to classification, regression, and sequence prediction. For prior/logit adjustment and Balanced Softmax, they specifically address class-prior effects in classification. However, FedTau provides a broader observation-likelihood formulation that can naturally recover related corrections under specific assumptions. For classification, FedTau derives the client-specific logit offset bk,c=log⁡π~k(c)−log⁡π~⋆(c)b_{k,c}=\log\widetilde{\pi}_k(c)-\log\widetilde{\pi}^\star(c) and incorporates it into local supervision. When the reference prior is uniform, the objective recovers the client-side Balanced Softmax formulation. When the local and reference priors are identical (bk=0b_k=0), the correction vanishes. Thus, Balanced Softmax can be understood as a special case of FedTau's classification formulation, while FedTau’s observation-likelihood principle can be represented to other forms of target mismatch, including right-censored regression and truncated sequence prediction.

The table below shows the methodological differences between FedTau and existing approaches.

Representative methodsMain focusKey difference from FedTau
FedAvg, FedAdam, FedYogiParameter aggregation and server-side updatesThese methods operate on updates that have already been generated. FedTau determines which response distribution each client should learn from its observations.
FedProx, SCAFFOLD, FedDynProximal constraints, control variates, and dynamic regularizationThese methods coordinate optimization under predefined local objectives. FedTau constructs these objectives according to client-specific sampling and recording mechanisms to align them with the reference task.
FedNovaEffective objective reweighting caused by heterogeneous local training workloadsFedNova corrects the relative contributions of local objectives to global optimization. FedTau corrects the response distribution targeted by each local objective.
MOON, FedProto, FedTGPRepresentation consistency and class prototypesThese methods constrain or share learned representations. FedTau identifies the supervision event represented by each observation before training.
FedGen, FedFTG, FedLMGGenerative information and cross-client knowledge transferThese methods provide training signals through generation or distillation. FedTau directly interprets existing observations through their observation mechanisms to construct local supervision.
AuthorsOpenReview ↗

Response by Authors (Part 2)

Q1: Following the response to W1, FedTau changes the starting point for analyzing client drift (heterogeneity) in federated learning. Existing methods start from predefined local objectives and investigate how to improve local training, coordinate client updates, or aggregate models more effectively. FedTau takes one step further back and asks a more fundamental question: Have client-specific sampling and recording mechanisms already changed the prediction target before training? Once local supervision is directed toward an incorrect target, optimizing the loss more thoroughly only leads to a more accurate fit to that incorrect target. The key contribution of FedTau is to prevent observation-induced target mismatch from entering the training process when local supervision is first constructed. The loss correction is the means of implementing this idea, rather than the core conceptual contribution itself.

Consider right-censored regression as an example. Suppose a client can record response values only up to 80. For a censored record, the observed value of 80 indicates that the true response is at least 80. If this value is treated as an exact target, even perfect local optimization would still learn the conditional mean of the capped response rather than that of the original response. FedProx can constrain local updates, SCAFFOLD can correct update directions, and adaptive aggregation can adjust the contribution of these updates at the server. In contrast, FedTau directly corrects the supervision used to produce these updates: instead of treating 80 as an exact response, it trains the model using the likelihood of the observed event that the true response is at least 80. In this way, clients can learn a predictor for the original, uncensored response using only the information they actually observe.

Q2: Of course yes. We have stated in the main paper that FedTau corrects supervisions by only changing the local training objectives. It can be naturally implemented in optimization methods that rely on pre-defined training objectives.

We conduct an experiment on CIFAR-10 with a Dirichlet partition (α=0.1\alpha=0.1), 100 clients, and a 10% participation rate per communication round. Comparing the results in Table 1 in the main paper, we have the following results (FedTau performs 79.14 ± 2.05%).

MethodOriginal (%)Combined with FedTau (%)Gain over original (%)Difference from FedTau (%)
FedProx65.34 ± 4.6179.52 ± 1.43+14.18+0.38
FedProto66.08 ± 5.4280.03 ± 2.47+13.95+0.89
MOON61.57 ± 5.8079.49 ± 2.29+17.92+0.35
FedGen66.79 ± 5.1280.94 ± 2.63+14.15+1.80
FedRG65.73 ± 2.3180.42 ± 2.77+14.69+1.28
FedDyn64.42 ± 2.6677.83 ± 2.25+13.41−1.31
FedLMG65.59 ± 5.7978.58 ± 3.02+12.99−0.56
FedCPD63.79 ± 3.0176.37 ± 3.64+12.58−2.77
FedTGP66.82 ± 3.6177.19 ± 2.86+10.37−1.95
FedFTG64.04 ± 4.1476.95 ± 3.30+12.91−2.19

We observe that after combining with FedTau, all existing methods further improve their performance, demonstrating the scalability and reliability of FedTau. Surprisingly, we observe that some methods even outperform pure FedTau (because FedTau implements a simple FedAvg aggregation rule). Some other methods can still further improve performance but not beyond FedTau. Therefore, FedTau can be directly applied to existing methods with relatively simple designs that do not interfere with model learning and aggregation at the server side. However, for more complex methods, we still need efforts to address possible optimization conflicts between them and FedTau, and this is an interesting future research direction.

Thank you for the detailed response. The additional experiments combining FedTau with existing federated learning methods are very useful, and I encourage the authors to incorporate these results into the revised paper. I understand that the main contribution of FedTau lies in providing a unified observation-aware framework for identifying and correcting target mismatch before local optimization. However, I remain concerned that the methodological novelty is limited, as the task-specific corrections are closely related to established techniques. Nevertheless, I appreciate the authors’ efforts to address my concerns, particularly through the additional experiments, and have raised my score accordingly.

We thank the reviewer for the positive feedback and increasing the score. We are working on the paper modification, including putting the combination of FedTau with existing baselines. We still need to clarify that we provide a brand-new perspective to address the non-i.i.d. issue in federated learning. After theoretical analysis (mathematical transformation and analysis), we surprisingly find that existing established techniques can be represented by the reduced version of FedTau under specific conditions (e.g., FedTau reduces to Balanced Softmax when the reference prior is uniform). This is another contribution that unifies (at a higher level) existing approaches.

Once again, we thank the reviewer for increasing the rating to 3.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We appreciate the reviewers' thoughtful evaluations and constructive suggestions. We have provided detailed point-by-point responses and additional experimental results addressing the main concerns. Here, we briefly summarize our key clarifications and new evidence.

  1. The main novelty of FedTau is that we are the first to address the client-drift issue (caused by non-i.i.d. data) at its source (observation-induced target mismatch), before local optimization begins, while existing methods optimize training from a given specified local objective. This is the core gap that existing methods cannot jump over. FedTau provides a brand new perspective to address the client-drift issue, which has a fundamental impact in federated learning community. Proposition 1 establishes the common-optimum property under the stated assumptions, while Appendix A.5 analyzes identifiability and mechanism-estimation errors.

  2. In response to the reviewers' suggestions, we have conducted new experiments. First, combining FedTau with ten existing FL methods improves all ten methods over their standalone versions, with five combinations outperforming standalone FedTau. Second, the EOS-masking ablation demonstrates that continuation-aware supervision provides additional benefits beyond simply removing incorrect EOS targets. Third, we provide additional five-seed sequence-prediction results using validation-selected checkpoints, supporting the robustness of FedTau's improvements under this evaluation protocol.

  3. We further clarify how observation mechanisms are obtained from available training statistics or recording metadata. Our theoretical bounds quantify the effects of mechanism-estimation errors, while controlled ablations demonstrate the importance of correctly matching observation mechanisms to client data.

We hope these clarifications and additional results address the reviewers' main concerns. We would greatly appreciate any further feedback and would be happy to discuss or clarify any remaining questions.

AuthorsOpenReview ↗

Paper update

Dear all,

The revisions and additional experiments have been incorporated into the updated paper, highlighted in blue color.

The changes are as follows:

  1. We highlight the novelty of FedTau in the Introduction section and Methodology section, emphasizing that FedTau corrects observation-induced target mismatch when constructing local supervision and aligning clients with a shared reference prediction target.
  2. We add results for joint optimization of FedTau with 10 baselines. All ten combinations achieve higher accuracy than their original methods, and five also exceed standalone FedTau.
  3. We add experiments comparing artificial EOS, EOS masking, and continuation-event supervision.

We hope our response addresses all reviewers' concerns.

Thank you again for raising so many meaningful questions related to our paper.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.