acceptodds
Under review as a conference paper at ICLR 2027

Asking the Crowd the Right Question: Bias-Cancelling Weights for Federated Learning

Abstract

A federated objective is a weighted sum of client risks, and the weights are almost always fixed in advance. We treat them instead as the only instrument of a wisdom-of-crowds mechanism: clients are noisy views of one truth, each seeing it through an independent distortion that is unbiased across the crowd. That the optimal weights are inversely proportional to the clients' error energies is classical; we begin at the question that answer presupposes, which energies belong there and whether a crowd can recover them from itself. Excess risk on the truth is of the exact order of the aggregate bias energy, so no optimizer can repair a bad weight vector; the truth itself is identifiable only up to a linear tilt, so subtracting estimated client biases provably reproduces uniform weighting. The expected per-client second moments, however, are exactly identified from the law of the crowd's disagreement by a well-conditioned linear inversion, a step random-effects meta-analysis cannot take because a source reports once; their realized counterparts are estimable up to an incoherence floor the algorithm can measure. This yields CROWD, which reads the disagreement off the optimization trajectory at no extra cost and matches a Bayesian minimax lower bound in the same constant: per instance as the horizon grows, and unconditionally as the prior becomes diffuse. For arbitrary distortions it stays competitive with the optimal weights, at a ratio governed by a geometric incoherence the algorithm can measure. On real scans split into sites with their own miscalibrated detectors it attains the oracle excess risk; on a companion federation that pulls bias and noise apart, weighting by noise variance is worse than not weighting at all, and CROWD is not.

Primary area:
optimization
Keywords:
federated learning, client heterogeneity, model aggregation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_G3cjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a new problem formulation (Equation 1) and a corresponding method (Algorithms 1 and 2), supported by substantial theoretical guarantees. The theoretical results appear sound, although I have not examined the proofs in the appendices.

Given its potentially novel perspective on federated learning and solid theoretical analysis, I lean toward acceptance if my concerns are adequately addressed. I would also consider the feedback from other reviewers.

Critical strengths

The paper introduces an interesting perspective on federated learning and proposes a corresponding algorithm with substantial theoretical guarantees. The theoretical analysis appears solid.

Critical weaknesses
  1. Restrictive assumptions. The problem formulation and main theoretical results primarily focus on strongly convex objectives and linear tilts. The proposed estimation procedure also relies on independence and mean-zero assumptions. These restrictions may limit the applicability of the proposed framework.

  2. Limited generalizability. My main concern is generalizability. Some possible directions to generalize it:

    (i) General convex and nonconvex objectives: Can the problem be formulated for general convex or even nonconvex objectives? In particular, how should the learning objective be defined in the nonconvex setting, and how would the algorithm change? Such extensions could attract broader research interest.

    (ii) Multiple local updates: The proposed algorithm is primarily based on minibatch SGD with a single update per round. Can it be extended to local SGD with multiple local updates?

    If these extensions are currently difficult, I would appreciate a brief discussion of possible approaches and the associated technical challenges.

Questions for authors
  1. Novelty of the proposed perspective. The authors describe their formulation as a new "wisdom of crowds" perspective on federated learning. Could they clarify its novelty compared with existing formulations? At first glance, it introduces a client-specific perturbation hih_i into each local objective, which appears straightforward.
  2. Algorithm 1, Line 4. Since the update (x^k\gets\Pi_{\mathcal X}(\cdots)) is performed by the server, should it be placed under "Server: statistics from the same reports" rather than the client operations?
4 comments

Thank you for the positive assessment and the concrete suggestions. We start with generalizability, which you name as your main concern, and then turn to the assumptions and your two questions.

Weakness #2(i) (non-convex objectives). Yes. The algorithm is unchanged, the target becomes a local minimiser x0⋆x_0^\star of f0f_0, and new Proposition C.13 (App. C.7) keeps the same bias term under a local Polyak–Łojasiewicz inequality. The other assumptions are those of Theorem 4.4, plus the condition that the iterates stay in the basin of x0⋆x_0^\star. Schematically, for every ϵ>0\epsilon>0,

E[f0(x^)−f0⋆]≤(1+ϵ)(1+κ)2μ(Ψ^(bT)+2Δ)+O(T−1/2),\mathbb E\big[f_0(\hat x)-f_0^\star\big]\le\frac{(1+\epsilon)(1+\kappa)}{2\mu}\big(\hat\Psi(bT)+2\Delta\big)+O\big(T^{-1/2}\big),

where the first term is the bias term of Theorem 4.4. The estimator does not involve f0f_0, so only the optimisation term changes.

Weakness #2(ii) (multiple local updates). Yes, empirically: CROWD works with local steps, and best with a drift-corrected optimiser such as SCAFFOLD. New Table 2 (App. H) runs FedAvg with 1, 5 and 20 local steps and SCAFFOLD with 5 and 20 on both real-scan federations, with the SCAFFOLD control variate weighted by ww. Excess risk relative to uniform weighting with the same optimiser, medians:

optimiser, EEsensors: inverse-noise / CROWDconventions: inverse-noise / CROWD
FedAvg, 10.778 / 0.7871.069 / 0.875
FedAvg, 50.403 / 0.4081.220 / 0.954
FedAvg, 200.584 / 0.5781.413 / 1.065
SCAFFOLD, 50.231 / 0.2440.953 / 0.833
SCAFFOLD, 200.202 / 0.2081.000 / 0.878

On the convention federation, client drift under FedAvg removes CROWD's gain at 20 steps, while SCAFFOLD keeps it. On the sensor federation CROWD stays close to inverse-noise weighting. Weights and drift correction address different errors, so they should be combined.

Two extensions remain directions for future research: general convex objectives without a growth condition, and a theory for several local steps. We find both settings interesting in their own right, because each changes what the crowd's disagreement can reveal. For general convex f0f_0, a small gradient bias can move the minimiser far along flat directions, so the bias energy no longer controls the excess risk; a new link between bias and risk is needed, for example through a Hölderian error bound. For local steps, the reports are no longer gradients at one query point, and client drift adds a client-specific term to each report. The open step is to show that the residual statistics of Theorem 4.1 stay unbiased when computed from local updates; given that, a SCAFFOLD drift analysis could be combined with our bias bound. Neither is a routine extension of the present proofs: the first needs a new link between bias and risk, the second new estimation results, which is why we leave them for future work.

Weakness #1 (restrictive assumptions). The sharpest results do use linear tilts and a strongly convex truth, but several results do not, and each result now states its assumptions. Lemmas 4.2 and 4.3 hold for arbitrary tilts. Theorem C.7 covers curved crowds with client-specific curvature. Proposition 4.7 gives a guarantee without independence, centring or linearity. Independence and centring cannot be dropped entirely: correlation destroys identifiability (Prop. C.4), and centring cannot be tested (Prop. C.1). A new paragraph at the start of §4 lists the assumptions of each result.

Question #1 (novelty). The formulation is a simple step, but it leads to results that existing formulations do not give. The common objective cannot be identified from one federation (Prop. 3.1, Thm. 4.5). Yet the per-client bias energies can be estimated from the disagreement along the optimisation path (Thm. 4.1). The resulting excess risk matches a Bayesian minimax lower bound in the same constant (Cor. 4.6). And no check computed from the data can certify that a weighting beats uniform weighting (Prop. C.14), which is why the guarantee is stated under A1.

Question #2 (Algorithm 1). Corrected: the server update now sits in the server block of Algorithm 1.

Dear Reviewer G3cj,

Thank you again for the constructive suggestions. On generalizability, we added a non-convex extension (Proposition C.13) and experiments with several local steps under FedAvg and SCAFFOLD (Table 2).

We would appreciate your view on whether these additions answer your questions, and we remain available for any further discussion.

Best regards, The Authors

Thanks for the detailed responses.

  • My weakness 1: I appreciate the clarification of the scope/assumptions for each result.
  • My weakness 2: I appreciate the newly added content with respect to Polyak–Łojasiewicz inequality.

The authors explains why the extension can be difficult. However, the mentioned concerns of the extension for general convex/non-convex cases, and some restrictive assumptions still exist. These restrictions may limit the applicability. So I keep my original positive rating.

Thank you for reading our responses and for keeping your positive rating. We agree that these limitations remain, and we will state them explicitly in the final version, together with the possible approaches as directions for future work.

Reviewer_bWrMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CROWD studies federated learning when clients give biased views of a common objective. It estimates client bias energies from cross-fitted gradient disagreements and uses these estimates to choose aggregation weights. The paper analyzes identifiability, estimation error, and excess risk, and gives a Gaussian Bayesian minimax lower bound. Experiments use synthetic clients, distorted digit data, and natural FEMNIST writers.

Critical strengths

• The distinction between persistent client bias and stochastic gradient noise is useful. It explains why weighting clients by noise variance alone may miss an important source of error.

• The disagreement statistic gives a concrete route to estimating bias energies without observing the common objective. The experiments also report settings where the assumptions fail and the method loses its advantage.

Critical weaknesses

• A4 requires client biases to be sufficiently incoherent (kappa < 1), with smaller kappa needed for the stronger near-optimality results. Biases from shared acquisition or labeling processes may be correlated. When is this assumption reasonable beyond high parameter dimension?

• The gains are less consistent for some client groups and architectures. They disappear with 40 natural FEMNIST writers (Figure 2c). For the CNN, median excess risk is 1.057 times that of uniform weighting, with improvement in only one of six runs (Table 1).

• The activation guard can be misled by underestimation of kappa. In Appendix H.2(c), the estimated values fall below 1 while the true values exceed 1, incorrectly passing the activation threshold. How can this estimation error be controlled to make the guard reliable?

Questions for authors

Please see weaknesses.

3 comments

Thank you for engaging with the failure cases. We answer the three points of your review in turn.

Question #1 (when is κ<1\kappa<1 reasonable beyond high dimension?). κ<1\kappa<1 is reasonable when the heterogeneity has many independent causes compared with the number of clients, as with per-site instrument calibration. It is not reasonable when sites share a labelling or acquisition process.

The dimension alone does not matter (App. H.2(a)). With rr independent causes, the largest crowd with κ<1\kappa<1 grows from about 5 at r=8r=8 to about 37 at r=256r=256. On the detector arrays of App. H.3, where each site has its own calibration, κ=0.48\kappa=0.48. A shared process correlates the biases (Prop. C.4). We test this on purpose with a miscalibrated scanner shared by some FEMNIST writers (App. H.7, Table 7). Unguarded CROWD can then lose up to 58%, while guarded CROWD stays within 3% of uniform weighting on all 108 seeds.

Question #2 (inconsistent gains). In both cases you mention, no weighting can gain much, and the guard keeps uniform weights. With 40 natural writers the bias of uniform weighting has already decayed as AM/n\mathrm{AM}/n, and even the oracle gains only 2%. On the CNN κ>1\kappa>1, and the oracle weights are themselves worse than uniform weighting. The guard falls back to uniform weighting in all 12 FEMNIST seeds and in 5 of 6 CNN seeds (Table 5).

Question #3 (how can the estimation error be controlled to make the guard reliable?). It cannot be controlled on every federation: new Prop. C.14 (App. C.8) proves that no check computed from the data can certify that a weighting beats uniform weighting. What can be controlled is the trade-off between gain and risk, through the threshold, and we now measure it.

The reason is the following identity. With βˉ=1n∑iβi\bar\beta=\frac1n\sum_i\beta_i and β~i=βi−βˉ\tilde\beta_i=\beta_i-\bar\beta, for linear tilts and every weight vector ww,

∥b(w)∥2−∥b(wunif)∥2=∥∑iwiβ~i∥2+2∑iwi⟨β~i,βˉ⟩.\|b(w)\|^2-\|b(w_{\rm unif})\|^2=\Big\|\sum_iw_i\tilde\beta_i\Big\|^2+2\sum_iw_i\langle\tilde\beta_i,\bar\beta\rangle .

The first term is visible in the transcript and never negative. The second, the alignment of each client's deviation with the crowd's common error, is invisible: shifting the truth by c=βˉc=\bar\beta leaves the transcript unchanged and makes uniform weighting exact. All the gain of reweighting comes from this term. CROWD predicts it through A1, and κ^\hat\kappa relies on the same assumption, so the two fail together.

This explains the failure of App. H.2(c). A new diagnostic splits the under-estimate of κ\kappa into two parts. One is a term that Prop. C.1 shows to be invisible in the data. The other is staleness: the probe has settled at the uniform minimiser, far from the optimum. Shifting the truth by c=−∇f0(x)c=-\nabla f_0(x) makes any probe point xx the minimiser without changing the transcript, so no data can tell a stale probe from a well-placed one. We corrected Remark C.2, which claimed that this failure is always detectable.

The threshold 1 comes from the condition κ<1\kappa<1 of the theory and was not tuned on our results. Over all 148 seeds of Table 5, guarded CROWD relative to uniform weighting (the row for 1 and the means for 0.5 are in App. H.7; the rest is computed from the supplementary results):

thresholdseeds worse than uniformlargest lossmeanmean of worst 10%
κ^<0.5\hat\kappa<0.586.3%0.9851.010
κ^<1\hat\kappa<1 (paper)146.5%0.9591.020
κ^<3\hat\kappa<33131.1%0.9391.076

A stricter threshold gives up gain, and a looser one adds risk. On the curved digits crowd, the values of κ^\hat\kappa overlap those of the real-scan federations where CROWD reaches the oracle, so no threshold separates them. A wrong acceptance there is cheap: on that crowd guarded CROWD has a median of 0.950 of the uniform excess risk and at most 1.065. New Table 6 shows that curvature alone barely changes the false-accept rate of the guard.

Dear Reviewer bWrM,

Thank you again for engaging with the failure cases. On the reliability of the guard, Proposition C.14 shows what no data-based check can certify, and App. H.7 measures the resulting trade-off between gain and risk.

If any part of this answer is unclear or incomplete, we would welcome your feedback, and we will gladly provide further details while the discussion is open.

Best regards, The Authors

Thank you for the clarification. After considering the response, I believe my initial assessment remains appropriate and will maintain my original score.

Reviewer_JfQiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to choose aggregation weights in federated learning when clients observe distorted versions of one unobserved objective. The authors introduce CROWD that estimates persistent disagreement between client gradients, corrects the resulting energy estimates, and gives less weight to clients with larger estimated errors.

The paper provides estimation and excess-risk bounds, a Gaussian lower bound, and a guarantee under an incoherence condition. Experiments include synthetic problems, constructed digit federations, natural FEMNIST clients, and small neural networks. The strongest guarantees concern linear distortions of a strongly convex objective where client gradients differ from the common gradient by constant offsets. Nonlinear extensions are discussed but required further assumptions. The experiments show benefits in some small federations but also failures as crowd size or model complexity increases.

Critical strengths

The paper clearly identifies an error that better optimization cannot fix. Its analysis shows that a method can accurately optimize the chosen weighted objective while still performing poorly on the underlying objective. Comparisons with FedProx and SCAFFOLD show why changing aggregation weights can matter even when the optimizer converges successfully. This is an interesting distinction between improving convergence and choosing what to optimize.

The paper connects its proposed estimator to the final learning error. It bounds how accurately CROWD estimates client error energies, how estimation errors affect the aggregation weights, and how those weights affect excess risk. This provides a theoretical justification for the complete procedure under the stated assumptions.

Critical weaknesses
  1. The main assumptions limit the paper’s relevance to practical federated learning

The strongest results assume that clients share the same loss curvature and differ only by constant gradient offsets. A1 also assumes independent distortions centered around one common objective. These conditions seem to exclude many differences caused by distinct client populations and feature distributions. The constructed experiments impose these properties rather than establish that they occur naturally. The natural-client experiments provide some support, but gains disappear as the federation grows, and CROWD sometimes performs worse than uniform weighting. The authors should identify a convincing application where these assumptions are justified, or provide stronger evidence that the method remains robust against realistic deviations from the assumptions.

  1. The claim that the truth cannot be identified needs a clearer justification

Proposition 3.1 changes the common objective and adjusts every client’s distortion so that the observed client objectives stay the same. But applying this adjustment to the distortion distributions changes their expected gradients and seem to be violating A1. The argument therefore does not establish that two different truths produce the same observation distribution while both satisfy the assumptions. I believe this distinction matters because the paper uses the impossibility claim to motivate its method. The authors should clarify whether the claim concerns the full probabilistic model or only what can be recovered from one fixed federation.

Questions for authors
  1. In what real federated-learning setting would A1 and the linear crowd model be reasonable? What evidence supports this?
  2. In Proposition 3.1, doesn’t shifting the client distortions break A1’s zero-mean assumption? If so, what exactly does the proposition establish?
  3. What does Theorem 4.1 add beyond classical methods that recover individual error variances from disagreement?
  4. On the existing experiments, how does the complete recommended method compare with uniform and inverse-noise weighting?
2 comments

Thank you for the detailed review, and especially for the careful reading of Proposition 3.1: your point about centring is correct and led to a clearer statement and to the new Proposition C.14.

Question #1 (a real setting for A1 and the linear crowd, and the evidence). A1 is reasonable when the clients are a random sample of the target population and their biases have independent sources, as with per-site instruments in multi-site measurement. This is the site-effect model of multi-site harmonisation: ComBat and ISO 5725 treat each site's bias as an independent offset around a common mean. The Limitations paragraph now states this intended setting, and that we do not evaluate a deployed cross-institution federation.

Our evidence covers both sides of A1. On natural FEMNIST writers the clients are a random sample of the writers whose average defines f0f_0, so A1 holds up to the finite size of that pool. The pool of 400 writers predicts a crowd ratio of (1+n/400)/(1+1/400)(1+n/400)/(1+1/400) instead of 1. The measured ratio is 0.98 to 1.31 for n=10n=10 to 160, against a prediction of 1.02 to 1.40 (App. H.5). We also break A1 on purpose: some writers pass through one shared miscalibrated scanner (App. H.7, Table 7). Unguarded CROWD can then lose up to 58%, because the weights collapse onto one writer. The guard detects this collapse, and guarded CROWD stays within 3% of uniform weighting on all 108 seeds. The claim is therefore conditional on A1, and Prop. C.14 (Question #2) shows that this condition cannot be removed.

Question #2 (Proposition 3.1 and A1). Yes, the shift breaks centring, so Proposition 3.1 is a statement about one realised federation, not about the full probabilistic model. It does not claim that two centred models give the same law of observations. It claims that for any realisation of the tilts and any cc, replacing f0f_0 by f0+⟨c,⋅⟩f_0+\langle c,\cdot\rangle and each hih_i by hi−⟨c,⋅⟩h_i-\langle c,\cdot\rangle leaves every client risk, and hence the transcript, unchanged. A protocol observes only this realisation, so it cannot tell the two truths apart.

In the full model the truth is identified as n→∞n\to\infty. At fixed nn, the quantitative statement is Theorem 4.5: under a diffuse prior every algorithm pays at least Ψ0/(2μ)\Psi_0/(2\mu), and CROWD attains this constant in the limits of Cor. 4.6. Prop. 3.1 and a new paragraph after it now say this explicitly, and the abstract, §1 and the Ethics statement say "from one federation".

New Prop. C.14 states the consequence for the method. With βˉ=1n∑iβi\bar\beta=\frac1n\sum_i\beta_i and β~i=βi−βˉ\tilde\beta_i=\beta_i-\bar\beta, for linear tilts and every weight vector ww,

∥b(w)∥2−∥b(wunif)∥2=∥∑iwiβ~i∥2+2∑iwi⟨β~i,βˉ⟩.\|b(w)\|^2-\|b(w_{\rm unif})\|^2=\Big\|\sum_iw_i\tilde\beta_i\Big\|^2+2\sum_iw_i\langle\tilde\beta_i,\bar\beta\rangle .

The first term is visible in the transcript and never negative. The second is invisible: the shift c=βˉc=\bar\beta leaves the transcript unchanged and makes uniform weighting exact. So every gain over uniform weighting rests on a term the data cannot see, and any guarantee must rest on an assumption like A1.

Question #3 (what Theorem 4.1 adds). For one scalar, the inversion of Theorem 4.1 is Grubbs's estimator, which we now cite with the three-cornered hat. What is new is the setting. The gradient field moves with the query point and cancels in the residual at every iterate (Lemma E.1). Cross-fitting removes the noise without knowing σi\sigma_i. And Lemma 4.2 gives a floor on what can be recovered, which does not decay with the number of rounds.

Question #4 (complete method). The complete guarded procedure is rarely worse than uniform weighting and has a much lighter tail than inverse-noise weighting; it does not beat inverse-noise weighting in the median, which is lower in half of the 14 settings. New Table 5 (App. H.7) applies the κ^\hat\kappa test to all 148 seeds of our studies. Guarded CROWD is worse than uniform weighting in 14 seeds, by at most 6.5%, against 62 seeds for inverse-noise weighting. Over the worst 10% of seeds, its excess risk is 1.02 times that of uniform weighting, against 1.40 for inverse-noise weighting.

Finally, one remark on Weakness #1, beyond what is covered above. Shared curvature is not needed, since Theorem C.7 allows client-specific curvature. The gains shrink as the natural federation grows because the bias of uniform weighting decays as AM/n\mathrm{AM}/n, so there is less to gain; with amplified writer styles, where the bias does not vanish, CROWD keeps a gain up to n=40n=40 (0.929).

Dear Reviewer JfQi,

Thank you again for your careful reading of the theory. Following your point on Proposition 3.1, we clarified its statement and added Proposition C.14, which explains why any guarantee must rest on an assumption like A1.

We would be glad to hear whether this addresses your concerns, and we are happy to discuss any remaining points before the end of the discussion period.

Best regards, The Authors

Comments on the paper

1 comment

We thank the reviewers for their careful reviews. All changes are in blue, and the main text uses the tenth page allowed during the discussion. The numbers behind each change are given in the individual responses.

The reviews share one question: when does the crowd hypothesis hold, and what happens when it does not? Our answer has two parts. First, no method can identify the truth from one federation, and no check computed from the data can certify that a weighting beats uniform weighting. Any guarantee must therefore be conditional on an assumption such as A1. Second, under this assumption CROWD matches the minimax lower bound in the same constant. Where the assumption fails, the guarded procedure stays close to uniform weighting in all our studies. This is evidence on the cases we tested, not a guarantee.

The main changes:

  1. Proposition 3.1 is clarified. It concerns one realised federation, and Theorem 4.5 is its quantitative form for the full model. The abstract, §1 and the Ethics statement are qualified accordingly (JfQi).
  2. We state when Assumption A1 is expected to hold: a few to a dozen sites whose heterogeneity has many independent causes, as in multi-site harmonisation. We also check its finite-sample prediction on natural FEMNIST writers (App. H.5, Limitations) (JfQi, bWrM).
  3. We cite Grubbs (1948) and the three-cornered hat, and say what Theorem 4.1 adds to them (JfQi).
  4. New App. H.7 evaluates the complete guarded procedure on all seeds of our studies and compares it with uniform and inverse-noise weighting. It also calibrates the guard away from the linear regime, and breaks A1 on purpose on FEMNIST (JfQi, bWrM).
  5. New Proposition C.14 shows why the guard cannot be made reliable on every federation. A diagnostic explains the guard failure of App. H.2(c), and Remark C.2 is corrected (bWrM).
  6. New Proposition C.13 extends the end-to-end bound to a non-convex truth under a local Polyak–Łojasiewicz inequality, with the same bias constant (G3cj).
  7. New Table 2 combines CROWD with FedAvg and SCAFFOLD with several local steps on real data (G3cj).
  8. §4 now lists the assumptions of each result, §1 states what is new in the formulation, and Algorithm 1 is corrected (G3cj, JfQi).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.