acceptodds
Under review as a conference paper at ICLR 2027

Incentivizing Federated Learning: Agreement is enough

Abstract

Federated learning enables collaborative training on private data, but attracting clients with valuable data requires incentives for effortful training and honest reporting. Computing these rewards must be inexpensive enough for every training round. Knowledge-Free Correlated Agreement (KFCA) addresses this need with fixed Dasgupta-Ghosh agreement-minus-chance scoring, avoiding correlation estimation and penalizing unilateral label flips under categorical and task assumptions. Federated LLM tuning, industrial inspection, and runtime comparisons assess rewards and efficiency; a smart-contract prototype illustrates auditable execution.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
mechanism design, incentive compatibility, federated learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CFvJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies low-cost incentives for federated learning using a fixed agreement-minus-chance peer-prediction rule. It scores model predictions or quantized updates, provides conditional reporting guarantees, and evaluates reward behavior and computational efficiency across federated learning settings, including LLM fine-tuning.

Critical strengths

The paper addresses a practical need for inexpensive reward computation in federated learning without labeled validation data. Its fixed agreement-minus-chance score is simple to implement, and the matched runtime comparison usefully isolates the computational savings from avoiding repeated correlation estimation.

Critical weaknesses
  • Unverified experimental pipeline. The LLM results do not establish whether reports use adapter states or actual updates, and the historical MNIST scoring implementation differs from the stated algorithm.
  • Insufficient support for fresh-training incentives. Table 7 shows that replaying stale updates can earn higher scores than fresh training. The conditional reporting guarantees do not cover this practically important deviation.
  • Limited incremental contribution. The core scoring rule and truthfulness guarantee follow established DG/MSDG results, while the efficiency gains over cached CA are modest and lack end-to-end validation.
Questions for authors
  1. Do the main experimental conclusions hold with verified local-update signs and an implementation consistent with Algorithm 1?
  2. How can KFCA incentivize fresh training when stale updates earn higher rewards in Table 7, especially after accounting for training costs?
  3. What substantive advance does KFCA provide over fixed categorical CA/MSDG beyond the measured computational savings?
Reviewer_CfcXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes KFCA. Building on the classical Dasgupta–Ghosh agreement scoring rule and multi-task peer prediction theory, KFCA replaces the correlation matrix estimation required by traditional Correlated Agreement (CA) mechanisms with a fixed agreement scoring function under the categorical-world condition, thereby reducing reward computation overhead. The paper introduces two implementations: KFCA-D, based on predicted labels, and KFCA-QP, based on model update signs. Experiments on federated LLM fine-tuning, MNIST, and industrial PCB inspection evaluate the computational efficiency and reward allocation properties of the proposed method.

Critical strengths
  1. The paper addresses a problem with significant practical relevance.
  2. The proposed KFCA mechanism is simple and computationally efficient.
  3. The paper combines theoretical analysis with experiments across multiple scenarios to evaluate the method's reward allocation, robustness against label-flipping attacks, and computational efficiency.
Critical weaknesses
  1. The core method is largely based on existing DG/MSDG theory.
  2. The theoretical guarantees rely on strong independence and categorical-world assumptions, making them difficult to generalize directly to complex real-world federated learning environments.
  3. Some experiments exhibit inconsistencies between the report definitions and their actual implementations.
Questions for authors
  1. Compared with existing DG/MSDG mechanisms and CA-based methods for federated learning, what are the substantive innovations of KFCA in terms of algorithm design and theoretical properties?

  2. The categorical-world condition is a key assumption underlying the theoretical guarantees. How can the authors ensure that this condition remains valid under highly non-IID data distributions and complex neural network parameter updates?

  3. The paper mainly establishes the optimality of truthful reporting. However, how does the proposed mechanism ensure that clients are incentivized to continuously perform genuine local training rather than obtaining rewards by reusing stale updates or copying reports from other clients?

  4. Appendix A.14.1 shows that some clients receive higher rewards when reusing stale updates than when submitting fresh updates. How do the authors explain this phenomenon, and does it undermine the practical effectiveness of the proposed incentive mechanism?

  5. Some experiments exhibit inconsistencies between parameter update signs and parameter-state signs, as well as discrepancies between the experimental code and the algorithm description. Could the authors re-evaluate the main experimental results using a consistent implementation and clarify whether these issues affect the validity of the reported conclusions?

Reviewer_ZUMFOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Knowledge-Free Correlated Agreement (KFCA) for federated learning, using fixed agreement-minus-chance scoring. KFCA-D scores predictions on shared unlabeled inputs, while KFCA-QP scores sign-quantized parameter updates. The paper examines conditional reporting guarantees, categorical structure, computational cost, and reward behavior.

Critical strengths
  • Rewarding honest, effortful participation in federated learning without ground-truth labels is practically important, and peer-prediction mechanisms are a principled fit.

  • Building on DG/MSDG gives the mechanism a categorical agreement guarantee rather than a purely heuristic scoring rule, and the paper shows that the sign condition holds empirically in the FL setting.

  • The paper turns the theory into a complete FL workflow, covering report construction from model updates, quantization, peer sampling, and scoring, which makes the approach easier to adopt and test.

Critical weaknesses
  • The agreement-minus-chance rule and categorical agreement guarantee are inherited from DG/MSDG. Related FL constructions already use shared unlabeled inputs, quantized model-update reports, and sampled peers. The contribution is mainly the FL pipeline and experiments on computational cost and reward behavior.

  • Effortful training is not sufficiently supported. In Table 7, own stale R4 increments receive higher scores than fresh increments: 0.07615/0.06695 versus 0.05596. The categorical sign condition holds empirically, yet stale updates score more. If reusing the stored increment is cheaper, no positive payment multiplier repairs this comparison.

  • Report definitions and implementation provenance remain unresolved. The recorded four-domain diagnostics do not establish whether they use local-increment signs or adapter-state signs. The archived peer-scoring code also differs from Algorithm 1 in quantization and penalty-index handling, and the code revision used for historical figures has not been established.

  • The computational saving is limited. Fixed scoring takes 123 ms, compared with 371 ms for estimating and applying a CA matrix and 154 ms for applying a cached matrix. Both approaches remain linear in report length at fixed alphabet and peer budgets.

  • Presentation needs improvement. The formatting and figure layout lack the polish expected at ICLR. In Figure 6, the dense computation–fidelity comparison and small axis labels and legends make the results difficult to assess.

Questions for authors
  1. Table 7 shows stale own R4 increments scoring higher than fresh ones (0.07615/0.06695 vs. 0.05596). Since reusing a stored increment is cheaper, is there any payment scaling or mechanism parameter under which fresh training strictly dominates stale reuse? If not, what does the categorical sign condition actually guarantee about effort?

  2. For the four-domain diagnostics, were reports computed from local-increment signs or adapter-state signs? The archived peer-scoring code also differs from Algorithm 1 in quantization and penalty-index handling. Which code revision produced each reported figure, and do the results hold when Algorithm 1 is implemented exactly as written?

  3. Beyond the FL instantiation, what is technically new? Does the agreement guarantee need any new proof steps or assumptions for the FL setting, and how does the mechanism differ, formally and empirically, from prior FL schemes that also use shared unlabeled inputs, quantized update reports, and sampled peers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.