- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies low-cost incentives for federated learning using a fixed agreement-minus-chance peer-prediction rule. It scores model predictions or quantized updates, provides conditional reporting guarantees, and evaluates reward behavior and computational efficiency across federated learning settings, including LLM fine-tuning.
Critical strengths
The paper addresses a practical need for inexpensive reward computation in federated learning without labeled validation data. Its fixed agreement-minus-chance score is simple to implement, and the matched runtime comparison usefully isolates the computational savings from avoiding repeated correlation estimation.
Critical weaknesses
- Unverified experimental pipeline. The LLM results do not establish whether reports use adapter states or actual updates, and the historical MNIST scoring implementation differs from the stated algorithm.
- Insufficient support for fresh-training incentives. Table 7 shows that replaying stale updates can earn higher scores than fresh training. The conditional reporting guarantees do not cover this practically important deviation.
- Limited incremental contribution. The core scoring rule and truthfulness guarantee follow established DG/MSDG results, while the efficiency gains over cached CA are modest and lack end-to-end validation.
Questions for authors
- Do the main experimental conclusions hold with verified local-update signs and an implementation consistent with Algorithm 1?
- How can KFCA incentivize fresh training when stale updates earn higher rewards in Table 7, especially after accounting for training costs?
- What substantive advance does KFCA provide over fixed categorical CA/MSDG beyond the measured computational savings?