- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates participation bias in decentralized language model fine-tuning, where infrequently participating nodes may be underrepresented during training. The authors particularly study a loss-adaptive participation setting, in which higher local loss reduces future participation, potentially creating a negative feedback loop. To address this issue, they propose RUMOR, a reweighting method that combines participation history, loss-dependent weight caps, and model disagreement with neighboring nodes to adjust local gradient updates. The method is evaluated on a small transformer and seven pretrained language models up to 12B parameters, under both static and loss-adaptive participation settings. Experimental results show that RUMOR effectively reduces loss disparities under strong participation feedback and generally avoids the severe loss increases observed with aggressive inverse-probability weighting, while remaining competitive under static participation.
Critical strengths
- The paper explores an interesting problem in decentralized language model fine-tuning, where poor model performance may discourage node participation and further worsen training outcomes. Although the practical prevalence of this feedback mechanism is not entirely clear, I find the hypothesis thought-provoking and worth investigating. The proposed RUMOR method also provides a reasonable way to address this issue using only locally available information.
- The paper provides extensive experiments across multiple language models, including models up to 12B parameters, under both static and loss-adaptive participation settings. The comparisons with various baselines, ablation studies, and independent-seed evaluations make the experimental findings relatively convincing. In particular, RUMOR demonstrates strong participation-bias correction while avoiding the severe loss increases observed with aggressive reweighting methods, which is a meaningful practical advantage.
Critical weaknesses
- The paper assumes that nodes with higher local loss are less likely to participate in future training, creating a negative feedback loop. While this is an interesting hypothesis, I am not entirely convinced that such behavior commonly occurs in real decentralized learning systems. In practice, node participation may be more strongly influenced by computational resources, network conditions, or device availability. The paper provides limited empirical evidence supporting its central assumption, which raises concerns about whether the proposed method addresses a practically significant problem or mainly performs well under a specifically designed simulation setting.
- RUMOR combines existing ideas such as inverse-probability weighting, adaptive weight capping, and neighbor model disagreement. While the combination is reasonable, its advantages over simpler alternatives are not consistently demonstrated. For example, the fixed-cap variant achieves comparable or even better results in several experiments, making the contribution of loss-dependent adaptation less convincing. Furthermore, incorporating neighbor disagreement requires an additional model exchange, potentially doubling model communication traffic. Without a more comprehensive evaluation of communication overhead and end-to-end training efficiency, it remains unclear whether the additional complexity is justified by the observed performance improvements.
Questions for authors
- Could the authors provide more justification or empirical evidence for the assumption that nodes with higher local loss are less likely to participate in future training? In real decentralized learning systems, participation is often determined by computational resources, network conditions, or device availability. I am particularly interested in whether the proposed feedback mechanism has been observed in practical systems or is primarily a theoretical assumption designed for this study.
- The fixed-cap variant achieves comparable or even better performance than the full RUMOR method in several experiments. For example, at β = 4 on the small transformer, the fixed-cap variant achieves a 97.2% reduction compared with 93.0% for RUMOR. Could the authors clarify under which conditions the loss-dependent cap provides a substantial advantage over simpler weighting strategies? I would also appreciate a clearer explanation of how much each component contributes to the overall improvement.
- The proposed method requires an additional model exchange to measure disagreement between neighboring nodes, potentially doubling model communication traffic. Considering that communication efficiency is important in decentralized learning, could the authors clarify whether the performance improvements justify this additional overhead? In particular, how would RUMOR compare with simpler history-based weighting methods under the same communication budget?
- The paper primarily emphasizes reductions in the loss gap between high- and low-participation groups. However, a smaller gap does not necessarily indicate better overall model performance, as it may also result from increased loss in the high-participation group. Although the paper reports absolute losses and uniform node-average losses, could the authors clarify whether RUMOR consistently improves overall model quality compared with the strongest baselines? This distinction would help assess the practical significance of the reported gains.
I think the authors have done some really interesting work. I still have a few questions that I hope the authors can address in the rebuttal, and I would be happy to raise my score if their responses resolve my concerns.
4 comments
Response to Reviewer DjfB (2/2): Communication and overall model quality
[Q3. Communication overhead and matched-budget performance]
Section 7 now clarifies our intended regime of compute-intensive fine-tuning over high-bandwidth links. Your communication concern motivated new comparisons under limited budgets. A neighbor can change its model through gossip mixing after sending it, so the stored copy may be outdated. Therefore, RUMOR exchanges models again to calculate disagreement using current neighbor models, as Appendix B.2 explains.
To test correction under this constraint, we added cached RUMOR. This ablation retains the weighting rule but reuses previously received neighbor models without another exchange. We count bytes used to transmit the trainable LoRA matrices added to the pretrained model. Each method receives the same byte budget within a model and seed and stops before another complete round would exceed it. Participation determines messages and updates per round, so completed rounds and local updates can differ.
Under matched communication budgets on Qwen2.5-7B at , cached RUMOR lowers uniform loss by nats versus cap-20 IPW, versus its fixed-cap ablation, and versus its no-disagreement ablation with the same adaptive cap. All three seeds favor cached RUMOR in each comparison. All methods use ordinary gossip without an extra exchange (Appendix H.3, Figure 13 and Table 4).
On Mistral-Nemo 12B at , two IPW runs spike and finish worse than cached RUMOR. It avoids these excursions and lowers mean final uniform loss by nats, with the third seed tied. Figure 13 in Appendix H.3 shows these trajectories. These tests establish correction and stability benefits under a communication constraint. We report bytes and local updates, while end-to-end timing remains unmeasured (Section 7).
We also compared both RUMOR versions at equal byte budgets on both models at . Figure 13 shows that cached RUMOR has lower loss in every seed in five of six settings and completes more local updates in every pair. Original RUMOR has lower mean loss on Mistral-Nemo 12B at , due to one cached-run spike.
[Q4. Overall model quality and the group gap]
Your question helped us clarify objective reporting in Section 6 and Appendix I and motivated a new paired analysis of existing ablation runs (Appendix I.1, Table 11). Group-gap reduction is our primary measure of participation-related disparity. Uniform node-average loss measures the training objective with equal node weights. We examine it and both group losses alongside the gap to distinguish disparity reduction from improvements that benefit both groups.
For example, on GPT-2 medium at , RUMOR with its original neighbor exchange lowers uniform loss by nats versus the same adaptive-cap rule without disagreement. Both groups improve in all ten seeds, so this gain does not come from worsening the better-served group. These are the ablation runs in Figure 10, Appendix H.1, with the paired uniform-loss analysis in Table 11.
Against interval-average cutoff 50, RUMOR lowers uniform loss by and nats on GPT-2 medium and Qwen 0.5B at (Appendix L.2). After comparable tuning at , RUMOR also achieves lower mean uniform loss and mean loss in both language groups than tuned IPW on Mistral-Nemo 12B and tuned interval-average on Qwen2.5-7B, as Table 5 in Appendix H.4 reports. Although some baselines achieve lower loss in particular settings, they can leave participation bias insufficiently corrected or suffer severe loss increases in others. RUMOR's strength is the balance it achieves between uniform loss, participation correction and training stability across the evaluated settings.
References
Cho et al. (2024). Maximizing Global Model Appeal in Federated Learning. TMLR.
Wu et al. (2024). Incentive-Aware Federated Learning with Training-Time Model Rewards. ICLR.
Goetze et al. (2025). FLOSS: Federated Learning with Opt-Out and Straggler Support. PACMI.
Hashimoto et al. (2018). Fairness Without Demographics in Repeated Loss Minimization. ICML.
Thank you for helping us strengthen the communication analysis and clarify the value of our method. If these revisions address your concerns, we would be grateful if you would consider increasing your score. We would be happy to discuss any remaining questions.
Thank you for the detailed responses. I will maintain my current rating.
Dear reviewer, thank you for your feedback. We believe our response and revised manuscript address the main concerns raised in your review. Could you clarify which specific concerns remain unresolved so we can address them and you can reconsider your rating?
Response to Reviewer DjfB (1/2): Motivation and correction benefits
Thank you for your insights on communication costs and for helping us further clarify why this research direction matters.
We have expanded the manuscript by more than a quarter in response to the reviews. The revision adds a conditional analysis of our weight corrections and a finite-time convergence analysis, together with communication-budget, sensitivity, tuning and loss-calibration studies. We also clarify how disparity and absolute losses should be interpreted together. Substantive changes in the revised manuscript are shown in blue.
[Q1. Motivation for benefit-dependent participation]
We expanded Sections 1–2 to explain why a shared model's local usefulness may affect willingness to contribute to the training. In MaxFL (Cho et al., 2024), clients can continue participating in training only if the global model meets their local performance requirements. Similarly, IAFL (Wu et al., 2024) models contributions through model rewards relative to resource costs, and FLOSS (Goetze et al., 2025) links willingness to share updates to satisfaction with model performance. These are direct precedents for studying benefit-dependent contributions in federated learning.
Hashimoto et al. (2018) study how poorly served users leaving can reduce their groups' representation in subsequent training. Their survey of 1,000 users testing keyboard next-word suggestions measured satisfaction and stated willingness to continue. Their survey-based simulations predicted that better serving minority users raised retention from 70% with average-loss training to 85% with robust training. These works motivate our model in which nodes participate less when the shared model performs poorly on their local data. In practice, resource constraints and node availability can also affect participation. Here, we focus on how local model performance influences participation in training. We examine its effect through a controlled loss-adaptive participation rule and study how to correct the resulting imbalance in training.
[Q2. When the adaptive cap and other components help]
Your small-transformer β=4 example identifies a setting where a conservative fixed cap works well. RUMOR's loss-dependent cap allows larger update weights for infrequently participating nodes when their local loss stays near its reference. Under static participation, RUMOR lowers uniform loss versus RUMOR, FIXED CAP by (1.34±0.05)×10−2 nats on GPT-2 small and (4±0.3)×10−3 on Qwen 1.5B (Figure 15, Table 11).
The reason to adapt the cap is that a large weight has two effects: it restores an infrequent node's influence, but it also amplifies fluctuations in its minibatch update. A small fixed cap limits this amplification, yet can leave useful contributions underweighted. RUMOR allows stronger history correction near the local loss reference and lowers the cap when recent loss rises above that reference. Neighbor disagreement provides additional information because nodes that participate equally often can still differ in how far their models are from their neighbors' models.
We added analysis to explain when these weight changes improve training. Theorem 1 compares RUMOR and capped IPW from the same local models, history and active set. It gives sufficient conditions under which reducing a weight helps because further amplification would increase loss. It also identifies when increasing a weight helps because additional progress along the update direction outweighs the effect of loss curvature. Under these conditions, RUMOR has lower expected uniform training loss after one update. Proposition 4 also shows that if neither correction changes the IPW weights, the next network-average models are identical. When weight changes are small, it bounds the resulting model and expected-loss differences. Section 5 summarizes this reasoning, with the conditions and proofs in Appendix K.3.
To test these roles during training, we compare a fixed-cap ablation retaining disagreement and a no-disagreement ablation retaining the adaptive cap. Both components improve Qwen2.5-7B uniform loss at β=6 under an equal byte budget (Appendix H.3, Table 4).