acceptodds
Under review as a conference paper at ICLR 2027

RUMOR Has It: Correcting Participation Bias in Decentralized Language Model Fine-Tuning

Abstract

Decentralized fine-tuning of language models can leave infrequent participants' data poorly represented. Motivated by prior user participation models that take into account model quality, we study a controlled setting in which a node becomes less likely to train and exchange its model as its loss rises relative to the network mean. This dependence can create a positive feedback loop: missed updates and model exchanges can keep the node's loss high, which can in turn further reduce its participation. To correct this imbalance using only local information, we reweight updates based on recent participation history, use past local loss as a reference to limit excessive amplification, and use disagreement with active neighbors' models as an additional signal for increasing the update weight. We call the resulting scheme RUMOR, Reference-guided Updates using Memory, Observed loss, and Reweighting. We evaluate a small transformer and seven pretrained models up to 12 billion parameters under both loss-adaptive participation and fixed participation probabilities. RUMOR substantially reduces the loss disparities associated with infrequent participation, with particularly clear benefits under strong feedback. Under weaker feedback and fixed participation probabilities, it remains competitive with aggressive participation reweighting while avoiding the severe loss increases observed for such baselines in the evaluated large-model runs.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
decentralized learning, gossip learning, federated learning, distributed optimization, participation bias, loss-adaptive participation, language model fine-tuning, LoRA

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DjfBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates participation bias in decentralized language model fine-tuning, where infrequently participating nodes may be underrepresented during training. The authors particularly study a loss-adaptive participation setting, in which higher local loss reduces future participation, potentially creating a negative feedback loop. To address this issue, they propose RUMOR, a reweighting method that combines participation history, loss-dependent weight caps, and model disagreement with neighboring nodes to adjust local gradient updates. The method is evaluated on a small transformer and seven pretrained language models up to 12B parameters, under both static and loss-adaptive participation settings. Experimental results show that RUMOR effectively reduces loss disparities under strong participation feedback and generally avoids the severe loss increases observed with aggressive inverse-probability weighting, while remaining competitive under static participation.

Critical strengths
  1. The paper explores an interesting problem in decentralized language model fine-tuning, where poor model performance may discourage node participation and further worsen training outcomes. Although the practical prevalence of this feedback mechanism is not entirely clear, I find the hypothesis thought-provoking and worth investigating. The proposed RUMOR method also provides a reasonable way to address this issue using only locally available information.
  2. The paper provides extensive experiments across multiple language models, including models up to 12B parameters, under both static and loss-adaptive participation settings. The comparisons with various baselines, ablation studies, and independent-seed evaluations make the experimental findings relatively convincing. In particular, RUMOR demonstrates strong participation-bias correction while avoiding the severe loss increases observed with aggressive reweighting methods, which is a meaningful practical advantage.
Critical weaknesses
  1. The paper assumes that nodes with higher local loss are less likely to participate in future training, creating a negative feedback loop. While this is an interesting hypothesis, I am not entirely convinced that such behavior commonly occurs in real decentralized learning systems. In practice, node participation may be more strongly influenced by computational resources, network conditions, or device availability. The paper provides limited empirical evidence supporting its central assumption, which raises concerns about whether the proposed method addresses a practically significant problem or mainly performs well under a specifically designed simulation setting.
  2. RUMOR combines existing ideas such as inverse-probability weighting, adaptive weight capping, and neighbor model disagreement. While the combination is reasonable, its advantages over simpler alternatives are not consistently demonstrated. For example, the fixed-cap variant achieves comparable or even better results in several experiments, making the contribution of loss-dependent adaptation less convincing. Furthermore, incorporating neighbor disagreement requires an additional model exchange, potentially doubling model communication traffic. Without a more comprehensive evaluation of communication overhead and end-to-end training efficiency, it remains unclear whether the additional complexity is justified by the observed performance improvements.
Questions for authors
  1. Could the authors provide more justification or empirical evidence for the assumption that nodes with higher local loss are less likely to participate in future training? In real decentralized learning systems, participation is often determined by computational resources, network conditions, or device availability. I am particularly interested in whether the proposed feedback mechanism has been observed in practical systems or is primarily a theoretical assumption designed for this study.
  2. The fixed-cap variant achieves comparable or even better performance than the full RUMOR method in several experiments. For example, at β = 4 on the small transformer, the fixed-cap variant achieves a 97.2% reduction compared with 93.0% for RUMOR. Could the authors clarify under which conditions the loss-dependent cap provides a substantial advantage over simpler weighting strategies? I would also appreciate a clearer explanation of how much each component contributes to the overall improvement.
  3. The proposed method requires an additional model exchange to measure disagreement between neighboring nodes, potentially doubling model communication traffic. Considering that communication efficiency is important in decentralized learning, could the authors clarify whether the performance improvements justify this additional overhead? In particular, how would RUMOR compare with simpler history-based weighting methods under the same communication budget?
  4. The paper primarily emphasizes reductions in the loss gap between high- and low-participation groups. However, a smaller gap does not necessarily indicate better overall model performance, as it may also result from increased loss in the high-participation group. Although the paper reports absolute losses and uniform node-average losses, could the authors clarify whether RUMOR consistently improves overall model quality compared with the strongest baselines? This distinction would help assess the practical significance of the reported gains.

I think the authors have done some really interesting work. I still have a few questions that I hope the authors can address in the rebuttal, and I would be happy to raise my score if their responses resolve my concerns.

4 comments
AuthorsOpenReview ↗

Response to Reviewer DjfB (1/2): Motivation and correction benefits

Thank you for your insights on communication costs and for helping us further clarify why this research direction matters.

We have expanded the manuscript by more than a quarter in response to the reviews. The revision adds a conditional analysis of our weight corrections and a finite-time convergence analysis, together with communication-budget, sensitivity, tuning and loss-calibration studies. We also clarify how disparity and absolute losses should be interpreted together. Substantive changes in the revised manuscript are shown in blue.

[Q1. Motivation for benefit-dependent participation]

We expanded Sections 1–2 to explain why a shared model's local usefulness may affect willingness to contribute to the training. In MaxFL (Cho et al., 2024), clients can continue participating in training only if the global model meets their local performance requirements. Similarly, IAFL (Wu et al., 2024) models contributions through model rewards relative to resource costs, and FLOSS (Goetze et al., 2025) links willingness to share updates to satisfaction with model performance. These are direct precedents for studying benefit-dependent contributions in federated learning.

Hashimoto et al. (2018) study how poorly served users leaving can reduce their groups' representation in subsequent training. Their survey of 1,000 users testing keyboard next-word suggestions measured satisfaction and stated willingness to continue. Their survey-based simulations predicted that better serving minority users raised retention from 70% with average-loss training to 85% with robust training. These works motivate our model in which nodes participate less when the shared model performs poorly on their local data. In practice, resource constraints and node availability can also affect participation. Here, we focus on how local model performance influences participation in training. We examine its effect through a controlled loss-adaptive participation rule and study how to correct the resulting imbalance in training.

[Q2. When the adaptive cap and other components help]

Your small-transformer β=4\beta=4 example identifies a setting where a conservative fixed cap works well. RUMOR's loss-dependent cap allows larger update weights for infrequently participating nodes when their local loss stays near its reference. Under static participation, RUMOR lowers uniform loss versus RUMOR, FIXED CAP by (1.34±0.05)×10−2(1.34\pm0.05)\times10^{-2} nats on GPT-2 small and (4±0.3)×10−3(4\pm0.3)\times10^{-3} on Qwen 1.5B (Figure 15, Table 11).

The reason to adapt the cap is that a large weight has two effects: it restores an infrequent node's influence, but it also amplifies fluctuations in its minibatch update. A small fixed cap limits this amplification, yet can leave useful contributions underweighted. RUMOR allows stronger history correction near the local loss reference and lowers the cap when recent loss rises above that reference. Neighbor disagreement provides additional information because nodes that participate equally often can still differ in how far their models are from their neighbors' models.

We added analysis to explain when these weight changes improve training. Theorem 1 compares RUMOR and capped IPW from the same local models, history and active set. It gives sufficient conditions under which reducing a weight helps because further amplification would increase loss. It also identifies when increasing a weight helps because additional progress along the update direction outweighs the effect of loss curvature. Under these conditions, RUMOR has lower expected uniform training loss after one update. Proposition 4 also shows that if neither correction changes the IPW weights, the next network-average models are identical. When weight changes are small, it bounds the resulting model and expected-loss differences. Section 5 summarizes this reasoning, with the conditions and proofs in Appendix K.3.

To test these roles during training, we compare a fixed-cap ablation retaining disagreement and a no-disagreement ablation retaining the adaptive cap. Both components improve Qwen2.5-7B uniform loss at β=6\beta=6 under an equal byte budget (Appendix H.3, Table 4).

AuthorsOpenReview ↗

Response to Reviewer DjfB (2/2): Communication and overall model quality

[Q3. Communication overhead and matched-budget performance]

Section 7 now clarifies our intended regime of compute-intensive fine-tuning over high-bandwidth links. Your communication concern motivated new comparisons under limited budgets. A neighbor can change its model through gossip mixing after sending it, so the stored copy may be outdated. Therefore, RUMOR exchanges models again to calculate disagreement using current neighbor models, as Appendix B.2 explains.

To test correction under this constraint, we added cached RUMOR. This ablation retains the weighting rule but reuses previously received neighbor models without another exchange. We count bytes used to transmit the trainable LoRA matrices added to the pretrained model. Each method receives the same byte budget within a model and seed and stops before another complete round would exceed it. Participation determines messages and updates per round, so completed rounds and local updates can differ.

Under matched communication budgets on Qwen2.5-7B at β=6\beta=6, cached RUMOR lowers uniform loss by (5.33±2.32)×10−3(5.33\pm2.32)\times10^{-3} nats versus cap-20 IPW, (1.37±0.381)×10−3(1.37\pm0.381)\times10^{-3} versus its fixed-cap ablation, and (1.81±0.421)×10−3(1.81\pm0.421)\times10^{-3} versus its no-disagreement ablation with the same adaptive cap. All three seeds favor cached RUMOR in each comparison. All methods use ordinary gossip without an extra exchange (Appendix H.3, Figure 13 and Table 4).

On Mistral-Nemo 12B at β=6\beta=6, two IPW runs spike and finish worse than cached RUMOR. It avoids these excursions and lowers mean final uniform loss by 0.743±0.4320.743\pm0.432 nats, with the third seed tied. Figure 13 in Appendix H.3 shows these trajectories. These tests establish correction and stability benefits under a communication constraint. We report bytes and local updates, while end-to-end timing remains unmeasured (Section 7).

We also compared both RUMOR versions at equal byte budgets on both models at β=2,4,6\beta=2,4,6. Figure 13 shows that cached RUMOR has lower loss in every seed in five of six settings and completes more local updates in every pair. Original RUMOR has lower mean loss on Mistral-Nemo 12B at β=4\beta=4, due to one cached-run spike.

[Q4. Overall model quality and the group gap]

Your question helped us clarify objective reporting in Section 6 and Appendix I and motivated a new paired analysis of existing ablation runs (Appendix I.1, Table 11). Group-gap reduction is our primary measure of participation-related disparity. Uniform node-average loss measures the training objective with equal node weights. We examine it and both group losses alongside the gap to distinguish disparity reduction from improvements that benefit both groups.

For example, on GPT-2 medium at β=2\beta=2, RUMOR with its original neighbor exchange lowers uniform loss by (6.82±0.615)×10−3(6.82\pm0.615)\times10^{-3} nats versus the same adaptive-cap rule without disagreement. Both groups improve in all ten seeds, so this gain does not come from worsening the better-served group. These are the ablation runs in Figure 10, Appendix H.1, with the paired uniform-loss analysis in Table 11.

Against interval-average cutoff 50, RUMOR lowers uniform loss by (7.8±0.4)×10−3(7.8\pm0.4)\times10^{-3} and (1.1±0.3)×10−3(1.1\pm0.3)\times10^{-3} nats on GPT-2 medium and Qwen 0.5B at β=2\beta=2 (Appendix L.2). After comparable tuning at β=6\beta=6, RUMOR also achieves lower mean uniform loss and mean loss in both language groups than tuned IPW on Mistral-Nemo 12B and tuned interval-average on Qwen2.5-7B, as Table 5 in Appendix H.4 reports. Although some baselines achieve lower loss in particular settings, they can leave participation bias insufficiently corrected or suffer severe loss increases in others. RUMOR's strength is the balance it achieves between uniform loss, participation correction and training stability across the evaluated settings.

References

Cho et al. (2024). Maximizing Global Model Appeal in Federated Learning. TMLR.

Wu et al. (2024). Incentive-Aware Federated Learning with Training-Time Model Rewards. ICLR.

Goetze et al. (2025). FLOSS: Federated Learning with Opt-Out and Straggler Support. PACMI.

Hashimoto et al. (2018). Fairness Without Demographics in Repeated Loss Minimization. ICML.

Thank you for helping us strengthen the communication analysis and clarify the value of our method. If these revisions address your concerns, we would be grateful if you would consider increasing your score. We would be happy to discuss any remaining questions.

Thank you for the detailed responses. I will maintain my current rating.

Dear reviewer, thank you for your feedback. We believe our response and revised manuscript address the main concerns raised in your review. Could you clarify which specific concerns remain unresolved so we can address them and you can reconsider your rating?

Reviewer_AH3POpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies participation bias in decentralized language-model fine-tuning, particularly a controlled setting where nodes with higher loss relative to the network mean participate less frequently. It proposes RUMOR, which combines inverse estimated participation-frequency weighting, a loss-dependent cap based on a historical local-loss reference, and an additional correction derived from disagreement with active neighbors. The analysis establishes properties of participation-history statistics, gossip averaging, and update-weight bounds. Experiments cover a small synthetic transformer and LoRA fine-tuning of seven pretrained models up to 12B parameters, under static and loss-adaptive participation. The reported benefits primarily concern reduced group-loss disparities and avoidance of severe loss increases exhibited by some aggressive reweighting baselines.

Critical strengths
  1. A relevant problem with a locally implementable intervention. The interaction between poor local performance and reduced participation is a coherent and potentially consequential failure mode. RUMOR uses locally available statistics and neighboring models without requiring the learner to access the network-wide loss used by the simulator. This makes the proposed intervention relevant to decentralized training.

  2. Substantial experimental transparency. The paper includes repeated seeds, independent-seed replication on the small transformer, component ablations, absolute losses, and paired comparisons. It explicitly acknowledges that smaller group gaps can result from degradation of the better-performing group, and retains unstable baseline runs in its summaries. These disclosures make the evidence considerably easier to assess.

Critical weaknesses

The headline improvements do not establish a compelling advantage on the paper’s stated optimization objective.

  • Section 3 specifies minimization of the uniformly weighted average of node losses. However, the prominent normalized group-gap metric can favor a method even when it produces worse losses on both groups.

  • Table 7 provides a concrete example. On GPT-2 medium at (\beta=2), RUMOR obtains low-/high-participation losses of 4.995/3.287, whereas RUMOR, NO HISTORY obtains 4.982/3.263. Thus, the simpler variant improves both groups in the reported means, yet receives a substantially worse headline gap-reduction score. Its uniform average loss is also lower across all three models in Table 7: 4.122 versus 4.141, 3.268 versus 3.289, and 3.422 versus 3.432. This undermines the interpretation of the headline ranking as evidence for better optimization of the stated objective.

  • Appendix L.2 raises a related concern about the stability argument. Interval-average weighting with cutoff 50 has no flagged runs across sixty small-transformer feedback runs, while achieving lower uniform losses than RUMOR by 0.011, 0.024, and 0.037 nats at (\beta=2,4,6), respectively. The favorable comparison against unstable windowed IPW therefore does not establish a general advantage over the stronger stable alternatives already evaluated.

  • Finally, some apparently large normalized gains have small absolute scale: Figure 4 states that 100 reduction points on Qwen 7B correspond to only 0.0039 nats of gap improvement. The authors disclose these issues, but the resulting evidence still leaves the principal contribution unclear: improved optimization of the uniform objective, improved low-participation performance, and smaller between-group gaps are distinct outcomes. The acceptance case needs to be supported consistently against the chosen objective.

The added value of the specific RUMOR construction remains insufficiently established beyond participation weighting and update stabilization.

  • Participation-history weighting and loss-based stabilization have established precedents, which the paper appropriately acknowledges through Wang and Ji (2024) and Ede and Beanland (2020). The novelty therefore rests substantially on the particular combination and its demonstrated benefit.

  • The theoretical results provide limited support for that combination. Proposition 1 assumes matching conditional participation probabilities and establishes indistinguishability of the corresponding histories. Proposition 2 describes properties of Metropolis averaging. The cap inequalities bound update multipliers, but do not establish reduced optimization bias or stability of RUMOR under endogenous participation. In particular, they do not explain when decreasing the cap as local loss rises improves the tradeoff between stability and underrepresentation. These limited guarantees place greater weight on empirical isolation of the proposed components.

  • That empirical evidence is mixed. The fixed-cap variant closely matches or exceeds RUMOR in several feedback settings. Figure 10 shows essentially no benefit from the disagreement component on GPT-2 medium at (\beta=6) or on the static small-transformer task. These results support setting-dependent usefulness, but leave the necessity of the complete construction uncertain.

  • Moreover, Appendix B.2 states that fresh-neighbor observations double model traffic relative to history-only weighting. The main comparisons match rounds rather than communication expenditure, and Appendix D acknowledges that actual update counts can also differ. The earlier-round communication control is useful, but does not establish an advantage at matched communication budgets. Similarly, the selected learning-rate controls in Appendix H.2 do not resolve whether the large-model stability advantage survives joint tuning of learning rate and cap for the strongest baselines. Consequently, the evidence does not yet distinguish a broadly useful correction mechanism from a particular stabilization configuration within the tested simulator.

Questions for authors
  1. Clarification of the target outcome: Which outcome should determine whether participation bias has been successfully corrected: uniform node-average loss, low-participation-group loss, or the group-loss gap? How should the GPT-2 medium result in Table 7—where NO HISTORY improves both groups over RUMOR—be reconciled with the headline ranking? Paired uncertainty estimates for these comparisons would materially affect my assessment.

  2. Evidence for the complete mechanism: Can the authors identify settings where the complete method provides a reproducible improvement in the stated uniform objective over both the fixed-cap variant and IPW with the same loss-dependent cap? In particular, are those improvements sufficient to justify the extra communication required by the disagreement component?

  3. Comparison protocol: How were learning rates and caps selected for the large-model IPW and interval-average baselines? Does the advantage persist with comparable tuning budgets and when performance is measured against cumulative communication bytes and local updates, rather than rounds alone?

  4. Robustness of the feedback mechanism: Appendix J.1 acknowledges that language-specific loss offsets alter participation without changing gradients, while Appendix B.1 describes a substantial cap transient caused by initializing the recent-loss EMA at 5.0. Have the principal comparisons been repeated with calibrated language-relative losses and initialization from observed local losses? This would help determine how much the conclusions depend on these simulator choices.

4 comments
AuthorsOpenReview ↗

Response to Reviewer AH3P (1/2): Optimization objective and correction benefits

Thank you for the detailed review.

We expanded the manuscript by more than a quarter to add conditional correction and finite-time convergence analyses, communication-budget, sensitivity, tuning and loss-calibration studies. We also clarify how disparity and absolute losses should be read together. Substantive changes in the revised manuscript are shown in blue.

[Q1. Optimization objective and group disparity]

We agree that NO HISTORY has lower uniform loss on all three models in the original Table 7. On GPT-2 medium at β=2\beta=2, it improves both groups and the stated training objective. Its worse gap score reflects a larger gain for the high-participation group, so its final gap remains larger than RUMOR's.

Original Table 7 is now Table 10. Its RUMOR and NO HISTORY entries report mean absolute losses and sample standard deviations across ten seeds. Table 11 adds the GPT-2 paired comparison. In 10−310^{-3} nats, NO HISTORY lowers uniform, low-participation-group and high-participation-group losses by 18.525±0.49718.525\pm0.497, 13.306±0.46113.306\pm0.461 and 23.744±0.70123.744\pm0.701, respectively, with paired SEs. All ten seeds favor NO HISTORY in all three outcomes. These estimates separate seed variability from uncertainty in paired comparisons.

On the other hand, RUMOR lowers Turkish/Swahili loss versus NO HISTORY in all ten seeds on Qwen 0.5B at β=2\beta=2 and Qwen 7B under static participation. Here, NO HISTORY's lower uniform loss accompanies worse low-participation-group performance. Therefore, removing history does not consistently benefit both groups, as Appendix I.1 and Table 11 show.

These comparisons explain why we distinguish overall loss from group disparity. Uniform node-average loss remains our training objective, while group-gap reduction is our primary disparity measure. In our participation model, unequal local usefulness can reinforce underrepresentation. The gap captures this aspect but does not establish lower average loss.

Performance-fairness criteria motivate this disparity measure. Ditto uses the standard deviation of client test losses (Li et al., 2021), q-FFL studies performance uniformity across devices (Li et al., 2020), and GIFAIR-FL penalizes differences between group-average losses (Yue et al., 2023). Hu et al. (2024) also discuss how equalizing group losses can sacrifice predictive utility. Our normalized score measures gap reduction relative to uncorrected and full-participation training. We clarified this distinction in Section 6 and Appendix I.1.

For overall optimization, RUMOR also lowers uniform loss against interval-average cutoff 50 on GPT-2 medium and Qwen 0.5B at β=2\beta=2 by (7.8±0.4)×10−3(7.8\pm0.4)\times10^{-3} and (1.1±0.3)×10−3(1.1\pm0.3)\times10^{-3} nats, respectively. Appendix L.2 retains the small-transformer settings where interval-average performs better. On Mistral-Nemo 12B at β=6\beta=6, Table 14 reports three flagged runs for cutoff 50 and none for RUMOR. Cutoff 3 is also stable with similar mean uniform loss.

[Q2. Benefits of the complete method and its communication cost]

Your concern led us to test RUMOR's corrections and the value of its additional neighbor exchange. Section 7 clarifies our intended regime of compute-intensive fine-tuning over high-bandwidth links.

First, we tested whether disagreement improves optimization beyond adaptive-cap participation weighting. On GPT-2 medium at β=2\beta=2, original RUMOR lowers uniform loss by (6.82±0.615)×10−3(6.82\pm0.615)\times10^{-3} nats versus the same adaptive-cap rule without disagreement. Both groups improve in all ten seeds. Appendix I.1, Table 11 reports this new paired analysis of the existing Figure 10 runs.

For communication-limited settings, we tested cached RUMOR. It retains the weighting rule but reuses previously received neighbor models without an additional exchange. On Qwen2.5-7B at β=6\beta=6, we compare it with two ablations under equal byte budgets. The fixed-cap ablation retains disagreement but removes loss-dependent cap adaptation. The no-disagreement ablation removes disagreement but keeps the identical adaptive cap. Table 4 in Appendix H.3 reports uniform-loss gains of (1.37±0.381)×10−3(1.37\pm0.381)\times10^{-3} and (1.81±0.421)×10−3(1.81\pm0.421)\times10^{-3} nats over these ablations, respectively. All three seeds favor cached RUMOR in both comparisons. Therefore, both corrections help without the extra exchange.

To assess that exchange directly, we compare original and cached RUMOR under equal byte budgets on Qwen2.5-7B and Mistral-Nemo 12B at β=2,4,6\beta=2,4,6 in Appendix H.3, Figure 13. Results vary by setting but generally favor cached RUMOR. Therefore, the benefit of fresh observations depends on the setting and available communication. Cached RUMOR retains both corrections and allows more local updates within the same budget. This makes it a practical option when communication limits training.

AuthorsOpenReview ↗

Response to Reviewer AH3P (2/2): Theory, tuning, and loss calibration

[Q2. Continued: theoretical explanation]

To explain when the corrections help, we added Theorem 1 in Appendix K.3. From the same models, history and active set, it gives sufficient conditions for lower expected uniform training loss than capped IPW after one update. These conditions identify when reducing weights through the cap or increasing them through disagreement improves that loss.

[Q3. Comparable tuning and resource budgets]

The original pretrained comparisons use learning rate 0.020.02, IPW caps 3.33,203.33,20 and interval-average cutoffs 3,503,50. The cap endpoints were retained from comparisons of tighter and looser IPW caps during method development.

Your feedback led us to add joint learning-rate/cap tuning on Mistral-Nemo 12B and Qwen2.5-7B at β=6\beta=6. Within each model, each method received six configurations, each trained for 1,000 rounds on one shared selection seed. We selected by final validation loss and fixed configurations before evaluation on three separate seeds. Appendix H.4 describes the protocol, and Table 5 and Figure 14 report both studies.

All methods selected learning rate 0.020.02 on both models. RUMOR selected upper cap 10 and IPW fixed cap 20. Interval-average selected cutoff 3 on Mistral-Nemo 12B and cutoff 50 on Qwen2.5-7B.

The table reports mean final uniform-loss differences after tuning, with paired SEs across three evaluation seeds. Differences are RUMOR minus the comparator, in nats. Negative values favor RUMOR.

ModelVersus tuned IPWVersus tuned interval-average
Mistral-Nemo 12B−0.0657±0.0475-0.0657\pm0.0475+0.00825±0.00735+0.00825\pm0.00735
Qwen2.5-7B−0.00074±0.00425-0.00074\pm0.00425−0.7087±0.5903-0.7087\pm0.5903

On Mistral-Nemo 12B, RUMOR has lower mean loss than tuned IPW, while tuned interval-average has the lowest mean loss. Figure 14 shows two tuned IPW runs losing progress and finishing above RUMOR.

On Qwen2.5-7B, RUMOR and tuned IPW have similar mean loss. RUMOR avoids the large late loss increases observed in two tuned interval-average runs, and its loss decreases at every recorded checkpoint in all three seeds.

Together, these studies support RUMOR's balance between model quality and training stability after comparable tuning. Q2 complements these fixed-round studies with equal-byte-budget comparisons. Under that constraint, cached RUMOR generally achieves lower loss than original RUMOR and completes more local updates.

[Q4. Loss calibration and measured initialization.]

We added the study in Appendix J.2 to test whether gains depend on initial language-loss differences or loss-history initialization. We compared the original setup with adjusted loss inputs alone, measured initialization alone, and both changes together.

Language-relative inputs subtract each language's initial loss from its later losses and add the initial node-average loss. These inputs control participation and the adaptive cap. They remove starting differences while preserving subsequent loss changes.

We measure initial losses on 64 training sequences per language and use them to initialize the recent, slow and reference histories. With language-relative inputs, these histories start from the initial node-average loss. The default recent-loss average instead starts at 5.05.0. Training gradients and test losses retain the original cross-entropy throughout these controls.

We evaluated GPT-2 medium at β=2,6\beta=2,6 and Qwen2.5-7B at β=6\beta=6, with three paired seeds and 1,000 rounds per configuration. With both changes, RUMOR lowers uniform loss versus cap-20 IPW in every seed across all three settings. Both group means also improve, as Tables 12–13 report. Therefore, these gains persist after removing initial language offsets and initializing loss histories from measured values.

References

Li et al. (2021). Ditto: Fair and Robust Federated Learning Through Personalization. ICML.

Li et al. (2020). Fair Resource Allocation in Federated Learning. ICLR.

Yue et al. (2023). GIFAIR-FL: A Framework for Group and Individual Fairness in Federated Learning. INFORMS Journal on Data Science.

Hu et al. (2024). Fair Federated Learning via Bounded Group Loss. SaTML.

We hope the additional controls and clearer loss comparisons address your concerns. If they do, we would be grateful if you would consider increasing your score. We would be glad to clarify any remaining points.

Thanks for your response, I have increased my overall score from 2 to 3.

Dear Reviewer AH3P, thank you for reassessing our work and for your helpful feedback. We appreciate your time and consideration.

Reviewer_1f3LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Motivation: The paper addresses a vicious cycle of participation bias in decentralized fine-tuning: a node that participates less makes fewer local updates and model exchanges, so its data has less influence on the model; the model then performs worse on that node, the node becomes less willing to participate, and participation drops further. The common fix, inverse-probability weighting (IPW), can make training unstable or even cause loss blowups when the weights are large, while a small cap undercompensates. Moreover, participation counts alone cannot tell whether low participation actually comes with a deteriorating model.

Method: (1) RUMOR estimates each node's participation rate from its recent participation history and, when an infrequent node comes online, scales up its update by the inverse of that rate to compensate for missed training. (2) It tracks the local loss against its historical best: when the loss is stable, a large weight cap is allowed (up to 20); when the loss deteriorates, the cap is tightened toward 3.33 so that overly large updates do not destabilize training. (3) It compares the node's model with those of its active neighbors; a large discrepancy is taken as a sign that the node may have fallen behind, and the update is further increased when the participation history also supports this. The whole method relies only on local information, but the neighbor-disagreement module requires one extra model exchange.

Main contributions: The paper proposes a safe weighting strategy that is more stable than aggressive IPW and uses extensive ablations and controls to show the role of each component. The core novelty is using the local loss to adaptively adjust the weight cap.

Experimental design: The task is language model fine-tuning with a next-token cross-entropy loss. The paper compares two participation mechanisms, static participation and loss-adaptive participation, the latter at several feedback strengths. The testbeds are a small Transformer on 16 nodes with synthetic data, and seven pretrained models from GPT-2 to Mistral-Nemo 12B fine-tuned with LoRA on 8 nodes, with English and German as the high-participation group and Turkish and Swahili as the low-participation group. Baselines include several IPW variants, disagreement-based weighting methods, and component ablations of RUMOR. Metrics include the loss gap between the low- and high-participation groups, the absolute loss of each group, the uniform node-average loss, and the number of severe failures.

Conclusion: In the environments the authors construct, RUMOR is an effective engineering trade-off: under strong feedback it brings the small Transformer's group gap close to the full-participation level, and it avoids the loss blowups that aggressive IPW suffers on large models. However, its advantage over IPW with a small fixed cap is limited, it has not been validated under real user behavior, and it comes with no convergence or unbiasedness guarantee.

Critical strengths
  1. The problem is practically meaningful. The paper studies the negative feedback loop in which poor model performance reduces participation and reduced participation further degrades performance. The proposed method relies only on participation history, local loss, and neighbor models, preserving decentralization while balancing bias correction and training stability, and it markedly reduces loss blowups compared with aggressive IPW.

  2. The experiments are comprehensive. They cover scales from a small model up to 12B and both static and adaptive participation, with thorough baselines, component ablations, paired comparisons, and replication on independent seeds. The paper also reports per-group losses and severe failure cases and is candid about the settings where the method falls behind, which makes its benefits, costs, and scope clear.

Critical weaknesses
  1. The method is largely heuristic. The exponential form of the loss-dependent cap, the use of the historical minimum loss as the reference, the disagreement weight, and several thresholds are mostly set empirically, without a systematic sensitivity analysis. The theory only argues that these signals may be complementary; it does not show that RUMOR recovers full-participation training without bias, and there is no convergence analysis. The neighbor-disagreement module only takes effect at intermediate participation rates, so it does little for the least-participating nodes, yet it doubles communication.

  2. The experimental setting and the gains remain limited. The experiments use only 8 to 16 nodes on a fixed topology, and the participation mechanism is synthetic and may not reflect real user behavior. The absolute improvements on pretrained models are small , and the normalized metric amplifies the differences. Compared with IPW using a small fixed cap, the advantage under feedback is unclear; RUMOR is also not the best on the uniform node-average loss that the paper states as its objective, and part of the gap reduction comes from higher loss on the high-participation group.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1f3L (1/2): Design, convergence, and communication

Thank you for highlighting where RUMOR's design and evaluation needed stronger support. Your feedback led us to add sensitivity, convergence and communication-budget studies.

We have expanded the manuscript by more than a quarter in response to the reviews. The revision adds a conditional analysis of our weight corrections and a finite-time convergence analysis, together with communication-budget, sensitivity, tuning and loss-calibration studies. We also clarify how disparity and absolute losses should be interpreted together. Substantive changes in the revised manuscript are shown in blue.

[Design choices and sensitivity]

Your concern about empirical design choices led us to add sensitivity tests on GPT-2 medium, Qwen2.5-7B and Mistral-Nemo 12B. Our exponential rule lowers the cap smoothly between its preset limits as loss rises above its reference. The historical minimum records the best observed smoothed local loss. To test sensitivity to these choices, we varied the cap scale and disagreement threshold. We also replaced the historical minimum with the minimum over each node's 50 most recent smoothed-loss observations.

These changes produced small shifts in mean uniform loss. We also confirmed that both corrections were active in every default 7B seed. The results demonstrate robustness to the tested parameter and loss-reference choices (Appendix H.5, Table 6).

[Convergence and correction benefits]

Your feedback encouraged us to develop a convergence analysis for our method and explain when its corrections benefit optimization. Theorem 2 now bounds the average expected squared gradient norm of the uniform training objective under its stated assumptions. It accounts for unequal weighted contributions, differences between local models and bias introduced by gradient clipping. Appendix K.4 provides the assumptions, proof and interpretation of this bound.

Theorem 1 complements this analysis. It gives conditions under which our loss-dependent cap and disagreement correction improve expected uniform training loss relative to capped IPW after one update. The comparison starts from the same models, history and active set, so it isolates the effect of the weight changes. We present these conditions and their proof in Appendix K.3 now.

[Disagreement and communication]

The least-participating nodes already receive history weights at or above the maximum disagreement weight. Their history correction and adaptive cap still apply. Disagreement does not further increase these already-large weights (Appendix K.2, Corollary 2).

Our intended regime is compute-intensive fine-tuning over high-bandwidth links. Your feedback led us to evaluate an alternative for settings with limited communication budgets. Therefore, we tested an ablation that reuses previously received neighbor models. It keeps RUMOR's weighting rule and avoids the additional exchange. We call this ablation cached RUMOR. We compared it with the baselines under equal communication budgets.

Under these matched budgets, cached RUMOR lowers uniform loss on Qwen2.5-7B at β=6\beta=6 versus cap-20 IPW and both its fixed-cap and no-disagreement ablations in every seed. On Mistral-Nemo 12B at the same feedback strength, it also avoids the prolonged loss increases observed in two IPW runs, as shown in Appendix H.3, Figure 13 and Table 4.

We also compare original and cached RUMOR under equal byte budgets on both models at β=2,4,6\beta=2,4,6. Relative performance varies across settings, although these comparisons generally favor cached RUMOR. Therefore, the benefit of fresh observations depends on the setting and available communication. These results support cached RUMOR as a practical option when communication limits training, since it retains both corrections while allowing more local updates within the same budget.

AuthorsOpenReview ↗

Response to Reviewer 1f3L (2/2): Communication and overall model quality

[Participation modeling, evaluation scope and model quality]

Your feedback helped us further clarify the motivation for our participation model. In this model, poor local performance makes nodes less likely to contribute to training. We explain this modeling choice in Sections 1–2. MaxFL (Cho et al., 2024) links continued participation to the shared model meeting clients' local performance requirements. Similarly, IAFL (Wu et al., 2024) models resource contributions through model benefits relative to contribution costs. These works provide precedents for our controlled loss-adaptive participation scenario.

We also clarified how our evaluation distinguishes disparity reduction from improvements in average performance. Group-gap reduction remains our primary disparity metric, while uniform and group losses show whether the model improves overall and who benefits. On the small transformer at β=4\beta=4, RUMOR improves integrated-gap reduction over small-cap IPW, as shown in Table 3. Separately, on GPT-2 medium at β=2\beta=2, RUMOR with its original neighbor exchange lowers uniform loss by (6.82±0.615)×10−3(6.82\pm0.615)\times10^{-3} nats versus the same adaptive-cap rule without disagreement. The paired analysis in Appendix I.1, Table 11 shows that both groups improve in every seed. Thus, this gain does not come from worsening the better-served group. Appendix L.2 also shows that RUMOR lowers uniform loss relative to interval-average cutoff 50 on GPT-2 medium and Qwen 0.5B at β=2\beta=2. In the new joint-tuning study, RUMOR also lowers mean uniform loss versus interval-average on Qwen2.5-7B at β=6\beta=6 by 0.7087±0.59030.7087\pm0.5903 nats, with paired SE across three seeds, as Table 5 in Appendix H.4 reports. Figure 14 shows the large late loss increases in two interval-average runs that RUMOR avoids.

References

Cho et al. (2024). Maximizing Global Model Appeal in Federated Learning. TMLR.

Wu et al. (2024). Incentive-Aware Federated Learning with Training-Time Model Rewards. ICLR.

Thank you again for helping us strengthen the analysis and evaluation of RUMOR. If these revisions address your concerns, we would be grateful if you would consider increasing your score. We would be happy to clarify any remaining points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.