acceptodds
Under review as a conference paper at ICLR 2027

Self-Play Enhancement via Advantage-Weighted Refinement in Online Federated LLM Fine-Tuning

Abstract

Recent works have advanced feedback-based learning systems, whereby a foundation model is able to intake incoming feedback (e.g., from a user) to self-improve, creating a self-loop system of training. However, existing works are limited in needing to consider an offline setup to allow for such feedback-based methods, requiring privileged ground-truth contexts for training. Moreover, there is limited consideration of federated learning (FL), which is particularly well-suited for incorporating external feedback across large networks of end users, for example, but requires methods to be efficient for training on resource-constrained edge devices. Therefore, we introduce SPEAR (Self-Play Enhancement via Advantage-Weighted Refinement), an efficient online learning algorithm for federated LLM fine-tuning. SPEAR utilizes a feedback-guided self-play loop to construct naturally contrastive pairs per prompt which are used to train the model with (i) standard maximum likelihood on satisfactory completions and (ii) confidence-weighted unlikelihood on tail tokens of unsatisfactory completions. SPEAR requires only an interaction-level binary satisfaction signal together with non-answer feedback, without requiring a ground-truth answer to be provided, and does not need expensive group generations, meaning training both online and in a resource-efficient manner is feasible. We validate SPEAR across various benchmark datasets, demonstrating its superior performance in comparison to relevant baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Foundation Models, Large Language Models, Self-Play, Federated Learning, Online

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bZT8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates online learning from interaction-level binary satisfaction signals and partial textual feedback in federated LLM fine-tuning. SPEAR first has client models generate responses and attempt to revise unsatisfactory responses based on feedback. It then applies maximum-likelihood training to satisfactory responses and an unlikelihood loss to tail tokens of unsuccessful responses whose probabilities exceed a confidence threshold. The server aggregates local models with weights proportional to the number of satisfactory responses obtained by each client. The paper also analyzes the log-probability margin induced by the joint objective under specific assumptions. The main experiments evaluate Llama3.2-3B and Qwen2.5-1.5B on four benchmarks in a simulated federated environment with 50 clients, reporting accuracy improvements over Feedback SFT, GRPO, and other baselines, alongside lower training times in most settings.

Critical strengths

S1. The method provides a coherent design for learning from both successful and unsuccessful responses. SPEAR not only retains satisfactory responses obtained through feedback-guided revision but also uses initially unsuccessful responses as negative training signals. Pairing revised responses with the original prompts allows improvements obtained through feedback to translate into the model’s ability to answer without additional hints. Confidence gating and tail-token selection restrict the scope of negative penalties, rather than indiscriminately suppressing entire unsuccessful responses. These components address a common objective and form a relatively simple, understandable training mechanism.

S2. Comparisons with Feedback SFT provide evidence for the effectiveness of the joint objective. In Table 1, SPEAR achieves the highest mean accuracy across all combinations of the two models and four tasks. In particular, compared with Feedback SFT, which also uses feedback to obtain satisfactory training responses but does not apply unlikelihood training, the average accuracy across tasks increases from 62.66% to 69.61% for Llama3.2-3B and from 63.87% to 70.68% for Qwen2.5-1.5B, corresponding to improvements of 6.95 and 6.81 percentage points, respectively.

S3. The efficiency claims are supported by measured training times. Figure 2 shows that SPEAR achieves the shortest training time in most settings. In settings where Feedback SFT is faster, the time difference is small. Together with the accuracy results in Table 1, these measurements indicate that the performance improvements are not simply attributable to longer training.

Critical weaknesses

W1. The information content of the feedback in the main experiments is in tension with the claim of “no privileged context.” Section 5 states that the feedback for ARC-Challenge and HellaSwag includes the first two or three words of the correct option, while MathMCQA uses the beginning of the reference reasoning, and satisfaction is determined by task correctness. In a multiple-choice setting, a short prefix of the correct option may already be sufficient to identify it uniquely. Therefore, withholding the complete reference answer does not necessarily mean withholding information that identifies the correct answer. This protocol supports the method’s effectiveness under partial feedback constructed from reference information, but does not by itself establish the “no privileged context” setting emphasized in the contributions. The paper should clarify how much answer information the feedback is allowed to contain and the scope of the practical claims supported by this setting.

W2. The applicability of the joint objective under erroneous satisfaction judgments remains unclear. The main experiments construct binary satisfaction signals from task correctness. However, incomplete or noisy textual feedback is distinct from errors in the satisfaction judgments themselves. When responses are incorrectly accepted or rejected, both the positive and negative objectives may produce misleading updates. The main experiments support SPEAR’s effectiveness under reliable satisfaction judgments, but do not directly establish whether its advantage over Feedback SFT is maintained when the binary signal contains a small proportion of errors and negative training is added.

Questions for authors

Q1. For what proportion of the training examples in ARC-Challenge and HellaSwag do the provided first two or three words uniquely identify the correct option? Could the authors compute this statistic using the existing data and clarify the operational definitions of “non-answer feedback” and “no privileged context”?

Q2. How would SPEAR’s advantage over Feedback SFT be expected to change when the binary satisfaction signal contains occasional false acceptances or false rejections? Could combining positive and negative training increase sensitivity to such errors, and to what extent can the current design mitigate this effect?

3 comments
AuthorsOpenReview ↗

Authors Response to Reviewer bZT8

We thank the reviewer for the thoughtful review and suggestions. Below, we answer each concern.

  • (W1) We thank the reviewer for the observation. To address this concern, we would like to direct the reviewer to Appendix B.12 of the original submitted manuscript. Here, we considered a weaker feedback type called "Process Critique". This feedback only gave generic feedback (e.g., "The answer is unsatisfactory. Find the first error and continue from there."). With the results from Appendix B.12, we demonstrated that SPEAR still obtains better performance than the baselines in Table 1 of the manuscript, meaning SPEAR can still achieve good performance under generic advice. To better highlight these results, we have moved this experiment from Appendix B.12 to the main text of the manuscript, in Sec. 5 (added section colored under blue text).

  • (W2/Q2) We thank the reviewer for the observation. To address this concern, we present results on a noisy satisfaction signal in conjunction with Process Critique feedback in Table A below. Here, the binary satisfaction signal can be incorrect occasionally (we explore no error, 5% error, and 10% error) on Qwen2.5-1.5B. From the results, we demonstrate that under generic process critique feedback and noisy satisfaction signals, SPEAR still obtains good performance, indicating SPEAR's robustness to such issues. We see even under this dual challenging scenario, SPEAR obtains better performance than non-error and original partial feedback-based Feedback SFT on all settings besides 10% error on ARC-Challenge in Table 1 of the manuscript (even in that scenario, the difference is small: 73.54 vs. 73.89). When placing Feedback SFT under the same constraints, it is always worse than SPEAR, as seen in Table A.

Table A: Generic Process Critique and Error Signal-based SPEAR for Qwen2.5-1.5B

MethodDatasetNo Error5% Error10% Error
SPEARARC-Challenge76.79±0.8476.25±0.7073.54±0.58
Feedback SFTARC-Challenge74.17±0.7474.06±1.6772.63±0.70
SPEARHellaSwag82.15±0.8881.97±0.8279.91±1.23
Feedback SFTHellaSwag80.38±0.2479.34±0.5277.25±0.78
SPEARStrategyQA64.92±1.8964.68±0.2563.65±0.72
Feedback SFTStrategyQA51.52±0.0051.52±0.0051.52±0.00
  • (Q1) We thank the reviewer for the question. We note the first two/three words uniquely identify the correct option in 75.07% of ARC-Challenge examples and 92.39% of HellaSwag examples, using case-insensitive prefix matching. These statistics concern potential hints across training examples, rather than only interactions that received feedback. Although none of the HellaSwag prefixes reproduce the entire answer text, they frequently identify the correct option. Despite the statistics found above, as discussed in our response to (W1), Appendix B.12 from the original submitted manuscript demonstrated meaningful learning with generic process critique feedback. For example, Qwen2.5-1.5B achieved 76.79% on ARC-Challenge and 82.15% on HellaSwag, with Llama3.2-3B being 76.19% and 87.57%, which is importantly better than all baselines in Table 1 of the manuscript. This means that even when specific information pertaining to an individual sample is not given in the feedback, SPEAR is still able to learn the task well. This supports the claim that SPEAR can operate under very challenging non-privileged context scenarios, while still receiving satisfaction-based supervision.

    Lastly, in light of these observations and the reviewer's comment on defining "no privileged context" and "non-answer feedback", we unify their definition to "non-privileged context" and define it as feedback that does not directly reveal the answer within the feedback (e.g., "The answer is A."). This has been updated in the manuscript in "Challenges" section of the Introduction (colored in blue).

We appreciate your time and effort in reviewing our paper. We would be happy to address any remaining questions or concerns the reviewer might have.

Reviewer bZT8,

We thank you for taking the time to review our work. We kindly request that you consider our rebuttal in response to your concerns. If the above responses resolve the remaining concerns, we would be grateful if you could maintain, or if appropriate further strengthen, the positive assessment.

Reviewer bZT8,

Once again, we thank you for taking the time to review our work. As the rebuttal will end soon, we kindly request that you consider our rebuttal in response to your concerns. If the above responses resolve the remaining concerns, we would be grateful if you could maintain, or if appropriate further strengthen, the positive assessment.

Reviewer_DUx9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SPEAR (Self-Play Enhancement via Advantage-Weighted Refinement), an efficient online learning algorithm for federated LLM fine-tuning. First, the authors formulate a feedback-guided environment for fine-tuning of LLMs in an FL setting. They then propose a novel feedback-guided self-play enhanced methodology for LLM fine-tuning, which seeks to separate samples into win and lose traces to create contrasting samples for training. Via hyperparameters that control the confidence and number of tail tokens targeted for unsatisfactory lose traces, SPEAR’s unlikelihood auxiliary loss discourages high-confidence tokens associated with unsatisfactory outputs. The authors conducted extensive experiments on a diverse set of benchmarks to prove the superiority of the method.

Critical strengths

1)This paper addresses a valuable question by exploring how to leverage interactive satisfaction signals and incomplete feedback for federated fine-tuning, reducing the need for users to provide complete standard answers. This aligns with the reality that feedback is readily available while complete demonstrations are difficult to obtain, and also links feedback learning with client-side computational constraints.

2)The solution proposed in this paper is clear: SPEAR uses maximum likelihood training on successful trajectories and applies unlikelihood penalties to failed trajectories, effectively utilizing both types of interactive data. In particular, training by pairing successfully revised answers with the original questions transforms feedback-assisted error correction capabilities into initial answering capabilities that require no additional feedback.

3)The experimental section of this paper effectively demonstrates the superiority of the method, achieving a performance of 69.61 on Llama3.2-3B and 70.68 on Qwen2.5-1.5B, as shown in Table 1. The wall-clock time experiment also proves the efficiency of the method.

Critical weaknesses

1)The feedback construction may reveal ground-truth answer information indirectly. For example, the feedback for ARC-Challenge and HellaSwag includes the first 2–3 words of the correct option, while all options are already present in the input; therefore, this hint may be sufficient to uniquely locate the correct option. Similarly, for StrategyQA, given a valid binary prediction, a reliable incorrectness signal effectively identifies the opposite answer as correct. Therefore, these findings are insufficient to prove that the experiment does not rely on answer information.

2)During aggregation, the aggregation weight is proportional to the number of wins by the client, which may continuously amplify clients with simple tasks or easier-to-satisfy feedback, while weakening valuable failure learning signals from difficult clients. The paper does not separately ablate this aggregation scheme against alternatives, such as weighting by the total number of interactions or assigning uniform weights to participating clients.

3)Negative examples that were successfully revised and failed to be revised were assigned α=1 and α=0.5, respectively. However, since both types of initial responses are labeled unsuccessful by the same binary satisfaction function, whether subsequent revisions were successful does not directly reflect the difference in reliability of the initial failure labels. The paper did not compare other weight designs, therefore the contribution of this weighting mechanism cannot yet be clearly defined.

4)Minor formatting issues: Inconsistent writing style. For example, in line 312, only the first letter of the first word is capitalized in the bolded part, but in line 326, the first letter of each word is capitalized.

Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Authors Response to Reviewer DUx9

We thank the reviewer for the thoughtful review and suggestions. Below, we answer each concern.

  • (W1) We thank the reviewer raising this point. To address this concern, we would like to direct the reviewer to Appendix B.12 of the original submitted manuscript. There, we considered a weaker feedback type called "Process Critique". This feedback only gave generic feedback (e.g., "The answer is unsatisfactory. Find the first error and continue from there."). With these results, we demonstrated that SPEAR still obtains better performance than the baselines in Table 1 of the manuscript, meaning SPEAR can still achieve good performance under generic advice. To better highlight these results, we have moved this experiment from Appendix B.12 to the main text of the manuscript, as the last experimental results of Sec. 5 (colored in blue text).

  • (W2) We thank the reviewer for the observation. To address this concern, we present results on both FedAvg-sample based and uniform aggregation in Table A on Qwen2.5-1.5B. We see that the proposed scheme achieves better performance, giving validity to our methodology.

Table A: SPEAR vs. Differing Aggregation Schemes on Qwen2.5-1.5B

Method / AblationARC-ChallengeHellaSwagStrategyQAAvg.
Uniform Weighting76.45±0.3981.34±0.1865.57±1.0974.45
Standard FL Weighting75.17±1.2179.94±0.9964.48±0.5273.20
SPEAR76.93±1.4180.65±0.7966.08±3.2074.55
  • (W3) We thank the reviewer for the observation. To address this, we perform an ablation of giving every lose trace a weight of 1. The results can be found below in Table B on Qwen2.5-1.5B. We can see that even under this ablation, that SPEAR still obtains higher performance with the lose trace weighting than without it.

Table B: SPEAR vs. Lose Weight Always 1 on Qwen2.5-1.5B

Method / AblationARC-ChallengeHellaSwagStrategyQAAvg.
No Loss Weighting76.65±0.3382.22±0.1364.38±0.7674.42
SPEAR76.93±1.4180.65±0.7966.08±3.2074.55
  • (W4) We thank the reviewer for indicating this formatting inconsistency. We have updated the manuscript to remedy this.

We appreciate your time and effort in reviewing our paper. We would be happy to address any remaining questions or concerns the reviewer might have.

Reviewer DUx9,

We thank you for taking the time to review our work. As the discussion deadline approaches, we kindly request that you consider our rebuttal in response to your concerns and hope that you can reconsider your score in light of our clarifications and additional experimental evidence.

Reviewer DUx9,

Once again, we thank you for taking the time to review our work. As the rebuttal will end soon, we wanted to follow up again and check whether we have addressed your concerns. If so, we hope you can reconsider your score in light of our clarifications and additional experimental evidence. We would be grateful for an opportunity to address any additional questions you have.

Reviewer_3QDKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SPEAR, an online federated LLM fine-tuning method that uses binary satisfaction signals and partial feedback. It collects successful and unsuccessful responses through self-correction and trains the model using maximum-likelihood learning for successful responses and selective unlikelihood learning for unsuccessful ones. The method also uses the number of successful responses to weight client updates during aggregation. Experiments on several benchmarks and LLM backbones show improvements in accuracy and training efficiency over existing baselines.

Critical strengths
  1. The paper studies a useful setting in which clients provide satisfaction signals and limited feedback rather than complete reference answers. SPEAR turns the resulting self-corrections into win and lose trajectories and trains on both. This makes use of unsuccessful interactions without relying on costly group-based sampling.
  2. On the main four-task evaluation, SPEAR improves average accuracy over Feedback SFT from 62.66% to 69.61% on Llama3.2-3B and from 63.87% to 70.68% on Qwen2.5-1.5B. The wall-clock comparisons and additional federated settings also help support the practical value of the approach.
Critical weaknesses
  1. The paper varies the confidence threshold, tail length, and loss weight, but does not fully isolate the effects of the proposed components. A direct comparison with vanilla unlikelihood training without these token-selection mechanisms is missing, as are separate tests of trace-level confidence weighting and win-count-weighted aggregation. These controls would clarify which parts of SPEAR account for its performance gains.
  2. Most main experiments use Qwen2.5-1.5B and Llama3.2-3B. The appendix adds Qwen3.5-2B and Qwen2.5-7B, but these evaluations use fewer baselines, and the Qwen3.5-2B results do not report variability. The average gain over Feedback SFT on Qwen2.5-7B is only 0.54 percentage points. More comprehensive results on recent backbones would help establish how well SPEAR generalizes to stronger models.
Questions for authors

same to weakness

3 comments
AuthorsOpenReview ↗

Authors Response to Reviewer 3QDK

We thank the reviewer for the thoughtful review and suggestions. Below, we answer each concern.

  • (W1) We thank the reviewer for the constructive feedback. In Table A below, we present additional ablations of (i) vanilla unlikelihood training (UL), (ii) removal of the confidence threshold (No Confidence Thres.) / unlikelihood loss weighting (No Loss Weighting), and (iii) uniform / standard FL-example based weighting at the server in Table A. In addition, we present results of no tailing on MathMCQA and StrategyQA with Qwen2.5-1.5B in Table B, which exhibits longer responses and where tailing is more relevant. From these results, we demonstrate that SPEAR maintains better performance than these ablations, justifying our methodological setup.

Table A: Additional ablations on Qwen2.5-1.5B

Method / AblationARC-ChallengeHellaSwagStrategyQAAvg.
UL76.37±0.3080.72±0.0165.26±0.1874.12
No Confidence Thres.73.00±3.7282.78±1.3565.25±0.6573.68
No Loss Weighting76.65±0.3382.22±0.1364.38±0.7674.42
Uniform Weighting76.45±0.3981.34±0.1865.57±1.0974.45
Standard FL Weighting75.17±1.2179.94±0.9964.48±0.5273.20
SPEAR76.93±1.4180.65±0.7966.08±3.2074.55

Table B: No Tail Ablation on MathMCQA and StrategyQA for Qwen2.5-1.5B

MethodMathMCQAStrategyQAAvg.
No Tail56.74±0.6464.57±0.5660.66
SPEAR59.07±1.1666.08±3.2062.58
  • (W2) We thank the reviewer for the observation. We have added standard deviations for the Qwen3.5-2B results in Table 14 of Appendix B.11 in the updated manuscript. In addition, we note in general better gains for SPEAR in comparison to Feedback SFT across varying models, such as seen with other models we evaluated: Qwen2.5-1.5B (70.68 vs. 63.87 in Table 1), Llama 3.2-3B (69.61 vs. 62.66 in Table 1), and Qwen3.5-2B (70.73 vs. 67.22 in Table 14 of the manuscript). In addition, better performance on these smaller scales demonstrates SPEAR's suitability to FL scenarios, where client resources are a significant consideration.

    Lastly, we have extended the results of Table 14 (updated table presented below in Table C) from Appendix B.11 to include RLTF-SD and OPSD. These updated results present a more comprehensive picture on recent backbones.

Table C: Extended Results on More Recent Architecture: Qwen3.5-2B

AlgorithmARC-ChallengeStrategyQAMathMCQAHellaSwagAvg.
OPSD43.94±19.6224.89±24.890.00±0.0024.77±0.0223.40
RLTF-SD83.02±0.2664.11±1.9751.28±0.2683.42±0.3270.46
Online Feedback DPO80.56±0.0439.80±0.2231.34±1.7060.44±0.0453.04
Feedback SFT82.59±0.4248.47±0.0052.60±0.6685.21±0.2067.22
SPEAR (Ours)82.79±0.0959.83±8.3054.38±0.9285.92±0.2570.73

We appreciate your time and effort in reviewing our paper. We would be happy to address any remaining questions or concerns the reviewer might have.

Reviewer 3QDK,

We thank you for taking the time to review our work. As the discussion deadline approaches, we kindly request that you consider our rebuttal in response to your concerns and hope that you can reconsider your score in light of our clarifications and additional experimental evidence.

Reviewer 3QDK,

Once again, we thank you for taking the time to review our work. As the rebuttal will end soon, we wanted to follow up again and check whether we have addressed your concerns. If so, we hope you can reconsider your score in light of our clarifications and additional experimental evidence. We would be grateful for an opportunity to address any additional questions you have.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.