- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates online learning from interaction-level binary satisfaction signals and partial textual feedback in federated LLM fine-tuning. SPEAR first has client models generate responses and attempt to revise unsatisfactory responses based on feedback. It then applies maximum-likelihood training to satisfactory responses and an unlikelihood loss to tail tokens of unsuccessful responses whose probabilities exceed a confidence threshold. The server aggregates local models with weights proportional to the number of satisfactory responses obtained by each client. The paper also analyzes the log-probability margin induced by the joint objective under specific assumptions. The main experiments evaluate Llama3.2-3B and Qwen2.5-1.5B on four benchmarks in a simulated federated environment with 50 clients, reporting accuracy improvements over Feedback SFT, GRPO, and other baselines, alongside lower training times in most settings.
Critical strengths
S1. The method provides a coherent design for learning from both successful and unsuccessful responses. SPEAR not only retains satisfactory responses obtained through feedback-guided revision but also uses initially unsuccessful responses as negative training signals. Pairing revised responses with the original prompts allows improvements obtained through feedback to translate into the model’s ability to answer without additional hints. Confidence gating and tail-token selection restrict the scope of negative penalties, rather than indiscriminately suppressing entire unsuccessful responses. These components address a common objective and form a relatively simple, understandable training mechanism.
S2. Comparisons with Feedback SFT provide evidence for the effectiveness of the joint objective. In Table 1, SPEAR achieves the highest mean accuracy across all combinations of the two models and four tasks. In particular, compared with Feedback SFT, which also uses feedback to obtain satisfactory training responses but does not apply unlikelihood training, the average accuracy across tasks increases from 62.66% to 69.61% for Llama3.2-3B and from 63.87% to 70.68% for Qwen2.5-1.5B, corresponding to improvements of 6.95 and 6.81 percentage points, respectively.
S3. The efficiency claims are supported by measured training times. Figure 2 shows that SPEAR achieves the shortest training time in most settings. In settings where Feedback SFT is faster, the time difference is small. Together with the accuracy results in Table 1, these measurements indicate that the performance improvements are not simply attributable to longer training.
Critical weaknesses
W1. The information content of the feedback in the main experiments is in tension with the claim of “no privileged context.” Section 5 states that the feedback for ARC-Challenge and HellaSwag includes the first two or three words of the correct option, while MathMCQA uses the beginning of the reference reasoning, and satisfaction is determined by task correctness. In a multiple-choice setting, a short prefix of the correct option may already be sufficient to identify it uniquely. Therefore, withholding the complete reference answer does not necessarily mean withholding information that identifies the correct answer. This protocol supports the method’s effectiveness under partial feedback constructed from reference information, but does not by itself establish the “no privileged context” setting emphasized in the contributions. The paper should clarify how much answer information the feedback is allowed to contain and the scope of the practical claims supported by this setting.
W2. The applicability of the joint objective under erroneous satisfaction judgments remains unclear. The main experiments construct binary satisfaction signals from task correctness. However, incomplete or noisy textual feedback is distinct from errors in the satisfaction judgments themselves. When responses are incorrectly accepted or rejected, both the positive and negative objectives may produce misleading updates. The main experiments support SPEAR’s effectiveness under reliable satisfaction judgments, but do not directly establish whether its advantage over Feedback SFT is maintained when the binary signal contains a small proportion of errors and negative training is added.
Questions for authors
Q1. For what proportion of the training examples in ARC-Challenge and HellaSwag do the provided first two or three words uniquely identify the correct option? Could the authors compute this statistic using the existing data and clarify the operational definitions of “non-answer feedback” and “no privileged context”?
Q2. How would SPEAR’s advantage over Feedback SFT be expected to change when the binary satisfaction signal contains occasional false acceptances or false rejections? Could combining positive and negative training increase sensitivity to such errors, and to what extent can the current design mitigate this effect?
3 comments
Reviewer bZT8,
We thank you for taking the time to review our work. We kindly request that you consider our rebuttal in response to your concerns. If the above responses resolve the remaining concerns, we would be grateful if you could maintain, or if appropriate further strengthen, the positive assessment.
Reviewer bZT8,
Once again, we thank you for taking the time to review our work. As the rebuttal will end soon, we kindly request that you consider our rebuttal in response to your concerns. If the above responses resolve the remaining concerns, we would be grateful if you could maintain, or if appropriate further strengthen, the positive assessment.
Authors Response to Reviewer bZT8
We thank the reviewer for the thoughtful review and suggestions. Below, we answer each concern.
(W1) We thank the reviewer for the observation. To address this concern, we would like to direct the reviewer to Appendix B.12 of the original submitted manuscript. Here, we considered a weaker feedback type called "Process Critique". This feedback only gave generic feedback (e.g., "The answer is unsatisfactory. Find the first error and continue from there."). With the results from Appendix B.12, we demonstrated that SPEAR still obtains better performance than the baselines in Table 1 of the manuscript, meaning SPEAR can still achieve good performance under generic advice. To better highlight these results, we have moved this experiment from Appendix B.12 to the main text of the manuscript, in Sec. 5 (added section colored under blue text).
(W2/Q2) We thank the reviewer for the observation. To address this concern, we present results on a noisy satisfaction signal in conjunction with Process Critique feedback in Table A below. Here, the binary satisfaction signal can be incorrect occasionally (we explore no error, 5% error, and 10% error) on Qwen2.5-1.5B. From the results, we demonstrate that under generic process critique feedback and noisy satisfaction signals, SPEAR still obtains good performance, indicating SPEAR's robustness to such issues. We see even under this dual challenging scenario, SPEAR obtains better performance than non-error and original partial feedback-based Feedback SFT on all settings besides 10% error on ARC-Challenge in Table 1 of the manuscript (even in that scenario, the difference is small: 73.54 vs. 73.89). When placing Feedback SFT under the same constraints, it is always worse than SPEAR, as seen in Table A.
Table A: Generic Process Critique and Error Signal-based SPEAR for Qwen2.5-1.5B
(Q1) We thank the reviewer for the question. We note the first two/three words uniquely identify the correct option in 75.07% of ARC-Challenge examples and 92.39% of HellaSwag examples, using case-insensitive prefix matching. These statistics concern potential hints across training examples, rather than only interactions that received feedback. Although none of the HellaSwag prefixes reproduce the entire answer text, they frequently identify the correct option. Despite the statistics found above, as discussed in our response to (W1), Appendix B.12 from the original submitted manuscript demonstrated meaningful learning with generic process critique feedback. For example, Qwen2.5-1.5B achieved 76.79% on ARC-Challenge and 82.15% on HellaSwag, with Llama3.2-3B being 76.19% and 87.57%, which is importantly better than all baselines in Table 1 of the manuscript. This means that even when specific information pertaining to an individual sample is not given in the feedback, SPEAR is still able to learn the task well. This supports the claim that SPEAR can operate under very challenging non-privileged context scenarios, while still receiving satisfaction-based supervision.
Lastly, in light of these observations and the reviewer's comment on defining "no privileged context" and "non-answer feedback", we unify their definition to "non-privileged context" and define it as feedback that does not directly reveal the answer within the feedback (e.g., "The answer is A."). This has been updated in the manuscript in "Challenges" section of the Introduction (colored in blue).
We appreciate your time and effort in reviewing our paper. We would be happy to address any remaining questions or concerns the reviewer might have.