- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper argues that the text gradients generated during federated prompting are not universally useful, which provides clear motivation, and then the authors propose a new approach called Scoped FedTextGard, which splits local prompt updates into individual rules so that FedTextGard can estimate their usefulness for each client. Experiments over 6 language models show improvement in average accuracy and prompt efficiency.
Critical strengths
- S1: The problem is intersting and the formulation is clear
- S2: Emprical anaylsis is strong. it's intersting to see that only of rules are classified as globally beneficial.
- S3: Table 7 reports a very good improvments over local TextGard and existing aggregation approaches
- S4 The ablation study of evalauting different prompt budgets and model families is intersting as well
Critical weaknesses
-
W1: As far as i understand the main idea is that each client evaluates proposed textual rules on its own validation data and then retains the more informative ones. That basically sounds like a mechanism that mechansims resembles personalized model selection using local validation performance! So it's not clear to me what advantage capability the explicit scope representation presents over straightforward personalized validation based selection.
-
W2: I would argue also that estimating each client scope requires additional computational and communication cost since it requires evaluating each candidate rule on each target client validation data. with high number of clients the candidate rules pool will explode and the claim of 73% reduction in prompt tokens basically refere to the final prompt length not the total communication and computational cost. I believe the authors should report end-to-end computational costs, GPU run time, communication volume and total inference tokens.
-
W3: As per my understanding, that the experiment setup uses 20 validation examples per client for predicting the text gradient? And this means that one correct example will increase the accuracy of 5%. I think it is very important to report the uncertainty intervals for scope aggregation in federated runs and see if confidence aware routing provides any advantages over the zero threshold.
-
W4: The main motivation of all federated learning approaches is to solve the privacy issue and the prpoesed approach could be strongly leak cleint information data. I would argue that sharing textual gradients could contain sensitive information of local examples and repeated utility measurments could expose properties of client validation distribution as well.
Side note: I would recommend changing the fonts of Figure 2 to be much more readable. The BIG- Bench Hard (BBH) is not citated so please check. Also the citation of some used LLMs are missined ( Qwen3, Llama 3.1 and Phi 3.5)
Questions for authors
check weaknesses
Dear Reviewer,
As we approach the final hours of the author-reviewer discussion period, I would greatly appreciate it if you could take a moment to review and respond to the authors’ rebuttal.
Please briefly indicate which concerns have been addressed and which remain, and consider updating your scores if needed.
Thank you again for your time and valuable contributions!
Best regards, AC