acceptodds
Under review as a conference paper at ICLR 2027

Textual Gradients Have Scope: Rethinking Federated Textual Optimization

Abstract

Federated textual optimization enables clients to improve prompts locally and aggregate natural-language updates without centrally pooling data. Existing approaches, however, typically compress distributed textual updates into a single prompt shared by all clients, implicitly treating useful textual gradients as broadly applicable. We show that this assumption often fails under heterogeneous client distributions. Across 2,070 textual-gradient rules from 108 federated runs over six language models, only 0.4% are empirically classified as globally beneficial under the default validation protocol, while 85.4% exhibit partial client scope. Motivated by this observation, we introduce textual-gradient scope, which captures the client-wise support of rule-level utility, and propose SCOPED-FEDTEXTGRAD, a scope-preserving framework for federated textual optimization. SCOPED-FEDTEXTGRAD atomizes local prompt revisions into independently routable rules, measures their target-client utility, and composes only scope-compatible updates under a hard prompt-token budget. A validation-gated variant, SAFE-SCOPED, further falls back to the local solution when scoped sharing is not supported by validation evidence. SCOPED-FEDTEXTGRAD improves average accuracy over Local TextGrad, while using approximately 73% fewer prompt tokens than unconstrained concatenation. The gains persist across model scales and families, and validation gating reduces negative transfer by more than half. These results suggest that federated textual gradients should be shared according to where they are useful, rather than compressed into a universally shared description. Anonymous code is available at https://anonymous.4open.science/r/Scope-FedTextual.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Federated Learning, Textual Gradients, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gK3mOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper argues that the text gradients generated during federated prompting are not universally useful, which provides clear motivation, and then the authors propose a new approach called Scoped FedTextGard, which splits local prompt updates into individual rules so that FedTextGard can estimate their usefulness for each client. Experiments over 6 language models show improvement in average accuracy and prompt efficiency.

Critical strengths
  • S1: The problem is intersting and the formulation is clear
  • S2: Emprical anaylsis is strong. it's intersting to see that only 0.40.4% of rules are classified as globally beneficial.
  • S3: Table 7 reports a very good improvments over local TextGard and existing aggregation approaches
  • S4 The ablation study of evalauting different prompt budgets and model families is intersting as well
Critical weaknesses
  • W1: As far as i understand the main idea is that each client evaluates proposed textual rules on its own validation data and then retains the more informative ones. That basically sounds like a mechanism that mechansims resembles personalized model selection using local validation performance! So it's not clear to me what advantage capability the explicit scope representation presents over straightforward personalized validation based selection.

  • W2: I would argue also that estimating each client scope requires additional computational and communication cost since it requires evaluating each candidate rule on each target client validation data. with high number of clients the candidate rules pool will explode and the claim of 73% reduction in prompt tokens basically refere to the final prompt length not the total communication and computational cost. I believe the authors should report end-to-end computational costs, GPU run time, communication volume and total inference tokens.

  • W3: As per my understanding, that the experiment setup uses 20 validation examples per client for predicting the text gradient? And this means that one correct example will increase the accuracy of 5%. I think it is very important to report the uncertainty intervals for scope aggregation in federated runs and see if confidence aware routing provides any advantages over the zero threshold.

  • W4: The main motivation of all federated learning approaches is to solve the privacy issue and the prpoesed approach could be strongly leak cleint information data. I would argue that sharing textual gradients could contain sensitive information of local examples and repeated utility measurments could expose properties of client validation distribution as well.

Side note: I would recommend changing the fonts of Figure 2 to be much more readable. The BIG- Bench Hard (BBH) is not citated so please check. Also the citation of some used LLMs are missined ( Qwen3, Llama 3.1 and Phi 3.5)

Questions for authors

check weaknesses

Reviewer_B3HLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates textual optimization through the client-dependent scope of textual gradients in federated textual optimization. The authors propose Scoped-FedTextGrad, which decomposes local prompt updates into reusable rules and selectively shares them based on their utility to individual clients under a token budget. A validation-gated variant further reduces negative transfer. Experiments across different tasks, data distributions, and language models demonstrate improvements in accuracy and prompt efficiency.

Critical strengths
  • The framework is simple and practical, sharing updates selectively under a bounded prompt length.
  • The experimental evaluation covers different data distributions, model families, model scales, and prompt budgets.
Critical weaknesses
  • The practical value for more capable LLMs remains unclear. Stronger LLMs may already achieve competitive performance with simple prompts, reducing the need for additional textual optimization.
  • Interactions among rules are not analyzed. Individually beneficial rules may interfere with each other when combined into a single prompt, potentially reducing overall performance.
  • Limited evaluation scale. The experiments use relatively small client datasets (16–20 training and 20 validation examples per client), raising concerns about scalability and generalizability.
  • The additional validation overhead may limit scalability. Scope estimation adds substantial validation inference, which grows with the number of clients. End-to-end computational, communication, and token costs should be provided.
Questions for authors
  1. How does the method handle interactions between textual-gradient rules?
  2. How should clients select representative validation samples to ensure reliable textual-gradient scope estimation?
  3. Does the benefit of federated textual optimization still hold as LLM capabilities increase? The experiments cover models up to Qwen3-32B, but stronger LLMs may already perform well with simple prompts. Could the authors investigate whether the proposed method still provides meaningful improvements over the original prompt on more capable models?
1 comment

Dear Reviewer,

As we approach the final hours of the author-reviewer discussion period, I would greatly appreciate it if you could take a moment to review and respond to the authors’ rebuttal.

Please briefly indicate which concerns have been addressed and which remain, and consider updating your scores if needed.

Thank you again for your time and valuable contributions!

Best regards, AC

Reviewer_qudVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the concern that, in federated optimization, a universal prompt or textual update may not benefit all clients. It proposes Scoped-FedTextGrad to customize rules and prompts for each client during optimization. The method extracts concise, reusable rules from locally improved prompts, evaluates each rule on each client's validation data, and builds personalized prompts using effective rules within a token budget. Its Safe-Scoped variant falls back to local prompts when validation performance is poor. The main experiments on three BIG-Bench Hard (BBH) tasks show that Scoped-FedTextGrad substantially outperforms both Local TextGrad and FedTextGrad.

Critical strengths
  1. The method is well motivated and clearly structured.
  2. The paper is generally well written and easy to follow.
Critical weaknesses

Client-specific prompt selection is well motivated by differences in clients' data distributions. Breaking updates down into rules also facilitates fine-grained selection and combination. However, the experiments more directly support client-specific utility-based selection and do not separately evaluate the benefits of atomic rule extraction itself. It therefore remains unclear whether atomic rule extraction provides sufficient benefit over selecting raw textual updates to justify the additional complexity.

Questions for authors
  1. How much improvement does extracting atomic rules provide compared with applying client-specific utility-based selection directly to whole local prompts or unprocessed textual updates under comparable token and validation-query budgets?
  2. Given that the evaluation is restricted to BBH tasks, could the authors provide evidence that the method generalizes to broader applications?
  3. Could the authors provide a complete end-to-end runtime comparison with all baselines across the evaluated benchmarks?
  4. Could the extracted rules or shared utility scores reveal information about clients' local data?
1 comment

Dear Reviewer,

As we approach the final hours of the author-reviewer discussion period, I would greatly appreciate it if you could take a moment to review and respond to the authors’ rebuttal.

Please briefly indicate which concerns have been addressed and which remain, and consider updating your scores if needed.

Thank you again for your time and valuable contributions!

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.