- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies server-side expert retrieval for personalized federated prompt learning. Each client maintains a local prompt, while the server selects a limited number of nonlocal prompts for download and mixture through a gating mechanism. The authors argue that symmetric prompt similarity cannot represent directed transfer utility, and formulate retrieval as a communication-constrained online transfer problem. pFedAER combines signed loss-improvement feedback, attention-weighted credit assignment, directed utility memory, exploration and freshness bonuses, an uncertainty-dependent retrieval budget, and a guard against persistent negative transfer. The paper evaluates the method on nine datasets with two CLIP backbones. Besides, in a controlled study, pFedAER reduces mean regret and the rate of harmful selections relative to Euclidean KNN, while using 74.2 percent of the budget.
Critical strengths
-
The paper frames server-side expert retrieval as a directed, nonstationary, communication-constrained online transfer problem. This makes the choice of which prompts to communicate an explicit part of personalization rather than a fixed nearest-neighbor preprocessing step. The problem formulation is valuable.
-
In the controlled study, pFedAER reduces regret and harmful selections while using less than the full budget. This provides evidence that directed utility learning with exploration can identify better experts in a controlled setting.
Critical weaknesses
-
The adaptive communication budget is largely inactive in the main experiments. Figure 3(c) shows that 9,067 of 9,075 selections deliver nine experts. The ablation variants in Table 13 also have an average K of approximately 8.99. Under the default configuration, Equations 9 through 11 therefore have little effect on the number of downloaded experts, and the main results do not demonstrate the claimed adaptive communication behavior. The communication savings in Table 5 come from manually lowering K_max, rather than from the adaptive mechanism automatically identifying redundant experts.
-
The reward in Equation 4 may not identify whether individual experts help or harm performance. Within a client minibatch, the assigned reward for each selected expert is the overall loss improvement multiplied by a nonnegative attention weight. Therefore, the rewards share the sign of the overall loss improvement. This raises concerns about whether the per-expert utility estimates and the negative-transfer guard can reliably identify harmful experts.
Questions for authors
-
Under the default configuration, the adaptive K is almost always saturated at nine. Could the authors calibrate or normalize the uncertainty scale, for example by adjusting tau_u or normalizing u_ij, and report whether Equations 9 through 11 can automatically reduce communication while maintaining accuracy and P10? Please also compare against pFedMoAP and pFedMMA at matched communication budgets, or clarify the differences in downloaded bytes, since this affects the assessment of the communication contribution.
-
Could the authors clarify the component-level effects in Table 13? In particular, please explain why removing some components yields comparable or better results, and report the uncertainty of these differences.
7 comments
Response to Weakness 2
Response. Thank you for raising this concern about individual credit and the guard’s interpretation. The shared-sign limitation is real. Within a minibatch, nonnegative attention weights and sign-preserving clipping make each nonzero reward inherit the sign of the mixture’s loss improvement over the local expert. Attention changes the magnitude of an expert’s assigned credit, not its sign. A useful expert can therefore receive negative feedback when the selected mixture performs poorly, even if removing that expert would further increase the loss.
We quantified this issue in three-seed audits on DTD and DomainNet. On the first eligible minibatch of each client-round, we measured the change in cross-entropy loss after removing each selected expert, holding model parameters fixed in evaluation mode and recomputing the gate. A positive loss change means that retaining the expert helps within that selected set. The assigned reward and this removal-based measure have opposite signs in of 2,430 DTD observations and of 3,903 DomainNet observations; mean within-seed Pearson correlations are and , respectively (Table 3.3). These diagnostic measurements were not used to update prompts or retrieval memory. They show that the reward does not reliably reproduce individual removal effects, which themselves remain conditional on the selected set.
Table 3.3. Conditional credit audit: attention reward versus expert-removal gain on the first eligible minibatch of each client-round, with fixed parameters and the gate recomputed after removal. All audited pairs have nonzero rewards and removal gains. These are dependent observations within three training seeds, not independent experimental replicates. The removal gain is conditional on the selected set; it is not a causal or Shapley attribution.
| Dataset | Seed | Batches | Expert pairs | Opposite signs | Pearson |
|---|---|---|---|---|---|
| DTD / RN50 | 1 | 90 | 810 | 44.44% | +0.009 |
| 2 | 90 | 810 | 47.53% | -0.069 | |
| 3 | 90 | 810 | 49.88% | +0.002 | |
| DomainNet / ViT-B/16 | 1 | 145 | 1301 | 49.04% | +0.041 |
| 2 | 145 | 1301 | 49.42% | +0.025 | |
| 3 | 145 | 1301 | 49.19% | +0.001 |
The guard acts on persistent negative observed feedback, so it inherits this attribution limitation. Its persistence and running-mean conditions are intended to avoid exclusion after an isolated negative observation; they do not establish that an expert has negative marginal utility. The DTD ablation also distinguishes filtering from actual exclusion at delivery (Table 3.4): all 166 guard activations among 270 training retrievals exhaust the candidate pool and invoke fallback. Fallback restores the pool, so both guard-on and guard-off deliver all nine remote experts. The observed -point accuracy difference therefore does not establish a predictive benefit from excluding harmful experts.
Table 3.4. DTD/RN50 guard ablation over three seeds. Predictive values are percentages; event counts pool training retrievals.
| Guard | Accuracy | P10 | Mean | Filter events | Fallback events |
|---|---|---|---|---|---|
| On | 9.00 | 166/270 | 166/270 | ||
| Off | 9.00 | 0/270 | 0/270 |
Consistent with the paper’s credit-assignment discussion and limitations, we interpret the reward as inexpensive, policy-dependent feedback rather than a causal attribution. The guard responds to repeated negative feedback under that estimator; neither the reward nor the guard guarantees reliable identification or prevention of individual harmful transfer.
Response to Question 1 (Part 1)
Response. Thank you for proposing uncertainty calibration and comparisons of communication costs. Adjusting the uncertainty threshold reduces downloads at a fixed hard cap, but the results show a trade-off rather than unchanged accuracy and P10. With , increasing from 0.5 to 4 reduces training download volume by on CIFAR10 and on DomainNet (Table 3.2). Relative to the new default runs, mean accuracy decreases by and percentage points, respectively, while P10 decreases by and points. The paired 95% intervals for these accuracy changes are for CIFAR10 and for DomainNet; the corresponding intervals for P10 changes are and . These three-seed measurements demonstrate substantial communication reduction with small observed mean-accuracy changes, but do not establish equivalent predictive performance.
To compare methods at equal download volume, we replayed each adaptive run’s expert counts for every client and selection in both training and evaluation. We used two controls: nearest-prompt retrieval within pFedAER’s client-state procedure, and pFedMoAP with its own procedure. All 36 controls match their adaptive counterparts’ client schedules and training download bytes. At , pFedAER exceeds the pFedMoAP control on DomainNet by accuracy points and P10 points, but falls below it on CIFAR10 by and points, respectively (Table 3.5). Thus, the DomainNet advantage persists at matched communication. The nearest-retrieval controls that retain pFedAER’s client-state procedure show no consistent advantage for adaptive ranking, so this complete-method gain should not be attributed solely to the retrieval score.
Table 3.5. Nearest-retrieval controls replaying each adaptive run’s exact per-client/per-round counts in training and evaluation. Accuracy and P10 are percentages. is adaptive pFedAER accuracy minus the control. Every pair has the same active-client sequence, matching selection keys and counts, and equal training payload bytes. The nearest/pFedAER-state control isolates retrieval within the pFedAER client-state procedure; pFedMoAP retains its own procedure. Download payloads equal the corresponding rows in Table 3.2.
| Control | Accuracy | P10 | (points) | |
|---|---|---|---|---|
| CIFAR10 / RN50 | ||||
| 0.5 | Nearest, pFedAER state | |||
| 0.5 | pFedMoAP | |||
| 2 | Nearest, pFedAER state | |||
| 2 | pFedMoAP | |||
| 4 | Nearest, pFedAER state | |||
| 4 | pFedMoAP | |||
| DomainNet / ViT-B/16 | ||||
| 0.5 | Nearest, pFedAER state | |||
| 0.5 | pFedMoAP | |||
| 2 | Nearest, pFedAER state | |||
| 2 | pFedMoAP | |||
| 4 | Nearest, pFedAER state | |||
| 4 | pFedMoAP |
Response to Weakness 1
Response. Thank you for distinguishing hard-cap savings from automatic budget adaptation. We agree that the main experiments do not demonstrate substantial adaptive reduction in expert counts at the default setting. With τu=0.5, the uncertainty ratio uˉi/τu exceeds one in all 9,075 training selections, with a minimum of 1.3831 (Table 3.1). Equation (9) therefore always requests the maximum available budget, explaining why nine experts are delivered in 99.91% of selections. The default is consequently an approximately fixed-K=9 operating point. Savings obtained by manually lowering Kmax demonstrate the effect of a tighter communication constraint, not automatic contraction under the default configuration.
Table 3.1. Original main-campaign uncertainty ratio uˉi/τu over training decisions. Every ratio is at least one; this table describes the submitted operating point.
To test adaptation separately from the hard cap, we kept Kmax=9 and varied the uncertainty normalization threshold τu (Table 3.2). Increasing τu from 0.5 to 4 reduces mean delivered K from 9.00 to 4.72 on CIFAR10 and from 8.97 to 4.14 on DomainNet, reducing training download volume by 47.6% and 53.9%, respectively. Relative to the new default runs, mean accuracy decreases by 0.118 and 0.212 percentage points, while P10 decreases by 0.482 and 0.733 points. These results show that the rule can use substantially less than the maximum budget, with small observed changes in mean accuracy but larger changes in lower-tail performance. They do not establish equivalent predictive performance.
Table 3.2. Threshold calibration at fixed Kmax=9. Accuracy and P10 are percentages; reductions are relative to the new τu=0.5 runs. Mean K uses actual training retrievals and excludes local-only initialization. MiB/round averages aggregate training downlink payloads over all scheduled rounds, including initialization; it excludes evaluation, uplink, and protocol overhead.
The distinction is that Kmax limits how many experts may be downloaded, whereas τu determines how the current uncertainty translates into a requested count. We varied τu experimentally; within each setting, the rule determines the count for each client and round. The evidence therefore supports the paper’s claim that adaptive communication depends on the uncertainty scale and should be assessed through both prediction and download cost. It does not support attributing savings at the submitted default to automatic identification of redundant experts.