acceptodds
Under review as a conference paper at ICLR 2027

pFedAER: Adaptive Expert Retrieval for Personalized Federated Prompt Learning

Abstract

Personalized federated prompt learning (pFedPL) exploits clients' lightweight prompts as experts to exchange knowledge for performance improvement while keeping the local vision-language model (VLM) frozen. In general, the effectiveness of pFedPL hinges on how well the server routes which remote prompts are worth communicating to each client. Existing approaches rely on fixed-size retrieval of nearest-neighbor prompts based on a symmetric and time-invariant metric of prompt proximity. However, performance gain in reality is directed, evolves over the course of training, and could be negative. To bridge this gap, we formulate prompt retrieval as a problem of communication-constrained online transfer and propose an adaptive expert retrieval method (pFedAER) for personalized prompt mixture. Specifically, on the client side, each participant measures the signed loss improvement of the gated mixture over its local expert and distributes this credit across selected experts via class-conditional gate attention. On the server side, this feedback is used to estimate directed client-expert utility, to balance exploitation against uncertainty and freshness, and to adapt the number of downloaded experts under a hard communication budget. Furthermore, a persistent negative-transfer guard filters client-expert pairs that are repeatedly harmful. Evaluated on nine datasets and two backbones, pFedAER outperforms competing baselines in several settings, especially under combined feature and label shift. In addition, in a controlled study with observable transfer utilities, pFedAER reduces mean regret from to relative to Euclidean retrieval, lowers the rate of harmful selections from to , and consumes only of the communication budget. Our implementation code will be available publicly.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Personalized federated prompt learning, Adaptive expert retrieval, Directed utility feedback, Harmful expert quarantine

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aKBNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies server-side expert retrieval for personalized federated prompt learning. Each client maintains a local prompt, while the server selects a limited number of nonlocal prompts for download and mixture through a gating mechanism. The authors argue that symmetric prompt similarity cannot represent directed transfer utility, and formulate retrieval as a communication-constrained online transfer problem. pFedAER combines signed loss-improvement feedback, attention-weighted credit assignment, directed utility memory, exploration and freshness bonuses, an uncertainty-dependent retrieval budget, and a guard against persistent negative transfer. The paper evaluates the method on nine datasets with two CLIP backbones. Besides, in a controlled study, pFedAER reduces mean regret and the rate of harmful selections relative to Euclidean KNN, while using 74.2 percent of the budget.

Critical strengths
  1. The paper frames server-side expert retrieval as a directed, nonstationary, communication-constrained online transfer problem. This makes the choice of which prompts to communicate an explicit part of personalization rather than a fixed nearest-neighbor preprocessing step. The problem formulation is valuable.

  2. In the controlled study, pFedAER reduces regret and harmful selections while using less than the full budget. This provides evidence that directed utility learning with exploration can identify better experts in a controlled setting.

Critical weaknesses
  1. The adaptive communication budget is largely inactive in the main experiments. Figure 3(c) shows that 9,067 of 9,075 selections deliver nine experts. The ablation variants in Table 13 also have an average K of approximately 8.99. Under the default configuration, Equations 9 through 11 therefore have little effect on the number of downloaded experts, and the main results do not demonstrate the claimed adaptive communication behavior. The communication savings in Table 5 come from manually lowering K_max, rather than from the adaptive mechanism automatically identifying redundant experts.

  2. The reward in Equation 4 may not identify whether individual experts help or harm performance. Within a client minibatch, the assigned reward for each selected expert is the overall loss improvement multiplied by a nonnegative attention weight. Therefore, the rewards share the sign of the overall loss improvement. This raises concerns about whether the per-expert utility estimates and the negative-transfer guard can reliably identify harmful experts.

Questions for authors
  1. Under the default configuration, the adaptive K is almost always saturated at nine. Could the authors calibrate or normalize the uncertainty scale, for example by adjusting tau_u or normalizing u_ij, and report whether Equations 9 through 11 can automatically reduce communication while maintaining accuracy and P10? Please also compare against pFedMoAP and pFedMMA at matched communication budgets, or clarify the differences in downloaded bytes, since this affects the assessment of the communication contribution.

  2. Could the authors clarify the component-level effects in Table 13? In particular, please explain why removing some components yields comparable or better results, and report the uncertainty of these differences.

7 comments
AuthorsOpenReview ↗

Response to Weakness 1

Response. Thank you for distinguishing hard-cap savings from automatic budget adaptation. We agree that the main experiments do not demonstrate substantial adaptive reduction in expert counts at the default setting. With τu=0.5\tau_u=0.5, the uncertainty ratio uˉi/τu\bar u_i/\tau_u exceeds one in all 9,075 training selections, with a minimum of 1.3831 (Table 3.1). Equation (9) therefore always requests the maximum available budget, explaining why nine experts are delivered in 99.91%99.91\% of selections. The default is consequently an approximately fixed-K=9K=9 operating point. Savings obtained by manually lowering Kmax⁡K_{\max} demonstrate the effect of a tighter communication constraint, not automatic contraction under the default configuration.

Table 3.1. Original main-campaign uncertainty ratio uˉi/τu\bar u_i/\tau_u over training decisions. Every ratio is at least one; this table describes the submitted operating point.

DatasetDecisionsMinimumP10MedianP90
Caltech1012701.38311.38311.66512.1899
CIFAR1033001.66513.03813.81454.2042
CIFAR10033001.66512.81713.81454.2042
DomainNet4351.66511.66512.82143.4616
DTD2701.38311.38311.66512.1899
Food1012701.38311.38311.66512.1899
Office-Caltech106901.46111.66512.07672.7334
Flowers1022701.38311.38311.66512.1899
OxfordPets2701.38311.38311.66512.1899

To test adaptation separately from the hard cap, we kept Kmax⁡=9K_{\max}=9 and varied the uncertainty normalization threshold τu\tau_u (Table 3.2). Increasing τu\tau_u from 0.5 to 4 reduces mean delivered KK from 9.00 to 4.72 on CIFAR10 and from 8.97 to 4.14 on DomainNet, reducing training download volume by 47.6%47.6\% and 53.9%53.9\%, respectively. Relative to the new default runs, mean accuracy decreases by 0.1180.118 and 0.2120.212 percentage points, while P10 decreases by 0.4820.482 and 0.7330.733 points. These results show that the rule can use substantially less than the maximum budget, with small observed changes in mean accuracy but larger changes in lower-tail performance. They do not establish equivalent predictive performance.

Table 3.2. Threshold calibration at fixed Kmax⁡=9K_{\max}=9. Accuracy and P10 are percentages; reductions are relative to the new τu=0.5\tau_u=0.5 runs. Mean KK uses actual training retrievals and excludes local-only initialization. MiB/round averages aggregate training downlink payloads over all scheduled rounds, including initialization; it excludes evaluation, uplink, and protocol overhead.

τu\tau_uAccuracyP10Mean KKMiB/roundReduction
CIFAR10 / RN50
0.580.93±0.4880.93\pm0.4872.79±0.3672.79\pm0.369.001.289±0.0001.289\pm0.0000.0%
280.90±0.4780.90\pm0.4772.27±0.9772.27\pm0.978.271.185±0.0011.185\pm0.0018.1%
480.82±0.6080.82\pm0.6072.31±0.3172.31\pm0.314.720.675±0.0010.675\pm0.00147.6%
DomainNet / ViT-B/16
0.538.10±0.3738.10\pm0.3725.56±0.6225.56\pm0.628.970.813±0.0000.813\pm0.0000.0%
237.77±0.0237.77\pm0.0224.56±0.2524.56\pm0.256.930.628±0.0020.628\pm0.00222.8%
437.89±0.8637.89\pm0.8624.82±1.2224.82\pm1.224.140.375±0.0010.375\pm0.00153.9%

The distinction is that Kmax⁡K_{\max} limits how many experts may be downloaded, whereas τu\tau_u determines how the current uncertainty translates into a requested count. We varied τu\tau_u experimentally; within each setting, the rule determines the count for each client and round. The evidence therefore supports the paper’s claim that adaptive communication depends on the uncertainty scale and should be assessed through both prediction and download cost. It does not support attributing savings at the submitted default to automatic identification of redundant experts.

AuthorsOpenReview ↗

Response to Weakness 2

Response. Thank you for raising this concern about individual credit and the guard’s interpretation. The shared-sign limitation is real. Within a minibatch, nonnegative attention weights and sign-preserving clipping make each nonzero reward inherit the sign of the mixture’s loss improvement over the local expert. Attention changes the magnitude of an expert’s assigned credit, not its sign. A useful expert can therefore receive negative feedback when the selected mixture performs poorly, even if removing that expert would further increase the loss.

We quantified this issue in three-seed audits on DTD and DomainNet. On the first eligible minibatch of each client-round, we measured the change in cross-entropy loss after removing each selected expert, holding model parameters fixed in evaluation mode and recomputing the gate. A positive loss change means that retaining the expert helps within that selected set. The assigned reward and this removal-based measure have opposite signs in 47.28%47.28\% of 2,430 DTD observations and 49.22%49.22\% of 3,903 DomainNet observations; mean within-seed Pearson correlations are −0.019-0.019 and 0.0220.022, respectively (Table 3.3). These diagnostic measurements were not used to update prompts or retrieval memory. They show that the reward does not reliably reproduce individual removal effects, which themselves remain conditional on the selected set.

Table 3.3. Conditional credit audit: attention reward versus expert-removal gain on the first eligible minibatch of each client-round, with fixed parameters and the gate recomputed after removal. All audited pairs have nonzero rewards and removal gains. These are dependent observations within three training seeds, not independent experimental replicates. The removal gain is conditional on the selected set; it is not a causal or Shapley attribution.

DatasetSeedBatchesExpert pairsOpposite signsPearson rr
DTD / RN5019081044.44%+0.009
29081047.53%-0.069
39081049.88%+0.002
DomainNet / ViT-B/161145130149.04%+0.041
2145130149.42%+0.025
3145130149.19%+0.001

The guard acts on persistent negative observed feedback, so it inherits this attribution limitation. Its persistence and running-mean conditions are intended to avoid exclusion after an isolated negative observation; they do not establish that an expert has negative marginal utility. The DTD ablation also distinguishes filtering from actual exclusion at delivery (Table 3.4): all 166 guard activations among 270 training retrievals exhaust the candidate pool and invoke fallback. Fallback restores the pool, so both guard-on and guard-off deliver all nine remote experts. The observed 0.0560.056-point accuracy difference therefore does not establish a predictive benefit from excluding harmful experts.

Table 3.4. DTD/RN50 guard ablation over three seeds. Predictive values are percentages; event counts pool training retrievals.

GuardAccuracyP10Mean KKFilter eventsFallback events
On91.398±0.17891.398\pm0.17884.826±0.91184.826\pm0.9119.00166/270166/270
Off91.343±0.17091.343\pm0.17084.652±0.77184.652\pm0.7719.000/2700/270

Consistent with the paper’s credit-assignment discussion and limitations, we interpret the reward as inexpensive, policy-dependent feedback rather than a causal attribution. The guard responds to repeated negative feedback under that estimator; neither the reward nor the guard guarantees reliable identification or prevention of individual harmful transfer.

AuthorsOpenReview ↗

Response to Question 1 (Part 1)

Response. Thank you for proposing uncertainty calibration and comparisons of communication costs. Adjusting the uncertainty threshold reduces downloads at a fixed hard cap, but the results show a trade-off rather than unchanged accuracy and P10. With Kmax⁡=9K_{\max}=9, increasing τu\tau_u from 0.5 to 4 reduces training download volume by 47.6%47.6\% on CIFAR10 and 53.9%53.9\% on DomainNet (Table 3.2). Relative to the new default runs, mean accuracy decreases by 0.1180.118 and 0.2120.212 percentage points, respectively, while P10 decreases by 0.4820.482 and 0.7330.733 points. The paired 95% intervals for these accuracy changes are [−0.574,0.339][-0.574,0.339] for CIFAR10 and [−2.403,1.979][-2.403,1.979] for DomainNet; the corresponding intervals for P10 changes are [−2.113,1.149][-2.113,1.149] and [−3.476,2.009][-3.476,2.009]. These three-seed measurements demonstrate substantial communication reduction with small observed mean-accuracy changes, but do not establish equivalent predictive performance.

To compare methods at equal download volume, we replayed each adaptive run’s expert counts for every client and selection in both training and evaluation. We used two controls: nearest-prompt retrieval within pFedAER’s client-state procedure, and pFedMoAP with its own procedure. All 36 controls match their adaptive counterparts’ client schedules and training download bytes. At τu=4\tau_u=4, pFedAER exceeds the pFedMoAP control on DomainNet by 1.9881.988 accuracy points and 2.3112.311 P10 points, but falls below it on CIFAR10 by 1.4011.401 and 1.9981.998 points, respectively (Table 3.5). Thus, the DomainNet advantage persists at matched communication. The nearest-retrieval controls that retain pFedAER’s client-state procedure show no consistent advantage for adaptive ranking, so this complete-method gain should not be attributed solely to the retrieval score.

Table 3.5. Nearest-retrieval controls replaying each adaptive run’s exact per-client/per-round counts in training and evaluation. Accuracy and P10 are percentages. Δ\Delta is adaptive pFedAER accuracy minus the control. Every pair has the same active-client sequence, matching selection keys and counts, and equal training payload bytes. The nearest/pFedAER-state control isolates retrieval within the pFedAER client-state procedure; pFedMoAP retains its own procedure. Download payloads equal the corresponding rows in Table 3.2.

τu\tau_uControlAccuracyP10Δ\Delta (points)
CIFAR10 / RN50
0.5Nearest, pFedAER state80.94±0.5280.94\pm0.5273.18±0.4273.18\pm0.42−0.002-0.002
0.5pFedMoAP82.33±0.5682.33\pm0.5674.20±0.4474.20\pm0.44−1.400-1.400
2Nearest, pFedAER state80.98±0.4680.98\pm0.4673.05±0.2473.05\pm0.24−0.071-0.071
2pFedMoAP82.28±0.5782.28\pm0.5774.06±1.0074.06\pm1.00−1.375-1.375
4Nearest, pFedAER state81.05±0.5481.05\pm0.5472.99±0.1872.99\pm0.18−0.236-0.236
4pFedMoAP82.22±0.6182.22\pm0.6174.30±0.5374.30\pm0.53−1.401-1.401
DomainNet / ViT-B/16
0.5Nearest, pFedAER state37.73±0.8337.73\pm0.8324.35±0.5824.35\pm0.58+0.366+0.366
0.5pFedMoAP35.95±0.7735.95\pm0.7721.63±2.9221.63\pm2.92+2.148+2.148
2Nearest, pFedAER state37.92±0.5337.92\pm0.5326.04±0.2826.04\pm0.28−0.154-0.154
2pFedMoAP35.95±0.9935.95\pm0.9921.65±1.3621.65\pm1.36+1.819+1.819
4Nearest, pFedAER state38.08±0.6938.08\pm0.6926.21±0.5626.21\pm0.56−0.189-0.189
4pFedMoAP35.90±1.0035.90\pm1.0022.51±0.6322.51\pm0.63+1.988+1.988
Reviewer_6YuuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The propose presents an adaptive expert retrieval method (pFedAER), which aims to address the issue in federated personalised prompt learning where the symmetric distance is used to measure transfer value. The method formulates server-side expert retrieval as a communication-constrained, directed, non-stationary online migration problem. It employs signed client-weighted feedback to learn ordered utilities, adaptively adjust the number of experts, and suppress persistent harmful migration.This results in improved performance in dual-heterogeneous scenarios.

Critical strengths
  1. The propose highlights the current issue in federated personalised prompt learning, whereby the value of transfer is measured using symmetric distance, and provides a mathematical proof for directed transfer.
  2. The paper proposes an adaptive expert retrieval method that utilises directed scalars, based on client feedback, to maintain and update the pool of retrieval candidate prompts. This results in improved performance in dual-heterogeneous scenarios.
Critical weaknesses
  1. While some improvements were observed in the double-heterogeneity experiments, the results from experiments conducted solely under the disjoint-class label shift condition and solely under the Dirichlet label shift condition performed poorly and did not demonstrate the general effectiveness of this method in prompting expert learning.
  2. In the important mechanism described in Eq. (4), when the local client computes the signed loss improvement git(B)g_i^t(B), this may result in good experts being assigned in the opposite direction.
Questions for authors
  1. In Table 1, the standard deviation for pFedAER on the Flowers102 dataset using the ViT-B/16 backbone network is 0.00. Does this imply that numerical precision truncation occurred in the experiments conducted with multiple random seeds?
  2. Using rijr_{ij}, it is possible to calculate the retrieval score for expert j with regard to client i. With regard to Eq. 8, does the weight (wu,wc,β,wf,wvw_{u} , w_{c} ,{\beta}, w_{f} , w_{v}) of each element in the retrieval score sijts^t_{ij} sensitive to the retrieval results across different scenarios?
8 comments
AuthorsOpenReview ↗

Response to Weakness 1

Response. Thank you for raising the question of effectiveness across heterogeneity settings. The results differ between the two label-shift settings. Under disjoint-class label shift, pFedAER improves over pFedMoAP on four of five datasets with both RN50 and ViT-B/16. The largest gains occur on DTD, at 0.700.70 and 0.550.55 percentage points, respectively; on Food101, pFedAER exceeds both pFedMoAP and pFedMMA under both backbones. These results show benefits beyond the combined-shift setting, although the gains are modest and OxfordPets remains below pFedMoAP. We therefore do not interpret the disjoint-class results as uniform superiority over all baselines.

Under Dirichlet label shift on CIFAR10/100, pFedAER is less accurate than both pFedMoAP and pFedMMA with either backbone. This is a limitation of the evaluated method. In contrast, pFedAER achieves the highest reported mean accuracy on DomainNet and Office-Caltech10 under both backbones, exceeding the strongest comparator in each setting by 0.530.53–3.573.57 percentage points. The paper’s emphasis on combined feature and label shift reflects this observed pattern, rather than a claim that directed retrieval improves every personalization task.

The additional experiments test whether this contrast persists at matched communication. Table 2.1 reports pFedAER’s operating points, and Table 2.2 reports controls using the same expert counts for each client and selection, with identical training download bytes. At τu=4\tau_u=4, pFedAER exceeds the pFedMoAP control on DomainNet by 1.9881.988 accuracy points and 2.3112.311 P10 points, but falls below it on CIFAR10 by 1.4011.401 and 1.9981.998 points, respectively. Thus, the DomainNet advantage cannot be explained by downloading more experts. Together, the submitted and matched-download results support task-dependent benefits, with the clearest evidence under combined shift. Because the datasets and client populations also differ, these comparisons do not isolate feature shift as the cause of the gains. The gains over pFedMoAP concern the complete method and should not be attributed solely to its retrieval score.

Table 2.1. Threshold calibration at fixed Kmax⁡=9K_{\max}=9. Accuracy and P10 are percentages; reductions are relative to the new τu=0.5\tau_u=0.5 runs. Mean KK uses actual training retrievals and excludes local-only initialization. MiB/round averages aggregate training downlink payloads over all scheduled rounds, including initialization; it excludes evaluation, uplink, and protocol overhead.

τu\tau_uAccuracyP10Mean KKMiB/roundReduction
CIFAR10 / RN50
0.580.93±0.4880.93\pm0.4872.79±0.3672.79\pm0.369.001.289±0.0001.289\pm0.0000.0%
280.90±0.4780.90\pm0.4772.27±0.9772.27\pm0.978.271.185±0.0011.185\pm0.0018.1%
480.82±0.6080.82\pm0.6072.31±0.3172.31\pm0.314.720.675±0.0010.675\pm0.00147.6%
DomainNet / ViT-B/16
0.538.10±0.3738.10\pm0.3725.56±0.6225.56\pm0.628.970.813±0.0000.813\pm0.0000.0%
237.77±0.0237.77\pm0.0224.56±0.2524.56\pm0.256.930.628±0.0020.628\pm0.00222.8%
437.89±0.8637.89\pm0.8624.82±1.2224.82\pm1.224.140.375±0.0010.375\pm0.00153.9%

Table 2.2. Nearest-retrieval controls replaying each adaptive run’s exact per-client/per-round counts in training and evaluation. Accuracy and P10 are percentages. Δ\Delta is adaptive pFedAER accuracy minus the control. Every pair has the same active-client sequence, matching selection keys and counts, and equal training payload bytes. The nearest/pFedAER-state control isolates retrieval within the pFedAER client-state procedure; pFedMoAP retains its own procedure. Download payloads equal the corresponding rows in Table 2.1.

τu\tau_uControlAccuracyP10Δ\Delta (points)
CIFAR10 / RN50
0.5Nearest, pFedAER state80.94±0.5280.94\pm0.5273.18±0.4273.18\pm0.42−0.002-0.002
0.5pFedMoAP82.33±0.5682.33\pm0.5674.20±0.4474.20\pm0.44−1.400-1.400
2Nearest, pFedAER state80.98±0.4680.98\pm0.4673.05±0.2473.05\pm0.24−0.071-0.071
2pFedMoAP82.28±0.5782.28\pm0.5774.06±1.0074.06\pm1.00−1.375-1.375
4Nearest, pFedAER state81.05±0.5481.05\pm0.5472.99±0.1872.99\pm0.18−0.236-0.236
4pFedMoAP82.22±0.6182.22\pm0.6174.30±0.5374.30\pm0.53−1.401-1.401
DomainNet / ViT-B/16
0.5Nearest, pFedAER state37.73±0.8337.73\pm0.8324.35±0.5824.35\pm0.58+0.366+0.366
0.5pFedMoAP35.95±0.7735.95\pm0.7721.63±2.9221.63\pm2.92+2.148+2.148
2Nearest, pFedAER state37.92±0.5337.92\pm0.5326.04±0.2826.04\pm0.28−0.154-0.154
2pFedMoAP35.95±0.9935.95\pm0.9921.65±1.3621.65\pm1.36+1.819+1.819
4Nearest, pFedAER state38.08±0.6938.08\pm0.6926.21±0.5626.21\pm0.56−0.189-0.189
4pFedMoAP35.90±1.0035.90\pm1.0022.51±0.6322.51\pm0.63+1.988+1.988
AuthorsOpenReview ↗

Response to Weakness 2

Response. Thank you for pointing out this limitation of the credit-assignment rule. Yes, a useful expert can receive negative credit. Within a minibatch, the normalized attention weights are nonnegative and clipping does not reverse the sign. Each nonzero reward in Eq. (4) therefore shares the sign of the mixture’s loss improvement over the local expert. If the mixture performs worse than the local expert, a selected expert receives negative credit even if removing it would further increase the mixture’s loss. Attention determines how much credit an expert receives, but cannot distinguish beneficial from harmful contributions within that mixture.

This is why the paper describes the reward as a policy-dependent credit estimator, rather than an expert’s causal marginal contribution. The design associates observed mixture performance with the participating experts using their gate attention, without the additional forward passes required by expert-removal evaluations. It provides inexpensive, recipient-specific feedback for repeated retrieval, but does not guarantee the correct sign of each expert’s individual contribution.

We assessed this limitation in six additional audit runs, comprising three seeds each on DTD and DomainNet. On the first eligible minibatch of each client-round, we froze the model parameters, used evaluation mode, and measured dij=LB(Si∖{j})−LB(Si),d_{ij}=\mathcal L_B(S_i\setminus\{j\})-\mathcal L_B(S_i), where LB\mathcal L_B is the cross-entropy loss on the same minibatch and the gate is recomputed after removing expert jj. A positive dijd_{ij} means that removal increases loss, so retaining that expert helps within the current selected set. The assigned reward and dijd_{ij} have opposite signs in 47.28%47.28\% of 2,430 DTD observations and 49.22%49.22\% of 3,903 DomainNet observations; their mean within-seed Pearson correlations are −0.019-0.019 and 0.0220.022, respectively (Table 2.3). These removal measurements are conditional on the selected set and were used only for diagnosis, not for training or retrieval-memory updates.

Table 2.3. Conditional credit audit: attention reward versus expert-removal gain on the first eligible minibatch of each client-round, with fixed parameters and the gate recomputed after removal. All audited pairs have nonzero rewards and removal gains. These are dependent observations within three training seeds, not independent experimental replicates. The removal gain is conditional on the selected set; it is not a causal or Shapley attribution.

DatasetSeedBatchesExpert pairsOpposite signsPearson rr
DTD / RN5019081044.44%+0.009
29081047.53%-0.069
39081049.88%+0.002
DomainNet / ViT-B/161145130149.04%+0.041
2145130149.42%+0.025
3145130149.19%+0.001

This substantial sign disagreement confirms the limitation raised by the reviewer. Accordingly, the reward should be interpreted as attention-weighted mixture feedback, not as a reliable test of whether an individual expert helps or harms. The negative-transfer guard acts on persistent negative feedback under this estimator; it does not guarantee exclusion of experts with negative marginal contributions.

AuthorsOpenReview ↗

Response to Question 1

Response. Thank you for noticing the ambiguity in the reported precision. The reported 0.000.00 results from rounding the standard deviation to two decimal places for presentation; it does not mean that the three runs produced identical accuracies. Following the paper’s evaluation protocol, we first average accuracy over the final five rounds of each run, then compute the mean and sample standard deviation across the three seeds. The seed-level accuracies are 99.50428912%99.50428912\%, 99.50875462%99.50875462\%, and 99.51424800%99.51424800\%, giving a mean of 99.50909724%99.50909724\% and a nonzero sample standard deviation of 0.004988280.00498828 percentage points.

Because this standard deviation is below 0.0050.005 percentage points, it rounds to 0.000.00 at the table’s two-decimal precision. Reporting the same results as 99.509±0.005%99.509\pm0.005\% at three-decimal precision makes the observed variation visible without changing the underlying results or evaluation protocol.

Reviewer_ZHLQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies server-side expert retrieval in personalized federated prompt learning and points out that existing symmetric nearest-neighbor retrieval cannot capture directed, nonstationary, and negative transfer. The authors formulate it as a communication-constrained online transfer problem and propose pFedAER. Clients return signed, attention-weighted scalar utility; based on this, the server learns a directed utility memory, adaptively adjusts the number of downloaded experts, and filters persistently harmful client-expert pairs. On nine datasets and two backbones, pFedAER shows the clearest gains under combined feature and label shift. Controlled experiments show lower regret and harmful-selection rate with a lower communication budget.

Critical strengths

In Section 2.2, Proposition 1, and Eqs. (1)–(2), the paper shows that symmetric prompt similarity cannot encode directed transfer, and formulates expert retrieval as a communication-constrained, directed, nonstationary online transfer problem. This elevates “which experts to communicate to whom” to an independent decision layer in personalized federated prompt learning. The conceptual contribution is clear and the motivation is practical, rather than being yet another prompt-mixture variant.

Critical weaknesses

The negative-transfer guard lacks targeted ablation on tasks where it actually becomes active. The adaptive number of experts is almost inactive at the default operating point.

Questions for authors
  1. Evidence for the negative-transfer guard. Table 13 ablates the guard on CIFAR10 and DomainNet, but Table 14 shows that the guard is almost never triggered on these two datasets; the guard is actually active on DTD, and to a lesser extent on Flowers102/OxfordPets/Caltech101. Could you provide guard on/off results at least on DTD? If this is not feasible, please explain why these two datasets were chosen for the guard ablation and how the current evidence supports the guard’s contribution.

  2. Adaptive budget at the default operating point. Figure 3(c) shows that 9,067 of 9,075 selections deliver 9 experts, and Table 15 reports Mean K≈9.00. Could you provide the distribution of uˉi/τu\bar{u}_i/\tau_u over rounds and clients to explain why the adaptive mechanism almost never contracts? Could you add a comparison at matched average download volume? This would help clarify whether the default configuration should be interpreted as fixed K=9K=9 and how the “adaptive” claim should be read.

  3. Tuning fairness and the fact that the method does not consistently outperform strong baselines. Please clarify whether the default hyperparameters of pFedAER in Table 7 were selected on a validation set, and whether pFedMoAP/pFedMMA received a comparable tuning budget. For pFedMMA, please summarize the differences from the official implementation. In addition, pFedAER is lower than pFedMMA on CIFAR10/100 in Table 2, and lower than pFedMoAP on OxfordPets in Table 1. Please clarify whether this reflects the applicability boundary of the method and how the claim of “improvements on six of nine datasets” in the abstract should be interpreted accurately. This would affect my judgment of the fairness of the main comparisons and the positioning of the method.

10 comments

No author response was provided during the rebuttal period. Therefore, the main concerns raised in my original review remain unresolved, and I maintain my score of 2.

AuthorsOpenReview ↗

Response to Weakness 1 and Experimental Settings in Rebuttal Period

Response. Thank you for identifying these gaps in the experimental evidence. We added the requested DTD guard ablation and fixed-cap threshold study. The guard activates on DTD but is frequently overridden by fallback (Q1). Calibrating the uncertainty threshold activates communication reduction on both CIFAR10 and DomainNet (Q2).

Rebuttal experimental settings.

Unless stated otherwise, new predictive results are mean ±\pm sample standard deviation across seeds 1–3, after averaging each run’s final five rounds. DTD uses RN50, 10 clients and 10 rounds; CIFAR10 uses RN50, 100 clients and 120 rounds; DomainNet uses ViT-B/16, 30 clients and 25 rounds. Participation is respectively 11, 0.10.1, and 0.250.25, with five local epochs throughout. We retain the submitted data partitions and training settings. New experiments use their contemporaneous default runs as references; they do not replace the submitted benchmark values. Paired intervals are descriptive 95% Student-tt intervals over three seeds, without multiplicity correction.

AuthorsOpenReview ↗

Response to Question 1

Response. Thank you for requesting an ablation where the guard is active. We added a guard-on/off comparison on DTD with RN50, using the same training protocol and three seeds. The original CIFAR10 and DomainNet ablations covered label shift and combined feature–label shift, respectively, but their negligible filtering activity limits what they establish about the guard.

Table 1.1 reports the targeted DTD results. With the guard enabled, accuracy is 91.398%91.398\% versus 91.343%91.343\% without it, and P10 is 84.826%84.826\% versus 84.652%84.652\%. The paired accuracy difference is +0.056+0.056 percentage points, with a descriptive 95% interval of [0.028,0.083][0.028,0.083]; the P10 interval includes zero. These are small predictive differences, whose interpretation depends on whether filtering changes the delivered expert set.

Table 1.1. Guard-on/off comparison on DTD using RN50 across three random seeds. Accuracy and P10 are reported as mean ± standard deviation (%). Filtering and fallback events are counted across all training retrievals from the three runs.

GuardAccuracyP10Mean KKFilter eventsFallback events
On91.398±0.17891.398\pm0.17884.826±0.91184.826\pm0.9119.00166/270166/270
Off91.343±0.17091.343\pm0.17084.652±0.77184.652\pm0.7719.000/2700/270

On DTD, the guard activates in 166 of 270 training retrievals, but every activation excludes the entire candidate pool and invokes the fallback described in the paper. Fallback restores the pool and ranks candidates by empirical utility; both variants consequently deliver all nine remote experts at every retrieval. Thus, guard activation does not produce a reduced delivered set in this experiment, and the accuracy difference cannot be taken as evidence that excluding harmful experts improves prediction. These findings match the paper’s operational account: the guard excludes experts with persistently negative feedback, whereas fallback preserves expert availability when all experts are excluded, allowing retrieval to continue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.