acceptodds
Under review as a conference paper at ICLR 2027

Trust Without Aggregation: Robust Expert Pools for Personalized Federated Prompt Learning

Abstract

Personalized federated prompt learning can preserve client heterogeneity by retaining uploaded prompts as separately retrievable experts instead of averaging them into a single model. However, this design worsens robustness: an expert pool delivers each prompt intact to its recipients, so a malicious prompt that remains geometrically close to benign ones can pass selection while encoding targeted behavior. We formulate this setting as robust expert-pool learning and propose pFedREP, a staged trust-governance mechanism that matches each defense to the evidence available at its stage, i.e., neighborhood-relative screening at cold start, recipient-directed counterfactual utility and distinct-reporter reputation as interaction evidence accumulates, diversity-aware retrieval, and prompt-independent capped routing that bounds reliance on any selected expert. Under fixed retrieval and ordering, we prove that replacing one normalized non-local expert shifts each class logit by at most a cap-controlled constant, yielding a margin condition for top-1 stability. Empirically, across multiple benchmarks, the trust governance of pFedREP does not sacrifice clean accuracy under heterogeneous settings. More importantly, under the primary attack, clean and worst-decile accuracy change by only 0.11 and 0.08 percentage points. In addition, a 20-client study isolates the individual trust mechanisms: relative to geometric screening alone, the full method reduces targeted exposure from 65.3% to 1.3%; and relative to nearest-prompt retrieval, it reduces severe negative transfer from 17.3% to 0.6%. These results support staged trust and bounded reliance as complementary principles for governing unaggregated expert pools. The code will be open-sourced.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated prompt learning, Anti-attack, Trust governance, Robust expert pool

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_drXWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PFEDREP, a method to improve the security of federated prompt learning when some clients may upload malicious prompts. It combines prompt screening, client feedback, reputation scores, diverse expert selection, and limits on each expert's influence. The authors also provide theoretical analysis and evaluate the method on several datasets and under different attack settings.

Critical strengths
  1. This paper addresses an important security problem in federated prompt learning. The proposed method can improve security without sharing raw client data.
  2. Selecting diverse experts and limiting their influence are useful ideas. They can reduce the risk of one malicious prompt having too much influence on the final prediction.
  3. The author/authors discuss important limitations and assumptions, which helps readers understand when the proposed method may work well.
Critical weaknesses
  1. The evaluation does not clearly show how well the proposed method works against adaptive attackers that change their behavior to avoid detection.
  2. Malicious prompts may still enter the retrieved expert pool. More experiments are needed to check whether the defense remains effective when attackers have more influence or work together.
  3. Some theoretical guarantees depend on fixed retrieval and ordering conditions. It is unclear whether these guarantees still hold when the expert pool changes during operation.
  4. The proposed method assumes that the server is honest and client identities are authenticated. These assumptions may not always hold in real-world federated learning systems.
Questions for authors
  1. Can the author/authors evaluate PFEDREP against stronger adaptive and colluding attackers to check whether it remains effective when malicious clients change their prompts or coordinate their attacks?
  2. How well does the proposed method scale when the number of clients and candidate experts increases significantly?
  3. Since malicious experts may still be retrieved during inference, can the authors provide more experiments to measure their impact under different attack settings and show which defense components are most effective in limiting their influence?
  4. How sensitive is PFEDREP to its screening thresholds and reputation parameters when the proportion of malicious clients increases?
  5. Can the author/authors evaluate the computational and communication overhead of PFEDREP compared with standard federated prompt learning, especially as the number of clients increases?
  6. Can you clarify the assumptions behind the theoretical guarantees and explain whether they still hold when the retrieved experts or their ordering change dynamically during inference?
8 comments
AuthorsOpenReview ↗

Response to Weakness 3 and 4

Response to Weakness 3. Theorem 1 can be applied to a fixed prediction state at any round, but it does not certify the transition between different states. It does not require the expert pool to remain unchanged throughout training. Instead, it compares two predictions for the same image after fixing the retrieved identities, their slot ordering, and the router parameters, with only one non-local slot’s representation replaced and the local branch unchanged.

Under this comparison, prompt-independent routing keeps the slot weight ai,j(x)a_{i,j}(x) unchanged. With the theorem’s norm bounds, replacing that representation changes each class logit by at most 2sai,j(x)≤2sαˉ2s a_{i,j}(x)\leq2s\bar\alpha. This is the paper’s conditional influence guarantee: it bounds reliance on one retained expert, regardless of how the system arrived at that fixed state.

If a pool update changes retrieval, ordering, or trained model parameters, the resulting prediction is not the single-slot comparison covered by the theorem. An earlier margin certificate therefore cannot be carried over automatically. At the updated state, the bound and margin condition must be evaluated again using the current retrieved set, router, effective cap, and logits. The guarantee remains conditional at each such evaluation; it does not establish stability across pool updates or end-to-end robustness to a poisoned training trajectory.

Response to Weakness 4. We agree that these assumptions limit applicability. pFedREP studies malicious clients within a federation that has trusted coordination and controlled participation; it does not establish those conditions itself. The paper explicitly assumes an honest server, authenticated client identities, and an uncompromised frozen backbone. Its robustness claims should therefore not be extended to deployments where the coordinator is compromised or adversaries can create unrestricted identities.

The honest-server assumption is needed because the server maintains the expert pool and feedback records and executes screening and retrieval. A malicious coordinator could alter those records or bypass the prescribed selection procedure. pFedREP does not include a mechanism for verifying a dishonest server’s execution, so protection against server compromise is outside the stated threat model.

Authenticated identities serve a different purpose: they let the system associate repeated reports with the same client. Distinct-reporter accounting prevents one identity from increasing the reporter count merely by submitting more reports; it does not establish that the reports are truthful or that different reporters act independently. Enrolled malicious clients can still collude, and Proposition 1 separately requires a bound on their fraction within each expert’s reporter set. If an attacker can create additional accepted identities, that premise need not hold. Thus identity enrollment/authentication and coordinator integrity must be supplied by the deployment. pFedREP governs prompt sharing under these assumptions; it does not provide Sybil resistance or remove the need to trust the coordinator.

AuthorsOpenReview ↗

Response to Question 1

Response. We completed 48 runs comparing full pFedREP with a governance-free control under adaptive and colluding attacks. The eight settings combine 20%/40% malicious clients, proximity radii 0.1/1.0, and independent/colluding attackers, with three seeds per policy. All runs use Food101/RN50, ten clients, ten rounds, five local epochs, full participation, and λ=0.5\lambda=0.5. Each full/control pair uses the same honest evaluation cohort.

For each malicious update, the attacker refines its prompt pp using

\mathcal L_{\rm clean}(p,y),\qquad \|p-c\|_2\leq r,$$ where $c$ is the pool median, fixed during that refinement. The trigger term promotes the target label, while the equally weighted clean term encourages preservation of clean performance. We use 20 gradient steps at step size 0.01, projecting onto the radius-$r$ ball after each step; the event audit confirms completion of all requested steps. Radius 1.0 relaxes the proximity constraint relative to 0.1. Optimization uses the attacker’s own training data and fixed current mixture as a recipient surrogate, not honest clients’ data or gradients through future retrieval and training. In the colluding setting, attackers initialize from the mean of available malicious prompts and coordinate feedback: they release the maximum permitted positive rewards for malicious experts and negative rewards for honest experts. This tests coordinated prompt initialization and report manipulation under the existing authenticated identities; it does not introduce additional identities. Table 3.2: All adaptive/colluding Food101/RN50 comparisons (48 runs). F/U denote full/governance-free policies. Values are percentages, mean $\pm$ sample SD over three seeds. Each F/U pair uses the same honest cohort; the 20% and 40% contamination blocks use different honest cohorts. | Malicious % | $r$ | Collude | CA (F) | CA (U) | ASR (F) | ASR (U) | |:---|:--:|:--:|:--:|:--:|:--:|:--:| | 20 | 0.1 | No | $93.36\pm0.07$ | $93.33\pm0.10$ | $0.062\pm0.018$ | $0.066\pm0.011$ | | 20 | 0.1 | Yes | $93.27\pm0.17$ | $93.31\pm0.09$ | $0.068\pm0.017$ | $0.060\pm0.009$ | | 20 | 1.0 | No | $93.30\pm0.18$ | $93.20\pm0.18$ | $0.055\pm0.015$ | $0.076\pm0.022$ | | 20 | 1.0 | Yes | $93.28\pm0.23$ | $93.37\pm0.14$ | $0.060\pm0.013$ | $0.070\pm0.009$ | | 40 | 0.1 | No | $94.36\pm0.25$ | $94.28\pm0.14$ | $0.105\pm0.031$ | $0.073\pm0.015$ | | 40 | 0.1 | Yes | $94.32\pm0.22$ | $94.24\pm0.32$ | $0.092\pm0.040$ | $0.105\pm0.040$ | | 40 | 1.0 | No | $94.33\pm0.27$ | $94.32\pm0.16$ | $0.121\pm0.043$ | $0.124\pm0.026$ | | 40 | 1.0 | Yes | $94.37\pm0.23$ | $94.40\pm0.21$ | $0.104\pm0.024$ | $0.111\pm0.009$ | Table 3.2 shows that both policies retain high clean accuracy, with no consistent ASR advantage for full pFedREP. At 20% contamination, its ASR ranges from $0.055\%$ to $0.068\%$, versus $0.060\%$ to $0.076\%$ without governance; at 40%, the respective ranges are $0.092$–$0.121\%$ and $0.073$–$0.124\%$. These results support accuracy preservation under the specified surrogate attacks, not a general claim that governance defeats adaptive or colluding adversaries. The 20% and 40% settings evaluate different honest cohorts, so higher CA in the latter cannot be interpreted as an improvement caused by more attackers.
AuthorsOpenReview ↗

Response to Question 2

Response. We measured scaling at 20, 50, and 100 clients on CIFAR10/RN50, comparing full pFedREP with a governance-free control at each client count. Each run uses ten rounds, 10% participation, five local epochs, and ten total expert slots. Three seeds per policy give 18 runs, all of which passed the GPU-isolation audit on an NVIDIA L20. This is a short scaling diagnostic, not the paper’s 120-round CIFAR accuracy benchmark. Table 3.3 reports the measured costs.

Table 3.3: CIFAR10/RN50 scaling: ten rounds, 10% participation, isolated L20 GPU per run. CA is final-round percentage; times are total recorded seconds divided by ten rounds. Peak is allocated GPU memory in GiB, not host memory. Means ±\pm sample SD over three seeds.

ClientsPolicyCATrain s/roundRound sSelect s/roundPeak GiB
20Full76.59±1.6276.59\pm1.6272.3±2.172.3\pm2.1100.6±1.9100.6\pm1.90.084±0.0050.084\pm0.0051.816±0.0001.816\pm0.000
20Governance-free76.00±0.9076.00\pm0.9071.3±2.471.3\pm2.499.3±2.299.3\pm2.2Not logged1.816±0.0001.816\pm0.000
50Full76.00±1.0976.00\pm1.0978.5±3.778.5\pm3.7124.6±3.2124.6\pm3.20.628±0.0540.628\pm0.0541.818±0.0001.818\pm0.000
50Governance-free75.97±0.8175.97\pm0.8178.1±3.778.1\pm3.7120.8±3.5120.8\pm3.5Not logged1.817±0.0001.817\pm0.000
100Full75.51±1.1275.51\pm1.1287.1±0.887.1\pm0.8177.9±2.0177.9\pm2.03.210±0.1743.210\pm0.1741.820±0.0001.820\pm0.000
100Governance-free75.17±1.3175.17\pm1.3187.2±1.187.2\pm1.1154.4±1.0154.4\pm1.0Not logged1.819±0.0001.819\pm0.000

The added overhead increases with client count. Mean paired round-time overhead relative to the control is 1.37%1.37\%, 3.17%3.17\%, and 15.28%15.28\% at 20, 50, and 100 clients, respectively. At 100 clients, full pFedREP takes 177.94177.94 seconds per round versus 154.37154.37 for the control. Its measured selection time rises from 0.0840.084 to 3.2103.210 seconds per round between 20 and 100 clients. Because participation is fixed at 10%, the number of clients trained per round also increases; total runtime growth should therefore not be attributed entirely to pool management. The matched policy comparison at each client count measures the additional cost of governance.

Peak allocated GPU memory stays near 1.821.82 GiB for both policies, but this does not measure the server’s host-memory cost. For MM available prompts of dimension dd, pairwise geometry requires O(M2d)O(M^2d) work, and the current broadcast implementation can allocate an O(M2d)O(M^2d) temporary. Directed recipient–expert history can require O(N2)O(N^2) storage for NN clients. Moreover, with partial participation, the available pool can contain fewer than NN prompts early in training. The results therefore demonstrate execution through 100 clients under this protocol, with increasing governance overhead; they do not establish linear scaling or feasibility at thousands of clients.

Reviewer_ESEjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript investigates the robustness issue of unaggregated expert pools in Personalized Federated Prompt Learning against backdoor and poisoning attacks. To address this issue, the authors propose PFEDREP, a trust governance framework incorporating multi-stage defense: (1) In the cold start phase, geometric anomaly screening and pruning are performed based on neighborhood median and median absolute deviation; (2) During the interaction process, clients calculate directed counterfactual utility and perform local isolation on persistently harmful experts; (3) The server aggregates the mean utility of each independent reporter and constructs a group reputation score against Byzantine contamination using the median and lower quantile; (4) Diversity-aware retrieval is performed by combining geometric affinity, directed utility, group reputation, and maximum redundancy penalty; (5) In the inference and routing phase, a prompt-agnostic bounded router is adopted to limit the weight of any non-local expert to αˉ\bar{\alpha}, and an upper bound on the logit shift under single expert substitution is provided. The authors evaluated the method on multiple visual-language small-sample and cross-domain benchmark datasets, as well as a 20-client synthetic simulator.

Critical strengths

The starting point of the manuscript is the security issue of non-aggregated expert pool in personalized federated prompt learning, which has practical research value.

Critical weaknesses

(1) There are issues with the literature, which makes me question whether the author used AI for literature processing. For example, in the author's list of references, there is "Learning Generative Visual Models from Few Training Examples". Although this reference can be found on Google Scholar and is listed as published at a workshop of CVPR in 2024, the full title of this reference is actually "Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories", and it includes three authors, as can be verified on its publisher, IEEE.

(2) If the receiving party calculates counterfactual utility on a local clean training batch without triggers, it may not be able to perceive backdoor triggering behavior, resulting in all evaluations from malicious experts in Figure 2 being positive? Would isolation and reputation mechanisms completely fail?

Questions for authors

(1) Please explain the specific differences in dimensions and calculation methods between zfull(x)z_{full}(x), vi,cfullv_{i,c}^{full} and z(x)z(x), vi,cv_{i,c} in formula (1) of the PROBLEM FORMULATION.

(2) On the Food101 or Flowers102 test set, what percentage of test samples can satisfy m(x)>4sαˉm(x) > 4s\bar{\alpha}? If the satisfaction rate is 0, what is the practical significance of this theoretical guarantee?

(3) In the results presented in Table 4, why did the clean accuracy of the honest client plummet from 93.20% to 38.19%? Is it because the neighborhood median/MAD clipping was contaminated by malicious large-norm hints, and instead, the hints of benign clients were mistakenly clipped?

(4) How can the counterfactual utility calculated on a clean local batch be used to detect backdoor attacks in principle?

6 comments
AuthorsOpenReview ↗

Response to Weakness 1 and 2

Response to Weakness 1. The reviewer is correct that our Caltech101 citation is incomplete: it omits the subtitle and two coauthors, Rob Fergus and Pietro Perona. The complete reference is:

Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories. 2004 Conference on Computer Vision and Pattern Recognition Workshop, p. 178. DOI: 10.1109/CVPR.2004.383.

The publication year is 2004, not 2024; our submitted bibliography already gives 2004. The required corrections are therefore to the title and authorship. We include the corrected BibTeX entry in the response package. This corrects the attribution of the Caltech101 dataset reference without changing the dataset or reported experimental results.

Regarding AI assistance, the manuscript includes an AI-use disclosure. Regardless of the tools used, checking the accuracy and completeness of references is the authors’ responsibility. We accept responsibility for this error and provide the full citation and DOI so that the correction can be independently verified.

Response to Weakness 2. Yes. A backdoored expert can receive positive utility if it improves predictions on the recipient’s clean feedback batch while behaving maliciously only on triggered inputs. The leave-one-expert-out reward measures the change in that batch’s loss, not the presence of a backdoor. Positive malicious-expert utilities in Figure 2 therefore indicate beneficial transfer on the observed clean data; they are not evidence that the expert is safe on triggered data.

The corrected experiments confirm that the rejection mechanisms do not activate in this regime. Across all 276 rebuttal runs, honest-recipient selections contain no directed-utility or quarantine rejections, and no new quarantine event is recorded. In the three primary attacked runs, 13 of 48 observed honest-to-malicious pair means are negative, but the minimum is only −0.000663-0.000663, far above the rejection threshold of −0.30-0.30. Thus even these negative means do not provide a rejection signal at the configured threshold, and the low measured ASR cannot be attributed to utility-based exclusion or quarantine.

This is a limitation of the available feedback, rather than evidence that reputation and quarantine have no role in any setting. Quarantine suppresses a recipient–expert pair after sufficiently persistent negative utility. Reputation aggregates distinct recipients’ observations into a ranking prior; it does not independently test for a trigger. When harmful transfer is visible in feedback, these mechanisms can use that evidence, as evaluated in the paper’s controlled negative-transfer study. When the backdoor leaves the observed clean behavior benign, neither mechanism can identify its hidden harm from those observations alone. The supported claim is therefore governance of observed transfer, not universal backdoor detection; capped routing supplies a separate conditional influence limit but does not resolve this detection blind spot.

AuthorsOpenReview ↗

Response to Question 1 and 2

Response to Question 1. The two pairs use the same frozen CLIP encoders but differ in feature dimension and role. The full image feature zfull(x)z_{\rm full}(x) is CLIP’s normalized image embedding. The full class-text feature vi,cfullv^{\rm full}_{i,c} is the normalized projected end-of-text embedding for class cc, constructed using client ii’s prompt. Both lie in CLIP’s shared image–text space of dimension D=1024D=1024 for RN50 or D=512D=512 for ViT-B/16.

The routed features are coordinate subsets of these full features. In the default configuration, a fixed operator PDP_D retains every kkth coordinate, starting at coordinate zero:

v_{j,c}=P_Dv^{\rm full}_{j,c},\qquad (P_Du)_r=u_{kr},\quad r=0,\ldots,127.$$ Here $k=8$ for RN50 and $k=4$ for ViT-B/16, so both routed features have dimension 128. This is deterministic slicing, not another encoder or a learned projection. For $j=i$, $v_{i,c}$ is the sliced local class-text feature; for $j\in S_i^t$, it is computed with the selected non-local expert’s prompt. Equation (1) therefore combines a full-dimensional private prediction with a lower-dimensional expert mixture: $$\begin{aligned} \ell_{i,c}(x) &=\lambda s\langle z_{\rm full}(x),v^{\rm full}_{i,c}\rangle &&\text{(private local branch)}\\\\ &\quad+s\Bigl\langle z(x),\sum_{j\in\\{i\\}\cup S_i^t}a_{i,j}(x)v_{j,c}\Bigr\rangle &&\text{(routed expert branch).} \end{aligned}$$ The local prompt contributes to both terms, but only the second includes retrieved experts. The default coefficient is $\lambda=0.5$. If no non-local expert is available, the implementation falls back to the full local logits directly. Normalization occurs before slicing, with no subsequent renormalization. Hence $\\|z(x)\\|_2\leq1$ and $\\|v_{j,c}\\|_2\leq1$, although the sliced vectors need not have unit norm. This is the qualification needed for the problem formulation’s normalization sentence; it is consistent with the norm assumptions used in Theorem 1. **Response to Question 2.** The measured fraction satisfying $m(x)>4s\bar\alpha$ is $0\%$ on both Food101 and Flowers102, for clean and triggered inputs. This holds at the default cap of 0.35 and at target caps 0.10 and 0.20. Table 2.1 reports the certified/eligible counts for final-round routed predictions on honest recipients, summed over seeds 1–3. Both input conditions evaluate the same proximity-attacked RN50 models; triggered counts exclude true target-class examples, as in ASR. Table 2.1: Certificate sweep on proximity-attacked RN50 models. CA/ASR are percentages, mean $\pm$ sample SD over three seeds. Counts are certified/eligible final-round predictions summed over seeds. Every listed clean and triggered coverage is 0%. The effective cap is $\max\{\alpha,1/E\}$; the Food101 target-0.1 mean effective cap is 0.10324 because some retrieved sets are smaller. | Dataset | Target $\alpha$ | CA | ASR | Clean count | Triggered count | |:---|:--:|:--:|:--:|:--:|:--:| | Food101 | 0.1 | $93.27\pm0.08$ | $0.053\pm0.006$ | $0/72,000$ | $0/71,100$ | | Food101 | 0.2 | $93.30\pm0.04$ | $0.060\pm0.020$ | $0/72,000$ | $0/71,100$ | | Food101 | 0.35 | $93.33\pm0.10$ | $0.070\pm0.017$ | $0/72,000$ | $0/71,100$ | | Flowers102 | 0.1 | $99.04\pm0.14$ | $0.190\pm0.060$ | $0/5,817$ | $0/5,781$ | | Flowers102 | 0.2 | $99.07\pm0.13$ | $0.086\pm0.108$ | $0/5,817$ | $0/5,781$ | | Flowers102 | 0.35 | $99.06\pm0.22$ | $0.121\pm0.079$ | $0/5,817$ | $0/5,781$ | The criterion is conservative relative to the observed margins. Here $m(x)$ is the gap between the largest and second-largest logits. With $s=100$ and $\bar\alpha=0.35$, the per-class perturbation bound is 70, so the sufficient top-1 test requires a margin greater than 140. Mean clean margins are approximately 4.30 on Food101 and 4.40 on Flowers102. Even the smallest tested target cap gives a sufficient margin threshold of at least 40, and no evaluated prediction passes the test. At these settings, there is no demonstrated top-1 certification benefit. Theorem 1 instead gives the enforceable bound $|\Delta\ell_c|\leq2s\bar\alpha$ for one selected non-local expert, assuming norm-bounded features and fixed image, retrieval, ordering, and router parameters. This formalizes the paper’s bounded-reliance claim even if selection admits a harmful expert; it does not certify the observed decisions or explain their low ASR. Because the margin test is sufficient rather than necessary, failing it also does not prove vulnerability. Neither the logit bound nor its margin corollary is an end-to-end guarantee against training-time poisoning.
AuthorsOpenReview ↗

Response to Question 3 and 4

Response to Question 3. The corrected reruns do not reproduce the drop to 38.19%38.19\%, and the available evidence does not support attributing it specifically to contaminated median/MAD clipping. The implementation audit identified a separate problem: the sequential client simulation restored each client’s prompt and router weights but did not fully isolate optimizer and scheduler history. Shared optimization state permits one client’s updates to affect another client’s training, independently of whether the former’s prompt passes the expert-selection gates. The submitted collapse is therefore not reliable evidence of a failure caused by geometric clipping.

We repeated the Food101/RN50 replacement experiment with client-specific optimization state for both full pFedREP and the no-geometry control, using scales 1, 5, and 10 and seeds 1–3 (18 runs). Table 2.2 reports the corrected results.

Table 2.2: Corrected Food101/RN50 replacement sweep (%), with private optimization state for each client. Local CA evaluates the full-dimensional local branch of the same trained model; it is not a separately trained local baseline. Means ±\pm sample SD over seeds 1–3.

ScalePolicyMixture CALocal CAASRMExp
1.0Full93.34±0.1993.34\pm0.1992.99±0.2392.99\pm0.230.089±0.0220.089\pm0.02222.64±0.5822.64\pm0.58
1.0No geometry93.33±0.0393.33\pm0.0392.88±0.1592.88\pm0.150.076±0.0170.076\pm0.01722.22±0.0022.22\pm0.00
5.0Full93.30±0.1493.30\pm0.1492.91±0.2292.91\pm0.220.066±0.0140.066\pm0.0140.00±0.000.00\pm0.00
5.0No geometry93.40±0.1693.40\pm0.1693.05±0.1893.05\pm0.180.063±0.0210.063\pm0.02122.22±0.0022.22\pm0.00
10.0Full93.32±0.1193.32\pm0.1192.92±0.2192.92\pm0.210.070±0.0060.070\pm0.0060.00±0.000.00\pm0.00
10.0No geometry93.36±0.1293.36\pm0.1293.05±0.1693.05\pm0.160.060±0.0050.060\pm0.00522.22±0.0022.22\pm0.00

At scales 5 and 10, full pFedREP achieves CA 93.30%93.30\% and 93.32%93.32\%, respectively, with zero malicious-slot exposure. Disabling geometry gives CA 93.40%93.40\% and 93.36%93.36\%, with exposure of 22.22%22.22\% at both scales. The local-branch accuracies also remain near 93%93\%. Thus geometry excludes the large-norm uploads in these corrected runs without reproducing the reported accuracy collapse; high clean accuracy is also retained by the no-geometry control.

The diagnostics do show some benign filtering. At each of scales 5 and 10, all 60 malicious expert-round snapshots and eight of 240 honest snapshots exceed the geometry threshold across the three seeds. The filter is therefore not perfectly selective, yet these runs retain near-93%93\% final accuracy. We supersede the affected 38.19%38.19\% result with the corrected measurements. Because the reruns include multiple state-handling corrections, they do not isolate optimizer momentum as the sole cause of the earlier collapse. The supported conclusion is limited to the observed screening and accuracy behavior under the corrected protocol.

Response to Question 4. Clean-batch counterfactual utility is not, by itself, a backdoor detector. It provides a rejection signal when retaining an expert increases the recipient’s observed loss. In Eq. (5), the reward is the clipped difference between the loss without the expert and the loss with it, with the remaining routing weights renormalized. If removing the expert reduces the batch loss, the reward is negative; sufficiently negative and persistent rewards can trigger directed quarantine. This detects harmful transfer on the observed data, not the existence of a trigger or malicious intent.

The limitation follows directly from the information available to the statistic. Suppose a benign expert and a backdoored expert produce identical logits on every observed clean feedback example. With the other experts and routing weights fixed, they yield identical counterfactual rewards, even if their predictions differ on triggered inputs. A decision rule using only these rewards cannot distinguish that pair. Reputation aggregates the same observations and cannot supply the missing trigger-specific evidence.

The paper’s supported claim is therefore recipient-specific governance of observed transfer. Exposure to a harmful expert can be reduced when it causes sufficiently negative observed utility, but a backdoor that preserves clean utility can remain undetected. Negative utility also does not uniquely identify an attacker: an honest expert can transfer poorly to a particular recipient. This is why quarantine is directed to the recipient–expert pair rather than treated as proof of global maliciousness. The separate routing cap limits conditional influence; it does not turn clean utility into a general backdoor-detection guarantee.

Reviewer_rF9sOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the robustness of personalized federated prompt learning when retained prompts are identifiable and retrievable as separate experts. The proposed mechanism consists of components that govern the process from cold start to final aggregation. Specifically, it leverages neighborhood-relative robust screening to keep different prompt centers, followed by a series of actions, including maintaining the reputation of experts and quarantining harmful experts. The paper also derives a conditional bound on the logit influence of replacing one selected expert. The experiments use five few-shot datasets with 10 clients each and various PromptFL baselines. The method is also evaluated under poisoning attacks, showing no degradation in clean accuracy and robustness to malicious prompts.

Critical strengths

1.The setting and task formulation are practical. A conventional aggregation pipeline does not involve decisions about expert admission, recipient-specific selection, and prediction influence.

2.Personalization provides a strong motivation for recipient-specific trust. Trust-based selection and bounded reliance are coherent and natural design choices for moving from population-level reputation to heterogeneous specialists.

3.The experimentation is meaningful in terms of reporting malicious-expert exposure alongside ASR, distinguishing selection behavior from predictive outcomes. The theoretical analysis appears correct under its stated conditions.

Critical weaknesses

1.The proposed mechanism does not show added value for the defense system. The ASR is marginal in settings with or without the proposed mechanism. The results do not show an empirical improvement in ASR for this system.

2.The controlled simulation does not close the evidence gap for detecting backdoor behavior. Eq. (5) shows the loss-based utility signal in actual backdoor attacks, but in the image-based experiment, the negative-utility gates and quarantine are not activated. An expert can improve performance on local examples while being harmful on a rare trigger distribution.

3.The technical contribution is marginal and lies primarily in the integration of mechanisms. The practical value of the theoretical guarantee is not established.

4.The novelty of this work lies more on the system side, in formulating and combining screening, recipient-specific utility, reputation, diversified retrieval, and bounded routing. The limited differentiation among the real-data ablations leaves the necessity and complementarity of the full design insufficiently demonstrated. There is also no end-to-end guarantee for training-time poisoning that can change multiple experts, retrieval decisions, and subsequent optimization.

Questions for authors

1.What is the performance of the primary attack against a matched expert-pool model with all proposed governance components removed? I am also wondering about the CA and ASR of malicious experts before and after imposing the proximity constraints.

2.What are the separate effects of each component under domain-shift attacks with matched controls?

3.In the few-shot protocol, is the attack target included in every evaluated recipient’s prediction vocabulary?

4.What fraction of clean and triggered test inputs satisfies the stated margin condition under the default cap?

8 comments
AuthorsOpenReview ↗

Response to Weakness 1 and 2

Response to Weakness 1. We agree that low ASR alone does not establish the added value of the governance mechanism. To address this concern, we compared full pFedREP with a governance-free expert pool under matched training and evaluation conditions. Table 1.1 distinguishes two comparisons: the effect of the attack on each policy, and the effect of governance under the same attack. All results use the same eight honest recipients and three seeds.

Table 1.1: Matched Food101/RN50 controls, final round (%). Means ±\pm sample SD over three seeds. Controls and attacks use the same eight honest recipients. Control ASR evaluates patched inputs without attack training. MExp is cumulative malicious non-local-slot exposure; it is undefined for local-only prediction.

PolicyTrainingCAW10ASRMExp
Full pFedREPControl93.37±0.1593.37\pm0.1589.13±0.4289.13\pm0.420.063±0.0110.063\pm0.01123.18±0.9123.18\pm0.91
Full pFedREPAttack93.33±0.1093.33\pm0.1089.23±0.3289.23\pm0.320.070±0.0170.070\pm0.01722.94±1.1022.94\pm1.10
Governance-freeControl93.30±0.1693.30\pm0.1688.98±0.1888.98\pm0.180.062±0.0150.062\pm0.01522.22±0.0022.22\pm0.00
Governance-freeAttack93.25±0.1693.25\pm0.1689.13±0.3989.13\pm0.390.063±0.0040.063\pm0.00422.22±0.0022.22\pm0.00
Local-onlyControl92.87±0.1592.87\pm0.1588.80±0.3388.80\pm0.330.079±0.0140.079\pm0.014—
Local-onlyAttack92.87±0.1592.87\pm0.1588.80±0.3388.80\pm0.330.079±0.0140.079\pm0.014—
Prompt-content routerControl92.88±0.4692.88\pm0.4689.16±0.6189.16\pm0.610.110±0.0300.110\pm0.03022.22±0.0022.22\pm0.00
Prompt-content routerAttack92.94±0.4092.94\pm0.4089.27±0.6789.27\pm0.670.104±0.0170.104\pm0.01722.22±0.0022.22\pm0.00

Relative to its unattacked control, full pFedREP’s clean accuracy changes from 93.37%93.37\% to 93.33%93.33\%, while ASR changes from 0.063%0.063\% to 0.070%0.070\%. These measurements support the paper’s claim that prediction performance remains near the unattacked control under this attack. However, the governance-free pool also has low attacked ASR (0.063%0.063\%). Thus the matched comparison does not establish an ASR benefit from governance; we do not use near-control ASR as evidence of such a benefit.

The additional evidence for governance concerns expert selection, which is a separate part of the paper’s claim. In the corrected DomainNet ablation, full pFedREP reduces the fraction of selected non-local slots occupied by malicious experts from 59.40%59.40\% to 15.78%15.78\%, with clean accuracy of 96.22%96.22\% versus 96.25%96.25\% for the governance-free pool. This is an exposure reduction, not an ASR reduction: the respective ASRs are 0.638%0.638\% and 0.548%0.548\%. Accordingly, the empirical conclusion is that pFedREP can reduce malicious-expert exposure in this setting while retaining similar clean accuracy, but the evaluated attacks do not establish a consistent ASR advantage.

Response to Weakness 2. We agree that the controlled study does not establish detection of image backdoors. The utility in Eq. (5) measures an expert’s contribution to the recipient’s loss on the observed batch: a positive value means that retaining the expert lowers that loss. It does not measure behavior on an unseen trigger distribution. An expert can therefore receive positive clean utility while remaining harmful on triggered inputs. Aggregating these rewards into reputation does not remove this limitation.

The image-backdoor results must be interpreted accordingly. Across all 276 rebuttal runs, the audit records no directed-utility rejection, quarantine rejection, or new quarantine event for honest recipients. In the three primary attacked runs, the lowest observed honest-to-malicious pair mean is −0.000663-0.000663, far above the rejection threshold of −0.30-0.30. Thus the feedback does not provide a sufficiently negative signal to activate these gates. Their inactivity is not evidence that the uploaded experts are benign, and the low measured ASR cannot be attributed to rejection by these mechanisms.

The controlled 20-client study addresses a different question: whether the controller suppresses repeated harmful transfer when harm is observable in recipient-specific feedback. Malicious experts are assigned negative utility for targeted recipients but positive utility for other recipients, explicitly testing why a population-level judgment can miss personalized harm. Under this simulated feedback model, full pFedREP reduces targeted exposure from 65.3%65.3\% with geometry alone to 1.3%1.3\%. This supports the paper’s claim that directed feedback can govern observable negative transfer; it does not demonstrate detection of a backdoor that preserves clean utility. The image experiments and the simulation therefore provide different evidence, and we do not use the simulation to close the hidden-trigger detection gap.

AuthorsOpenReview ↗

Response to Weakness 3 and 4

Response to Weakness 3. The contribution is a governance formulation for unaggregated prompt experts, rather than a new median/MAD estimator or diversity criterion. Because uploaded prompts remain separately retrievable, an expert can benefit one recipient while harming another, and a selected harmful expert continues to contribute directly to predictions. pFedREP therefore separates two decisions: recipient-specific evidence determines which experts to select, while prompt-independent capped routing limits reliance on an expert even if selection fails to exclude it. This separation is the technical rationale for the staged design, not a claim that its individual statistical tools are new.

Theorem 1 formalizes the second decision. With the image, retrieved identities, slot ordering, and router parameters fixed, replacing an expert cannot change its slot weight because the router does not depend on prompt content. For representations with norm at most one, the resulting class-logit change satisfies ∣Δℓc∣≤2saj(x)≤2sαˉ|\Delta\ell_c|\leq2s a_j(x)\leq2s\bar\alpha; the unchanged local branch cancels. The cap thus imposes an explicit influence limit without assuming that the selected expert has been correctly identified as benign. This is a conditional single-expert bound, not a guarantee against changes to retrieval or the subsequent training trajectory.

The practical certification result is more limited. Top-1 invariance requires a margin greater than 4sαˉ4s\bar\alpha. At the default s=100s=100 and αˉ=0.35\bar\alpha=0.35, this threshold is 140, whereas the mean observed clean margins are approximately 4.30 on Food101 and 4.40 on Flowers102. Accordingly, measured certificate coverage is zero on both clean and triggered inputs, including the additional target-cap settings 0.10 and 0.20. The theorem specifies an enforceable conditional influence constraint, but these experiments do not establish a useful top-1 certificate or an ASR improvement from that constraint. We retain the paper’s bounded-reliance claim with this explicit distinction between the logit bound and its empirical certification coverage.

Response to Weakness 4. pFedREP’s contribution is a system-level design for governing identifiable experts at three stages: admission, recipient-specific selection, and prediction. Geometry supplies an upload-time screening signal before behavioral feedback exists. Directed utility and distinct-reporter reputation incorporate observed transfer, while diversity discourages concentration on similar experts. Capped routing then limits a selected expert’s influence without requiring selection to be infallible. These roles explain why the components are combined; they do not, by themselves, establish that every component is necessary in every attack setting.

The corrected ablations provide setting-specific evidence. Removing diversity raises malicious-slot exposure from 15.78%15.78\% to 65.14%65.14\% on DomainNet and from 40.53%40.53\% to 43.91%43.91\% on Office. In the replacement experiments at scales 5 and 10, geometry reduces malicious-slot exposure from 22.22%22.22\% to zero, with clean accuracy remaining near 93%93\%. These results demonstrate effects on admission and exposure, not a consistent ASR improvement. Conversely, removing quarantine reproduces the full model’s domain results because quarantine never activates. The controlled study evaluates utility and quarantine jointly under observable negative transfer; it does not establish quarantine’s separate benefit against image backdoors. We therefore interpret complementarity as distinct, condition-dependent roles; the current evidence does not establish universal necessity or a synergistic ASR benefit from the full pipeline.

The reviewer is also correct that the theorem is not an end-to-end poisoning guarantee. Theorem 1 bounds the class-logit change from replacing one non-local expert only when the image, retrieved identities, slot ordering, and router parameters are fixed. Training-time poisoning can alter several of these quantities and subsequent client updates, so its cumulative effect is outside the theorem’s scope. Consistent with the paper’s bounded-reliance claim, the guarantee concerns a selected expert’s conditional influence; robustness of the evolving federated system remains an empirical question.

AuthorsOpenReview ↗

Response to Question 1

Response. Under the primary attack, the governance-free expert pool achieves CA 93.25±0.16%93.25\pm0.16\% and ASR 0.063±0.004%0.063\pm0.004\%, compared with 93.33±0.10%93.33\pm0.10\% and 0.070±0.017%0.070\pm0.017\% for full pFedREP. Table 1.2 reports the matched attacked and unattacked runs using the same Food101/RN50 protocol, eight honest recipients, and seeds 1–3. The governance-free control disables geometric screening/clipping, behavioral feedback, reputation, quarantine, diversity, robust cold-start initialization, and the routing cap. It retains the prompt-independent router and client-state isolation to keep those architectural and training factors fixed. We also report a separate prompt-content-router control and an independently trained local-only control. Thus the low deployed ASR is not unique to full pFedREP.

Table 1.2: Matched Food101/RN50 controls, final round (%). Means ±\pm sample SD over three seeds. Controls and attacks use the same eight honest recipients. Control ASR evaluates patched inputs without attack training. MExp is cumulative malicious non-local-slot exposure; it is undefined for local-only prediction.

PolicyTrainingCAW10ASRMExp
Full pFedREPControl93.37±0.1593.37\pm0.1589.13±0.4289.13\pm0.420.063±0.0110.063\pm0.01123.18±0.9123.18\pm0.91
Full pFedREPAttack93.33±0.1093.33\pm0.1089.23±0.3289.23\pm0.320.070±0.0170.070\pm0.01722.94±1.1022.94\pm1.10
Governance-freeControl93.30±0.1693.30\pm0.1688.98±0.1888.98\pm0.180.062±0.0150.062\pm0.01522.22±0.0022.22\pm0.00
Governance-freeAttack93.25±0.1693.25\pm0.1689.13±0.3989.13\pm0.390.063±0.0040.063\pm0.00422.22±0.0022.22\pm0.00
Local-onlyControl92.87±0.1592.87\pm0.1588.80±0.3388.80\pm0.330.079±0.0140.079\pm0.014—
Local-onlyAttack92.87±0.1592.87\pm0.1588.80±0.3388.80\pm0.330.079±0.0140.079\pm0.014—
Prompt-content routerControl92.88±0.4692.88\pm0.4689.16±0.6189.16\pm0.610.110±0.0300.110\pm0.03022.22±0.0022.22\pm0.00
Prompt-content routerAttack92.94±0.4092.94\pm0.4089.27±0.6789.27\pm0.670.104±0.0170.104\pm0.01722.22±0.0022.22\pm0.00

To measure malicious-expert potency separately from mixture prediction, we evaluate each of the two final malicious prompts as a standalone classifier immediately before and after the attacker’s radius-0.1 proximity projection. Each prompt is tested on its owner’s cohort and all eight honest cohorts, without fitting to test data. The attack poisons 20% of the malicious clients’ training examples using a 4×44\times4 patch and target label 0. Table 1.3 reports both CA and ASR; its “Before” and “After” rows describe the upload constraint, not removal or application of the server’s defense.

Table 1.3: Standalone malicious-prompt potency before/after the attacker’s radius-0.1 upload projection (%). Each seed evaluates both malicious prompts. CA averages the relevant expert–recipient cohort accuracies; ASR pools non-target counts within seed. Own denotes each attacker’s own test cohort; honest denotes all eight honest cohorts. These are standalone classifiers, not deployed mixtures.

Training policyProjectionOwn CAOwn ASRHonest CAHonest ASR
FullBefore69.17±12.1869.17\pm12.1823.35±16.6423.35\pm16.6426.64±21.6526.64\pm21.6557.10±26.2857.10\pm26.28
FullAfter78.95±0.6078.95\pm0.601.86±0.441.86\pm0.4465.88±5.1265.88\pm5.122.43±0.442.43\pm0.44
Governance-freeBefore77.84±7.9877.84\pm7.9814.83±11.5514.83\pm11.5532.34±16.6932.34\pm16.6943.61±19.8643.61\pm19.86
Governance-freeAfter78.77±2.0378.77\pm2.031.87±0.911.87\pm0.9166.70±4.4066.70\pm4.402.44±1.162.44\pm1.16

On honest cohorts, projection changes the full-pFedREP malicious experts’ CA from 26.64%26.64\% to 65.88%65.88\% and ASR from 57.10%57.10\% to 2.43%2.43\%. For experts trained with the governance-free pool, CA changes from 32.34%32.34\% to 66.70%66.70\% and ASR from 43.61%43.61\% to 2.44%2.44\%. The proximity constraint therefore substantially weakens the standalone attack before the prompts enter the deployed mixture. These diagnostics explain why strong pre-projection ASR should not be treated as evidence of equally strong uploaded experts. Together with the matched controls, they support the paper’s near-control prediction result under the specified attack, but do not establish that trust governance causes the low deployed ASR.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.