- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FedPTR, a framework for federated online soft-prompt routing under corrupted evaluation feedback. The method aims to reduce the influence of manipulated feedback before evaluation scores are observed. It compares local and federated uncertainty to measure dependence on cross-client information and adds a federation-aware attenuation term to standard uncertainty clipping. The paper also provides theoretical guarantees and experiments showing how corrupted feedback can affect clients with clean evaluation channels.
Critical strengths
The paper is well-written, and the proposed method is easy to follow. The problem is interesting and relevant to federated online learning systems that use automatic evaluators.
The paper clearly demonstrates how corrupted evaluation feedback can affect other clients through shared federated updates. The matched local and federated comparisons help illustrate this issue.
The theoretical analysis is clearly presented, and the assumptions are explicitly stated.
I appreciate that the authors separate trusted utility from observed evaluator scores. This makes the experimental results more meaningful under corrupted feedback.
Critical weaknesses
- Novelty of the proposed method: The proposed method is interesting, but I have some concerns about the extent of its methodological novelty. FedPTR builds on the uncertainty-weighted estimator of He et al. (2022), with the main modification being an additional denominator based on the difference between local and federated uncertainty.
While this extension is reasonable and the theoretical analysis provides additional insights, I would like to better understand how much of the improvement comes specifically from this new component.
Also, the connection to LLMs and soft-prompt routing seems mainly to come from the application setting. The routing algorithm operates on frozen query–prompt representations, and it is not entirely clear which parts of the method are specific to LLM prompt routing.
- Possible inconsistency between Figure 6 and Table 1: Based on my understanding, there seems to be an inconsistency between Figure 6 and Table 1.
Section C.7 mentions that the utility results in Figure 6 are for T5-Flan-base on AG News under attack, which appears to be the same model and dataset as the first column of Table 1. However, the reported values are quite different. For example, FedPTR achieves around 0.847 in Figure 6 compared with 0.785 in Table 1, while Local Weighted achieves around 0.828 in Figure 6 compared with 0.613 in Table 1. These differences seem significant, especially for Local Weighted. It would be helpful if the authors could clarify whether the two results use different corruption budgets, evaluation settings, or other experimental conditions.
-
Missing federated uncertainty-weighted baseline in the main comparison: The paper describes a 2×2 comparison between local and federated methods and weighted and unweighted methods. However, Table 1 does not separately include a federated version of the uncertainty-weighted method of He et al. (2022) without the proposed attenuation term. Figure 6 includes a comparison with γ = 0, which removes the federation-aware denominator while keeping standard uncertainty clipping. However, this comparison is not reported across the datasets and models in Table 1. Including this baseline in the main results would help distinguish the benefit of applying existing robust weighting in a federated setting from the additional improvement provided by the proposed denominator.
-
Minor presentation issues: There are some typos in the figures, including "rount," "AItered," and "Local–federatedpre-feedback influence contro" in Figures 1 and 3.
Also, some important algorithmic and implementation details are provided only in the appendix. Including a more complete description of the algorithm in the main paper would improve clarity.
Questions for authors
Figure 6 and Table 1: Could the authors clarify why the attacked-utility values for FedPTR and Local Weighted differ substantially between Figure 6 and Table 1 for T5-Flan-base on AG News? Are the corruption budgets, evaluation metrics, and experimental settings the same?
Federated weighted baseline: Could the authors report the performance of federated uncertainty weighting with γ = 0 across the settings in Table 1? This would help clarify how much improvement comes from standard uncertainty clipping and how much comes from the proposed federation-aware attenuation term.
Methodological contribution: Could the authors explain more clearly how the proposed uncertainty-gap term differs from existing uncertainty-weighted approaches beyond its use in a federated setting?
Connection to LLMs: Which parts of FedPTR specifically depend on LLMs or soft-prompt routing? Would the same method apply directly to general federated contextual-bandit problems with frozen feature representations?