- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies federated skill learning for LLM‑based agents under client‑side task heterogeneity. Existing federated skill‑learning methods such as FedSkill share skills mainly based on embedding similarity, which may introduce irrelevant or conflicting skills when clients hold different task distributions. The authors propose P‑FedSkill with three components: Local Skill Extraction (LSE) that extracts skill‑patches grouped by task type, Personalized Skill Selection (PSS) leveraging Bayesian Beta posterior with uncertainty penalty to build client‑specific skill sets, and Task‑Aware Skill Routing (TASR) for adaptive on‑the‑fly skill selection at inference time. The paper also provides finite‑sample sub‑optimality bounds for PSS
Critical strengths
Meaningful practical motivation. The paper addresses an important real‑world scenario: edge agents generate private interaction trajectories that cannot be directly shared, and clients exhibit heterogeneous task distributions. It points out the limitation of similarity‑only skill sharing in existing federated skill learning.
Critical weaknesses
- Missing centralized‑oracle upper‑bound baseline. All experiments run under federated task partitioning where no single client can access the full training set. There is no centralized‑oracle control: a single agent with access to all training tasks, using the same LSE pipeline and evaluated on the identical held‑out test set. Without this reference, we can only confirm relative improvements versus other federated baselines. We cannot quantify how much performance degradation is purely caused by federated data fragmentation, making it hard to judge how close P‑FedSkill approaches the ideal full‑data capability. This gap exists across all four benchmarks.
- Confounding factors in non‑IID simulation setup. The paper simulates client‑side category heterogeneity (different clients prefer different business task types) by partitioning tasks according to raw task types via Dirichlet sampling. However, across the adopted benchmarks (especially -Bench), different task types have large intrinsic differences in difficulty. As a result, non‑IID partitioning conflates category heterogeneity (target of the paper) and difficulty heterogeneity (confounder) — these are two distinct phenomena. Worse, training trajectories are dynamically rollout‑generated by each client’s local agent. Clients assigned many hard tasks will produce more failed low‑quality trajectories and inferior skill‑patches, further amplifying the confounder. The counter‑intuitive result that non‑IID accuracy slightly exceeds IID accuracy cannot be reliably attributed to the benefit of personalized skill selection; difficulty‑distribution shift provides an alternative plausible explanation. Only IID comparisons are relatively clean.
- Uneven experimental evidence across benchmarks
- Component‑wise ablations and canary‑based privacy leakage evaluation are only performed on DS‑1000 and GTA. No ablation or privacy evaluation is conducted on ‑Bench and ALFWorld. Evidence for component effectiveness and privacy protection is extrapolated rather than directly measured on these datasets.
- On -Bench, only aggregated dataset‑level accuracy numbers are reported (59.17% IID / 60.83% non‑IID). Per‑domain numerical results for Retail / Airline / Telecom are only shown in figures without tabulated values. Therefore it is unclear whether performance gains hold uniformly across easy and hard sub‑domains, in particular the challenging Telecom domain.
Questions for authors
Q1. How large is the performance gap between P‑FedSkill and the centralized oracle in your simulation setting? Could this gap explain most of the remaining error under federation? Q2. In your non‑IID setup, to what extent is client performance correlated with the fraction of intrinsically hard tasks assigned to that client? Could the slightly higher non‑IID accuracy be explained purely by difficulty distribution shifts? Q3. On -Bench, does the improvement persist in the hardest Telecom sub‑domain, or is the overall gain mainly driven by easier Retail cases? Q4. What fraction of test episodes actually activate the adaptive‑branch logic in TASR? How many samples effectively reduce to fixed‑K selection?