acceptodds
Under review as a conference paper at ICLR 2027

PFedMA: Who Should Join Next as Federated Multi-Agent Orchestration Evolves?

Abstract

Federated Multi-Agent Systems (FedMAS) offer a promising paradigm for collaborative reasoning among LLM-based agents while keeping raw client data decentralized. However, reliable multi-round collaboration remains challenging. Agent selection must take into consideration evolving task requirements and the agent’s historical reliability, while privacy rules must adapt to changing collaboration contexts and meet the client’s personalized needs. To address these challenges, we propose PFedMA, a federated multi-agent framework for feedback driven multi-round collaboration under heterogeneous privacy requirements. Across rounds, PFedMA maintains historical capability and reliability states to guide agent selection, evaluates task responses before aggregation, and refines subsequent queries according to accumulated collaboration feedback. PFedMA enables bidirectional local to global and global to local private rule evolution across heterogeneous clients. Clients generalize locally updated privacy rules and selectively incorporate globally maintained rules, thereby facilitating cross-client privacy rule evolution while preserving personalized constraints. Extensive experiments across reasoning, privacy sensitive, and adversarial settings show that PFedMA improves task performance, reduces measured privacy leakage, and improves system robustness. The implementation code for PFedMA is available at https://anonymous.4open.science/r/paper_PFedMA-2F44/.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Federated Multi-Agent Systems, Large Language Models, Feedback Driven Orchestration, Agent Selection, Privacy Rule Co-Evolution.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_25eUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

PFedMA addresses multi-round collaboration among clients with private local information and heterogeneous privacy requirements. It combines historical capability and reliability states for client selection, structured intent/evidence/consensus evaluation before aggregation, feedback-guided query refinement, and bidirectional privacy-rule sharing with local adaptation. The paper reports improvements in reasoning performance, measured privacy leakage, and prompt-injection robustness. Its most promising idea is to treat reusable privacy knowledge as an evolving part of collaboration. However, substantial discrepancies between the paper and the released training/inference paths leave the main experimental claims insufficiently attributable to the proposed method.

Critical strengths

Relevant problem and useful perspective. Agent selection and information disclosure interact: privacy processing changes the information available for reasoning, while response quality should affect subsequent participation. Studying their joint evolution is worthwhile.

Meaningful mechanism-level comparisons. Figure 2 includes ablations of the routing states, query refinement, and reliability dimensions, together with static, local-only, and one-way privacy-rule evolution. Figure 9c provides a particularly relevant comparison between bidirectional evolution and simpler alternatives. These experiments go beyond reporting aggregate performance alone.

Broad reported evaluation and explicit scope statements. The paper evaluates multiple reasoning benchmarks, privacy-sensitive tasks, adversarial settings, heterogeneity, and efficiency. It explicitly disclaims formal privacy guarantees, excludes policy poisoning from its threat model, and clarifies that evidence verifiability is a text-based judge assessment. These boundaries are appropriate for an empirical systems contribution.

Critical weaknesses

The distinct contribution is not sufficiently isolated from existing mechanisms. Multi-round RL routing and aggregation are already present in Router-R1, historical performance feedback is central to KABB, and iterative federated answer refinement is established by Fed-ICL. The contribution therefore rests primarily on the particular integration with personalized rule evolution. Table 3 should distinguish security-specific anomaly feedback from historical performance feedback when characterizing KABB. The existing ablations support individual components but do not clearly establish what the proposed coupling adds beyond combining these established mechanisms with a strong privacy filter.

The released training and inference paths do not implement the same method described in the paper. Section 3.2.2 assigns Fusion to the trainable orchestrator, identified as Qwen2.5-3B in Appendix C.2. In contrast, the default inference path uses a separate MiniMax-M2.5 aggregator (infer/run.py:164; infer/server.py:1284), while the training routing service uses Kimi-K2.5 (my/llm_agent/route_service.py:113). Moreover, the PPO routing service constructs clients afresh and does not maintain the explicit capability/reliability states or run the bidirectional rule-evolution loop used by the inference server. The query-replacement branch is disabled in the main generation loop. Appendix D also compares a client's current retained response with its own previous retained response, whereas infer/server.py:920–959 estimates contribution to changes in the global state using a frozen judge and updates capability values for all evaluated responses. These differences affect the core method and the attribution of its gains.

Privacy-rule effectiveness feedback is not independently validated, and its implementation differs from the stated update. In infer/LLMClient.py:145–157, recording a reported rule application increments both trigger_count and valid_count, without a separate check that enforcement was correct. This equates application with successful enforcement. Further, private_protect/server.py:236–238 updates the historical score as (previous_score + sum(current_scores)) / (1 + number_of_current_scores), rather than the cumulative mean in Appendix F, Equation 59. An isolated two-round example with feedback 1 and then 0 yields scores 0.50 and 0.25; the stated equation yields 1.00 and 0.50. These are material concerns for an approach whose claimed advantage depends on learning from rule effectiveness.

The released configurations and evaluation defaults do not substantiate the reported experimental setting. The supplied infer/expert_ability.json contains one expert using one backbone, rather than the 20 profiles and seven backbones in Section 4.1. With that configuration, the default round(N × adversarial_ratio) calculation selects zero attackers at all four reported ratios of 10–40%. The documented data preparation randomly allocates approximately 70% of math.parquet to training, while default inference evaluates the original complete math.parquet, creating train/evaluation overlap if the README workflow is followed. This establishes a problem in the released defaults, not that the paper necessarily used overlapping data. The manuscript also needs clearer sample counts, repeated-run counts, uncertainty definitions, and the exact configurations behind its tables.

Isolated execution exposes failures in the documented paths and a mismatch in the training objective. The unescaped JSON braces in the f-string inside infer/LLMClient.py:225–255 raise ValueError before the model call. The scoring template in infer/qa_em.py raises KeyError when formatted. The training environment requires a search action with at least two colons, whereas the format reward accepts exactly one (my/llm_agent/generation.py:644–649 versus verl/trainer/main_ppo.py:113–122). Finally, the progress reward uses the policy's self-reported draft_before rather than consistently using the previous aggregated state. Keeping two correct observations fixed and changing only the second self-reported draft from the correct answer to 'wrong' increases the computed reward from 1.5 to 2.5 (verl/utils/reward_score/qa_em.py:372–395). The existing logic tests do not exercise these execution paths. These findings require reconciliation with the actual experimental version.

Some privacy and robustness conclusions exceed the demonstrated boundaries. The soft utility comparison in Equations 10–11 does not ensure that adopted rules preserve every personalized constraint; aggregate improvements do not exclude regressions for particular clients. The leakage protocol in Appendix H does not sufficiently specify the detector or whether the evaluated information includes the server-visible rule candidates, feedback, and accumulated communications described in Appendix K. Appendix I primarily evaluates context-ignoring prompt injection, so broader claims about unreliable or malicious responses should remain qualified. I do not consider the absence of differential privacy or policy-poisoning experiments a defect, given the stated scope.

Questions for authors

Which exact code version generated the reported results? Please provide the corresponding revision, full expert/backbone configuration, trained checkpoint reference, benchmark sample identifiers and splits, number of runs, and definition of the reported uncertainty. Is the current one-expert configuration only a minimal example? How were nonzero attacker counts ensured in the reported experiments?

What components are actually trained and used during evaluation? Please reconcile the Qwen2.5-3B Fusion description with the MiniMax/Kimi aggregation calls. Were all baselines evaluated with the same aggregation model and comparable auxiliary-model budgets? How are the explicit routing states and bidirectional rule evolution incorporated into PPO trajectories?

Which capability-state definition is intended? Is improvement measured against the same client's last retained response, or against changes in global reasoning state? Are capability increments assigned to filtered responses, and are initial capability states zero or preset? Please also reconcile the reliability-update domain in main-text Equation 2 with Appendix D.

How is successful rule enforcement verified? Please explain why the released code increments valid_count whenever an application is recorded, and reconcile the historical-score implementation with Equation 59. What empirical evidence shows that global adaptation preserves local requirements, including clients whose policies conflict with a promoted rule?

Can you supply a minimal end-to-end execution trace from the experimental version? This should resolve the prompt-formatting errors, incompatible search-action formats, and self-reported-draft reward issue, while showing the actual state, risk, and rule updates. A representative existing benchmark case would be sufficient to clarify the implementation.

What exactly does the leakage evaluation observe? Please specify the evaluated communication channels, leakage detector, sensitive-item annotations, and any validation of detector accuracy. Does the measurement cover the full information available to the honest-but-curious server, or a restricted set of generated outputs?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 25eU (1/3)

Dear Reviewer 25eU,

Thank you very much for your valuable and constructive comments and questions. Below, we address them point by point. We have revised the manuscript and updated the code to address the identified issues. We will also release the complete code used for the reported experiments.

W1: Distinct Contributions Beyond Existing Mechanisms

  1. The key contribution of PFedMA lies in addressing reliable multi-round collaboration under federated data isolation and heterogeneous privacy requirements. Although Fed-ICL supports iterative federated answer refinement, it lacks historical reliability-aware client selection and adaptive privacy rule evolution. Meanwhile, centralized methods such as Router-R1 and KABB focus on dynamic routing and historical expert performance, respectively, but do not explicitly address client data isolation, unreliable responses, and heterogeneous privacy constraints in federated settings. In contrast, PFedMA integrates reliability-aware client selection, response filtering, and bidirectional privacy rule evolution within a federated framework. As shown in Table 1, PFedMA improves average task performance from 76.34 to 78.86 and reduces the leakage rate from 32.60 to 26.70 compared with Fed-ICL. Furthermore, Figures 2b and 2c validate the contributions of feedback-driven orchestration and privacy rule co-evolution, respectively.

  2. We have revised the paper and added the table below to further demonstrate the contributions of these components. Compared with Decoupled PFedMA, PFedMA improves MedQA accuracy from 75.71 to 77.91 and reduces LR from 30.05 to 26.70, demonstrating that cross-module feedback provides additional benefits beyond simply combining the two mechanisms.

MethodAdaptive RoutingRule EvolutionCross-module FeedbackMedQA ↑LR ↓
Static Routing + Static Rules×××61.1936.90
Adaptive Routing Only✓××71.4341.70
Rule Evolution Only×✓×62.5033.80
Decoupled PFedMA✓✓×75.7130.05
PFedMA✓✓✓77.9126.70

W2 & Q2: Implementation and Evaluation Clarification

  1. The Fusion operation in the main experiments is performed by the trainable Qwen2.5-3B orchestrator. MiniMax-M2.5 and Kimi-K2.5 serve as expert backbones within the heterogeneous expert pool. Their presence in the publicly released aggregation and routing scripts comes from an earlier fallback design intended to maintain system functionality when Qwen2.5-3B was unavailable.

  2. The previously released code provided only a partial implementation. We have updated the repository to include the multi-round collaboration pipeline and clarify the responsibilities of its components. Policy training is implemented in verl/trainer/main_ppo.py and verl/trainer/ppo/ray_trainer.py. The routing interface in my/llm_agent/route_service.py connects policy-generated actions to a persistent collaboration environment, which reuses infer/server.py for capability and reliability state updates and bidirectional privacy rule evolution across rounds.

  3. The query-replacement branch is disabled in the released code for the "w/o Query Update" ablation, rather than being absent from the full method. Query updating is enabled in the main experiments and disabled only for this ablation.

  4. The frozen judge evaluates responses without updating its parameters during training, so capability improvements do not come from training the judge itself. We also distinguish between historical capability and reliability. All evaluated responses update reliability records, including those filtered out, while capability records are updated only when a response passes the filter and shows improvement over a previous reliable response.

AuthorsOpenReview ↗

Response to Reviewer 25eU (2/3)

W3: Privacy Rule Effectiveness Feedback

  1. trigger_count represents the number of times a rule is selected and applied, while valid_count represents the number of times the rule execution process is successfully completed and produces a valid protection mapping. Here, "valid" refers to successful system-level execution rather than semantic correctness verified through independent annotation or external evaluation. Therefore, when a rule is successfully triggered and generates a protection result, both counters increase simultaneously.

  2. The released code only provides a simplified example, while our experiments use cumulative evaluation counts for the calculations. We have updated the code accordingly.

W4 & Q1: Experimental Configuration

  1. The released single-expert configuration is only a minimal example. The main experiments use 20 expert profiles across 7 backbone models.

  2. We did not use the default experimental settings provided in the example. Instead, we split the MATH dataset into 70% for reinforcement learning training and 30% for independent testing, with K-fold cross-validation to ensure reliable evaluation.

  3. Results are reported as mean ± standard deviation over multiple independent runs. All methods use consistent data splits, expert configurations, and evaluation protocols.

W5 & Q5: Implementation and Reward Mechanisms

  1. This issue may occur when an agent fails to generate outputs in the specified prompt format or when the parser fails to extract valid JSON. We encountered this issue previously, but it has not occurred in our recent experiments. It is possible that some outdated code was inadvertently included in the released version. We will fix these issues in the publicly released code.

  2. The two-colon check in my/llm_agent/generation.py is used for the internal routing request format, whereas the single-colon check in main_ppo.py is used for reward-side format parsing. These two checks serve different purposes rather than representing conflicting requirements within the same protocol. The experiments reported in the paper use a unified routing and reward configuration, in which internal routing requests are processed according to the experimental protocol before entering reward computation.

  3. In our experiments, draft_before is not freely self-reported by the policy. Instead, it is generated by the environment based on the aggregated state saved from the previous round and serves as a fixed baseline for reward computation. Therefore, modifying the text in the model's output does not affect quality_before and cannot yield the additional reward suggested in the reviewer's example.

W6: Privacy and Robustness Clarification

  1. Equations (10)–(11) do not force clients to adopt the globally optimal rule. Instead, each client compares the utility of global candidate rules with that of its local rules based on its own relevance and conflict statistics. A global rule is adopted only if it provides higher local utility; otherwise, the client retains its existing personalized rules. Therefore, PFedMA preserves client-specific requirements through a local rule retention and veto mechanism.

  2. The leakage protocol in Appendix H evaluates whether sensitive information remains in the generated outputs after privacy processing. Specifically, the detector identifies sensitive information leakage in the agent's response text, rather than performing inference attacks on the complete server-side communication records and privacy rules. The server-visible rule candidates, compact feedback, and accumulated communications described in Appendix K constitute protocol metadata rather than the output text evaluated in Appendix H. Therefore, they should not be directly treated as inputs to the same leakage metric.

Q3: Which Capability-State Definition Is Intended?

The initial capability scores are predefined rather than zero. As described in Appendix D, capability improvement is evaluated by comparing a client's current retained response with its previous retained response, rather than changes in the global reasoning state. Capability scores are updated only when the response passes the reliability filter, a previous retained response exists, and an improvement is observed. The first retained response serves only as a reference. Responses rejected by the reliability filter do not affect capability scores, but their anomaly scores still update the historical reliability state. Thus, capability updates reflect improvements in accepted responses, while reliability updates capture the risk history of all evaluated responses.

AuthorsOpenReview ↗

Response to Reviewer 25eU (3/3)

Q4: How Is Successful Rule Enforcement Verified?

  1. We use a client-side detection LLM to evaluate privacy rule enforcement and determine whether the generated outputs satisfy the corresponding privacy protection requirements. Specifically, valid_count records the number of times a rule is successfully executed and produces a valid protection mapping. Global rules are designed to provide reusable privacy knowledge rather than directly replace local rules. Each client can selectively adopt and adapt global rules according to its own privacy requirements, thereby preserving personalized constraints.

  2. As shown in Fig. 2(c), the paper compares full bidirectional rule evolution with Local Evolution Only (LE), Global-to-Local Adaptation Only (G2L), and Local-to-Global Aggregation Only (L2G). Together with the evaluation under heterogeneous client settings, these results provide indirect evidence supporting the effectiveness of this mechanism.

Q6: What Exactly Does the Leakage Evaluation Observe?

  1. In our setting, privacy rules specify information disclosure constraints without containing sensitive client data. Privacy leakage evaluation focuses solely on client-generated responses intended for upload to the server, rather than privacy rules or other communication content. We use an LLM-based detector to identify potential leakage based on each client's task context and personalized privacy requirements, rather than a uniform list of sensitive fields.

  2. We further adopted the Beyond Memorization approach [1] to evaluate privacy inference risks associated with shared privacy rules by simulating multi-round server-side privacy inference attacks on a medical dataset. The results show that PFedMA substantially reduces privacy inference risks compared with sharing raw privacy rules, achieving privacy protection close to the no-rule-sharing baseline.

Rule Sharing StrategySensitive Inference Accuracy ↓Sensitive Field Recovery EM ↓
No Rule Sharing0.290.24
Raw Local Privacy Rules0.500.35
PFedMA Rule Sharing0.300.24

[1] Beyond Memorization: Violating Privacy Via Inference with Large Language Models. ICLR 2024.

Reviewer_wiC4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper conducts research on appropriate agent selection and privacy optimization in a multi-agent framework in a federated environment, and proposes PFedMA. In this method, historical capabilities and reliable response information are used to guide agent selection for the problem of agent selection. To address privacy protection issues, bidirectional updates of privacy rules are used to achieve rule sharing and personalized adaptation between different clients. The experimental results on tasks such as mathematical reasoning, medical question answering, privacy protection, and adversarial attacks demonstrate the advantages of the method.

Critical strengths

The framework considers both collaboration effectiveness and privacy protection. And design from these two aspects. Firstly, the paper combines the dynamic selection of intelligent agents, response reliability evaluation, and privacy rule updates, and adjusts subsequent choices by referencing the agent's previous performance to minimize the impact of unreliable answers. Meanwhile, different clients can share useful privacy rules and adjust them according to their respective needs, making the entire collaboration process more flexible.

Critical weaknesses

The paper reduces sensitive information leakage by dynamically updating privacy rules and achieves good experimental results. However, during multiple rounds of collaboration, clients will continuously share processed answers and privacy rules, which may pose additional potential privacy risks as the number of interactions increases. The author needs to further analyze the privacy protection effect in this situation.

Questions for authors

Section 3.3 of the paper achieves cross client collaboration by sharing generalized local privacy rules. Although it avoids directly uploading complete local rules, the generalized rules may still contain semantic features related to sensitive information, which poses a potential risk of information leakage. The existing experiments mainly evaluate the privacy leakage situation in the generated output. Has the author considered the risk of servers inferring sensitive client information through shared generalization rules? Is there any relevant experiment that can verify the privacy and security of this sharing process?

3 comments
AuthorsOpenReview ↗

Response to Reviewer wiC4

Dear Reviewer wiC4

Thank you very much for your new comments and questions. We are happy to address them.

W1 & Q1: Does multi-round rule sharing suffer from privacy inference risks?

  1. First, PFedMA explicitly distinguishes privacy rules from raw private data held by clients. Privacy rules primarily define protection constraints governing information access, usage, and sharing, such as restricting the disclosure of specific categories of sensitive information, specifying the scope of information sharing under different collaboration contexts, and controlling the granularity of information disclosed by clients. Unlike sensitive data such as personal identities and medical records, privacy rules are intended to specify how information should be protected rather than to directly store or transmit the information being protected. Therefore, PFedMA enables cross-client privacy rule collaboration without requiring clients to upload their raw private data.

  2. Second, PFedMA mitigates potential privacy leakage risks through client-side rule generalization and server-side de-identification. As described in Section 3.3, clients retain complete personalized privacy rules locally and share only generalized rule representations, reducing client-specific details while preserving reusable protection constraints. Furthermore, as detailed in Appendix F, the server limits each client's contribution to one candidate per semantic cluster, removes explicit client identifiers, and randomly shuffles shared candidates. These mechanisms reduce the exposure of client-specific information and the direct association between shared rules and their originating clients.

  3. Third, we provide empirical evidence to evaluate potential privacy leakage risks. As shown in Table 1, PFedMA achieves a leakage rate of 26.70, compared with 32.60 for Fed-ICL, representing a reduction of 5.90 percentage points. These results demonstrate that PFedMA effectively reduces measured privacy leakage while maintaining strong privacy rule enforcement.

  4. Finally, we adopted the Beyond Memorization approach [1] to evaluate privacy inference risks associated with shared privacy rules by simulating multi-round server-side privacy inference attacks on a medical dataset. The results are reported in the table below in the revised manuscript. Compared with directly sharing complete local privacy rules, PFedMA reduces answer-option inference accuracy from 0.50 to 0.30 and sensitive-field recovery Exact Match (EM) from 0.35 to 0.24. These results indicate that rule generalization and de-identification can effectively mitigate the server's ability to indirectly infer sensitive client information from shared privacy rules under the evaluated experimental setting. [1]Beyond Memorization: Violating Privacy Via Inference with Large Language Models. ICLR 2024.

Rule Sharing StrategySensitive Inference Accuracy ↓Sensitive Field Recovery EM ↓
No Rule Sharing0.290.24
Raw Local Privacy Rules0.500.35
PFedMA0.300.24

Thank you for your response. Having reviewed the feedback, my concerns have been addressed, and I will maintain my rating.

AuthorsOpenReview ↗

Thank You to Reviewer wiC4

Thank you for your response. We sincerely appreciate your recognition and support. We are also deeply grateful for your insightful and constructive suggestions, which have provided valuable guidance for our future improvements.

Reviewer_WN1uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PFedMA trains a server LLM with RL to select client agents that keep their data local. The selection uses records of agent performance and reliability. An LLM judge filters responses before they are combined. Clients and the server exchange generalized privacy rules. The authors report an average task score above Fed-ICL and the proposed method has less data leakage and fewer successful attacks.

Critical strengths

This method is overall interesting. Clients exchange privacy rules adapted for reuse and adopt rules fitting their local requirements. This could share privacy knowledge without centralizing data if the design is validated.

Critical weaknesses
  1. There could be some claim-evidence misalignment. For example, the authors claim that PFedMA protects client data from inference by the server and preserves personalized client privacy rules. However, neither claim is tested directly. So the privacy contribution seems unsupported. Appendix H measures leakage on generated outputs. But there is no experiment which specifies how client requirements differ. And the released code gives every client the same default rules. The rule-enforcement metrics appear to come from Mind2Web-SC. The authors could simulate an inference attacker at the server and give clients distinct privacy requirements.
  2. It is appreciated that the authors provide the code for reproducibility and verification. After I double checked the code, it seems that the released code departs from the described protocol in ways that may inflate accuracy. Table 1 may not measure what the paper describes. In more details, a separate model with an expert backbone combines responses into the final answer, though Section 3.2.2 assigns this step to the server LLM. That model judges correctness using an unstated threshold. Scores of at least 0.5 count as correct. So one large model both writes the final answer and grades it. In addition, by default in the released code, inference covers the full MATH file, of which a random 70% has also been used for RL training. It would be great if the authors provide some clarifications.
  3. There may be some claim-evidence misalignment. Some key arguments may not be sufficiently supported. For example, the paper credits gains to revising agent selection across rounds within a task, but Figure 9b reports at most 1.45 average API calls per sample. So most queries may end after one call. The performance record starts at zero and increases only when an agent improves on its earlier answer (See Appendix D, eq. 33), so it should rarely change. Removing this record nevertheless lowers MATH accuracy by about 9.4 points (See Figure 2b). The released code initializes it with preset ability scores, so the ablation may remove prior knowledge about experts. It would be great if the authors provide some clarifications
  4. It seems that the comparison with the strongest single backbone and majority voting is missing. So it cannot show whether federated orchestration outperforms querying the best pool member. The Single Agent row appears to use the server LLM. The pool includes DeepSeek-R1, whose report (the paper's reference Guo et al., 2025) gives 79.8% accuracy for one sampled answer on AIME 2024 under its own protocol, above PFedMA's 76.67. Figure 6c shows Agent 11 selected often on every benchmark, and the agents' backbones are not reported.
Questions for authors
  1. How were client privacy requirements made to differ, who judged leakage What could the server infer from the uploaded responses and rule candidates?
  2. Which client count, data partition, per-round agent limit, and adversarial ratio produced Tables 1 and 2? Why does Figure 9d show GSM8K accuracy near 0.60?
  3. What accuracy does each backbone reach alone, which backbone does Agent 11 use? Were the learned baselines trained on the same splits and expert pool?
3 comments
AuthorsOpenReview ↗

Response to Reviewer WN1u (1/2)

Dear Reviewer WN1u

Thank you very much for your valuable and constructive comments and questions. Below, we address them point by point. We have updated the paper and code to reflect the changes.

W1 & Q1: Privacy Protection Clarification

  1. PFedMA allows each client to maintain its own privacy rules based on its local data and task-specific requirements. During initialization, all clients are initialized with the same default rule repository. As collaboration progresses, each client gradually develops a personalized rule repository based on its task-specific strengths by independently updating local rules and selectively adopting shared ones, rather than following a uniform policy. Furthermore, the rule enforcement metrics are not derived exclusively from Mind2Web-SC.

  2. Privacy leakage is assessed through an independent evaluation process based on client-specific privacy requirements and ground-truth sensitive information, rather than being determined by the server. Appendix H evaluates sensitive information disclosure in generated responses using the Leakage Rate (LR).

  3. We adopted the Beyond Memorization approach [1] to evaluate potential privacy leakage risks associated with sharing privacy rules by simulating multi-round server-side privacy inference attacks on a medical dataset. The results are reported in the table below in the revised manuscript.Compared with directly sharing raw privacy rules, PFedMA reduces Sensitive Inference Accuracy from 0.50 to 0.30 and Sensitive Field Recovery EM from 0.35 to 0.24, approaching the no rule sharing baseline.

[1] Beyond Memorization: Violating Privacy Via Inference with Large Language Models. ICLR 2024 |Rule Sharing Strategy|Sensitive Inference Accuracy ↓|Sensitive Field Recovery EM ↓| |---|---:|---:| |No Rule Sharing|0.29|0.24| |Raw Local Privacy Rules|0.50|0.35| |PFedMA|0.30|0.24|

W2: Implementation and Evaluation Clarification

  1. First, we clarify that PFedMA separates answer aggregation and evaluation into two independent stages. The trained coordinator integrates expert responses to generate the final answer, while a separate evaluation Agent assesses response quality. These two stages are handled by different models. Second, we designed a tiered scoring mechanism in the evaluation prompt: scores above 0.7 indicate correct final answers, scores between 0.5 and 0.7 indicate incorrect final answers with valid reasoning, and scores below 0.5 indicate errors in both. Importantly, in our multi-expert collaboration framework, valid intermediate reasoning steps can still contribute valuable information to subsequent expert collaboration and answer aggregation, even when the final answer is incorrect. Therefore, our scoring mechanism considers both final answer correctness and reasoning quality.

  2. We did not use the default experimental settings provided in the example. Instead, we split the MATH dataset into 70% for reinforcement learning training and 30% for independent testing, with K-fold cross-validation to ensure reliable evaluation.

W3:API call metrics

  1. The API call metric in Figure 9b is calculated as: CAPI=CtMNC_{API} = \frac{C_t}{MN}, where CtC_t is the total number of API calls, MM is the number of clients, and NN is the number of test samples. This metric measures Average API Calls per Client–Sample Pair, not the number of collaboration rounds, as multiple agents may be selected within a single round.

  2. The ablation in Figure 2b only disables historical state updates while retaining the predefined expert capability scores for initialization. Therefore, the 9.4 point accuracy drop reflects the contribution of online updates rather than the removal of expert priors. We clarify this setting in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer WN1u (2/2)

W4 & Q3: Lack of single baselines

  1. To isolate the contribution of the orchestration strategy, we have revised the manuscript by adding Table 4, which presents comparisons with the strongest single agent and majority voting under identical datasets, model pools, prompts, and sampling budgets. As shown in the table below, PFedMA consistently outperforms all single agents and majority voting across six benchmarks, demonstrating the effectiveness of its orchestration strategy.
MethodMATHGSM8KAIME24MedQAAlpacaEval 2.0MedMCQA
MiniMax2.581.0086.0176.2077.0070.3067.04
DeepSeek-V382.8084.3839.4073.5070.8064.00
Qwen2-72B-Instruct62.1086.0013.3360.7738.1063.30
GPT-4o75.9085.309.3076.1050.7050.76
DeepSeek-R183.2083.6174.8075.6976.7062.56
LLaMA-3-70B-Instruct51.0885.8031.6067.8038.1062.39
Kimi-K2.582.1079.5369.6076.4074.2065.31
Majority Voting82.5185.5771.1076.9073.6068.40
PFedMA84.5386.6076.6777.9177.6569.80
  1. Agent 11 uses Kimi-K2.5 as its backbone and is configured to specialize in mathematical reasoning, complex multi-skill problem solving, and medical knowledge. Its frequent selection is primarily driven by the alignment between these predefined capabilities and the evaluated tasks, rather than the backbone model alone.

  2. PFedMA fine-tunes only the central coordinator using LoRA, while expert models remain frozen. Training uses a consistent data split and expert pool.

  3. DeepSeek-R1's officially reported 79.8% accuracy and PFedMA's 76.67% accuracy on AIME 2024 are based on different model configurations and evaluation protocols, making a direct comparison inappropriate. Furthermore, under our current experimental setup, DeepSeek-R1 achieves an accuracy of 74.80%, indicating that directly using its officially reported result would not constitute a fair comparison. Moreover, PFedMA dynamically selects experts rather than relying solely on DeepSeek-R1.

Q2: Are the experimental settings clear and the results reasonable?

  1. Table 1 uses the same complete expert pool, consisting of 20 experts across 7 backbone models. It presents the main comparison of performance, privacy, and utility under attack-free conditions, with an adversarial client ratio of 0%. The minimum number of proxies per round is set to 1, while the maximum is set to 1/5 of the total number of clients. The data are evenly distributed across clients. Table 2 adopts the same expert pool, number of clients, data partitioning strategy, and maximum number of proxies per round, with the only difference being an adversarial client ratio of 15%.

  2. Figure 9d primarily investigates the impact of client data heterogeneity on performance, rather than reproducing the results in the main table. Specifically, the accuracy on GSM8K under the IID setting is 0.600, while the accuracy varies across different levels of heterogeneity and gradually approaches the IID performance as α increases. Since the heterogeneity experiments also simulate client differences by adjusting expert capability scores, their experimental configurations and routing outcomes differ from those in the main table. Therefore, the accuracy results are not directly comparable.

AuthorsOpenReview ↗

Official Comment by Authors

With the discussion period nearing its end, we would greatly appreciate your feedback on our responses to your recent questions. If you have any additional questions or comments, we would be more than happy to address them. If our responses have satisfactorily addressed your concerns, we would be grateful if you could consider revisiting your evaluation and potentially increasing your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.