- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PFedMA addresses multi-round collaboration among clients with private local information and heterogeneous privacy requirements. It combines historical capability and reliability states for client selection, structured intent/evidence/consensus evaluation before aggregation, feedback-guided query refinement, and bidirectional privacy-rule sharing with local adaptation. The paper reports improvements in reasoning performance, measured privacy leakage, and prompt-injection robustness. Its most promising idea is to treat reusable privacy knowledge as an evolving part of collaboration. However, substantial discrepancies between the paper and the released training/inference paths leave the main experimental claims insufficiently attributable to the proposed method.
Critical strengths
Relevant problem and useful perspective. Agent selection and information disclosure interact: privacy processing changes the information available for reasoning, while response quality should affect subsequent participation. Studying their joint evolution is worthwhile.
Meaningful mechanism-level comparisons. Figure 2 includes ablations of the routing states, query refinement, and reliability dimensions, together with static, local-only, and one-way privacy-rule evolution. Figure 9c provides a particularly relevant comparison between bidirectional evolution and simpler alternatives. These experiments go beyond reporting aggregate performance alone.
Broad reported evaluation and explicit scope statements. The paper evaluates multiple reasoning benchmarks, privacy-sensitive tasks, adversarial settings, heterogeneity, and efficiency. It explicitly disclaims formal privacy guarantees, excludes policy poisoning from its threat model, and clarifies that evidence verifiability is a text-based judge assessment. These boundaries are appropriate for an empirical systems contribution.
Critical weaknesses
The distinct contribution is not sufficiently isolated from existing mechanisms. Multi-round RL routing and aggregation are already present in Router-R1, historical performance feedback is central to KABB, and iterative federated answer refinement is established by Fed-ICL. The contribution therefore rests primarily on the particular integration with personalized rule evolution. Table 3 should distinguish security-specific anomaly feedback from historical performance feedback when characterizing KABB. The existing ablations support individual components but do not clearly establish what the proposed coupling adds beyond combining these established mechanisms with a strong privacy filter.
The released training and inference paths do not implement the same method described in the paper. Section 3.2.2 assigns Fusion to the trainable orchestrator, identified as Qwen2.5-3B in Appendix C.2. In contrast, the default inference path uses a separate MiniMax-M2.5 aggregator (infer/run.py:164; infer/server.py:1284), while the training routing service uses Kimi-K2.5 (my/llm_agent/route_service.py:113). Moreover, the PPO routing service constructs clients afresh and does not maintain the explicit capability/reliability states or run the bidirectional rule-evolution loop used by the inference server. The query-replacement branch is disabled in the main generation loop. Appendix D also compares a client's current retained response with its own previous retained response, whereas infer/server.py:920–959 estimates contribution to changes in the global state using a frozen judge and updates capability values for all evaluated responses. These differences affect the core method and the attribution of its gains.
Privacy-rule effectiveness feedback is not independently validated, and its implementation differs from the stated update. In infer/LLMClient.py:145–157, recording a reported rule application increments both trigger_count and valid_count, without a separate check that enforcement was correct. This equates application with successful enforcement. Further, private_protect/server.py:236–238 updates the historical score as (previous_score + sum(current_scores)) / (1 + number_of_current_scores), rather than the cumulative mean in Appendix F, Equation 59. An isolated two-round example with feedback 1 and then 0 yields scores 0.50 and 0.25; the stated equation yields 1.00 and 0.50. These are material concerns for an approach whose claimed advantage depends on learning from rule effectiveness.
The released configurations and evaluation defaults do not substantiate the reported experimental setting. The supplied infer/expert_ability.json contains one expert using one backbone, rather than the 20 profiles and seven backbones in Section 4.1. With that configuration, the default round(N × adversarial_ratio) calculation selects zero attackers at all four reported ratios of 10–40%. The documented data preparation randomly allocates approximately 70% of math.parquet to training, while default inference evaluates the original complete math.parquet, creating train/evaluation overlap if the README workflow is followed. This establishes a problem in the released defaults, not that the paper necessarily used overlapping data. The manuscript also needs clearer sample counts, repeated-run counts, uncertainty definitions, and the exact configurations behind its tables.
Isolated execution exposes failures in the documented paths and a mismatch in the training objective. The unescaped JSON braces in the f-string inside infer/LLMClient.py:225–255 raise ValueError before the model call. The scoring template in infer/qa_em.py raises KeyError when formatted. The training environment requires a search action with at least two colons, whereas the format reward accepts exactly one (my/llm_agent/generation.py:644–649 versus verl/trainer/main_ppo.py:113–122). Finally, the progress reward uses the policy's self-reported draft_before rather than consistently using the previous aggregated state. Keeping two correct observations fixed and changing only the second self-reported draft from the correct answer to 'wrong' increases the computed reward from 1.5 to 2.5 (verl/utils/reward_score/qa_em.py:372–395). The existing logic tests do not exercise these execution paths. These findings require reconciliation with the actual experimental version.
Some privacy and robustness conclusions exceed the demonstrated boundaries. The soft utility comparison in Equations 10–11 does not ensure that adopted rules preserve every personalized constraint; aggregate improvements do not exclude regressions for particular clients. The leakage protocol in Appendix H does not sufficiently specify the detector or whether the evaluated information includes the server-visible rule candidates, feedback, and accumulated communications described in Appendix K. Appendix I primarily evaluates context-ignoring prompt injection, so broader claims about unreliable or malicious responses should remain qualified. I do not consider the absence of differential privacy or policy-poisoning experiments a defect, given the stated scope.
Questions for authors
Which exact code version generated the reported results? Please provide the corresponding revision, full expert/backbone configuration, trained checkpoint reference, benchmark sample identifiers and splits, number of runs, and definition of the reported uncertainty. Is the current one-expert configuration only a minimal example? How were nonzero attacker counts ensured in the reported experiments?
What components are actually trained and used during evaluation? Please reconcile the Qwen2.5-3B Fusion description with the MiniMax/Kimi aggregation calls. Were all baselines evaluated with the same aggregation model and comparable auxiliary-model budgets? How are the explicit routing states and bidirectional rule evolution incorporated into PPO trajectories?
Which capability-state definition is intended? Is improvement measured against the same client's last retained response, or against changes in global reasoning state? Are capability increments assigned to filtered responses, and are initial capability states zero or preset? Please also reconcile the reliability-update domain in main-text Equation 2 with Appendix D.
How is successful rule enforcement verified? Please explain why the released code increments valid_count whenever an application is recorded, and reconcile the historical-score implementation with Equation 59. What empirical evidence shows that global adaptation preserves local requirements, including clients whose policies conflict with a promoted rule?
Can you supply a minimal end-to-end execution trace from the experimental version? This should resolve the prompt-formatting errors, incompatible search-action formats, and self-reported-draft reward issue, while showing the actual state, risk, and rule updates. A representative existing benchmark case would be sufficient to clarify the implementation.
What exactly does the leakage evaluation observe? Please specify the evaluated communication channels, leakage detector, sensitive-item annotations, and any validation of detector accuracy. Does the measurement cover the full information available to the honest-but-curious server, or a restricted set of generated outputs?
7 comments
Response to Reviewer 25eU (2/3)
W3: Privacy Rule Effectiveness Feedback
trigger_count represents the number of times a rule is selected and applied, while valid_count represents the number of times the rule execution process is successfully completed and produces a valid protection mapping. Here, "valid" refers to successful system-level execution rather than semantic correctness verified through independent annotation or external evaluation. Therefore, when a rule is successfully triggered and generates a protection result, both counters increase simultaneously.
The released code only provides a simplified example, while our experiments use cumulative evaluation counts for the calculations. We have updated the code accordingly.
W4 & Q1: Experimental Configuration
The released single-expert configuration is only a minimal example. The main experiments use 20 expert profiles across 7 backbone models.
We did not use the default experimental settings provided in the example. Instead, we split the MATH dataset into 70% for reinforcement learning training and 30% for independent testing, with K-fold cross-validation to ensure reliable evaluation.
Results are reported as mean ± standard deviation over multiple independent runs. All methods use consistent data splits, expert configurations, and evaluation protocols.
W5 & Q5: Implementation and Reward Mechanisms
This issue may occur when an agent fails to generate outputs in the specified prompt format or when the parser fails to extract valid JSON. We encountered this issue previously, but it has not occurred in our recent experiments. It is possible that some outdated code was inadvertently included in the released version. We will fix these issues in the publicly released code.
The two-colon check in
my/llm_agent/generation.pyis used for the internal routing request format, whereas the single-colon check inmain_ppo.pyis used for reward-side format parsing. These two checks serve different purposes rather than representing conflicting requirements within the same protocol. The experiments reported in the paper use a unified routing and reward configuration, in which internal routing requests are processed according to the experimental protocol before entering reward computation.In our experiments,
draft_beforeis not freely self-reported by the policy. Instead, it is generated by the environment based on the aggregated state saved from the previous round and serves as a fixed baseline for reward computation. Therefore, modifying the text in the model's output does not affectquality_beforeand cannot yield the additional reward suggested in the reviewer's example.
W6: Privacy and Robustness Clarification
Equations (10)–(11) do not force clients to adopt the globally optimal rule. Instead, each client compares the utility of global candidate rules with that of its local rules based on its own relevance and conflict statistics. A global rule is adopted only if it provides higher local utility; otherwise, the client retains its existing personalized rules. Therefore, PFedMA preserves client-specific requirements through a local rule retention and veto mechanism.
The leakage protocol in Appendix H evaluates whether sensitive information remains in the generated outputs after privacy processing. Specifically, the detector identifies sensitive information leakage in the agent's response text, rather than performing inference attacks on the complete server-side communication records and privacy rules. The server-visible rule candidates, compact feedback, and accumulated communications described in Appendix K constitute protocol metadata rather than the output text evaluated in Appendix H. Therefore, they should not be directly treated as inputs to the same leakage metric.
Q3: Which Capability-State Definition Is Intended?
The initial capability scores are predefined rather than zero. As described in Appendix D, capability improvement is evaluated by comparing a client's current retained response with its previous retained response, rather than changes in the global reasoning state. Capability scores are updated only when the response passes the reliability filter, a previous retained response exists, and an improvement is observed. The first retained response serves only as a reference. Responses rejected by the reliability filter do not affect capability scores, but their anomaly scores still update the historical reliability state. Thus, capability updates reflect improvements in accepted responses, while reliability updates capture the risk history of all evaluated responses.
Response to Reviewer 25eU (3/3)
Q4: How Is Successful Rule Enforcement Verified?
We use a client-side detection LLM to evaluate privacy rule enforcement and determine whether the generated outputs satisfy the corresponding privacy protection requirements. Specifically,
valid_countrecords the number of times a rule is successfully executed and produces a valid protection mapping. Global rules are designed to provide reusable privacy knowledge rather than directly replace local rules. Each client can selectively adopt and adapt global rules according to its own privacy requirements, thereby preserving personalized constraints.As shown in Fig. 2(c), the paper compares full bidirectional rule evolution with Local Evolution Only (LE), Global-to-Local Adaptation Only (G2L), and Local-to-Global Aggregation Only (L2G). Together with the evaluation under heterogeneous client settings, these results provide indirect evidence supporting the effectiveness of this mechanism.
Q6: What Exactly Does the Leakage Evaluation Observe?
In our setting, privacy rules specify information disclosure constraints without containing sensitive client data. Privacy leakage evaluation focuses solely on client-generated responses intended for upload to the server, rather than privacy rules or other communication content. We use an LLM-based detector to identify potential leakage based on each client's task context and personalized privacy requirements, rather than a uniform list of sensitive fields.
We further adopted the Beyond Memorization approach [1] to evaluate privacy inference risks associated with shared privacy rules by simulating multi-round server-side privacy inference attacks on a medical dataset. The results show that PFedMA substantially reduces privacy inference risks compared with sharing raw privacy rules, achieving privacy protection close to the no-rule-sharing baseline.
Rule Sharing Strategy Sensitive Inference Accuracy ↓ Sensitive Field Recovery EM ↓ No Rule Sharing 0.29 0.24 Raw Local Privacy Rules 0.50 0.35 PFedMA Rule Sharing 0.30 0.24 [1] Beyond Memorization: Violating Privacy Via Inference with Large Language Models. ICLR 2024.
Response to Reviewer 25eU (1/3)
Dear Reviewer 25eU,
Thank you very much for your valuable and constructive comments and questions. Below, we address them point by point. We have revised the manuscript and updated the code to address the identified issues. We will also release the complete code used for the reported experiments.
W1: Distinct Contributions Beyond Existing Mechanisms
W2 & Q2: Implementation and Evaluation Clarification