- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FedRuleMem proposes a federated memory system for frozen LLM agents that exchanges executable action-precondition rules instead of trajectories or model updates. Clients synthesize rules from local failures, check executability and consistency with observed outcomes, and prune them using a greedy MDL objective. The server performs exact and comment-based semantic deduplication before broadcasting a shared library. During interaction, rules provide feedback for bounded action revision, with an exploration bypass.
The paper evaluates the system on Crafter and ALFWorld over nine independent end-to-end runs. Compared with the matched FedTextRule control, it reports improvements from 11.4 to 13.5 in Crafter Score and from 62.4% to 64.8% in ALFWorld success. Additional experiments examine efficiency, component contributions, model sensitivity, client heterogeneity, cross-client transfer, and injected conflicts.
Critical strengths
The executable memory interface is concrete and useful for frozen agents. Rules provide explicit action checks and localized corrective feedback without requiring parameter updates or repeatedly interpreting long trajectory histories. The comparison with FedTextRule matches federation, MDL filtering, and the correction budget, making it more informative than comparisons against no-memory agents alone. On Crafter, the reported end-to-end token cost also decreases from 69.6k to 61.9k per task.
The evaluation includes meaningful controls for the benefits of sharing and verification. The paper reports nine complete runs, separately ablates behavioral verification, MDL, and deduplication, and tests heterogeneous client distributions. The experiment withholding a task family from one recipient improves success from 44.2% with Local Rules to 58.6% using rules learned by other clients, providing direct evidence of cross-client transfer. These experiments support the usefulness of the overall pipeline, even though they do not fully resolve the reliability concerns below.
Critical weaknesses
The aggregation and recipient-side rejection mechanisms do not adequately establish reliable reuse across heterogeneous clients. Equation (11) clusters rules using embeddings of their comments and retains the rule with the longest comment. Similar descriptions do not establish equivalent executable predicates, so consolidation can discard complementary conditions or retain a less appropriate representative. Appendix E.2 acknowledges this possibility, but the evaluation does not directly quantify coverage lost or false rejections introduced by semantic merging. Moreover, Equation (1) and Algorithm 1 replace every client's acting library with the global broadcast. A locally rejected rule can therefore return if another client retains and uploads it; the manuscript does not specify a persistent recipient-specific veto. Table 12 demonstrates mitigation of injected conflicts, but does not sufficiently explain whether recurring, client-dependent conflicts survive rebroadcast. This is consequential because controlling the propagation of incorrect rules is a central motivation for the federated design.
The evidence for reusable rule validity remains limited by the learning and evaluation protocol. New rules are synthesized from current-round failures and checked against positives and failures from the same round. Passing this gate establishes consistency with observed evidence, rather than correctness on unseen states. Appendix G also states that all 134 ALFWorld tasks are evaluated each round and redistributed across clients. This is a reasonable online-adaptation protocol, but redistribution does not create a globally untouched evaluation set. The recipient-held-out-family experiment supports transfer between clients; it does not by itself establish generalization to task instances unseen by every client. Independent measurements of rule false-rejection rates and failure coverage would help distinguish reusable preconditions from constraints fitted to repeatedly encountered tasks. Given the modest improvement over the closest textual control, these unresolved questions limit the strength of the contribution and motivate my weak-rejection recommendation.
Questions for authors
-
Recipient-side rejection: If client A rejects a rule but client B retains it, does the next broadcast cause A to execute that rule again? Is there a persistent local rejection list or applicability mask? Please explain how Table 12 handles repeated rebroadcast of rules that are valid for some clients but invalid for others.
-
Semantic consolidation: What changes in failure coverage and false-rejection rate occur before and after semantic deduplication? Is there evidence that choosing the longest-comment representative preserves the behavior of discarded rules? The exact-only ablation measures aggregate performance, but does not directly establish merge correctness.
-
Generalization protocol: Are there results with a frozen learned library evaluated on task instances or Crafter seeds unseen by all clients during synthesis and verification? For the recipient-held-out-family experiment, please specify whether recipient evaluation outcomes subsequently enter rule learning or validation, and whether the reported result measures the first exposure.
5 comments
Rebuttal to Reviewer 7N4t (2/3)
Q2. (a) What changes in failure coverage and false-rejection rate occur before and after semantic deduplication? (b) Is there evidence that the longest-comment representative preserves discarded rules' behavior? (Critical Weakness, paragraph 1; supplementary evidence: library size and task performance)
(a) Diagnostic changes. Semantic selection reduces false rejection of successful actions from 1.5% to 1.2% while retaining 87.5% failure coverage across source and recipient contexts. Both libraries use identical independent ALFWorld pairs from episodes/configurations excluded from synthesis, selection, and tuning. Labels come from environment execution, including actions the rules would block.
Rebuttal Table R10. Independent behavioral diagnostics of semantic consolidation.
Source: New ALFWorld diagnostic; mean ± sample standard deviation across nine learned libraries. “Before” follows exact deduplication. Each run uses 1,000 failed and 2,000 successful actions, including 500 failed and 1,000 successful recipient actions.
| Evaluation scope | Metric (%) | Before semantic selection | After semantic selection |
|---|---|---|---|
| Source + recipient | Failure coverage ↑ | 89.4 ± 2.2 | 87.5 ± 2.3 |
| Source + recipient | False acceptance of incorrect actions ↓ | 10.6 ± 2.2 | 12.5 ± 2.3 |
| Source + recipient | False rejection of successful actions ↓ | 1.5 ± 0.6 | 1.2 ± 0.5 |
| Recipient only | Failure coverage ↑ | 86.8 ± 2.8 | 84.4 ± 3.0 |
| Recipient only | False rejection of successful actions ↓ | 2.0 ± 0.8 | 1.6 ± 0.7 |
Any rejecting rule constitutes library rejection. Coverage and false acceptance partition failed actions; false rejection uses successful actions. Source + recipient counts are pooled within each run.
(b) Behavioral retention. Coverage decreases by 1.9 points and false rejection by 0.3 points; recipient-only decreases are 2.4 and 0.4 points, respectively (Rebuttal Table R10). This quantifies the tradeoff of the longest-comment heuristic, which selects one existing rule per action-conditioned cluster.
The separate end-to-end ablation gives a 32.2% smaller mean library with higher mean task success: exact-only versus full aggregation yields versus rules and versus ALFWorld success (paper Table 6). These complete runs complement the fixed-pool behavioral diagnostic.
Q3. (a) Are there frozen results on tasks or Crafter seeds unseen by all clients? (b) Was recipient-held-out evaluation first exposure? (c) Did its feedback later enter learning? (Critical Weakness, paragraph 2: generalization protocol)
(a) Original FedRuleMem achieves 63.7 ± 1.3% on globally unseen ALFWorld configurations, exceeding Local Rules by 6.4 points and original FedTextRule by 6.3 (Rebuttal Table R11). All 134 test instances and their 47 configurations are excluded from every client's construction data. We learn with 16 clients, 10 rounds, and 134 attempts per round, then freeze memory; test feedback supports only the current episode and never enters learning, verification, or cross-task updates.
Rebuttal Table R11. New configuration-disjoint frozen ALFWorld experiment.
| Method | Success (%) |
|---|---|
| Local Rules | 57.3 ± 1.9 |
| Original FedTextRule | 57.4 ± 1.6 |
| Original FedRuleMem | 63.7 ± 1.3 |
Our original repeated-attempt ALFWorld setting follows an established paradigm used by Reflexion, Retroformer, QuBE with Reflexion, and CDMem. The new experiment additionally tests globally unseen configurations.
(b) Yes: the earlier result measures A's first evaluation batch on a held-out family. A excludes that family in rounds 1–9; other clients can learn it. In round 10, the round-9 broadcast yields 58.6 ± 1.6%, versus 44.2 ± 2.4% with A's local memory from other families (paper Appendix B.4). Memory remains fixed across the batch; feedback is used within episodes.
(c) Only after the whole batch is scored do its trajectories enter cumulative evidence for round-10 learning and verification.
Critical Weakness, paragraph 2: independent rule validity
Independent diagnostics give 87.5% failure coverage / 1.2% successful-action false rejection, and 84.4% / 1.6% on recipients (Rebuttal Table R10). Online verification uses cumulative evidence including synthesis examples; the diagnostic pairs exclude synthesis, selection, and tuning episodes/configurations. Together with frozen transfer (Rebuttal Table R11), these results assess validity beyond learning evidence.
Rebuttal to Reviewer 7N4t (3/3)
Critical Weakness, paragraph 2: gains over the textual control
The new verified-text control confirms gains of 1.9 Crafter Score points and 3.2 ALFWorld points: FedRuleMem scores 13.5 / 64.8% (paper Table 1), versus 11.6 / 61.6% in the new rebuttal experiment. Both gates require cumulative failure coverage and no rejection of observed successful actions. Crafter cost per task is 61.9k tokens / 60.8 seconds (paper Table 16), versus 84.6k / 84.0 for new verified text: 26.8% fewer tokens and 27.6% less time in complete runs.
Rebuttal to Reviewer 7N4t (1/3)
Thank you for the constructive review. Tables R9–R11 below are rebuttal tables. Results are mean ± sample standard deviation over nine runs. Manuscript revisions are blue.
Q1. (a) Can rejected rules return? (b) Is there a persistent rejection list or mask? (c) How does the paper's Table 12 handle repeated rebroadcast? (Critical Weakness, paragraph 1; supplementary evidence: sustained-conflict experiment)
(a) Rule return. A rule rejected by A can return through B's upload. Original FedRuleMem revalidates inherited rules against cumulative evidence after interaction and before upload. This raises affected-recipient success from 49.2% to 58.9% under recurring conflicts, a 9.7-point gain (Rebuttal Table R9).
(b) Persistent masking. The original method has no persistent pre-use mask; interaction uses bounded regeneration and the ϵ bypass. Without this extension, it already achieves 64.8% main ALFWorld success and 63.7% frozen-transfer success (paper Table 1; Rebuttal Table R11).
(c) Protocol of the paper's Table 12. At 30% injection, original FedRuleMem achieves 61.2±1.6% versus 50.4±2.5% without recipient revalidation (paper Table 12). Conflicts are injected once at the end of round 8, after server deduplication and before broadcast. Clients use the perturbed library in round 9, then revalidate inherited rules against cumulative evidence before upload and aggregation. “Rejected at next local gate” measures this check, which the control omits. Round-10 scoring uses the round-9 broadcast, pooling all 134 attempts across 16 clients.
Supplementary evidence for (a)–(c): sustained conflicts. In the new experiment, fixed-identity conflicts repeatedly replace approximately 30% of a source client's unique canonicalized upload rules in rounds 2–10, preserving pool size. Injection precedes server deduplication; the rules are valid in designated source scenarios and invalid in designated recipient scenarios.
Rebuttal Table R9. Recipient performance under repeated rebroadcast of conflicting rules.
Each run pools the actual round-10 attempts of eight affected recipients (approximately 67), using the round-9 broadcast.
The persistent-exclusion extension blocks rules contradicted by successful local outcomes on later broadcasts. Identity uses action type, schema version, and canonicalized code; modified programs are rechecked against stored counterexamples.
Persistent exclusion further raises success to 62.5%, adding 3.6 points (Rebuttal Table R9).