- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes LBF-Gate, which organizes historical failure experiences into structured rules, matches them to current tasks using predefined conditions, and provides the matched rules to an LLM as decision constraints. Experiments show that the method outperforms standard structured memory and free-text reflection on two language models.
Critical strengths
- The paper conducts a series of experiments and is clearly written and easy to follow.
Critical weaknesses
W1. LBF-Gate essentially performs rule matching based on predefined conditions and prompt-level constraint injection, without introducing a fundamentally new learning or reasoning algorithm. And it generalization maybe a main concern.
W2. Experiments are restricted to a synthetic rescue environment and relatively small memory banks. The Gemma experiments do not demonstrate an advantage of LBF-Gate over standard structured memory. Moreover, the additional benefit of gating nearly disappears in the ANCHOR setting, where the relevant experience is explicitly identified.
Questions for authors
- Can you provide sufficient comparisons with more competitive experience-selection methods, such as LLM-based selection, dense retrieval, and learned reranking.
6 comments
Thank you for the additional experiments and clarifications. However, my main concerns remain unresolved.
The authors explicitly acknowledge that the evaluation is restricted to a synthetic rescue environment, small memory banks (8/16 records), predefined symbolic applicability predicates, and exactly one matching lesson per evaluated case. Generalization to realistic settings, uncertain applicability, and multiple matching experiences remains untested. These limitations directly affect the generality and practical significance of the proposed method.
Furthermore, the authors have not evaluated dense retrieval or learned reranking, which are important competitive baselines for the proposed experience-selection mechanism. The additional LLM-based selection experiment provides useful evidence, but does not sufficiently address this gap. The Gemma results also fail to demonstrate an advantage over standard typed memory.
I appreciate the authors' efforts in expanding the analysis. However, these limitations concern the fundamental validation and generalizability of the proposed approach. Addressing them would require substantial additional experiments rather than further explanations or reinterpretations of the existing results.
In my view, the current evidence remains insufficient to meet the acceptance threshold. I therefore maintain my original score.
Thank you for taking the time to reconsider our response.
We respect your concerns, but hope you can also appreciate the scope of a focused conference paper. We have conducted extensive controlled experiments addressing our research questions. Broader generalization, experience retrieval, and multiple matching lessons are important research directions, but each could warrant a substantial investigation of its own. Relevant foundations, including SagaLLM, Mnemosyne, TRW-Trace, and the AGI Trilogy, have also been independently documented.
We have made substantial efforts to clarify and strengthen the paper within the conference page limit. If a thorough study of a focused question is expected to resolve several additional research problems, perhaps the community should reconsider what can reasonably fit into a conference paper.
We recognize that you may still recommend rejection. However, receiving scores of 2, 1, 1, 1 across Presentation, Informativeness, Soundness, and Overall, despite the extensive experiments, revisions, and additional evidence, is deeply discouraging. Such scores convey an impression that the work has almost no scientific merit. We respectfully believe that this assessment is disproportionately severe.
We are not asking you to overlook the paper's limitations. We simply hope that the scientific effort, demonstrated findings, and potential for further research can receive some recognition and encouragement, even if you ultimately maintain your recommendation. A critical review can still acknowledge the value of a scientific contribution.
Thank you again for your consideration.
Author Response
Thank you for challenging the novelty and generalization of the work and for requesting a stronger selection baseline. The simple matching implementation was prominent in our earlier presentation, while the scientific question it isolates was not. We understand why that made the contribution difficult to distinguish from ordinary rule matching.
W1: Contribution beyond matching. The match operation is intentionally transparent: holding the acquired lesson bank and model weights fixed lets us isolate a question that memory retrieval alone cannot answer: when does the same remembered experience produce a correct, feasible action that prevents harm before a deadline? Memory-to-action benchmarks such as Mem2ActBench evaluate memory-guided tool choice and parameter grounding. LBF measures the downstream preventive outcome and the failure stages in between. Figure 2 demonstrates why that additional endpoint matters. Claude's gated planner chooses 4,477 correct actions, more than GPT's 4,333, yet prevents only 3,653 events versus GPT's 4,316; 788 correct, validly scheduled Claude actions finish late. Action accuracy would miss that reversal. Wrong-mark controls reveal another failure mode: misleading guidance often triggers costly escalation and delay instead of immediate execution of an incorrect correction.
The requested competitive selector is now tested. Section 5.8/Table 3 evaluates a two-stage LLM selector that sees the full lesson bank and public observations with no match hint. Its chosen lesson is passed unchanged to the planner; the extra model call is not charged to simulated deadlines. Over 250 registered paired blocks per class/model, GPT's selector identifies the applicable record in 84.56% of usable cases. Its loss is 0.11769 versus 0.08824 with gating (gain 0.02945; prespecified D10 lower bound 0.00213; protected checks pass). Claude self-selects correctly in 99.53% and essentially ties gating (0.15991 versus 0.15997). This model contrast is a finding: the value of external applicability help depends on the planner's existing selection ability.
We also compare BM25 top-four lexical retrieval on matched descriptive panels, where gating has lower loss by 0.0132 (GPT) and 0.0274 (Claude). BM25 changes the readable set and therefore is not equivalent to the full-bank comparison. Dense retrieval and learned reranking are not evaluated (Sec. 5.7; Conclusion); those remain stronger future baselines.
W2: Why ANCHOR and Gemma are informative. We recognize why the near-zero ANCHOR gain looked like a weak result without the missing premise: ANCHOR already supplies the identified lesson to every arm. Once source identification is solved, the gate is predicted to add little; its descriptive advantage falls to 0.0003 on GPT and 0.0050 on Claude. Gemma demonstrates why solving a prior stage does not guarantee prevention: gating increases correct actions from 1,315 to 1,343, but direct preventions decline from 1,024 to 1,009, with more invalid schedules (73 to 115 among unsuccessful correct actions). This points to schedule feasibility as a specific model-interface improvement target, not a proven repair.
Our constructed rescue task, eight-/sixteen-record banks, supplied symbolic predicates, and unique-match cases make the controlled comparisons interpretable and provide private prevention deadlines. They do not establish independent-domain transfer, multiple-match robustness, or superiority over learned retrieval. We now distinguish those future tests from what the completed evidence supports. We hope the revised paper makes the evaluation and diagnosis of preventive competence clear enough to assess on its scientific merits.