- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose GovernMem, a framework that aims to improve memory repair by learning from past experiences. A fast loop repairs memory, and a slow loop learns from this experience, and guides how memories are repaired in the future.
Critical strengths
(1) The research question is timely and relevant: Persisting errors can cause failure, and being able to reliably identify and fix memory failures could address this failure mode and meaningfully advance the state of the art.
Critical weaknesses
(1) My main concern is with the experimental setup. The authors write:
The results show that agents make better repair choices when they reuse past repair experience. Compared with starting without prior experience, using repair experience collected by another model on the same cases and in the same order improves repair utility by 6.39 points on MetaLlama-3.1-8B-Instruct and 6.63 points on Qwen3-14B.
If I understand correctly, the authors employed the same cases to gather repair experience and measure whether repair experience helps. If this is indeed the case, it would limit the applicability of the proposed solution significantly. It would be unclear if the method "recalls" specific failure cases (overfitting), or generalizes properly.
(2) The authors do not adequately discuss the limitations of their method.
Questions for authors
(1) Can the authors please clarify whether their experimental results yield any evidence how their method generalizes beyond the training manifold? It the experimental results do not yield any such evidence, could the authors please (a) include experiments that show that their method generalizes, or (b) clarify why there is a legitimate use case for their method regardless, and re-scope their writing to reflect this premise?
(2) Could the authors please share their code to ensure full reproducibility?
(3) Can the authors please include a dedicated Limitations section?
(4) Figure 3 is somewhat difficult to interpret. Could the authors enhance the visibility of the MemFail results?
3 comments
Thank you for the response and the additional experiments. Unfortunately, they do not resolve my main concerns.
Generalization. The new result compares repair with no repair. The experiments do not test whether the learned experience transfers beyond the cases it was collected on. To show generalization, the experience would need to be collected on some cases and evaluated on held-out ones.
Code. The code is still not available. I can only assess what is available during the review, not planned releases.
I will therefore maintain my score.
Dear Reviewer,
Thank you again for your careful review and constructive feedback.
We have posted our responses and revised the manuscript to address the concerns you raised. As the discussion period draws to a close, we would greatly appreciate your thoughts on whether our revisions have adequately addressed your main concerns.
We would be happy to clarify any remaining points.
Thank you again for your time and consideration!
Q1: Generalization
We really apologize for not describing the transfer procedure clearly enough, which makes you confused. The downstream system was initialized with the saved GovernRouter posterior and skill library accumulated using Qwen2.5 in the original experiments. We kept this source state frozen during downstream evaluation with Qwen3. The downstream experiments therefore evaluate a system using previously accumulated repair experience.
Q2: Code availability
Our source code is now publicly available on GitHub for inspection during the review period. The current release includes the core implementation, selected evaluation scripts, and experiment manifests and does not yet include some subsequent commits from our local development branch. These files expose the repair workflow, routing updates, and experience-transfer procedure. We are organizing the remaining experimental artifacts and will add them to the repository.
Thank you for your constructive comments. We appreciate your concerns about the experimental setup and generalization claim. We have conducted additional evaluations and will clarify the scope of our claims and improve the presentation. We believe our response has addressed most of your concerns and ask if you're comfortable to re-consider the rating in light of our efforts and dedication.
Q1: Concerns about the experimental setup and generalization.
Indeed you make it right that the original experience-reuse experiments use the same cases for source experience collection and target evaluation. This protocol measures cross-model experience reuse on shared cases. We may not have made this distinction clear enough.
So we add downstream evaluations on PersonaMem and LongMemEval-derived QA cohorts, keeping source experience frozen during target evaluation. On the 240-question controlled memory-fault evaluation, repair improves QA accuracy from 32.08% to 65.00%, a gain of 32.92 percentage points with a paired 95% confidence interval of [26.25, 39.58]. These results demonstrate downstream repair value.
Q2: Figure 3 is difficult to interpret, particularly the MemFail results.
Thank you for pointing this out. In version 1, we reduced the figure size to meet the page limit while trying to preserve the visibility of the results. We will expand the caption and accompanying discussion to explain the metrics, aggregation procedure, and MemFail results more clearly, making the figure easier to interpret.
Q3: Insufficient discussion of limitations.
We will add a dedicated Limitations section in Appendix. We will also align the abstract and conclusions with these evaluated settings, so that the scope is clear throughout the paper.
Q4: Code availability and reproducibility.
We will publicly release the code and experimental artifacts after the rebuttal period. The release will include evaluation scripts, processed evaluation data and preparation scripts, case and family manifests, configurations, random seeds, and the routing snapshots required for the reported comparisons.
We will document the commands and dependencies needed to reproduce each table, including the utility definitions, aggregation rules, and confidence-interval procedures.
We sincerely hope that these clarifications, additional downstream results, and planned revisions will help you reconsider your assessment.