- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This submission appears to be AI-generated and probably doesn't meet the quality standards for acceptance.
Critical strengths
N/A
Critical weaknesses
Likely to be AI-generated.
Questions for authors
N/A
3 comments
The Reviewer Has Exercised Independent Judgment and Recommends Rejection
I strongly disagree with the authors' response. My concerns are about the fundamental research quality of this submission, not merely its writing style. Even a brief examination reveals several critical weaknesses that, in my assessment, cannot be adequately addressed during the rebuttal period.
1. Inadequate benchmarking and external validation. The main experiments rely almost entirely on ProcStream-RSI, a benchmark constructed by the authors themselves. Although the paper provides implementation details and supplementary materials, its central claims remain largely unvalidated on independently established benchmarks. The only external evaluation uses a subset of just 32 HumanEval+ tasks, despite the availability of a substantially larger benchmark, and does not even evaluate the main Scoped-ORC approach. Why are there no evaluations on established agent memory or continual learning benchmarks?
2. Weak and insufficiently grounded baselines. The main comparisons rely on author-implemented variants such as Latest-only, Replay, and Self-judge, rather than strong, established methods from the literature. Although related work is cited, the experimental baselines are not adequately connected to published reference implementations or competitive existing approaches. Without such comparisons, the claimed improvements provide limited evidence of advancement over the state of the art.
3. Insufficient experimental evidence for the claimed contributions. The original ORC method fails to outperform the Static baseline, while the main improvements arise from family-scoped retrieval. The paper does not adequately establish whether its elaborate verification mechanism offers meaningful benefits beyond straightforward per-family memory management.
Beyond these experimental concerns, the manuscript requires substantial improvements in clarity, organization, and scientific presentation to meet the standards expected of an ICLR submission. I encourage the authors to examine accepted ICLR papers for examples of how to present methodological contributions and support them with rigorous, independently verifiable experiments.
To be clear, my recommendation for rejection is based on these substantive scientific weaknesses. Whether or not the manuscript was AI-assisted does not change the authors' responsibility to establish novelty, conduct convincing comparisons, and substantiate their claims.
I do not believe these fundamental shortcomings can be adequately resolved through a rebuttal. I therefore maintain my recommendation for rejection and encourage the Area Chair to evaluate the submission accordingly.
Reply to Reviewer: Clarification of the Expanded Evidence and Contribution
Thank you for clarifying your concerns. The revision adds experiments that directly test memory capacity, retrieval scope, and update selection.
Benchmark and external validation. ProcStream-RSI enables a controlled test of cross-family interference: the fixed-output intervention changes retrieval scope while holding proposals, acceptance decisions, and task completions fixed. The HumanEval+ experiment provides a separate transfer diagnostic. Established memory benchmarks would broaden external validation; our central evidence identifies the deployment mechanism through controlled comparisons.
Capacity-matched baselines. We added multiversion retrieval with matched storage capacity, metadata allowance, active skill length, editor, and gate. Scoped retrieval improves trajectory utility by 0.026 [0.012, 0.043] and final-bank utility by 0.060 [0.026, 0.099]. A shared-prompt-bank comparison further supports selecting the relevant skill. These experiments directly address whether additional memory capacity alone explains the gains.
Scope and selection. On the same 27 streams, Scoped-ORC exceeds Static by 0.055 [0.040, 0.071] in trajectory utility. The added gate controls separate this retrieval benefit from selection: private-only matches ORC on these streams, while the broader comparisons characterize the utility–regression trade-off. Against a scoped cumulative one-shot control with the same slot budget, sequential adaptation improves final-bank utility by 0.041 [0.023, 0.062].
These results strengthen our central contribution: identifying cross-family interference and showing how scope-matched memory preserves useful skills across recurring tasks. The assessment that scope, capacity, and selection remain experimentally unseparated does not reflect these added controls.
Thank you for considering these clarifications. We invite you to review the latest revision and its expanded experimental evidence, and to reconsider your assessment in light of the added controls and findings.
The reviewer states, “This submission appears to be AI-generated and probably doesn't meet the quality standards for acceptance.” We stand by the AI-use disclosure on page 11 of the submitted manuscript. If the reviewer disputes its accuracy, we ask the reviewer to specify what use was allegedly omitted and cite evidence for that claim. The quoted comment offers an impression of the prose as grounds for recommending rejection. We ask the area chair to give that inference no weight in the decision. The research community’s trust in the conference depends on reviewers exercising independent judgment and giving reasons that authors can answer and area chairs can verify.