- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces PDSA, a framework that separates an agent’s external memory from its adaptable policy to distinguish learning from simply storing information or trying more answers. Through experiments, it tests whether feedback-driven policy updates and selective memory storage improve performance under limited resources. The results are largely negative. Its main contribution is a careful diagnostic framework showing that meaningful feedback and measurable policy changes do not necessarily translate into better solutions, with limited candidate coverage remaining a major bottleneck.
Critical strengths
- Matched candidate budgets, frozen-policy controls, and hidden audit cases help distinguish actual benefits from gains due to additional attempts. The paper also reports its negative findings transparently.
Critical weaknesses
-
Experiment focuses on arithmetic and list-program repair, with only 32 problems in the final comparison.
-
Policy adaptation and memory admission are evaluated separately, so the experiments do not establish whether their interaction is useful.
-
Poor candidate coverage identifies where errors persist, but does not explain why learning fails to improve that coverage. With at most three updates per problem in the final study, the results cannot clearly separate limitations of the approach from insufficient optimization or exploration.
Questions for authors
See weakness
2 comments
Thank you for the authors' response. I recommend authors to submit their work to the next venue with a substantial change.
We thank the reviewer for recognizing the core diagnostic value of our study—particularly the matched candidate budgets, frozen-policy controls, hidden audit cases, and transparent reporting of negative findings. Since our primary contribution is a diagnostic separation rather than a new SOTA system, we address the three weaknesses and presentation concerns through that lens below.
Task scope and sample size (32 problems). We agree that 32 synthetic problems on a single 1B model cannot support broad generalization claims or high-precision effect estimates, and we frame all conclusions as strictly conditional on this tested setting. However, this bounded scale was a deliberate trade-off to make exhaustive post-hoc auditing feasible: we scored every unique saved program across all arms (1,429 distinct problem-program items, 3,696 exposures, 26,063 executor calls; Appendix D.1) on 20 held-out audit cases per problem. That complete evaluation is what enables an exact separation between candidate-generation failure and public-selection failure without sampling approximations.
Separate evaluation of policy adaptation and memory admission. We agree that the experiments do not establish whether their joint interaction is useful. We isolated the two components intentionally as a diagnostic filter before running a combined system (Table 28). Specifically, on the policy side, online updates do not outperform equal-budget frozen sampling (§5.1). On the memory side (§5.3, Table 30), the label-selected write action improves over no-write (0.43125 vs. 0.5140625) but merely matches a simple relevance/source-trust heuristic (0.43125). Testing a joint Phi x M system before either component clears its standalone baseline would confound unresolved single-component bottlenecks. Instead, we specify the prospective joint-effect factorial protocol in Appendix O as the falsifiable next step once component-level gains are established.
Candidate coverage under at most three updates. We fully agree that at most three optimizer steps per problem cannot separate intrinsic limitations of the approach from insufficient optimization or exploration (as we note in §6). Within that budget, however, our diagnostics isolate two concrete findings that rule out simpler explanations: (a) Learning signals and behavioral shifts did occur: Dense feedback drives 83 optimizer steps for arm D, shifts attained-prefix distributions (28/29 applicable probes have positive TV), and causes candidate-token streams to diverge from frozen/exact/sham baselines on 27/25/28 of 32 problems (§5.1, Table 21). (b) Public selection did not miss good solutions: Post-hoc pool scoring (§5.2, Table 5) shows that whenever an audit-perfect program appeared in any arm's saved pool (2/32 problems), the public selector already chose it. Even an oracle audit-best selector leaves mean loss at 0.609-0.684 because 30/32 pools contain no audit-perfect candidate. Thus, while why coverage remained low (e.g., 3-step budget vs. reward proxy misalignment vs. exploration) remains open, our decomposition rules out both "updates had no behavioral effect" and "good answers were generated but overlooked," pinpointing candidate-pool coverage (L_pool) as the exact bottleneck to target next.
Addressing Presentation (Rating 1). We appreciate the candid feedback that the manuscript is difficult to follow. Because the underlying diagnostic separations are useful, we want the presentation to surface them clearly rather than burying them in internal execution logs. In the revision, we will: (a) Add an overview diagram of the dual-state (Phi, M) separation, concise pseudocode of the main round-by-round loop (moving it from Appendices D.2/Q.2), and a concrete worked example of an arithmetic/list repair task in Sections 1-3. (b) Move operational provenance details (execution dates, engineering-run accounting, and byte/hash inventories) out of the main text into a dedicated Reproducibility appendix, and replace dense multi-decimal text with visual summaries of L_pool vs. R_select.