acceptodds
Under review as a conference paper at ICLR 2027

PDSA: Separate Policy and Memory State for Resource-Bounded Adaptation

Abstract

An adaptive agent can improve by retaining facts, changing its policy, or trying more answers. Predictive Dual-State Adaptation (PDSA) separates bounded memory from an independently restorable numerical policy and proposes action-conditioned future utility for admission. We study controlled arithmetic and list-program repair with a frozen shared semantic base, public execution feedback, and hidden audit cases excluded from updates and selection. Five studies examine shared semantic qualification, candidate/update budget, calibrated update strength, policy-carrier design, and feedback resolution. Larger candidate budgets with evolving public feedback improve selected answers equally in updating and frozen regimes; neither calibrated strength nor an internal carrier establishes its prespecified feedback-specific gain. In the final 32-problem comparison, dense feedback distinguishes every active candidate batch, drives 83 optimizer steps across problems (at most three per problem), and changes attained-prefix distributions and most candidate streams. Yet dense-learning audit loss is 0.703125, versus 0.7015625 for frozen selection, 0.653125 for exact-reward updates, and 0.73125 for dense cyclic mismatch; none of the registered dense-learning contrasts meets the materiality and uncertainty rule. Configured search reaches 0.028125. Subsequent post-hoc CPU scoring finds no missed audit-perfect solutions in the saved pools: all four methods retain only two fully solved problems under pool-best selection. Pool-best mean loss remains 0.609375–0.684375 on the same audit cases, locating substantial residual error in observed candidate coverage. Separately, admission improves over no-write but not a strong rule. These results distinguish feedback variability, policy reuse, candidate coverage and selection from incremental task benefit; learned predictive admission and full PDSA advantage remain unestablished.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Predictive Dual-State Adaptation, agent memory, test-time adaptation, semantic grounding, causal evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_eARZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces PDSA, a framework that separates an agent’s external memory from its adaptable policy to distinguish learning from simply storing information or trying more answers. Through experiments, it tests whether feedback-driven policy updates and selective memory storage improve performance under limited resources. The results are largely negative. Its main contribution is a careful diagnostic framework showing that meaningful feedback and measurable policy changes do not necessarily translate into better solutions, with limited candidate coverage remaining a major bottleneck.

Critical strengths
  1. Matched candidate budgets, frozen-policy controls, and hidden audit cases help distinguish actual benefits from gains due to additional attempts. The paper also reports its negative findings transparently.
Critical weaknesses
  1. Experiment focuses on arithmetic and list-program repair, with only 32 problems in the final comparison.

  2. Policy adaptation and memory admission are evaluated separately, so the experiments do not establish whether their interaction is useful.

  3. Poor candidate coverage identifies where errors persist, but does not explain why learning fails to improve that coverage. With at most three updates per problem in the final study, the results cannot clearly separate limitations of the approach from insufficient optimization or exploration.

Questions for authors

See weakness

2 comments

We thank the reviewer for recognizing the core diagnostic value of our study—particularly the matched candidate budgets, frozen-policy controls, hidden audit cases, and transparent reporting of negative findings. Since our primary contribution is a diagnostic separation rather than a new SOTA system, we address the three weaknesses and presentation concerns through that lens below.

  1. Task scope and sample size (32 problems). We agree that 32 synthetic problems on a single 1B model cannot support broad generalization claims or high-precision effect estimates, and we frame all conclusions as strictly conditional on this tested setting. However, this bounded scale was a deliberate trade-off to make exhaustive post-hoc auditing feasible: we scored every unique saved program across all arms (1,429 distinct problem-program items, 3,696 exposures, 26,063 executor calls; Appendix D.1) on 20 held-out audit cases per problem. That complete evaluation is what enables an exact separation between candidate-generation failure and public-selection failure without sampling approximations.

  2. Separate evaluation of policy adaptation and memory admission. We agree that the experiments do not establish whether their joint interaction is useful. We isolated the two components intentionally as a diagnostic filter before running a combined system (Table 28). Specifically, on the policy side, online updates do not outperform equal-budget frozen sampling (§5.1). On the memory side (§5.3, Table 30), the label-selected write action improves over no-write (0.43125 vs. 0.5140625) but merely matches a simple relevance/source-trust heuristic (0.43125). Testing a joint Phi x M system before either component clears its standalone baseline would confound unresolved single-component bottlenecks. Instead, we specify the prospective joint-effect factorial protocol in Appendix O as the falsifiable next step once component-level gains are established.

  3. Candidate coverage under at most three updates. We fully agree that at most three optimizer steps per problem cannot separate intrinsic limitations of the approach from insufficient optimization or exploration (as we note in §6). Within that budget, however, our diagnostics isolate two concrete findings that rule out simpler explanations: (a) Learning signals and behavioral shifts did occur: Dense feedback drives 83 optimizer steps for arm D, shifts attained-prefix distributions (28/29 applicable probes have positive TV), and causes candidate-token streams to diverge from frozen/exact/sham baselines on 27/25/28 of 32 problems (§5.1, Table 21). (b) Public selection did not miss good solutions: Post-hoc pool scoring (§5.2, Table 5) shows that whenever an audit-perfect program appeared in any arm's saved pool (2/32 problems), the public selector already chose it. Even an oracle audit-best selector leaves mean loss at 0.609-0.684 because 30/32 pools contain no audit-perfect candidate. Thus, while why coverage remained low (e.g., 3-step budget vs. reward proxy misalignment vs. exploration) remains open, our decomposition rules out both "updates had no behavioral effect" and "good answers were generated but overlooked," pinpointing candidate-pool coverage (L_pool) as the exact bottleneck to target next.

  4. Addressing Presentation (Rating 1). We appreciate the candid feedback that the manuscript is difficult to follow. Because the underlying diagnostic separations are useful, we want the presentation to surface them clearly rather than burying them in internal execution logs. In the revision, we will: (a) Add an overview diagram of the dual-state (Phi, M) separation, concise pseudocode of the main round-by-round loop (moving it from Appendices D.2/Q.2), and a concrete worked example of an arithmetic/list repair task in Sections 1-3. (b) Move operational provenance details (execution dates, engineering-run accounting, and byte/hash inventories) out of the main text into a dedicated Reproducibility appendix, and replace dense multi-decimal text with visual summaries of L_pool vs. R_select.

Thank you for the authors' response. I recommend authors to submit their work to the next venue with a substantial change.

Reviewer_np8QOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors want to explore which of two cases is more advantageous for a model: updating its parameters at test time, or sampling enough times to pick the best. They tested with a small model and found that the two are about the same on the vast majority of the problems they tested.

Critical strengths
  1. The topic has practical significance. It can help answer whether, in industrial scenarios, a model should use some compute to keep updating (updating at test time), or use more parallelism to select more possible results and then pick the best among these possible results. This is the only strength I can see.
Critical weaknesses
  1. The sampling scope is too small. There are only 32 problems, and they are not from a very well-known, widely used benchmark, so it is quite possible that this effect only appears in this one place. 2. Too few methods are compared, so it cannot show at all whether the results really are as the authors claim. 3. The amount of experiments is too small, far below the standard required by this conference.
Questions for authors
  1. About a glossary: this paper is very hard to understand and is full of all kinds of uncommon terms (terms from various disciplines). I hope the authors can provide a glossary that converts these uncommon terms into easy-to-understand equivalents. 2. About additional experiments: I hope more experiments can be added in a later submission.
2 comments

Thank you for recognizing the practical relevance of the question and for asking for clearer terminology.

1. Diagnostic question and findings (Sec. 5.1–5.2, Tables 3–5)

We ask whether updating the policy within a problem improves held-out performance over frozen sampling and selection when both use the same public feedback, selection rule, and candidate ceiling.

In 66 of the dense arm’s 115 active batches, dense rewards varied while exact rewards were all zero. The dense arm took 83 optimizer steps, changed next-token distributions in 28 of 29 applicable probes, and produced different candidate streams from frozen selection on 27 of 32 problems. Mean audit loss was 0.7031 for dense updates and 0.7016 for frozen selection. The adjusted frozen-minus-dense interval was [−0.0625, 0.0656], which includes the prespecified 0.05 improvement threshold.

Table 5 shows what better selection could recover from the saved pools. In each arm, the public selector picked an audit-perfect program in both pools that contained one. The other 30 pools contained none. Even choosing the audit-best candidate in every pool leaves a mean loss of 0.609–0.684 across arms. Public selection adds a further 0.044–0.064. The analysis covers 1,429 distinct problem–program items, of which 1,325 were newly scored in 26,063 executor calls (App. D.1).

2. Study scope and benchmark choice (Sec. 4, Table 2)

We agree that 32 problems, one paired sampling stream per problem, and one fitted base limit the final comparison. Table 2 reports five studies addressing distinct questions about shared semantic grounding, candidate budget, update strength, adapter design, and feedback resolution. The final 32-problem comparison addresses feedback resolution. The other studies have their own units of analysis, and Apps. J–K report separate memory-admission experiments.

We constructed the program-repair tasks to control which facts are available to the model, which examples provide public feedback, and which cases are held out for evaluation. This lets us examine how changes in feedback and update design affect both the generated pools and the selected programs. Our conclusions are scoped to the evaluated task families and fitted base.

3. Comparators (Tables 2–6, Apps. J–K)

The final neural comparison includes frozen selection, exact-reward updates, dense-reward updates, and cyclic reward-mismatch updates. These arms share a public selection rule and candidate ceiling. The earlier studies examine performance across candidate budgets and compare original versus publicly calibrated learning rates and output-layer versus internal adapters. Configured symbolic search provides a separate reference with direct operand binding from resident facts. The memory experiments use their own no-write and heuristic controls.

4. Glossary and readability

We will add a short glossary and a five-study overview in the revision.

TermPlain-language meaning
Policy carrier (Φ)Independently restorable policy state that can be updated within a problem.
Memory (M)A separate store of validated records available to the model.
Public / audit casesIn program repair, four visible input–output examples provide feedback and guide selection. Twenty held-out cases score the selected program after the choice is locked.
Cyclic mismatchRotates rewards among candidates during updates while keeping the correct scores for public selection.
Attained-prefix probeCompares next-token probabilities under two policy states at the same prefix reached during generation.
Pool loss / selection regretThe lowest audit loss among the saved candidates, and the additional audit loss incurred by choosing the public winner.
Rebinding pairTwo instances with the same workflow and a changed resident fact, used to check whether the output follows that change.
Configured searchSymbolic enumeration of short programs with operands bound directly from stored facts.

We would appreciate your reconsidering the recommendation in light of these clarifications.

Thank you for the authors' response. However, since the expected experiments were not added, I will maintain my score.

Reviewer_xyJPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper considers whether numerical updates to the policy within the problem could lead to better results than sampling and selecting public feedback. The process is divided into an immutable base, a policy adapter that can be refitted, and limited memory. All components are examined through the application to math problems and list processing. The contribution of the study is primarily theoretical and consists of developing the method of verification. There is no evidence from experiments about the superiority of the whole PDSA framework. In one experiment with 32 problems, learning with abundant feedback affects token selection and choice. The loss of the proposed approach is 0.7031, of frozen selection - 0.7016, and of exact-reward learning - 0.6531. No proposed comparisons revealed any superiority. Experiments testing memory access separately performed better than without memory access and than the compared approach.

Critical strengths
  • The matched frozen comparator plus shared semantic base provides a credible test of the implemented online regimes. The consistent constrained sampling/replay distribution, policy-restoration interventions, and audit withheld from selection all help in this comparison. The paper correctly recognizes that cyclic reward mismatch has expected raw gradient −g/7, not zero, and does not treat it as a neutral placebo. These controls help the internal interpretation of the results.
  • The complete-pool diagnosis provides useful additional information to a mere null comparison. For each method, the two pools containing an audit-perfect answer already have an audit-perfect public winner (in the other pool). The other 30 contain no perfect answer on the fixed audit cases. Even with audit-best selection, substantial loss remains. Within the observed pools, overlooked perfect candidates therefore cannot be the sole explanation for the result. This conclusion is appropriately limited to finite-pool coverage rather than the intrinsic model capacity.
Critical weaknesses
  • The updater maximizes expected completion reward, but evaluation measures the hidden exact loss of a publicly selected candidate. Dense similarity changes reward alignment as well as reward sparsity. Variable rewards and divergent samples therefore do not establish that the updates applied useful optimization pressure.
  • The main comparison uses one fitted base, one sampling stream per problem, and at most three updates per problem. The adjusted frozen-minus-dense interval, [−0.0625, 0.0656], still includes the paper’s material improvement threshold of 0.05.
  • Configured search binds operands directly from resident facts, whereas neural methods must generate the operands themselves. Its much stronger performance therefore does not distinguish difficulty with operation composition from difficulty with parameter binding.
  • The admission experiment evaluates eight single-event prefixes, selects actions using sampled label futures, and disables subsequent writes. It does not evaluate a learned deployment-time admission policy. The method matches the relevance/source-trust rule, and joint efficacy remains unmeasured.
  • Transient LoRA adaptation already appears in Policy of Thoughts, and downstream-reward memory management appears in Memory-R1. PDSA’s case for acceptance therefore rests mainly on its diagnostic insight. Its frozen comparator receives aggregate scores without the corresponding candidate program or correction history.
Questions for authors
  • Do retained measurements demonstrate improvement in the public training objective at fixed prompts beyond positive total variation and candidate divergence?
  • Does the coverage/adaptation gap persist when operands are constrained to the appropriate resident facts, leaving operation composition to the model?
  • Do the teacher and strong heuristic choose the same actions, or do they merely receive identical audit losses? Among the already evaluated actions, is there descriptive headroom over the heuristic?
2 comments

Thank you for the detailed review and for recognizing the value of the matched frozen control, policy-restoration interventions, and complete-pool analysis. Our main finding is that dense-reward updates changed later token distributions and candidate streams, while the registered comparisons did not establish a material gain in the held-out audit loss of the publicly selected program.

1. Scope and interpretation of the feedback-resolution results (W1–W2)

Table 2 reports five studies, including the final 32-problem feedback-resolution study. The earlier studies characterize shared semantic grounding, candidate budget, update strength, and adapter design. The uncertainty in the final comparison reflects its own design: one fitted base, one paired sampling stream per problem, and at most three updates per problem.

For the registered frozen-minus-dense contrast (F − D), the point estimate is −0.0015625 and the adjusted interval is [−0.0625, 0.065625] (Table 4). The interval includes both zero and the prespecified 0.05 material-improvement threshold, so a gain of that size remains compatible with the data.

The exact-public-reward arm (E) has the lowest observed neural audit loss (0.653125, versus 0.701563 for F and 0.703125 for D). Its pool-best loss and public-selection regret are also descriptively lower than F's (Tables 3, 5, and 14). F − E was not a registered primary contrast, the adjusted E − D interval includes zero, and the cyclic-mismatch control uses dense rewards. These comparisons leave the contribution of reward alignment unresolved.

The saved pools give a more direct result about coverage. In each neural arm, the public rule found an audit-perfect program on both problems whose pools contained one. The other 30 pools in each arm contained none. Even retrospective selection of the best saved program left mean audit losses of 0.609–0.684 across the neural arms (Table 5).

2. Expected public reward at fixed prompts (W1, Q1)

The retained measurements provide no paired pre/post estimate of expected public completion reward at the same prompt. The logged RLOO objective is computed on the update batch, and the later prefix probes measure token-probability changes. Dense feedback turned 66 of 115 active batches from exact-all-zero to dense-variable, produced 83 optimizer steps, and changed token distributions in 28 of 29 applicable round-two probes. D's candidate-token streams differed from F, E, and S on 27, 25, and 28 of 32 problems, respectively (Section 5.1 and Table 21). These measurements establish behavioral change. The exact and dense regimes differ in both reward definition and update frequency, so they leave the effect of reward sparsity unresolved.

3. Operand binding and operation composition (W3, Q2)

The operand-constrained neural repair comparison remains untested. In the offline qualification with workflow order supplied, the shared base achieved 16/16 audit-perfect greedy programs and 8/8 successful rebinding pairs in each stratum (Appendix H, Table 27). In compositional repair, configured search binds operands from resident facts, so its advantage leaves the relative contributions of operand binding and operation composition unresolved. F, E, D, and S share the public task interface and grammar rules, so this attribution is unnecessary for the comparisons among the neural arms.

4. Admission action agreement and headroom (W4, Q3)

The saved records underlying Table 30 show that the label-selected teacher and the relevance/source-trust rule chose the same slot or NULL on all eight prefixes. Taking the best legal action retrospectively on the audit futures improves one prefix (Arithmetic 3, from 0.3125 to 0.2500). This lowers the mean from 0.43125 to 0.4234375, a descriptive gap of 0.0078125 on those same futures. The label-selected teacher matched the strong rule on independent audit and improved over NULL (0.51406). Appendix O.1 calls for useful independent-audit headroom before training a future-blind learner. The completed study disables subsequent writes and leaves learned deployment-time admission and its interaction with online policy updates unmeasured.

5. Prior work and comparator scope (W5)

Section 2 credits Policy of Thoughts for transient policy adaptation and Memory-R1 for reward-based memory management. Our contribution is the diagnostic comparison of changes in later behavior, coverage in the saved candidate pool, and the extra loss from public selection. The frozen and updating arms receive the same types of aggregate feedback under a common selection rule. Giving the frozen policy access to candidate programs and per-case correction histories would test an additional repair baseline.

Thank you for the authors' response. I will maintain my score.

Comments on the paper

1 comment

We have uploaded a revised manuscript with clarifications throughout the paper on the study design, terminology, and interpretation of the results. The experimental results and supplementary code are unchanged. We apologize for posting the revision so close to the end of the discussion period. We were unfamiliar with this year’s revision workflow and appreciate the reviewers’ and area chair’s time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.