- Rating
- 4
- Confidence
- 3(−1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents PACT-Agent, a self-evolving language-agent framework that replaces purely retrospective memory construction with prospective mechanism testing. The key motivation is that a successful trajectory may be consistent with multiple environmental mechanisms, so a rule distilled after the fact may fit the observed outcome without actually having predicted it. PACT-Agent addresses this by maintaining competing mechanism hypotheses, using predictive contracts to precommit mechanism-conditioned outcomes, selecting short-horizon actions through an Expected Free Energy objective, and using a Variational Free Energy gate for post-hoc memory revision. The paper is technically well motivated, and I find the core idea—distinguishing retrospective fit from prospective validation—interesting and potentially important for self-evolving agents. The benchmark choices are also well aligned with the proposed problem, particularly WebArena and AndroidWorld, where local interaction semantics are often ambiguous. However, I currently have several concerns that prevent me from recommending acceptance in the current form.
Critical strengths
The problem formulation is compelling. The paper correctly identifies an underexplored weakness of many self-evolving agent systems: trajectories are usually generated to solve tasks, not to distinguish competing explanations of environment dynamics. As a result, retrospective reflection may produce rules that explain a trajectory but are not actually identifiable from the available evidence. I also like the separation between memory admission and prospective validation. A retrospectively proposed mechanism may be stored as unresolved, but it is not promoted to a resolved rule until it has been prospectively tested. This is a conceptually clean design choice. The experimental settings are generally well chosen. WebArena, AndroidWorld, and ALFWorld contain many cases where environment behavior is only partially observable or where superficially similar actions can have different effects. The ablations on information gain, belief updating, complexity regularization, and prospective validation are useful, and the reported gap between prospectively validated and retrospective-only mechanisms is suggestive.
Critical weaknesses
Presentation of the figures. The overview figures are somewhat over-stylized and have a noticeable AI-generated appearance, which reduces technical clarity. I encourage the authors to replace them with a cleaner and more conventional system architecture diagram during rebuttal.
Incomplete related-work discussion. The paper misses closely related work on failure modes of self-evolving agent memory, particularly Obsessive Experience Poisoning (OEP). OEP studies how locally valid but non-transferable experiences can induce over-generalized persistent rules, which is complementary to PACT's distinction between retrospective fitting and prospective validation. This connection and the difference in problem scope should be discussed.
Lack of direct mechanism-identification validation. The current experiments mainly demonstrate downstream performance and future prediction accuracy, but do not directly verify the paper's central claim that prospective testing improves mechanism identification. A controlled experiment with known ground-truth mechanisms, reporting identification accuracy or false-resolution rate, would substantially strengthen the causal attribution of the gains.
Questions for authors
Would the authors consider replacing the current overview figure with a cleaner and more conventional system architecture diagram? I believe this would substantially improve the technical clarity of the paper.
Related work. How does PACT-Agent relate to Obsessive Experience Poisoning (OEP), which studies how locally valid but non-transferable experiences can induce over-generalized persistent rules in self-evolving agents? In particular, how do the authors distinguish PACT's prospective mechanism validation from OEP's concern about rule scope and cross-task transferability?
Direct validation of mechanism identification. Can the authors provide a controlled experiment with known ground-truth mechanisms to directly verify that prospective testing improves mechanism identification, rather than only downstream task success or future prediction accuracy? Metrics such as mechanism identification accuracy and false-resolution rate would be especially helpful.
I currently lean toward accept.
We sincerely thank the reviewer for the strong support and for framing the retrospective-vs-prospective distinction so clearly.
W1 / Q1 (Overview figure). We replaced the overview figure with a cleaner, conventional architecture diagram (Worker contract → contracted EFE → execution → VFE-gated Builder → versioned memory with unresolved/resolved states). We hope this improves technical readability.
W2 / Q2 (Relation to Obsessive Experience Poisoning, OEP). We added discussion of OEP. The problems are complementary:
OEP: locally correct experiences can yield over-broad persistent rules that fail on transfer—an attack/robustness perspective on rule scope and cross-context misuse.
PACT: under a rule’s stated applicability, multiple mechanisms may equally explain seen trajectories. Before the next action, PACT records each candidate’s predictions and uses observations to discriminate them (e.g., ReadBack tests whether Save alone persists). Validation supports that rule version under tested conditions, not automatic extrapolation to new edit modes.
PACT records explicit applicability; widening scope creates a new version and requires evidence on the expanded conditions. Our experiments focus on mechanism prediction and task success; OEP-style cross-context poisoning robustness is important but requires dedicated evaluation—we cite OEP and clarify scope rather than claiming to solve poisoning.
W3 / Q3 (Direct mechanism-identification validation). We added a controlled Zendo study (following the compositional hypothesis-setting of Grounding Compositional Hypothesis Generation in Specific Instances): each episode has a known hidden rule and an initial scene compatible with multiple hypotheses (e.g., “one blue one red” fits both “exactly one blue” and “at least one blue”). The agent may test up to seven additional scenes; we save PACT’s pre-action predictions and at episode end compare memory rules marked resolved against the hidden rule (domain-equivalent rules count as correct). Gemini-2.5-flash; three variants:
Full PACT substantially exceeds both controls: active EFE-based testing helps find the true rule, and prospective validation reduces false resolution given the same queries. We will include full protocol details in the appendix / supplemental material.
We hope these additions address the remaining concerns and would be grateful if the reviewer considers whether the revised empirical and conceptual clarity supports the positive assessment.