acceptodds
Under review as a conference paper at ICLR 2027

From Local Impasses to Grounded Recovery: Experience Reuse for LLM Agents

12 views

Abstract

LLM agents can exhaust their interaction budgets by repeating unproductive actions, even when relevant past experience is available. Effective experience reuse must therefore do more than retrieve a similar task: it must identify a supported way to restore progress under current conditions. We propose Policy–Memory–Recovery (PMR), a framework for recovering from local impasses without updating policy parameters. PMR builds on a key observation: the experience revealing a difficulty need not contain its solution, and evidence for a useful local effect can occur in an otherwise unsuccessful episode. PMR separately establishes context and effect evidence, then links compatible evidence across episodes into conditional recovery records. This allows unsuccessful interaction to supply useful evidence without treating its behavior as a solution. At deployment, retrieved experience advises the policy, while corrective execution requires current-state checks and observation-based confirmation of progress, without auxiliary recovery-generation calls. Across multiple long-horizon interactive environments, PMR consistently achieves the highest mean task success, outperforming strong baselines by – percentage points while using fewer test-time actions and model calls. Controlled support removals and a fixed-bank execution intervention show that the gain follows relevant evidence and checked execution rather than memory volume alone, establishing evidence-grounded recovery as a more effective alternative to task-level experience retrieval.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, episodic memory, experience reuse, long-horizon tasks

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_y7SHOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

LLM agents often get stuck in long-horizon tasks, repeating unproductive actions until the budget runs out. PMR improves recovery by providing a corrective next step when agent got stuck. It splits past trajectories into two kinds of local evidence: where the agent was stuck (context) and which actions produced useful effects (effect). These can come from different episodes, including failed ones, and are compiled offline into conditional recovery records. On ALFWorld, TextCraft, and BabyAI, PMR achieves the highest task success while using fewer actions and model calls than memory-based baselines.

Critical strengths
  1. The paper is clear and well-motivated. Its key idea is that “where the agent got stuck” and “what action helped” can be learned separately, so useful steps inside failed episodes can still be reused instead of being discarded.

  2. PMR has strong results on all benchmarks, achieving higher results while using less tool calls.

  3. This paper has extremely careful ablation and intervention designs, these go far beyond what most agent memory papers provide.

Critical weaknesses
  1. I think it is unclear whether the gains actually come from evidence linking. The paper claims that connecting past experiences, including failed ones, to the current situation helps agents recover. However, Table 2 shows that removing the direct-effect branch reduces PMR from 77.6% to 55.0%, close to retrieval alone (56.5%). In contrast, removing information recovery only reduces performance by 3.7 points.

  2. The paper lacks a baseline that uses the same interface without past experience. PMR uses a manually designed adapter that already provides useful information like task goals and progress detection. However, the baselines do not have access to this interface. A simple rule-based agent using the same information, but without any past experience, might already perform well.

Questions for authors

I have only one question: How does a rule-only controller perform when it uses the same PHI, the adapter that provides the useful information but no compiled experience?

5 comments
AuthorsOpenReview ↗

Response to Reviewer y7SH (1/3): Overview of our response

Thank you for recognizing the separation of context and effect evidence, the benchmark results, and the care taken in the intervention designs. We particularly appreciate your close reading of the ablations.

We address your concerns in the following two comments:

  1. Same-interface rule controls — Weakness 2 and your question: PH/PMH test whether the shared interface and manual rules account for the gains.
  2. Execution and relevant evidence — Weakness 1: the branch ablations and controlled support removals distinguish execution utility from dependence on relevant experience.
AuthorsOpenReview ↗

Response to Reviewer y7SH (2/3): Same-interface rule controls

Same-interface rule controls — Weakness 2 and your question.

We conduct this baseline on ALFWorld, the environment underlying the Table 2 ablation motivating your concern, so that the same-interface comparison can be interpreted alongside the existing execution and support interventions. Specifically, we evaluated two controls on the same 134 unseen ALFWorld tasks, using the same frozen Qwen3-4B-Instruct-2507 policy and 50-action budget:

  • PH (policy + human rules) uses the same adapter Φ\Phi and bounded execution framework, with manually specified rules for delivery/placement, heating, cooling, cleaning, resource/hand management, and information search. For instance, the rules cover the following cases:

    • Delivery / placement: if the target is held and a valid placement location is observed, move there and place it.
    • Heating / cooling / cleaning: if an object must be heated, cooled, or cleaned, go to the corresponding appliance and apply the action once the object is held and the appliance is reachable.
    • Resource / hand management: if the hand holds an object that is not immediately needed, put it down before another pickup or action.
    • Information search: if the target object has not yet been observed and repeated actions produce no new information, systematically open unexplored receptacles (such as cabinets, drawers, or the fridge) rather than repeating the same action.

    PH retains the frozen policy as the default proposer, and manual rules may replace that proposal after current-state prerequisite checks. PH uses Φ\Phi and reads no experience artifacts. This matches PMR's execution split. The question concerns the shared adapter without compiled experience, so the proposer stays fixed; removing the policy would change the proposer as well as the experience store.

  • PMH (PH + memory) keeps those rules and adds retrieved advice from the mixed300 cards and index compiled by PMR (the same cards and index used by the PMR reference in this comparison). Retrieved experience advises the policy, while corrective overrides remain governed by the manual rules.

The new PH/PMH controls are compared with the mixed300 PMR reference from the original submission (Tables 8 and 10). Each configuration has three runs:

ControllerSuccesses out of 134Success rate, mean ± SD
PH86 / 92 / 8866.17 ± 2.28%
PMH93 / 96 / 9169.65 ± 1.88%
PMR104 / 96 / 9874.13 ± 3.11%

The shared-interface rule controller is effective, reaching 66.17%. PMR reaches 74.13%, 7.96 percentage points above PH, and it is ahead on every run (104/96/98 versus 86/92/88).

PMH adds the same mixed300 cards and index, while the manual rules keep the override: the retrieved cards only advise the policy. Success moves from 66.17% to 69.65% (+3.48). Those cards therefore add little when they keep this advisory role and the manual rules retain the authority to replace the submitted action. The existing mixed300 retrieval-only arm disables recovery and keeps the cards as advice. It succeeds on 57/66/63 tasks (46.3%), so this source experience does not by itself produce PMR's result. PMR gives compiled evidence authority to replace the proposal and reaches 74.13%, 4.48 points above PMH, where the manual rules still hold that authority.

In this experiment, many different rule sets are possible, and no single set constitutes a unique baseline. To more thoroughly test the contribution of the interface, when designing the rules, we considered variants that differ in two practical respects: (1) whether resource/hand-management rules are included (placing a held object before picking up another), and (2) the information-search strategy (systematic opening of unexplored receptacles versus only retrying recent actions). We adopted the version that includes both, as these address frequent sources of stalled progress in ALFWorld. We also note that the comparison is not fully symmetric. The human rules require the designer to already identify which situations lead to stalled progress and to specify appropriate corrective actions in advance. In contrast, PMR starts from raw interaction traces with no hand-specified recovery rules and compiles conditional recovery opportunities using evidence from both successful and unsuccessful episodes.

Under this setting, PMR still outperforms both PH and PMH. Thus, the benefit extends beyond both manual rule control and supplementing that control with retrieved experience. The third comment examines the role of the supporting evidence directly through the fixed-executor support-removal interventions.

Added: Section 4.3, “Same-interface rule controls” (p. 8), and Appendix F.2 / Table 8 (p. 37) define PH/PMH and report their per-run outcomes alongside the existing mixed300 PMR reference. In the revised PDF, blue marks text that existed in the original submission and was moved or lightly edited; red marks text that is new or substantively changed.

AuthorsOpenReview ↗

Response to Reviewer y7SH (3/3): Execution and relevant evidence

Execution utility and evidence linking — Weakness 1.

The direct-effect branch has the largest configuration-level impact in Table 2: removing it reduces success from 77.6% to 55.0%, near retrieval alone at 56.5%. Information recovery is one execution branch; removing it leaves context-linked chains and the separately supported direct-effect branch intact. Its 3.7-point decrement measures the contribution of that branch, rather than the contribution of all evidence linking. Direct effects can intervene when a supported goal effect is applicable, whereas information recovery requires stagnation.

Two interventions distinguish execution utility from dependence on relevant evidence:

  1. Hold the bank fixed and change execution (Table 4). Full PMR and direct-effect-off share compiled support, cards, and the retrieval index. Enabling direct-effect execution raises mean success from 60.70% to 74.13% in the mixed300 cohort: 13.43 percentage points with the available evidence held fixed.
  2. Hold execution capabilities fixed and change support (Figure 3b). At 22 checkpoints, the executor and effects are fixed and retrieval is off. Removing matching support reduces escapes from 22/22 to 5/22; removing an equal amount of unrelated support preserves 22/22. This identifies the contribution of relevant evidence beyond having an executor or retaining a particular volume of support.

The deployed-event intervention corroborates this result: across 17 matched continuations, relevant-support removal reduces task success from 14/17 to 5/17, while equal-size unrelated removal preserves 14/17 (Figure 3c).

Together, these results explain why the large direct-effect ablation is consistent with the value of evidence linking: compiled evidence supports recovery through corrective execution. The information-branch ablation removes one recovery route; it does not remove all evidence-supported recovery. When execution capabilities remain fixed, removing relevant support sharply reduces recovery, whereas removing equally much unrelated support does not. The gains therefore depend on which experience supports the intervention, beyond the availability of an executor or the amount of stored experience. Combined with the same-interface rule controls in the preceding comments, these findings indicate that the improvements come from compiling and grounding relevant local evidence rather than from the shared interface or recovery framework alone.

Added: Section 4.3, “Configurations and purpose” and “From components to mechanisms” (pp. 8–9), connects branch utility to the support interventions. Appendix F.1.2 (p. 36) states the retained branches.

Reference: The intervention results discussed above are those reported in the original submission; the revision clarifies how they complement the component comparisons.

We appreciate your recognition of PMR’s central idea and the care taken in the experimental design. Your question motivated the additional same-interface controls and a clearer explanation of how the existing interventions distinguish execution utility from the contribution of relevant experience.

Reviewer_Z4DvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper addresses a critical challenge in LLM agents: the tendency to repeat ineffective actions and exhaust interaction budgets when encountering local impasses. To mitigate this without altering policy parameters, the authors propose Policy-Memory-Recovery (PMR). This framework mines past episodes to link unresolved contexts with actions that reliably induce progress. Crucially, PMR employs conditional records that decouple evidence of an unresolved context from evidence of a useful effect. Furthermore, it utilizes a recovery controller that separates policy advice from execution authority, verifying candidate actions against the current state before allowing them to override the policy's proposal.

Critical strengths

1- The paper tackles a limitation in standard RL and LLM agent frameworks, where sparse terminal rewards often lead to the unjustified reinforcement of entire trajectories, including ineffective sub-policies.

2- PMR addresses this by isolating and reusing locally effective transitions. This draws strong conceptual parallels to Hindsight Experience Replay (HER) in RL, but innovatively adapts the concept for subgoal-level recovery and evidence grounding in LLM agents.

3- f the empirical claims hold, this represents a system with significant practical utility for improving the reliability and efficiency of long-horizon agent tasks.

Critical weaknesses

1- Lack of Formal Grounding and Basic Definitions. The manuscript lacks a clear formal grounding in its operating context. It is initially difficult to discern whether the proposed memory mechanisms operate on textual representations, neural embeddings, or symbolic structures. While the paper references "policies," it does not establish a clear Markov Decision Process (MDP) or Partially Observable MDP (POMDP) framework. Furthermore, terms such as "interaction budget," "evidence," "experience," and "recovery" are used ambiguously. A concise, formal definition of these terms upfront would significantly improve readability and rigor.

2- Underspecified Terminology and Phrasing. The text frequently employs ambiguous or underspecified terminology, making the methodology difficult to follow. For example, in Section 3.2 ("Inducing supported effects"), the authors state: "Their representation follows the interface: where typed arguments are available, normalization retains relevant types while replacing instance-specific arguments with roles." It is never clarified what constitutes a "typed argument" in this context, what "normalization" operationally entails, or what baseline it is being compared to. The mapping of "instance-specific arguments" to "roles" is not defined. It remains unclear to the reader whether these interface capabilities stem from specific environment affordances (e.g., ALFWorld vs. BabyAI). This level of ambiguity recurs throughout the methodology section.

3- Premature Notation without Definition. Notation is often introduced prematurely or without concrete instantiation. In Figure1, Δgt\Delta g_t is used without defining gg or tt. Similarly, BB, MM, and JJ are introduced in the caption with only weak definitions. Section2 (Problem Formulation) attempts to clarify this, it would be more helpful to explicitly state, for example, ``Let Δgt\Delta g_t denote goal-relevant (gg) progress observed at timestep (tt) after action ata_t.'' In Equation~(1), the observable interface zt=(Ft,Ut,At)z_t = (F_t, U_t, A_t) is introduced. However, the components FtF_t (facts), UtU_t (obligations), and AtA_t (action interface) are never concretely instantiated for any specific environment, leaving the reader to guess what ztz_t actually contains in practice.

4- Obscure algorithmic details. The main text references Algorithms 1 and 2 (e.g., in Section 3.2) without clear cross-referencing to the Appendix. More importantly, the core mechanics of the proposed method remain opaque. The text does not explicitly detail how context classes or matching equivalence classes are formed, nor does it explain the exact decision logic of Resolve Φ.

5- Ambiguous Experimental Setup and Data Splits. The data splits and experimental setup are unclear in the main text. It is not explicitly stated what data is used for tuning, building the memory bank, and final evaluation. While the Appendix provides some details, contradictions or ambiguities remain: -- Appendix B.1 describes a 600-episode mechanism bank (300 successful) -- Section 4.2 mentions a separate 300-episode mixed pool (150 successful / 150 unsuccessful). -- The caption for Figure 2 refers to the "primary policy." While Section 4.1 mentions Qwen3-4B-Instruct-2507, it is not explicitly confirmed whether this exact policy and configuration were used uniformly across all three environments.

​ .

Questions for authors

To help clarify the above points, I request the authors address the following in their rebuttal: a- Episode Selection: What are the precise, step-by-step criteria for selecting and differentiating episodes to populate the experience pool? b- Authority vs. Advice: What is the precise operational distinction between "policy advice" and "action authority" within the recovery controller? c- Candidate Checking: What specific criteria and thresholds are evaluated when checking candidates during the resolution phase? d- How exactly are context classes and matching equivalence classes formed? e- Algorithmic Transparency: What are the concrete implementations of the abstract functions Project Compose, and others in Algorithms 1 and 2? Please provide more detail than the high-level descriptions in Appendix E.1.

f- Data Splits: Which specific data splits are used for building the memory, tuning hyperparameters, and final evaluation? Which exact source pool yielded the main 77.6% result?

8 comments
AuthorsOpenReview ↗

Response to Reviewer Z4Dv (1/5): Interaction, representations, and terminology

Thank you for recognizing the value of locally effective experience for agent reliability and efficiency. PMR's formulation, mechanisms, and results are unchanged. The revisions relocate four existing specifications and add one new control.

We address your concerns in five comments:

  1. Interaction, representations, and terminology — Weaknesses 1–3.
  2. Matching and construction operators — Weakness 4; Questions d, e.
  3. Execution checks and authority — Weakness 4; Questions b, c.
  4. Source pools and training-side validation — Weakness 5; Questions a, f.
  5. Primary policy and inference settings — Weakness 5.

Blue marks moved or lightly edited text from the original submission; red marks newly updated text. Page references refer to the revised PDF; original-submission references are identified explicitly.

Interaction and representations — Weaknesses 1–3.

Operating setting and memory roles — Weakness 1. Here, policy means the LLM's action-selection rule conditioned on the available interaction history. This usage does not require a Markov assumption on the current observation: ReAct, Section 2 likewise defines a policy over observation–action context. PMR uses a history-conditioned policy and checks recovery against observed evidence; it requires neither an explicit environment transition model nor a Markov-state interpretation of its observable projection.

Building on the history-based, partially observable interaction in the original Section 2 and the memory mechanisms in Section 3, the revised Section 2 collects their roles in one place. The frozen policy conditions on goal gg, observation/action history hth_t, and retrieved text. The controller checks the observation-derived interface

zt=Φ(g,ht)=(Ft,Ut,At)z_t=\Phi(g,h_t)=(F_t,U_t,\mathcal A_t)

alongside its execution state. Here ztz_t is an observable projection, not an assumed Markov state; it reads received observations without hidden state or future outcomes. Structured support B\mathcal B stores conditional records and effect evidence; textual cards M\mathcal M advise the policy; the dense embedding index J\mathcal J retrieves cards. Similarity selects advice; evidence and current-state checks determine execution authority.

In the revised manuscript, Section 2 (p. 3) collects the interaction setting, representation roles, and policy/controller inputs in one place.

Definitions — Weaknesses 1–2. An experience is a historical episode trace. Evidence is observation-supported information from local context/effect windows, including non-effects and source references. A recovery is a checked corrective action or bounded sequence whose progress is assessed through subsequent observations. The interaction budget counts submitted environment actions, including corrective actions.

Reference: Section 2 (p. 3) already names these four terms in “Interaction and historical experience,” “Evidence and memory representations,” and “Local evidence reuse for recovery,” alongside the objects they describe.

Notation and concrete instantiation — Weakness 3. In Figure 1, gg denotes the goal, tt the interaction step, ztz_t the observable projection, and Δtg\Delta_t^g goal-relevant progress observed after action ata_t. The mug-cooling-and-placement example instantiates Equation (1): seeing a mug on a countertop supplies its observed type/location facts in FtF_t; UtU_t tracks the cooling and placement obligations and remaining counts; At\mathcal A_t supplies currently available commands.

Reference: Figure 1's caption (p. 3) already places the symbol definitions beside the overview. Section 2, immediately after Equation (1) (p. 3), instantiates the interface components on the mug example.

Read together, these statements identify what the agent observes, what each memory representation stores, and how that information enters policy advice and recovery decisions.

Typed arguments and normalization — Weakness 2.

There is no baseline in this sentence: normalization is a representation operation, applied to source actions before induction. It maps instance-specific identifiers to argument roles. In ALFWorld, mug 1 has the adapter-parsed type mug; holding(mug 1) becomes holding($arg0), where $arg0 is the first action argument. Type constraints remain in the context and grounding checks, and execution binds the variable to a current entity. TextCraft uses observable item/recipe representations; BabyAI uses mission/spatial representations.

Reference: The role-based normalization is in the original §3.2; in the revised manuscript, Section 3.2 (p. 5) instantiates it on argument substitution and current-entity binding. Appendices C.1 (p. 19) and C.5 (p. 22) state the context fields and environment-specific interfaces.

AuthorsOpenReview ↗

Response to Reviewer Z4Dv (2/5): Matching and construction operators

Context and matching classes — Weakness 4; Question d.

Classes are equivalence classes formed by equality over specified, canonicalized fields. In ALFWorld, effect-induction contexts encode direct-affordance absence and resource occupancy: empty, occupied by a pending-role object, or occupied by another object. Contract contexts retain pending target types and the carrying-resource state, including the held object's type and whether it matches a pending role.

Two class definitions serve different purposes:

  • Matching classes: equal complete query keys group records with the same canonical projected fields and branch requirements. A chain query checks the requested effect sequence; a start query compares its requested effect with the first effect of a contract. Single-stage queries restrict records to one-effect chains; relation-state queries also check pending obligations when supplied. Source identity does not determine membership in a matching class.
  • Source-analysis classes: the 69 groups use projected trigger condition, ordered effect identifiers, and terminal events. Source identity and final outcome do not enter this key; successful/unsuccessful labels summarize the sources after grouping.

Thus, matching classes determine which records can support a recovery request, while source-analysis classes describe where the evidence came from. Class membership is a deterministic comparison over these fields: no learned clustering, no similarity threshold, and no supervision.

Reference: Appendix C.1 (p. 19) already specifies the projected context fields. Added: Appendix E.2 (pp. 24–25) expands the original matching-key definition into branch-specific comparisons and distinguishes matching classes from descriptive source-analysis groups.

Concrete operators — Weakness 4; Question e.

The construction operates on observed transitions and bounded prefixes:

  • Project: parse observed facts, obligations, types, affordances, and the action at each source step; attach observed relation additions/deletions, action legality, and source references. Final episode outcomes are excluded from this transition view.
  • Discover / Induce: locate recurrence/repetition windows without goal-relevant progress; group eligible transitions and bounded prefixes by normalized context/action representations. Count positive and non-effect observations and apply the distinct transition/prefix retention criteria in Appendix C.1.
  • Compose: connect an added relation to a later prerequisite when predicate, arity, and arguments admit consistent variable binding. Enumerate paths from a transition effect to a progress option, up to the configured depth and without repeated effect identifiers. The selection order prefers shorter paths, an earlier observable milestone, a shorter median observed terminal length, higher held-out and fit support, and a deterministic identifier tie-break. Algorithm 1 selects the first path.
  • Suffix selection and Link: select the longest suffix of that path compatible with an impasse context, then emit a context-source record referencing the suffix's effect identifiers. Effect entries retain their own support and execution fields.
  • Auxiliary support and assembly: construct direct-effect and information support and validation artifacts; reject failed required gates, including when the chain is empty; package accepted support into B\mathcal B.
  • Cards and retrieval: render event-bounded segments from successful sources linked to chain records; retain source links and build J\mathcal J. Filter cards for compatibility, rank by cosine similarity, and deduplicate event signatures. Observe ingests actual feedback for subsequent stage checks.

These operations specify how observations become retained effects, how compatible effects form a path, and how that path is linked to the context that needs recovery. The runtime checks in Part 3 then determine whether this compiled evidence permits execution in the current state.

Reference: These operators are those implemented in the submitted code and specified in Algorithms 1–2 and Appendix E.1 (pp. 23–24). Section 3.2 (p. 4) points directly to these algorithms. Added: Appendix E.2 (pp. 24–25) is the only new specification; Appendix C.2 (pp. 19–20) expands path selection.

AuthorsOpenReview ↗

Response to Reviewer Z4Dv (3/5): Execution checks and authority

Advice and authority — Question b.

Advice changes the policy input; authority determines the submitted action. Retrieved cards enter the frozen policy's prompt. Equation (3) submits a controller-admitted candidate when authorized and otherwise retains the policy proposal. The two actions can agree. In the mug-finding example, an unresolved obligation is linked to a supported opening effect. Current observations must ground a legal cabinet action; feedback then governs completion or release. Historical cabinet locations do not become current-state facts.

The original Section 3.3 states this distinction directly: “Retrieved cards advise the policy, while recovery can consult structural support outside the retrieved top-kk. … This separates advisory relevance from execution authority.” Equation (3) in Section 3.4 is its operational form: advice enters the prompt, authority is the admission bit γt\gamma_t. In the revision this distinction is the opening sentence of Section 3.4 (p. 5). The mug example in Section 3.2 illustrates how supported effects are grounded for execution.

Reference: In the revised manuscript, Section 2 (pp. 3–4) collects the representation roles and policy/controller inputs, and Section 3.4 (pp. 5–6) connects the advice/authority distinction to the operational checks in Appendix C.4 (p. 21).

Resolution order and checks — Weakness 4; Question c.

In ALFWorld, active information recovery continues its phase. Otherwise, Resolve checks chains first, then direct effects if the chain neither authorizes an action nor was active at entry. New information recovery follows only when neither higher-priority branch occupies control, preserving the chain's handback step.

Admission checks retained support, matching context, grounded prerequisites, preservation, action legality, and remaining budgets. During execution:

  • A chain advances when its terminal is observed and releases on completion, exhausted stage bounds, failed prerequisites, or unavailable grounding.
  • A direct effect requires a pending obligation, an admitted effect, and a legal grounded action. New activation requires a legal policy proposal, retained if it already supports the effect. Each activation has a four-action bound and two unchanged-state checks, followed by same-state retry suppression.
  • Information recovery requires repetition without an observed event, an open obligation, no target sighting or blocking resource condition, a matching contract, and a supported legal information action. It rechecks legality and unused affordances, releasing on target sighting, no authorized action, or compiled action/source limits.

Reference: The conditions themselves are those listed in §3.4 of the original submission. Added: Appendix C.4 (p. 21) gives the branch ordering as a numbered list and the admission and release conditions as a table.

Offline thresholds versus online checks — Question c.

Single-transition retention requires at least five positive fit observations, one positive held-out training observation, and consistency of at least 0.90 in each partition. Bounded-prefix retention requires five legal positive fit prefixes and one held-out prefix under its separate criteria. Online checks use current applicability and execution limits: chain stage bounds accompany effects, and information limits accompany compiled support.

Reference: The retention thresholds in the original Appendix C.1 are unchanged; the online branch checks are in Appendix C.4 (p. 21), separating them from the offline retention criteria.

Reviewer_NVZ2OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper tackles the problem of LLM agents getting stuck in local impasses. The authors propose Policy-Memory-Recovery (PMR), a framework that breaks down past interactions (both successes and failures) to separate the context of a trap from the actual solution (recovery effect). At test time, PMR relies on offline-compiled symbolic rules to check the current state. If a recovery candidate matches, it directly overrides the agent's proposed action. Because this relies on condition-checking rather than prompting the LLM to generate a recovery strategy on the fly, it requires zero auxiliary LLM calls. The authors evaluate PMR on ALFWorld, TextCraft, and BabyAI, showing large gains in task success rates and massive reductions in computational overhead compared to recent baselines.

Critical strengths

Clever use of failure data: Decoupling the trap's context from the recovery action is a genuinely fresh take on experience reuse. Mining failed episodes specifically to build context evidence is a very smart way to leverage data that usually just gets thrown away.

Zero-generation recovery: The condition-checked execution is a huge architectural win. By avoiding auxiliary LLM calls during the recovery phase, PMR slashes online token costs, model calls, and environment actions while still hitting state-of-the-art success rates.

Analytical depth: I really appreciated the experiments here—it’s not just a benchmark dump. The controlled support removal and the mathematical retention law do an excellent job of causally linking specific local evidence to recovery success, rather than just hand-waving that "more memory is better."

Critical weaknesses

Heavy reliance on manual adapters: PMR requires developers to handcraft environment-specific adapters (Φ\Phi) to define state predicates, extraction rules, and branch arbitration. This severely bottlenecks its ability to generalize out-of-the-box to messier, open-ended domains like WebArena.

Brittleness of exact-matching: Authorizing a recovery requires exact equality over canonical projected relation states. This is fine in clean, discrete simulators, but it will almost certainly break in continuous, noisy, or dynamic real-world environments.

Missing foundational baselines: The evaluation compares PMR against strong recent models (AdaMEM, Synapse) but entirely skips foundational failure-recovery methods like Reflexion or ExpeL. Leaving these out makes it harder to situate PMR relative to the most recognized approaches in this exact subfield.

Questions for authors

Can you quantify the human effort (e.g., lines of code or time) it actually took to implement the adapter Φ\Phi for the three evaluated environments?

How do you see PMR adapting to continuous state spaces or noisy observations where exact state matching is practically impossible? Is there a way to integrate probabilistic grounding or similarity-based matching without losing the efficiency of the symbolic check?

Why exclude baselines like Reflexion, ExpeL, or LATS? How would PMR hold up against them regarding success rates and token efficiency?

6 comments
AuthorsOpenReview ↗

Response to Reviewer NVZ2 (1/4): Adapter effort and reuse

Thank you for the thoughtful assessment and for recognizing the reuse of failure evidence, recovery without auxiliary generation, and the depth of the intervention analysis.

We address your three concerns in four comments:

  1. Adapter effort and reuse — Weakness 1 / Question 1: environment-specific responsibilities, implementation size, and reuse of the common mechanisms.
  2. Matching and observation changes — Weakness 2 / Question 2: matching diagnostics and the proposed extension to uncertain observations.
  3. ReAct and Reflexion — Weakness 3 / Question 3: the added manuscript comparisons and cumulative retry costs.
  4. ExpeL and LATS — Weakness 3 / Question 3: the additional ALFWorld comparisons and search-budget analysis.

Adapter effort and reuse — Weakness 1 / Question 1.

PMR requires only a modest deployment-side adaptation: environment interfaces connect to shared evidence-compilation, retrieval, and recovery-control engines. Environment-specific work covers semantic interfaces and candidate construction. Developers specify observable facts, goals, action grounding, and environment-native candidates; training trajectories determine the retained effects, recovery records, context–effect associations, and support. Recovery records therefore do not have to be manually authored one by one.

ALFWorld supplies household relations and object/device binding; TextCraft supplies inventory quantities and recipe grounding; BabyAI supplies mission clauses, observed spatial facts, and local action sequences. Online interfaces interpret observations, track obligations, and ground checks and actions. Offline interfaces convert sources and construct candidates for training-side admission. The common engines use these artifacts without auxiliary online recovery-generation calls.

The deployment-side code is small in all three environments. We count physical Python source lines in the provided adapters and runtime connections, excluding blanks, comments, docstrings, delimiter-only lines, and base methods overridden without delegation on the deployed path:

EnvironmentDeployment-side adapter and integration lines
ALFWorld847
TextCraft595
BabyAI342

These counts exclude offline-only construction, shared engines/utilities, simulator code, model serving, experiment runners, tests, and other baselines. Offline source conversion and candidate construction are separate adaptation components. Environment-specific ordering configures the common controller. Porting therefore concentrates effort at these interfaces. The supplied adapters and builders are reusable in all three supported environments.

Added: Section 3 (p. 4) makes interface-based adaptation and common-engine reuse explicit. Appendix C.5 (p. 22) details these responsibilities and the deployment-side counting scope, with offline construction described separately. In the revised PDF, blue marks text that existed in the original submission and was moved or lightly edited; red marks text that is new or substantively changed.

Does the adapter alone explain the gains? Two controls share PMR's environment interface, bounded execution framework, frozen Qwen3-4B-Instruct-2507 policy, 134 unseen ALFWorld tasks, and 50-action budget:

  • PH (policy + human rules) retains the policy for ordinary action proposals. Manual rules for delivery, heating, cooling, cleaning, resource recovery, and information search can override proposals after current-state prerequisite checks. PH reads no experience artifacts.
  • PMH (policy + memory + human rules) keeps those rules and adds retrieved advice from the mixed300 cards and index shared with PMR. Experience advises the policy; corrective overrides remain governed by manual rules.

Both are compared with the original submission's mixed300 PMR reference, compiled from 150 successful and 150 unsuccessful source episodes. Each configuration has three runs:

ControllerSuccess rate, mean ± SD over three runs
PH66.17 ± 2.28%
PMH69.65 ± 1.88%
PMR, mixed30074.13 ± 3.11%

PMR improves mean success by 7.96 percentage points over PH and 4.48 points over PMH. The advantage over PH and PMH shows that the shared interface and manual rules do not account for the full gain, even when those rules use the same retrieved-experience library.

Added: Section 4.3, “Same-interface rule controls” (p. 8), and Appendix F.2 / Table 8 (p. 37) define these controls and report per-run outcomes.

AuthorsOpenReview ↗

Response to Reviewer NVZ2 (2/4): Matching and observation changes

Matching under observation changes — Weakness 2 / Question 2.

Proposed extension: probabilistic grounding and similarity-based retrieval, followed by symbolic admission. This would preserve PMR's separation between finding a relevant recovery and authorizing its execution. The four parts of the extension are:

  • Continuous states: map continuous measurements to recovery-relevant relations, such as whether an object is within reach or inside a container. Records would match these conditions, without requiring identical coordinates or full-state equality. Bind the candidate action to the currently observed entities and geometry.
  • Noisy observations and probabilistic grounding: attach confidence to the inferred relations and require sufficient confidence in the predicates needed for execution. Uncertain evidence could still inform policy advice; corrective action would wait for adequate support and be checked against subsequent observations.
  • Similarity-based matching: retrieve a bounded set of potentially relevant records using similarity, then check each candidate's current bindings, prerequisites, legality, and budget. Similarity would broaden candidate discovery without itself granting execution authority.
  • Efficiency: compute the observation grounding once per step and reuse it across the bounded candidate set. Final admission would remain a compact set of predicate and budget checks, without an auxiliary recovery-generation call. Grounding and retrieval overhead would be measured separately from admission cost; this design preserves the lightweight symbolic check while adding uncertainty-aware inputs.

Current mechanism and verified diagnostics. The extension above is a design proposal. The following probes evaluate the existing discrete controller's matching and authorization behavior.

PMR applies exact comparison to normalized, recovery-relevant relations, such as pending target types and resource occupancy, rather than to the full observation text. It then binds the supported effects to current entities. The intended behavior is to preserve matching when these relations remain unchanged and reassess execution when a required condition changes. We tested both behaviors with fixed-checkpoint controller probes.

We first replayed all 3,553 recorded checkpoints from 134 unseen ALFWorld tasks, reproducing the original actions and control metadata. Each probe independently starts from its original checkpoint, with the memory bank and controller fixed and the recorded policy proposal retained. For entity renaming, identifiers are changed consistently across the observation, controller state, and proposal. We applied two such maps and, separately, two observation additions: “The room is quiet.” and “There is nothing else to report.” Each transformation family therefore contains 3,553 checkpoints × two variants = 7,106 probes.

Controlled changeProbesResult
Consistent entity renaming7,106Projection, matching outputs, and admission decisions preserved in all probes
Task-irrelevant observation text7,106Projection, matching outputs, admission decisions, and action preserved in all probes
Withdraw admitted action's legal permission200Targeted recovery action no longer authorized in all probes

Behavior at recovery decisions. The 200 checkpoints that originally authorized recovery yield 400 probes for each transformation family. Matching outputs and admission decisions were preserved in all 400/400 probes in each family. In 24 renaming probes, changing identifiers changed the ordering used to break ties between equally ranked candidates. The selected action used a different object of the same type, with the same operator and supported effect; both actions were legal, and matching evidence and admission status remained unchanged.

The permission-withdrawal test covers 200 recovery decisions across 65 tasks and all three branches. The targeted recovery action lost authorization in every case: 110 probes selected another legal recovery action, and 90 authorized no recovery action. Together, these checks show stable matching under the tested surface changes and renewed authorization checks when an execution condition changes.

Reference: Section 3.2 (p. 5), Appendix C.1 (p. 19), and E.2 (pp. 24–25) cover normalization, binding, and matching.

The proposed extension would retain the distinction demonstrated by these diagnostics: matching remains stable under the tested surface-form changes, while changes to execution conditions require renewed authorization.

AuthorsOpenReview ↗

Response to Reviewer NVZ2 (3/4): ReAct and Reflexion

Foundational baselines — Weakness 3 / Question 3 (ReAct and Reflexion).

We have added ReAct and Reflexion (up to three attempts) to the revised manuscript's main comparison, with detailed results in the appendix.

The primary comparisons use the frozen Qwen3-4B-Instruct-2507 policy and the evaluation tasks from the original submission. Each added setting has three independent runs. SR is mean ± sample SD; costs are means per task including unsuccessful tasks, and K denotes thousands of tokens. PMR values are those reported in the original submission.

ReAct and Reflexion. ReAct interleaves reasoning with actions in a single attempt. Reflexion can make up to three attempts on the same task: after a failed attempt, the environment is reset and verbal reflections are carried into the next attempt. Each attempt retains the 50/30/128-action limit on ALFWorld/TextCraft/BabyAI. Allowing up to three attempts therefore gives Reflexion a cumulative ceiling of 150/90/384 actions per task; the per-attempt limit is unchanged. Evaluation stops as soon as the task succeeds, so solved tasks receive no further attempts. Final success counts tasks solved within these three attempts, and reported costs accumulate all attempts actually used, including reflection requests.

SettingMethodSR (%)Actions/taskCalls/taskTokens/task (K)
ALFWorld seenReAct51.43 ± 4.3434.9035.0373.99
ALFWorld seenReflexion (≤3 attempts)59.29 ± 1.8968.8170.52169.95
ALFWorld seenPMR76.9 ± 1.825.425.462.8
ALFWorld unseenReAct53.73 ± 2.9932.8833.0070.85
ALFWorld unseenReflexion (≤3 attempts)67.91 ± 4.4862.3363.75147.72
ALFWorld unseenPMR77.6 ± 1.325.125.174.9
TextCraftReAct69.33 ± 1.5314.7514.7518.74
TextCraftReflexion (≤3 attempts)75.33 ± 2.0830.4931.0645.69
TextCraftPMR80.0 ± 0.013.513.523.0
BabyAIReAct47.67 ± 1.5380.9680.9656.18
BabyAIReflexion (≤3 attempts)54.33 ± 0.58197.26202.18200.42
BabyAIPMR58.0 ± 1.073.573.552.5

Reflexion benefits from additional attempts, as the cumulative success rates show:

Setting / policy≤1 attempt≤2 attempts≤3 attempts
ALFWorld seen46.19 ± 2.5156.90 ± 1.6559.29 ± 1.89
ALFWorld unseen52.49 ± 4.1165.42 ± 3.7667.91 ± 4.48
TextCraft69.00 ± 5.2074.00 ± 2.0075.33 ± 2.08
BabyAI46.67 ± 2.5251.67 ± 1.5354.33 ± 0.58
TextCraft / Llama-3.2-3B23.00 ± 0.0024.67 ± 0.5824.67 ± 0.58
TextCraft / Qwen2.5-3B27.33 ± 1.1528.33 ± 2.0828.67 ± 2.52

On ALFWorld unseen, cumulative actions rise from 32.82 to 49.64 to 62.33 per task. PMR reaches 77.6%, versus 67.91% for final Reflexion, with 59.7% fewer actions, 60.6% fewer calls, and approximately 49.3% fewer tokens. Across all four primary settings, PMR has higher success and lower action, call, and token costs than three-attempt Reflexion. The cross-policy TextCraft comparisons also favor PMR in success: 40.3%/37.3% versus Reflexion's 24.67%/28.67% under Llama-3.2-3B/Qwen2.5-3B. The manuscript reports final outcomes; the table above makes the retry gains explicit.

PMR also exceeds ReAct in success on all four primary splits while using fewer environment actions and model calls. The added comparisons therefore show that the gains extend to both single-attempt reasoning and multi-attempt recovery, with all retries and reflection requests included in the reported costs.

Because Reflexion enters the main comparison, the margin to the runner-up is narrower than in the original submission; PMR's own success rates are unchanged. Concretely, the abstract range changes from 3.3–22.6 to 3.3–17.6 percentage points: the upper end is now PMR (76.9%) minus Reflexion (59.29%) on ALFWorld seen (17.6 pp), where Reflexion replaces AdaMEM-HIGH as runner-up. Every PMR result in the manuscript is the same number as in the original submission.

Added: Sections 4.1–4.2 (p. 7) specify the methods and cumulative accounting; Figure 2 and Table 1 (p. 8) add ReAct and final Reflexion. Appendix B.2–B.3 (pp. 16–18) gives the protocol; Appendix F.1 (Tables 5–7, pp. 34–36; Figure 12, p. 35) and Tables 9, 14–17 (pp. 38, 41–44) give aggregate and per-run results.

The attempt-wise breakdown is reported here. Part 4 presents ExpeL and both LATS budgets.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank You and Closing Remarks

Dear Area Chair and Reviewers,

Thank you for carefully reviewing our paper and taking the time to discuss it with us. As the discussion concludes, we would like to briefly restate the central message of our work.

PMR addresses a limitation of relevance-based experience reuse: a related example or useful strategy does not by itself establish which local effect is supported by past observations, or whether a corrective action is applicable now. PMR compiles local evidence into conditional recovery records and uses current-state checks to authorize corrective actions, helping agents recover from local impasses and restore progress.

Two distinctions guide the design. Evidence of a difficulty and evidence of a useful effect are established separately, then linked under compatible conditions across episodes. The experience revealing an obstacle need not contain its solution, and unsuccessful episodes can contribute useful local evidence. Retrieved advice is also distinct from execution authority: supported recovery candidates can replace the policy’s proposed action after current-state checks, with subsequent observations determining whether progress occurred.

Three findings support this approach. Mixing successful and unsuccessful episodes achieved higher task success than either source alone at the same episode count. With execution capabilities fixed, removing relevant support reduced successful escapes at fixed checkpoints, while equal-size unrelated removal left escape outcomes unchanged. With compiled evidence and the retrieval index fixed, enabling direct-effect recovery improved task success while reducing model calls and online tokens.

Across the diverse long-horizon tasks evaluated, PMR achieves higher mean task success with fewer online actions and model calls than the evaluated baselines, without policy updates or auxiliary recovery-generation calls.

Together, these results demonstrate the practical value of PMR: helping LLM agents recover from local impasses and restore progress while reducing wasted interaction.

Thank you again for your thoughtful consideration and constructive discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.