acceptodds
Under review as a conference paper at ICLR 2027

Learning Compact State Abstractions for LLM Agents

Abstract

LLM agents commonly use the full interaction history as a proxy for state, retaining all past observations and actions regardless of whether they remain relevant. We instead ask what information an agent actually needs to carry forward. We define agent state as a compact abstraction of interaction history that preserves the competence of a fixed reference agent, with the minimal such abstraction as the ideal. Because matching the reference agent's behavior is unnecessarily restrictive and difficult for black-box policies, we define sufficiency through task performance rather than behavior matching. We represent state abstractions as deterministic recursive programs and search over them using an LLM-guided genetic algorithm. Across ALFWorld and ScienceWorld, the learned abstractions preserve near-reference task success while reducing mean policy input by up to 71% and 57%, respectively. The resulting programs are directly inspectable, providing an explicit account of what information is retained, updated, and discarded during interaction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
State Abstraction, Reinforcement Learning, LLM Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hgFDOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a method to search for a python program to compress the text history of LLM in a task specific way. In the setup, a LLM work as an agent that receive text observation and return text action, which form the memory. The python program is used to process the memory into a more compact form but still aim to finish the task at hand. The program modification is done via another LLM.

The experiment results show that the memory length is reduced significantly while keeping a high task success rate.

Critical strengths
  1. The method produces a practical, low-cost method. The learned memory is a plain Python function that runs without any LLM call at deployment. Summarization approaches such as ACON and the LLM Summarizer instead require online LLM inference whenever they compress.

  2. The experiments show that the resulting memory is still informative and rich for specific tasks,

  3. Also the memory in text as well as the python program is interpretable and bring insights on tasks at hand.

Critical weaknesses
  1. The methodology has unclear novelty. Using LLM to search for program for memory manipulation is not novel. For example: Learning to Continually Learn via Meta-learning Agentic Memory Designs (ALMA)

  2. With the main goal being the memory saving, the tasks used all induce pretty managable full history length. In this case, it is not clear to me the point of trading memory reduction at the cost of even 1% lower success rate.

  3. It seems that modifying text history is not KV cache friendly and may induce re-computation.

  4. What do the existing search records show about improvement beyond the first LLM-generated structured updater, and which improvements depend on population-based search rather than ordinary execution-guided code repair? If the task specification is given in great detail, it is possible that an LLM can directly give a pretty good program in one go?

Questions for authors
  1. How does the method differ from ALMA beyond the choice of a deterministic recursive update?

  2. Can authors discuss the impact on the usage of KV cache and inference efficiency?

  3. Can authors provide some abliation on the effectivenss of different search components?

3 comments
AuthorsOpenReview ↗

Response by Authors [1/2]

Opening statement. We thank the reviewer for the review, which credits the method as practical, low-cost, and inspectable. The revision addresses each concern. We first summarize the main points of our response, and then address each weakness and question in detail.

  • ALMA. Motivated by the state abstraction literature in RL, our contribution is learning a policy-relative, within-episode state that is sufficient for control and replaces the interaction history. ALMA learns memory designs for continual learning across tasks.
  • Success versus memory. Three runs on all 254 held-out instances show no systematic difference in success from full history. On BabyAI, ϕ\phi cuts mean policy input tokens by 95% with no detectable loss in success.
  • KV cache. Full history allows prefix-cache reuse. ϕ\phi instead keeps the context and KV memory bounded.
  • Search. The first generated program is feasible in only 2 of 10 families. All 26 later improvements originate from evolutionary proposals. Of these, 16 require reflector-guided repair.

We hope these additions address the reviewer's concerns and would be grateful if the reviewer would reconsider their score.

AuthorsOpenReview ↗

Response by Authors [2/2]

W1. Novelty relative to ALMA.

Our contribution is learning a policy-relative, within-episode state sufficient for control. A candidate state is admissible only if it preserves the fixed reference agent's success within a tolerance. Among admissible states, we optimize compactness. This follows a policy-relative notion of sufficiency that builds on classical state abstraction (see Sections 2.2–3). Proposition 1 (Section 3.1) shows that this criterion is a relaxation of both behavior and performance matching.

ALMA addresses a different problem. It meta-learns memory designs for continual learning across tasks and selects programs based on how well they help an agent improve over a stream of tasks.

Our method asks what information from the current episode must be retained so that the same fixed LLM agent preserves its competence when the raw history is removed. We added this distinction to Sec. 2.4.

W2. Manageable history lengths and trading success for memory.

  • Success is preserved (see Appendix D.1). Across 254 held-out ALFWorld and ScienceWorld instances, ϕ\phi is +1.7 pp relative to full history when averaged over three runs. The 95% CI is [−1.8, +5.3].
  • Long horizons (see Appendix E). On BabyAI GoTo, full history reaches 27.5k tokens per call while ϕ\phi remains at about 250. Mean policy input tokens fall by 95%, with no detectable loss in success (exact McNemar test, p=0.18p = 0.18).
  • Savings scale with the horizon (see Appendix E.1). Full history grows by 24–36 tokens per step in every original family. In 7 of 10 families, the state produced by ϕ\phi grows by only 2–8 tokens per step.

Memory saving is not the paper's primary goal. It is a consequence of identifying a compact state sufficient for control.

W3. KV-cache friendliness.

Full history is cache-friendly because its prefix never changes. However, its KV cache and attention context grow at every step.

With ϕ\phi, the static instructions and task remain cacheable. Only the bounded recurrent state is re-encoded. On BabyAI, this state is about 250 tokens compared with 27.5k tokens for full history. The trade-off is prefix reuse versus bounded context and KV memory.

W4. Improvement beyond the first program and the role of search.

  • One-shot generation is generally insufficient. The first LLM-generated program satisfies the feasibility constraint in only 2 of 10 task families.
  • Every later improvement comes from evolutionary search (see Appendix F.1). Of the 26 best-so-far improvements, 10 come directly from mutation or crossover proposals. The other 16 improve only after reflector-guided repair of such a proposal. In several families, including Pick & Place, Heat & Place, and Cool & Place, the final program appears only after multiple generations.

Q1. Difference from ALMA.

ALMA meta-learns memory designs for continual learning across tasks. We learn a policy-relative, within-episode state that must preserve a fixed agent's success, optimizing compactness among such states (see W1 and Section 2.4).

Q2. KV cache and inference efficiency.

Full history allows prefix-cache reuse, but its KV cache and attention context grow at every step. With ϕ\phi, the static prefix stays cacheable and the recurrent state stays bounded. On BabyAI, the state is about 250 tokens compared with 27.5k tokens for full history (see W3).

Q3. Effectiveness of search components.

The first generated program is feasible in only 2 of 10 families. Among the 26 later improvements, 10 arise directly from evolutionary proposals and 16 require reflector-guided repair. This shows that the gains are not explained by one-shot generation and that both evolutionary proposal generation and repair contribute (see W4 and Appendix F.1).

As the discussion period closes today, we would be grateful if the reviewer could take a look at our response and the revised paper. We are happy to answer any remaining questions.

Reviewer_297yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for learning code-based compact state abstraction functions for LLM agents optimized through LLM-guided genetic search. Experiments on ALFWorld and ScienceWorld show that the learned abstractions largely preserve task success while reducing average policy input tokens by up to 71% and 57% for some subtasks.

Critical strengths
  1. The paper presents an interesting application of LLM-guided evolutionary search to learning compact state abstraction functions for LLM agents.
  2. The paper is well-written and easy to follow.
Critical weaknesses
  1. State abstraction and structured memory for LLM agents have been widely studied, including both prompted and learned approaches with different representations (e.g., MCMA and StateAct). However, results in this paper lack sufficiently discussing and evaluating these closely related approaches in terms of test-time token efficiency, learning efficiency, code abstraction vs other structures, state as additional input vs having state totally replacing observations, etc.

  2. Evaluation is done with only one LLM backbone on two benchmarks. Actually benchmarks like Alfworld do not feature long-horizon trajectories, and observations are rather short. Therefore they do not adequately demonstrate the benefits of state abstraction in long-horizon or observation-heavy environments (web navigation, for example).

  3. At the same time, ALFWorld and ScienceWorld have relatively structured observations that can be parsed with deterministic rules. It remains unclear whether the proposed program-based approach can effectively handle more open-ended tasks with diverse and semantically complex observations.

  4. Despite the relatively structured tasks, several learned programs are surprisingly long (e.g., 428 LOC for Heat & Place). This raises concerns about unnecessary complexity and potential overfitting to environment-specific observation patterns. The paper lacks sufficient analysis of program complexity and robustness.

Questions for authors
  1. How does the method perform on longer-horizon tasks or environments with substantially larger and more diverse observations? Does it remain effective with different LLM backbones?
4 comments
AuthorsOpenReview ↗

Response by Authors [1/3]

Opening statement. We are thankful for the reviewer's feedback. The revision addresses each concern. We first summarize the main points of our response, and then address each weakness and question in detail.

  • MCMA and StateAct. Motivated by the state abstraction literature in RL, our contribution is learning a policy-relative, within-episode state that is sufficient for control and replaces the interaction history.
  • Backbone and horizon. On ALFWorld, a ϕ\phi learned with Luna, used unchanged with Qwen, shows no systematic difference in success from Qwen with full history, while reducing mean policy token input by up to 69%. On a long-horizon BabyAI task, ϕ\phi reduces mean policy input tokens by 95%.
  • Open-ended observations. ϕ\phi is a general-purpose program. It can parse machine-generated observations directly or use a language model for free-form text.
  • Overfitting. On ALFWorld, success on environments unseen during search (0.86) is no lower than on seen ones (0.81), and shows no systematic difference from full history on either, so there is no evidence of overfitting. Hiding seven of eight state-field groups also significantly lowers success.

We hope these additions address the reviewer's concerns and would be grateful if the reviewer would reconsider their score.

AuthorsOpenReview ↗

Response by Authors [2/3]

W1. Relation to MCMA and StateAct.

Our contribution is not a new memory mechanism, but a new problem formulation. We learn the state representation itself. The learned state is a compact within-episode representation that is sufficient for a fixed policy to preserve its task success, and it replaces the raw interaction history. This builds on classical state abstraction (see Section 2.2). Reduced policy input is a consequence rather than the primary objective.

DimensionMCMAStateActOurs (ϕ\phi)
What is representedCross-episode procedural memoryPrompted task stateWithin-episode state sufficient for a fixed policy
RoleAdded to historyAdded to historyReplaces history
Test-time tokensIncrease policy inputIncrease policy inputUp to 71% less on ALFWorld, 57% on ScienceWorld, and 95% on BabyAI
Learning4B copilot with SFT + DPOAnnotated few-shot promptingBlack-box search with no gradients or model weights
RepresentationLearned memory modelGenerated languageExecutable, inspectable program

Neither MCMA nor StateAct formulates state identification under our sufficiency-and-compactness criterion. We therefore treat them as related work rather than direct baselines.

ACON follows the same comparison logic. It is the context-compression baseline closest to our setting and compares against history-reduction methods rather than StateAct. We added this discussion to Sec. 2.1.

W2. One LLM backbone, short horizons.

  • Backbone (see Appendix D.4). On ALFWorld, Luna-learned ϕ\phi programs are used unchanged with Qwen-3.5-397B. They show no systematic difference from Qwen full history. The pooled difference is −1.5 pp with a 95% CI of [−5.9, +2.5] over n=202n=202 instances. Mean policy input tokens are reduced by up to 69%.
  • Scope. Transfer is not guaranteed because sufficiency is defined relative to the policy used during search. A different policy may rely on information that ϕ\phi was not required to preserve. This mismatch may be especially pronounced for a weaker agent. A weaker agent may depend more heavily on explicit information that a stronger agent can infer. Rerunning the search is the intended approach when changing agents.
  • Long horizon (see Appendix E). On BabyAI GoTo with a 576-step cap, full history grows to 27.5k tokens per call while ϕ\phi remains at about 250. Mean policy input tokens fall by 95%, with no detectable loss in success (exact McNemar test, p=0.18p = 0.18).
  • Original benchmarks (see Appendix E.1). Full history grows by 24–36 tokens per step in every family. In 7 of 10 families, the state produced by ϕ\phi grows by only 2–8 tokens per step.
  • Observation diversity. ScienceWorld includes experimental progress, object properties, and scientific measurements. These require different state information from ALFWorld.

W3. Structured observations.

The program class is not restricted to structured observations. Programs are a highly expressive representation class. Since ϕ\phi is a general-purpose program, it can in principle implement any computable state update, using arbitrary control flow, data structures, libraries, and tools.

Machine-generated observations such as simulator text, HTML pages, and API responses can be parsed directly. For free-form text, ϕ\phi can call a language model to extract the information needed by the state. This trades some of its zero-inference cost for broader coverage.

W4. Program length and overfitting.

  • Program length is not state size. Heat & Place has 428 program lines but only 18 state fields.

  • Its rules apply to the whole task family, not to specific rooms or episodes. It handles goal phrasing variants, normalizes object and receptacle names, detects action outcomes, and tracks container and object state. It also tracks the current task phase.

  • No evidence of overfitting to training environments (see Appendix D.3).

    ALFWorld splitnϕ\phi − full history95% CI
    Seen68+1.5 pp[−4.4, +7.4]
    Unseen134−1.5 pp[−7.5, +4.5]

    The agent using ϕ\phi reaches 0.81 success on seen environments and 0.86 on environments never encountered during search. Success on unseen environments is no lower than on seen ones, and matches full history on both, so there is no evidence of overfitting to the environments used during search.

  • Overall success is preserved (see Appendix D.1). Across all 254 held-out instances, ϕ\phi is +1.7 pp relative to full history when averaged over three runs. The 95% CI is [−1.8, +5.3].

  • The maintained state contributes to performance (see Appendix F.4). Hiding seven of eight field groups significantly lowers success. The largest drops come from progress and control, recent observations, and exploration memory.

AuthorsOpenReview ↗

Response by Authors [3/3]

Q1. Longer horizons, larger observations, other backbones.

BabyAI provides the long-horizon evaluation. Qwen provides the cross-backbone evaluation. ScienceWorld provides more diverse observation semantics. W3 explains how the program representation extends to more open-ended observations.

We hope these additions address the reviewer's concerns and would be grateful if the reviewer would reconsider their score.

As the discussion period closes today, we would be grateful if the reviewer could take a look at our response and the revised paper. We are happy to answer any remaining questions.

Reviewer_PnMuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

LLM agents usually reread their whole history at every step. This paper asks what the agent actually needs to remember. It defines the agent's "state" as a short summary of the history that lets a fixed LLM agent succeed about as often as it does with the full history. The summary is a Python program that updates the state after each step and discards the raw text. An LLM-guided genetic search writes and improves these programs. A second LLM explains failures, and the search keeps only programs that stay within 10% of full-history success while using fewer tokens.

Contributions

  • A definition of agent state for black-box LLM agents, based on keeping task success rather than copying behavior. It is linked to classic state abstraction work in RL.
  • A search method that learns these state-update programs from interaction alone.
  • Experiments showing token savings, plus inspection of what the learned programs keep.
Critical strengths
  • Good framing. Asking "what must the agent remember?" instead of "how do we shrink the context?" is a clean and well-motivated idea, grounded in prior work on state abstraction.
  • Practical success test. Judging the state by task success, not by matching actions, makes sense for black-box models where many paths can succeed.
  • Cheap to run. The state is plain code applied from the first step, with no extra LLM calls. Summarizer baselines need an LLM call each time they compress.
  • Readable. You can look at the program and see what is kept and what is dropped.
  • Honest. The paper shows the cases where the method failed to compress and admits that token counts and success affect each other.
Critical weaknesses
  • The main motivation is not tested. The paper is motivated by histories that become too long. But full-history inputs here are only about 260 to 560 tokens per step. There is no long-horizon experiment, where the method would matter most.
  • Results may be noise. Test sets are small (12 instances for Forces and Genetics), and there is one seed, one rollout per candidate, and no error bars. Many gaps of 2 to 8 points could be chance. Forces is a warning sign. Its learned state is just the last 40 steps, almost the same as full history, yet success jumps from 0.42 to 0.75.
  • Missing simple baselines. There is no "keep the last k steps" baseline and no hand-written state tracker. Two learned programs are "keep the last k steps," so these baselines are needed.
  • Some claims go too far. "Minimal" state is never measured, and no test removes fields one at a time. "Interpretable" is claimed but not evaluated, and a 400-line regex program is hard to read. The paper uses the language of formal state abstraction but gives no theoretical result.
Questions for authors
  • How does the method do on tasks with much longer histories (thousands of tokens)?
  • What do results look like across multiple seeds, with error bars?
  • How does it compare to keeping the last k steps, for several values of k?
  • How much does the search cost in LLM calls and environment runs?
  • Does a program learned with one LLM work with another LLM?
  • Which state fields matter most? What happens when each one is removed?
4 comments
AuthorsOpenReview ↗

Response by Authors [1/3]

Opening statement. We are thankful for the reviewer's feedback. The review credits the framing as "clean and well-motivated" and the method as practical and honest. The revision directly addresses each concern. We first summarize the main points of our response, and then address each weakness and question in detail.

  • Long horizons. On BabyAI, full history reaches 27.5k tokens per call while ϕ\phi remains at about 250, a 95% reduction in mean policy input tokens, with no detectable loss in success.
  • Noise. Across three runs on all 254 held-out instances, the agent using ϕ\phi matches full-history success (+1.7 pp, 95% CI [−1.8, +5.3]). ϕ\phi also has the lowest run-to-run variance.
  • Token-matched recent-history baseline. ϕ\phi beats a token-matched last-kk window baseline by +10.0 pp with a 95% CI of [+5.9, +13.9].
  • State field importance. Hiding seven of eight field groups significantly lowers success, with drops of up to 36.1 pp.
  • Search cost. The search uses 860 meta-LLM calls (calls to the reflector and proposer) and 5,948 evaluation rollouts across the 10 families.
  • Transfer to another LLM. On ALFWorld, a Luna-learned ϕ\phi is used unchanged with Qwen-3.5-397B. It shows no systematic difference in success from Qwen with full history while reducing mean policy token input by up to 69%.

The core contribution is a policy-relative notion of agent state that builds on classical state abstraction. We learn a recursively maintained abstraction of the history that is sufficient for a fixed agent's success and replaces the raw history. Reduced policy input is a consequence of identifying this state rather than the primary objective. We hope these additions address the reviewer's concerns and would be grateful if the reviewer would reconsider their score.

AuthorsOpenReview ↗

Response by Authors [2/3]

W1. The main motivation is not tested (short histories).

  • BabyAI GoTo (see Appendix E.3). We evaluate 3×3 rooms with a 576-step cap. Full history grows by 48.5 tokens per step and reaches 27.5k tokens per call at the cap. This is roughly 50× the history length in the original benchmarks.
  • ϕ\phi remains bounded. From step 25 onward, it stays at about 250 tokens per call. This reduces mean policy input tokens by 95% and reduces per-episode input from 1.89M to 66k tokens.
  • Success. There is no detectable loss in success relative to full history (exact McNemar test, p=0.18p = 0.18).
  • Original benchmarks (see Appendix E.1). Full history grows by 24–36 tokens per step in every family. In 7 of 10 families, the state produced by ϕ\phi grows by only 2–8 tokens per step.

W2. Results may be noise (Forces).

MethodSuccess (mean ± std, 3 runs)ϕ\phi − method (95% CI)Per run
ϕ\phi0.819 ± 0.007––
Full history0.802 ± 0.010+1.7 pp [−1.8, +5.3]+1.2 / +0.4 / +3.5
Token-matched window0.719 ± 0.018+10.0 pp [+5.9, +13.9]+10.6 / +10.6 / +8.7

Across all 254 held-out instances, we do not detect a systematic difference between ϕ\phi and full history. ϕ\phi also has the smallest pooled run-to-run standard deviation, and its standard deviation is lower than full history's in 6 of 10 families, equal in 3, and higher in only 1.

Forces is noise, as the reviewer suspected. ϕ\phi retains the last 40 steps, and episodes are capped at 40 steps. Thus, ϕ\phi and full history carry the same content. Full history alone scores 0.42, 0.92, and 0.58 across the three runs. The original 0.42-to-0.75 gap therefore should not be interpreted as an improvement from the abstraction.

W3. Missing simple baselines.

  • Token-matched last-kk (see Appendix D.2). We choose kk separately for each family so that the window matches ϕ\phi's token budget. ϕ\phi wins by +10.0 pp with a 95% CI of [+5.9, +13.9] over n=254n=254 instances. The advantage holds in every run.
  • Largest gains. Pick & Place improves by +25.5 pp with k=2k=2. Heat & Place improves by +24.7 pp with k=6k=6. Cool & Place improves by +11.7 pp with k=6k=6.
  • Hand-written tracker. A hand-written tracker requires domain knowledge about what each task family needs the agent to remember. It also requires programming expertise to extract this information reliably from the observations. This effectively supplies the task-specific state representation that our method is designed to discover. We therefore add a domain-agnostic recent-history baseline that does not inject this supervision.

W4. Some claims go too far.

  • "Minimal." Sec. 3.1 defines the minimal state as the ideal. The paper explicitly states that the search does not certify minimality.

  • Field importance (see Appendix F.4).

    Hidden field groupChange in success95% CI
    Progress & control−36.1 pp[−42.3, −29.7]
    Recent observations−29.3 pp[−36.6, −22.1]
    Exploration memory−29.1 pp[−35.6, −22.7]
    Object & container knowledge−15.5 pp[−20.3, −10.6]
    Last action / result−10.0 pp[−17.1, −2.9]
    Inventory−8.5 pp[−12.8, −4.3]
    Current location−5.2 pp[−9.7, −0.8]
    Target / goal+0.5 pp[−3.3, +4.1]

    Seven of eight groups significantly affect success. On Clean & Place, hiding the flags that record which objects are already clean drops success from 0.73 to 0.04. Target / goal is the only group with no measurable effect, as the task prompt already states the goal.

  • "Interpretable." The paper now consistently uses inspectable. Each state variable and update rule can be read directly from the program.

  • Theory. The revision adds a formal result in Sec. 3.1 (Proposition 1, proof in Appendix G). Behavior matching, performance matching and our success criterion form a sequence of relaxations with nested feasible sets. Our criterion therefore admits every abstraction the stricter criteria admit, and its optimal state size is no larger (Corollary 1).

AuthorsOpenReview ↗

Response by Authors [3/3]

Q1. Longer histories.

On BabyAI, full history reaches 27.5k tokens per call while ϕ\phi remains at about 250, a 95% reduction in mean policy token input, with no detectable loss in success (see W1 and Appendix E).

Q2. Multiple seeds, error bars.

We evaluate ϕ\phi, full history, and the token-matched window three times. We report mean ± std per family and pooled, together with paired bootstrap confidence intervals (see W2 and Appendix D.1).

Q3. Last kk for several kk.

We use one token-matched kk per family rather than a sweep. This keeps the token budget fixed. Across families, matched kk ranges from 2 to 47. ϕ\phi's advantage is largest on Pick & Place (+25.5 pp at k=2k=2) and Heat & Place (+24.7 pp at k=6k=6) (see W3 and Appendix D.2).

Q4. Search cost.

We define a meta-LLM call as an LLM call that proposes, reflects on, or repairs candidate programs during search. Across the 10 families, the search makes 860 meta-LLM calls using 17.7M tokens. It also uses 5,948 environment rollouts to evaluate candidate programs. On BabyAI GoTo, it makes 52 meta-LLM calls and 324 rollouts. Appendix F.2 reports the cost per family.

Q5. Transfer to another LLM.

On ALFWorld, a ϕ\phi learned with GPT-5.6 Luna is used unchanged with Qwen-3.5-397B. It shows no systematic difference in success from Qwen with full history. The pooled difference is −1.5 pp with a 95% CI of [−5.9, +2.5] over n=202n=202 instances. Mean policy token input is reduced by up to 69%.

Transfer is not guaranteed because sufficiency is policy-relative. Another policy may rely on information that ϕ\phi was not required to preserve. This mismatch may be especially pronounced for a weaker agent. A weaker agent may depend more heavily on explicit information that a stronger agent can infer. Rerunning the search is therefore the intended approach when changing agents.

Q6. Which fields matter.

Hiding seven of eight groups significantly lowers success. The largest drops come from progress and control (−36.1 pp), recent observations (−29.3 pp), and exploration memory (−29.1 pp). Only target / goal has no measurable effect (see W4 and Appendix F.4).

As the discussion period closes today, we would be grateful if the reviewer could take a look at our response and the revised paper. We are happy to answer any remaining questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response by Authors

We thank the reviewers for their careful reading and thoughtful feedback. The revision adds new experiments and analyses that address the shared concerns. We present the main facts below. Full details are in the revised paper, with all changes marked in blue.

  1. Long-horizon evaluation (see Appendix E). On BabyAI GoTo, full history reaches 27.5k tokens per call while our approach ϕ\phi remains at about 250 tokens. ϕ\phi reduces mean policy input tokens by 95% and uses 29$\times$ fewer input tokens per episode, with no detectable loss in success relative to full history (exact McNemar test, p=0.18p = 0.18). On ALFWorld and ScienceWorld, full history grows by 24–36 tokens per step in every family. In 7 of 10 families, the state produced by ϕ\phi grows by only 2–8 tokens per step.

  2. Success is preserved, with the lowest run-to-run variance (see Appendix D.1).

    MethodSuccess (mean ± std, 3 runs)ϕ\phi − method (95% CI)
    ϕ\phi0.819 ± 0.007–
    Full history0.802 ± 0.010+1.7 pp [−1.8, +5.3]
    Token-matched window0.719 ± 0.018+10.0 pp [+5.9, +13.9]

    The agent using ϕ\phi shows no systematic difference in success from full history. It also has the smallest run-to-run standard deviation of the three methods.

  3. ϕ\phi beats a token-matched recent-history baseline by 10.0 pp (see Appendix D.2). The advantage holds in every run. The largest gains are on Pick & Place (+25.5 pp), Heat & Place (+24.7 pp), and Cool & Place (+11.7 pp).

  4. The learned state carries what the agent needs (see Appendix F.4). Hiding seven of eight field groups significantly lowers success. The largest drops come from progress tracking (−36.1 pp), recent observations (−29.3 pp), and exploration memory (−29.1 pp). On Clean & Place, hiding the flags that record which objects are already clean drops success from 0.73 to 0.04.

  5. Generalization to unseen environments (see Appendix D.3). On ALFWorld, success shows no systematic difference from full history on either seen or unseen environments.

  6. The search does real work (see Appendices F.1 and F.2). The first LLM-generated program is feasible in only 2 of 10 families. All 26 later improvements originate from evolutionary proposals. Of these, 16 require reflector-guided repair.

  7. Formulation and related work (see Sections 2 and 3.1). We now discuss MCMA, StateAct, and ALMA. Our objective is a policy-relative state that is sufficient for control and replaces the interaction history. This differs from cross-episode memory and prompted state tracking. The revision also adds a result in Sec. 3.1 showing that our success criterion relaxes performance matching, which in turn relaxes behavior matching, so it admits more compact states than either.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.