acceptodds
Under review as a conference paper at ICLR 2027

Actions Should EARN Their Credit: Test-Time Adaptation and Route Consolidation for Frozen LLM Agents

Abstract

LLM agents can use feedback from repeated attempts to improve their decisions without updating model parameters. However, positive action credits can continue to favor actions that repeatedly achieve the same score, even when higher scores are achievable. Under a fixed interaction budget, trying new actions in the final attempts can also lower performance. We introduce EARN, a method that uses past experience to guide further progress and retain earlier performance through conditional reuse of effective action sequences. EARN reduces action credits for reaching scores repeatedly attained in earlier episodes and assigns larger credits to action segments that reach higher scores. These credits adjust the probabilities of LLM-proposed actions. During the final episodes, EARN directly replays stored actions when the current context matches the recorded context and the actions remain available, bypassing LLM candidate generation. We evaluate performance at the end of the interaction budget using the average score over the final four episodes. Across Jericho, ScienceWorld, and Crafter, EARN achieves the best score on this metric in 31 of 33 task–budget settings among the evaluated methods, including ties. Across the evaluated Jericho budgets, replay bypasses 78.9–86.3% of candidate-generation calls in the final four episodes, averaged across games.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test Time Learning, Large Language Models, Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_43VGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces EARN, a training-free adaptation method for frozen LLM agents. It combines progress-relative block-wise credits, step-wise feedback, and conditional route replay to improve performance over repeated attempts. Across Jericho, ScienceWorld, and Crafter, EARN achieves the highest or tied-highest final-four average score in 31 of 33 task–budget settings.

Critical strengths
  1. EARN addresses both progress beyond previously attained scores and reliable reuse of effective behavior. Combining action-level credit adjustment with conditional route replay provides a practical approach without updating model parameters.
  2. Evaluation across three complementary environments supports the approach’s empirical value. The Jericho ablations indicate that both block-wise credit and replay contribute to performance.
Critical weaknesses
  1. Although all methods use frozen models and matched interaction budgets, EARN explicitly uses score-based credit updates, while AWM relies on workflow memory. The paper does not adequately clarify whether baselines receive comparable feedback and adaptation opportunities, or whether their ScienceWorld implementations are appropriately adapted. This limits attribution of the gains to EARN’s design.
  2. The evidence does not sufficiently separate improved adaptation from route reproduction. Replay is enabled in precisely the final-four window used for the primary metric, while ablations cover only two Jericho games.
Questions for authors
  1. In ScienceWorld, do AWM and the other baselines receive the same score feedback and opportunities to accumulate experience as EARN? How is AWM’s workflow induction and retrieval adapted to changing task instances, and what are the respective inference and memory budgets?
  2. How much of EARN’s advantage remains when the strongest baselines are equipped with the same conditional route replay mechanism? ScienceWorld results would help distinguish gains from credit assignment versus route reuse.
3 comments

Weakness 1: "...EARN explicitly uses score-based credit updates, while AWM relies on workflow memory. The paper does not adequately clarify whether baselines receive comparable feedback and adaptation opportunities, or whether their ScienceWorld implementations are appropriately adapted."

Question 1: "How is AWM’s workflow induction and retrieval adapted to changing task instances, and what are the respective inference and memory budgets?"

We clarify that all methods use the same frozen backbone, ScienceWorld scoring protocol, task-instance schedule, and interaction budgets: 10, 20, or 30 episodes per task, with at most 100 actions per episode. Memory persists across episodes within each run, allowing every method to accumulate experience as instances change. We retain each method’s own inference and memory settings, including its auxiliary reflection, workflow-induction, or evaluation calls. Thus, environment-interaction budgets are matched, while each method retains its own inference and memory settings.

AWM receives per-step rewards and scores, together with the final episode score. After each episode, it updates the task’s workflow using the previous workflow and the new trajectory. And AWM calls an LLM to assess the episode’s outcome and update the existing workflow using the trajectory and score feedback. AWM reuses the workflow across different variations of the same task type, providing it alongside the current task description and observation. The agent therefore uses earlier experience while adapting its actions to the current situation.

Question 2: "How much of EARN’s advantage remains when the strongest baselines are equipped with the same conditional route replay mechanism? ScienceWorld results would help distinguish gains from credit assignment versus route reuse."

Response: We have added EvoTest + conditional replay and JitRL + conditional replay. All methods use the same final-four replay window and route-selection, context-matching, and action-availability rules. Each method resumes its native action-selection policy when replay is unavailable. ScienceWorld evaluates different task variations across episodes. As a result, stored routes trigger replay less often than in Jericho and WebShop. We therefore report the replay-controlled comparison on these two datasets.

For Jericho, each entry reports Last-4 / Last / Best-round, where Best-round is the highest episode score within the interaction budget.

EpisodesGameEvoTest + replayJitRL + replayEARN
30DeepHome14.75 / 15 / 1513.50 / 12 / 2019.25 / 20 / 20
30Detective100 / 100 / 100165 / 170 / 180190 / 190 / 190
30Zork I26 / 5 / 4040 / 40 / 4047.50 / 50 / 50
30Zork III1 / 2 / 33 / 3 / 34 / 4 / 4
30ZTUU2.50 / 0 / 59.75 / 12 / 1720.75 / 22 / 22
50DeepHome16 / 15 / 1918.75 / 20 / 2323 / 23 / 23
50Detective232.50 / 120 / 310320 / 320 / 320340 / 340 / 340
50Zork I10 / 10 / 1050 / 50 / 5054 / 54/ 54
50Zork III2.75 / 3 / 33 / 3 / 35 / 5 /5
50ZTUU5 / 5 / 50 / 0 / 527 / 27 / 27

EARN achieves strictly higher Last-4 and Last scores in all 10 settings and the highest Best-round score in every setting, including two ties. Thus, its advantage persists after the strongest baselines receive the same replay mechanism.

We conducted an additional experiment on 10 fixed shopping tasks from a 1,000-product WebShop subset. Each method receives 20 attempts per task, with at most 20 steps per attempt, using the same frozen backbone and task instances. The environment provides a graded score in [0,1][0,1] upon purchase and no intermediate task rewards.

For WebShop, we apply final-four replay to each method. All metrics are averaged equally across tasks.

MethodLast-4 without replay ↑\uparrowLast-4 with replay ↑\uparrowBest-round without replay ↑\uparrowBest-round with replay ↑\uparrow
EvoTest0.86670.86670.86670.8667
JitRL0.87500.93330.93330.9333
EARN0.78330.96670.96670.9667

Replay improves JitRL’s Last-4 score from 0.8750 to 0.9333 and narrows its gap with full EARN from 0.0917 to 0.0333. Best-round scores remain unchanged for all three methods. EARN’s higher Best-round score is already present without final replay. EARN retains the highest Last-4 score when replay is available to all three methods.

Weakness 2: The evidence does not sufficiently separate improved adaptation from route reproduction. Replay is enabled in precisely the final-four window used for the primary metric.

Response: We thank you for raising this question. EARN addresses two complementary objectives: discovering higher-scoring behavior through progress-relative credit assignment and retaining discovered behavior through conditional consolidation. The credits reduce reinforcement of repeatedly attained scores and encourage further progress. During consolidation, stored routes are replayed only while their execution checks remain valid; otherwise, memory-guided action selection resumes.

Consolidation timing. Using the existing Jericho runs, we examined whether each run’s best score had already appeared by episode K−KcK-K_c, where KK is the total episode budget and KcK_c is the consolidation window.

Episode budget KKConsolidation windows KcK_cGames whose best score was observed by episode K−KcK-K_c
304, 63/5, 2/5
504, 6, 8, 104/5, 3/5, 3/5, 2/5
704, 6, 8, 10, 12, 144/5 for every window

These results reveal a timing trade-off: starting consolidation too early may preserve a lower-scoring route. At K=70K=70, 4/5 games had reached their best score before every tested window, allowing more flexible timing. Thus, KcK_c should balance sufficient time for discovery with enough episodes for reliable replay.

Alternative evaluation criteria. We also computed best-found score, max⁡1≤k≤KSk\max_{1 \leq k \leq K} S_k, and average score across all episodes, 1K∑k=1KSk\frac{1}{K}\sum_{k=1}^{K} S_k, for all six methods. The table reports the number of Jericho settings in which each method achieves the highest or tied-highest score.

MethodBest-found score ↑\uparrowAverage score across all episodes ↑\uparrow
FIFO Memory1/153/15
Reflexion0/150/15
AWM0/150/15
EvoTest1/154/15
JitRL6/153/15
EARN13/155/15

EARN achieves the most top-or-tied settings under both criteria. Its best-found advantage is substantial (13/15 versus 6/15 for the next-best method), while its advantage in all-episode average is smaller (5/15 versus 4/15 for EvoTest).

We also examine whether EARN’s full-run best score had already been discovered before different prospective consolidation windows. This occurs in 11/15 settings before the final 4 episodes, 9/15 before the final 6 or 8 episodes, and 7/15 before the final 10 episodes (more than 6/15 for the next-best method).

EARN evaluation cutoffSettings where the full-run best score was already observed
Before the final 4 episodes11/15
Before the final 6 episodes9/15
Before the final 8 episodes9/15
Before the final 10 episodes7/15

These results show that many of EARN’s best-scoring trajectories are already available before consolidation could begin. As the window grows and its starting point moves earlier, fewer settings have reached their eventual best score, highlighting the trade-off between allowing further discovery and beginning consolidation. Together, the results demonstrate EARN’s overall best-found advantage and characterize when its best observed trajectories become available for reuse.

Dear Reviewer 43VG,

As the discussion phase is ending soon, we would greatly appreciate it if you could review our rebuttal and let us know if any concerns remain.

Thank you for your time and thoughtful feedback!

Reviewer_5iP3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EARN, a training-free framework for test-time adaptation in frozen LLM agents. The method addresses two challenges in repeated agent-environment interactions: (1) previously rewarded actions may continue to be reinforced even when they no longer lead to progress beyond an already achieved score, and (2) agents may fail to reproduce previously successful action sequences in later episodes. EARN combines progress-relative block-wise credit attribution, step-wise feedback from all trajectories, and support-controlled action probability adjustment. It introduces conditional route replay in the final episodes, allowing the agent to directly execute previously successful actions. Experiments on Jericho, ScienceWorld, and Crafter show improvements over several memory-based and test-time adaptation baselines.

Critical strengths
  1. The paper is well-motivated. The paper identifies a meaningful limitation of experience-based action selection: actions that repeatedly yield positive rewards may be continually reinforced despite providing little further information about how to surpass previously achieved performance.
  2. EARN enables an LLM agent to adapt its action-selection policy through persistent memory and numerical credit adjustment, without updating the underlying model parameters. Also, its credit signals are constructed from existing environment feedback without additional LLM-based reward evaluation, which makes the method computationally practical.
Critical weaknesses
  1. The primary evaluation metric is the average score over the final four episodes, while the proposed method explicitly activates route replay during exactly these episodes. This design naturally favors the preservation of previously discovered high-performing trajectories. Although optimizing terminal performance is a legitimate objective, the current evaluation does not establish whether EARN improves general learning efficiency or whether its advantage primarily reflects a policy specifically tailored to the evaluation window. Results under alternative consolidation schedules and metrics, such as best-found performance or performance averaged across the entire interaction budget, would help clarify the generality of the improvement.
  2. EARN relies on environment-provided score increases to identify progress-bearing segments and construct action credits. This assumption is relatively well aligned with the selected benchmarks, where task progress is represented through numerical scores or achievement signals. However, many realistic agent tasks provide sparse, delayed, or binary feedback, making it difficult to identify intermediate progress events. The paper does not establish whether its credit assignment remains effective when intermediate score changes are unavailable. An evaluation under sparse-reward conditions would better characterize the method's applicability.
Questions for authors
  1. How sensitive is EARN to the choice of the final consolidation window? Does it maintain its advantage under alternative evaluation criteria, such as best-found score or performance averaged across all episodes?
  2. Could the authors evaluate the proposed framework under sparse-reward conditions would better characterize the method's applicability?
3 comments

Weakness 1/Question 1: "...the current evaluation does not establish whether EARN improves general learning efficiency or whether its advantage primarily reflects a policy specifically tailored to the evaluation window."

Response: We thank you for raising this question. EARN addresses two complementary objectives: discovering higher-scoring behavior through progress-relative credit assignment and retaining discovered behavior through conditional consolidation. The credits reduce reinforcement of repeatedly attained scores and encourage further progress. During consolidation, stored routes are replayed only while their execution checks remain valid; otherwise, memory-guided action selection resumes.

Consolidation timing. Using the existing Jericho runs, we examined whether each run’s best score had already appeared by episode K−KcK-K_c, where KK is the total episode budget and KcK_c is the consolidation window.

Episode budget KKConsolidation windows KcK_cGames whose best score was observed by episode K−KcK-K_c
304, 63/5, 2/5
504, 6, 8, 104/5, 3/5, 3/5, 2/5
704, 6, 8, 10, 12, 144/5 for every window

These results reveal a timing trade-off: starting consolidation too early may preserve a lower-scoring route. At K=70K=70, 4/5 games had reached their best score before every tested window, allowing more flexible timing. Thus, KcK_c should balance sufficient time for discovery with enough episodes for reliable replay.

Alternative evaluation criteria. We also computed best-found score, max⁡1≤k≤KSk\max_{1 \leq k \leq K} S_k, and average score across all episodes, 1K∑k=1KSk\frac{1}{K}\sum_{k=1}^{K} S_k, for all six methods. The table reports the number of Jericho settings in which each method achieves the highest or tied-highest score.

MethodBest-found score ↑\uparrowAverage score across all episodes ↑\uparrow
FIFO Memory1/153/15
Reflexion0/150/15
AWM0/150/15
EvoTest1/154/15
JitRL6/153/15
EARN13/155/15

EARN achieves the most top-or-tied settings under both criteria. Its best-found advantage is substantial (13/15 versus 6/15 for the next-best method), while its advantage in all-episode average is smaller (5/15 versus 4/15 for EvoTest).

We also examine whether EARN’s full-run best score had already been discovered before different prospective consolidation windows. This occurs in 11/15 settings before the final 4 episodes, 9/15 before the final 6 or 8 episodes, and 7/15 before the final 10 episodes (more than 6/15 for the next-best method).

EARN evaluation cutoffSettings where the full-run best score was already observed
Before the final 4 episodes11/15
Before the final 6 episodes9/15
Before the final 8 episodes9/15
Before the final 10 episodes7/15

These results show that many of EARN’s best-scoring trajectories are already available before consolidation could begin. As the window grows and its starting point moves earlier, fewer settings have reached their eventual best score, highlighting the trade-off between allowing further discovery and beginning consolidation. Together, the results demonstrate EARN’s overall best-found advantage and characterize when its best observed trajectories become available for reuse.

Weakness 2/Question 2: "...many realistic agent tasks provide sparse, delayed, or binary feedback, making it difficult to identify intermediate progress events. The paper does not establish whether its credit assignment remains effective when intermediate score changes are unavailable."

Response: We agree that agent tasks often provide sparse feedback. We note that EARN’s step-wise credit channel does not require dense environment rewards.

We conducted an additional experiment on 10 fixed shopping tasks from a 1,000-product WebShop subset. Each method receives 20 attempts per task, with at most 20 steps per attempt, using the same frozen backbone and task instances. The environment provides a graded score in [0,1][0,1] upon purchase and no intermediate task rewards.

The following table compares all methods without final consolidation (Kc=0K_c=0). Scores are averaged equally across tasks. Best-found is the mean of each task’s highest episode score.

MethodBest-foundAll-episode averageTasks reaching full score
FIFO Memory0.86670.82338/10
Reflexion0.76670.48677/10
AWM0.76670.69507/10
EvoTest0.86670.82508/10
JitRL0.93330.82468/10
EARN0.96670.78679/10

EARN achieves the highest best-found score and discovers a full-score solution on nine tasks, compared with at most eight for the baselines. Thus, its ability to discover high-scoring behavior extends to this setting with terminal-only environment feedback.

We next equipped all six methods with the same conditional route-replay mechanism. The table below reports all-episode average / Last-4 episode average, with a fixed Last-4 evaluation window.

MethodKc=0K_c=0Kc=2K_c=2Kc=4K_c=4Kc=6K_c=6
FIFO Memory0.8233 / 0.86670.8233 / 0.86670.8233 / 0.86670.8233 / 0.8667
Reflexion0.4867 / 0.43330.5217 / 0.60830.5533 / 0.76670.5817/ 0.7667
AWM0.6950 / 0.70830.7017 / 0.74170.7067 / 0.76670.7083 / 0.7667
EvoTest0.8250 / 0.86670.8250 / 0.86670.8250 / 0.86670.8250 / 0.8667
JitRL0.8246 / 0.87500.8279 / 0.89170.8362 / 0.93330.8396 / 0.9333
EARN0.7867 / 0.78330.7983 / 0.84170.8233 / 0.96670.8333 / 0.9667

With Kc=4K_c=4 or 66, EARN achieves the highest Last-4 score of 0.9667, compared with 0.9333 for JitRL. EARN’s best-found score remains 0.9667 across all windows.

Overall, these results support EARN’s ability to discover effective behavior under sparse terminal feedback and retain it through appropriately timed consolidation.

Thank you for the additional experiments. The best-found results and comparisons with replay-equipped baselines address my concern about dependence on the final-four evaluation window. The terminal-feedback WebShop experiments also provide useful evidence beyond environments with intermediate score increments. The responses sufficiently address my main concerns. I am raising my score from 2 to 3 (weak acceptance).

Reviewer_RDq1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EARN learns an external memory of action credits and useful trajectories. The LLM’s weights remain frozen. Trajectory scores are feedback used to update that memory. Its central idea is to reuse effective action sequences near the end of the interaction budget. Even after discovering a good trajectory, the agent may fail to reproduce it because it generates different actions on subsequent attempts. EARN assigns credit through two complementary channels: block-wise credit and step-wise credit. The frozen LLM proposes five actions and reports confidence scores. EARN retrieves relevant historical credits and reweights those preferences. Evidence based on few records or inconsistent block-wise credits is weakened. It then replays successful routes during the final four episodes. This work uses FIFO Memory, Reflexion, Agent Workflow Memory, EvoTest, and JitRL as baselines. On ZTUU, EARN achieves 20.8, vs. 6.3 for JitRL.

Critical strengths
  1. The approach of progress-relative credit assignment and conditional sequence replay is intuitive.
  2. The proposed approach addresses the common issue of how to overcome progress plateaus.
Critical weaknesses

This paper misses related work that could serve as strong baselines:

  1. Feng, Erhu, et al. "Get experience from practice: Llm agents with record & replay." arXiv preprint arXiv:2505.17716 (2025) (AgentRR) AgentRR records agent interaction traces, converts them into reusable experiences, and replays those experiences on similar tasks. It introduces check functions that verify execution-flow integrity, state preconditions, and safety invariants during replay. https://arxiv.org/pdf/2505.17716

  2. Tran, Quan Hung, et al. "Sample Efficient Alignment Learning With Episodic Control." Findings of the Association for Computational Linguistics: EMNLP 2025. 2025. https://aclanthology.org/2025.findings-emnlp.560.pdf

  3. Ecoffet, Adrien, et al. "First return, then explore." Nature 590.7847 (2021): 580-586.

Questions for authors

Could you add the above baselines (see weaknesses) for comparison in the results table?

4 comments

Weakness 1/Question: "...related work that could serve as strong baselines"

Response: We thank you for pointing out these relevant works. We have added all three to our related work discussion and clarified their relationships to EARN.

AgentRR [1] summarizes interaction traces into multi-level experiences for reuse in similar tasks, with check functions constraining execution. By contrast, EARN uses progress-relative credits to discover higher-scoring behavior and directly executes applicable stored action sequences during consolidation. We have clarified this in the related work. The available implementation (in MobiAgent) targets mobile GUI tasks, and adapting the full framework to our benchmarks would require re-implementing its experience construction, level selection, and environment-specific checks, making a faithful numerical comparison infeasible at this stage.

ALEC [2] uses episodic memory to guide text generation under reward-model feedback, with evaluation on helpfulness, harmlessness, and summarization tasks. In our setting, actions change the environment and determine subsequent observations, and experience is accumulated across repeated episodes to improve task scores. ALEC's experimental setting differs from ours. It evaluates the quality of generated text using reward-model feedback on helpfulness, harmlessness, and summarization tasks. Our experiments evaluate agents that execute actions, change the environment, and use experience across repeated episodes to improve task scores.

Go-Explore [3] is a training-based reinforcement-learning method that involves policy training, and its simulator-based variant relies on restoring previously visited environment states. Our method is training-free, keeps the LLM parameters fixed, and does not require arbitrary state restoration. These differences prevent a direct comparison under the same protocol, so we discuss Go Explore as related work rather than include it as a directly comparable baseline.


[1] Feng, Erhu, et al. "Get experience from practice: Llm agents with record & replay." arXiv preprint arXiv:2505.17716 (2025) https://arxiv.org/pdf/2505.17716

[2] Tran, Quan Hung, et al. "Sample Efficient Alignment Learning With Episodic Control." Findings of the Association for Computational Linguistics: EMNLP 2025. 2025. https://aclanthology.org/2025.findings-emnlp.560.pdf

[3] Ecoffet, Adrien, et al. "First return, then explore." Nature 590.7847 (2021): 580-586.

Dear Reviewer RDq1,

As the discussion phase is ending soon, we would greatly appreciate it if you could review our rebuttal and let us know if any concerns remain.

Thank you for your time and thoughtful feedback!

Following our earlier response, we have completed additional WebShop experiments with adaptations of AgentRR and ALEC and a frozen-LLM Go-Explore baseline. These experiments were delayed by GPU scheduling, and we now report the results below.

In the following tables, all three methods use the same frozen LLM backbone and ten shopping tasks, with 20 episodes per task and at most 20 steps per episode. We evaluate final consolidation windows of 2, 4, and 6 episodes and report the all-episode average, final-four average, and best-found scores.

MethodBest-foundAll-episode averageTasks reaching full score
AgentRR0.53330.53335/10
ALEC0.53330.51505/10
Go-Explore0.63330.56546/10
EARN0.96670.78679/10

EARN achieves the highest best-found score and discovers a full-score solution on nine tasks. These results support its ability to discover high-scoring behavior under terminal-only environment feedback.

The table below reports all-episode average / Last-4 average in each cell. �0��0� denotes the final consolidation window length; the evaluation window remains fixed at the final four episodes. | Method | �1��1� | �2��2� | | :--- | :---: | :---: | :---: | :---: | | AgentRR | 0.5333 / 0.5333 | 0.5333 / 0.5333 | 0.5333 / 0.5333 | 0.5333 / 0.5333 | | ALEC | 0.5150 / 0.5167 | 0.5150 / 0.5167 | 0.5183 / 0.5333 | 0.5183 / 0.5333 | | Go-Explore| 0.5654 / 0.5646 | 0.5712 / 0.6021 | 0.5775 / 0.6333 | 0.5850 / 0.6333 | | EARN | 0.7867 / 0.7833 | 0.7983 / 0.8417 | 0.8233 / 0.9667 | 0.8333 / 0.9667 |

We additionally evaluated these methods with conditional final route replay. The table reports all-episode average / Last-4 episode average, keeping the Last-4 evaluation window fixed. With �3��3� or 66, EARN achieves the highest Last-4 score of 0.9667. EARN’s best-found score remains 0.9667 across all windows.

We discuss below how we adapt each method.

AgentRR adaptation. We store concrete interaction traces and use the LLM to summarize positive-score purchases into procedural guidance. Full-success traces can be replayed in subsequent episodes when the recorded context and available actions match the current environment. Execution outcomes are checked after each replayed action. When replay is unavailable or a check fails, the LLM generates actions using the highest-scoring stored experience. This implements concrete replay and abstract experience guidance, with WebShop-specific execution checks.

ALEC adaptation. We adapt ALEC’s episodic value memory to WebShop action selection. At each step, the frozen LLM generates five candidate actions, whose action–context representations are used for similarity-based memory retrieval. The original method uses $$\epsilon)−greedyexplorationduringmemoryconstructionandconfidence−basedselectionduringtest−timedecoding.Inouronlinesetting,memoryconstructionandactionselectionoccuracrossrepeatedepisodes:withprobability()-greedy exploration during memory construction and confidence-based selection during test-time decoding. In our online setting, memory construction and action selection occur across repeated episodes: with probability (\epsilon=0.3=0.3, the agent selects a candidate uniformly at random; otherwise, it selects the highest estimated-value candidate when the confidence criterion is met, or the highest-likelihood candidate under the LLM when it is not. After each episode, discounted purchase scores update the memory values for executed actions. Thus, environment feedback replaces reward-model feedback, while the LLM parameters remain fixed.

Go-Explore adaptation. We maintain an archive of visited pages and the action sequences used to reach them. At the start of an episode, the agent selects an archived state, favoring less frequently selected entries, and attempts to return by executing its recorded action sequence from the environment reset. Return actions count toward the step budget and undergo context and availability checks. The frozen LLM then continues exploration, with previously attempted actions included in its guidance. This adaptation uses neither arbitrary simulator-state restoration nor policy training.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We thank the reviewers for their thoughtful feedback and the Area Chair for their service to the research community. We appreciate the positive assessments of our motivation (RDq1), the practicality of using environment feedback for action credits (5iP3), and the empirical evidence (43VG). Below, we summarize the clarifications and additional results reported in the discussion.

Evaluation protocol and related methods

  • Feedback and adaptation opportunities (43VG). We clarified the shared ScienceWorld scoring protocol, frozen backbone, task-instance schedule, and interaction budgets. AWM receives score feedback and updates its workflow after every episode. Each method retains its own inference and memory settings.
  • Additional related methods (RDq1). We evaluated WebShop adaptations of AgentRR, ALEC, and Go-Explore using the same frozen backbone and interaction budgets. Across ten tasks, EARN's mean best-found score was 0.9667, compared with 0.5333, 0.5333, and 0.6333, respectively.

Additional experiments and analyses

  • Performance when baselines also replay (43VG, 5iP3). We gave JitRL and EvoTest the same conditional replay mechanism. Across five Jericho games with budgets of 30 and 50 episodes, EARN achieved strictly higher final-four average and final-episode scores than both baselines in all ten settings.
  • Performance beyond the final-four metric (5iP3, 43VG). EARN achieved the highest or tied-highest best-found score in 13/15 Jericho settings, versus 6/15 for JitRL. For the all-episode average, EARN led or tied in 5/15 settings, compared with 4/15 for EvoTest.
  • Sparse environment feedback (5iP3). On WebShop, reward is provided only at purchase. Without final consolidation, EARN found full-score solutions on 9/10 tasks, compared with at most 8/10 for the baselines.
  • Choice of replay window (5iP3, 43VG). We tested 0, 2, 4, and 6 replay episodes on WebShop, keeping evaluation fixed at the final four episodes. With replay available to all methods, EARN achieved the highest final-four average with either four or six replay episodes: 0.9667, versus 0.9333 for JitRL.

Significance and scope

Across Jericho and WebShop, EARN’s clearest advantage is its ability to discover high-scoring solutions. On Jericho, EARN retained its terminal lead in all ten settings where JitRL and EvoTest could replay their own routes. These results suggest that each agent’s own method for selecting actions remains important when replay cannot continue during the final consolidation window. EARN also performed well under WebShop’s sparse feedback, where agents receive a graded reward only at purchase. It discovered full-score solutions on 9 of 10 tasks without final consolidation and achieved the highest final-four average (0.9667) with either four or six replay episodes.

Reviewer 5iP3 confirmed that the responses sufficiently addressed their main concerns and raised their score from 2 to 3 (weak acceptance). Supporting details and tables are available in the respective reviewer threads.

Thank you for your consideration.

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.