- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces EARN, a training-free adaptation method for frozen LLM agents. It combines progress-relative block-wise credits, step-wise feedback, and conditional route replay to improve performance over repeated attempts. Across Jericho, ScienceWorld, and Crafter, EARN achieves the highest or tied-highest final-four average score in 31 of 33 task–budget settings.
Critical strengths
- EARN addresses both progress beyond previously attained scores and reliable reuse of effective behavior. Combining action-level credit adjustment with conditional route replay provides a practical approach without updating model parameters.
- Evaluation across three complementary environments supports the approach’s empirical value. The Jericho ablations indicate that both block-wise credit and replay contribute to performance.
Critical weaknesses
- Although all methods use frozen models and matched interaction budgets, EARN explicitly uses score-based credit updates, while AWM relies on workflow memory. The paper does not adequately clarify whether baselines receive comparable feedback and adaptation opportunities, or whether their ScienceWorld implementations are appropriately adapted. This limits attribution of the gains to EARN’s design.
- The evidence does not sufficiently separate improved adaptation from route reproduction. Replay is enabled in precisely the final-four window used for the primary metric, while ablations cover only two Jericho games.
Questions for authors
- In ScienceWorld, do AWM and the other baselines receive the same score feedback and opportunities to accumulate experience as EARN? How is AWM’s workflow induction and retrieval adapted to changing task instances, and what are the respective inference and memory budgets?
- How much of EARN’s advantage remains when the strongest baselines are equipped with the same conditional route replay mechanism? ScienceWorld results would help distinguish gains from credit assignment versus route reuse.
3 comments
Weakness 2: The evidence does not sufficiently separate improved adaptation from route reproduction. Replay is enabled in precisely the final-four window used for the primary metric.
Response: We thank you for raising this question. EARN addresses two complementary objectives: discovering higher-scoring behavior through progress-relative credit assignment and retaining discovered behavior through conditional consolidation. The credits reduce reinforcement of repeatedly attained scores and encourage further progress. During consolidation, stored routes are replayed only while their execution checks remain valid; otherwise, memory-guided action selection resumes.
Consolidation timing. Using the existing Jericho runs, we examined whether each run’s best score had already appeared by episode , where is the total episode budget and is the consolidation window.
| Episode budget | Consolidation windows | Games whose best score was observed by episode |
|---|---|---|
| 30 | 4, 6 | 3/5, 2/5 |
| 50 | 4, 6, 8, 10 | 4/5, 3/5, 3/5, 2/5 |
| 70 | 4, 6, 8, 10, 12, 14 | 4/5 for every window |
These results reveal a timing trade-off: starting consolidation too early may preserve a lower-scoring route. At , 4/5 games had reached their best score before every tested window, allowing more flexible timing. Thus, should balance sufficient time for discovery with enough episodes for reliable replay.
Alternative evaluation criteria. We also computed best-found score, , and average score across all episodes, , for all six methods. The table reports the number of Jericho settings in which each method achieves the highest or tied-highest score.
| Method | Best-found score | Average score across all episodes |
|---|---|---|
| FIFO Memory | 1/15 | 3/15 |
| Reflexion | 0/15 | 0/15 |
| AWM | 0/15 | 0/15 |
| EvoTest | 1/15 | 4/15 |
| JitRL | 6/15 | 3/15 |
| EARN | 13/15 | 5/15 |
EARN achieves the most top-or-tied settings under both criteria. Its best-found advantage is substantial (13/15 versus 6/15 for the next-best method), while its advantage in all-episode average is smaller (5/15 versus 4/15 for EvoTest).
We also examine whether EARN’s full-run best score had already been discovered before different prospective consolidation windows. This occurs in 11/15 settings before the final 4 episodes, 9/15 before the final 6 or 8 episodes, and 7/15 before the final 10 episodes (more than 6/15 for the next-best method).
| EARN evaluation cutoff | Settings where the full-run best score was already observed |
|---|---|
| Before the final 4 episodes | 11/15 |
| Before the final 6 episodes | 9/15 |
| Before the final 8 episodes | 9/15 |
| Before the final 10 episodes | 7/15 |
These results show that many of EARN’s best-scoring trajectories are already available before consolidation could begin. As the window grows and its starting point moves earlier, fewer settings have reached their eventual best score, highlighting the trade-off between allowing further discovery and beginning consolidation. Together, the results demonstrate EARN’s overall best-found advantage and characterize when its best observed trajectories become available for reuse.
Dear Reviewer 43VG,
As the discussion phase is ending soon, we would greatly appreciate it if you could review our rebuttal and let us know if any concerns remain.
Thank you for your time and thoughtful feedback!
We clarify that all methods use the same frozen backbone, ScienceWorld scoring protocol, task-instance schedule, and interaction budgets: 10, 20, or 30 episodes per task, with at most 100 actions per episode. Memory persists across episodes within each run, allowing every method to accumulate experience as instances change. We retain each method’s own inference and memory settings, including its auxiliary reflection, workflow-induction, or evaluation calls. Thus, environment-interaction budgets are matched, while each method retains its own inference and memory settings.
AWM receives per-step rewards and scores, together with the final episode score. After each episode, it updates the task’s workflow using the previous workflow and the new trajectory. And AWM calls an LLM to assess the episode’s outcome and update the existing workflow using the trajectory and score feedback. AWM reuses the workflow across different variations of the same task type, providing it alongside the current task description and observation. The agent therefore uses earlier experience while adapting its actions to the current situation.
Response: We have added EvoTest + conditional replay and JitRL + conditional replay. All methods use the same final-four replay window and route-selection, context-matching, and action-availability rules. Each method resumes its native action-selection policy when replay is unavailable. ScienceWorld evaluates different task variations across episodes. As a result, stored routes trigger replay less often than in Jericho and WebShop. We therefore report the replay-controlled comparison on these two datasets.
For Jericho, each entry reports Last-4 / Last / Best-round, where Best-round is the highest episode score within the interaction budget.
EARN achieves strictly higher Last-4 and Last scores in all 10 settings and the highest Best-round score in every setting, including two ties. Thus, its advantage persists after the strongest baselines receive the same replay mechanism.
We conducted an additional experiment on 10 fixed shopping tasks from a 1,000-product WebShop subset. Each method receives 20 attempts per task, with at most 20 steps per attempt, using the same frozen backbone and task instances. The environment provides a graded score in [0,1] upon purchase and no intermediate task rewards.
For WebShop, we apply final-four replay to each method. All metrics are averaged equally across tasks.
Replay improves JitRL’s Last-4 score from 0.8750 to 0.9333 and narrows its gap with full EARN from 0.0917 to 0.0333. Best-round scores remain unchanged for all three methods. EARN’s higher Best-round score is already present without final replay. EARN retains the highest Last-4 score when replay is available to all three methods.