acceptodds
Withdrawn from ICLR 2027

Remember Right, Look Ahead: History Notes with Future Prediction for LLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language-model agents trained with multi-turn reinforcement learning (RL) often see only their last few steps, so they lose what they observed earlier. An agent can keep this information in a short text note that it writes at every step and reads at the next, but RL rewards the agent for completing the task and never checks whether the note is correct. We introduce History Notes with Future Prediction (ForeNote), which supervises the note directly. In the text environments we study, everything an agent learns during an episode comes from text feedback, such as a confirmed pickup or an opened drawer, so a small rule-based tracker can read the same feedback and write the correct note, a reference note. In both supervised fine-tuning and RL, ForeNote uses this reference as two training signals: it trains the agent to write its current note correctly (remember right) and to predict the next note after each action (look ahead). At test time, the agent writes and reads its own note without the tracker. With Qwen2.5-3B-Instruct, ForeNote reaches 94.6% and 95.2% success on seen and unseen ALFWorld scenes and solves 59.1% of ScienceWorld tasks. Our analysis on ALFWorld shows where these gains come from. Without a note, an agent is twice as likely to revisit a searched place just after it leaves its five-step window, whereas the note of ForeNote still lists 99.6% of such places, and the agent relies on it: erasing the note at test time costs 8.5 points of unseen success. Without the two signals during RL, unseen success falls by 20.8 points, and giving the agent a correct note at test time does not bring it back. Unseen success also falls by 4.8 points without next-note prediction in RL, and by 3.9 and 15.7 points when the agent instead predicts the next observation or another game's next note.

Primary area:
reinforcement learning
Keywords:
LLM agents, reinforcement learning, agent memory, state tracking, auxiliary prediction, partial observability, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8YpfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ForeNote, which maintains a compact JSON history note to prevent an agent with a limited context window from forgetting searched locations, target locations, or completed steps. During training, an environment-specific rule tracker constructs current and next-step reference notes. History-Note (HN) supervision scores the current note and contributes a note advantage to GRPO, while Future Prediction (FP) asks the model to predict how the note will change after its action and supplies an auxiliary cross-entropy target during SFT/RL. At test time the tracker is removed and the agent writes its own note. The experiments are extensive and include many baselines, training-stage/objective ablations, note-erasure and tracker-note interventions, seen/unseen ALFWorld scenes, WebShop, and ScienceWorld. The results are strongest on ALFWorld, while gains are smaller or less clear on the other environments.

Critical strengths
  1. The empirical study is thorough for an agent-memory paper. It includes many comparison methods, controlled HN/FP ablations, note erasure, tracker-note replacement, seen/unseen ALFWorld evaluation, and behavioral diagnostics such as repeated searches and taking back already placed objects.
  2. FP directly supervises which state fields an action will change, addressing the shared-omission blind spot of comparing only current notes. The intervention results also show that a test-time tracker provides only a small additional gain over the agent's own note, suggesting that the model can internalize the structured state representation under the proposed training recipe.
Critical weaknesses
  1. The core contribution appears to be an environment-specific training recipe that combines manually designed note schemas, rule-based trackers, SFT, and RL, rather than a substantially new learning principle. The paper should more clearly distinguish the contribution of the note interface, HN reward, and FP from the broader claim of general agent memory learning.
  2. Training and evaluation remain within the same benchmark families, simulator semantics, observation language, and action interfaces. Although unseen ALFWorld scenes provide limited layout generalization, the experiments do not establish transfer to a new environment or to settings where a reliable tracker is unavailable, such as a real-world environment. The method therefore primarily demonstrates tracker-to-self internalization within a fixed environment family, not sim-to-real or tracker-free generalization.
Questions for authors
  1. Can the authors evaluate ForeNote under a meaningful environment shift, such as a different simulator, altered observation/action formatting, or a real-world setting where the manually designed tracker is unavailable? At minimum, please report whether a new environment requires a newly hand-designed note schema and tracker, and provide a tracker-free transfer experiment or discuss the associated development cost and generality limitations.
Reviewer_cbF2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper sits on the observation that only seeing the last few steps (as a consequence of keeping prompt short) hinders the performance of language agents. A natural remedy is to use a short text note at every step containing a dense information regarding what has been tried for example, and consider it as a 'scratchpad' that the task reward will shape on its own. The paper argues that this note should be explicitly trained. More specifically, the proposed approach (ForeNote) supervises the agent to predict both a current and next-step note during both SFT and RL stage and the groud truth note is obtained from a rule-based tracker rebuilds from the agent's interactions so far. And the notes are represented by JSON object with hand-designed field, recording three kinds of information that a short window loses: what the agent has already tried, where its targets are, and which steps of the task are done. The authors compare ForeNote with on multi-turn environment such as ALFWorld, SceinceWorld and WebShop and show the improvements.

Critical strengths
  • It is expected that a note summarizing the past interactions is helpful, but I found it interesting that the note has to be directly trained as shown by the author. I would have thought that it's enough to let the note be trained 'implicitly' with the task reward.

  • The ablation study is relatively comprehensive where why the note has to be supervised and how the supervision changes the behavior are shown.

Critical weaknesses
  • Since the note is simply a compact JSON object recording the state of the interactions, I am a bit puzzled by the effectiveness of predicting current and next notes . It seems that the model can easily predict the current and next note by copying tokens and inferring some deterministic effects of the action since most parts of the note would not change between steps. In this case, I also find the 'look ahead' framing is a bit oversold because there seems not many genuinely unknown future outcomes to be predicted (if any, the loss would be dominated by the unchanged part in the note, and simply copying those can already bring the loss down quite a bit). Would the authors be able to clarify this?

  • Secondly, can the author clarify the novelty of their work over other works that maintain a textual belief and trains it? Reading from the second paragraph of related work, Lanchantin et al. 2023, Lidayan et al. 2025 (ABBEL) and Wang et al. 2025 (VAGEN) looks quite similar.

  • The fact that the rule-based note tracker has to be hand-designed, and cannot work out of the box for similar environments and environments with feedback which is harder to parse does not warrant a stronger recommendation.

Questions for authors

(Just minor points on the presentation):

  • The UU in equation 1 seems undefined (I assume it's the tracker). Also L_PG in equation 5.

  • Consider expand the L_expert and L_PG in equation 4 and 5 so that the HN part is explicitly shown. I believe that this will make the paper clearer.

Reviewer_eDyVOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This article proposes History Notes with Future Prediction (ForeNote) to address the problem that LLM forgets earlier interactions with limited context window. It uses a rule-based tracker to convert environment feedback into structured reference notes, which will be used to construct two training signals for both SFT and RL phase. During inference, the agent relies on the self-generated history notes and the tracker is deprecated.

The authors evaluate ForeNote using Qwen2.5-3B-Instruct and Qwen3-1.7B on three benchmarks: ALFWorld, ScienceWorld, and WebShop. ForeNote achieves success rates of 94.6% and 95.2% respectively on ALFWorld seen and unseen scenes, a 59.1% task-solving rate on ScienceWorld, and an 81.9% success rate on the WebShop clean test set using Qwen2.5-3B-Instruct. Ablation studies and behavioral analyses further examine the contributions of the two training signals. Removing both HN and FP during RL reduces ALFWorld unseen success from 95.2% to 74.4%, while removing only FP reduces it to 90.4%. The authors also show that insufficient or inappropriate memory supervision can cause agents to forget history information. Using the auxiliary supervision loss can help address this problem.

Critical strengths
  1. The paper is well written and the experiments are well-designed.
Critical weaknesses
  1. Lack of novelty compared to ABBEL and ReBel. And lack of empirical comparisons with these works.

  2. The method relies on manually designed, environment-specific note schemas and parsing rules to generate reference supervision. This requires additional engineering for new environments, and its applicability to noisy or free-form feedback remains unvalidated.

Questions for authors
  1. What are the independent advantages of Next-Note Prediction over Current-Note Supervision? The results in Table 12 show no statistically significant advantage over Current-Note Supervision on ALFWorld Unseen.

  2. See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.