acceptodds
Withdrawn from ICLR 2027

Predictions About Unchosen Actions Improve Learning From Agent Trajectories

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Interactive agents generate trajectories that are later reused for training, yet each decision records an outcome only for the action that was executed. We ask whether the same realized interaction becomes more useful when it also retains pre-action predictions about actions that were available but not chosen. We introduce Candidate logging, which stores the consequence predicted by the trajectory-generating model (the Source) for each candidate action without executing the alternatives. In a controlled latent-rule environment, where an unobserved rule maps mode–action pairs to stochastic consequences, Candidate logging improves held-out consequence prediction from 69.0% to 75.7% (+6.7 percentage points) compared with retaining a prediction only for the executed action. The improvement transfers to autonomous behavior, with no logged predictions or reasoning available at evaluation: relative to the same baseline, task success rises by 5.1 points on ALFWorld valid_unseen (to 52.5%) and by 4.7 points on WebShop (to 39.7%). On both tasks, it also outperforms a baseline that retains the Source’s native pre-action reasoning. Control conditions that keep the logged fields but remove or misalign their action-specific content do not reproduce the controlled gain, which increases as predictions for more unchosen actions are retained and decreases as downstream model capacity and training data grow. These patterns are consistent with explicit predictions reducing the need to recover action–consequence relationships from limited realized experience.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent trajectories, language agents, trajectory logging, action-conditioned prediction, unchosen actions, auxiliary supervision, offline learning, model-based learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_T861OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors consider fine-tuning language agents on AlfWorld, WebShop, and a warmup synthetic task, exploring the impact of either supervising on the next action; on intermediate CoT; and on intermediate predictions for counterfactual actions, ultimately championing the last option. The intuition is that such intermediate predictions serve as a "fallible" proxy of what would have actually happened had such actions been taken. They find the modest benefits that decay with increasing model and data scale.

Critical strengths

I like the warmup toy task as a way of gaining intuition, and the experimental results appear to be well analyzed for statistical significance.

Critical weaknesses

Poor clarity -- eg, the problem setup does not even say what the learner is actually doing (how is it supervised and evaluated?), and is better thought of as an incomplete methods stub.

Poor evaluation -- AlfWorld and WebShop are old benchmarks that wouldn't be considered very challenging / a good signal of progress at this point, certainly if one is standing on the shoulders of foundation models, as this paper does.

Small effect size -- the approach championed by the paper confers modest gain on outdated benchmarks. I don't htink this is adequate validation, and the decay of the effect size with model and data scale means the benefits wash out in the limit.

Questions for authors

Does the benefit of your method also wash out with scale on alfworld and webshop? Can you test on harder agentic tasks?

Reviewer_aG3vOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that conventional trajectories record only the executed actions and their realized outcomes, and that retaining information about the consequences of alternative actions can improve downstream learning. It introduces Candidate logging, which augments a fixed realized trajectory with model-predicted immediate consequences for candidate actions at each decision state, using only pre-action information. These predictions serve as auxiliary supervision for the Learner. Compared with Chosen, Candidate improves held-out consequence-prediction accuracy from 69.0% to 75.7% in the controlled latent-rule environment, and increases autonomous task success from 47.5% to 52.5% on ALFWorld valid_unseen and from 35.0% to 39.7% on WebShop.

Critical strengths
  • The approach is intuitive and well motivated. Retaining predictions about unchosen actions enriches the supervision available from fixed realized trajectories.
  • The empirical results are solid, showing consistent improvements across multiple evaluation environments.
Critical weaknesses
  • Limited novelty. The core idea is incremental relative to Agent Learning via Early Experience. The main distinction is replacing executed outcomes of alternative actions with Source predictions. While avoiding additional environment interaction is useful, the advantage over prior work remains unclear without a direct comparison.
  • Unclear applicability to harder-to-predict tasks. The method relies on Source predictions, but the effect of prediction quality is insufficiently studied. Its effectiveness on tasks with hidden states or complex tool outcomes remains unclear.
  • Limited comparison with alternative distillation methods. Budget-matched comparisons with other forms of teacher supervision, such as action-distribution distillation, are missing. A comparison with OPD under a compatible setting would also help establish the practical value of Candidate logging.
Questions for authors

Please refer to the Weaknesses section.

Reviewer_gJMrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Candidate logging, which augments agent trajectories with teacher-generated predictions of the consequences of unchosen actions. The authors train smaller models using these predictions as auxiliary supervision and report improvements in a synthetic latent-rule environment, ALFWorld, and WebShop.

Critical strengths

The paper addresses an interesting problem of leveraging predictions about unchosen actions to improve agent learning. The experiments are reasonably well-designed, and the method shows consistent improvements across both controlled and autonomous agent tasks.

Critical weaknesses
  1. The writing is unnecessarily complicated and really difficult to follow.

  2. Limited novelty and insufficient insights beyond existing work. The proposed method seems relatively straightforward, and many of the experimental findings are largely predictable. For example, it is not surprising that additional consequence-prediction supervision benefits smaller models or models trained with less data. And that larger benefits with less training data or smaller student models are consistent with standard expectations for knowledge distillation. Although the paper includes several ablation studies, I do not think they provide much deeper insight into why the method works or under what conditions it is genuinely useful.

  3. Unclear informativeness and reliability of teacher-generated candidate predictions.It is unclear what genuinely new information the teacher contributes through its predictions about unchosen actions. For example, the paper illustrates a candidate action go to fridge 1 with the predicted consequence (navigate, fridge 1, at:fridge 1, success). This consequence seems largely predictable from the action description itself, potentially even by the smaller student model without additional teacher supervision. I would like to understand whether the proposed method teaches meaningful state-dependent environment dynamics or merely reinforces relatively trivial action–effect associations. The paper reports does not adequately evaluate the quality of predictions for unchosen actions or investigate how teacher prediction errors affect downstream learning.

  4. Missing critical baselines for world-model learning and student-generated predictions. The current baselines do not adequately establish that the improvement specifically requires predictions generated by a larger teacher model. In particular, I would like to see comparisons with:

a. Student-generated candidate predictions: Have the student model generate its own predictions about unchosen actions and use them for auxiliary training. Does the stronger teacher actually provide additional useful knowledge?

b. World-model supervision from realized transitions: Train the student to predict actual next observations or consequences using the existing executed trajectories. How much additional benefit comes specifically from the unexecuted alternatives?

c. It would also be interesting to compare against explicitly asking the student to predict candidate consequences before selecting an action at inference time.

Without these comparisons, I find it difficult to determine which component is responsible for the reported gains.

  1. Insufficient differentiation from existing world-model approaches.

The proposed approach appears closely related to existing work on world modeling and action-consequence prediction for language agents. For example, Web Agents with World Models (Chae et al., https://arxiv.org/abs/2410.13232) already explores predicting the consequences of candidate actions to improve agent decision-making.

More closely related to the training setup, Policy and World Modeling Co-Training for Language Agents (Lu et al., 2026, https://arxiv.org/abs/2606.02388) uses auxiliary world-modeling supervision to improve policy learning without requiring additional inference-time computation.

While the proposed method differs in using teacher-generated predictions for unexecuted actions, I do not find the paper's discussion sufficient to establish why this distinction represents a meaningful advance over existing approaches.

Questions for authors
  1. How accurate are the teacher's predictions of unchosen actions?

  2. What additional information does the teacher provide that the student does not already know?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.