acceptodds
Under review as a conference paper at ICLR 2027

ReplayBind: Training Language World Models to Generate Action-Faithful Futures

Abstract

An agent deciding what to do next needs to distinguish the consequences of its alternatives. In language world models, however, the change that matters may occupy only a few words in an otherwise unchanged observation. Predicting that observation well can therefore leave the association between an action and its consequence weakly learned. We introduce ReplayBind, which learns this association from two observed branches sharing a visible state. Its crossed-likelihood objective favors the recorded action–outcome assignment over the assignment obtained by exchanging the outcomes, using forward prediction and inverse action reconstruction within one generator. Broad observation restoration and balanced branch replay carry this relational supervision into complete observation generation. Matched comparisons show improvements in action–outcome assignment, action reconstruction, and generated branch separation. These results show that ReplayBind better preserves action-induced changes across successive predictions, providing more reliable descriptions of future outcomes for distinguishing the consequences of different actions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language world models, LLM agents, action-conditioned generation, model-based planning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_yXdLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ReplayBind, a training framework for language world models intended to strengthen the association between actions and their resulting textual observations. Given two observed action–outcome branches sharing the same visible state, ReplayBind compares the likelihood of the recorded action–outcome assignment against a crossed assignment in which the outcomes are exchanged. It combines forward outcome prediction and inverse action reconstruction, followed by observation restoration and balanced replay to retain full-observation generation ability.

The method is evaluated on ALFWorld, SciWorld, TextWorld, and WebShop using action–outcome assignment margins, action reconstruction, generated branch separation, observation Token F1, and ten-step recursive prediction. The paper reports improvements over matched controls and several external language/world-model baselines.

Critical strengths
  1. The matched ablations are reasonably careful.

  2. The paper provides several diagnostics beyond overall observation overlap. Branch margin, both-correct rate, distinctive-content F1, inverse action reconstruction, and recursive prediction attempt to examine action-dependent differences rather than relying solely on Token F1.

Critical weaknesses
  1. The shared-state assumption is not adequately verified.
    ReplayBind relies on comparing alternative actions from the same state, but only ALFWorld provides replay-verified branches. Other environments use identical visible observations, which do not guarantee identical underlying states. This undermines the interpretation of the learned differences as action-induced effects.

  2. The methodological novelty is limited.
    The core contribution is essentially a crossed pairwise likelihood objective with forward and inverse supervision, followed by conventional observation restoration and replay. Its distinction from existing contrastive or ranking-based training methods is insufficient to establish a substantial methodological advance.

  3. The experiments do not demonstrate actual planning benefits.
    The evaluation focuses on observation reconstruction, likelihood margins, and textual branch separation. Even recursive prediction follows fixed logged actions rather than model-guided decisions. Consequently, improved prediction fidelity is not shown to translate into better planning or closed-loop agent performance.

  4. The headline performance gains are confounded by extensive prior training.
    ReplayBind inherits substantial domain-specific adaptation and counterfactual supervision that external baselines do not receive. Although matched ablations isolate the crossed-loss contribution, their improvements are considerably smaller than the headline gains, making the actual contribution of ReplayBind difficult to assess.

  5. The branch evaluation provides limited evidence of generalization.
    The primary TextWorld evaluation contains only 15 branch pairs from two conversations. The additional holdout expands coverage but does not verify identical underlying states across branches. This limits the evidence for reliable action-conditioned prediction across diverse environments.

Questions for authors
  1. For SciWorld, TextWorld, and WebShop, how can the authors establish that two branches with the same visible text actually correspond to the same underlying environment state rather than different hidden histories or latent states?

  2. What capability does the crossed-assignment objective provide beyond standard pairwise ranking or contrastive supervision over positive and negative action–outcome pairs?

  3. Can the authors provide evidence that ReplayBind improves closed-loop planning or task success when its generated futures are actually used by an agent to choose actions, rather than only reconstructing fixed logged trajectories?

  4. How should the large gains over external models in Table 1 be interpreted when ReplayBind inherits extensive domain-specific adaptation and counterfactual supervision that those external models do not receive?

1 comment

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the replay-verified state comparisons, additional objective-level baselines, and closed-loop planning evaluation.

However, several fundamental concerns remain. First, while the crossed-assignment objective demonstrates some empirical advantages, its methodological distinction from existing contrastive and assignment-based learning objectives remains limited, particularly given the relatively modest improvements over matched alternatives. Second, although the new closed-loop experiments provide useful evidence, the observed task-success improvements are small and their robustness remains unclear. Third, the shared-state assumption is verified primarily in ALFWorld, leaving the interpretation of action-induced differences in the other environments insufficiently established.

Overall, I acknowledge the improvements introduced during rebuttal, but I remain unconvinced that the current evidence sufficiently establishes the methodological novelty and practical significance of the proposed framework. I therefore maintain my original rating of 1 (Clear Rejection).

Reviewer_LUcqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why language world models, which predict the next textual observation given an action, learn the association between an action and its consequence only weakly. Its premise is that the consequential change often occupies a few words inside an observation that is otherwise shared across alternatives, so ordinary next-observation likelihood is dominated by persistent scene text and the correspondence between competing actions and their futures is left implicit. It proposes ReplayBind, which takes two observed branches sharing a visible state and forms a crossed-likelihood objective: the recorded pairing of actions to outcomes is compared against the pairing obtained by exchanging the outcomes, with a hinge on the normalized margin, applied in both the forward direction of predicting the outcome and the inverse direction of recovering the action, within a single generator. Because each outcome and each conditioning action appears once in both assignments, additive outcome-only and action-only terms cancel in the difference, so improving the margin requires changing the relative compatibility of action-outcome combinations rather than the fluency of either piece. Two further stages, broad observation restoration in the model's native chat template and balanced replay mixing branch pairs with broad transitions, carry this relational signal into complete observation generation. It is tested on four text environments.

Critical strengths
  • The objective is well motivated and the derivation behind it is clean. Because each outcome and each conditioning action appears exactly once in both the recorded and the exchanged assignment, additive preferences for an action alone or an outcome alone cancel in the difference, so the margin can only be improved by changing the relative compatibility of action-outcome combinations. That makes the loss a targeted signal for the association the paper argues is missing, rather than a generic contrastive term, and the extension to permutations over more candidates follows from the same structure.

  • The control design is good. Matched arms share initialization, data, ordering, seeds, optimizer, and update count so that only the hinge differs, and the comparison set includes forward-only and inverse-only deletions plus a per-condition ranking alternative, which is the closest competing formulation. Separate holdouts exclude source identifiers used in any adapter-training stage, and a stricter cohort removes local transitions appearing anywhere in training. The paper also states openly where the coefficient was selected and where an analysis was added after inspecting outputs.

Critical weaknesses
  • No reported result carries a variance estimate and nothing indicates more than one training run per condition, yet the differences the paper rests on are very small.

  • The quantities that improve most are not independent evidence for the claim. The assignment margins are exactly what the hinge optimizes, so their increase confirms that training worked rather than that a relation was learned, and on the paper's own stricter cohort the crossed term changes mean observation fidelity by a negligible amount.

Questions for authors
  • Running the matched pair with several training seeds would resolve my main concern. For the three comparisons the paper leans on most, which are action reconstruction exact match, both-correct rate, and four-candidate inverse assignment, what is the spread across seeds for both the crossed and the zero-crossed arm, and does the ordering hold in every seed?
Reviewer_kPMKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReplayBind which trains language world models to distinguish the outcomes of different actions. Given two observed action-outcome pairs with the same visible observation, its crossed-likelihood loss favors the recorded pairings over swapped pairings. The model learns both forward outcome prediction and inverse action reconstruction, followed by observation restoration and branch replay. Experiments evaluate outcome assignment, generated branch separation, and recursive prediction across four environments.

Critical strengths
  1. The motivation is clear. Predicting shared background text can hide errors in the few words that describe an action’s effect, while comparing alternative action-outcome pairings directly addresses this problem.

  2. The objective comparisons are carefully controlled. The crossed-loss and no-crossed-loss models share initialization, data, computation, and later training stages. Forward-only, inverse-only, and row-wise ranking controls further examine the proposed objective

Critical weaknesses
  1. The generation metrics mainly measure lexical similarity. They can miss incorrect entity relations or state changes. In particular, "both-correct" only requires each prediction to resemble its own reference more than the alternative. The state-level evaluation would better support the action-faithfulness claim.

  2. The planning benefit remains unverified. Recursive prediction follows fixed logged actions, while goal selection uses recorded outcomes and their progress labels. Neither directly tests whether planning with model-generated futures improves task completion.

  3. The gains from the crossed loss lack uncertainty estimates. On the stricter fidelity cohort, F1 changes from 0.7795 to 0.7796, new-source action reconstruction improves from 22.65 to 23.02%. Some Textworld gains are larger, but multi-seed results or confidence intervals are needed to establish reliability.

Questions for authors

NA

1 comment

Thank you for the additional results and clarifications. The semantic evaluations and online planning results addressed part of my original concerns. However, I agree with reviewer yXdL that the validity of the shared-state assumption beyond ALFWorld remain concerns, together with the limited method novelty and robustness of the planning gains. I appreciate the strengthened empirical results, but I still think the overall contribution insufficient for acceptance, thus I keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.