acceptodds
Withdrawn from ICLR 2027

Action2Delta: Decision-Aligned Dynamics Mid-Training for Language Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning action consequences from environmental interactions, typically through next-state prediction, is becoming an important approach to improving language agents’ decision-making. However, high prediction quality can obscure confusion between the consequences of candidate actions and may not reflect how modeling errors affect action selection. We find that conventional generative objectives focus on fitting successor descriptions, without explicitly comparing action consequences from the same starting state or accounting for their effects on decisions. To address this, we propose Action2Delta, a framework for learning from environmental experience to improve decisions. It organizes supervision around action–consequence discrimination and decision relevance through two methods: Contrastive Delta Dynamics (CDD) and Verified Regret Alignment (VRA). CDD executes different actions from the same real environment state and represents their observable consequences as state deltas. It contrasts the likelihood of the same observed delta under different action conditions, strengthening the correspondence between actions and consequences. Under a fixed value evaluator, VRA uses real rollout returns to verify decision gains from replacing predicted successors with real ones. These gains weight dynamics training, while the returns supervise the policy. On ALFWorld and ScienceWorld, static Action2Delta achieves a success rate of 86.57% and a macro-average score of 72.73, respectively, outperforming uniform training with the same real-environment feedback by 1.74 percentage points and 1.80 score points. Evaluation uses only a single actor. Mechanistic analyses further show improved action–consequence discrimination and reduced candidate decision regret.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language agents, world models, dynamics learning, mid-training, contrastive learning, decision-aware learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_heeeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors propose Action2Delta, a two LLM framework, where actor agent tries to finish the tasks, and the dynamics agent tries to estimate the consequence of the model. CDD teach the dynamics model to learn what an action changes and VRA train the actor agent based on the training signals provided by the dynamics model. The authors demonstrate the effectiveness of this paper in various tasks.

Critical strengths
  1. Well motivated object, a dynamic model as the world model is a good starting point.
  2. Explicit limitation, the authors make a clear discussion on the limitations of this paper.
Critical weaknesses
  1. The gains on multiple tasks still lack adequate uncertainty estimates.
  2. Some results need to clarify. In Table 37, all 30 ScienceWorld task scores satisfy exactly B11=0.82B00+22.9308\mathrm{B11}=0.82\mathrm{B00}+22.9308 In Table 36, improvements are identical across all six ALFWorld task families.
Questions for authors
  1. Can you provide uncertainty of some of the most important results?
  2. Are Tables 36 and 37 direct aggregations of evaluated episodes? Please explain their exact numerical relationships and provide the underlying task/seed outcomes and aggregation procedure.
Reviewer_x6DEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a new method for fine-tuning language models on text simulators with recoverable states that brings model-based RL concepts to LLM fine-tuning. At each decision root, the method assesses candidate actions using predicted and observed outcomes, then performs a weighted contrastive update that emphasizes consequence-prediction errors affecting decision quality. A separate actor update uses real rollout returns and reference-policy probabilities to define action targets, alongside an auxiliary contrastive dynamics objective. The authors evaluate performance gains over SFT and other baselines on ALFWorld and ScienceWorld and provide an ablation study of the algorithm’s key design choices.

Critical strengths
  1. The authors provide an extensive ablation study investigating the contributions of Action2Delta’s key components and design choices.
  2. The paper provides a detailed technical formulation of the method, specifying its training objectives, decision-repair signal, and separate world-model and actor updates.
  3. The shared-feedback comparisons hold starting states, candidate actions, rollout returns, and policy targets fixed, helping isolate the contribution of the proposed dynamics-weighting scheme.
  4. Authors report resource requirements and compute overhead, and the overhead is concentrated in the training phase rather than inference
Critical weaknesses
  1. Authors design their approach around text simulators with recoverable states whereas extending the method to software engineering requires defining observable action deltas, reproducible branch states, and reliable continuation returns; the paper leaves these choices open
  2. The method introduce branching, a world model, a value evaluator, repeated continuations, but its complexity (71x for ALFWorld and 36x for ScienceWorld) needs justification against a strong RL baseline with matched total resources.
  3. ALFWorld and ScienceWorld are good as controlled testbeds, but results on those alone are weak evidence for broad, practical agent capability, stronger alternatives may include DeepSWE, Terminal-Bench, WebArena e t c
  4. Tables 1 and 2 report point estimates without confidence intervals or variability across training seeds. Consequently, the reported differences do not establish statistical significance, and the stability of the improvements across training runs remains difficult to assess.
  5. The diminishing gains from VRA at larger model sizes raise questions about its scalability and cost-effectiveness. At 32B, VRA improves over uniform feedback by only one ALFWorld success across 402 evaluations and shows no ScienceWorld improvement at the reported precision. Without uncertainty estimates, it remains unclear whether this incremental benefit persists reliably as model capacity increases.
  6. Reproducibility would benefit with providing method’s source code
Questions for authors
  1. Table 9 includes GRPO and GiGPO comparisons and a longer-GRPO baseline intended to account for upstream costs. Could you clarify which comparisons match total training FLOPs and environment interactions, including dynamics-data collection and continuation rollouts? A corresponding GiGPO comparison with matched total resources would help assess whether Action2Delta offers a cost-effective improvement over direct RL.
  2. Can you propose a practically implementable pipeline for transferring Action2Delta to software engineering tasks?
  3. Is it possible to evaluate agent on stronger banchmarks such as Terminal-Bench, WebArena to further study its practical capability?
  4. Could you report individual seed results, variability across training seeds, and paired confidence intervals for the main performance differences, particularly VRA versus uniform feedback and the larger-model comparisons?
Reviewer_xLNsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Even when an agent can accurately describe the environment after an action, it may still choose the wrong action, because a small prediction error can change the decision. To address this issue, the authors propose Action2Delta, which consists of two mechanisms. First, Contrastive Delta Dynamics (CDD) executes different actions from the same environment state and teaches the model to distinguish their consequences. For example, opening a cabinet reveals a cup, whereas inspecting the cabinet may not. Second, Verified Regret Alignment (VRA) replaces predicted successor states with real ones while keeping the value evaluator fixed. It then uses returns from real continuation rollouts to verify whether the replacement improves action selection and increases the training weights of the corresponding samples. Finally, supervision from real returns and auxiliary training transfers this knowledge to a policy model that directly selects actions. Using Qwen3-4B as the backbone, the method achieves an 86.57% success rate on ALFWorld and a 72.73 macro-average score on ScienceWorld, outperforming a uniformly weighted training baseline using the same real feedback by 1.74 percentage points and 1.80 score points, respectively.

Critical strengths
  1. The paper focuses on prediction errors that cause agents to choose the wrong actions and assigns training weights accordingly. In the experiments, VRA performs slightly worse than uniform training on some prediction accuracy metrics, yet achieves lower decision regret and better task performance, supporting this design.

  2. CDD holds the starting state and target delta fixed while changing only the action condition, directly encouraging the model to learn which action produces which consequence. The method also excludes actions with equivalent observed consequences from negative pairs, avoiding incorrectly treating equally effective actions as negatives.

  3. VRA keeps the evaluator fixed and replaces only the successor-state input, then uses real returns to verify whether action selection improves. This links the training signal to observable decision gains.

Critical weaknesses
  1. Limited gains and statistical support. VRA improves over NLL-priority weighting by only 0.50 percentage points on ALFWorld and 0.90 points on ScienceWorld, with the former corresponding to just two additional successes across three training seeds. Could the authors provide paired confidence intervals to establish whether these gains are statistically reliable?

  2. Evaluator dependence of the repair signal. Holding the evaluator fixed does not eliminate its representation-dependent biases, so VRA measures evaluator-specific replacement gains rather than the intrinsic importance of dynamics errors. How stable are the sample-weight rankings and downstream improvements across different evaluators and continuation policies?

  3. Unclear necessity of static world-model updates. Appendix D.11 indicates that updated world-model parameters are not transferred to the actor during static training, while policy targets remain fixed. Could the authors clarify whether freezing the world model after computing the repair weights would yield equivalent actor performance, and report the resulting computational savings?

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.