acceptodds
Withdrawn from ICLR 2027

Imagine Before You Call: Factored World Models for Tool-Using Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tool-using agents extend the capabilities of large language models by interacting with external APIs and environments. However, reliable tool use remains challenging because interaction histories contain substantial task-irrelevant information, while the value of a tool call often depends on consequences that are unavailable before execution. Existing reward models score tool calls directly from the current context without explicitly modeling how each call changes the environment, leaving the agent's understanding of environment dynamics under-constrained. To tackle these issues, we introduce **FOLD**, a factored world-modeling framework for tool-using agents that predicts the consequences of candidate calls through one-step latent dynamics. FOLD partitions the latent state according to two functional properties: (1) whether state information changes in response to the agent's action; and (2) whether it affects future task reward. This yields four latent blocks with distinct transition and decision roles. Based on this representation, FOLD jointly learns latent transitions, process rewards, and state values. The process reward provides step-level supervision for standard policy optimization, while the predicted successor states support test-time look-ahead before a tool is invoked. We further provide a block-wise identifiability analysis for the learned factorization. Experiments on ALFWorld, -Bench, and DataSearch show that FOLD achieves 97.6% average accuracy on pairwise tool-call preference decisions, raises downstream task success by 3.7 points, and improves test-time tool-selection accuracy to 95.7% on average.

Primary area:
reinforcement learning
Keywords:
tool-using, world model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1VFsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how process rewards and action-consequence prediction can improve policy training and decision-making in LLM agents. It proposes FOLD to provide step-level tool-call evaluations beyond final task outcomes and predict candidate calls’ consequences before execution. FOLD uses Qwen3-4B-Instruct-2507 as its backbone, with LoRA adaptation and jointly trained history encoding, transition, reward, and value models. Interaction histories are encoded into four latent blocks according to action dependence and reward relevance. The transition model predicts each successor block, while the reward and value models read the reward-relevant blocks. Training combines candidate reward ranking, prediction of representations of observed successor histories, and regression to a value proxy. During agent training, FOLD is frozen and supplies process rewards for GRPO. At inference, calls are selected using immediate reward and the value of the predicted successor state. Across three environments, the paper reports average task success of 73.6%, compared with 64.1% for outcome-only GRPO and 69.9% for RWML. In offline candidate selection, look-ahead improves average hit rate from 91.8% with reward-only scoring to 95.7%.

Critical strengths
  1. The research problem is meaningful. Tool calls need to be evaluated for both their immediate utility and their effects on subsequent decisions. Combining process rewards with consequence prediction to support policy training and call selection addresses a practical need. Although these are established problems, studying them within a shared framework is worthwhile.
  2. The method offers a useful perspective. Organizing interaction information by action dependence and reward relevance provides a concrete approach to deciding what should be predicted and what should inform decisions. The resulting transition, reward, and value modules form a technically reasonable framework. The paper is generally well organized, and its diagrams and core equations make the overall training and inference procedures understandable.
Critical weaknesses
  1. The factorization is a central contribution, but its implementation and practical role are not sufficiently established. The method defines how blocks interact with the prediction modules, but essential implementation details remain unclear, particularly how the backbone representation is extracted and transformed into four blocks. Formal definitions and theoretical conditions receive substantial attention, while the connection to the implemented model is less clearly explained. Beyond implementation, the link between the architectural restrictions and the intended semantic separation remains insufficiently supported. The theoretical analysis relies on additional conditions whose relationship to practical training is not established. The synthetic experiment reports block recovery in a controlled setting, but does not establish the factorization’s contribution on the target tasks. The key missing ablation is an unfactored model with matched backbone, total latent dimensionality, data, and training objectives. The existing reward-only ablation reports offline metrics without comparing downstream agents trained using the respective reward models. Consequently, the main experiments report gains for the complete system without adequately identifying the contributions of factorization and joint transition learning. This limits the new understanding provided about the paper’s central design.
  2. The experiments do not fully connect consequence prediction to actual task gains, and incomplete reporting limits assessment of the main comparisons. The main experiment evaluates agents trained with FOLD’s process rewards. The look-ahead experiment instead measures whether the selected call has the highest process label, without reporting final task success when this selection strategy is deployed. Together, these experiments do not sufficiently establish the contribution of successor-state prediction to downstream task outcomes. Value evaluation has a related limitation. The main text describes value as cumulative return, whereas the implementation uses the maximum process label within the current candidate group as a proxy. The relationship between this proxy and actual future return is neither adequately explained nor empirically validated. The results support proxy prediction and offline ranking, but do not sufficiently establish that the predicted values capture future task utility or improve task completion. The paper states that policy backbones, policy-learning rollout budgets, and optimization configurations are matched within each environment. However, the actual TRM configuration, RWML adaptations, and parts of the DataSearch evaluation protocol remain insufficiently specified. The main results also lack uncertainty estimates. These reporting gaps limit assessment of comparability and the stability of the gains.
Questions for authors

A Implementation and contribution How is h extracted and aggregated, what is the architecture of q, and how are the four blocks produced? How are actions incorporated into the transition and reward modules? Could you provide a matched unfactored comparison and downstream policy results using reward-only versus full FOLD reward models? B Main experimental protocol The environment description specifies the telecom split of τ²-Bench, but the ReAct, GRPO, and RWML numbers match the cross-domain aggregates in RWML (Yu et al., 2026, Table 1). Were these results taken from that paper or obtained from new runs? Were the evaluation domains, user simulators, and protocols identical across methods? Please specify the TRM model and training configuration used in these experiments, the adaptations made to RWML, and the DataSearch train/test split, interaction procedure, and success criterion. C Interpretation of representation and inference experiments In the synthetic experiment, is the fourth block sampled independently at every step? Does the recovery probe target the current or next ground-truth block? If the target is independently sampled next-step noise, how is the reported recovery score of 0.96 explained? Is there evidence connecting the value proxy to actual future return? Under the same policy and candidate budget, does improved offline selection translate into higher final task success? D Robustness to model error Did you examine divergence between predicted scores and actual performance, or whether look-ahead preferentially selects candidates with larger transition-prediction errors? This would help assess whether policy optimization or call selection exploits FOLD’s prediction errors.

Reviewer_4XhROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FOLD, a factored world-modeling framework for tool-using agents. The latent interaction state is partitioned into four blocks by action dependence and reward relevance; the transition model is structured so that only action-dependent blocks are conditioned on the call, and the reward and value models read only the reward-relevant state. The framework is trained jointly with a listwise process-reward objective, a blockwise transition objective, and a value objective, then used both to provide step-level process rewards for GRPO policy optimization and to perform one-step latent look-ahead for test-time tool selection. A blockwise identifiability result is provided, and experiments on three benchmarks show gains in tool-call assessment, downstream task success.

Critical strengths
  1. The motivation is clear. The paper proposes an imagine-before-call decision paradigm.
  2. The design is reasonable. The paper designs a four-block state factorization, where the structured transition decouples action-dependent blocks, and the reward/value heads read only the reward-relevant state.
  3. The paper provides a blockwise identifiability theory with proofs
  4. The experiments are comprehensive.
Critical weaknesses
  1. The introduction motivates the work by verbose, task-irrelevant histories that make tool selection difficult, but the policy still consumes the full history. The factored state is used only for scoring and reranking and is never fed back into the policy’s context. The framework filters information for the decision signal but does not reduce what the policy read.
  2. Given that the encoder state already shows substantial cross-block leakage in Fig. 4a, the paper lacks a control that feeds the undivided 64-dim state directly into the transition/reward/value models, to justify the necessity and effectiveness of the factorization.
  3. In downstream policy optimization, only the reward head participates, while the transition and value models are used exclusively for offline reranking at inference. The paper gives no rationale for this restriction. In particular, the predicted future utility from them could be naturally used to refine turn-level advantage estimation in Eq. (58).
  4. The network design of the representation head and the three functional heads () is never specified.
  5. The transition accuracy in Fig. 3b (cosine similarity) is reported without any comparison baseline, making its absolute level hard to interpret.
Questions for authors
  1. Which model serves as the LLM judge in Eq. (50), and have its scores been validated against human judgments?
  2. Please specify the architecture of the representation head and the three functional heads.
  3. Eq. (14) states the value model regresses the true return-to-go, while Appendix C.2 states it regresses the group-optimal label as a return proxy. Please provide an explanation for this.
  4. The 8B comparison in Appendix D.5 uses the 4B-tuned configuration (same LoRA rank, steps, and learning rate), so the conclusion that “increasing backbone scale alone does not improve reward modeling” is confounded with under-fitting. Please clarify whether the 8B backbone was given scale-appropriate capacity and training budget, or soften the claim accordingly.
Reviewer_rocNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the authors tackle a scenario where language-model agents need to select tool calls without observing their consequences beforehand, and the interaction histories contain task-irrelevant information. In such a scenario, the authors argue that predicting the consequences of candidate calls could improve tool selection, and propose FOLD, a factored world-modeling framework. The procedure is to encode the interaction history into four blocks based on whether the information is affected by the agent’s actions and whether it is relevant to reward. These representations are used to jointly train transition, process-reward, and value models. The process-reward model provides supervision for policy optimization using GRPO. During inference, the transition model predicts the next state for each candidate call. The calls are scored by combining their process rewards with the values of their predicted next states, and the highest-scoring call is executed. They evaluate the approach on ALFWorld, the telecom domain of τ2-Bench, and DataSearch, and report improvements in task success and tool-call selection.

Critical strengths

The paper addresses a useful problem: assessing the consequences of a tool call before execution. The approach combines process supervision for policy training with latent look-ahead for tool selection.

The authors evaluate across three different environments and compare against relevant baselines, including GRPO, TRM, and RWML. Table 2 reports consistent improvements in task success across all three environments.

The authors propose using predicted next states to compare candidate tool calls before execution. This is useful when tool calls are costly or change the environment.

Critical weaknesses
  1. The turn-level advantage estimation when using the reward model trained is a little worrisome. As the turns increase, while calculating the advantage in a rollout for turn t, the authors are comparing rewards at turn t across rollouts, even as their histories diverge. This can potentially also learn to increase in policy-gradient variance (which is a pivotal reason why baseline reduction is done).

  2. The value target described in the appendix does not match the main paper. Eq. (14) describes predicting discounted future returns, but Appendix C.2 uses the maximum process label among the current candidates. Appendix C.3 evaluates against the same proxy. Predicting the best local label does not establish prediction of future returns, so the interpretation of the value term in look-ahead needs clarification.

  3. The takeaway from the theory is not clear. The authors write about the theory that each learned block is identifiable up to a C^1 invertible transformation. But what is the practical takeaway from it? Also, empirical experimentation supporting this theory is also not presented in the paper.

  4. The look-ahead evaluation does not establish improved task success during interaction. Table 3 measures offline candidate selection, while Table 2 evaluates policy training with process rewards. The authors should compare reward-only selection and look-ahead during interactive rollouts using the same frozen policy and candidate budget.

  5. The key presentation problem with the paper is that the training procedure is difficult to follow. The notation and freezing details appear in Table 4 and Appendix C, but Section 3.2 does not clearly state which parameters each loss updates. The authors should distinguish the FOLD backbone from the actor and summarize the parameter definitions, gradient flow, and components frozen during policy optimization properly in the main paper.

Questions for authors

Can the authors evaluate whether comparing rewards across increasingly different rollout histories increases policy-gradient variance at later turns? Does a baseline conditioned on the current history improve training stability?

Is the value model trained on discounted future returns or the maximum process label among the current candidates? Can the authors ensure consistency in Section 3.2 with Appendix C.2 and compare look-ahead performance using these two targets?

What does Theorem 1 establish about the representations learned by the proposed training objective? Which assumptions hold in the main experiments, and what empirical evidence supports the claimed separation of the latent blocks?

Can the authors clearly specify in the main paper which models and parameters each loss updates? A table showing the parameter definitions, gradient flow, and components frozen during policy optimization would help clarify the training procedure.

Can the authors justify the reason for conducting ALFWorld experiments with an instruct model and the other two with a reasoning model (Qwen3-8B)? Can the authors showcase the broader utility to both the both across all benchmarks they evaluated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.