- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how process rewards and action-consequence prediction can improve policy training and decision-making in LLM agents. It proposes FOLD to provide step-level tool-call evaluations beyond final task outcomes and predict candidate calls’ consequences before execution. FOLD uses Qwen3-4B-Instruct-2507 as its backbone, with LoRA adaptation and jointly trained history encoding, transition, reward, and value models. Interaction histories are encoded into four latent blocks according to action dependence and reward relevance. The transition model predicts each successor block, while the reward and value models read the reward-relevant blocks. Training combines candidate reward ranking, prediction of representations of observed successor histories, and regression to a value proxy. During agent training, FOLD is frozen and supplies process rewards for GRPO. At inference, calls are selected using immediate reward and the value of the predicted successor state. Across three environments, the paper reports average task success of 73.6%, compared with 64.1% for outcome-only GRPO and 69.9% for RWML. In offline candidate selection, look-ahead improves average hit rate from 91.8% with reward-only scoring to 95.7%.
Critical strengths
- The research problem is meaningful. Tool calls need to be evaluated for both their immediate utility and their effects on subsequent decisions. Combining process rewards with consequence prediction to support policy training and call selection addresses a practical need. Although these are established problems, studying them within a shared framework is worthwhile.
- The method offers a useful perspective. Organizing interaction information by action dependence and reward relevance provides a concrete approach to deciding what should be predicted and what should inform decisions. The resulting transition, reward, and value modules form a technically reasonable framework. The paper is generally well organized, and its diagrams and core equations make the overall training and inference procedures understandable.
Critical weaknesses
- The factorization is a central contribution, but its implementation and practical role are not sufficiently established. The method defines how blocks interact with the prediction modules, but essential implementation details remain unclear, particularly how the backbone representation is extracted and transformed into four blocks. Formal definitions and theoretical conditions receive substantial attention, while the connection to the implemented model is less clearly explained. Beyond implementation, the link between the architectural restrictions and the intended semantic separation remains insufficiently supported. The theoretical analysis relies on additional conditions whose relationship to practical training is not established. The synthetic experiment reports block recovery in a controlled setting, but does not establish the factorization’s contribution on the target tasks. The key missing ablation is an unfactored model with matched backbone, total latent dimensionality, data, and training objectives. The existing reward-only ablation reports offline metrics without comparing downstream agents trained using the respective reward models. Consequently, the main experiments report gains for the complete system without adequately identifying the contributions of factorization and joint transition learning. This limits the new understanding provided about the paper’s central design.
- The experiments do not fully connect consequence prediction to actual task gains, and incomplete reporting limits assessment of the main comparisons. The main experiment evaluates agents trained with FOLD’s process rewards. The look-ahead experiment instead measures whether the selected call has the highest process label, without reporting final task success when this selection strategy is deployed. Together, these experiments do not sufficiently establish the contribution of successor-state prediction to downstream task outcomes. Value evaluation has a related limitation. The main text describes value as cumulative return, whereas the implementation uses the maximum process label within the current candidate group as a proxy. The relationship between this proxy and actual future return is neither adequately explained nor empirically validated. The results support proxy prediction and offline ranking, but do not sufficiently establish that the predicted values capture future task utility or improve task completion. The paper states that policy backbones, policy-learning rollout budgets, and optimization configurations are matched within each environment. However, the actual TRM configuration, RWML adaptations, and parts of the DataSearch evaluation protocol remain insufficiently specified. The main results also lack uncertainty estimates. These reporting gaps limit assessment of comparability and the stability of the gains.
Questions for authors
A Implementation and contribution How is h extracted and aggregated, what is the architecture of q, and how are the four blocks produced? How are actions incorporated into the transition and reward modules? Could you provide a matched unfactored comparison and downstream policy results using reward-only versus full FOLD reward models? B Main experimental protocol The environment description specifies the telecom split of τ²-Bench, but the ReAct, GRPO, and RWML numbers match the cross-domain aggregates in RWML (Yu et al., 2026, Table 1). Were these results taken from that paper or obtained from new runs? Were the evaluation domains, user simulators, and protocols identical across methods? Please specify the TRM model and training configuration used in these experiments, the adaptations made to RWML, and the DataSearch train/test split, interaction procedure, and success criterion. C Interpretation of representation and inference experiments In the synthetic experiment, is the fourth block sampled independently at every step? Does the recovery probe target the current or next ground-truth block? If the target is independently sampled next-step noise, how is the reported recovery score of 0.96 explained? Is there evidence connecting the value proxy to actual future return? Under the same policy and candidate budget, does improved offline selection translate into higher final task success? D Robustness to model error Did you examine divergence between predicted scores and actual performance, or whether look-ahead preferentially selects candidates with larger transition-prediction errors? This would help assess whether policy optimization or call selection exploits FOLD’s prediction errors.