- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a reinforcement learning framework for learning internal world models for coding agents. The main innovation is that before each action, the agent outputs a “belief” representing the agent’s thinking of the state of the world, a prediction about what will happen next, and a decision whether to trust in the prediction or abstain. The predictions are benchmarked against the actual outcomes and used to assign credit to different steps, and at inference time, to select that best-forecast action or resample if the decision is to abstain. In the experiments, the method is compared with agents that do not explicitly output their forecasts before action.
Critical strengths
The method seems sensible and a good number of out-of-domain datasets is analysed in the experiments, with promising results.
Critical weaknesses
The difference to the closest methods has not been explained clearly, nor demonstrated in the experiments. Appendix A lists related work, and a reader is left with an impression that there are several previous works that do something similar even if not in exactly the same way. What is the delta to the closest methods, why is it significant, and is it possible to demonstrate the benefit in the experiments?
The presentation and figures are not clear enough. For example, looking at figures 1 and 2, the fonts are so small that it’s almost impossible to even see what is written, and it does not help that there are maybe 10 different colors, several font types, and decoration which is not always informative. The math in the main text is also not always clear, and the propositions do not seem to add much value to the story.
Questions for authors
Question: How did you set the hyperparameters?
Question: The paper writes that when assigning credits, BAPO compares “steps with the same forecast across rollouts of the same task, regardless of trajectory position”. Could you elaborate the intuition of this?
8 comments
Response to Reviewer zcGj (2/5)
[W1-2. Closest related work: methodological differences and external comparisons]
We have discussed the comparison with ECHO, including our reproduction on the same backbone and RL pool (Section 3.2). Given space constraints, further discussion and additional comparisons are provided in the appendix. We have also discussed these related methods and their differences from ANTE in the Introduction (Section 1).
- ECHO — our closest external baseline. In terminal environments, ECHO co-trains policy optimization with an auxiliary cross-entropy loss on environment-observation tokens. The auxiliary loss trains the same model to predict terminal responses, while policy credit remains trajectory-level (ECHO, Section 3). ANTE trains an execution-grounded code world model whose verified beliefs shape policy optimization. Beyond prediction supervision, BAPO assigns step-level policy credit from forecast verification and trust/abstain decisions (Section 2.3).
- CWM learns execution behavior through trace-based mid-training, followed by post-training. ANTE learns explicit forecasts during policy RL and uses their verification for step credit.
- SWE-World trains separate transition and reward models to simulate execution feedback and evaluation. ANTE obtains these signals from real execution, without a separate learned simulator or critic.
- VAGEN trains internal world models for VLM agents in visual environments, using language-model judging to supervise state estimation and transition prediction.
Comparison with ECHO. Our reproduction uses Qwen3.5-9B, the same RL pool and scaffold on TB 2.0 (Table 11):
| Method | Resolved (%) |
|---|---|
| ECHO, our reproduction | 21.4 |
| ANTE | 24.7 (+3.3) |
ANTE improves over ECHO by 3.3 points. This comparison tests the additional benefit of explicit belief learning and BAPO beyond auxiliary observation prediction; the reward-preserving experiments above further isolate the step-credit construction.
Comparison with other code-agent baselines. The reported 32B SBV results specify each method's backbone and scaffold (Table 10):
| Method | Backbone | Scaffold | SBV |
|---|---|---|---|
| Meta-CWM-32B | From scratch | CWM | 53.9 |
| SWE-World-32B | Qwen2.5-Coder-32B | R2E-Gym | 55.0 |
| ANTE-32B | Qwen3-32B | R2E-Gym | 57.2 |
ANTE reaches 57.2%, above the reported CWM and SWE-World results.
Training configurations. CWM reports 5T mid-training tokens before post-training (CWM, Section 4.2). SWE-World uses 5.7K policy-SFT trajectories for five epochs, with separate 32B transition/reward models trained on 26K/21K examples for four/two epochs (SWE-World, Appendices B–C). ANTE uses 12.5K trajectories for three SFT epochs and 6.6K tasks for one RL epoch (Section 3.1; Appendix G.2, Table 9).
[W2-1. Readability of Figures 1 and 2] Improve the readability of Figures 1 and 2, including text size and colors.
Thank you for your valuable feedback on the presentation. Following your thoughtful suggestion on the font, we have enlarged the text and standardized the typography in Figures 1 and 2 in the revised version. We can further increase the text size in the camera-ready version. We would also like to respectfully clarify the content and organization of Figures 1 and 2 below (the colors follow a consistent coding scheme within each figure to distinguish the components and their roles).
-
Figure 1: behavior and empirical context. The upper panels compare Reactive reasoning with Retrospective memory; the ANTE panel shows an explicit repository belief and prospective reasoning. The bottom panels show task success and forecast accuracy during RL. Reasoning about possible edits is distinct from the forecast fields verified by executing the chosen command (Sections 2.1–2.3).
- [Color code] Pink/brown, blue, and green distinguish thinking, memory, and ANTE belief; the method colors recur in the performance bars.
-
Figure 2: execution feedback and learning. From left to right, the figure shows execution-labeled BeliefTraj and SFT, the RL loop, and ANTE-TTO. Execution feedback supplies prediction targets and decision rewards; BAPO assigns grouped step credit alongside episode-level task credit (Section 2.3). The right panel shows how ANTE-TTO applies the learned beliefs to candidate selection at inference time (Section 2.4).
- [Color code] Blue denotes forecasts, green observed outcomes, and orange learning signals, tracing the flow from prediction and execution to policy updates.
Response to Reviewer zcGj (3/5)
[W2-2. Clarity of the math and role of the propositions] Clarify the main-text mathematics and explain how the propositions support the method. “The math in the main text is also not always clear … the propositions do not seem to add much value to the story.”
We appreciate your careful reading of the theoretical analysis. The five propositions explain the training stages, the BAPO construction, and how learned forecasts support action selection. Their roles are:
| Proposition | Role in the method |
|---|---|
| 1: Belief-augmented imitation | Separates imitation error from hindsight-label mismatch, motivating execution-grounded learning after SFT. |
| 2: Selection regret | Bounds the utility lost when candidate selection relies on imperfect forecasts. |
| 3: Anchored-advantage decomposition | Shows how the two baselines retain within-decision variation and trust/abstain credit. |
| 4: Prediction–reward tradeoff | Interprets the prediction-loss multiplier as an accuracy budget under the stated optimization assumption. |
| 5: Prediction-loss information bounds | Relates prediction loss to the outcome and environment-state information carried by the prediction context. |
-
Proposition 1: the distinct roles of SFT and execution-grounded RL. Fitting hindsight-labeled demonstrations and predicting outcomes on the policy's own executions are different learning requirements. The bound separates accumulated imitation error from a remainder due to hindsight labels (Section 2.2; Appendix D.3). SFT initializes the belief-generating policy; Stage II trains its predictions against outcomes from its own interactions. The corresponding RL stage improves SBV by 2.4 / 4.8 / 10.8 points over ANTE-SFT (Section 3.2, Table 1).
-
Proposition 2: the consequence of forecast error for selection. The result bounds the selected action's expected-utility loss relative to the best available candidate using candidate-specific forecast divergences (Section 2.3). Corollary 3 adds point-forecast spread for the single emitted predictions used by the actual decoder (Appendix D.5, Eq. 16). It quantifies how prediction error can translate into a worse action choice.
- In the additional 14B replay audit (our response to Reviewer 1, W4-2 & Q3), the mean estimated bound is 1.520, and 43/49 sets (87.8%) have estimates below the trivial maximum of 4. The mean measured loss is 0.122. A two-action example with true success probabilities (0.40, 0.60) and forecasts (0.501, 0.499) gives regret 0.800 versus a bound of 0.811. These measurements examine informativeness on real candidates and a concrete configuration where the bound closely tracks loss.
-
Proposition 3: the information retained by the two baselines. Centering returns separately within trust and abstain groups would make both groups' mean advantages zero, removing their average difference from that credit signal. BAPO combines forecast-relative and forecast–decision-relative streams. It retains both the step's deviation within its decision group and credit for choosing trust versus abstain, with empirical orthogonality before normalization and clipping (Appendix D.2, Eqs. 8–10).
-
The original ablation measures BAPO's combined reward and credit contribution. The additional experiment retains the reward and step-level returns: full ANTE exceeds the version without belief grouping by 1.2 / 1.6 / 2.2 points. This tests the practical benefit of the construction explained by Proposition 3. The reported learning-stage results are (Section 3.2, Table 1):
Method 7B 14B 32B ANTE-SFT 24.8 41.4 46.4 ANTE w/o BAPO 25.4 43.6 54.2 ANTE 27.2 46.2 57.2 ANTE + TTO 28.6 48.6 57.8
-
-
Proposition 4: the role of prediction weight. Under its idealized optimization assumption, the multiplier indexes a family of prediction-loss budgets: increasing it tightens that budget, with non-increasing task return (Appendix D.4). The prediction weight has an explicit tradeoff interpretation: it determines how strongly the objective favors prediction accuracy alongside task success.
-
Proposition 5: why predicting checkable outcomes can teach state information. Under the POMDP assumptions, expected prediction loss upper-bounds conditional outcome uncertainty and yields a lower bound on the information the prediction context carries about the next environment state (Appendix D.5, Eqs. 13–15). At fixed conditional outcome entropy, lower prediction loss raises this information lower bound. Execution outcomes therefore provide supervision for learning state-relevant information, giving a formal basis for the forecast component of the internal world model.
Response to Reviewer zcGj (1/5)
We appreciate the time and care you devoted to reviewing our work! We address the related-work comparisons, presentation, and methodological questions below.
Thank you for your careful review. Our closest comparisons are the Reactive and Retrospective baselines in our experiments (Introduction, Section 1; Experiments, Sections 3.1–3.2, Table 1; Appendix E.3). ANTE includes explicit forecasts, verification by real execution, and per-step policy credit within agentic RL (Section 1).
[W1-1. Closest comparisons: from thinking to memory and foresight]
Training recipe and Reactive. Recent code-agent training includes SFT on multi-turn coding trajectories containing reasoning, tool calls, and execution feedback, and RL through interaction with executable environments: the policy generates actions, observes their execution results, and updates from task-outcome rewards (e.g., SWE-Gym, R2E-Gym, and DeepSWE; Section 1). Reactive follows this SFT-then-RL recipe, learning the reasoning-and-tool-use loop without an explicit memory or belief block. Our tasks come from R2E-Gym, SWE-Gym, and SWE-rebench (Section 3.1; Appendix E.1). Reactive and Retrospective are defined in Section 3.1, with their paired corpus construction detailed in Appendix E.3.
Reactive → Retrospective → ANTE
These are close comparisons by construction. Each recorded SFT run is rendered in three forms with the same task query and tool calls. RL shares the backbone, task pool, and scaffold, with rollouts from each variant's SFT checkpoint (Section 3.1; Appendix E.3). ANTE uses the additional learning signals defined in Section 2.3.
The corresponding results are as follows (Section 3.2, Table 1):
Retrospective improves over Reactive by 1.4 points after RL. ANTE reaches 57.2%, improving over Reactive and Retrospective by 6.2 and 4.8 points, respectively.
Additional experiments on policy credit. The reported ANTE w/o BAPO variant retains the belief state and prediction loss but removes the decision reward and belief-conditioned step credit, using episode-level policy advantages (Section 3.1). We add two experiments that retain prediction loss, decision reward, RL task pool, and budgets while changing credit construction. The trajectory-level experiment gives every step the same centered trajectory-mean decision reward. The step-level experiment retains each eligible step's return-to-go and replaces both grouped baselines with the task mean, preserving normalization, clipping, and eligibility (Section 2.3; Appendix G.3).
ANTE adds 3.0 points over w/o BAPO, 2.4 over trajectory-level reward, and 2.2 over ungrouped step credit. Episode credit retains the task objective; grouped step credit distinguishes decisions conditioned on the forecast. Proposition 3 explains how the two streams retain within-decision variation and trust/abstain credit before normalization and clipping (Appendix D.2). Additional 32B rollout analysis also shows first-post-edit forecast accuracy rising from 30.89% to 62.60% (our response to Reviewer 2, Q1).