acceptodds
Under review as a conference paper at ICLR 2027

ANTE: Anticipate, Then Execute - Learning Internal World Models for Code Agents from Verified Beliefs

Abstract

Rapid advances in large language models have made software engineering agents capable of tackling complex coding tasks. Within a containerized environment, a code agent receives tool responses or execution results, and determines its later actions by those earlier results. However, most existing code agents never look ahead: their policies do not anticipate the consequences of actions, and the terminal reward barely attributes a failure to certain earlier steps, so an ill-judged edit can corrupt the repository state irreversibly before any signal arrives. Recent advances in code world models seek to tackle these challenges, yet they are learned offline or frozen before certain training stages, reduced to an auxiliary loss on observation tokens. We present ANTE (Anticipate, Then Execute), an agentic RL framework that verifies the policy's pre-action predictions by real execution and turns them into per-step credit for its actions and trust-or-abstain decisions. With an internal belief stated before each action, Belief-Anchored Policy Optimization (BAPO) turns the settled predictions into per-step advantages by anchoring credit on the policy's own forecasts. To turn the trained forecasts into better actions at inference, thereby facilitating effective test-time scaling, ANTE-TTO selects among candidate actions without an external scorer. ANTE raises Qwen3-32B from 25.0% to 57.2% pass@1 on SWE-bench Verified, 6.2% above a matched GRPO baseline trained without the belief state; its world model's accuracy climbs from 44% to 69% during RL, and the learned foresight capability transfers across harnesses and out-of-distribution domains. In summary, we aim to build a code world model that is omniscient over both time and environment state.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Code agent, World model, Large language model, Agentic reinforcement learning, Credit assignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zcGjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a reinforcement learning framework for learning internal world models for coding agents. The main innovation is that before each action, the agent outputs a “belief” representing the agent’s thinking of the state of the world, a prediction about what will happen next, and a decision whether to trust in the prediction or abstain. The predictions are benchmarked against the actual outcomes and used to assign credit to different steps, and at inference time, to select that best-forecast action or resample if the decision is to abstain. In the experiments, the method is compared with agents that do not explicitly output their forecasts before action.

Critical strengths

The method seems sensible and a good number of out-of-domain datasets is analysed in the experiments, with promising results.

Critical weaknesses

The difference to the closest methods has not been explained clearly, nor demonstrated in the experiments. Appendix A lists related work, and a reader is left with an impression that there are several previous works that do something similar even if not in exactly the same way. What is the delta to the closest methods, why is it significant, and is it possible to demonstrate the benefit in the experiments?

The presentation and figures are not clear enough. For example, looking at figures 1 and 2, the fonts are so small that it’s almost impossible to even see what is written, and it does not help that there are maybe 10 different colors, several font types, and decoration which is not always informative. The math in the main text is also not always clear, and the propositions do not seem to add much value to the story.

Questions for authors

Question: How did you set the hyperparameters?

Question: The paper writes that when assigning credits, BAPO compares “steps with the same forecast across rollouts of the same task, regardless of trajectory position”. Could you elaborate the intuition of this?

8 comments
AuthorsOpenReview ↗

Response to Reviewer zcGj (1/5)

We appreciate the time and care you devoted to reviewing our work! We address the related-work comparisons, presentation, and methodological questions below.

[W1. Discussion on closest work] What is the delta to the closest methods, why is it significant, and is it possible to demonstrate the benefit in the experiments?

Thank you for your careful review. Our closest comparisons are the Reactive and Retrospective baselines in our experiments (Introduction, Section 1; Experiments, Sections 3.1–3.2, Table 1; Appendix E.3). ANTE includes explicit forecasts, verification by real execution, and per-step policy credit within agentic RL (Section 1).

[W1-1. Closest comparisons: from thinking to memory and foresight]

Training recipe and Reactive. Recent code-agent training includes SFT on multi-turn coding trajectories containing reasoning, tool calls, and execution feedback, and RL through interaction with executable environments: the policy generates actions, observes their execution results, and updates from task-outcome rewards (e.g., SWE-Gym, R2E-Gym, and DeepSWE; Section 1). Reactive follows this SFT-then-RL recipe, learning the reasoning-and-tool-use loop without an explicit memory or belief block. Our tasks come from R2E-Gym, SWE-Gym, and SWE-rebench (Section 3.1; Appendix E.1). Reactive and Retrospective are defined in Section 3.1, with their paired corpus construction detailed in Appendix E.3.

Reactive → Retrospective → ANTE

  • Reactive: thinking and actions (the standard reasoning-and-tool-use paradigm in code-agent work; Section 1). The agent reasons over its interaction history and generates tool calls.
  • Retrospective: with memory. We retain ANTE's state-summary component as an explicit memory of past events and known repository state, without its foresight or trust/abstain decisions. The agent uses this memory when generating subsequent actions.
  • ANTE: with foresight and belief learning (including the state summary as memory). The summary is augmented with next-outcome forecasts and trust/abstain decisions. Real execution provides prediction targets, and BAPO uses verified decisions for belief-conditioned policy credit (Sections 2.1–2.3).

These are close comparisons by construction. Each recorded SFT run is rendered in three forms with the same task query and tool calls. RL shares the backbone, task pool, and scaffold, with rollouts from each variant's SFT checkpoint (Section 3.1; Appendix E.3). ANTE uses the additional learning signals defined in Section 2.3.

The corresponding results are as follows (Section 3.2, Table 1):

VariantSFTAfter RL
Reactive: thinking + actions45.851.0 ±0.6
Retrospective: thinking + memory48.252.4 ±0.4
ANTE46.457.2 ±1.1

Retrospective improves over Reactive by 1.4 points after RL. ANTE reaches 57.2%, improving over Reactive and Retrospective by 6.2 and 4.8 points, respectively.

Additional experiments on policy credit. The reported ANTE w/o BAPO variant retains the belief state and prediction loss but removes the decision reward and belief-conditioned step credit, using episode-level policy advantages (Section 3.1). We add two experiments that retain prediction loss, decision reward, RL task pool, and budgets while changing credit construction. The trajectory-level experiment gives every step the same centered trajectory-mean decision reward. The step-level experiment retains each eligible step's return-to-go and replaces both grouped baselines with the task mean, preserving normalization, clipping, and eligibility (Section 2.3; Appendix G.3).

Method32B SBV
ANTE w/o BAPO (Table 1)54.2
Trajectory-level decision reward54.8 ±0.64
Step-level, without belief grouping55.0 ±1.10
ANTE (Table 1)57.2 ±1.1

ANTE adds 3.0 points over w/o BAPO, 2.4 over trajectory-level reward, and 2.2 over ungrouped step credit. Episode credit retains the task objective; grouped step credit distinguishes decisions conditioned on the forecast. Proposition 3 explains how the two streams retain within-decision variation and trust/abstain credit before normalization and clipping (Appendix D.2). Additional 32B rollout analysis also shows first-post-edit forecast accuracy rising from 30.89% to 62.60% (our response to Reviewer 2, Q1).

AuthorsOpenReview ↗

Response to Reviewer zcGj (2/5)

[W1-2. Closest related work: methodological differences and external comparisons]

We have discussed the comparison with ECHO, including our reproduction on the same backbone and RL pool (Section 3.2). Given space constraints, further discussion and additional comparisons are provided in the appendix. We have also discussed these related methods and their differences from ANTE in the Introduction (Section 1).

  • ECHO — our closest external baseline. In terminal environments, ECHO co-trains policy optimization with an auxiliary cross-entropy loss on environment-observation tokens. The auxiliary loss trains the same model to predict terminal responses, while policy credit remains trajectory-level (ECHO, Section 3). ANTE trains an execution-grounded code world model whose verified beliefs shape policy optimization. Beyond prediction supervision, BAPO assigns step-level policy credit from forecast verification and trust/abstain decisions (Section 2.3).
  • CWM learns execution behavior through trace-based mid-training, followed by post-training. ANTE learns explicit forecasts during policy RL and uses their verification for step credit.
  • SWE-World trains separate transition and reward models to simulate execution feedback and evaluation. ANTE obtains these signals from real execution, without a separate learned simulator or critic.
  • VAGEN trains internal world models for VLM agents in visual environments, using language-model judging to supervise state estimation and transition prediction.

Comparison with ECHO. Our reproduction uses Qwen3.5-9B, the same RL pool and scaffold on TB 2.0 (Table 11):

MethodResolved (%)
ECHO, our reproduction21.4
ANTE24.7 (+3.3)

ANTE improves over ECHO by 3.3 points. This comparison tests the additional benefit of explicit belief learning and BAPO beyond auxiliary observation prediction; the reward-preserving experiments above further isolate the step-credit construction.

Comparison with other code-agent baselines. The reported 32B SBV results specify each method's backbone and scaffold (Table 10):

MethodBackboneScaffoldSBV
Meta-CWM-32BFrom scratchCWM53.9
SWE-World-32BQwen2.5-Coder-32BR2E-Gym55.0
ANTE-32BQwen3-32BR2E-Gym57.2

ANTE reaches 57.2%, above the reported CWM and SWE-World results.

Training configurations. CWM reports 5T mid-training tokens before post-training (CWM, Section 4.2). SWE-World uses 5.7K policy-SFT trajectories for five epochs, with separate 32B transition/reward models trained on 26K/21K examples for four/two epochs (SWE-World, Appendices B–C). ANTE uses 12.5K trajectories for three SFT epochs and 6.6K tasks for one RL epoch (Section 3.1; Appendix G.2, Table 9).

[W2-1. Readability of Figures 1 and 2] Improve the readability of Figures 1 and 2, including text size and colors.

Thank you for your valuable feedback on the presentation. Following your thoughtful suggestion on the font, we have enlarged the text and standardized the typography in Figures 1 and 2 in the revised version. We can further increase the text size in the camera-ready version. We would also like to respectfully clarify the content and organization of Figures 1 and 2 below (the colors follow a consistent coding scheme within each figure to distinguish the components and their roles).

  • Figure 1: behavior and empirical context. The upper panels compare Reactive reasoning with Retrospective memory; the ANTE panel shows an explicit repository belief and prospective reasoning. The bottom panels show task success and forecast accuracy during RL. Reasoning about possible edits is distinct from the forecast fields verified by executing the chosen command (Sections 2.1–2.3).

    • [Color code] Pink/brown, blue, and green distinguish thinking, memory, and ANTE belief; the method colors recur in the performance bars.
  • Figure 2: execution feedback and learning. From left to right, the figure shows execution-labeled BeliefTraj and SFT, the RL loop, and ANTE-TTO. Execution feedback supplies prediction targets and decision rewards; BAPO assigns grouped step credit alongside episode-level task credit (Section 2.3). The right panel shows how ANTE-TTO applies the learned beliefs to candidate selection at inference time (Section 2.4).

    • [Color code] Blue denotes forecasts, green observed outcomes, and orange learning signals, tracing the flow from prediction and execution to policy updates.
AuthorsOpenReview ↗

Response to Reviewer zcGj (3/5)

[W2-2. Clarity of the math and role of the propositions] Clarify the main-text mathematics and explain how the propositions support the method. “The math in the main text is also not always clear … the propositions do not seem to add much value to the story.”

We appreciate your careful reading of the theoretical analysis. The five propositions explain the training stages, the BAPO construction, and how learned forecasts support action selection. Their roles are:

PropositionRole in the method
1: Belief-augmented imitationSeparates imitation error from hindsight-label mismatch, motivating execution-grounded learning after SFT.
2: Selection regretBounds the utility lost when candidate selection relies on imperfect forecasts.
3: Anchored-advantage decompositionShows how the two baselines retain within-decision variation and trust/abstain credit.
4: Prediction–reward tradeoffInterprets the prediction-loss multiplier as an accuracy budget under the stated optimization assumption.
5: Prediction-loss information boundsRelates prediction loss to the outcome and environment-state information carried by the prediction context.
  • Proposition 1: the distinct roles of SFT and execution-grounded RL. Fitting hindsight-labeled demonstrations and predicting outcomes on the policy's own executions are different learning requirements. The bound separates accumulated imitation error from a remainder due to hindsight labels (Section 2.2; Appendix D.3). SFT initializes the belief-generating policy; Stage II trains its predictions against outcomes from its own interactions. The corresponding RL stage improves SBV by 2.4 / 4.8 / 10.8 points over ANTE-SFT (Section 3.2, Table 1).

  • Proposition 2: the consequence of forecast error for selection. The result bounds the selected action's expected-utility loss relative to the best available candidate using candidate-specific forecast divergences (Section 2.3). Corollary 3 adds point-forecast spread for the single emitted predictions used by the actual decoder (Appendix D.5, Eq. 16). It quantifies how prediction error can translate into a worse action choice.

    • In the additional 14B replay audit (our response to Reviewer 1, W4-2 & Q3), the mean estimated bound is 1.520, and 43/49 sets (87.8%) have estimates below the trivial maximum of 4. The mean measured loss is 0.122. A two-action example with true success probabilities (0.40, 0.60) and forecasts (0.501, 0.499) gives regret 0.800 versus a bound of 0.811. These measurements examine informativeness on real candidates and a concrete configuration where the bound closely tracks loss.
  • Proposition 3: the information retained by the two baselines. Centering returns separately within trust and abstain groups would make both groups' mean advantages zero, removing their average difference from that credit signal. BAPO combines forecast-relative and forecast–decision-relative streams. It retains both the step's deviation within its decision group and credit for choosing trust versus abstain, with empirical orthogonality before normalization and clipping (Appendix D.2, Eqs. 8–10).

    • The original ablation measures BAPO's combined reward and credit contribution. The additional experiment retains the reward and step-level returns: full ANTE exceeds the version without belief grouping by 1.2 / 1.6 / 2.2 points. This tests the practical benefit of the construction explained by Proposition 3. The reported learning-stage results are (Section 3.2, Table 1):

      Method7B14B32B
      ANTE-SFT24.841.446.4
      ANTE w/o BAPO25.443.654.2
      ANTE27.246.257.2
      ANTE + TTO28.648.657.8
  • Proposition 4: the role of prediction weight. Under its idealized optimization assumption, the multiplier indexes a family of prediction-loss budgets: increasing it tightens that budget, with non-increasing task return (Appendix D.4). The prediction weight has an explicit tradeoff interpretation: it determines how strongly the objective favors prediction accuracy alongside task success.

  • Proposition 5: why predicting checkable outcomes can teach state information. Under the POMDP assumptions, expected prediction loss upper-bounds conditional outcome uncertainty and yields a lower bound on the information the prediction context carries about the next environment state (Appendix D.5, Eqs. 13–15). At fixed conditional outcome entropy, lower prediction loss raises this information lower bound. Execution outcomes therefore provide supervision for learning state-relevant information, giving a formal basis for the forecast component of the internal world model.

Reviewer_BwwJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ANTE, an RL framework for code agents. Before each tool call, the agent writes a short belief state with three parts: a one-sentence progress summary, a forecast of the result of the check the action runs, and a trust/abstain decision. The forecast covers whether the target test passes, fails or gives no verdict, and whether the command exits cleanly. The model is first fine-tuned on solving trajectories from stronger models, with forecast labels filled in from the logged execution results. It is then trained with BAPO, which adds a prediction loss on the forecast, a reward for correctly trusted forecasts, and step-level credit from comparing steps in different rollouts that made the same forecast. At inference, ANTE-TTO resamples a few candidate actions when the agent abstains or predicts failure, and executes the one with the best self-forecast. The paper also gives several bounds that link forecast accuracy to imitation and action-selection quality. The method is tested on SWE-bench Verified (7B–32B) and three terminal benchmarks (9B), against baselines trained on the same trajectories without forecasts (thinking only, and thinking plus memory). ANTE beats both and reaches 57.2% on SWE-bench Verified at 32B. Forecast accuracy rises during RL, ANTE-TTO adds small further gains, and ANTE also improves on eleven out-of-domain reasoning benchmarks.

Critical strengths

The main comparison does not cleanly show that the forecasts cause the gains, and this is the main reason for my score. A larger evaluation budget is stated only for ANTE. In the main case study, a baseline run stops at the training step limit while it is still editing. If the baselines had a smaller budget, part of the gap comes from budget, not from the method. The headline gain is also measured against the only baseline trained without the format reward and budget warnings. Against the cleaner memory baseline on SWE-bench Verified, the forecast and its prediction loss add almost nothing at the two smaller scales. BAPO's extra few points come from a single training run, and the error bars only reflect re-evaluating one checkpoint.

Critical weaknesses

The claims go well beyond what the method actually does. The title, the abstract and Figure 1 all present the agent as foreseeing an edit's effect before it makes the edit. In Figure 1, the belief written before the edit compares two possible fixes. It predicts that changing the function's output would pass the target test but break other code that calls the function, while adding a default value keeps all tests passing, so the agent picks the second fix. The method as defined cannot produce such forecasts. The forecast that execution checks has only two discrete fields: the result of the check run in the current step, and whether the command exits cleanly. An edit step is always forecast as "no verdict" plus the exit code, and the summary gets no training signal in RL. So the forecast written before an edit never says whether the edit will make the tests pass or break other tests. The agent only forecasts a test result later, when it runs the tests, and by then the code has already been changed. For the same reason, ANTE-TTO cannot rank candidate edits by their effect on the tests. The rising hit rate may also partly come from easy checks on scripts the agent writes itself. The theory applies standard bounds under idealized assumptions that do not cover the algorithm as implemented. Calling this an internal world model, and the abstract's stated aim of one that is "omniscient over both time and environment state," goes far beyond this evidence.

Questions for authors
  1. The paper's key evidence that the world model gets better is the rising forecast hit rate on conclusive steps, that is, checks that return PASS or FAIL (Figure 5(a)). At 32B it goes from 44% to 69% during RL. These checks are tests that the agent chooses to run, including reproduction scripts it writes itself. The protocol also asks the agent to make each script print its own PASS or FAIL verdict (Appendix P). Some of these forecasts are easy. One example is predicting that a bug-reproduction script fails before any fix. Another is predicting that a script re-run with no code change gives the same result as last time. The paper's own example of the belief sharpening during RL (Appendix O.3) is of the first kind: the improved forecast is that the reproduction script fails on the still-unfixed code. BAPO also gives credit to the action at a step whose trusted forecast comes true, so the policy may even learn to run more such easy checks. Figure 18 shows that the share of scored forecasts stays stable, but not what kinds of checks they are. Could you split the hit rate in Figure 5(a) by type of check: checks run right after a code edit, reproduction runs before any fix, and re-runs with no code change in between? Could you also show how the share of each type changes during RL? Only the first type shows whether the agent can foresee the effect of its own edits.

  2. ANTE-TTO picks among candidates with a fixed score (Eq. 23). A forecast of PASS gets 3 points, "no verdict" gets 1, FAIL gets 0, and a predicted clean exit adds 1. Under the protocol, edits and views are always forecast as "no verdict", so they score at most 2, while a check forecast to pass scores 3 or 4. So any check predicted to pass beats every edit, and two different edits can only be told apart by their predicted exit code. In other words, the score favours optimistic forecasts: a useful check that is correctly expected to fail ranks below an unverified edit. The paper's own SWE example shows this (Figure 14 and Appendix O.1). At step 7, the drafted edit scored 2. The winning candidate ran the reproduction script and scored 4 because it forecast PASS, based on the wrong belief that the bug was not there. The script in fact failed. Had the forecast been correct (FAIL with a clean exit), this candidate would have scored only 1. The rule would then have kept the unverified edit that the paper says ANTE-TTO avoided. The TTO gains in Table 1 are also small (+1.4, +2.4 and +0.6 points) and have no error bars. At 7B and 32B they are smaller than ANTE's own run-to-run standard error. On the same candidates, how does ANTE-TTO compare with simple rules that ignore the forecast? Examples are picking a test-running candidate whenever there is one, or picking at random. How often is the winning candidate's forecast correct? Please report these with error bars.

6 comments
AuthorsOpenReview ↗

Response to Reviewer BwwJ (1/5)

We appreciate the time and care you devoted to reviewing our work and providing detailed feedback! We address each point below with experimental results, theoretical analysis, and clarifications.

  • [W1. Pre-action belief, edit effects, and the contribution of ANTE] Clarify what is predicted before an edit, what execution verifies, and how these capabilities support ANTE’s claimed contribution. “In Figure 1, the belief written before the edit compares two possible fixes. … The method as defined cannot produce such forecasts. … only two discrete fields … the summary gets no training signal in RL … ANTE-TTO cannot rank candidate edits by their effect on the tests.”

Thank you for your careful and insightful review. ANTE learns edit actions through task-success credit and belief-conditioned policy generation; its two forecast fields supervise the next command's verifiable outcome. These roles explain how execution prediction can support better editing (Sections 2.1–2.3).

  • What is explicitly predicted.
    • At an edit step, the check field is NA and the exit field predicts whether the edit command executes cleanly. The realized exit result still enters the prediction loss on verifiable edit steps (Section 2.3, Eq. 3; Appendix F.2, Table 8).
    • A PASS/FAIL forecast is made before executing a check on the current code. It does not certify the test outcomes of multiple unexecuted patches. Candidate edits with identical NA/exit forecasts tie under TTO, with preference for the draft (Appendix H.1).

The comparison of possible fixes in Figure 1 illustrates the policy's reasoning about the repository. The policy can reason about alternatives using its history and belief; the harness supervises the outcome of the command actually executed.

  • How the model learns better edits.
    • Edit actions receive episode-level policy credit. The final task outcome trains reasoning and actions, including edits whose check field is NA; edit quality is not learned from exit prediction alone (Section 2.3, Eqs. 3–5).
    • Prediction and action generation share model parameters. SFT trains reasoning, summaries, forecasts, and actions together. During RL, prediction supervision updates the same model that generates edits, while action generation conditions on the history, reasoning, and belief (Sections 2.1–2.2, Eq. 1).
    • Later checks inform subsequent revisions. Execution feedback enters the history used to generate the next belief and action. The summary is learned during SFT; masking its tokens from the RL losses does not remove this context or freeze the shared model (Appendix F.1, Figure 11).

Decision rewards and grouped step credit concentrate on conclusive checks. NA steps retain prediction supervision and episode credit, without earning a correctness reward merely for choosing an edit or view (Appendix F.2, Eq. 18). Proposition 3 explains the within-decision and trust/abstain components before normalization and clipping (Appendix D.2).

Experiments separating reward from credit construction. Alongside Table 1, we report two additional experiments on SBV with prediction loss, decision reward, RL task pool, and budgets fixed. The step-level experiment replaces both conditional baselines with the task mean and retains normalization, clipping, and eligibility rules (Section 2.3; Appendix G.3).

Credit construction7BΔ14BΔ32BΔ
ANTE w/o BAPO25.4—43.6—54.2—
Trajectory-level decision reward25.8 ±0.69↑0.444.0 ±0.53↑0.454.8 ±0.64↑0.6
Step-level, without belief grouping26.0 ±0.99↑0.644.6 ±0.87↑1.055.0 ±1.10↑0.8
Full ANTE27.2 ±1.7↑1.846.2 ±0.9↑2.657.2 ±1.1↑3.0

Δ: points over ANTE w/o BAPO; ±: s.e. over three evaluations of one checkpoint.

  • Belief-anchored credit adds 1.4 / 2.2 / 2.4 points over trajectory-level reward and 1.2 / 1.6 / 2.2 over ungrouped step credit. These comparisons isolate the contribution of credit construction with prediction supervision retained.
  • The benefit extends beyond a state summary: ANTE exceeds Retrospective-RL by 2.0 / 2.8 / 4.8 points. At 32B, ANTE w/o BAPO reaches 54.2 versus Retrospective-RL's 52.4; full ANTE reaches 57.2, a 10.8-point gain over ANTE-SFT (Section 3.2, Table 1).

For example, on psf__requests-1724, ANTE tracks the native-string fix, distinguishes pre-existing network failures from patch-induced failures, and preserves 56/56 previously passing tests while passing 6/6 target tests (Section 3.5; Appendix O.2). The results and case illustrate improved editing through the complete prediction-and-policy training recipe, without requiring explicit test-outcome forecasts for every hypothetical patch.

AuthorsOpenReview ↗

Response to Reviewer BwwJ (2/5)

  • [W2. How the theory connects to the implemented algorithm] Explain how the theoretical assumptions and bounds connect to the algorithm actually implemented. “standard bounds under idealized assumptions … do not cover the algorithm as implemented.”

We appreciate your valuable feedback on the theoretical analysis. The propositions analyze identifiable parts of ANTE: learning from execution, choosing an action from emitted forecasts, and assigning belief-conditioned credit. The appendix connects their assumptions to the implemented procedure; the added experiments examine their practical consequences.

  • Learning after SFT: why real execution matters. Proposition 1 separates imitation error from a remainder caused by hindsight labels (Section 2.2; Appendix D.3). Stage II obtains labels from the policy's own executions, so forecasts are trained on the outcomes of actions it actually takes.

  • Candidate selection: accounting for the actual decoder. Proposition 2 relates selection loss to candidate-specific forecast divergence (Section 2.3).

    • Corollary 3 covers a single emitted forecast, as used by TTO, through the additional point-forecast spread term (Appendix D.5, Eq. 16).
    • The conditioning identity accounts for writing the forecast before the action. Its information term captures the difference between predicting an outcome at the writing context and predicting the outcome of a particular action (Appendix D.5, Eq. 17).
    • The analysis uses the paper's bounded execution utility, instantiated by the 0–4 TTO score (Appendix H.1, Eq. 23). Task-level benefits are measured separately by the selection experiments.

Measured terms on real candidates. We replay the draft and three alternatives from each saved repository state at 14B. Candidate KL compares predicted and replay-estimated outcome distributions; point-forecast spread compares emitted scores with their predictive means. Selection loss compares the selected candidate with the best available candidate under the same execution utility. The nonlinear bound is calculated per set before averaging (Appendix D.5, Eq. 16).

MeasurementANTE-14BInterpretation
Valid / sampled candidate sets49 / 50Four candidates per valid set
Candidates with disagreement across initial replays0/196Observed execution repeatability
Mean maximum candidate KL (nats)0.129 [0.056, 0.216]Forecast mismatch for each candidate's own action
Point-forecast spread, ζ^\hat\zeta0.070 [0.037, 0.109]Emitted score versus its predictive mean
Mean estimated selection loss, rˉ\bar r0.122 [0.041, 0.224]Utility lost relative to the best available candidate
Mean estimated bound, bˉ\bar b1.520 [1.069, 2.051]Estimated upper bound on that loss
Mean bound minus loss1.398 [1.029, 1.828]Conservativeness of the estimate
Sets with estimated bound below 443/49; 87.8% [77.6, 95.9]Estimates better than the trivial maximum

Brackets show confidence intervals conditioned on the saved environment states.

The estimated bound is below the trivial maximum of 4 for 43/49 sets (87.8%). Its mean is 1.520, with 1.380 contributed by candidate divergence and 0.140 by point-forecast spread. Mean measured loss is 0.122; the 1.398 gap quantifies conservativeness. Identical-action sets have zero selection loss by construction; comparisons with Random assess ranking quality separately.

  • When the bound closely tracks loss. In the two-action experiment with true success probabilities (0.40, 0.60) and forecasts (0.501, 0.499), the forecast ranking is reversed. Exact regret is 0.800 versus a bound of 0.811, within 1.4%. With correct predictive probabilities (0.10, 0.90), expected-value selection incurs zero loss, but a greedy original forecast and sampled alternative incur expected point-selection loss 0.320. The corresponding point-forecast spread is 0.720 and the bound is 1.440. The extra term is relevant even when candidate KL is zero. These calculations instantiate Proposition 2 and Corollary 3 under the bounded-utility assumptions.

  • Grouped credit: the decomposition used by BAPO. Proposition 3 decomposes the raw advantage into within-decision variation and credit for trust versus abstain, with empirical orthogonality before normalization and clipping (Appendix D.2, Eqs. 8–10; Section 2.3, Eqs. 4–5). The implemented algorithm then applies the normalization, clipping, and fallback rules of Appendix G.3. With these rules and the decision reward retained, full ANTE improves by 1.2 / 1.6 / 2.2 points over ungrouped step credit.

The numerical audit evaluates the selection bound; the ablations evaluate the implemented credit construction. Together they give quantitative evidence for the specific mechanisms analyzed by the theory.

AuthorsOpenReview ↗

Response to Reviewer BwwJ (3/5)

  • [W3. What the internal world model represents and what the evidence shows] Clarify the scope of the internal world model and align the framing with the evidence. “internal world model … omniscient over both time and environment state … goes far beyond this evidence.”

Thank you for your thoughtful and valuable review. ANTE's execution-grounded internal world model maintains an explicit repository belief and predicts outcomes that execution can verify. The state summary supplies context for action generation; the forecast supplies a supervised prediction target (Sections 2.1–2.3).

  • What is represented and updated.
    • Current state: the summary records established facts, modifications, verified behavior, and unresolved failures. It is learned during SFT and regenerated throughout interaction (Sections 2.1–2.2; Appendix F.1).
    • Anticipated outcome: the forecast predicts the next command's check result and exit status; trust/abstain expresses whether the policy commits to that forecast (Section 2.1).
    • Subsequent revision: execution verifies the forecast and updates the history used for the next belief. BAPO uses verified decisions during training, and ANTE-TTO uses candidate forecasts for inference-time selection (Sections 2.3–2.4).

The forecast fields are the directly supervised predictive component of a broader belief-conditioned policy. They do not enumerate every possible repository transition. Action generation also uses the state summary, reasoning, and accumulated observations (Section 2.1, Eq. 1).

  • Why this is a world-modeling formulation. VAGEN [1] combines state estimation with transition prediction in an explicit internal belief. ANTE similarly maintains state information and predicts future outcomes, but verifies its forecast fields through execution. MuZero [2] learns predictions of reward, policy, and value for planning, illustrating that a useful environment model can focus on decision-relevant quantities rather than reconstructing every detail. These precedents motivate the representation; ANTE's experiments establish its usefulness in code-agent learning.

  • Theoretical support for state-relevant prediction. Under the paper's POMDP assumptions, the prediction loss provides a formal connection to environment-state information: I(ct;st+1∣at)≥I(ct;yt∣at)≥H(yt∣at)−Lˉt(θ)I(c_t;s_{t+1}\mid a_t)\ge I(c_t;y_t\mid a_t)\ge H(y_t\mid a_t)-\bar{\mathcal L}_t(\theta) (Section 2.3, Eq. 3; Appendix D.5, Proposition 5, Eqs. 13–14). Here ctc_t is the prediction context, st+1s_{t+1} the next state, yty_t the verified outcome, ata_t the action, and Lˉt\bar{\mathcal L}_t the expected prediction cross-entropy. At fixed conditional outcome entropy, reducing prediction loss raises this lower bound on state information. Thus, checkable outcomes provide supervision for learning information about the environment.

For example, case analysis of psf__requests-1724. The trajectory illustrates state tracking, failure attribution, and verification of the repair (Section 3.5; Appendix O.2):

Evidence on psf__requests-1724Observed behavior or outcome
Failure attributionDistinguishes pre-existing network failures from patch-induced failures
Tracking the modificationRecords the native-string fix in the evolving belief
Target behavior after repair6/6 target tests pass
Previously passing behavior56/56 tests remain passing

The belief distinguishes what was already broken from what the patch changed, and retains what execution has verified. The trajectory resolves the issue in 27 steps, with both the repaired target behavior and previously passing tests verified by execution.

Evidence beyond this case.

  • Forecast hit rate rises from 44% to 69% during RL (Section 3.5, Figure 5(a)). The added breakdown shows the largest increase on the first check after a source-code change: 30.89% to 62.60%.
  • ANTE exceeds Retrospective-RL, which already maintains a state summary, by 2.0 / 2.8 / 4.8 points (Section 3.2, Table 1). The gains measure the usefulness of the complete prospective belief-training recipe beyond retaining a summary.

We will replace “omniscient” with execution-grounded internal world model, specifying the state information maintained and outcomes predicted. Forecast hit rate measures prediction accuracy; task results and case analyses establish the broader usefulness of the belief representation.

References

[1] Wang et al. VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents. NeurIPS, 2025.

[2] Schrittwieser et al. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature, 2020.

Reviewer_3RuaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ANTE makes each agent step start with a belief state: a summary plus a prediction of the execution outcome (test PASS/FAIL/NA, abnormal exit) and a TRUST/ABSTAIN decision. Predictions are grounded by real execution labels, used for supervised warm-up, then optimized jointly with the task reward; BAPO groups trajectories by prediction × decision to anchor step-level credit, with an additional decision reward. At inference, ANTE-TTO samples K=3 candidates and selects by predicted score, without an external scorer. On SWE-bench Verified the method improves 7B/14B/32B models from 24.4/41.8/51.0 to 27.2/46.2/57.2, and prediction accuracy rises from 44% to 69%. The paper also reports Terminal-Bench results, 11 out-of-domain benchmarks, a cost analysis, and a theoretical appendix.

After rebuttal: Thanks for the detailed response, the score is already positive.

Critical strengths
  1. The mechanism is concrete and falsifiable: predict → verify by execution → convert into step-level credit → select at inference. Each interface is machine-readable, unlike another reflection prompt.
  2. The controlled baselines are well designed for this literature: Reactive and Retrospective variants with paired SFT corpora and a paired RL pool, three model scales, two harnesses.
  3. The theory appendix is not decorative: it explicitly carries residual and conditioning terms (Appendix D.5, Eq. 16–17) instead of presenting a clean bound that ignores them.
  4. Cost transparency: the 1.31× token overhead of TTO is reported, including discarded candidates.
Critical weaknesses
  1. BAPO is not isolated. The "w/o BAPO" condition removes both the decision reward and belief-anchored grouping, so the reported drop supports the combination but not the claimed novelty of grouping by belief. Fixing the prediction loss and decision reward while only changing the advantage construction would settle this.
  2. The TTO gain is close to noise and lacks the decisive control. Gains are +1.4/+2.4/+0.6, while the reported error bars come from repeated evaluation of one checkpoint, not training seeds; +0.6 is within noise. More importantly, there is no control at the same trigger positions, candidate count, and budget using random or simple selection, so it is unclear how much of the gain comes from belief ranking rather than extra sampling.
  3. "World model" overstates what is modeled. What is predicted is a discrete test outcome and exit status — a result predictor, not a state model of the environment. The out-of-domain +6.2 average (MMLU-Pro +10.4) is presented with a stronger causal reading than a single belief intervention supports.
  4. Theory-to-practice gap. Proposition 2 depends on inter-candidate KL, a mutual-information term, and the point-prediction error ζ; none of these are estimated empirically, so validity of the bound does not imply tightness.
Questions for authors
  1. Can you report a BAPO isolation run: same prediction loss, same decision reward, same data and budget, only the step-level advantage construction changed? How much of the +1.8–3.0 remains?
  2. For TTO, can you compare against random/simple candidate selection at identical trigger positions, K, and token budget, with variance over training seeds?

3)Can you estimate the extra terms in Proposition 2 (candidate divergence, ζ) on a small sample, to indicate whether the bound is tight enough to be informative?

4)Would you consider narrowing the "world model" framing to execution-result prediction, and qualifying the cross-domain causal claim accordingly?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 3Rua (1/6)

We sincerely thank you for your careful review and valuable feedback! We address each point below with additional experiments, theoretical analysis, and clarifications.

  • [W1-1 & Q1. additional step-level advantage construction comparison] Fixing the prediction loss and decision reward while only changing the advantage construction (belief-anchored grouping), only the step-level advantage construction changed

Thank you for your thoughtful feedback. Belief-anchored grouping provides additional gains when the decision reward is retained at either the trajectory or step level. Alongside the ANTE and w/o BAPO results (Section 3.2, Table 1), we report two additional experiments on SBV with the prediction loss and decision reward fixed. All variants use a shared RL task pool and the same training/evaluation budgets (Section 3.1; Appendix G.2, Table 9).

  • [A. Trajectory-level experiment] In the trajectory-level experiment, we average decision rewards within each rollout and center these means across rollouts of the same task. Every step then receives the same added credit.
  • [B. Step-level experiment] In the step-level experiment, we retain each eligible step's return-to-go and replace both conditional baselines with the task mean (Section 2.3, Eq. 4). We keep the normalization, clipping, and eligibility rules unchanged (Appendix G.3), isolating the grouping construction when compared with full ANTE.
MethodDecision rewardCredit construction7BΔ14BΔ32BΔ
ANTE (w/o BAPO)✗Episode credit25.4—43.6—54.2—
ANTE (trajectory-level decision reward)✓Episode + centered trajectory-mean reward25.8 ±0.69↑0.444.0 ±0.53↑0.454.8 ±0.64↑0.6
ANTE (step-level, w/o belief grouping)✓Episode + ungrouped step credit26.0 ±0.99↑0.644.6 ±0.87↑1.055.0 ±1.10↑0.8
ANTE✓Episode + belief-anchored step credit27.2 ±1.7↑1.846.2 ±0.9↑2.657.2 ±1.1↑3.0

Δ: points over ANTE w/o BAPO; ±: s.e.

Trajectory-level reward adds 0.4 / 0.4 / 0.6 points, and ungrouped step credit adds 0.6 / 1.0 / 0.8 points. Full ANTE adds a further 1.2 / 1.6 / 2.2 points over the step-level method without belief grouping. Decision-reward supervision is useful, and belief-anchored credit provides a substantial further gain.

Averaging rewards over a trajectory scales each check's contribution by rollout length and distributes it uniformly across steps. Step credit retains local differences; grouping further conditions the comparison on the forecast and decision.

As shown in Proposition 3, BAPO retains both within-decision variation and credit for trust versus abstain, with empirical orthogonality before normalization and clipping (Appendix D.2, Eqs. 8–10).

AuthorsOpenReview ↗

Response to Reviewer 3Rua (2/6)

  • [W2 & Q2. ANTE-TTO seed & selection control & tokens] For TTO, can you compare against random/simple candidate selection at identical trigger positions, K, and token budget, with variance over training seeds?

Thank you for your careful reading. Belief ranking improves mean task success over Random both across repeated evaluations of one checkpoint and across independently trained checkpoints. We add these experiments to distinguish useful selection from additional candidate sampling, following the TTO protocol (Section 2.4; Appendix H.1). Besides, we would like to respectfully clarify that TTO triggers when the draft forecasts FAIL or chooses ABSTAIN. This rule acts on the model's outputs at each step; trigger positions are not manually chosen (Appendix H.1, Algorithm 3). Further ANTE-TTO exp settings:

  • Matched candidate sampling. K=3 means four candidates in total including the original one. Random and ANTE-TTO retain the same tasks, trigger rule, candidate-generation settings, and budget; Random selects uniformly.
  • Evaluation and training seeds. Our reported ANTE scores come from multi-run evaluation of one trained checkpoint (Section 3.2, Table 1), following a typical evaluation protocol in code-agent work such as DeepSWE and CWM (Section 7.2). We evaluate Random and ANTE-TTO through multi-run evaluation on our original checkpoint (training seed = 42). We additionally conduct a training-seed experiment by adding two checkpoints trained with seeds 43 and 44, giving three checkpoints independently trained with different training seeds.
Method7B14B32B
ANTE (Table 1)27.2 ±1.746.2 ±0.957.2 ±1.1
Random (multi-run)27.13 ±0.7446.07 ±0.8756.53 ±2.57
ANTE-TTO (multi-run)29.00 ±1.4048.80 ±1.1058.20 ±1.63
ANTE (3 training seeds)27.20 ±1.3946.40 ±1.4557.07 ±0.93
Random (3 training seeds)26.47 ±1.6245.93 ±1.0556.47 ±1.39
ANTE-TTO (3 training seeds)29.13 ±1.6448.87 ±1.3958.67 ±1.73

ANTE-TTO exceeds Random by 1.87 / 2.73 / 1.67 points in multi-run evaluation and 2.67 / 2.93 / 2.20 points across training seeds; Random has slightly lower means than ANTE. The benefit comes with belief-guided selection, rather than appearing automatically from extra sampling. The dependence of selection quality on forecast error is analyzed in Proposition 2 and its point-forecast extension (Section 2.3; Appendix D.5, Corollary 3).

We have reported both total and marginal token costs in Section 3.4, Figure 4, and Appendix H.1. Total token cost measures tokens used relative to one greedy attempt; MTC measures extra tokens per percentage point gained; nMTC expresses that per-point cost in greedy-token budgets.

TTS-4/8 use best-of-n sampling over four/eight full trajectories. The following averages cover SBV-7B/14B/32B and Terminal-9B, including prompts, generations, and discarded candidates.

Cost metricANTE-TTOTTS-4TTS-8
BtokB_{\mathrm{tok}}1.6×4.1×8.0×
nMTC0.433.235.65

Local selection with learned beliefs substantially reduces token cost per point gained. ANTE-TTO's average MTC is approximately 1/10 that of TTS-4 and 1/17 that of TTS-8 (Section 3.4, Figure 4). On SBV-7B, total token ratios are 1.31× versus 4.42× for ANTE-TTO and TTS-4 (Section 3.4).

  • Candidate-count sensitivity. For each model scale, we compare K=3, 5, and 7 in three runs on the reported checkpoint. The original candidate is generated using greedy decoding; the additional candidates are sampled at temperature 1.0. K=3 uses the same multi-run results as the selection experiment.
K7B7B BtokB_{\mathrm{tok}}14B14B BtokB_{\mathrm{tok}}32B32B BtokB_{\mathrm{tok}}
329.00 ±1.40≈1.32×48.80 ±1.10≈1.45×58.20 ±1.63≈1.38×
527.47 ±0.96≈1.54×49.07 ±1.22≈1.76×59.20 ±1.14≈1.64×
729.33 ±2.08≈1.75×48.27 ±1.11≈2.06×58.40 ±0.81≈1.89×

K=3 provides an efficient default: additional candidates cost more without consistently improving success. ANTE-TTO further improves ANTE’s strong task performance through efficient belief-guided selection at inference time.

AuthorsOpenReview ↗

Response to Reviewer 3Rua (3/6)

  • [W3-1 & Q4-1. Scope of the term "world model"] Clarify what the internal world model represents beyond a discrete execution-result predictor.

We appreciate your careful reading of our work. ANTE's execution-grounded internal world model combines an explicit belief about the repository state with verifiable predictions of action consequences. The forecast is its directly supervised predictive component; the belief also supplies context for action generation (Sections 2.1–2.3).

  • The belief represents repository state as well as the expected execution outcome.

    The belief contains a state summary, outcome forecast, and trust/abstain decision, with distinct roles:

    • State summary. The summary records established facts, modifications, verified behavior, and unresolved failures. It is learned during SFT and generated throughout interaction, providing context for subsequent reasoning and action generation (Sections 2.1–2.2; Appendix F.1).
    • Outcome forecast and decision. The forecast predicts the next command's verifiable execution outcome; the trust/abstain decision indicates whether the agent commits to that prediction. After execution, the observed result verifies the forecast and updates the history used to generate subsequent beliefs (Sections 2.1–2.3).
  • The forecast is used to assign training credit and choose candidate actions.

    The predictive component has a specific role in both training and inference:

    • During training. BAPO uses verified decisions to construct step-level policy credit, comparing decision returns within forecast and forecast–decision groups (Section 2.3).
    • During inference. ANTE-TTO uses candidate forecasts to select the action to execute, allowing the agent to use its predictions when choosing among alternatives (Section 2.4).
  • ANTE improves over an agent that already maintains a state summary.

    The experiments and trajectory analysis examine task performance, forecast accuracy, and the information recorded in the belief:

    • Task performance. ANTE exceeds Retrospective-RL, which maintains a state summary, by 2.0 / 2.8 / 4.8 points on SBV at 7B / 14B / 32B (Section 3.2, Table 1).
    • Forecast accuracy. Forecast hit rate rises from 44% to 69% at 32B during RL, showing that the execution-outcome predictions become more accurate during training (Section 3.5, Figure 5(a)).
    • Repository state tracking. For example, on psf__requests-1724, the belief distinguishes pre-existing network failures from patch-induced failures and tracks the native-string fix. The trajectory resolves the issue in 27 steps, with 6/6 target tests passing and 56/56 previously passing tests preserved (Section 3.5; Appendix O.2).

    The comparison with Retrospective-RL establishes the task-level benefit of the full ANTE method, while the training curve and case study show how its forecasts improve and its belief tracks repository changes.

  • Predicting execution outcomes provides information about the environment state.

    • Theoretical interpretation. Under the stated POMDP assumptions, Proposition 5 relates expected prediction loss to a lower bound on the information the prediction context carries about the next environment state (Appendix D.5, Eqs. 13–15). At fixed conditional outcome entropy, reducing the loss raises that lower bound. This provides theoretical support for learning information about the environment by predicting checkable outcomes.
    • Related world-model formulations. This interpretation is consistent with VAGEN's state estimation and transition prediction [1], RWML's textual action-consequence modeling [2], and MuZero's prediction of decision-relevant future quantities [3].

ANTE makes its forecast target directly checkable through real execution.

References

[1] Wang et al. VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents. NeurIPS, 2025.

[2] Yu et al. Reinforcement World Model Learning for LLM-based Agents. arXiv:2602.05842, 2026.

[3] Schrittwieser et al. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. arXiv:1911.08265, 2019.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Sincerely Looking Forward to the Reviewers' Feedback 🙏

Dear reviewers,

We sincerely appreciate your careful evaluation and thoughtful feedback. We hold the utmost respect for your hard work!

As the review discussion stage is drawing to a close, we humbly seek to ascertain whether our responses adequately address your concerns. We are deeply grateful for your attention to this matter.

We summarize our responses to each reviewer as below:

(W: weakness; Q: question)

  • We supplemented our analysis with trajectory-level and ungrouped step-level experiments that retain decision rewards to isolate the contribution of belief-anchored advantage construction. (Reviewer 3Rua [W1-1 & Q1]; Reviewer zcGj [Q2])
  • We provided further analysis of forecast accuracy and check-type proportions during RL, distinguishing post-edit checks, pre-fix reproductions, and unchanged-code reruns. (Reviewer BwwJ [Q1])
  • We included additional multi-run and training-seed evaluations to further assess the robustness of the ANTE-TTO results reported in the paper. (Reviewer 3Rua [W2 & Q2]; Reviewer BwwJ [Q2]; Reviewer zcGj [Q1])
  • We added Random/Test-first comparisons, candidate-count ablations, and winner-forecast accuracy measurements for ANTE-TTO. (Reviewer 3Rua [W2 & Q2]; Reviewer BwwJ [Q2]; Reviewer zcGj [Q1])
  • We clarified the world-model formulation, learning signals for edits, and interpretation of cross-domain transfer using the paper's results and case analyses. (Reviewer 3Rua [W3-1 & Q4-1, W3-2 & Q4-2]; Reviewer BwwJ [W1, W3])
  • We estimated candidate divergence and point-forecast error through a real-candidate replay audit, added two-action experiments on bound tightness, and highlighted how Propositions 1–5 support the method. (Reviewer 3Rua [W4-1–W4-3 & Q3]; Reviewer BwwJ [W2]; Reviewer zcGj [W2-2])
  • We explained the progression from Reactive to Retrospective and ANTE, and elaborated on the differences from ECHO and other closely related methods using the comparisons reported in the paper. (Reviewer zcGj [W1, Q2])
  • We have enlarged and standardized the fonts in Figures 1–2, and explained their color coding and the hyperparameter choices documented in the paper. (Reviewer zcGj [W2-1, Q1])
  • We highlight ANTE's core contribution in belief learning and policy optimization, and explain ANTE-TTO's additional inference-time benefits through the paper's performance and token-cost comparisons with TTS-4/8. (Reviewer 3Rua [W2 & Q2]; Reviewer BwwJ [Q2])

If you have any additional questions or require further clarification, please do not hesitate to inform us. We value the opportunity to continue the discussion if necessary. 🙏

Best regards,

The authors

AuthorsOpenReview ↗

Sincerely Looking Forward to the Reviewers' Feedback Once Again

Dear reviewers,

We genuinely appreciate the depth of your insightful comments and the value of your feedback. Your dedicated efforts are truly esteemed.

Due to the deadline is approaching, we sincerely hope you can take a look at it🙏🙏. We faithfully hope our response adequately address your concern.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.