acceptodds
Under review as a conference paper at ICLR 2027

Software World Models: From Consequence Prediction to Decision Value

Abstract

A coding agent may safely modify one repository while silently breaking downstream services, libraries, or datastores that depend on it. Exhaustively running integration tests after every agent action is impractical, so the agent must predict these failures before executing them. Existing software world models predict the agent’s own observations, while static change-impact analysis only identifies where a change may propagate. We instead introduce the Software World Model (SWM), which models the broader system affected by a code change and predicts its blast set: the components that the change will break. SWM follows three stages: explore, learn, and act. Explore executes candidate changes from restored system states, prioritizing regions where observed failures contradict the dependency graph. Learn fine-tunes a language model on these execution outcomes to predict downstream breakage. Act converts sampled predictions into per-consumer break probabilities for change ranking, proactive migration, and deciding when another execution is worth its cost. On held-out synthetic systems, SWM improves blast-set F1 from 0.431 for static reachability to 0.571 ± 0.037, more than halves ranking regret, and improves migration return at all nine evaluation checkpoints. The two methods are complementary: reachability is stronger on dependencies represented in the graph, while SWM recovers failures caused by couplings the graph misses. Experiments on held-out real libraries further show that predicting structured failure outcomes, rather than only scalar risk, is important for downstream decision quality.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models, LLM Agents, Software Engineering, Partial Observability, Environment Modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hVXyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a practical problem in software engineering agents: a change may pass all tests in the repository being edited while still breaking downstream services or libraries. The authors propose a Software World Model (SWM) to predict which downstream components will fail after a proposed change. The proposed pipeline has three parts. First, the system executes candidate changes from restored software states and collects their downstream consequences. It gives more attention to cases where the dependency graph disagrees with observed execution. Second, a language model is fine-tuned on these execution records to predict the affected components, failure type, and possible failure channel. Finally, the predictions are converted into per-consumer failure probabilities and used to rank changes, decide whether to migrate a consumer in advance, or determine whether an additional check is worthwhile. The results are generally positive. SWM improves over typed reachability on the synthetic benchmarks and also reports gains on the real-library experiments. The paper also makes a useful distinction between prediction accuracy and the value of the decisions induced by a predictor.

Critical strengths

The problem is well motivated. Current coding agents usually validate the repository they modify, but this does not say much about the behavior of downstream consumers. The paper focuses on a failure mode that is easy to overlook and has clear practical importance. The same-state branching procedure is one of the stronger parts of the work. By executing several candidate changes from the same restored state, the authors make the comparison between alternatives more meaningful. The targeted exploration of cases where static reachability is wrong is also sensible, especially when the execution budget is limited.

Critical weaknesses

My main concern is that the “world model” framing is somewhat broader than what is actually evaluated. The main agent experiment does not show a coding agent freely proposing and implementing arbitrary patches over a long software-engineering trajectory. The interface is closer to identifying affected consumers and selecting preventive migrations for a given change. This is a useful task, but the paper should be more precise about what kind of agent improvement is being demonstrated. The method also relies on executed consequences from resettable environments. This is reasonable for collecting training data, but it may be difficult to reproduce in a large production system where downstream services are not all accessible and failed changes cannot always be safely rolled back. The synthetic propagation rules are also designed by the authors, and the real-library evaluation covers only a limited range of projects. I am therefore not fully convinced that the current evidence supports broad generalization.

Questions for authors

Could the authors clarify the exact interface used in the main agent experiment? Is the model generating code changes, or is it given a change and asked to identify and migrate affected consumers? The current description sometimes sounds broader than the actual evaluation.

How much of the improvement comes from the proposed consequence-learning objective rather than ordinary task adaptation? A matched SFT baseline with the same data, inputs, and output format would be helpful.

Why is a language model with GRPO preferable to a simpler structured risk model in this setting? The paper should compare them under the same decision interface, rather than only comparing their prediction scores under different protocols.

How does SWM perform on completely unseen libraries, new versions, or dependency patterns that are not represented in the training worlds?

5 comments
AuthorsOpenReview ↗

Response to Reviewer hVXy (Part 1 of 3)

We thank the reviewer for recognizing the practical importance of downstream breakage prediction, the value of same-state counterfactual execution, and our targeted exploration of failures missed by static reachability. We also appreciate the constructive comments.


[Weaknesses] "... the 'world model' framing is somewhat broader than what is actually evaluated ... the method relies on executed consequences from resettable environments ... the synthetic propagation rules are designed by the authors ... the real-library evaluation covers only a limited range of projects ..."

Response:

  • Agent evaluation scope (Q1): In our Explore–Learn–Act pipeline, the policy model generates code changes, while SWM predicts their downstream consequences to guide preventive migration and verification. We clarify this division of responsibilities and the specific decision interface evaluated in our experiments.
  • Consequence learning versus task adaptation (Q2): We provide matched supervised fine-tuning controls to distinguish the benefits of downstream consequence supervision from generic task adaptation, and further evaluate the contribution of the complete SWM training recipe.
  • Comparison with structured risk models (Q3): We extend our comparisons to strong relation/action risk predictors, larger SWM backbones, and downstream decision outcomes, clarifying the advantages and limitations of each approach.
  • Generalization and practical applicability (Q4): We clarify our world-level train/test separation and report evidence from three held-out real libraries, including failures missed by extracted dependency graphs. We also discuss the role of resettable execution environments in offline data acquisition and the limitations of extrapolating these results to unrestricted production systems.

[Q1] Could the authors clarify the exact interface used in the main agent experiment? Is the model generating code changes, or is it given a change and asked to identify and migrate affected consumers? The current description sometimes sounds broader than the actual evaluation.

Response: Our Explore–Learn–Act pipeline distinguishes two roles: the policy model proposes code changes, while SWM learns to predict their downstream consequences from execution-verified experience. The learned knowledge can improve the acting agent through two integration mechanisms:

\boxed{\text{Policy Model: Propose Code Changes}} \\\\ \downarrow \\\\ \boxed{\text{Explore: Execute Changes and Observe Consequences}} \\\\ \downarrow \\\\ \boxed{\text{Learn: Train SWM}} \\\\ \downarrow \\\\ \begin{array}{c|c} \text{Internalized SWM} & \text{External SWM Guidance} \\\\ \hline \boxed{ \begin{array}{c} \text{SWM-Trained Acting Agent} \\\\ \downarrow \\\\ \text{Identify Affected Consumers} \\\\ \downarrow \\\\ \text{Select Preventive Migrations} \\\\ \downarrow \\\\ \text{Agent Return} \end{array} } & \boxed{ \begin{array}{c} \text{SWM Predicts Consequences} \\\\ \downarrow \\\\ \text{Provide Predictions to Policy} \\\\ \downarrow \\\\ \text{Policy Selects Actions} \end{array} } \end{array} \end{array}$$ Our main experiment demonstrates **how SWM with consequence learning improves coding-agent decisions**, while code-change generation is handled by the policy model itself. We have clarified this distinction in Section 4.1 and the corresponding experimental protocol.
AuthorsOpenReview ↗

Response to Reviewer hVXy (Part 2 of 3)

[Q2] How much of the improvement comes from the proposed consequence-learning objective rather than ordinary task adaptation? A matched SFT baseline with the same data, inputs, and output format would be helpful.

Response: We conducted matched SFT ablations using the same 3B backbone, training data, input representations, optimization budget, and two-stage agent evaluation interface, varying only the supervision objective. All checkpoints act directly as agents under the same output protocol, without an external SWM advisor.

\hline \text{Method} & \text{Training Objective} & \text{Return (40)} & \text{Return (360)} \\\\ \hline \text{Frozen agent} & \text{None} & 0.721 & 0.668 \\\\ \text{Observation-only SFT} & \text{Observations} & 0.558 & 0.436 \\\\ \text{Blast-set SFT} & \text{Downstream consequences} & 0.754 & 0.725 \\\\ \text{Structured SFT} & \text{Four-field consequences} & 0.732 & 0.670 \\\\ \hline \text{Full SWM} & \text{SFT + GRPO} & \color{green}{0.773} & \color{green}{0.755} \\\\ \hline \end{array}$$ Observation-only SFT underperforms the frozen agent, whereas blast-set SFT improves return from **0.721 to 0.754** on 40 tasks and from **0.668 to 0.725** on 360 tasks. The **full SWM recipe** further improves return to **0.773 and 0.755**, respectively. The improvement also extends to 7B, where consequence-only SFT achieves 0.790 and full SWM reaches 0.813, compared with 0.695 for the frozen agent. We have incorporated these controls and their experimental protocols into the revised appendix. --- > **[Q3]** Why is a language model with GRPO preferable to a simpler structured risk model in this setting? The paper should compare them under the same decision interface, rather than only comparing their prediction scores under different protocols. **Response:** We clarify **(1) why consequence-aware GRPO offers a richer learning objective than structured risk scoring**, and **(2) how the two approaches compare under the same agent decision interface**, including a concrete failure case. **1. Mechanism: consequence learning versus structured risk scoring.** The structured risk model estimates independent per-consumer break probabilities from fixed graph and action features. It is effective when these features capture the relevant dependencies, but cannot directly exploit semantic evidence absent from its feature representation. In contrast, SWM interprets code-change context and consumer-specific usage evidence, learning structured downstream consequences beyond extracted dependency paths. GRPO aligns training with execution-verified outcomes. This directly optimizes consequence fidelity at the set level, encouraging accurate identification of affected consumers while penalizing false alarms. **2. Quantitative comparison under the same decision interface.** We evaluate both models through a fixed two-stage agent interface: the same actor first identifies candidate consumers, then receives per-consumer risk guidance and selects preventive migrations. On 40 identical held-out tasks, only the guidance source changes: $$\begin{array}{l|c} \hline \text{Guidance Source} & \text{Agent Return} \\\\ \hline \text{No guidance} & 0.771 \\\\ \text{Relation/action logistic} & 0.779 \\\\ \text{SWM (3B)} & 0.796 \\\\ \hline \end{array}$$ Our subsequent scaling and internalized-agent evaluations provide complementary evidence: - **Consequence prediction:** SWM achieves synthetic blast-set F1 of **0.761 (3B)** and **0.784 (7B)** versus **0.671** for logistic, and real-library F1 of **0.718 (7B)** versus **0.672**. - **Agent decisions:** The full SWM-trained agent achieves return of **0.773 (3B)** and **0.813 (7B)**, compared with **0.794** for the explicit supervised-risk plan on the same tasks. The 3B difference is not statistically significant, while the 7B result is a higher point estimate. Unlike the explicit risk plan, the SWM-trained agent must identify candidate consumers itself. - **Training recipe:** Consequence-only SFT achieves return of **0.754 (3B)** and **0.790 (7B)**; the complete SFT + GRPO recipe raises these point estimates to **0.773** and **0.813**, respectively.
AuthorsOpenReview ↗

Response to Reviewer hVXy (Part 3 of 3)

3. Case study: avoiding unnecessary migrations.

Consider a synthetic software world with five downstream services, where an upstream change causes no actual failures. The logistic risk model nevertheless assigns billing a break probability of 0.490, triggering an unnecessary migration. GRPO trains SWM across other software worlds to distinguish actual breakage from apparent dependency risks, rewarding correct no-break predictions over false alarms.

\boxed{\text{Training: Executed Changes Across Different Worlds}} \\\\ \downarrow \\\\ \boxed{\text{Experience: Changes Without Downstream Breakage}} \\\\ \downarrow \\\\ \text{GRPO: Compare Sampled Consequence Predictions} \\\\ \begin{array}{cc} \text{Correct: Empty Blast Set} & \text{Incorrect: False Break Prediction} \\\\ \color{green}{F_1=1} & \color{red}{F_1=0} \end{array} \\\\ \downarrow \\\\ \color{blue}{\boxed{\text{Learned Knowledge: Risk Does Not Imply Breakage}}} \end{array}$$ By learning from execution-verified no-break cases across training worlds, GRPO reinforces the distinction between potential dependency risks and actual downstream failures. This learned distinction can help SWM avoid unnecessary interventions in unseen software environments. $$\begin{array}{c} \boxed{\text{Held-Out World: Upstream Change Causes No Breakage}} \\\\ \downarrow \\\\ \begin{array}{c|c} \text{Logistic-Guided Agent} & \text{SWM-Guided Agent} \\\\ \hline \text{Fixed Graph/Action Features} & \color{blue}{\boxed{\text{Learned Knowledge: Risk Does Not Imply Breakage}}} \\\\ \downarrow & \downarrow \\\\ \text{Billing Risk: }0.490 & \text{Recognize No-Break Pattern} \\\\ \downarrow & \downarrow \\\\ \boxed{\text{Migrate Billing}} & \boxed{\text{No Migration}} \\\\ \downarrow & \downarrow \\\\ \text{Unnecessary Migration Cost} & \text{Avoided Migration Cost} \\\\ \end{array} \end{array}$$ --- > **[Q4]** How does SWM perform on completely unseen libraries, new versions, or dependency patterns that are not represented in the training worlds? **Response:** We evaluate generalization through **world-level held-out splits** in both real-library and synthetic environments, while distinguishing unseen software worlds from entirely unseen failure mechanisms. **1. Generalization to unseen real libraries.** We train on three upstream-library worlds (jsonschema, Jinja2, and Werkzeug) and evaluate on Attrs, Requests, and Starlette. - The 0.5B SWM achieves 0.644 blast-set F1 versus 0.671 for typed reachability, showing that smaller SWM models do not consistently outperform structural baselines. - Among 457 breaking records, **137 are missed by extracted-graph reachability**. Incorporating consumer-specific usage context improves SWM's hidden-channel F1 from 0.142 to 0.198, compared with 0.137 for reachability. **2. Generalization to unseen synthetic worlds.** We extend the main evaluation from **40 tasks across 7 held-out worlds** to **480 migration tasks across 60 held-out worlds**. These worlds are generated using the same deterministic software-world generator, but with a separate evaluation seed offset, ensuring **no overlap with training worlds**. Each world contributes six migration tasks. The frozen 3B model achieves return 0.651, while SWM (3B) checkpoint achieves **0.724–0.729** across three seeds, with positive paired improvement. These experiments establish transfer to unseen world instances under the same generator, not generalization to arbitrary new dependency distributions. We have clarified these distinctions and the corresponding limitations in the revised manuscript.
Reviewer_RwJrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Software World Models (SWM) to predict which downstream components will break after a locally valid software change. Its Explore–Learn–Act framework collects counterfactual execution outcomes from restored states, trains a consequence predictor, and uses predicted break probabilities for change ranking, preventive migration, and selective checking. Experiments cover procedural systems and held-out real libraries. The paper reports improved agent returns and fewer downstream failures, and analyzes how incomplete dependencies, exploration policies, and prediction errors affect decision quality.

Critical strengths
  1. It addresses a relevant problem with a well-motivated data collection design. Predicting downstream breakage under incomplete dependency information addresses an important limitation of repository-local verification. Executing alternative changes from the same restored state makes their consequences meaningfully comparable. The controlled synthetic study provides supporting evidence: all four full-budget training corpora improve return over the base agent, with positive paired world-bootstrap intervals in Table 2.

  2. It conducts useful analysis of when predictions improve decisions. Figures 8–9 examine direct versus hidden propagation and show that similar blast-set F1 can produce different ranking and migration outcomes. The paper also openly reports stronger supervised controls and cases where additional execution is preferable. These analyses make the paper informative beyond its headline performance gains.

Critical weaknesses
  1. The distinctive benefit of the proposed modeling approach remains insufficiently isolated. Relevant precedents include Predictive Test Selection, which learns failure probabilities from historical execution outcomes, and SWE-World, which generates structured test feedback and uses predicted outcomes for solution selection. SWM’s distinctive contribution therefore rests particularly on downstream consumers, incomplete dependencies, and targeted acquisition. arxiv.org However, the paper’s own relation/action risk control achieves higher synthetic forecast F1 than the headline SWM recipe (0.671 versus 0.571, Figure 8a), and higher belief-planning return (0.794 versus 0.757, Figure 9a). These are companion experiments; the stronger controls have yet to be compared through the main agent interface. Consequently, Table 1 establishes benefits from consequence training more clearly than benefits attributable to SWM’s particular formulation.

  2. The real-library agent evaluation is insufficiently specified to assess the broader claims. The main synthetic study contains 40 tasks from seven worlds and uses a constrained consumer-listing/migration-selection interface. The real-library results could materially strengthen the paper, but their agent-task population, complete project/version splits, and migration mechanics are unclear. The 984 records in Figure 6 belong to a separate prediction evaluation. Moreover, Table 1’s real-library returns are difficult to reconcile with the stated cost model and the four downstream packages per named world in Table 9. This requires clarification before the reported decision gains can be interpreted confidently.

Questions for authors
  1. How do supervised-only consequence fine-tuning and the strongest per-consumer risk predictor perform under the same data, training budget, and acting interface as Table 1? Which real-library experiment supports the abstract’s claim that structured failure outcomes improve decisions over scalar risk?

  2. Please specify the real-library agent evaluation population and reconcile its return accounting. Under the stated return formula without paid feedback, Table 1 implies approximately 8.0 migrations per task for the base agent and 5.2 for SWM, whereas Table 9 describes four downstream packages per named world. Are additional consumers, repeated actions, or other costs involved? This is an unresolved accounting question, rather than evidence by itself of incorrect results.

5 comments
AuthorsOpenReview ↗

Response to Reviewer RwJr (Part 1 of 3)

We thank the reviewer for recognizing the importance of downstream failure prediction under incomplete dependencies, our restore-and-execute data collection design, and our analysis connecting prediction quality to agent decisions. We appreciate the constructive feedback and have incorporated additional experimental evidence and clarifications into the revised manuscript and appendix.


[W1] "The distinctive benefit of the proposed modeling approach remains insufficiently isolated ... the relation/action risk control achieves higher synthetic forecast F1 ... and higher belief-planning return ... the stronger controls have yet to be compared through the main agent interface."

Response: We thank the reviewer for suggesting these comparisons. We clarify SWM's contributions along three dimensions:

  • Methodological novelty: How SWM differs from existing approaches in its prediction target, targeted acquisition, and downstream decision-making.
  • Performance against strong supervised controls: How model scaling changes the comparison in consequence prediction and agent decision quality.
  • Consequence supervision and the full pipeline: How we disentangle the benefits of learning downstream consequences from generic fine-tuning.

1. Methodological novelty beyond existing approaches

Our contribution is not execution-based prediction alone, but learning and acting on downstream consequences under incomplete dependency information.

MethodPrediction targetDecision objective
Predictive Test SelectionP(yjfail=1∣d,H)P(y_j^{fail}=1\mid d,H)Prioritize tests
SWE-WorldP(ot+1tool,r∣ht,at)P(o_{t+1}^{tool},r\mid h_t,a_t)Simulate feedback; select solutions
CWMP(τexec∣c)P(\tau_{exec}\mid c)Predict code execution
Qwen-AgentWorldP(ot+1∣ht,at,c)P(o_{t+1}\mid h_t,a_t,c)Simulate environment observations
SWMP(B(st,at),ot+1cons∣ht,at)P(B(s_t,a_t),o_{t+1}^{cons}\mid h_t,a_t)Predict downstream failures; prevent breakage

Here dd is a code change, HH execution history, τexec\tau_{exec} an execution trace, and B(st,at)B(s_t,a_t) the downstream blast set. Formulas summarize modeling targets rather than exact training losses.

CapabilityPTSSWE-WorldCWMAgentWorldSWM
Execution-grounded learning●●●●●
Explicit downstream blast-set prediction○○○○●
Same-state counterfactual execution○○○○●
Targeted acquisition of graph blind spots○○○○●
Preventive migration and cost-aware checks○○○○●

● Explicitly addressed; ○ Not an explicit target or evaluated mechanism in the cited work.

SWM differs in three essential aspects:

  • Prediction: Models which downstream consumers break after a locally passing change, rather than only test failure likelihood or next environment observations.
  • Acquisition: Executes alternative changes from restored states and prioritizes discrepancies between observed consequences and extracted-graph reachability.
  • Decision: Translates predicted consequences into candidate ranking, preventive migrations, and selective paid verification.

These components form our Explore–Learn–Act framework. We have added detailed comparisons with Predictive Test Selection, SWE-World, CWM, and Qwen-AgentWorld to the revised appendix.

2. Comparison with the relation/action risk control.

We extend this comparison to larger predictors, held-out real libraries, and downstream decisions, using the same frozen evaluation sets within each setting. With larger backbones, SWM surpasses the supervised risk scorer and improves on held-out real libraries.

\hline \text{Evaluation} & \text{Relation/action Risk} & \text{SWM} & \text{Difference} \\\\ \hline \text{Forecast F1, 0.5B} & 0.671 & 0.571 & \color{red}{-0.100} \\\\ \text{Forecast F1, 0.5B SFT} & 0.671 & 0.608 & \color{red}{-0.063} \\\\ \text{Forecast F1, 3B} & 0.671 & 0.761 & \color{green}{+0.090} \\\\ \text{Forecast F1, 7B} & 0.671 & 0.784 & \color{green}{+0.113} \\\\ \hline \text{Real-library F1, 7B} & 0.672 & 0.718 & \color{green}{+0.046} \\\\ \hline \text{Belief-plan return, 0.5B} & 0.794 & 0.757 & \color{red}{-0.037} \\\\ \text{Belief-plan return, 3B} & 0.794 & 0.791 & \color{red}{-0.003} \\\\ \text{Agent return, 3B} & 0.794 & 0.773 & \color{red}{-0.021} \\\\ \text{Agent return, 7B} & 0.794 & 0.813 & \color{green}{+0.019} \\\\ \hline \end{array}$$ At 3B/7B, SWM achieves higher forecast F1 (0.761/0.784 vs. 0.671), with paired 95% CIs excluding zero, and improves real-library F1 (0.718 vs. 0.672) by capturing graph-missed failures. In downstream decisions, the 3B SWM nearly matches the supervised-risk plan (0.773 vs. 0.794; difference not significant), while the 7B agent exceeds it in point estimate (0.813 vs. 0.794). **These results show that the baseline's small-scale advantage does not persist at larger model scales**.
AuthorsOpenReview ↗

Response to Reviewer RwJr (Part 2 of 3)

3. Contributions Beyond Consequence Supervision

We thank the reviewer for recognizing the contribution of learning downstream consequences. SWM's novelty extends beyond this objective to the complete Explore–Learn–Act pipeline, combining (1) targeted acquisition of graph-missed failures through same-state counterfactual execution, (2) consequence-aware training for downstream blast-set prediction, and (3) decision-aware integration for preventive migration and selective verification.

To further isolate consequence supervision from generic agent adaptation, we supplement our existing experiments with matched 3B SFT controls, holding training data, backbone, acting interface, and optimization budget fixed while varying the supervision objective.

\hline \text{Training Recipe} & \text{Training Objective} & \text{Return (40)} & \text{Return (360)} \\\\ \hline \text{Frozen} & \text{None} & 0.721 & 0.668 \\\\ \text{Observation-only SFT} & \text{Observation} & 0.558 & 0.436 \\\\ \text{Delta-only SFT} & \text{Blast set} & 0.754 & 0.725 \\\\ \text{Structured SFT} & \text{Four-key consequences} & 0.732 & 0.670 \\\\ \hline \text{Full SWM} & \text{SFT + GRPO} & 0.773 & 0.755 \\\\ \hline \end{array}$$ **The results confirm that consequence supervision provides benefits beyond generic fine-tuning:** Moreover, SWM's contribution is not limited to supervision: its **targeted counterfactual acquisition** provides informative experience under incomplete dependencies, while **decision-aware integration** converts predicted consequences into preventive actions. Our main experiments (Tables 1–2) further demonstrate the effectiveness of the complete pipeline, including stronger agent returns than CWM and Qwen-AgentWorld. We have supplemented the revised appendix with these methodological distinctions. --- > **[W2]** "The real-library agent evaluation is insufficiently specified ... agent-task population, complete project/version splits, and migration mechanics are unclear ... returns are difficult to reconcile with the stated cost model."\ > **[Q2]** "Please specify the real-library agent evaluation population and reconcile its return accounting ... approximately 8.0 migrations per task for the base agent and 5.2 for SWM ... four downstream packages per named world ... Are additional consumers, repeated actions, or other costs involved?" **Response:** **Response:** We thank the reviewer for identifying these ambiguities. We clarify **(1) the real-library evaluation settings and world construction**, and **(2) the migration mechanics and return accounting**. **1. Real-library evaluation settings and world construction.** Each executable world consists of an upstream library and its downstream dependent packages, all pinned to fixed commits. We establish a passing baseline, restore it before each intervention, retain upstream changes that pass their own tests, and execute downstream test suites to obtain ground-truth failure labels. The train/test split is defined by **upstream library**, with three training worlds and three held-out worlds: | Split | Upstream library | Downstream consumers | Records | |---|---|---|---:| | Train | jsonschema 4.26.0 | nbformat, openapi-spec-validator, openapi-schema-validator | 455 | | Train | Jinja2 3.1.6 | Flask, Cookiecutter | 475 | | Train | Werkzeug 3.1.8 | Flask, Flask-WTF, Flask-Login, Flask-CORS | 578 | | Test | Attrs 26.1.0 | environ-config, jsonschema, attrs-strict | 358 | | Test | Requests 2.34.2 | requests-mock, requests-toolbelt, requests-oauthlib | 357 | | Test | Starlette 1.6.0 | FastAPI, starlette-context | 269 | Two evaluation populations are distinguished: - **Consequence prediction (Figure 6):** The 984 held-out records correspond to independently executed upstream changes with verified downstream failure outcomes. Each change also defines an agent migration task, in which the agent anticipates affected consumers, selects preventive migrations, and receives a return based on remaining downstream failures and migration costs (Table 1). - **Agent migration (Table 1):** Each task provides a pending change; the agent first identifies affected consumers and then selects preventive migrations. On real libraries, a migration involves patching a consumer and verifying compatibility with its test suite, potentially requiring revisions.
AuthorsOpenReview ↗

Response to Reviewer RwJr (Part 3 of 3)

2. Migration mechanics and return accounting.

We define the return as J=1−0.25nbroken−0.05(nfixes+1)J = 1 - 0.25n_{\mathrm{broken}} - 0.05(n_{\mathrm{fixes}}+1). We clarify that nfixesn_{\mathrm{fixes}} counts migration attempts rather than distinct downstream consumers. In real-library environments, a consumer patch may fail validation and require multiple revisions and resubmissions, each charged separately. Thus, the approximately 8.0 and 5.2 attempts per task implied by Table 1 can exceed the number of downstream packages.

We acknowledge that the original description of nfixesn_{\mathrm{fixes}} as the number of migrated consumers was potentially misleading. We have clarified its definition in the revised manuscript:

-0.05\left(\sum_{u\in C}m_u+1\right), \qquad n_{\mathrm{fixes}}=\sum_{u\in C}m_u.$$ where $m_u$ denotes the number of patch attempts for consumer $u \in C$, including unsuccessful attempts, and $n_{\mathrm{broken}}$ counts consumers remaining broken. The additional unit represents applying the upstream change. **This reformulation makes attempt-level accounting explicit without changing the original return function.** We have updated **Section 4.1 and the corresponding appendix sections** to clarify the real-library world construction, complete project/version splits, evaluation populations, migration mechanics, and attempt-level return accounting. These revisions explicitly distinguish prediction records from agent episodes and migration attempts from distinct downstream consumers, resolving the ambiguities identified by the reviewer. --- > **[Q1]** "How do supervised-only consequence fine-tuning and the strongest per-consumer risk predictor perform under the same data, training budget, and acting interface ... Which real-library experiment supports the abstract's claim ... over scalar risk?" **Response:** **Response:** We address **(1) supervised-only training and the strongest per-consumer risk baseline**, and **(2) the real-library evidence supporting structured consequence prediction**. **1. Supervised-only fine-tuning and per-consumer risk controls.** We evaluate supervised-only consequence training using the same 3B/7B backbones, and 40 held-out migration tasks as Table 1. $$\begin{array}{l|l|c|c} \hline \text{Method} & \text{Training Configuration} & \text{Return 3B} & \text{Return 7B} \\\\ \hline \text{Base agent} & \text{No fine-tuning} & 0.721 & 0.695 \\\\ \text{Blast-set SFT} & \text{Consequence target} & 0.754 & 0.790 \\\\ \text{Full SWM} & \text{SFT + GRPO} & \color{green}{0.773} & \color{green}{0.813} \\\\ \hline \end{array}$$ Compared with the **per-consumer risk plan (0.794)**, the 3B SWM achieves statistically comparable return (0.773), while the **7B SWM exceeds it in point estimate (0.813)**, despite having to identify candidate consumers itself. Consequence-only SFT improves agent performance over the base agent at both scales, while the complete SWM recipe (SFT + GRPO) further increases return from 0.754 to 0.773 at 3B and from 0.790 to 0.813 at 7B, demonstrating the additional value of the full training pipeline in point estimates. **2. Real-library evidence and the structured-versus-scalar claim.** We acknowledge that our original abstract overstated the evidence for the decision-level superiority of structured predictions over scalar risk in all cases. Nevertheless, **our real-library experiments establish the following important advantages**: - **Stronger consequence prediction:** The 7B SWM achieves 0.718 blast-set F1, outperforming the relation/action logistic (0.672) and typed reachability (0.671). - **Recovery of graph-missed failures:** Incorporating consumer-specific usage context improves SWM's hidden-channel F1 from 0.142 to 0.198, compared with 0.137 for reachability, demonstrating its ability to anticipate downstream failures beyond extracted dependency paths. These findings provide direct real-library evidence for SWM's prediction advantages, while distinguishing them from an unisolated decision-level superiority claim. To more accurately reflect these findings, **we have revised the abstract** to state: *"Experiments on held-out real libraries further demonstrate SWM's ability to predict downstream consequences and recover failures missed by incomplete dependency information."*
Reviewer_LUGwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Software World Models (SWM), which predict which downstream components a code change will break. SWM explores counterfactual changes in executable worlds, fine-tunes a language model on the executed outcomes, and uses sampled break probabilities for change ranking, preventive migration, and paid checks. On synthetic and real-library worlds, it reports gains over a base agent, typed reachability, and reimplemented CWM and Qwen-AgentWorld baselines.

Critical strengths
  • It targets an important, underexplored problem: breakage outside the agent’s workspace that repository tests miss.
  • It evaluates forecasts by their decision value, not just accuracy.
Critical weaknesses
  • Simple baselines match or beat SWM: logistic regression has higher blast-set F1 (0.671 vs ≤0.608), and a supervised risk scorer plans better (0.794 vs 0.757).
  • On real libraries, typed reachability (0.671) matches or exceeds the learned variants (≤0.668), so most gains rest on synthetic worlds with hand-specified propagation.
Questions for authors
  • Which backbone was used where, given that Section 4.1 says 3B but Table 13 lists 0.5B?

  • How does an agent fine-tuned on the same data without consequence targets compare?

  • How was the fidelity of the CWM and Qwen-AgentWorld reimplementations validated?

4 comments
AuthorsOpenReview ↗

Response to Reviewer LUGw (Part 1 of 3)

We thank the reviewer for the careful evaluation and constructive feedback. We address all concerns with additional experimental evidence and methodological clarifications. We have incorporated these clarifications and additional results into the revised manuscript and appendix.


[W1] Simple baselines match or beat SWM: logistic regression has higher blast-set F1 (0.671 vs ≤0.608), and a supervised risk scorer plans better (0.794 vs 0.757).

Response: The supervised risk baseline is competitive in small-scale, in-distribution settings, but its advantage does not persist as SWM scales. SWM benefits from increased model size:

1. Scaling reverses the forecast comparison. SWM substantially improves with model scale, surpassing the supervised risk baseline at 3B and 7B on the same frozen evaluation set.

\hline \text{Forecast Evaluation} & \text{Logistic} & \text{SWM} & \text{Difference} \\\\ \hline \text{Synthetic F1, 0.5B} & 0.671 & 0.608 & \color{red}{-0.063} \\\\ \text{Synthetic F1, 3B} & 0.671 & 0.761 & \color{green}{+0.090} \\\\ \text{Synthetic F1, 7B (s0)} & 0.671 & 0.784 & \color{green}{+0.113} \\\\ \hline \text{Real-library F1, 7B} & 0.672 & 0.718 & \color{green}{+0.046} \\\\ \text{Real hidden-channel F1} & 0.137 & 0.198 & \color{green}{+0.061} \\\\ \hline \end{array}$$ SWM surpasses the logistic baseline at 3B (+0.090, 95% CI [+0.042, +0.140]) and 7B (seed 0: +0.113 [+0.048, +0.177]; seed 1: +0.106 [+0.043, +0.169]). All three reported paired intervals exclude zero, confirming the forecast advantage at larger model scales. **2. Scaling closes the decision-performance gap.** At 3B, SWM nearly matches the supervised-risk plan, while the 7B SWM agent exceeds it in point estimate. $$\begin{array}{l|c|c|l} \hline \text{Decision Evaluation} & \text{Risk Plan} & \text{SWM} & \text{Difference} \\\\ \hline \text{Belief-plan return, 0.5B} & 0.794 & 0.757 & \color{red}{-0.037} \\\\ \text{Belief-plan return, 3B} & 0.794 & 0.791 & \color{red}{-0.003} \\\\ \hline \text{Agent return, 3B} & 0.794 & 0.773 & \color{red}{-0.021} \\\\ \text{Agent return, 7B} & 0.794 & 0.813 & \color{green}{+0.019} \\\\ \hline \end{array}$$ Importantly, the supervised-risk plan receives candidate consumers directly, whereas the SWM agent must first identify affected consumers and then select migrations. Despite this more demanding interface, the 3B agent is statistically comparable to the risk plan (95% CI for the return difference: [-0.053, +0.011]), and the 7B agent achieves a higher point estimate. **Overall, the baseline's advantage at small scale does not generalize across model scales and evaluation settings.** SWM's forecast gains become substantial at 3B–7B, and translate into a narrowing decision gap and favorable large-scale agent-return estimates. --- > **[W2]** On real libraries, typed reachability (0.671) matches or exceeds the learned variants (≤0.668), so most gains rest on synthetic worlds with hand-specified propagation. **Response**: To clarify, our real-library evaluation is grounded in **execution-verified downstream failures, not synthetic propagation rules**. On these real libraries, SWM surpasses typed reachability at larger model scales and captures failures beyond the extracted dependency graph. **1. SWM outperforms structural baselines on real libraries at larger scales.** While the 0.5B predictor performs comparably to typed reachability, **the 7B SWM achieves higher aggregate F1**. Moreover, SWM identifies hidden-channel failures that static graph traversal systematically misses. $$\begin{array}{l|c|c|l} \hline \text{Real-library Evaluation} & \text{Baseline} & \text{SWM} & \text{Difference} \\\\ \hline \text{Overall F1, 0.5B} & 0.671 & 0.668 & \color{red}{-0.003} \\\\ \text{Overall F1, 7B} & 0.671 & 0.718 & \color{green}{+0.047} \\\\ \text{Overall F1, 7B vs. logistic} & 0.672 & 0.718 & \color{green}{+0.046} \\\\ \text{Hidden-channel F1} & 0.137 & 0.198 & \color{green}{+0.061} \\\\ \hline \end{array}$$ Of the 984 held-out records, 457 exhibit downstream breakage, including **137 breaking records missed by the extracted dependency graph**. Incorporating consumer-specific usage context improves SWM's hidden-channel F1 from 0.142 to 0.198, compared with 0.137 for typed reachability. This improvement demonstrates the value of richer semantic inputs, while the 7B results establish SWM's advantage in aggregate consequence prediction.
AuthorsOpenReview ↗

Response to Reviewer LUGw (Part 2 of 3)

2. Case study: predicting failures beyond extracted dependencies. Consider the archived Marshmallow C3_06 perturbation, which renames Schema.TYPE_MAPPING to FIELD_TYPE_MAPPING. Although all 1,184 upstream tests pass, the change breaks downstream consumers through dependencies missed by the static extractor.

\boxed{\text{Marshmallow: schema mapping renamed}} \\\\ \downarrow \\\\ \boxed{\text{Upstream: 1,184 tests passed}} \\\\ \downarrow \\\\ \boxed{\text{Unobserved downstream dependency}} \\\\ \downarrow \\\\ \boxed{\text{environs: 15 failures; SQLAlchemy: 66 failures + 22 errors}} \end{array}$$ **SWM addresses this by combining semantic usage evidence with execution-grounded consequence learning**, improving hidden-channel F1 by 45% relative to static reachability on the separate held-out real-library evaluation: $$\begin{array}{c|c} \text{Static Reachability} & \text{SWM + Semantic Usage} \\\\ \hline \boxed{\text{Extracted graph only}} & \boxed{\text{Graph + consumer usage}} \\\\ \downarrow & \downarrow \\\\ \text{Missing dependency path} & \text{Learned execution consequences} \\\\ \downarrow & \downarrow \\\\ \text{Cannot detect hidden path} & \text{Can predict beyond graph paths} \\\\ \hline \text{Held-out hidden F1: }0.137 & \text{Held-out hidden F1: }0.198 \end{array}$$ --- > **[Q1]** Which backbone was used where, given that Section 4.1 says 3B but Table 13 lists 0.5B? **Response:** We thank the reviewer for identifying this inconsistency. **The correct backbone size for the main experiments is 3B for both the policy agent and the Software World Model (SWM).** The 0.5B entry in Table 13 was an outdated configuration from our earlier experiments exploring different SWM model sizes. To clarify the distinction between the two components: - **Policy agent (Section 4.1):** Uses the model as the backbone for interactive decision-making, with learned consequence knowledge internalized into the agent rather than supplied by a separate advisor. - **Software World Model (Table 13):** Uses the model for downstream consequence prediction, supporting blast-set forecasting, candidate ranking, and consequence-aware planning. We have updated Section 4.1, Table 13, and the corresponding experimental descriptions to consistently identify the 3B configuration used in the main experiments and distinguish it from the exploratory model-size settings. --- > **[Q2]** How does an agent fine-tuned on the same data without consequence targets compare? **Response:** **Response:** We conducted controlled experiments to isolate the contribution of consequence supervision from generic agent fine-tuning. Experimental settings: - **Backbone and interface:** All methods use the same Qwen2.5-Coder-3B-Instruct backbone and identical two-stage agent interface (candidate identification followed by migration decisions). Trained checkpoints act directly as agents, without a separate advisor. - **Training protocols:** We distinguish two settings: - **Matched SFT controls:** The same training data, matched inputs, optimization budgets, and three training seeds. Only the supervision objective differs. - **Full SWM:** Our main consequence-learning recipe uses 360 executed records per acquisition corpus, with 200 SFT warm-up steps followed by 250 GRPO steps. Four acquisition corpora and three seeds per corpus are evaluated. - **Compared methods:** - **Frozen:** Original 3B backbone without fine-tuning. - **Observation-only SFT:** Fine-tuned to reconstruct probe observations without explicit downstream consequence targets. - **Delta-only SFT:** Fine-tuned on executed downstream blast sets. - **Structured SFT:** Fine-tuned on four fields (delta, outcome, observation, mechanism). - **Full SWM:** Our complete SFT + GRPO consequence-learning pipeline. - **Evaluation:** All methods are compared on **40 held-out migration tasks**. The matched SFT controls are additionally evaluated on **360 hidden-channel-heavy tasks** from 60 held-out worlds. Training and evaluation worlds are disjoint. $$\begin{array}{l|c|c|c} \hline \text{Training Recipe} & \text{Training Objective} & \text{Return (40)} & \text{Return (360)} \\\\ \hline \text{Frozen} & \text{None} & 0.721 & 0.668 \\\\ \text{Observation-only SFT} & \text{Observation} & 0.558 & 0.436 \\\\ \text{Delta-only SFT} & \text{Blast set} & 0.754 & 0.725 \\\\ \text{Structured SFT} & \text{Four-key consequences} & 0.732 & 0.670 \\\\ \hline \text{Full SWM} & \text{SFT + GRPO} & 0.773 & 0.755 \\\\ \hline \end{array}$$ The new comparison shows that observation-only fine-tuning substantially underperforms consequence-supervised training. We have incorporated this additional ablation and its experimental details into the revised paper.
AuthorsOpenReview ↗

Response to Reviewer LUGw (Part 3 of 3)

[Q3] How was the fidelity of the CWM and Qwen-AgentWorld reimplementations validated?

Response: We adopt controlled 3B versions of CWM and Qwen-AgentWorld to compare their world-modeling formulations under matched conditions. Directly comparing their released checkpoints would confound differences in model scale, pretraining data, computational budget, and training environments. Our adaptations instead standardize the backbone, executed software-world data, training environment, and two-stage agent evaluation protocol, while preserving the defining objectives of the respective methods.

  • CWM: It learns code and environment dynamics through large-scale mid-training on Python execution traces and agent–environment interactions, followed by instruction SFT and multi-task RL using a modified asynchronous, multi-turn GRPO. Our reduced-scale adaptation replaces large-scale mid-training with supervised learning on the available action–observation trajectories, followed by same GRPO setting of our SWM. This preserves the emphasis on predicting observable environment transitions.
  • Qwen-AgentWorld: We adapt its CPT → SFT → RL pipeline by performing lightweight CPT on our executed software trajectories, using next-token prediction over serialized action–observation sequences to learn environment transitions, followed by SFT + GRPO under the same training configuration as SWM. This preserves the core next-observation modeling objective while standardizing the downstream optimization protocol for a controlled comparison.

Our adaptations preserve each method's core world-modeling objective and training-stage design, while standardizing the 3B backbone, training data, computational budget, agent interface, and evaluation metrics to ensure comparability with SWM. We have expanded Appendix D.6 to document the adaptations, their departures from the original recipes, and the corresponding evaluation protocols.

AuthorsOpenReview ↗

Follow-up on Rebuttal

We sincerely thank the reviewer again for the thoughtful and constructive feedback. As the discussion phase is approaching its end, we would like to briefly follow up on our rebuttal.

We have provided additional experiments and methodological clarifications to address your specific concerns:

  • Strong supervised-risk baselines: New 3B/7B scaling experiments and downstream decision comparisons.
  • Real-library generalization: Execution-verified results on held-out libraries, including failures missed by static dependency analysis.
  • Backbone and training configurations: Clarification of model sizes and experimental settings.
  • Matched fine-tuning controls: Additional ablations isolating consequence supervision from generic task adaptation.
  • CWM and Qwen-AgentWorld adaptations: Detailed explanations of their training objectives, implementation differences, and controlled comparison protocols.

We hope these additions satisfactorily address your concerns. If any questions remain, we would be very happy to provide further clarification or discussion before the discussion phase concludes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary

We sincerely thank all reviewers and the Area Chair for their careful evaluations, constructive feedback, and time throughout the discussion. We summarize the strengths highlighted in the reviews and the additional evidence provided in response.

1. Strengths Recognized by the Reviewers

  • LUGw: Identified downstream breakage outside an agent's workspace as an important, underexplored problem, and valued evaluating forecasts by their downstream decision utility rather than accuracy alone.
  • RwJr: Recognized the well-motivated restore-and-execute counterfactual data collection, positive agent-return evidence, and analyses explaining when predictive accuracy translates into better decisions.
  • hVXy: Highlighted the practical significance of the failure mode, same-state branching, and targeted exploration of failures missed by static dependency reachability.

2. Concerns Addressed Through Additional Experiments and Clarifications

  • Strong risk and structural baselines (LUGw W1–W2; RwJr W1/Q1; hVXy Q3): New scaling and controlled comparisons show synthetic blast-set F1 of 0.784 for 7B SWM versus 0.671 for logistic risk, and real-library F1 of 0.718 versus 0.672 for logistic risk and 0.671 for typed reachability. The 7B agent reaches 0.813 return versus 0.794 for the explicit risk-planning baseline, a favorable point estimate.

  • Consequence learning versus ordinary task adaptation (LUGw Q2; RwJr Q1; hVXy Q2): Matched SFT ablations control for backbone, data, budget, and acting interface. On 40 held-out tasks, consequence-only SFT improves return from 0.721 (frozen) to 0.754, and the full SFT+GRPO recipe reaches 0.773; additional evaluations cover 360 tasks.

  • Real-library evidence and generalization (LUGw W2; RwJr W2; hVXy Q4): We clarified three training and three held-out real-library worlds, with execution-verified downstream failures. Among 457 breaking held-out records, 137 are missed by extracted-graph reachability; consumer-usage evidence improves hidden-channel F1 to 0.198 versus 0.137 for reachability. We additionally evaluate held-out synthetic worlds while explicitly distinguishing world-level transfer from generalization to entirely unseen failure mechanisms.

  • Agent protocol and return accounting (RwJr W2/Q2; hVXy Q1): We specify the two-stage consumer-identification and migration interface, complete library/version splits, and why attempted patch revisions—not merely distinct consumers—are charged in the migration-cost term.

  • Experimental clarity and related methods (LUGw Q1/Q3; RwJr W1): We corrected the backbone configuration description, distinguished the roles of the policy and SWM, and documented the controlled CWM/Qwen-AgentWorld adaptations and the distinctions from predictive test selection and SWE-World.

3. Positive Reviewer Acknowledgments

The discussion has received positive acknowledgment from both responding reviewers

  • Reviewer RwJr thanked us for the detailed rebuttal and raised the rating following our additional comparisons and real-library evaluation clarifications.
  • Reviewer hVXy explicitly stated that the matched SFT comparisons and held-out-world evaluations addressed those concerns, appreciated the agent decision-process clarification, and maintained the positive rating.

Reviewer LUGw has not yet replied. However, their concerns about strong risk baselines, ordinary fine-tuning, real-library evidence, and model configurations substantially overlap with questions raised by RwJr and hVXy. These overlapping concerns were addressed with the same additional experimental evidence, which has already received positive acknowledgment from the other reviewers.

We sincerely appreciate the reviewers' and Area Chair's consideration of these substantive additions and clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.