- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies a practical problem in software engineering agents: a change may pass all tests in the repository being edited while still breaking downstream services or libraries. The authors propose a Software World Model (SWM) to predict which downstream components will fail after a proposed change. The proposed pipeline has three parts. First, the system executes candidate changes from restored software states and collects their downstream consequences. It gives more attention to cases where the dependency graph disagrees with observed execution. Second, a language model is fine-tuned on these execution records to predict the affected components, failure type, and possible failure channel. Finally, the predictions are converted into per-consumer failure probabilities and used to rank changes, decide whether to migrate a consumer in advance, or determine whether an additional check is worthwhile. The results are generally positive. SWM improves over typed reachability on the synthetic benchmarks and also reports gains on the real-library experiments. The paper also makes a useful distinction between prediction accuracy and the value of the decisions induced by a predictor.
Critical strengths
The problem is well motivated. Current coding agents usually validate the repository they modify, but this does not say much about the behavior of downstream consumers. The paper focuses on a failure mode that is easy to overlook and has clear practical importance. The same-state branching procedure is one of the stronger parts of the work. By executing several candidate changes from the same restored state, the authors make the comparison between alternatives more meaningful. The targeted exploration of cases where static reachability is wrong is also sensible, especially when the execution budget is limited.
Critical weaknesses
My main concern is that the “world model” framing is somewhat broader than what is actually evaluated. The main agent experiment does not show a coding agent freely proposing and implementing arbitrary patches over a long software-engineering trajectory. The interface is closer to identifying affected consumers and selecting preventive migrations for a given change. This is a useful task, but the paper should be more precise about what kind of agent improvement is being demonstrated. The method also relies on executed consequences from resettable environments. This is reasonable for collecting training data, but it may be difficult to reproduce in a large production system where downstream services are not all accessible and failed changes cannot always be safely rolled back. The synthetic propagation rules are also designed by the authors, and the real-library evaluation covers only a limited range of projects. I am therefore not fully convinced that the current evidence supports broad generalization.
Questions for authors
Could the authors clarify the exact interface used in the main agent experiment? Is the model generating code changes, or is it given a change and asked to identify and migrate affected consumers? The current description sometimes sounds broader than the actual evaluation.
How much of the improvement comes from the proposed consequence-learning objective rather than ordinary task adaptation? A matched SFT baseline with the same data, inputs, and output format would be helpful.
Why is a language model with GRPO preferable to a simpler structured risk model in this setting? The paper should compare them under the same decision interface, rather than only comparing their prediction scores under different protocols.
How does SWM perform on completely unseen libraries, new versions, or dependency patterns that are not represented in the training worlds?
5 comments
Response to Reviewer hVXy (Part 2 of 3)
[Q2] How much of the improvement comes from the proposed consequence-learning objective rather than ordinary task adaptation? A matched SFT baseline with the same data, inputs, and output format would be helpful.
Response: We conducted matched SFT ablations using the same 3B backbone, training data, input representations, optimization budget, and two-stage agent evaluation interface, varying only the supervision objective. All checkpoints act directly as agents under the same output protocol, without an external SWM advisor.
\hline \text{Method} & \text{Training Objective} & \text{Return (40)} & \text{Return (360)} \\\\ \hline \text{Frozen agent} & \text{None} & 0.721 & 0.668 \\\\ \text{Observation-only SFT} & \text{Observations} & 0.558 & 0.436 \\\\ \text{Blast-set SFT} & \text{Downstream consequences} & 0.754 & 0.725 \\\\ \text{Structured SFT} & \text{Four-field consequences} & 0.732 & 0.670 \\\\ \hline \text{Full SWM} & \text{SFT + GRPO} & \color{green}{0.773} & \color{green}{0.755} \\\\ \hline \end{array}$$ Observation-only SFT underperforms the frozen agent, whereas blast-set SFT improves return from **0.721 to 0.754** on 40 tasks and from **0.668 to 0.725** on 360 tasks. The **full SWM recipe** further improves return to **0.773 and 0.755**, respectively. The improvement also extends to 7B, where consequence-only SFT achieves 0.790 and full SWM reaches 0.813, compared with 0.695 for the frozen agent. We have incorporated these controls and their experimental protocols into the revised appendix. --- > **[Q3]** Why is a language model with GRPO preferable to a simpler structured risk model in this setting? The paper should compare them under the same decision interface, rather than only comparing their prediction scores under different protocols. **Response:** We clarify **(1) why consequence-aware GRPO offers a richer learning objective than structured risk scoring**, and **(2) how the two approaches compare under the same agent decision interface**, including a concrete failure case. **1. Mechanism: consequence learning versus structured risk scoring.** The structured risk model estimates independent per-consumer break probabilities from fixed graph and action features. It is effective when these features capture the relevant dependencies, but cannot directly exploit semantic evidence absent from its feature representation. In contrast, SWM interprets code-change context and consumer-specific usage evidence, learning structured downstream consequences beyond extracted dependency paths. GRPO aligns training with execution-verified outcomes. This directly optimizes consequence fidelity at the set level, encouraging accurate identification of affected consumers while penalizing false alarms. **2. Quantitative comparison under the same decision interface.** We evaluate both models through a fixed two-stage agent interface: the same actor first identifies candidate consumers, then receives per-consumer risk guidance and selects preventive migrations. On 40 identical held-out tasks, only the guidance source changes: $$\begin{array}{l|c} \hline \text{Guidance Source} & \text{Agent Return} \\\\ \hline \text{No guidance} & 0.771 \\\\ \text{Relation/action logistic} & 0.779 \\\\ \text{SWM (3B)} & 0.796 \\\\ \hline \end{array}$$ Our subsequent scaling and internalized-agent evaluations provide complementary evidence: - **Consequence prediction:** SWM achieves synthetic blast-set F1 of **0.761 (3B)** and **0.784 (7B)** versus **0.671** for logistic, and real-library F1 of **0.718 (7B)** versus **0.672**. - **Agent decisions:** The full SWM-trained agent achieves return of **0.773 (3B)** and **0.813 (7B)**, compared with **0.794** for the explicit supervised-risk plan on the same tasks. The 3B difference is not statistically significant, while the 7B result is a higher point estimate. Unlike the explicit risk plan, the SWM-trained agent must identify candidate consumers itself. - **Training recipe:** Consequence-only SFT achieves return of **0.754 (3B)** and **0.790 (7B)**; the complete SFT + GRPO recipe raises these point estimates to **0.773** and **0.813**, respectively.Response to Reviewer hVXy (Part 3 of 3)
3. Case study: avoiding unnecessary migrations.
Consider a synthetic software world with five downstream services, where an upstream change causes no actual failures. The logistic risk model nevertheless assigns billing a break probability of 0.490, triggering an unnecessary migration. GRPO trains SWM across other software worlds to distinguish actual breakage from apparent dependency risks, rewarding correct no-break predictions over false alarms.
\boxed{\text{Training: Executed Changes Across Different Worlds}} \\\\ \downarrow \\\\ \boxed{\text{Experience: Changes Without Downstream Breakage}} \\\\ \downarrow \\\\ \text{GRPO: Compare Sampled Consequence Predictions} \\\\ \begin{array}{cc} \text{Correct: Empty Blast Set} & \text{Incorrect: False Break Prediction} \\\\ \color{green}{F_1=1} & \color{red}{F_1=0} \end{array} \\\\ \downarrow \\\\ \color{blue}{\boxed{\text{Learned Knowledge: Risk Does Not Imply Breakage}}} \end{array}$$ By learning from execution-verified no-break cases across training worlds, GRPO reinforces the distinction between potential dependency risks and actual downstream failures. This learned distinction can help SWM avoid unnecessary interventions in unseen software environments. $$\begin{array}{c} \boxed{\text{Held-Out World: Upstream Change Causes No Breakage}} \\\\ \downarrow \\\\ \begin{array}{c|c} \text{Logistic-Guided Agent} & \text{SWM-Guided Agent} \\\\ \hline \text{Fixed Graph/Action Features} & \color{blue}{\boxed{\text{Learned Knowledge: Risk Does Not Imply Breakage}}} \\\\ \downarrow & \downarrow \\\\ \text{Billing Risk: }0.490 & \text{Recognize No-Break Pattern} \\\\ \downarrow & \downarrow \\\\ \boxed{\text{Migrate Billing}} & \boxed{\text{No Migration}} \\\\ \downarrow & \downarrow \\\\ \text{Unnecessary Migration Cost} & \text{Avoided Migration Cost} \\\\ \end{array} \end{array}$$ --- > **[Q4]** How does SWM perform on completely unseen libraries, new versions, or dependency patterns that are not represented in the training worlds? **Response:** We evaluate generalization through **world-level held-out splits** in both real-library and synthetic environments, while distinguishing unseen software worlds from entirely unseen failure mechanisms. **1. Generalization to unseen real libraries.** We train on three upstream-library worlds (jsonschema, Jinja2, and Werkzeug) and evaluate on Attrs, Requests, and Starlette. - The 0.5B SWM achieves 0.644 blast-set F1 versus 0.671 for typed reachability, showing that smaller SWM models do not consistently outperform structural baselines. - Among 457 breaking records, **137 are missed by extracted-graph reachability**. Incorporating consumer-specific usage context improves SWM's hidden-channel F1 from 0.142 to 0.198, compared with 0.137 for reachability. **2. Generalization to unseen synthetic worlds.** We extend the main evaluation from **40 tasks across 7 held-out worlds** to **480 migration tasks across 60 held-out worlds**. These worlds are generated using the same deterministic software-world generator, but with a separate evaluation seed offset, ensuring **no overlap with training worlds**. Each world contributes six migration tasks. The frozen 3B model achieves return 0.651, while SWM (3B) checkpoint achieves **0.724–0.729** across three seeds, with positive paired improvement. These experiments establish transfer to unseen world instances under the same generator, not generalization to arbitrary new dependency distributions. We have clarified these distinctions and the corresponding limitations in the revised manuscript.
Response to Reviewer hVXy (Part 1 of 3)
We thank the reviewer for recognizing the practical importance of downstream breakage prediction, the value of same-state counterfactual execution, and our targeted exploration of failures missed by static reachability. We also appreciate the constructive comments.
Response:
Response: Our Explore–Learn–Act pipeline distinguishes two roles: the policy model proposes code changes, while SWM learns to predict their downstream consequences from execution-verified experience. The learned knowledge can improve the acting agent through two integration mechanisms:
\boxed{\text{Policy Model: Propose Code Changes}} \\\\ \downarrow \\\\ \boxed{\text{Explore: Execute Changes and Observe Consequences}} \\\\ \downarrow \\\\ \boxed{\text{Learn: Train SWM}} \\\\ \downarrow \\\\ \begin{array}{c|c} \text{Internalized SWM} & \text{External SWM Guidance} \\\\ \hline \boxed{ \begin{array}{c} \text{SWM-Trained Acting Agent} \\\\ \downarrow \\\\ \text{Identify Affected Consumers} \\\\ \downarrow \\\\ \text{Select Preventive Migrations} \\\\ \downarrow \\\\ \text{Agent Return} \end{array} } & \boxed{ \begin{array}{c} \text{SWM Predicts Consequences} \\\\ \downarrow \\\\ \text{Provide Predictions to Policy} \\\\ \downarrow \\\\ \text{Policy Selects Actions} \end{array} } \end{array} \end{array}$$ Our main experiment demonstrates **how SWM with consequence learning improves coding-agent decisions**, while code-change generation is handled by the policy model itself. We have clarified this distinction in Section 4.1 and the corresponding experimental protocol.