- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes ASSERT, a framework for converting reference patches into executable training trajectories for software engineering agents. It combines dependency-constrained repair, replay verification, trajectory slicing, and rationale reconstruction with restricted information access. Experiments with two teachers on Python and Go tasks show improved trajectory yield and downstream student performance after SFT.
Critical strengths
-
The paper is trying to tackle a practical problem. Recovering useful training trajectories from tasks that ordinary rollouts fail to solve is well motivated. Clean replay and access isolation are sensible designs for improving supervision reliability.
-
The method is intuitive and (seems to be) sound. While the idea of "generate-then-modify" is very straightforward, I think the replay for ensuring the state transition is interesting. I like this idea.
-
The experimental results are clearly positive. The matched-task comparisons suggest that the improvements are not solely due to increased training data coverage. Component ablations and automated and human rationale audits provide complementary evidence for the usefulness of the generated data.
Critical weaknesses
-
The advantage under a comparable construction budget remains unclear. Allocating ASSERT’s additional repair and review budget to more RS sampling might allow the baseline to solve more training tasks, collect more successful trajectories, and achieve comparable student performance. Simple patch-conditioned sampling would additionally control for access to the reference solution. The independent-resampling ablation is useful, but it matches budget caps rather than actual computation and retains ASSERT’s downstream processing. A matched-cost comparison would better separate the method’s contribution from additional generation effort.
-
The contribution of the specific mechanisms could be better isolated. The ablations support slicing and rationale reconstruction as overall stages, but do not compare them with simpler alternatives, such as length-matched deduplication or an additional rationale-rewriting pass. These controls would help establish the additional value of dependency constraints, decision-distortion scoring, and access isolation beyond general data cleaning.
Questions for authors
-
How often does SFT truncation remove evidence required by retained supervised actions? I think that reporting the evidence-retention rate after truncation would clarify how well the construction-time guarantees carry over to the actual training inputs.
-
Could you provide more analysis of why ASSERT produces better training data? For example, what kinds of additional tasks does it cover compared with RS, what defects does DCTR repair, and what interactions does RCDS remove? This would help explain ASSERT’s gains and inform better sampling methods.
-
Could you provide a worked example showing how dependency edges are constructed, how a failure determines the editable region, and how a deletion candidate is accepted or rejected? This would help clarify the abstract operations in Algorithm 1.
-
Does lower UCR predict better downstream training performance? Figure 4 reports both metrics, but their relationship is not analyzed. I am curious about whether UCR is intended only as a grounding diagnostic or as a proxy for training-data quality.
-
The overall method is understandable, but the figures could communicate it more clearly. Could Figure 1 separate the main workflow from internal operations? Figure 4 would also be easier to interpret if training-data UCR and downstream Pass@1 were presented separately.
-
It is better to provide a small comparison when RS and ASSERT uses the same budget for data sampling. It just a recommendation which can benefit the paper, not mandatory.
-
Could you provide an artifact package for reproduction? The artifact is not available. The appendix said that the case examples are in
case_evidence/but there is no such file or directories. -
Could you use PDF or SVG in Figure 1? Figure 1 is PNG or JPG, which seems like AI-generated. Need more human inspections to revise this figure.
I am willing to raising my score if the response adequately addresses the major questions above.
6 comments
Response to Reviewer P4hq (2/4) — Weakness 2 and Question 1
Weakness 2: Ablations of RCDS and Rationale Reconstruction.
To further assess the value of RCDS, we replaced it with simple rule-based deduplication in the GPT-5.5/Python setting while keeping Rationale Reconstruction. We treat each tool call and its output as one step. Within each trajectory, we keep the first occurrence of a step and mark later occurrences as duplicates when the tool, arguments (excluding the textual summary), and output match. We also mark a later file view as a duplicate if the entire code excerpt it returns is contained in an earlier view of the same file. To match the number of retained training steps, we randomly remove as many duplicate steps as RCDS removed from the same trajectory, or all available duplicates if fewer are available. As the table shows, simple deduplication lowers Pass@1 by 3.00 percentage points relative to retaining the full trajectories and by 4.20 points relative to full ASSERT. The student also makes 0.79 more tool calls per evaluation task than it does without RCDS. By removing some training steps, it leaves fewer actions for Rationale Reconstruction. So the mean cost is $0.06 less than retaining the full trajectory. The result shows that simple deduplication can discard key evidence from a trajectory. For example, in getmoto__moto-6492, the same test reported 1 passed both before and after a new assertion about the Location header was added. Deleting the second run would remove the direct evidence that the new assertion passed. RCDS would retain this run if a later action relies on that confirmation. This example illustrates how RCDS approaches trajectory slicing more generally. It preserves the steps that later decisions depend on while removing as much redundant content as possible.
| Variant | Pass@1 (%) | Steps | Cost |
|---|---|---|---|
| Full ASSERT | 69.60 | 49.12 | 19.80 |
| Without RCDS slicing | 68.40 | 49.68 | 19.91 |
| Simple deduplication in place of RCDS | 65.40 | 50.47 | 19.85 |
| Rationale rewriting in place of Rationale Reconstruction | 66.20 | 50.32 | 19.87 |
To further assess the value of access isolation in Rationale Reconstruction, we kept RCDS and replaced reconstruction with a rewriting pass in which the teacher could read the full reference patch while revising the draft rationale for each retained action. This variant costs $0.07 more per retained training trajectory than full ASSERT, yet achieves 66.20% Pass@1, 3.40 percentage points below full ASSERT and 1.40 points below directly retaining the original rationale drafts. In this setting, an additional rewriting pass did not improve downstream performance. Access to the reference patch increases the risk that the teacher will use knowledge of the final fix to justify an earlier decision after the fact. Such rationales may teach the student to take similar actions before it has gathered sufficient evidence to support them.
Question 1: Evidence retention after SFT truncation.
We set the maximum SFT input length to 32,768 tokens to follow prior work on training software engineering agents from trajectories [1] and to keep training efficient by avoiding excessive training time on long trajectories. RCDS removes redundant steps to keep trajectories from becoming too long. However, when a trajectory exceeds the length limit, truncation can still remove evidence that a retained action depends on.
In the GPT-5.5/Python setting, ASSERT retained 2,081 training trajectories. We found that 82.36% of them fit within the limit, while the remaining 367 exceeded 32,768 tokens. Using the evidence certificates, we measured an evidence-retention rate of 94.45% after truncation. We have since improved the truncation procedure by using the certificates to actively protect cited evidence. As long as an action is retained for supervision after truncation, all evidence spans cited in its certificate are kept in the context preceding that action, so the student can still see the evidence supporting it.
Response to Reviewer P4hq (3/4) — Question 2
Question 2: Additional task coverage, DCTR repairs, and RCDS deletions.
(1) Additional task coverage. Our analysis identified three representative types of tasks in this additional coverage:
- Cross-library compatibility. These tasks require one library to handle data supplied by another. A challenge for RS is tracing a user-visible failure to the underlying interface that needs to change. In
bokeh__bokeh-13328, plotting a PolarsSeriesfailed, and the fix required Bokeh’s serializer to recognize the array protocol exposed by Polars. - CLI argument interactions. These tasks require a command to respect the scope defined by a combination of arguments. An RS rollout can verify the intended effect yet overlook changes to objects outside that scope. In
conan-io__conan-13622, Conan deleted the packages matching the filter, but also deleted nonmatching packages and the recipe itself. - Type narrowing. These tasks require a type checker to infer which types a variable can have in each branch of a condition. An RS rollout can fix the reported branch while overlooking valid subclass cases in the other. In
python__mypy-10284, mypy needed to narrowytoTwhentype(y) == Twas true while allowingyto have a subclass ofTwhen the condition was false.
(2) What defects does DCTR repair? DCTR targets five main types of trajectory defects.
- Missing evidence for an edit. When a plan jumps directly to a function, a reproduction or inspection step can establish why that function should change. In
iterative__dvc-5118, the traceback fromtest_get_dirpoints toout.save(), providing evidence for the subsequent removal of that call. - Incorrect operation order. An operation must follow the steps on which it depends. In
pandas-dev__pandas-50760, comparingNAwithpd.NAbefore importing pandas aspdraisedNameError. Importing pandas first allowed the same comparison to returnTrue. - Invalid command targets. When a command names a nonexistent test, the agent must identify a runnable target. In
bokeh__bokeh-13328, pytest could not find the specified test node, so the agent usedpytest --collect-onlyto list the available tests. - Missing runtime configuration. An incomplete configuration can stop a reproduction script before it reaches the behavior under investigation. In
getmoto__moto-5752, the script stopped atNoRegionError. Specifying the AWS region allowed it to reach the assertion that exposed the filter-order bug. - Missing post-edit verification. A plan that ends after an edit, or runs only unrelated tests, needs a check that exercises the original failure. In
iterative__dvc-5118, the agent reran the previously failingtest_get_dirafter removingout.save()and confirmed that it passed.
(3) What tool interactions does RCDS target for removal? RCDS considers four main types of interactions for removal.
- Unused exploration. A read-only operation may succeed without contributing information to later decisions. In
python__mypy-10284, an initial repository inspection printed the current Git commit hash, which subsequent edits and tests did not use. - Tool errors that provide no useful feedback. A failed call is a deletion candidate when it leaves the environment unchanged and does not guide recovery. The distinction matters in
pandas-dev__pandas-50760: a failed directory read prompted the agent to switch to a search, so that error provided a reason to retain the interaction. - Repeated source reads. Rereading code may add no information when an earlier edit result or diff already shows the relevant lines. In
bokeh__bokeh-12841, the agent reread two edited regions immediately after inspecting a diff that showed both changes. - Intermediate tests covered by a later run. A test subset is a deletion candidate when a later run covers the entire subset on unchanged code. In
bokeh__bokeh-13239, the agent ran eight tests and then the full 32-test module, which included all eight.
Response to Reviewer P4hq (1/4) — Weakness 1 and Question 6
We sincerely thank you for your constructive feedback and for suggesting a stronger RS baseline. Below, we present the additional experiments and address your questions about trajectory construction, evaluation, and presentation.
Weakness 1 / Question 6: Stronger RS and patch-conditioned baselines.
In fact, we tested up to six total RS attempts per training task with GPT-5.5 on Python during development. This increased the number of retained training tasks from 534 to 536 and the API cost per retained trajectory from USD 3.91 to USD 19.69. This coverage–cost trade-off motivated our use of one-attempt RS in the original main comparison. However, we believe it is important to clarify whether this trade-off holds across all four settings. Following your suggestion, we extended this comparison to the other three teacher–language settings. At six attempts, RS and ASSERT have comparable API costs per retained trajectory. On GPT-5.5/Python, raising the limit from six to seven increased this cost from USD 19.69 to USD 23.70 while student Pass@1 stayed at 63.80%. In the GPT-5.5/Python sweep, six attempts also came closest to ASSERT's cost per retained trajectory. We therefore set the maximum number of RS attempts per task to six. The results are as follows:
Task coverage and cost. Up to six attempts recover only 0–16 additional training tasks across the four settings (a 0–3.19% increase), while the API cost per retained trajectory rises to 5.02–5.80× its single-attempt level. ASSERT has a cost per retained trajectory comparable to that of six-attempt RS, but it retains 1.92–3.88 times as many trajectories across the four settings. On Go with GLM-5.2, retries recover no new tasks and leave downstream Pass@1 unchanged, while raising the cost per retained trajectory from USD 0.46 to USD 2.63 (5.72×).
Downstream performance and mechanism. Additional sampling improves student Pass@1 in three of the four settings (most notably +2.80 points for GPT-5.5/Python). Across all four settings, six-attempt RS still trails ASSERT by 5.20–7.24 percentage points. Even with the additional budget, RS still leaves many training tasks without a successful trajectory.
RS and ASSERT under the same total API budget. We tested whether RS could close the gap with ASSERT given the same total API budget for constructing training trajectories. With GPT-5.5 on Python, six-attempt RS retained 536 trajectories at a total cost of approximately USD 10.6k, compared with 2,081 trajectories at approximately USD 41.2k for ASSERT. Given this gap in both coverage and spending, we raised the RS limit to 24 attempts per training task to match ASSERT’s total trajectory construction cost. Under this budget, RS retained 542 trajectories at $76.01 each, and the resulting student achieved 64.20% Pass@1, still 5.40 percentage points below the student trained on ASSERT trajectories. Raising the RS limit from one to six attempts improved student Pass@1 by 2.80 points. Raising it from 6 to 24 yielded only six additional trajectories and a further 0.40-point improvement. Pan et al. [1] observed a similar decline in the coverage gained from additional samples. Increasing the number of sampled trajectories per task from 1 to 8 improved Pass@k by 17.2 percentage points, while increasing it from 8 to 16 added only 5.0 points.
We examined tasks where ASSERT produced a test-passing trajectory and RS exhausted its retries. Most cases involved incorrect or overly specific implementations. On
getmoto__moto-5752, all six RS attempts failed. Thedescribe_parametersAPI should return a parameter only if it matches both tag filters. Both parameters hadhello=world, but only one hadx=b. The original implementation returned one parameter whenx=bcame first and both whenhello=worldcame first. Using execution feedback and the reference patch, ASSERT’s Action-Plan Repair traced this order dependence to a prematurereturn Truein_match_filters. ASSERT removed the early return and added a regression test requiring both filter orders to return the same single parameter. The test passed, and ASSERT retained the trajectory for training. Our response to Question 2 describes other tasks ASSERT covered beyond RS.By the way, we use STaR as one of the baselines in our experiment, which is a simple patch-conditioned baseline. Given the reference patch, STaR generates a trajectory in a single response, synthesizing the intermediate actions and observations without executing them.