acceptodds
Under review as a conference paper at ICLR 2027

ASSERT: Synthesizing Replay-Verified Trajectories with Access Isolation for Software Engineering Agents

Abstract

Reference patches specify verified fixes, but do not provide the reliable interaction traces needed to train software engineering agents. Rejection sampling discards failed attempts, while patch-conditioned traces can contain unsupported reasoning and redundant steps. Therefore, we introduce ASSERT, a framework that converts reference patches into executable training trajectories across programming languages. Its central design separates privileged plan recovery from the execution evidence used to justify supervised decisions. Dependency-Constrained Trajectory Repair (DCTR) repairs failed plans within dependency regions identified by failure feedback, while Replay-Certified Decision-Sufficient Slicing (RCDS) removes redundant interactions subject to evidence, replay, and final-test constraints. A patch-blind runner supplies observations, and an isolated rationale writer explains fixed actions using preceding evidence. With GPT-5.5 and GLM-5.2 teachers, ASSERT retains up to 2.93 times as many trajectories as the largest baseline pool and improves student Pass@1 by up to 5.84 percentage points over the strongest baseline in each setting on SWE-bench Verified and Multi-SWE-bench. To compare supervision quality at matched task coverage, we also train a student for each method using one method-specific trajectory per task in each setting's four-method training intersection. ASSERT maintains a Pass@1 advantage of up to 6.07 percentage points over the strongest baseline and has the lowest unsupported-claim rate in every setting, supporting improvements in both supervision yield and training utility.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Software engineering agents, Trajectory synthesis, Supervised fine-tuning, Access isolation, Replay verification.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_P4hqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ASSERT, a framework for converting reference patches into executable training trajectories for software engineering agents. It combines dependency-constrained repair, replay verification, trajectory slicing, and rationale reconstruction with restricted information access. Experiments with two teachers on Python and Go tasks show improved trajectory yield and downstream student performance after SFT.

Critical strengths
  1. The paper is trying to tackle a practical problem. Recovering useful training trajectories from tasks that ordinary rollouts fail to solve is well motivated. Clean replay and access isolation are sensible designs for improving supervision reliability.

  2. The method is intuitive and (seems to be) sound. While the idea of "generate-then-modify" is very straightforward, I think the replay for ensuring the state transition is interesting. I like this idea.

  3. The experimental results are clearly positive. The matched-task comparisons suggest that the improvements are not solely due to increased training data coverage. Component ablations and automated and human rationale audits provide complementary evidence for the usefulness of the generated data.

Critical weaknesses
  1. The advantage under a comparable construction budget remains unclear. Allocating ASSERT’s additional repair and review budget to more RS sampling might allow the baseline to solve more training tasks, collect more successful trajectories, and achieve comparable student performance. Simple patch-conditioned sampling would additionally control for access to the reference solution. The independent-resampling ablation is useful, but it matches budget caps rather than actual computation and retains ASSERT’s downstream processing. A matched-cost comparison would better separate the method’s contribution from additional generation effort.

  2. The contribution of the specific mechanisms could be better isolated. The ablations support slicing and rationale reconstruction as overall stages, but do not compare them with simpler alternatives, such as length-matched deduplication or an additional rationale-rewriting pass. These controls would help establish the additional value of dependency constraints, decision-distortion scoring, and access isolation beyond general data cleaning.

Questions for authors
  1. How often does SFT truncation remove evidence required by retained supervised actions? I think that reporting the evidence-retention rate after truncation would clarify how well the construction-time guarantees carry over to the actual training inputs.

  2. Could you provide more analysis of why ASSERT produces better training data? For example, what kinds of additional tasks does it cover compared with RS, what defects does DCTR repair, and what interactions does RCDS remove? This would help explain ASSERT’s gains and inform better sampling methods.

  3. Could you provide a worked example showing how dependency edges are constructed, how a failure determines the editable region, and how a deletion candidate is accepted or rejected? This would help clarify the abstract operations in Algorithm 1.

  4. Does lower UCR predict better downstream training performance? Figure 4 reports both metrics, but their relationship is not analyzed. I am curious about whether UCR is intended only as a grounding diagnostic or as a proxy for training-data quality.

  5. The overall method is understandable, but the figures could communicate it more clearly. Could Figure 1 separate the main workflow from internal operations? Figure 4 would also be easier to interpret if training-data UCR and downstream Pass@1 were presented separately.

  6. It is better to provide a small comparison when RS and ASSERT uses the same budget for data sampling. It just a recommendation which can benefit the paper, not mandatory.

  7. Could you provide an artifact package for reproduction? The artifact is not available. The appendix said that the case examples are in case_evidence/ but there is no such file or directories.

  8. Could you use PDF or SVG in Figure 1? Figure 1 is PNG or JPG, which seems like AI-generated. Need more human inspections to revise this figure.

I am willing to raising my score if the response adequately addresses the major questions above.

6 comments
AuthorsOpenReview ↗

Response to Reviewer P4hq (1/4) — Weakness 1 and Question 6

We sincerely thank you for your constructive feedback and for suggesting a stronger RS baseline. Below, we present the additional experiments and address your questions about trajectory construction, evaluation, and presentation.

Weakness 1 / Question 6: Stronger RS and patch-conditioned baselines.

In fact, we tested up to six total RS attempts per training task with GPT-5.5 on Python during development. This increased the number of retained training tasks from 534 to 536 and the API cost per retained trajectory from USD 3.91 to USD 19.69. This coverage–cost trade-off motivated our use of one-attempt RS in the original main comparison. However, we believe it is important to clarify whether this trade-off holds across all four settings. Following your suggestion, we extended this comparison to the other three teacher–language settings. At six attempts, RS and ASSERT have comparable API costs per retained trajectory. On GPT-5.5/Python, raising the limit from six to seven increased this cost from USD 19.69 to USD 23.70 while student Pass@1 stayed at 63.80%. In the GPT-5.5/Python sweep, six attempts also came closest to ASSERT's cost per retained trajectory. We therefore set the maximum number of RS attempts per task to six. The results are as follows:

BenchmarkTeacherTraining tasks retained ↑Pass@1 (%) ↑Cost / traj. (USD) ↓
SWE-bench Verified (Python)GLM-5.2517 (+16)63.60 (+0.40)2.55 (+2.11)
GPT-5.5536 (+2)63.80 (+2.80)19.69 (+15.78)
Multi-SWE-bench (Go)GLM-5.2264 (+0)23.60 (+0.00)2.63 (+2.17)
GPT-5.5258 (+6)26.17 (+0.24)21.58 (+17.28)

Task coverage and cost. Up to six attempts recover only 0–16 additional training tasks across the four settings (a 0–3.19% increase), while the API cost per retained trajectory rises to 5.02–5.80× its single-attempt level. ASSERT has a cost per retained trajectory comparable to that of six-attempt RS, but it retains 1.92–3.88 times as many trajectories across the four settings. On Go with GLM-5.2, retries recover no new tasks and leave downstream Pass@1 unchanged, while raising the cost per retained trajectory from USD 0.46 to USD 2.63 (5.72×).

Downstream performance and mechanism. Additional sampling improves student Pass@1 in three of the four settings (most notably +2.80 points for GPT-5.5/Python). Across all four settings, six-attempt RS still trails ASSERT by 5.20–7.24 percentage points. Even with the additional budget, RS still leaves many training tasks without a successful trajectory.

RS and ASSERT under the same total API budget. We tested whether RS could close the gap with ASSERT given the same total API budget for constructing training trajectories. With GPT-5.5 on Python, six-attempt RS retained 536 trajectories at a total cost of approximately USD 10.6k, compared with 2,081 trajectories at approximately USD 41.2k for ASSERT. Given this gap in both coverage and spending, we raised the RS limit to 24 attempts per training task to match ASSERT’s total trajectory construction cost. Under this budget, RS retained 542 trajectories at $76.01 each, and the resulting student achieved 64.20% Pass@1, still 5.40 percentage points below the student trained on ASSERT trajectories. Raising the RS limit from one to six attempts improved student Pass@1 by 2.80 points. Raising it from 6 to 24 yielded only six additional trajectories and a further 0.40-point improvement. Pan et al. [1] observed a similar decline in the coverage gained from additional samples. Increasing the number of sampled trajectories per task from 1 to 8 improved Pass@k by 17.2 percentage points, while increasing it from 8 to 16 added only 5.0 points.

We examined tasks where ASSERT produced a test-passing trajectory and RS exhausted its retries. Most cases involved incorrect or overly specific implementations. On getmoto__moto-5752, all six RS attempts failed. The describe_parameters API should return a parameter only if it matches both tag filters. Both parameters had hello=world, but only one had x=b. The original implementation returned one parameter when x=b came first and both when hello=world came first. Using execution feedback and the reference patch, ASSERT’s Action-Plan Repair traced this order dependence to a premature return True in _match_filters. ASSERT removed the early return and added a regression test requiring both filter orders to return the same single parameter. The test passed, and ASSERT retained the trajectory for training. Our response to Question 2 describes other tasks ASSERT covered beyond RS.

By the way, we use STaR as one of the baselines in our experiment, which is a simple patch-conditioned baseline. Given the reference patch, STaR generates a trajectory in a single response, synthesizing the intermediate actions and observations without executing them.

AuthorsOpenReview ↗

Response to Reviewer P4hq (2/4) — Weakness 2 and Question 1

Weakness 2: Ablations of RCDS and Rationale Reconstruction.

To further assess the value of RCDS, we replaced it with simple rule-based deduplication in the GPT-5.5/Python setting while keeping Rationale Reconstruction. We treat each tool call and its output as one step. Within each trajectory, we keep the first occurrence of a step and mark later occurrences as duplicates when the tool, arguments (excluding the textual summary), and output match. We also mark a later file view as a duplicate if the entire code excerpt it returns is contained in an earlier view of the same file. To match the number of retained training steps, we randomly remove as many duplicate steps as RCDS removed from the same trajectory, or all available duplicates if fewer are available. As the table shows, simple deduplication lowers Pass@1 by 3.00 percentage points relative to retaining the full trajectories and by 4.20 points relative to full ASSERT. The student also makes 0.79 more tool calls per evaluation task than it does without RCDS. By removing some training steps, it leaves fewer actions for Rationale Reconstruction. So the mean cost is $0.06 less than retaining the full trajectory. The result shows that simple deduplication can discard key evidence from a trajectory. For example, in getmoto__moto-6492, the same test reported 1 passed both before and after a new assertion about the Location header was added. Deleting the second run would remove the direct evidence that the new assertion passed. RCDS would retain this run if a later action relies on that confirmation. This example illustrates how RCDS approaches trajectory slicing more generally. It preserves the steps that later decisions depend on while removing as much redundant content as possible.

VariantPass@1 (%)StepsCost
Full ASSERT69.6049.1219.80
Without RCDS slicing68.4049.6819.91
Simple deduplication in place of RCDS65.4050.4719.85
Rationale rewriting in place of Rationale Reconstruction66.2050.3219.87

To further assess the value of access isolation in Rationale Reconstruction, we kept RCDS and replaced reconstruction with a rewriting pass in which the teacher could read the full reference patch while revising the draft rationale for each retained action. This variant costs $0.07 more per retained training trajectory than full ASSERT, yet achieves 66.20% Pass@1, 3.40 percentage points below full ASSERT and 1.40 points below directly retaining the original rationale drafts. In this setting, an additional rewriting pass did not improve downstream performance. Access to the reference patch increases the risk that the teacher will use knowledge of the final fix to justify an earlier decision after the fact. Such rationales may teach the student to take similar actions before it has gathered sufficient evidence to support them.

Question 1: Evidence retention after SFT truncation.

We set the maximum SFT input length to 32,768 tokens to follow prior work on training software engineering agents from trajectories [1] and to keep training efficient by avoiding excessive training time on long trajectories. RCDS removes redundant steps to keep trajectories from becoming too long. However, when a trajectory exceeds the length limit, truncation can still remove evidence that a retained action depends on.

In the GPT-5.5/Python setting, ASSERT retained 2,081 training trajectories. We found that 82.36% of them fit within the limit, while the remaining 367 exceeded 32,768 tokens. Using the evidence certificates, we measured an evidence-retention rate of 94.45% after truncation. We have since improved the truncation procedure by using the certificates to actively protect cited evidence. As long as an action is retained for supervision after truncation, all evidence spans cited in its certificate are kept in the context preceding that action, so the student can still see the evidence supporting it.

AuthorsOpenReview ↗

Response to Reviewer P4hq (3/4) — Question 2

Question 2: Additional task coverage, DCTR repairs, and RCDS deletions.

(1) Additional task coverage. Our analysis identified three representative types of tasks in this additional coverage:

  1. Cross-library compatibility. These tasks require one library to handle data supplied by another. A challenge for RS is tracing a user-visible failure to the underlying interface that needs to change. In bokeh__bokeh-13328, plotting a Polars Series failed, and the fix required Bokeh’s serializer to recognize the array protocol exposed by Polars.
  2. CLI argument interactions. These tasks require a command to respect the scope defined by a combination of arguments. An RS rollout can verify the intended effect yet overlook changes to objects outside that scope. In conan-io__conan-13622, Conan deleted the packages matching the filter, but also deleted nonmatching packages and the recipe itself.
  3. Type narrowing. These tasks require a type checker to infer which types a variable can have in each branch of a condition. An RS rollout can fix the reported branch while overlooking valid subclass cases in the other. In python__mypy-10284, mypy needed to narrow y to T when type(y) == T was true while allowing y to have a subclass of T when the condition was false.

(2) What defects does DCTR repair? DCTR targets five main types of trajectory defects.

  1. Missing evidence for an edit. When a plan jumps directly to a function, a reproduction or inspection step can establish why that function should change. In iterative__dvc-5118, the traceback from test_get_dir points to out.save(), providing evidence for the subsequent removal of that call.
  2. Incorrect operation order. An operation must follow the steps on which it depends. In pandas-dev__pandas-50760, comparing NA with pd.NA before importing pandas as pd raised NameError. Importing pandas first allowed the same comparison to return True.
  3. Invalid command targets. When a command names a nonexistent test, the agent must identify a runnable target. In bokeh__bokeh-13328, pytest could not find the specified test node, so the agent used pytest --collect-only to list the available tests.
  4. Missing runtime configuration. An incomplete configuration can stop a reproduction script before it reaches the behavior under investigation. In getmoto__moto-5752, the script stopped at NoRegionError. Specifying the AWS region allowed it to reach the assertion that exposed the filter-order bug.
  5. Missing post-edit verification. A plan that ends after an edit, or runs only unrelated tests, needs a check that exercises the original failure. In iterative__dvc-5118, the agent reran the previously failing test_get_dir after removing out.save() and confirmed that it passed.

(3) What tool interactions does RCDS target for removal? RCDS considers four main types of interactions for removal.

  1. Unused exploration. A read-only operation may succeed without contributing information to later decisions. In python__mypy-10284, an initial repository inspection printed the current Git commit hash, which subsequent edits and tests did not use.
  2. Tool errors that provide no useful feedback. A failed call is a deletion candidate when it leaves the environment unchanged and does not guide recovery. The distinction matters in pandas-dev__pandas-50760: a failed directory read prompted the agent to switch to a search, so that error provided a reason to retain the interaction.
  3. Repeated source reads. Rereading code may add no information when an earlier edit result or diff already shows the relevant lines. In bokeh__bokeh-12841, the agent reread two edited regions immediately after inspecting a diff that showed both changes.
  4. Intermediate tests covered by a later run. A test subset is a deletion candidate when a later run covers the entire subset on unchanged code. In bokeh__bokeh-13239, the agent ran eight tests and then the full 32-test module, which included all eight.
Reviewer_nhqHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ASSERT, a framework for generating training trajectories for software engineering agents from reference patches. It combines dependency-constrained trajectory repair (DCTR), replay-certified trajectory slicing (RCDS), and access-isolated rationale reconstruction. Experiments on SWE-bench Verified and Multi-SWE-bench demonstrate improved trajectory yield and downstream agent performance.

Critical strengths

S1. The paper addresses a practical challenge in constructing reliable training data for software engineering agents. S2. It demonstrates consistent Pass@1 improvements across two teacher models and two programming languages. S3. It provides matched-task experiments and ablation studies supporting the effectiveness of the proposed components.

Critical weaknesses

W1. The overall approach builds upon existing reference-patch-guided trajectory synthesis methods, particularly P2T, which already incorporates process graphs, blinded execution, and groundedness checks. ASSERT introduces more elaborate repair, slicing, and rationale reconstruction mechanisms, but the extent of its conceptual advancement over prior approaches remains unclear. W2. Although the DCTR ablations demonstrate benefits over unrestricted repair and independent resampling, it remains unclear whether the proposed framework substantially outperforms simpler extensions of existing approaches. In particular, comparisons with P2T augmented with iterative repair or replay-verified trajectory pruning would help establish the necessity of the proposed mechanisms. W3. Rationale reconstruction conditions on the target action, potentially producing plausible post-hoc explanations. Though access isolation and evidence certificates improve grounding, they do not guarantee that the reconstructed reasoning is independently supported by preceding observations. W4. Although ASSERT improves cost per retained trajectory relative to P2T, the methods are not evaluated under matched total computational budgets. This limits conclusions about the computational efficiency of the additional repair and verification mechanisms.

Questions for authors

Address the weakness of W1-W4. The authors addressed my major concerns.

4 comments
AuthorsOpenReview ↗

Response to Reviewer nhqH (1/2) — Weaknesses 1 and 2

We sincerely thank you for your thoughtful review. We address these points below.

Weakness 1. How ASSERT differs from P2T.

Although ASSERT and P2T share the goal of converting reference patches into training trajectories, they construct those trajectories in fundamentally different ways. P2T works backward from the patch to identify facts and solution milestones, then uses the resulting process graph to guide and select a patch-blind teacher's next steps, one segment at a time [1]. ASSERT uses the patch to propose a complete candidate action plan, then executes that plan forward from the original repository state. This difference matters in two ways.

(1) P2T selects segments, while ASSERT refines complete trajectories. The shortest effective segment that P2T selects in the current window can still be a worse choice for the trajectory as a whole [1]. Skipping a diagnostic may save a step now but force the agent to investigate the same issue again later. On difficult tasks in particular, P2T gives its solver clues about how to reach the fix and favors the shortest effective segment that advances the current milestones. ASSERT takes the fix in the reference patch as its target from the outset and constructs and refines an action plan until the resulting patch passes the tests. Only after the trajectory is complete does RCDS determine which steps can be removed, because it can then identify which earlier diagnostics supplied evidence for later edits. Our trajectory counts and downstream results are consistent with this advantage. For example, in the GPT-5.5/Python setting, ASSERT retained 4.05 times as many training trajectories as P2T, and its student’s Pass@1 was 3.40 percentage points higher.

(2) P2T and ASSERT restrict access to the reference patch at different stages. P2T uses an LLM critic which can see the patches to judge whether a proposed graph node reveals the fix prematurely and checks whether earlier observations support curator-modified steps. ASSERT restricts access at the point where training explanations are written. The planner can consult the reference patch, but a patch-blind runner produces the recorded outputs of replayable tools. The separate rationale writer explains each fixed action using prior evidence, without access to the reference patch or that action's outcome. On matched GPT-5.5/Python tasks, the unsupported claim rate in the explanation preceding the first source-code edit was 4.58% for ASSERT versus 6.88% for P2T (Figure 4).

Weakness 2. ASSERT vs. simpler extensions of P2T.

To illustrate the necessity of the proposed mechanisms of ASSERT, we further compare ASSERT with augmented P2T.

For replay-verified trajectory pruning, P2T’s original workflow already uses a selection mechanism to choose the shortest path at each stage. Further pruning could therefore lead to excessive information loss. P2T lets the solver try several approaches and favors shorter candidate segments among those that make sufficient progress. A code-reading step retained in this way may explain why the next edit targets that particular location. Further pruning could remove this evidence. Our UCR experiments also show that P2T does indeed lose more information (GPT-5.5/Python tasks: 6.88% for P2T vs. 4.58% for ASSERT, Figure 4).

To extend the existing P2T workflow with iterative repair, we added iterative repair to P2T’s solver. In the original P2T, a candidate trajectory ends if its final patch fails the tests [1]. We therefore evaluated P2T + iterative repair. When a submitted patch fails the tests, the solver receives the failure report, revises the patch, and reruns the tests until the patch passes or the maximum number of iterations is reached.

With GPT-5.5 on SWE-Gym (Python), we set the maximum for iterative repair to 4、8 and 12. The results are as follows:

MethodTrajectories retainedStudent Pass@1 (%)
ASSERT2,08169.60
P2T51466.20
P2T + iterative repair 4 times60366.60
P2T + iterative repair 8 times62867.20
P2T + iterative repair 12 times63567.20

The results show that increasing the number of repair iterations yields more training trajectories and improves student Pass@1, but the gains are limited. Even after 12 iterations. Student Pass@1 reached 67.20% at 8 rounds and remained there at 12 rounds, 2.40 percentage points below ASSERT.

AuthorsOpenReview ↗

Response to Reviewer nhqH (2/2) — Weaknesses 3 and 4 and References

Weakness 3. Risk of post-hoc rationalization when the target action is known.

Your concern relates to the correctness of chain-of-thought reasoning. When the teacher knows the target action, it may write an explanation that sounds plausible but goes beyond what the preceding observations support. This raises a broader challenge in chain-of-thought research: how to assess whether a model’s explanation is faithful. Turpin et al. [2] found that models can justify an incorrect answer without mentioning the cue that biased them toward it. Parcalabescu and Frank [3] examined several tests of explanation faithfulness and found that agreement between a model’s explanation and its answer is insufficient to establish how the model reached that answer.

In ASSERT, we guide the teacher to explain each verified action using observations available before that action. To reduce the risk of the teacher constructing a post-hoc justification for a chosen action, we require the trajectory to establish the facts on which the action relies. If a plan jumps directly to editing a function, Action-Plan Repair can insert a diagnostic or code inspection step before the edit to show why the function warrants attention. The evidence certificate identifies the specific passage in the issue or an earlier observation that supports this judgment, and RCDS preserves that passage when slicing the trajectory. During Rationale Reconstruction, the teacher can therefore see both what the agent did and which earlier observations made that action reasonable at that point. This reduces the need for the teacher to infer a justification from the chosen action alone.

To check whether the preceding evidence supports the reconstructed rationales, we conducted a blinded human audit (Appendix F.4, Table 6). Two annotators independently examined 40 rationales per method, each immediately preceding the first production-code edit. The target action remained hidden until the annotators had judged whether each assessable factual claim was supported by the issue description or earlier observations. We used the human unsupported claim rate, which measures the proportion of assessable factual claims in each rationale that annotators could not substantiate from the issue description or earlier observations. We averaged these proportions within each setting and then equally across the four settings. Human unsupported claim rate was 4.66% for ASSERT, 6.48% for RS, and 8.82% for P2T. Even when the annotators could not use the chosen action as a clue, they found a lower proportion of unsupported factual claims in ASSERT’s rationales.

Weakness 4. P2T and ASSERT under the same total API budget.

We tested whether P2T could close the gap with ASSERT given the same total API budget for constructing training trajectories. In the GPT-5.5 experiment on SWE-Gym (Python) reported in Table 1, ASSERT’s total API cost was 2.81 times that of P2T. Given this gap in spending, we raised the P2T limit to 3 attempts per training task to match ASSERT’s total trajectory construction cost. Under this budget, P2T retained 582 trajectories at USD 70.79 each, an increase of 68 trajectories over the original P2T baseline. The resulting student achieved 66.40% Pass@1, still 3.20 percentage points below the student trained on ASSERT trajectories. These results show that P2T’s performance changed little, while its cost increased substantially.

References

[1] Ma et al. From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents. 2026. NeurIPS, 2026.

[2] Miles Turpin et al. Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS, 2023.

[3] Letitia Parcalabescu and Anette Frank. On Measuring Faithfulness or Self-consistency of Natural Language Explanations. ACL, 2024.

We sincerely thank you again for your suggestions! We will clarify the comparisons with P2T and better explain the evidence supporting our design choices, making ASSERT more convincing.

Thanks for detailed responses. Given that my primary concerns have been addressed, I am raising my score from 2 to 3.

Thank you for your kind words! We’re glad our response addressed your concerns!

Reviewer_v9FcOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses how to turn reference patches into supervised trajectories for software engineering agents without the reasoning leaking the answer or containing redundant steps. ASSERT separates privileged plan recovery, which may use the patch, from the evidence used to justify each decision: a patch-blind runner executes the plan to collect real observations, Dependency-Constrained Trajectory Repair fixes failed plans locally around the failure, Replay-Certified Decision-Sufficient Slicing removes steps not needed under replay and test constraints, and an isolated writer explains each fixed action using only preceding evidence. The resulting trajectories are used to fine-tune student models and compared against rejection sampling and patch-conditioned baselines.

Critical strengths
  • The comparison at matched task coverage, using one trajectory per commonly solved task, still shows gains, so the improvement is not only due to a larger data pool.
  • Gains are consistent across two teachers and two languages, and the ablation supports targeted repair over plain resampling.
Critical weaknesses
  • Access isolation is not measured at the action level: exported actions such as search queries and opened files still come from patch-aware components, which the paper acknowledges, and the description of who selects the evidence passed to the rationale writer is not fully consistent. Measuring how often early actions target patch locations without support from the issue or prior observations, compared with rejection sampling, would strengthen the central claim.
  • Results come from a single training seed, several gaps are small, and the pipeline is considerably more expensive than rejection sampling. Mean and variance over a few seeds and an equal-budget rejection-sampling baseline would make the differences easier to interpret."
Questions for authors
  • Access isolation is not measured at the action level: exported actions such as search queries and opened files still come from patch-aware components, which the paper acknowledges, and the description of who selects the evidence passed to the rationale writer is not fully consistent. Measuring how often early actions target patch locations without support from the issue or prior observations, compared with rejection sampling, would strengthen the central claim.
  • Results come from a single training seed, several gaps are small, and the pipeline is considerably more expensive than rejection sampling. Mean and variance over a few seeds and an equal-budget rejection-sampling baseline would make the differences easier to interpret."
3 comments
AuthorsOpenReview ↗

Response to Reviewer v9Fc (1/3) — Question 1 and Question 2(1)

We sincerely thank you for raising these questions. They have been very helpful to us. We address both questions below:

Question 1. Grounding early actions in prior evidence and selecting evidence for the rationale writer

(1) Grounding early actions in prior evidence. The concern you raise was one of our motivations for designing DCTR: an early action may rely on an unsupported premise, including the specific case you raise, where an early search or file open directly targets a location modified by the reference patch without any prior evidence pointing to it. During plan review, ASSERT checks whether the issue description or earlier tool outputs give the agent a reason to inspect that location. If the review identifies a missing premise, DCTR locally repairs the relevant steps before the trajectory can be accepted.

To assess whether the agent’s stated reasons for targeting a particular code location early in the trajectory are supported by the issue description and earlier tool outputs, we report the Unsupported Claim Rate (UCR) for the rationale immediately preceding the first source-code edit (Figure 4). For each trajectory, we evaluate the rationale immediately preceding the first production-source edit. We compute the fraction of assessable factual claims in that rationale that lack support from the issue description or preceding tool outputs, then average across trajectories with assessable claims. On the 292 shared GPT-5.5/Python tasks, UCR is 4.58% for ASSERT and 6.30% for RS. Thus, students learning from ASSERT trajectories are less likely to encounter unsupported claims in these pre-edit explanations.

(2) Who selects the evidence passed to the rationale writer? The ambiguity may stem from our use of the term reviewer in both DCTR (Action-Plan Repair) and RCDS (Execution Grounding) (Section 3.3.2 and Appendix E). Both operate before the rationale writer. During plan repair, Action-Plan Repair and Execution Grounding work iteratively. Execution Grounding returns replay observations and reports any replay or test failures, which Action-Plan Repair uses to revise the candidate plan. Once the trajectory can be replayed in full and the resulting patch passes the official tests, RCDS removes redundant steps while preserving the evidence needed for the retained actions. The resulting trajectory and its evidence certificates are then passed to the rationale writer. The DCTR reviewer can consult the reference patch when assessing whether a candidate action has sufficient prior support. The RCDS reviewer has no access to the reference patch and selects the supporting passages from the issue description or earlier tool outputs.

We are grateful to you for pointing this out and will revise the wording to make it clearer.

Question 2. Evaluation variability and the RS sampling budget

(1) Evaluation variability. Following the same fixed-seed protocol used in prior language model fine-tuning studies [1, 2], we used the same SFT seed (42) for all methods. After training, we evaluated each student checkpoint five times on the full test set, with one greedy rollout per task in each run. We report the mean Pass@1 across the five runs to reduce the influence of any single evaluation run on the final score. Averaging resolution rates across repeated runs is an established practice in software engineering agent evaluation [3, 4, 5].

AuthorsOpenReview ↗

Response to Reviewer v9Fc (2/3) — Question 2(2)

(2) RS at a comparable cost per retained trajectory. In fact, we tested up to six total RS attempts per training task with GPT-5.5 on Python during development. This increased the number of retained training tasks from 534 to 536 and the API cost per retained trajectory from USD 3.91 to USD 19.69. This coverage–cost trade-off motivated our use of one-attempt RS in the original main comparison. However, we believe it is important to clarify whether this trade-off holds across all four settings. Following your suggestion, we extended this comparison to the other three teacher–language settings. At six attempts, RS and ASSERT have comparable API costs per retained trajectory. On GPT-5.5/Python, raising the limit from six to seven increased this cost from USD 19.69 to USD 23.70 while student Pass@1 stayed at 63.80%. In the GPT-5.5/Python sweep, six attempts also came closest to ASSERT's cost per retained trajectory. We therefore set the maximum number of RS attempts per task to six. The results are as follows:

BenchmarkTeacherTraining tasks retained ↑Pass@1 (%) ↑Cost / traj. (USD) ↓
SWE-bench Verified (Python)GLM-5.2517 (+16)63.60 (+0.40)2.55 (+2.11)
GPT-5.5536 (+2)63.80 (+2.80)19.69 (+15.78)
Multi-SWE-bench (Go)GLM-5.2264 (+0)23.60 (+0.00)2.63 (+2.17)
GPT-5.5258 (+6)26.17 (+0.24)21.58 (+17.28)

Task coverage and cost. Up to six attempts recover only 0–16 additional training tasks across the four settings (a 0–3.19% increase), while the API cost per retained trajectory rises to 5.02–5.80× its single-attempt level. ASSERT has a cost per retained trajectory comparable to that of six-attempt RS, but it retains 1.92–3.88 times as many trajectories across the four settings. On Go with GLM-5.2, retries recover no new tasks and leave downstream Pass@1 unchanged, while raising the cost per retained trajectory from USD 0.46 to USD 2.63 (5.72×).

Downstream performance and mechanism. Additional sampling improves student Pass@1 in three of the four settings (most notably +2.80 points for GPT-5.5/Python). Across all four settings, six-attempt RS still trails ASSERT by 5.20–7.24 percentage points. Even with the additional budget, RS still leaves many training tasks without a successful trajectory.

RS and ASSERT under the same total API budget. We tested whether RS could close the gap with ASSERT given the same total API budget for constructing training trajectories. With GPT-5.5 on Python, six-attempt RS retained 536 trajectories at a total cost of approximately USD 10.6k, compared with 2,081 trajectories at approximately USD 41.2k for ASSERT. Given this gap in both coverage and spending, we raised the RS limit to 24 attempts per training task to match ASSERT’s total trajectory construction cost. Under this budget, RS retained 542 trajectories at $76.01 each, and the resulting student achieved 64.20% Pass@1, still 5.40 percentage points below the student trained on ASSERT trajectories. Raising the RS limit from one to six attempts improved student Pass@1 by 2.80 points. Raising it from 6 to 24 yielded only six additional trajectories and a further 0.40-point improvement. Pan et al. [6] observed a similar decline in the coverage gained from additional samples. Increasing the number of sampled trajectories per task from 1 to 8 improved Pass@k by 17.2 percentage points, while increasing it from 8 to 16 added only 5.0 points.

We examined tasks where ASSERT produced a test-passing trajectory and RS exhausted its retries. Most cases involved incorrect or overly specific implementations. On getmoto__moto-5752, all six RS attempts failed. The describe_parameters API should return a parameter only if it matches both tag filters. Both parameters had hello=world, but only one had x=b. The original implementation returned one parameter when x=b came first and both when hello=world came first. Using execution feedback and the reference patch, ASSERT’s Action-Plan Repair traced this order dependence to a premature return True in _match_filters. ASSERT removed the early return and added a regression test requiring both filter orders to return the same single parameter. The test passed, and ASSERT retained the trajectory for training.

We sincerely thank you again for your suggestions! They have been very helpful in improving the readability of the paper, and we will continue to refine it.

AuthorsOpenReview ↗

Response to Reviewer v9Fc (3/3) — References

References

[1] Jingwen Sun et al. Introducing Graph Context into Language Models through Parameter-Efficient Fine-Tuning for Lexical Relation Mining. ACL, 2025.

[2] Mathew Huerta-Enochian and Seung Yong Ko. Instruction Fine-Tuning: Does Prompt Loss Matter?. EMNLP, 2024.

[3] John Yang et al. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS, 2024.

[4] Yuntong Zhang et al. AutoCodeRover: Autonomous Program Improvement. ISSTA, 2024.

[5] Linghao Zhang et al. SWE-bench Goes Live!. NeurIPS, 2025.

[6] Jiayi Pan et al. Training Software Engineering Agents and Verifiers with SWE-Gym. ICML, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.