- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to collect coding-agent trajectories that are useful for downstream training. SWE-Trail compiles previous attempts on the same issue into frozen detector rules, local comparison objectives, and sampling schedules. When a new rollout reaches a detected state, the coding agent proposes actions from its ordinary context; the collector executes two candidates from the same checkpoint and selects one route using local execution evidence. History and collector messages are excluded from the agent context and exported training trajectory. With 1,500 trajectories per condition, H5 improves the 124B student's mean resolution over independent sampling by 2.33 percentage points on SWE-bench Multilingual and 1.60 points on SWE-bench Pro. Improvements also appear with Qwen3-14B-Base.
Critical strengths
-
The paper evaluates the training value of collected trajectories, which is a useful and consequential question beyond improving the collector's own issue-resolution rate. The experiments use the same collection issues and trajectory count, hold the remaining training setup fixed, and report three independent training runs. H5 improves over SeedBank on both benchmarks and with both student models. This makes the empirical result more convincing than a collection-success comparison alone.
-
The separation between collector-side history and agent-side generation is clearly implemented. Candidate actions come from the ordinary agent context, alternatives execute from the same checkpoint, and only one continuous route is exported. The randomized detection and retention controls are useful, and the appendix documents the prompts, information boundaries, and export checks in detail. These are substantive strengths of the proposed collection design.
Critical weaknesses
-
The distinct contribution over existing adaptive replay and branching methods is not sufficiently established empirically. SWE-Replay [1], already cited in the paper, uses archived trajectories to select intermediate states, restore the environment, and generate new continuations. Its step selection does not require inserting a distilled history-advice message into the actor's context. DARS [2] also explores alternatives at selected key decisions using previous execution feedback. SWE-Trail differs through frozen history-derived rules, a bounded two-route execution comparison, and its focus on student training utility. I recognize these differences, but the experiments mainly compare against independent sampling, curation adaptations, and an agent-visible experience baseline. H0 is a useful internal control, but does not establish the benefit over an existing replay-based collector. A comparison of the training value of such trajectories would materially strengthen the case for the new design.
-
The benefit of history-derived detection, beyond changing the amount and diversity of local search, is not fully isolated. Appendix B.5 gives H0 a fixed proposal temperature of 0.7, whereas H5 uses higher temperatures for mixed and never-solved histories. Table 22 also shows 968 executed comparisons for H5 versus 560 for H0. Thus, H5-H0 changes the location and frequency of comparisons, as well as candidate sampling. The random-detection ablation partially addresses this concern, but Appendix G.1 matches the allowed per-trajectory budget rather than reporting matched realized intervention counts. For the 124B student, the H5-H0 gains are 2.22/0.87 points, with task-bootstrap intervals of [-0.2, 4.6]/[-0.8, 2.5] (Table 16). The Qwen results provide additional positive evidence, so I do not view the method as ineffective. However, the current evidence does not cleanly identify how much comes from the history-derived allocation rules rather than more frequent branching or a different sampling schedule.
These two gaps in establishing the distinctive contribution are why I currently lean toward weak rejection, despite the useful downstream results.
[1] Ding and Zhang. SWE-Replay: Efficient Test-Time Scaling for Software Engineering Agents. arXiv:2601.22129, first posted January 29, 2026. https://arxiv.org/abs/2601.22129 [2] Aggarwal et al. DARS: Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree Traversal. arXiv:2503.14269, first posted March 18, 2025. https://arxiv.org/abs/2503.14269
Questions for authors
-
Provide a controlled comparison with a replay-based collector, particularly SWE-Replay, using the same collection model, issue pool, retained trajectory count, and student-training protocol. The relevant endpoint is downstream student resolution, rather than the test-time performance of the original method. This would help determine the practical value of the frozen detector and local execution-selection design.
-
What are the realized candidate-sampling and executed-comparison counts for random detection? Can a history-free control using the same candidate-temperature schedule and a matched intervention rate retain the H5 advantage? I would also appreciate uncertainty estimates for the randomized-control contrasts. These clarifications would directly affect my assessment of the history-derived detector.
6 comments
W2 and Q2: Separating comparison location from comparison frequency and temperature.
Separating comparison location from comparison frequency and temperature is indeed important. We address the reviewer's three questions in turn: (a) the realized counts of random detection, (b) a history-free control with H5's temperatures and a matched intervention rate, and (c) uncertainty estimates for the randomized controls.
(a) Realized counts of the randomized controls. To check how often random detection actually compares, Table R2 reports how often each collector samples candidate actions and how often it executes a comparison. Random detection does not use a detector. In every trajectory that has an eligible state, it picks one at random and samples candidates there, whereas H5 samples candidates only where its detector fires.
Table R2
| Collection | Location | Temperatures | Trajectories with candidate sampling | Executed comparisons |
|---|---|---|---|---|
| SWE-Trail-H0 | history-free detector | 0.7 | 664 | 560 |
| SWE-Trail-H5 | history-compiled detector | H5 schedule | 1147 | 968 |
| H5, random detection | random eligible state | H5 schedule | 1477 | 1248 |
Random detection samples candidates in 1477 trajectories, more than H5's 1147, and executes more comparisons (1248 vs. 968) under the same temperature schedule and budget. Yet its students are below H5's by 1.44 / 1.28 points on the 124B student and 2.00 / 1.78 on Qwen3-14B (Table R3).
(b) A history-free control with H5's temperature schedule and intervention rate. To test whether H5's advantage holds without its comparison placement or without its temperature schedule, we use two controls. Random detection is a randomized-location control with H5's temperature schedule. Its rate is higher than H5's rather than equal (1248 vs. 968 comparisons). As a converse test, we re-run H5 with its frozen detectors but with H0's fixed temperature of 0.7 for all candidates.
Table R3
| Collection | Location | Temperatures | Executed comparisons | Ling-124B Multilingual | Ling-124B SWE-Pro | Qwen3-14B Multilingual | Qwen3-14B SWE-Pro |
|---|---|---|---|---|---|---|---|
| SWE-Trail-H0 | history-free detector | 0.7 | 560 | 63.44 | 41.95 | 12.78 | 11.90 |
| H5, fixed temperature | history-compiled detector | 0.7 | 1004 | 65.33 | 42.41 | 14.00 | 13.22 |
| H5, random detection | random eligible state | H5 schedule | 1248 | 64.22 | 41.54 | 12.44 | 11.72 |
| SWE-Trail-H5 | history-compiled detector | H5 schedule | 968 | 65.67 | 42.82 | 14.44 | 13.50 |
With all candidate temperatures fixed at 0.7, H5 still outperforms H0 on both students and benchmarks, by 1.89 / 0.46 points on the 124B student and 1.22 / 1.32 points on Qwen3-14B. Random detection executes more comparisons than H5 (1248 vs. 968) but produces weaker students on all four metrics. These results support the value of history-guided comparison placement.
W2 and Q2 (continued)
(c) Uncertainty for the randomized-control contrasts. To quantify how reliable these contrasts are, we treat independently trained models as the unit and pool within-condition variance over all existing runs of each student, assuming a common variance across conditions. For the 124B student, the standard deviation within a condition is 1.09 points on Multilingual and 0.81 on SWE-Pro, so a difference between two 3-run means has a standard error of 0.89 and 0.66 points.
Table R4
| Contrast | Student | Benchmark | Δ | t | One-sided p |
|---|---|---|---|---|---|
| H5 − random detection | Ling-124B | Multilingual | +1.44 | 1.63 | 0.06 |
| H5 − random detection | Ling-124B | SWE-Pro | +1.28 | 1.94 | 0.03 |
| H5 − random detection | Qwen3-14B | Multilingual | +2.00 | 10.33 | < 0.001 |
| H5 − random detection | Qwen3-14B | SWE-Pro | +1.78 | 14.65 | < 0.001 |
| H5 − random retention | Ling-124B | Multilingual | +0.89 | 1.00 | 0.16 |
| H5 − random retention | Ling-124B | SWE-Pro | +1.19 | 1.80 | 0.04 |
The random-detection contrast is supported on both students. It is clear on Qwen3-14B, holds on the 124B student's SWE-Pro (p = 0.03), and is borderline on its Multilingual (p = 0.06). The random-retention contrast is weaker. It holds on SWE-Pro (p = 0.04) but not on Multilingual (p = 0.16), so we will describe it as suggestive. For H5 vs. H0, all three H5 runs on Qwen3-14B score higher than all three H0 runs on both benchmarks (Table 18). The stratified run-level permutation test across the two students gives p = 0.0025 on Multilingual and p = 0.0075 on SWE-Pro for H5 versus H0.
In summary, random detection uses H5's temperatures and compares more often than H5, yet stays close to H0. H5 with H0's temperature keeps most of its advantage. These results support the history-derived allocation rules, rather than more frequent branching or a different sampling schedule, as the main source of the gain.
We will add the following to the paper:
- A controlled SWE-Replay collector evaluated by downstream student resolution under the reviewer's protocol (Table R1), with DARS and SWE-Replay discussed in the related work.
- Realized intervention counts for H0, H5, and random detection (Table R2).
- A temperature-matched H5 collection (Table R3).
- Run-level uncertainty for the randomized controls (Table R4).
- The cross-student combined test for H5 vs. H0.
We hope these results address the reviewer's questions, and we would be glad to provide further analysis.
We thank the reviewer for the detailed reading of our appendices. We address each point with new experiments that follow the protocols the reviewer specified:
W1 and Q1: Controlled comparison with SWE-Replay.
As the reviewer requested, we compare SWE-Trail with SWE-Replay under the same collection model, issue pool, retained trajectory count, and student-training protocol, and evaluate by downstream student resolution. SWE-Replay is a test-time method that replays archived attempts from intermediate states chosen by state rarity and reasoning intensity, and then selects a final patch. We use it as a collector with three changes only. Its archive starts from the same five SeedBank attempts that H5 reads. All five new attempts per issue are kept for training instead of selecting one patch (1500 trajectories, as for H5). Each replayed attempt is exported as the archived prefix followed by the new continuation. Everything else follows the original method, and the collection model (GLM-5.1) and scaffold (OpenHands) match H5.
Table R1
Table R1b
H5 is the highest on both students and both benchmarks. SWE-Replay improves over SeedBank on Qwen3-14B (+1.78 Multilingual, +1.23 SWE-Pro) and on 124B Multilingual (+1.11), but stays below H5 on every column, by 1.22 / 1.55 on the 124B student and 1.33 / 1.19 on Qwen3-14B. SWE-Replay is a strong collector. It resolves slightly more trajectories than H5 (353 vs. 345, Table R1b), although over fewer issues (95 vs. 98). Its weaker students therefore do not come from fewer successes. We see two reasons for the gap. First, H5 spends its extra effort on the steps that decide success or failure, so its trajectories teach the student better choices at exactly those steps. SWE-Replay spends its extra effort on rarely visited states, which need not matter for the outcome, so its extra data teaches less. Random detection supports this: with H5's comparison placed at random, it is also below H5 (Table R3). Second, 28% of the steps in SWE-Replay's trajectories are copied from earlier attempts, so the student sees less new behavior. As the reviewer notes, collection success and training value are different endpoints. Under the reviewer's protocol, SWE-Trail's trajectories therefore train stronger students than those of a replay-based collector.