- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents SWE-LOOP, the method that combines resource-aware execution infrastructure with a failure-aware GRPO recipe for software engineering agents, reporting improved reliability, throughput, and benchmark performance.
Critical strengths
This paper addresses a practical relevant problem and carefully separates infrastructure failures from policy outcomes. Configuration is documented particularly well, which helps reproducibility in principle.
Critical weaknesses
The contribution is a collection of known engineering practices without new insights. This paper is an useful engineering report rather than the work with a broad research contribution. The headline claim should be explained carefully. With the same self-verification prompt, the starting checkpoint scores 68.8/60.6 on SWE-bench Verified/Pro (Table 5), whereas the Reduced-LR baseline falls to 63.2/57.2 (Table 6). Therefore, the full method’s gains are +10.8/+10.6 over baseline RL, but +5.2/+7.2 over the starting checkpoint. There is only one training run per variant, one eval sample per task (T=0.6). Many reported differences (e.g. 1–3 points on Terminal-Bench) are within sampling noise for 500-task benchmarks. Group-completion cutoff discards the slowest ~14% of groups every step. This introduces a systematic selection bias toward fast/short trajectories that is neither analyzed nor discussed.
Questions for authors
Can you report >3 eval runs per task with confidence intervals for Tables 5–7? Does cutoff systematically change the difficulty or length distribution of training examples?
2 comments
Thank you for the additional experiments. The three seeds, repeated evaluation, and no-cutoff runs largely address my questions on noise and the cutoff, and I appreciate the gain over the starting checkpoint (+5.1/+7.2) being reported separately.
However, my main concern remains. Seq-Agg is an existing option, the submission penalty is simple reward shaping, self-verification is a prompt change, and the cutoff is a simple rule that stops waiting for the slowest rollouts. The infrastructure part is standard systems engineering, compared only against the authors' own baseline. I still see the paper as a collection of existing methods applied to SWE tasks, which makes it a solid engineering report rather than a research contribution. I therefore keep my rating.
Thank you for requesting a clearer training-gain comparison, repeated evaluation, and a direct cutoff-selection analysis. We have completed all three.
Contribution. SWE-LOOP separates operation queues, protects capacity for short tools and hidden verification, and shares spare capacity. Group filtering and stopping connect execution reliability to efficient policy updates.
Gain beyond the starting checkpoint. Under the same self-verification prompt, the repeated starting-checkpoint evaluation scores 68.83/60.66 on SWE-bench Verified/Pro. The full method scores 73.97/67.83, giving +5.13/+7.17 percentage points beyond the starting model. Gains over the paper's Reduced-LR reference are separately +10.68/+10.50 points. This distinguishes improvement beyond initialization from improvement over baseline RL.
Three training seeds and four draws per task. We evaluate final checkpoints from three independent training runs at temperature 0.6, top-p 0.95, and top-k 20. Terminal-Bench 2.0, SWE-bench Verified, and SWE-bench Pro public v1 contain 89, 500, and 731 tasks, respectively. Starting-checkpoint seed blocks are evaluation repeats of the same model.
Let s index the three seed blocks, i index tasks, k index the four draws, N be the benchmark's task count, and ysik∈{0,1} indicate success. Pass@1 measures mean single-attempt success; Pass@4 measures success in at least one of four attempts:
\qquad P@4_s=\frac{100}{N}\sum_{i=1}^{N}\mathbf1\!\left[\sum_{k=1}^{4}y_{sik}\ge1\right].$$ For a matched comparison, let $d_s$ be its seed-level score difference, $\bar d$ its mean, and $s_d$ its sample standard deviation. We report pointwise approximate paired Student-t 95% confidence intervals across the three seed blocks: $$\bar{d} \pm 4.30265 \frac{s_d}{\sqrt{3}}$$ These quantify between-run variation on the fixed benchmarks. Let A denote the full method at LR $10^{-6}$ and KL coefficient $10^{-3}$; B removes KL and C raises LR to $3\times10^{-6}$, with other settings unchanged. | Comparison, percentage points [95% CI] | Terminal-Bench | Verified | Pro | | --- | ---: | ---: | ---: | | A−starting checkpoint, Pass@1 | 7.68 [2.78, 12.58] | 5.13 [4.82, 5.45] | 7.17 [6.96, 7.38] | | A−B, Pass@1 | 3.18 [2.78, 3.59] | 2.68 [1.32, 4.05] | 1.66 [1.18, 2.15] | | A−C, Pass@1 | 3.37 [2.67, 4.07] | 4.47 [2.04, 6.90] | 3.49 [2.67, 4.31] | | A−starting checkpoint, Pass@4 | 9.74 [5.47, 14.00] | 4.67 [4.38, 4.95] | 3.78 [3.59, 3.98] | Thus the gains beyond initialization and the displayed controlled LR/KL comparisons have positive lower bounds. Four draws per task meet the requested evaluation count, while the training repeats quantify run-to-run variation. **Direct uncensored cutoff audit.** We train three no-cutoff runs and replay the cutoff decision on their complete event histories. A group becomes ready only after all eight original attempts terminate and at least seven are valid. With 64 groups and completion fraction 0.85, the target is 55 ready groups; collection otherwise ends at exhaustion. We compare eventually eligible groups that would be excluded by cutoff against those retained. For group $g$ with task $x_g$, let $L_g$ be its eventual mean response length over valid members. Let $\hat p_0(x_g)$ be the fixed starting-model solve-rate estimate from eight filtering rollouts; difficulty is $d_g=1-\hat p_0(x_g)$. Let D and R denote the excluded and retained groups within an update, respectively, and overbars denote their group means. Define $$\Delta_L=\overline L_D-\overline L_R, \qquad \Delta_d=\overline d_D-\overline d_R.$$ We calculate these contrasts within each update, average updates within each seed, then average the three seeds equally: | Excluded-minus-retained diagnostic | Mean [95% CI] | | --- | ---: | | Eventual response length | 326.58 [−242.98, 896.14] tokens | | Starting-model difficulty | 0.0058 [−0.0010, 0.0126] | The difficulty contrast means excluded groups have a **0.58-percentage-point lower starting-model solve-rate proxy**. These measurements show small mean shifts toward shorter/easier retained groups. The end-to-end comparison establishes the associated performance–time tradeoff: at 50 completed updates, cutoff-on minus no-cutoff Verified performance is **−0.050 [−0.423, 0.323] points**; the paired geometric training-time ratio is **0.5794 [0.5700, 0.5890]**, or **42.1% less training time**. The ratio interval uses paired log ratios and exponentiation. The lower performance bound is above the specified −1-point margin. The cutoff therefore has a favorable measured efficiency–performance tradeoff, supported by both uncensored selection diagnostics and repeated model evaluation.