acceptodds
Under review as a conference paper at ICLR 2027

Reliable and Efficient Agent Loops for Reinforcement Learning on Software Engineering Tasks

Abstract

Software engineering (SWE) agents solve repository-level tasks through tool use, execution, verification, and iterative repair, making SWE an important and objectively verifiable domain for reinforcement learning (RL). Existing benchmarks, agent harnesses, and distributed RL frameworks establish key ingredients, but do not provide a unified infrastructure and training recipe for long-horizon agent loops under resource contention and execution failures. We present SWE-LOOP, which combines a failure-aware SWE agent-loop infrastructure—with bounded admission, class-aware scheduling, image affinity, and asynchronous preparation—with a GRPO recipe using submission-aware rewards, self-verification, group-completion cutoff, masked updates, and sequence-level aggregation. The infrastructure reduces Step-0 wall-clock time by 28.0%, while the full recipe improves SWE-bench Verified and SWE-Bench Pro by 10.8 and 10.6 points over the stable reduced-learning-rate baseline.

Primary area:
reinforcement learning
Keywords:
Software Engineering Agents, Agentic Reinforcement Learning, SWE Agent Loops, Failure-Aware RL Infrastructure, GRPO, Docker-Based Execution, Trajectory Filtering and Masking, Self-Verification, SWE-bench, Terminal-Bench

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wNPWOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents SWE-LOOP, the method that combines resource-aware execution infrastructure with a failure-aware GRPO recipe for software engineering agents, reporting improved reliability, throughput, and benchmark performance.

Critical strengths

This paper addresses a practical relevant problem and carefully separates infrastructure failures from policy outcomes. Configuration is documented particularly well, which helps reproducibility in principle.

Critical weaknesses

The contribution is a collection of known engineering practices without new insights. This paper is an useful engineering report rather than the work with a broad research contribution. The headline claim should be explained carefully. With the same self-verification prompt, the starting checkpoint scores 68.8/60.6 on SWE-bench Verified/Pro (Table 5), whereas the Reduced-LR baseline falls to 63.2/57.2 (Table 6). Therefore, the full method’s gains are +10.8/+10.6 over baseline RL, but +5.2/+7.2 over the starting checkpoint. There is only one training run per variant, one eval sample per task (T=0.6). Many reported differences (e.g. 1–3 points on Terminal-Bench) are within sampling noise for 500-task benchmarks. Group-completion cutoff discards the slowest ~14% of groups every step. This introduces a systematic selection bias toward fast/short trajectories that is neither analyzed nor discussed.

Questions for authors

Can you report >3 eval runs per task with confidence intervals for Tables 5–7? Does cutoff systematically change the difficulty or length distribution of training examples?

2 comments

Thank you for requesting a clearer training-gain comparison, repeated evaluation, and a direct cutoff-selection analysis. We have completed all three.

Contribution. SWE-LOOP separates operation queues, protects capacity for short tools and hidden verification, and shares spare capacity. Group filtering and stopping connect execution reliability to efficient policy updates.

Gain beyond the starting checkpoint. Under the same self-verification prompt, the repeated starting-checkpoint evaluation scores 68.83/60.66 on SWE-bench Verified/Pro. The full method scores 73.97/67.83, giving +5.13/+7.17 percentage points beyond the starting model. Gains over the paper's Reduced-LR reference are separately +10.68/+10.50 points. This distinguishes improvement beyond initialization from improvement over baseline RL.

Three training seeds and four draws per task. We evaluate final checkpoints from three independent training runs at temperature 0.6, top-p 0.95, and top-k 20. Terminal-Bench 2.0, SWE-bench Verified, and SWE-bench Pro public v1 contain 89, 500, and 731 tasks, respectively. Starting-checkpoint seed blocks are evaluation repeats of the same model.

Let ss index the three seed blocks, ii index tasks, kk index the four draws, NN be the benchmark's task count, and ysik∈{0,1}y_{sik}\in\{0,1\} indicate success. Pass@1 measures mean single-attempt success; Pass@4 measures success in at least one of four attempts:

\qquad P@4_s=\frac{100}{N}\sum_{i=1}^{N}\mathbf1\!\left[\sum_{k=1}^{4}y_{sik}\ge1\right].$$ For a matched comparison, let $d_s$ be its seed-level score difference, $\bar d$ its mean, and $s_d$ its sample standard deviation. We report pointwise approximate paired Student-t 95% confidence intervals across the three seed blocks: $$\bar{d} \pm 4.30265 \frac{s_d}{\sqrt{3}}$$ These quantify between-run variation on the fixed benchmarks. Let A denote the full method at LR $10^{-6}$ and KL coefficient $10^{-3}$; B removes KL and C raises LR to $3\times10^{-6}$, with other settings unchanged. | Comparison, percentage points [95% CI] | Terminal-Bench | Verified | Pro | | --- | ---: | ---: | ---: | | A−starting checkpoint, Pass@1 | 7.68 [2.78, 12.58] | 5.13 [4.82, 5.45] | 7.17 [6.96, 7.38] | | A−B, Pass@1 | 3.18 [2.78, 3.59] | 2.68 [1.32, 4.05] | 1.66 [1.18, 2.15] | | A−C, Pass@1 | 3.37 [2.67, 4.07] | 4.47 [2.04, 6.90] | 3.49 [2.67, 4.31] | | A−starting checkpoint, Pass@4 | 9.74 [5.47, 14.00] | 4.67 [4.38, 4.95] | 3.78 [3.59, 3.98] | Thus the gains beyond initialization and the displayed controlled LR/KL comparisons have positive lower bounds. Four draws per task meet the requested evaluation count, while the training repeats quantify run-to-run variation. **Direct uncensored cutoff audit.** We train three no-cutoff runs and replay the cutoff decision on their complete event histories. A group becomes ready only after all eight original attempts terminate and at least seven are valid. With 64 groups and completion fraction 0.85, the target is 55 ready groups; collection otherwise ends at exhaustion. We compare eventually eligible groups that would be excluded by cutoff against those retained. For group $g$ with task $x_g$, let $L_g$ be its eventual mean response length over valid members. Let $\hat p_0(x_g)$ be the fixed starting-model solve-rate estimate from eight filtering rollouts; difficulty is $d_g=1-\hat p_0(x_g)$. Let D and R denote the excluded and retained groups within an update, respectively, and overbars denote their group means. Define $$\Delta_L=\overline L_D-\overline L_R, \qquad \Delta_d=\overline d_D-\overline d_R.$$ We calculate these contrasts within each update, average updates within each seed, then average the three seeds equally: | Excluded-minus-retained diagnostic | Mean [95% CI] | | --- | ---: | | Eventual response length | 326.58 [−242.98, 896.14] tokens | | Starting-model difficulty | 0.0058 [−0.0010, 0.0126] | The difficulty contrast means excluded groups have a **0.58-percentage-point lower starting-model solve-rate proxy**. These measurements show small mean shifts toward shorter/easier retained groups. The end-to-end comparison establishes the associated performance–time tradeoff: at 50 completed updates, cutoff-on minus no-cutoff Verified performance is **−0.050 [−0.423, 0.323] points**; the paired geometric training-time ratio is **0.5794 [0.5700, 0.5890]**, or **42.1% less training time**. The ratio interval uses paired log ratios and exponentiation. The lower performance bound is above the specified −1-point margin. The cutoff therefore has a favorable measured efficiency–performance tradeoff, supported by both uncensored selection diagnostics and repeated model evaluation.

Thank you for the additional experiments. The three seeds, repeated evaluation, and no-cutoff runs largely address my questions on noise and the cutoff, and I appreciate the gain over the starting checkpoint (+5.1/+7.2) being reported separately.

However, my main concern remains. Seq-Agg is an existing option, the submission penalty is simple reward shaping, self-verification is a prompt change, and the cutoff is a simple rule that stops waiting for the slowest rollouts. The infrastructure part is standard systems engineering, compared only against the authors' own baseline. I still see the paper as a collection of existing methods applied to SWE tasks, which makes it a solid engineering report rather than a research contribution. I therefore keep my rating.

Reviewer_72gEOpenReview ↗
Rating
2(+1)
Confidence
4(+1)
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents SWE-LOOP, a framework for improving the reliability and efficiency of reinforcement learning for software engineering agents. It combines infrastructure-level optimizations, including resource scheduling, image affinity, and asynchronous prefetching, with training strategies such as submission-aware rewards, self-verification, group-completion cutoff, and sequence-level loss aggregation. Experiments on Terminal-Bench 2.0, SWE-bench Verified, and SWE-Bench Pro report improvements in execution efficiency and downstream task performance.

Critical strengths
  1. The paper addresses practical challenges in long-horizon software engineering agent training, particularly execution reliability, resource contention, and infrastructure failures.

  2. The paper provides detailed implementation configurations and reports useful empirical observations regarding execution efficiency and several training strategies.

Critical weaknesses
  1. Limited novelty and unclear core contribution. Most of the proposed components, including resource scheduling, image prefetching, reward shaping, and sequence-level aggregation, are established techniques or closely related to existing work. The paper does not clearly identify a sufficiently novel methodological contribution beyond their integration.

  2. Lack of a unified research narrative. The paper combines multiple infrastructure optimizations and training strategies, but their conceptual connections are not sufficiently developed. The introduction does not clearly motivate a central research question or explain why these components should be considered a coherent methodological contribution.

  3. Insufficient positioning against existing work. The discussion of related research is relatively limited, particularly regarding existing agentic RL frameworks and harness-native training approaches. The paper would benefit from a more explicit comparison with closely related methods to clarify its distinctive contributions.

  4. Insufficient experimental validation and ablation studies. Although several configurations are evaluated, the experiments do not adequately isolate the contributions of all the proposed components, especially group-level masking and completion-based cutoff. Stronger baselines and more systematic ablations are needed to support the effectiveness of the overall framework.

  5. Potential confounding factors in the infrastructure experiments. The cumulative infrastructure experiments modify timeout configurations alongside scheduling mechanisms, making it difficult to attribute the reported efficiency and reliability improvements to specific proposed techniques.

  6. Limited evidence for the generality and significance of the reported improvements. The experiments are conducted with a single model backbone and a relatively limited training setting, without repeated runs or statistical uncertainty estimates. Moreover, part of the reported improvement is measured against an RL baseline that underperforms the untrained model with self-verification, which makes the magnitude of the actual training benefit less clear.

Questions for authors
  1. What is the central methodological novelty of SWE-LOOP compared with existing agentic RL frameworks, such as LEGO-RL and SkyRL-Agent?

  2. Could the authors clarify the main research question and explain the underlying connection between the infrastructure optimizations and the proposed training strategies?

  3. Why does the reduced-learning-rate RL baseline underperform the untrained model with self-verification on the SWE benchmarks, and how would the full method compare against a more thoroughly optimized baseline?

  4. Could the authors provide additional ablations to isolate the contributions of group-level masking, group-completion cutoff, and the other training components?

  5. Could the authors evaluate the infrastructure optimizations under identical timeout configurations to better isolate the effects of the proposed scheduling mechanisms?

  6. How well do the reported improvements generalize across different model backbones, resource configurations, and training seeds?

2 comments
AuthorsOpenReview ↗

Operation-Level Scheduling: Core Contribution, Controlled Ablations, and Generality

Thank you for the questions. Our research question is: How can agent RL collect reliable learning trajectories efficiently when heterogeneous operations compete for constrained resources?

Core contribution. We schedule operations inside long agent executions. Within a shared resource pool, let cc index operation classes, ncn_c count active operations, ucu_c be class caps, rcr_c protected reservations, and CC total capacity:

∑cnc≤C,0≤nc≤uc,∑crc≤C.\sum_c n_c\le C,\qquad 0\le n_c\le u_c,\qquad \sum_c r_c\le C.

Classes below reservation receive admission priority as slots free; unused capacity is shared. This protects progress while allowing expensive operations to use spare capacity.

FrameworkRelevant design emphasis
SkyRL-AgentPipelines initialization, agent execution, and reward stages
LEGO-RLNative-harness integration, faithful rollout capture, and scalable execution
SWE-LOOPOperation-level contention control through protected capacity plus sharing, coupled to failure-aware group construction and completion stopping

Policy updates are synchronous; execution is asynchronous. Filtering defines usable group data, cutoff limits collection time, and the training recipe controls policy change and trajectory weighting. These connect execution efficiency to learning.

Controlled comparison. We fix Qwen3.5-35B-A3B, eight H200 GPUs, harness, sampling, effective deadlines, cache policy, and timing boundary: 64 prompts, eight trajectories each, three matched frozen-model repetitions. Timing includes preparation through cleanup. Seven services receive 20 or 10 CPUs each; GPU allocation and memory remain fixed.

Execution setting140 CPUs: valid trajectories/hour70 CPUs: valid trajectories/hour
SkyRL-Agent Async Pipeline342.69202.79
SkyRL-Agent + our scheduler451.85313.17
LEGO-RL407.23—

Paired geometric throughput ratios are 1.318 [1.108, 1.569] and 1.545 [1.481, 1.612] versus SkyRL-Agent at 140/70 CPUs, and 1.109 [1.055, 1.166] versus LEGO-RL. Brackets are approximate pointwise 95% paired Student-t intervals over three run blocks; ratios use the log scale. At 140 CPUs, mean per-run started-operation light/reward p95 waits fall from 144.93/242.73 to 60.83/96.17 seconds, and invalid-trajectory rate from 3.711% to 1.432%.

Why LR reduction alone is insufficient. Token mean gives longer trajectories more weight; sequence mean averages each trajectory first. We penalize valid non-submission by −0.1-0.1. Reduced-LR uses token mean without this penalty. The submitted factorial fixes self-verification, LR 10−610^{-6}, and KL coefficient 10−310^{-3}:

Submitted configurationVerified (%)Pro (%)
Token mean, no penalty63.257.2
Token mean, penalty68.263.2
Sequence mean, no penalty67.061.4
Sequence mean, penalty74.067.8

Verified/Pro denote SWE-bench Verified/Pro. Three-seed evaluations show Reduced-LR declining 67.82 to 63.28 on Verified from updates 10 to 50, while our recipe rises 70.35 to 73.97. With matched self-verification evaluation, initialization scores 68.83/60.66, versus our 73.97/67.83: +5.13/+7.17 points beyond initialization.

Matched controls retain our reward and aggregation. Let η\eta be learning rate and β\beta the KL coefficient. For (η,β)(\eta,\beta), A uses (10−6,10−3)(10^{-6},10^{-3}), B (10−6,0)(10^{-6},0), and C (3×10−6,10−3)(3\times10^{-6},10^{-3}). All have 50 completed updates, three training seeds, and four evaluation draws/task. Here pp means percentage points:

Backbone and contrastVerified gain, pp [95% CI]Pro gain, pp [95% CI]
Qwen3.5-35B-A3B, A−B2.68 [1.32, 4.05]1.66 [1.18, 2.15]
Qwen3.5-35B-A3B, A−C4.47 [2.04, 6.90]3.49 [2.67, 4.31]
Qwen3.5-9B, A−B2.37 [2.05, 2.68]0.99 [0.62, 1.36]
Qwen3.5-9B, A−C3.68 [2.29, 5.08]2.71 [1.65, 3.78]

Both effects have positive lower bounds on both SWE benchmarks and backbones. Final unweighted current-to-starting-policy KL on common fixed held-out histories is 0.0421/0.0999/0.1128 nats/token position for original-backbone A/B/C, consistent with controlled drift.

Filtering and cutoff ablations. We wait for all eight attempts to terminate, retaining groups with at least seven valid members; incorrect solutions remain valid. Excluded trajectories have zero actor/KL loss masks. With cutoff off, retaining siblings gains 1.37 [1.18, 1.56] Verified points over strict exclusion; replacement adds 1.47 [1.28, 1.66] for 14.87 extra hours. At equal updates, cutoff saves 42.1% training time, with Verified difference −0.05 [−0.423, 0.323] points. Training self-verification adds 2.07/1.40 Verified/Pro points for 3.41% more training time, using a common evaluation prompt.

Thank you for your detailed explanations and additional experimental results. These results provide some further empirical support for your approach. However, I still believe that the overall novelty is limited and the core contribution is not sufficiently clear or distinctive. Nevertheless, I am willing to raise my score from 1 to 2.

Reviewer_LneXOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents SWE-LOOP, provide a bag of tricks with a GRPO training recipe for software engineering agents. It improves execution efficiency and SWE benchmark performance through resource-aware scheduling, failure handling, and training refinements.

Critical strengths
  1. The framework addresses practical execution bottlenecks that affect training speed and reward reliability.
  2. The execution protocol distinguishes infrastructure failures from policy failures, preventing unreliable outcomes from entering training.
  3. The experiments demonstrate substantial efficiency gains and isolate the effects of submission penalties and loss aggregation.
Critical weaknesses
  1. The proposed framework primarily integrates established techniques, limiting methodological novelty. Some essential training changes, like the infrastructure failures & policy failures, lack ablation and comparisons.
  2. The infrastructure experiments change timeout settings alongside architectural modifications, weakening causal attribution.
  3. The paper does not evaluate whether discarding entire groups containing failed or truncated trajectories is preferable to retaining valid members or resampling failed trajectories.
  4. The trained model is mainly compared with other ablation variants and the base model, not baselines from other recent works. The training curve did not show a result of test performance with the training step.
Questions for authors
  1. Why does the full method compare with the baseline under the same total wall-clock training budget, rather than the same number of updates?
  2. How much additional execution cost does verification introduce?
1 comment
AuthorsOpenReview ↗

Controlled Framework Comparisons, Filtering Ablations, and Verification Cost

Reviewer 1

Thank you for identifying the comparisons needed to assess the contribution. We now provide controlled framework comparisons, filtering alternatives, held-out checkpoint evaluations, and three-seed training results.

Contribution and timeout-controlled comparison. SWE-LOOP schedules individual execution operations: short filesystem tools, expensive shell/public-test operations, and hidden verification receive distinct admission policies. Protected capacity preserves progress under contention; unused capacity is shared. This addresses contention inside agent execution, complementing stage-level rollout pipelines.

Our new scheduler comparison holds the model, harness, inference backend, task manifest, resources, effective deadlines, cache policy, and timing boundary fixed. Each repetition uses 64 prompts and eight trajectories per prompt, with a frozen Qwen3.5-35B-A3B model and eight H200 GPUs. No policy updates or completion cutoff are involved. Timing includes preparation, verification, result assembly, and cleanup. Three matched repetitions give:

FrameworkValid trajectories/hourInvalid trajectories (%)
SkyRL-Agent Async Pipeline342.693.711%
SkyRL-Agent + our scheduler451.851.432%
LEGO-RL407.231.563%

For VV valid trajectories collected in TT seconds, throughput is Q=3600V/TQ=3600V/T. Incorrect solutions remain valid when execution/scoring complete correctly. Paired geometric throughput ratios are 1.318 [1.108, 1.569] versus SkyRL-Agent and 1.109 [1.055, 1.166] versus LEGO-RL. Brackets are approximate pointwise 95% paired Student-t intervals across three run blocks; ratios use the log scale.

The mechanism is also measured: mean per-run p95 queue waits decrease from 144.93 to 60.83 seconds for light tools and 242.73 to 96.17 seconds for hidden verification. These are enqueue-to-start waits among started operations. Thus the controlled comparison attributes the improvement to scheduling under unchanged deadlines.

Retaining valid siblings versus exclusion or replacement. Let G=8G=8 be the original group size and ngn_g the valid terminal trajectory count in group gg. We wait for all eight original attempts to terminate. We clarify the implemented rule as retaining survivors when ng≥7n_g\ge7. Valid incorrect patches remain policy outcomes; timeout/infra-error trajectories are filtered.

The following comparison disables cutoff in every arm and matches 50 completed updates, three seeds, and all other training settings. Verified denotes SWE-bench Verified; scores average four evaluation draws per task.

Filtering ruleVerified Pass@1 (%)Training hours
Require all eight valid72.65149.20
Retain survivors when at least seven valid74.02153.27
Replace filtered members, then require at least seven valid75.48168.13

Replacement permits one fresh attempt per filtered original member under the same policy snapshot. Retaining siblings improves Verified by 1.37 [1.18, 1.56] percentage points over strict exclusion. Replacement adds 1.47 [1.28, 1.66] points for 14.87 [12.72, 17.01] additional hours. These results directly establish the quality–cost choices.

Equal updates and held-out learning. We impose no equal-wall-clock constraint. The new controlled training comparisons match 50 completed policy-update iterations and report elapsed time as an outcome. Let η\eta denote learning rate and β\beta the KL coefficient. For (η,β)(\eta,\beta), A is our setting (10−6,10−3)(10^{-6},10^{-3}); B removes KL, (10−6,0)(10^{-6},0); C increases LR, (3×10−6,10−3)(3\times10^{-6},10^{-3}). Other settings match. Reduced-LR is the paper's token-mean, no-submission-penalty reference at A's LR/KL. Mean Verified Pass@1 across three seeds is:

Completed updateABCReduced-LR
1070.3570.8071.2867.82
2071.4872.2872.9366.80
3072.4073.0772.5265.35
4073.2372.3271.1864.15
5073.9771.2869.5063.28

A finishes above B by 2.68 [1.32, 4.05] points and C by 4.47 [2.04, 6.90] points. This supplies the requested performance-versus-update evidence.

Verification cost. We additionally train three runs without the public self-verification instruction, preserving tools and mandatory hidden scoring. Both arms use the same verification-on final evaluation prompt. Enabling the instruction raises total training time from 85.87 to 88.80 hours, a paired geometric increase of 3.41% [0.75%, 6.14%]. It improves Pass@1 by 3.37 [2.67, 4.07] points on Terminal-Bench 2.0, 2.07 [1.15, 2.98] on Verified, and 1.40 [1.06, 1.74] on SWE-bench Pro. This measures the additional total training cost together with its resulting performance benefit.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion period is drawing to a close, could you please review the authors' rebuttal and provide your feedback at your earliest convenience?

While I would like to thank Reviewer wNPW for the response, feedback from the other two reviewers is still absent.

Active engagement in discussions with the authors is an essential part of the peer-review process and plays an important role in ensuring the success of the conference. Your participation would be greatly appreciated.

Thank you for your time and contributions.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.