acceptodds
Under review as a conference paper at ICLR 2027

XYZ-SWE: Revisiting the Long Horizon Agent Training by Modeling-System Co-Design

Abstract

Long-horizon agent training, including software engineering, is advancing rapidly, but the recipe and infrastructure needed for a strong reinforcement-learning baseline remain difficult to reproduce. This makes gains from new algorithms or data hard to distinguish from improvements to an under-tuned baseline. We present XYZ-SWE, a fully asynchronous RL framework that establishes such a baseline through model-system co-design, using overfitting on a small task pool as the testbed. This testbed exposes failures that early reward gains conceal and guides the integration of established components: gated reward, importance-weight control, aligned training and inference engines, and consistent state across policy updates. No single component is new, yet together they form the missing foundation. On raw R2E-Gym data, without new algorithms or data work, trains Qwen3.5-4B to 60.5% pass@1 on SWE-bench Verified. This is comparable to concurrent FrogNano (61.5%), which relies on online task synthesis, after about one-eighth as many training rollouts. Building on this foundation, we further filter and refine R2E-Gym, SWE-rebench V2, and Scale-SWE. With this curated subset, the 4B model improves from 47.4%/25.0% to 62.3%/38.0% on SWE-bench Verified/Pro, and the recipe also improves Qwen3.5-9B from 56.2%/30.8% to 63.8%/40.6% on Verified/Pro. Controlled comparisons show that both curation and test strengthening improve held-out performance, although stronger tests reduce apparent training-task mastery. provides a documented starting point for studying better algorithms, data and environments for asynchronous agent RL.

Primary area:
foundation or frontier models, including LLMs
Keywords:
SWE Agent, Reinforcement Learning, LLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oo56OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper discusses the issue of identifying strong reinforcement learning (RL) baselines for long-horizon agents, with the motivation to standardize the baselines in the literature and avoid people from using under-tuned baselines. To that end, the authors propose to use overfitting on a small scale dataset as a way to reveal the failure mode, and their findings suggest a list of modifications that can be made to improve its performance, such as gated reward on valid completion, matching training and inference numerics, and scheduling useful recent experiences.

=== post rebuttal ===

Thank you for the response. I feel my concern for (W1) remains since the paper in its current state feels like putting together a lot of things and sharing its observations, rather than exploring scientific insights. Thus, I maintain my score.

Critical strengths

The paper explores multiple ways to combine together different engineering practices together to form a better RL baseline, which could be informative to the community.

Critical weaknesses
  1. The paper is very confusing to read, and I feel its presentation is more like listing a group of things the authors attempted at the problem without connecting on why they are done this way. The first four pages are easy to follow, but the remaining is hard to read.
  2. As the authors have acknowledged in Section 6, most experiments in this paper only uses one seed over one harness only. Given that the claim of the paper is on a strong RL baseline that people can use to avoid from using an under-tuned weaker baseline, I think more experiments and some report in terms of variance should at least be attempted. It is understandable that compute budgets may restrain people from conducting larger scale experiments, but the authors can at least try two seeds instead of one.
  3. On section 3.1, the authors claim that “If an RL pipeline cannot reliably memorize a small, solvable subset of tasks, it cannot be expected to generalize over a large, noisy corpus.” While I agree with this statement, what I do not agree with is the inverse, which the authors implicitly assumed in the method’s design. It seems the authors mostly relied on using performance obtained from overfitting a model on a small testbed to establish whether an RL algorithm is usable or not, but it could also be the case that “if an RL pipeline reliably memorize a small solvable subset of tasks, it cannot generalize.” This scenario is possible for the authors’ proposed approach. Thus, I believe the study’s methodology needs more evidence, explanation, and clarification.
Questions for authors

Similar as Critical Weakness (3), I wonder what is the reason behind choosing overfitting as a stress test for the proposed approach?

1 comment
AuthorsOpenReview ↗

Response to Reviewer oo56

Thank you for taking the time to read our work.The soundness rating rests on a reading of the method that the paper does not support. We address it first, then presentation and seeds.

Soundness (necessary vs. sufficient). The review argues that fitting a small pool does not imply generalization. The paper never claims it does. Its logic has two steps, in opposite directions.

  • Holding the pool is necessary for full training to succeed, not sufficient: the submitted text opens the stress-test section with "If an RL pipeline cannot reliably memorize a small, solvable subset of tasks, it cannot be expected to generalize over a large, noisy corpus."

  • Our corrections are sufficient for the recipe to hold the pool, not necessary: other remedies may also work, and Appendix K notes that removing recovery or enforcing our particular format "is not a universal prescription". So the screen only rejects recipes, passing it is never taken as evidence of generalization, and every headline number comes from held-out SWE-bench Verified and Pro, none from the training pool. The gain–decay model in Appendix B.1 formalizes the first step: a recipe that keeps losing ground on the pool carries a permanent deficit into full training. The paper also reports the case the review raises, where training reward and held-out accuracy separate: past step 250 the 4B degrades on held-out data while its training reward barely moves, and the 9B, trained on the same deck under the same eviction rule, does not (Fig. 6). The revision adds an explicit statement of this screen-then-validate logic to Sections 3.1 and 6.

Presentation. Each component in Section 3 is introduced by the failure that motivates it, in the order in which the diagnostic exposes those failures:

  1. The Prime-RL recipe, which leaves hinted invalid turns unpenalized, collapses by epoch 11. Gated reward, which pays only for valid, explicitly submitted completions, reaches a 74.5% plateau and then collapses from epoch 42 with format errors (Fig. 1).
  2. Those errors get past the gate because parsers silently repair malformed output: the parsed gate lets 6 of 13 malformed turns through (Table 7). Checking the raw tokens closes that path, and reward holds around 80% (Fig. 1).
  3. The remaining gap is in the system layer, where numerical mismatch, stale KV caches and load-blind scheduling change the experience the optimizer sees. Fixing them brings the full recipe to 93–96% (Fig. 3). Neither layer is enough alone: with the system fixes but only gated reward, training still collapses near epoch 200 (faint line in Fig. 3).

The revision collects this chain in a roadmap paragraph and Table 1, which lists each failure with its fix and measured effect.

Each fix is an established technique, chosen for a failure the testbed exposed and weighed against the published alternatives that Appendix K compares with ours. For invalid turns, SkyRL-Agent continues with recovery hints, NeMo assigns negative advantages to invalid calls, and SWE-Master forces a submission; FrogNano and MiMo-V2.6 penalize length, while we use no length penalty. The testbed also tests alternatives directly: masking invalid turns instead of ending the rollout, run for this rebuttal, collapses three times and then plateaus about 15 points lower (see our reply to QkkX).

Seeds. Two new seeds of the proposed recipe agree with the submitted run within one point in every reported window. The stability claims of Section 3 concern training dynamics on the testbed, where repeated runs fit in the rebuttal period, so we reran the recipe there changing only the random seeds:

Full recipe (testbed)Epochs 100–150Epochs 200–230Peak
Seed 42 (submitted run)93.0%95.8%96.9%
Seed 43 (new)92.4%95.2%97.0%
Seed 44 (new)92.4%96.2%97.2%

The table uses the paper's metric and windows, and seed 42 reproduces the submitted numbers. Values are post-gate rewards averaged over completed rollouts, with peaks taken within the 240 plotted epochs; revised Fig. 3 shows the three seeds as a mean with a min–max band.

We did not rerun the failing recipes, because a single collapse already makes a recipe unreliable. Seeds on which it happens to survive would not make it safe to use. Gated reward alone has in fact collapsed in two independent runs, near epoch 42 (Fig. 1) and near epoch 200 (faint line in Fig. 3).

Since the soundness concern rests on a claim the paper does not make, and the proposed recipe is stable across seeds, we ask the reviewer to reconsider the rating.

Reviewer_QkkXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

XYZ-SWE presents an asynchronous RL training system built on Prime-RL for software engineering agents. The authors train on a small set of tasks for many epochs to find training problems, then improve reward handling, output-format checks, numerical precision, cache updates, and task scheduling. They also filter existing SWE datasets by checking whether task descriptions, tests, and reference patches agree, and separately study the benefits of stronger tests. The resulting training recipe improves Qwen3.5-4B and Qwen3.5-9B on SWE-bench Verified and Pro.

Critical strengths
  1. The paper documents useful practical problems in long-horizon RL training. Its analysis of output formatting, numerical differences, stale caches, and scheduling can help researchers understand why apparently successful training runs later become unstable.
  2. The training-data study provides concrete examples of incorrect patches passing tests and valid alternative patches failing them. The test-strengthening experiment keeps task identities fixed and changes the tests used for reward, providing useful evidence that better reward signals can improve held-out performance.
Critical weaknesses
  1. I feel the framework-level claim is somewhat overstated. The paper reads more like a training recipe developed for a specific SWE setup and evaluated on two benchmarks. Most detailed studies focus on Qwen3.5-4B, with additional results on 9B from the same family. Since some fixes directly concern the model’s output format and recurrent states, it is not clear how well the findings would transfer to other model families or training systems.
  2. I am not fully convinced that trajectories with malformed tool calls should be stopped rather than allowed to recover. Learning to recover from such errors can also be useful in long-horizon tasks. We could mask the invalid turn and still train on the remaining valid steps. A comparison with this alternative would help explain why preventing these recovery trajectories from becoming successful training examples is better.
  3. I appreciate the controlled data experiments in Figure 5, but the filtering comparison uses rejected tasks as the control. This shows a difference between accepted and rejected data, but does not directly show the gain over randomly sampling from the execution-valid pool. Also, the additional gains from curated RL in the final results are relatively small compared with the gains from RL on the original data. Since the data sources and some training settings change between these runs, the connection between Figure 5 and the final results needs more explanation. Testing the filtering on another model family would also strengthen the evidence.
Questions for authors
  1. What does XYZ-SWE add beyond the underlying Prime-RL framework, and which parts of the recipe are expected to work beyond the current SWE setup? Can the authors provide evidence that the key findings hold for other models or training settings?
  2. Have the authors compared stopping at a malformed tool call with allowing recovery while masking or penalizing only the invalid turn? How do these choices affect training stability, format-error rates, and final task performance?
  3. Can the authors explain why the gains from data curation in Figure 5 are larger than the additional gains from curated RL in the final results? Do the benefits also hold for another model family?
4 comments
AuthorsOpenReview ↗

Response to Reviewer QkkX, Part 1/2

Thank you for the review. Our reply comes in two parts because of the length limit.

"Framework" vs. recipe. The revision calls XYZ-SWE a recipe because it builds on Prime-RL rather than replacing it. It is not a recipe in the narrow sense of a loss and hyperparameters on an unchanged stack: the changes also reach the system layer, in the trainer's numerics and weight transfer, cache reconstruction in the inference engines after each weight refresh, and load-aware placement of requests across engines (Section 3.3, Appendix C), alongside reward gating and raw-token checking in the harness. The abstract accordingly describes a recipe built through model–system co-design. The contributions are (1) the overfitting diagnostic as a reusable method, (2) the corrections it motivates across the algorithm and system layers (gated reward, raw-token format checking, train/inference numerical alignment, cache reconstruction on weight refresh, and load-aware scheduling), and (3) the execution-grounded data audit.

Stop vs. mask-and-continue. We ran this comparison on the testbed: masking invalid turns is unstable early and then plateaus about 15 points below ending the rollout. Our recipe ends a rollout at its first invalid turn with zero reward (Section 3.2), so an invalid turn never shares in a positive reward. The new arm is identical except that each invalid turn (malformed or raw-envelope-invalid call, duplicate call, truncated response, non-exact submission) is masked from the loss, while the rollout continues with the harness's correction and keeps its final reward. Episode-level limits are unchanged. Results for seed 42, stopped at epoch 180 once the outcome was clear (Fig. 3, Appendix B.2):

  • Early training is unstable. Within 22 epochs reward collapses three times, to 1.7%, 9.2% and 16.4%, as rollouts fill with 66–76 masked turns each, most of the 100-turn budget. Each time it recovers only partly.
  • It then plateaus well below the full recipe: 77.6% over epochs 100–150 (peak 85.4%), against 92.4–93.0% for the full recipe across three seeds, and below the full recipe's band at every stage (Fig. 3). Its reward still credits rollouts that needed the harness's corrections, which favors it in this comparison.
  • Invalid turns do not die out. Around epoch 167, 14% of rollouts are still loops of malformed calls (52 of their 67 turns rejected; 46% solved, against 93% for clean rollouts), and successful rollouts that needed corrections still reinforce their valid turns.

Our claim is that ending the rollout is sufficient to prevent this collapse, not that it is the only remedy. The review's other suggestion, a negative reward for invalid turns, is a reasonable alternative that we did not test; NeMo assigns negative advantages to invalid calls, and MiMo-V2.6 penalizes faulty intermediate actions (Appendix K).

Masking also cannot act on errors the parser hides. Parsers silently repair malformed output: a call still executes for most of our 13 malformed test turns, and our parsed gate alone lets 6 through (Table 7). A turn the parser has already repaired looks valid and would not be masked, which is why our check reads the raw tokens.

That parsers execute malformed calls is not specific to Qwen's format: it happens in all seven other families we tested. We ran the same test on the tool-call formats of those families (Gemma 4, gpt-oss, Llama 3.1, Ministral 3, GLM-4.7, Granite 4.2 and MiniCPM5) through current vLLM and SGLang (Appendix B.3, Table 8). Every family has malformed turns that still execute on at least one server, up to 13 of 13 for Gemma 4 on vLLM. An argument given twice (ls, then rm -rf build) runs rm -rf build for six of the seven families, even where the server constrains decoding with a grammar by default; only MiniCPM5's parsers reject it. Granite 4.2 uses Qwen3.5's tool-call format and reproduces Table 7's server results exactly.

AuthorsOpenReview ↗

Response to Reviewer QkkX, Part 2/2

Filter control. Fig. 5a is designed to test whether our second pipeline works. Its control is 7,500 rows that pass the standard fail-to-pass gate, 91% of them rows that our audit rejects, so if the audit were wrong, the two arms would train equally well. The curated deck leads at all five matched steps: the pipeline tells good rows from bad ones and makes RL both better and more efficient. We agree that a uniform draw from the validated pool is a different baseline, and we expect a smaller gap against it. We could not run it within the rebuttal period. However, we note three points:

  • Curation improves efficiency. Over the first 250 steps, 60% of the control's rollouts fall in groups where every attempt fails, against 29% for the curated deck, so our pipeline makes the model better while wasting far fewer rollouts.
  • Our pipeline finds defects even in already-filtered datasets. Scale-SWE, SWE-rebench, and R2E-Gym are all filtered datasets (Appendix E.6), yet the audit still rejects 58% of the rows that pass our fail-to-pass gate. We believe the gain would be larger on raw GitHub pull requests that no prior filter has screened.
  • The control already includes accepted rows. R2E-Gym has too few rejected rows, so 657 of its control rows come from the accepted set (Appendix E.7).

Test strengthening needs no control set: both arms train on the same 4,889 instance IDs and differ only in the grading suite, and the strengthened arm leads at every matched step (Fig. 5c,d).

Why the end-to-end gain is smaller than in Fig. 5. Fig. 5a is the largest contrast the audit can produce, and full training absorbs part of the cost of bad rows. In full training, a task whose tests reject every attempt gives a group with zero advantage, which our zero-advantage filter drops before the update (Appendix A), so such rows cost compute without adding signal (the 60% above).

The end-to-end number does not isolate curation, but 3 points is still a large gain at this level. The 4B comparison (R2E-Gym only 62.0% → curated 65.0% on Verified, Table 5) also changes the data source. For scale, gpt-oss-120b has 5.6 times the parameters of gpt-oss-20b and scores 1.7 points higher on Verified (62.4% vs. 60.7%; OpenAI, arXiv:2508.10925). Because Verified is also our checkpoint-selection set, Pro is the cleaner check. There the gain shows on the Python subset of Pro-v2, which matches our Python-only training data (64.8% → 68.1%).

Other model families. We agree that a second family would strengthen the paper, and we explain below why most corrections are not tied to Qwen. We use Qwen3.5 because the prior 4B–9B SWE agents we compare against all start from it (Table 5), which keeps the headline comparison meaningful. Repeating the screen and the main runs on another family was beyond our compute, and the rebuttal compute went to the seed and masking runs. On whether the corrections are tied to Qwen:

  • The screen is defined relative to the base model: its pool is the tasks that model solves at least once in four attempts. It applies to any family and would show which corrections that family needs.
  • Gated reward, KV recomputation and load-aware placement act in the harness and the serving system and never touch the model.
  • The raw-token check depends on the model only through its chat template's tool-call grammar, and the parser failures it targets appear in every family we tested (part 1).
  • Numerical alignment is the one architecture-specific component, since which paths need FP32 depends on the layers. The problem itself is not specific to Qwen: MiniMax-M1, a hybrid lightning-attention model, reports a training–inference precision mismatch that stalled reward growth until its LM head moved to FP32, and notes that it did not appear in smaller dense models with softmax attention (MiniMax-M1, Section 3.2).

Training stacks built on other families have also arrived at the same fixes. MiMo-V2.6, which trains Xiaomi's own MiMo-V2 architecture, rebuilds the KV cache after policy updates, bounds staleness, filters zero-advantage groups, and aligns weights and expert routing between rollout and training (Appendix K); Magistral (Mistral) also filters zero-advantage groups. Several open families have recently released models with competitive SWE-bench scores at this size, such as Granite 4.2-8B and MiniCPM5-2B, and repeating the screen on one of them is the natural next step.

Curation reliability. The verdicts that decide reward validity are checked by execution. Across all judge transcripts, 97.3% of the 5,351 too_loose verdicts record a non-gold patch that the judge wrote as a wrong fix passing the graded suite, and 97.1% of the 3,143 over_restrictive verdicts record the fix the judge considered correct being executed and failing.

Thanks for the additional experiments. To clarify my comment on “masking invalid turns,” my concern was whether the current recipe could weaken the model’s ability to recover from errors. I wasn’t asking directly test my tentative suggestion, but I really appreciate the effort you put into exploring it. The additional results across multiple benchmarks have addressed most of my concerns, therefore i will raise my score to 3

We are glad hear that our responses have addressed your concerns. Thank you for your valuable feedbacks and efforts put into the review!

Reviewer_E1eTOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

this paper investigates why agentic RL receives misleading training signals and proposes solutions at both the infrastructure and dataset levels: Overfitting as a diagnostic tool for agentic RL, Correcting algorithmic feedback and execution, Semantic consistency auditing of SWE datasets.

Critical strengths

this paper not simply achieving a competitive SWE-bench score. It investigates why agentic RL receives misleading training signals and proposes solutions at both the infrastructure and dataset levels. especially: Overfitting as a diagnostic tool for agentic RL, Correcting algorithmic feedback and execution and Multiple infrastructure and optimization changes make it difficult to isolate individual contributions.Semantic consistency auditing of SWE datasets

Critical weaknesses

Multiple infrastructure and optimization changes make it difficult to isolate individual contributions. Can the LLM reliably distinguish incorrect solutions from valid alternative implementations? The quality of the curation pipeline depends heavily on this.

Questions for authors

Great paper in general. one question: Does better training reward translate into better real-world software engineering? High training reward alone does not establish better problem-solving ability.

1 comment
AuthorsOpenReview ↗

Response to Reviewer E1eT

Thank you for the positive assessment. Brief answers below.

Isolating contributions. The paper separates the components layer by layer, and the rebuttal adds seeds and a mask-and-continue arm. Figs. 1 and 3 add gated reward, then the raw-format check, then the system fixes, with each step as its own curve. Appendix B (fixed-pool runs, parser audit) and Appendix C (system fixes) give per-component detail, and the new Table 1 summarizes the chain. We also repeated the final recipe at three seeds, which agree within one point in every window, and added a mask-and-continue arm that isolates the choice to end the rollout at an invalid turn. It collapses three times early and then plateaus about 15 points lower (see our reply to QkkX).

Can the LLM tell wrong patches from valid alternatives? The verdicts that decide reward validity are checked by execution, not by the judge reading code. The judge writes a patch it considers wrong and the actual graded suite scores it, so a too_loose verdict needs that patch to really pass. Across all judge transcripts, 97.3% of the 5,351 too_loose verdicts record a non-gold patch that the judge wrote as a wrong fix passing the graded suite (in 83.8%, a patch written on the buggy code rather than on top of gold), and 97.1% of the 3,143 over_restrictive verdicts record the fix the judge considered correct being executed and failing. Whether a passing patch is truly wrong remains the judge's label; the strengthening experiment (Section 4.2) tests whether that label tracks something real by training the same instance IDs on both suites. Test strengthening is instructed to add only checks that follow from the issue or established public behavior, and every strengthened suite must still pass the gold patch.

Does training reward translate into capability? We agree that training reward alone does not establish capability, which is why we measure capability only on held-out tasks: every headline number is SWE-bench Verified or Pro, never training reward. We also examined how the agent's behavior changes after RL, not only its score.

We compared the evaluation traces of the reported 9B and its base on all 483 Verified tasks. The trained model resolves 65.7% of rollouts, against 55.7% and 57.2% for two base runs (Table 5's 56.2% lies between them). The behavior behind the gain:

  • It works from the repository's own tests. It runs the test module that the graded tests live in for 89% of rollouts (base: 80%), and before its first edit for 24% (base: 8%). The base instead writes throwaway test scripts in 41% of rollouts (trained: 5%) and leaves scratch files in the repository in 12% (trained: 3%).
  • It finishes. The base exhausts its 100-turn budget in 43% of rollouts and submits in only 57%; a quarter of its solves come from rollouts that never submitted. The trained model submits in 98%, and its rejected tool calls fall from 4.2% of turns to 0.01%.
  • It is more consistent. Tasks solved in all four attempts rise from 188 to 259, while tasks solved at least once rise only from 350 to 370, so RL mainly makes the agent reliable on problems it could sometimes solve.

Other model families. Most corrections are not tied to Qwen; our reply to QkkX gives the details. In short, the screen is defined relative to whichever base model is used, most corrections act in the harness or the serving system, and training stacks built on other families report the same numerics and cache problems (MiniMax-M1, MiMo-V2.6).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment to all reviewers

We thank all reviewers for their comments. We have uploaded a revised PDF with the following changes.

  • New runs on the diagnostic testbed (post-submission): two more seeds of the full recipe and a mask-and-continue variant. The three seeds agree within one point in every reported window (Fig. 3, Appendix B.2). The variant removes invalid turns from the loss instead of ending the rollout (Section 3.2, Fig. 3, Appendix B.2).
  • Other model families (post-submission). Appendix B.3 runs the malformed-turn test of Table 7 on the tool-call formats of seven other model families (Table 8).
  • Presentation. The revision makes the method's structure and claims explicit. Section 3 now opens with a roadmap and a summary table (Table 1). Sections 3.1 and 6 state that the overfitting check is a necessary-condition screen. The contributions are described as a diagnostic method, a set of model-agnostic corrections, and an execution-grounded data audit.
  • Updated evaluation. SWE-bench numbers are re-evaluated under a revised protocol, and none of the conclusions change. Each checkpoint is evaluated with its own sampling configuration (Appendix H.5.1), while the training curves in Fig. 6b use one fixed configuration (temperature 1.0, top-p 0.95, top-k 20). SWE-bench Pro is now reported as Pro-v1 and Pro-v2, together with the Python subset of Pro-v2. Under this protocol the R2E-Gym-only 4B is reported at update 340 with 62.0% on Verified (submitted: 60.5% at update 240), and the curated 4B and 9B reach 65.0% and 65.7% (submitted: 62.3% and 63.8%). FrogNano is re-evaluated in our harness (Section 5.3), and Appendix I.5 lists the 9B run's updated training settings.

As the discussion period is approaching to its end, we would like to ask once again for any additional feedbacks the reviewers may have. If there is any confusion unresolved, we would very much like to clarify and have further discussions! Thanks for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.