- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Lego-RL connects existing coding-agent harnesses to RL training while preserving their internal workflows. It captures tokens and metadata at the model API boundary, manages sandbox execution and reward checks, and provides tools for monitoring training and inspecting trajectories. Using Qwen3.5-35B-A3B, the paper reports improvements under OpenHands SDK, Claude Code, and OpenCode on SWE-bench Verified, with additional results on SWE-bench Pro and Multilingual.
Critical strengths
- Supporting RL training with existing agent harnesses is practically useful. The experiments cover three harnesses, and the transfer results suggest that the learned improvements are not limited to the harness used during training.
- The paper reports improvements across three SWE benchmarks. The mixed-harness experiment also shows that one shared model can achieve results comparable to separately trained models.
Critical weaknesses
- The paper lacks a comparison with models trained using other RL frameworks. Comparing LEGO-RL with the base model shows that RL training is effective, but does not establish the benefit of the proposed framework. A comparison with an existing framework using the same initial model, harness and training data would make the contribution more convincing.
- The main experiments use only one model, and controlled comparisons are needed to show whether the central design choices improve final task performance. The handling of history compaction also needs more justification. Other approaches preserve individual model calls as separate training samples when the history changes, while LEGO-RL falls back to lossy reconstruction. It is unclear how often this fallback is used, how much information is lost, and whether it affects training stability or final performance.
Questions for authors
- How does LEGO-RL compare with an existing RL training framework under the same model, harness, and training data?
- Can the authors provide controlled comparisons showing which central design choices improve final task performance?
- How often does history compaction trigger lossy reconstruction, and how does this affect learning? Why use this approach instead of keeping separate training samples before and after compaction?
Response to Reviewer 2qRn
W1 / Q1. Comparison with existing RL frameworks
Thank you for suggesting a matched comparison. We began developing LEGO-RL independently in February 2026, before the papers on Polar, OpenForgeRL, and Agent Lightning v1.0 appeared. Published evaluations differ in model configuration and scope: Polar and Agent Lightning v1.0 report Qwen3.5-4B and Qwen3.5-9B experiments, respectively, while LEGO-RL evaluates a 35B-total/3B-active MoE policy across OpenHands SDK, Claude Code, and OpenCode. Our end-to-end results and targeted alignment diagnostics demonstrate the system’s utility, but do not establish a performance advantage over another framework.
We have not conducted a matched comparison because the additional long-context agent rollouts and sandbox execution exceed our current compute budget. As resources permit, we plan to compare against an open-source framework using the same initial model, harness, training data, and training and evaluation budgets. We will make this limitation explicit and distinguish demonstrated utility from comparative claims in the future.
W2. Evidence from a second model
In addition to Qwen3.5-35B-A3B, we provide supplementary results for Qwen3-30B-A3B-Instruct-2507 trained through OpenHands SDK with SAO.
Table: SWE-bench Verified resolve rates for Qwen3-30B-A3B-Instruct-2507.
This result extends the evidence beyond the model used in the main experiments. We will include it in the revision as supplementary model-coverage evidence.
W2 / Q2. Component contributions to final performance
Our end-to-end experiments demonstrate the effectiveness of the complete framework, while the component diagnostics examine specific sources of error in its training signals. Termination-aware filtering, reward-integrity safeguards, and routing replay address different aspects of training correctness.
These mechanisms target specific failure modes: environment failures or timeouts entering task-group reward statistics, answer leakage inflating executable rewards, and discrepancies between rollout and trainer-side probability computation. We therefore view them as safeguards for valid RL training.
Reward hacking illustrates why this distinction matters. If a training task exposes its reference fix, an agent can obtain a higher reward by retrieving the answer, and RL may reinforce this shortcut. The resulting reward increase does not demonstrate transferable problem-solving ability on real tasks where reference solutions are unavailable. We will revise the manuscript to make their correctness objectives explicit and distinguish this evidence from downstream performance attribution.
W2 / Q3. History compaction and lossy reconstruction
Thank you for raising these questions. History reconstruction and context compaction follow different paths in our implementation, and we clarify them separately below.
History reconstruction after harness-managed edits. A harness may modify earlier messages when preparing a later model request, for example through internal context management or changes to injected content. The resulting history is no longer a prefix extension of the previously captured conversation. Reconstruction handles this divergence so that the training representation reflects the context supplied by the harness, while recovering original generation metadata for the unchanged token prefix. It cannot guarantee recovery of the original log-probabilities or routing records for rewritten spans.
In a supplementary OpenHands SDK run, history edits occurred in 88 of 29,448 training trajectories (0.30%). We have not separately quantified that token fraction or isolated its effect on learning. We will make clear that these edits arise from harness-managed history changes and should not be equated with compaction events.
Compaction: retaining separate training segments. Our current compaction implementation follows the principle proposed by the reviewer. It preserves pre-compaction segments with their original token IDs, log-probabilities, and routing records, and starts a new segment under the compacted context. Summary generation also participates in policy optimization. When the summary and retained history are reused as input to the next segment, they are treated as context and excluded from that segment's policy loss, avoiding duplicate loss contributions.
We will distinguish it from the reconstruction fallback described in the original paper.