acceptodds
Under review as a conference paper at ICLR 2027

Lego-RL: Harness-Native Reinforcement Learning for Coding Agents

Abstract

Reinforcement learning for coding agents increasingly relies on long-running agent harnesses to manage tool integration, repository contexts, and execution feedback. However, the native execution environments of these harnesses are inherently misaligned with policy-gradient training: environmental crashes and reward hacking corrupt outcome signals, while train–inference discrepancies decouple rollout behavior from policy updates. To address this, we present **Lego-RL**, a framework that bridges native coding-agent harnesses with scalable policy-gradient optimization without modifying their internal control flow. Lego-RL is built upon three pillars:**(1) faithful optimization** via in-process LLM proxying that captures raw generation streams for token-level alignment and robust trainer-side log-probability recomputation, even under harness-side compaction or re-serialization; **(2) reliable execution** via scalable sandbox orchestration featuring image caching and stage-wise defenses to mitigate reward hacking; and **(3) observable training** through an integrated plugin that automates validation and monitoring, paired with a Live UI for granular trajectory diagnostics. With GSPO, **LEGO-RL** improves Qwen3.5-35B-A3B's SWE-bench Verified resolve rates by , , and points under OpenHands SDK, Claude Code, and OpenCode, respectively, with rollout–training sampled-token probability correlations of at least . The gains generalize across harnesses; mixed-harness training and alternative RL algorithms further demonstrate the flexibility.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
reinforcement learning, coding agent, reward hacking, native harness

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_2qRnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Lego-RL connects existing coding-agent harnesses to RL training while preserving their internal workflows. It captures tokens and metadata at the model API boundary, manages sandbox execution and reward checks, and provides tools for monitoring training and inspecting trajectories. Using Qwen3.5-35B-A3B, the paper reports improvements under OpenHands SDK, Claude Code, and OpenCode on SWE-bench Verified, with additional results on SWE-bench Pro and Multilingual.

Critical strengths
  1. Supporting RL training with existing agent harnesses is practically useful. The experiments cover three harnesses, and the transfer results suggest that the learned improvements are not limited to the harness used during training.
  2. The paper reports improvements across three SWE benchmarks. The mixed-harness experiment also shows that one shared model can achieve results comparable to separately trained models.
Critical weaknesses
  1. The paper lacks a comparison with models trained using other RL frameworks. Comparing LEGO-RL with the base model shows that RL training is effective, but does not establish the benefit of the proposed framework. A comparison with an existing framework using the same initial model, harness and training data would make the contribution more convincing.
  2. The main experiments use only one model, and controlled comparisons are needed to show whether the central design choices improve final task performance. The handling of history compaction also needs more justification. Other approaches preserve individual model calls as separate training samples when the history changes, while LEGO-RL falls back to lossy reconstruction. It is unclear how often this fallback is used, how much information is lost, and whether it affects training stability or final performance.
Questions for authors
  1. How does LEGO-RL compare with an existing RL training framework under the same model, harness, and training data?
  2. Can the authors provide controlled comparisons showing which central design choices improve final task performance?
  3. How often does history compaction trigger lossy reconstruction, and how does this affect learning? Why use this approach instead of keeping separate training samples before and after compaction?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 2qRn

W1 / Q1. Comparison with existing RL frameworks

Thank you for suggesting a matched comparison. We began developing LEGO-RL independently in February 2026, before the papers on Polar, OpenForgeRL, and Agent Lightning v1.0 appeared. Published evaluations differ in model configuration and scope: Polar and Agent Lightning v1.0 report Qwen3.5-4B and Qwen3.5-9B experiments, respectively, while LEGO-RL evaluates a 35B-total/3B-active MoE policy across OpenHands SDK, Claude Code, and OpenCode. Our end-to-end results and targeted alignment diagnostics demonstrate the system’s utility, but do not establish a performance advantage over another framework.

We have not conducted a matched comparison because the additional long-context agent rollouts and sandbox execution exceed our current compute budget. As resources permit, we plan to compare against an open-source framework using the same initial model, harness, training data, and training and evaluation budgets. We will make this limitation explicit and distinguish demonstrated utility from comparative claims in the future.

W2. Evidence from a second model

In addition to Qwen3.5-35B-A3B, we provide supplementary results for Qwen3-30B-A3B-Instruct-2507 trained through OpenHands SDK with SAO.

Table: SWE-bench Verified resolve rates for Qwen3-30B-A3B-Instruct-2507.

ModelHarnessRL objectiveBase modelBest checkpoint (step 160)Difference
Qwen3-30B-A3B-Instruct-2507OpenHands SDKSAO22.8%30.2%+7.4 pp

This result extends the evidence beyond the model used in the main experiments. We will include it in the revision as supplementary model-coverage evidence.

W2 / Q2. Component contributions to final performance

Our end-to-end experiments demonstrate the effectiveness of the complete framework, while the component diagnostics examine specific sources of error in its training signals. Termination-aware filtering, reward-integrity safeguards, and routing replay address different aspects of training correctness.

These mechanisms target specific failure modes: environment failures or timeouts entering task-group reward statistics, answer leakage inflating executable rewards, and discrepancies between rollout and trainer-side probability computation. We therefore view them as safeguards for valid RL training.

Reward hacking illustrates why this distinction matters. If a training task exposes its reference fix, an agent can obtain a higher reward by retrieving the answer, and RL may reinforce this shortcut. The resulting reward increase does not demonstrate transferable problem-solving ability on real tasks where reference solutions are unavailable. We will revise the manuscript to make their correctness objectives explicit and distinguish this evidence from downstream performance attribution.

W2 / Q3. History compaction and lossy reconstruction

Thank you for raising these questions. History reconstruction and context compaction follow different paths in our implementation, and we clarify them separately below.

History reconstruction after harness-managed edits. A harness may modify earlier messages when preparing a later model request, for example through internal context management or changes to injected content. The resulting history is no longer a prefix extension of the previously captured conversation. Reconstruction handles this divergence so that the training representation reflects the context supplied by the harness, while recovering original generation metadata for the unchanged token prefix. It cannot guarantee recovery of the original log-probabilities or routing records for rewritten spans.

In a supplementary OpenHands SDK run, history edits occurred in 88 of 29,448 training trajectories (0.30%). We have not separately quantified that token fraction or isolated its effect on learning. We will make clear that these edits arise from harness-managed history changes and should not be equated with compaction events.

Compaction: retaining separate training segments. Our current compaction implementation follows the principle proposed by the reviewer. It preserves pre-compaction segments with their original token IDs, log-probabilities, and routing records, and starts a new segment under the compacted context. Summary generation also participates in policy optimization. When the summary and retained history are reused as input to the next segment, they are treated as context and excluded from that segment's policy loss, avoiding duplicate loss contributions.

We will distinguish it from the reconstruction fallback described in the original paper.

Reviewer_XBurOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LEGO-RL, an RL framework for coding agents. It integrates generation-time token capture, MoE routing replay, asynchronous training, sandbox orchestration, reward-integrity safeguards, and trajectory-level monitoring while preserving the control flow of existing agent harnesses. Built on verl and Harbor, the framework supports OpenHands SDK, Claude Code, and OpenCode. Experiments demonstrate improvements in SWE-bench performance across the three harnesses, cross-harness policy transfer, and mixed-harness training.

Critical strengths
  • The framework demonstrates that existing coding-agent harnesses can be integrated into large-scale RL training without replacing their native interaction loops. The improvements across multiple harnesses and benchmarks provide useful evidence of practical applicability.
  • The paper systematically addresses execution failures, reward integrity, token alignment, and MoE routing consistency. The associated diagnostics and ablations provide useful insights into implementation challenges that arise in real-world coding-agent RL training.
Critical weaknesses
  • Most core components build directly on existing techniques, including black-box harness integration, asynchronous RL, MoE routing replay, sandbox orchestration, and training observability. Closely related frameworks such as Polar, Agent Lightning, and OpenForgeRL already address harness-native RL, while OpenForgeRL also explores multi-harness training. The paper does not establish a sufficiently distinct technical contribution beyond integrating these existing components.
  • The experiments primarily demonstrate that the complete framework works, rather than identifying which mechanisms are responsible for the observed improvements. This limits the strength of the paper's technical contributions.
Questions for authors
  1. What are the main technical advances of LEGO-RL over Polar, Agent Lightning, and OpenForgeRL? Which components provide measurable improvements in downstream RL performance beyond existing implementations?
  2. Does enabling MoE routing replay improve final RL performance or training stability compared with disabling it, rather than merely improving probability alignment?
  3. How frequently do context compaction and history reconstruction occur across the three harnesses? What fraction of training tokens lack original log-probabilities or routing metadata, and how does this affect training stability?
  4. Could the authors report the actual distributions of input context lengths, generated tokens, and interaction turns per trajectory across harnesses, as well as the number of generated and admitted trajectories and the proportion contributing nonzero advantages?
  5. How sensitive are the reported improvements to checkpoint selection and training randomness? Can uncertainty estimates or independent evaluation results be provided to support the main comparisons?
3 comments
AuthorsOpenReview ↗

Response to Reviewer XBur part 1

W1 / Q1. Technical contribution and related frameworks

Thank you for asking us to clarify the contribution. LEGO-RL contributes infrastructure for training MoE coding agents through their native harnesses. Its technical content lies in the mechanisms connecting native execution, trajectory construction, and policy updates, evaluated through targeted diagnostics and end-to-end training.

Relationship to existing frameworks. We began designing and developing LEGO-RL independently in February 2026. The papers describing Polar, OpenForgeRL, and Agent Lightning v1.0 appeared during this development period. We distinguish these concurrent developments from the original Agent Lightning work, which predates LEGO-RL.

Technical mechanisms. LEGO-RL captures token IDs, behavior probabilities, and expert-routing records during generation. It uses tool-call identity to associate harness-reserialized histories with captured actions and aligns routing records with the tokens consumed by the trainer. It also distinguishes execution failures from ordinary task failures: execution failures are excluded from task-group reward statistics and receive zero reward-driven advantages, while ordinary task failures remain valid outcomes. These mechanisms address concrete problems in constructing training signals from native harness execution.

Empirical evidence. We evaluate the complete system by training a 35B-total/3B-active MoE policy through OpenHands SDK, Claude Code, and OpenCode. Under the reported best-checkpoint protocol, SWE-bench Verified resolve rates improve by 6.4, 5.8, and 9.4 percentage points, respectively. In targeted diagnostics, correctly aligned routing replay reduced mean absolute rollout–trainer probability discrepancy from 0.0062 to 0.00245, approximately 60%.

These results support system-level effectiveness and rollout–trainer consistency. These mechanisms are designed to enforce essential requirements for correct training rather than optional performance enhancements. Their value therefore cannot be assessed solely by changes in the final resolve rate: reward hacking, for example, can inflate measured rewards without improving genuine task-solving ability. We do not claim a performance advantage over other frameworks under matched conditions. We will revise the contribution statement and comparison to clarify these requirements and the evidence supporting them.

W2. Attribution of downstream gains

Our end-to-end experiments demonstrate the effectiveness of the complete framework, while the component diagnostics examine specific sources of error in its training signals. Termination-aware filtering, reward-integrity safeguards, and routing replay address different aspects of training correctness as we expalined in W1.

These mechanisms target specific failure modes: environment failures or timeouts entering task-group reward statistics, answer leakage inflating executable rewards, and discrepancies between rollout and trainer-side probability computation. We therefore view them as safeguards for valid RL training.

Reward hacking illustrates why this distinction matters. If a training task exposes its reference fix, an agent can obtain a higher reward by retrieving the answer, and RL may reinforce this shortcut. The resulting reward increase does not demonstrate transferable problem-solving ability on real tasks where reference solutions are unavailable. We will revise the manuscript to make their correctness objectives explicit and distinguish this evidence from downstream performance attribution.

Q2. Routing replay and final performance

We have not established that enabling routing replay improves final resolve rate or training stability relative to disabling it. Our rationale for including replay is to improve consistency between rollout generation and trainer-side probability computation. The diagnostic supports this specific objective: correctly aligned replay reduces the mean absolute probability difference from 0.0062 to 0.00245, approximately 60%. Misaligned replay increases it to 0.0954, showing that preserving token–routing alignment is important for replay to achieve improved rollout–trainer probability consistency.

AuthorsOpenReview ↗

Response to Reviewer XBur part 2

Q3. Compaction, reconstruction, and metadata coverage

Thank you for raising this question. We provide one OpenHands SDK experiment with context compaction:

MetricOpenHands SDK
Training trajectories29,448
Trajectories with compaction3,778 (12.83%)
Compaction events4,820
Trajectories with history edits88 (0.30%)

Compaction frequency depends on the configured token threshold and the length of agent interactions. This run used a 96k-token condensation threshold; the 12.83% rate is specific to this configuration and workload.

Compaction and training records. In our compaction experiments, trajectories are split into training segments at compaction boundaries. Pre-compaction segments retain their captured token IDs, log-probabilities, and routing records. Summary generation is also included in policy optimization. When the summary and retained history are reused as context in the next segment, they are excluded from that segment’s policy loss to avoid counting them twice. Compaction therefore preserves previously captured training records while changing the context for subsequent actions.

History reconstruction. Non-compaction history edits occurred in 0.30% of trajectories. This is a trajectory-level rate, not the fraction of training tokens missing original metadata. That fraction depends on the rewritten span and the metadata recovered from the unchanged prefix; retries and routing-capture failures also require separate accounting.

We will report these results as supplementary OpenHands SDK measurements.

AuthorsOpenReview ↗

Response to Reviewer XBur part 3

Q4. Trajectory distributions and effective training data

We report context lengths, generated tokens, interaction turns, and rollout and filtering counts separately below.

Table 1: Mean input context length per model call (tokens).

HarnessMean
OpenHands SDK56,356.4
Claude Code64,711.0
OpenCode54,354.4

Table 2: Peak input context length per trajectory (tokens).

HarnessMeanP50P90P95P99
Claude Code75,636.568,955.5122,814.7143,706.3166,433.7
OpenCode66,417.361,276.0111,718.0131,089.8169,871.8

Table 3: Generated tokens per training trajectory.

HarnessMeanP50P90P95P99
OpenHands SDK16,660.413,770.033,126.040,425.556,996.5
Claude Code19,010.713,952.040,369.753,691.583,513.2
OpenCode20,720.216,490.042,519.355,262.881,735.9

Table 4: Interaction turns per training trajectory.

HarnessMeanP50P90P95P99
OpenHands SDK58.854.0105.0122.0162.0
Claude Code58.149.0111.0137.0200.0
OpenCode73.865.0138.0167.0231.0

Table 5: Rollout records and termination-aware filtering.

HarnessRollout recordsBatch rowsRows passing filtering
OpenHands SDK66,89364,51263,002
Claude Code69,40265,02460,405
OpenCode72,09865,02460,925

Nonzero advantages. The paper provides related group-level statistics: zero-reward-variance groups increase from 44.7% to 51.4% across OpenHands SDK training and account for approximately 43.3% in OpenCode. We will summarize these results in the revised manuscript.

Q5. Checkpoint selection and uncertainty

We assess sensitivity to checkpoint selection and provide additional training results below.

Checkpoint selection. Checkpoints were selected on SWE-bench Verified and then evaluated on SWE-bench Pro and SWE-bench Multilingual. To show how the reported gains depend on checkpoint choice, we compare the best and last evaluated checkpoints from the training curves in the paper.

Table: SWE-bench Verified resolve rates at the initial, best, and last evaluated checkpoints. Differences are in percentage points (pp).

HarnessInitialBest checkpointLast checkpointLast − initialBest − last
OpenHands SDK64.0%70.4% (step 95)68.6% (step 125)+4.6 pp1.8 pp
Claude Code62.4%68.2% (step 110)66.4% (step 120)+4.0 pp1.8 pp
OpenCode57.2%66.6% (step 50)64.8% (step 120)+7.6 pp1.8 pp

The last evaluated checkpoints retain gains of 4.0–7.6 pp and are 1.8 pp below the respective best checkpoints. The improvements therefore persist beyond the selected peaks. The selected policies also improve on Pro and Multilingual, neither of which was used for checkpoint selection. We have not evaluated checkpoint selection on a separate held-out set for Verified; we will state this protocol explicitly and report both best and last evaluated scores.

Additional training runs. We conducted further OpenHands SDK and Claude Code experiments to examine whether policy improvement also occurs beyond the original runs. For OpenHands SDK, both runs start from Qwen3.5-35B-A3B and use GSPO. We compare step 50 in both runs rather than selecting a separate peak for each.

Table: SWE-bench Verified resolve rates before training and at step 50 in two OpenHands SDK runs. Parentheses indicate evaluation context limits.

RunBefore trainingStep 50Change
Original64.0% (200k)68.8% (200k)+4.8 pp
Subsequent60.8% (160k)66.6% (160k)+5.8 pp

Following a resource adjustment, the subsequent run used different context limits at baseline and step 50; timeout settings also differed. But Its score change shows the same trend. A further Claude Code run increased from 62.0% to 66.8% at step 80.

These experiments provide additional observations of higher scores after training. I hope these replies can solve most of your concerns.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer SNNE

Thank you for the careful review and for recognizing the practical value of training through native coding-agent harnesses. We address your concerns below.

W1 / Q1. Training randomness and robustness

Additional OpenHands SDK training. With Qwen3.5-35B-A3B and GSPO, both the original experiment and additional experiment starting from the base model show improvements at step 50:

OpenHands SDK experimentBefore trainingStep 50Improvement
Original paper run64.0% (200k)68.8% (200k)+4.8 pp
Subsequent run60.8% (160k)66.6% (200k)+5.8 pp

We report step 50 in both experiments to compare a common training step without independently selecting peak checkpoints. Following a resource adjustment, the subsequent experiment used 160k at baseline and 200k at step 50.

Uncertainty analysis. In the subsequent OpenHands SDK experiment, performance improved by 6.02 percentage points. To assess uncertainty, we resampled the evaluation tasks 100,000 times, comparing both models on the same tasks in each resample. The resulting 95% confidence interval for the improvement was [2.61, 9.44] percentage points. This interval excludes zero, providing additional statistical support for the observed gain.

Additional Claude Code training. Additional Claude Code harness traning increased SWE-Bench-Verified from 62.0% to 66.8% at step 80.

Together, these additional experiments strengthen the evidence for policy improvement across training trajectories. We will emphasize this repeated improvement while treating the smaller cross-harness, mixed-harness, and GSPO-versus-SAO differences as descriptive comparisons, since matched independent repeats are not yet available. Checkpoint selection is addressed separately in Q2.

W1. Evidence from a second model

We thank the reviewer for highlighting the limited model coverage. In addition to Qwen3.5-35B-A3B in the main experiments, we provide supplementary results for Qwen3-30B-A3B-Instruct-2507, trained through the OpenHands SDK harness with SAO. On the SWE-bench Verified, the performance improves from 22.8% to 30.2%, a gain of 7.4 percentage points.

ModelHarnessRL objectivestep 0Best evaluated checkpoint (step 160)Gain
Qwen3-30B-A3B-Instruct-2507OpenHands SDKSAO22.8%30.2%+7.4 pp

This result extends the empirical evidence beyond the single model used in the main experiments and shows that Lego-RL supports effective native-harness training with different models. We will incorporate this supplementary result into the revision.

W2 / Q2. Checkpoint selection

We thank the reviewer for raising this point. We agree that clarifying the checkpoint-selection procedure and reporting results from later checkpoints would help readers assess the reported gains.

In our experiments, checkpoints were selected based on SWE-bench Verified performance and further evaluated on the other two benchmarks. We have not conducted a separate experiment using an independent validation set to select checkpoints. We therefore acknowledge that the reported Verified scores are not independent of checkpoint selection.

Our motivation for evaluating intermediate checkpoints was to account for possible performance degradation during RL training. Related observations have been reported in DeepSWE, which introduces compact filtering to mitigate reward collapse, and in the GSPO study, which discusses instability and collapse during prolonged GRPO training. For additional context, h1 selects checkpoints using validation performance, while NVIDIA’s SWE RL report presents its best-performing checkpoint alongside later checkpoints with lower scores.

To assess the sensitivity of our results to checkpoint choice, we additionally report results for the last evaluated checkpoints from the training curves presented in the paper:

HarnessInitial scoreBest evaluated checkpointLast evaluated checkpointLast minus initialBest minus last
OpenHands SDK64.0%70.4% (step 95)68.6% (step 125)+4.6 pp1.8 pp
Claude Code62.4%68.2% (step 110)66.4% (step 120)+4.0 pp1.8 pp
OpenCode57.2%66.6% (step 50)64.8% (step 120)+7.6 pp1.8 pp

All three harnesses retain improvements of 4.0–7.6 percentage points at the last evaluated checkpoints, which are 1.8 points below their respective peaks. Thus, the observed gains remain evident beyond the selected peak checkpoints.

The selected policies also improve on the other two benchmarks, neither of which was used for checkpoint selection, providing complementary evidence that the learned improvements transfer beyond SWE-bench Verified. We will explicitly document the selection protocol and report both best and last evaluated results.

AuthorsOpenReview ↗

Response to Reviewer SNNE part 2

W3 / Q3. Component ablations

Thank you for raising this point. These components primarily safeguard RL training correctness. A small change in final resolve rate after disabling a safeguard would not establish that the underlying training signals remain correct. We therefore evaluate their roles through targeted consistency checks and failure-mode diagnostics:

Routing replay.

Routing conditionSampled-token probability correlationMean absolute probability difference
Replay disabled0.99460.0062
Correctly aligned replay0.99930.00245
Misaligned replay0.75030.0954

Correct alignment reduces the mean absolute probability difference by approximately 60% relative to disabling replay. This directly tests probability consistency, not final performance.

Termination-aware filtering. An analysis of the available historical training logs identified 1,510 filtered batch-row occurrences in OpenHands (1,098 environment failures and 412 timeouts), 4,063 in OpenCode (2,778 environment failures and 1,285 timeouts), and 4,619 in Claude Code (51 environment failures and 4,568 timeouts). For Claude Code, 7.10% of batch rows were classified as environment failures or timeouts. These rows are excluded from task-group reward statistics and assigned zero reward-driven advantages, while ordinary task failures remain valid training outcomes.

Generation-time capture and history alignment. For the Claude Code trajectories pairs selected for known history mismatches, exact prefix matching rejected all mismatch trajectories, while identity-aware tool-call matching accepted all trajectories, identifying an avoidable source of reconstruction.

Reward-integrity safeguards. An example of astropy-14309 found an agent retrieving the reference fix through git show and receiving reward 1, demonstrating how answer leakage can inflate rewards.

These diagnostics address the components’ correctness roles. Full component-off retraining would quantify downstream performance sensitivity, but an individually measurable resolve-rate gain is not a prerequisite for maintaining consistent training signals and reward integrity.

Q4. Lossy reconstruction after history rewriting

Our supplementary OpenHands SDK audit found non-compaction history edits in only 88 of 29,448 training trajectories (0.30%), indicating that this potential reconstruction trigger was rare in the previous run and affect little on training quality.

W4 / Q5. Matched comparison with existing frameworks

Thank you for the suggestion. We began developing LEGO-RL in February 2026. At that time, we were not aware of a publicly available implementation offering a directly comparable training setup for the unmodified native coding-agent harnesses targeted by our work. Closely related systems became public during our development—for example, Polar was released in May 2026. A matched comparison with these systems would constitute an additional empirical study.

W5. Infrastructure contribution

LEGO-RL contributes the infrastructure needed for policy-gradient training through unmodified native coding-agent harnesses. The central technical challenge is meeting the requirements of RL optimization while preserving each harness’s native tools, history management, and execution behavior.

Addressing this requires coordinated mechanisms for token generation and probability capture, history alignment, token-aligned routing replay, termination-aware filtering, and execution-level diagnostics. These mechanisms span rollout, data processing, and optimization; they are not automatically provided by connecting an existing harness to an RL trainer.

We demonstrate this capability across three native harnesses, mixed-harness training, and different RL algorithms. We will sharpen the presentation of the mechanisms developed at the harness–trainer boundary and distinguish them from the established algorithms and execution components we reuse.

AuthorsOpenReview ↗

Response to Reviewer SNNE part 3

W5. Observability evidence

Thank you for raising this point. We will clarify the systematic analysis workflow behind the case studies. LEGO-RL links four levels of observation—Run → Step → Task → Trajectory where training behavior can be examined from aggregate trends down to the executions.

Run-level analysis. The dashboard tracks the state of training through reward and validation curves, batch reward distributions, and training logs. This establishes when performance improves, plateaus, or deteriorates and identifies the training windows that need closer inspection.

Step-level analysis. For each logged training step, users can inspect reward, KL, entropy, and termination-reason distributions, together with rollout–trainer consistency check. These measurements help distinguish whether a change in aggregate performance coincides with optimization failure, timeouts, or environment failures.

Task-level analysis. The task solve-rate grid tracks how individual tasks change across epochs, making improvements and regressions visible even when the aggregate reward increases. For example, the dashboard view shown in our presentation covers 2,694 tasks, with 1,135 showing improved solve rates and 471 showing decreased solve rates between the first and last epochs. This provides a task-wide view of learning behavior from which individual cases can be selected for inspection.

Trajectory-level analysis. From an individual task, users can inspect its recorded trajectories, including tool calls, patches, and termination reasons, or download trajectories for further analysis. This connects a task-level change to concrete agent behavior: what the agent attempted, which tools it invoked, what it changed, and how execution ended.

The case studies illustrate how these levels work together. In the collapsed run in Appendix C, the decline in reward was accompanied by a decrease in mean interaction length from 18.9 to 0.96 turns, comparing the first and last five logged steps. Inspecting a late trajectory then revealed that the agent emitted shell commands in code blocks without invoking tools. The trajectory evidence explains the behavioral failure behind the aggregate change.

We will add this Run → Step → Task → Trajectory workflow and its task-wide measurements in the revision, with the cases illustrating how users investigate patterns identified through routine monitoring. This supports systematic analysis of the recorded training process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.