acceptodds
Withdrawn from ICLR 2027

MOLT: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agentic reinforcement learning requires rapid experimentation with agents and learning algorithms, yet large policies and long, multimodal trajectories demand substantial distributed infrastructure. We present MOLT, a lightweight, PyTorch- and Hugging Face-native framework that brings these goals together through four contributions. MOLT combines direct loading of Hugging Face models with experimentally validated trillion-parameter scalability in approximately 9.2K lines of framework code. Unified OpenAI- and Anthropic-compatible interfaces integrate existing agents with automatic handling of context compaction. Fully asynchronous training overlaps agent rollouts and policy optimization, accommodating variable agent execution times. Distributed experience storage removes centralized rollout-memory bottlenecks for long, multimodal trajectories. We experimentally validate the complete RL training pipeline on a one-trillion-parameter policy and demonstrate sustained learning with a 30B mixture-of-experts agent, establishing MOLT as a lightweight foundation for large-scale agentic RL research.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Models, Reinforcement Learning, Infrastructure, Agent

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ibtWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents MOLT, a PyTorch- and Hugging Face-native framework for agentic RL. It is about 9.2K lines of code, built on Ray, vLLM, and NeMo AutoModel with FSDP2. It claims four contributions:

scaling to 1T-parameter policies OpenAI- and Anthropic-compatible endpoints that record token IDs and log-probs, and split a rollout into segments when the agent rewrites its context fully asynchronous training with partial rollouts and importance correction experience storage in Ray's object store instead of a central gather The evaluation includes a line-count comparison with OpenRLHF, verl and slime, and one end-to-end training step on a 1T policy. It also includes a 1,464-update run with a 30B MoE math-tool agent and a 2-step check on a 753B model that the trainer's token probabilities match the serving engine's. The rest is a throughput comparison with slime, plus the speed and memory effects of turning serving and memory features on and off for a 35B model.

Critical strengths

It targets a real need with simple interfaces. Agent RL needs frequent changes to agents, rewards and estimators, and large frameworks make that slow. In MOLT, an existing OpenAI or Anthropic agent can be trained by pointing its client at a session URL (§3), and estimators and losses are plain Python functions. When an agent compacts its context, MOLT splits the rollout into segments that share one reward. That's a practical fix for long-horizon agents.

It runs at realistic scale. A 30B MoE tool agent trains stably for 1,464 asynchronous updates, with mean accuracy rising from 46.4% to 66.4%. On a 753B model, the gap between rollout and trainer log-probs stays below 5×10−45\times10^{-4}. Throughput is within 8% of a Megatron-based stack, while MOLT stays PyTorch-native.

Critical weaknesses

The contribution is mostly integration of known pieces, and the paper doesn't show what it buys over existing frameworks.

The pieces already exist. These are established techniques:

recording tokens at a provider-compatible endpoint (Agent Lightning, Polar, ProRL Agent, all cited) asynchronous RL with partial rollouts and importance correction (AReaL, slime, Laminar) passing Ray object references instead of gathering payloads Splitting rollouts at context compaction is the main new piece, and it's small.

No advantage is shown. A framework paper can still matter if it's faster, scales further, or makes research changes easier. Where MOLT is compared, it's slightly slower: 92% of slime's throughput (461 vs. 502 tokens/GPU/s), on one model with 16 GPUs.

"Lightweight" rests on a self-defined line count. MOLT is 9.2K lines vs. 7.2K for OpenRLHF, which MOLT came from. The count appears to exclude NeMo AutoModel, which MOLT builds on. It measures scope, not simplicity: verl is larger partly because it supports more backends.

The evaluation doesn't test the paper's own claims.

1T scalability is one sentence. §5.2 says a training update ran on a 1T policy, and gives no model, hardware, step time, memory or throughput. That shows the code path runs, not that it scales. Two of the four contributions have no experiment. No experiment uses an agent that compacts its context. There's also no memory or throughput comparison with and without the object store on the multimodal workloads that motivate it. Async training isn't ablated. There's no comparison of synchronous vs. asynchronous training. There's also no test of how policy lag (about 8 updates in the 30B run) affects learning. The learning run has no reference. Without the same recipe run in another framework, the 30B curve shows that learning happens, not that it matches other frameworks. The 753B check covers only 2 steps. Table 5 measures engine features. The 5.14× speedup from multi-token prediction and the optimizer-offload tradeoff come from vLLM and FSDP2, not from MOLT's design. In short, MOLT may be a convenient tool, but the paper shows that it runs, not that its design helps.

Questions for authors

Clarifications

For the 1T run: which model, how many GPUs, and what step time, memory and throughput? How many updates ran? What do asynchronous training and object-store experience storage buy? For example, what are step time and controller memory with each turned off, on a multimodal workload such as OSWorld? Will MOLT be open-sourced? Does the same 30B recipe reach similar accuracy in OpenRLHF or slime? Suggestions

Add a scaling curve (throughput vs. GPU count and model size) against verl and slime. Show one research change end to end, such as adding a new estimator, and how many lines it takes in MOLT vs. another framework.

Reviewer_rkoEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present MOLT, a ~9.2K-line PyTorch- and Hugging Face-native framework for asynchronous agentic RL. It loads HF models into an FSDP2-based actor with tensor, expert, and context parallelism, and reports end-to-end execution on a 1T-parameter policy. Agents connect through OpenAI- and Anthropic-compatible endpoints that capture tokens and log-probabilities, with trajectories split into segments when the agent compacts its context. Training is fully asynchronous, with importance correction for stale rollouts, and trajectory payloads are distributed through Ray's object store. Experiments cover code footprint, a ~1,500-update Qwen3-30B-A3B run, log-probability agreement on a 753B model, throughput against slime, and serving and memory measurements on a 35B MoE.

Critical strengths
  • Useful niche. A PyTorch/Hugging Face-native framework that loads HF checkpoints directly and composes FSDP2 with tensor, expert, and context parallelism is a reasonable alternative to Megatron-based stacks for researchers who want to modify algorithms easily.
  • Context compaction is considered. Splitting rollouts into segments when an agent rewrites its history, while sharing the terminal reward, is a sensible design for a problem many frameworks don’t address, though it isn’t evaluated experimentally.
  • Stable long training run. The ~1,500-update Qwen3-30B-A3B run shows the async pipeline trains stably in practice, albeit with a single seed and no baseline.
Critical weaknesses
  • The headline claim is unsupported. “1T scale” is backed by one sentence saying it ran. There are no numbers, no hardware details, and no throughput.
  • Lower throughput than the main baseline: MOLT reaches ~92% of slime’s tokens/GPU/s (Table 4).
  • MOLT has about 9.2K lines, OpenRLHF about 7.2K, verl about 62K, and slime about 25K. Line count is not really a meaningful metric to optimize on its own.
Questions for authors
  • What benefits do MOLT’s ~9.2K lines give over slime’s ~25K? Is there evidence that this actually translates into practical benefits?
  • What is the cause of the slower throughput relative to slime?
  • 1T execution: Which 1T model was used, on what hardware and parallelism configuration, and for how many training steps? Can you report step time, throughput, and rollout–trainer log-probability agreement at this scale, as you did for the 753B GLM-5.2 checkpoint?
Reviewer_AnwdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a new framework for asynchronous agentic RL training, supporting upto 1T parameters models including rollouts, grading, policy actor and env state handling. They explain in detail the compatibility with OpenAI and Claude's responseAPI, and present the framework details like distributed rollout storage and fully async training.

The comparisons have been made using Qwen3-30B-A3B and GLM-5.2. However neither of these are 1T parameter models as claimed.

Critical strengths

The paper proposes a new training framework for agentic RL with easily compatible interface with most popular training models. Adequate explanations have been presented into each module of the framework.

Critical weaknesses
  1. The biggest issue with the framework is there is no mention that it will be open-sourced or released to the public. Without this release, the contributions of the paper to the broader scientific and academic community is limited.

  2. Although each module of the paper is explained in great detail, it is not sufficiently motivated as to why we need yet another training framework, and what exact gaps in current landscape is this pipeline targeted to solve?

  3. The whole paper has limited technical contributions, and might not be a right interest to the targeted audience in ICLR (where this manuscript is currently submitted to).

Questions for authors
  1. Why is RAY backend chosen for PyTorch/FSDP2 actor coordination, what particular advantages does it have. are other frameworks experimented with as well?

  2. The paper claims multiple places about a trillion-parameter support, but the largest model experimented with is GLM 5.2, which is sub-trillion model. The support for larger models is not sufficiently explored.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.