- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper presents MOLT, a PyTorch- and Hugging Face-native framework for agentic RL. It is about 9.2K lines of code, built on Ray, vLLM, and NeMo AutoModel with FSDP2. It claims four contributions:
scaling to 1T-parameter policies OpenAI- and Anthropic-compatible endpoints that record token IDs and log-probs, and split a rollout into segments when the agent rewrites its context fully asynchronous training with partial rollouts and importance correction experience storage in Ray's object store instead of a central gather The evaluation includes a line-count comparison with OpenRLHF, verl and slime, and one end-to-end training step on a 1T policy. It also includes a 1,464-update run with a 30B MoE math-tool agent and a 2-step check on a 753B model that the trainer's token probabilities match the serving engine's. The rest is a throughput comparison with slime, plus the speed and memory effects of turning serving and memory features on and off for a 35B model.
Critical strengths
It targets a real need with simple interfaces. Agent RL needs frequent changes to agents, rewards and estimators, and large frameworks make that slow. In MOLT, an existing OpenAI or Anthropic agent can be trained by pointing its client at a session URL (§3), and estimators and losses are plain Python functions. When an agent compacts its context, MOLT splits the rollout into segments that share one reward. That's a practical fix for long-horizon agents.
It runs at realistic scale. A 30B MoE tool agent trains stably for 1,464 asynchronous updates, with mean accuracy rising from 46.4% to 66.4%. On a 753B model, the gap between rollout and trainer log-probs stays below . Throughput is within 8% of a Megatron-based stack, while MOLT stays PyTorch-native.
Critical weaknesses
The contribution is mostly integration of known pieces, and the paper doesn't show what it buys over existing frameworks.
The pieces already exist. These are established techniques:
recording tokens at a provider-compatible endpoint (Agent Lightning, Polar, ProRL Agent, all cited) asynchronous RL with partial rollouts and importance correction (AReaL, slime, Laminar) passing Ray object references instead of gathering payloads Splitting rollouts at context compaction is the main new piece, and it's small.
No advantage is shown. A framework paper can still matter if it's faster, scales further, or makes research changes easier. Where MOLT is compared, it's slightly slower: 92% of slime's throughput (461 vs. 502 tokens/GPU/s), on one model with 16 GPUs.
"Lightweight" rests on a self-defined line count. MOLT is 9.2K lines vs. 7.2K for OpenRLHF, which MOLT came from. The count appears to exclude NeMo AutoModel, which MOLT builds on. It measures scope, not simplicity: verl is larger partly because it supports more backends.
The evaluation doesn't test the paper's own claims.
1T scalability is one sentence. §5.2 says a training update ran on a 1T policy, and gives no model, hardware, step time, memory or throughput. That shows the code path runs, not that it scales. Two of the four contributions have no experiment. No experiment uses an agent that compacts its context. There's also no memory or throughput comparison with and without the object store on the multimodal workloads that motivate it. Async training isn't ablated. There's no comparison of synchronous vs. asynchronous training. There's also no test of how policy lag (about 8 updates in the 30B run) affects learning. The learning run has no reference. Without the same recipe run in another framework, the 30B curve shows that learning happens, not that it matches other frameworks. The 753B check covers only 2 steps. Table 5 measures engine features. The 5.14× speedup from multi-token prediction and the optimizer-offload tradeoff come from vLLM and FSDP2, not from MOLT's design. In short, MOLT may be a convenient tool, but the paper shows that it runs, not that its design helps.
Questions for authors
Clarifications
For the 1T run: which model, how many GPUs, and what step time, memory and throughput? How many updates ran? What do asynchronous training and object-store experience storage buy? For example, what are step time and controller memory with each turned off, on a multimodal workload such as OSWorld? Will MOLT be open-sourced? Does the same 30B recipe reach similar accuracy in OpenRLHF or slime? Suggestions
Add a scaling curve (throughput vs. GPU count and model size) against verl and slime. Show one research change end to end, such as adding a new estimator, and how many lines it takes in MOLT vs. another framework.