- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper describes SWAP, a method for improving the efficiency of on-policy massively parallel training in multi-task settings. Following the observation that adding more parallel environments has diminishing returns in single-task scenarios, even when diversity is added manually, SWAP leverages multi-task training to obtain naturally diverse policies and make better use of the available environments. To alleviate the high variance of using different policies at every time-step, SWAP follows the recently-popular chunked action paradigm.
Critical strengths
The idea of using excess environment capacity to train multi-task policies is intriguing - in effect, instead of lowering the parallelism rate and lengthening training, SWAP uses the available resources more effectively and therefore gains training efficiency. The fact that sample efficiency is gained as well is an additional strength.
The paper is clearly written and explains its motivations and methods well.
Critical weaknesses
The central weakness I see in this paper is the low diversity in the experiment section. While the experiments do support most of the claims made in the paper, it would be beneficial to show experimentation on additional environment domains.
In addition, I would have liked to see more discussion of the effects of bootstrapping with out-of-distribution actions - while these can assist with exploration, this is effective because the MT50 tasks are quite similar to each other. In other domains this behavior could be limiting and possibly slow convergence. This is another reason why additional experimentation could strengthen the paper.
Other concerns are outlined in the questions section below.
Questions for authors
- Why does SWAP underperform at low T? The authors mention this is “…most likely due to the relative difficulty of learning a Q function versus a value function”. Why is this more pronounced in short horizons?
- Rollouts seem to be of length T≤64 throughout the paper, but it seems like the episode is longer (Fig. 6, Appendix E). Could tha authors provide a clearer explanation of where the rollout length is used in the method?
- Sec 4.2 mentions the estimator for step TD(\lambda) is biased - how is that rectified?
- What is the architecture of the chunked policy networks? Do they predict the chunks as a single block or autoregressively?
- “The inference overhead incurred at chunk boundaries is minimal since it scales as O(K/N) forward passes” (Line 320): do you mean O(KT/N)? One (or actually two) forward pass per each of the K tasks at each T/N decision boundary?
- In the reported environments, what is the value of K (number of environments per task)? I may be missing this but can’t seem to find it in the experiments section.
- “Increasing the ratio \rho improves performance monotonically up until all environments are switching” - I don’t think this ablation study fully shows this as described, to test this claim I’d expect 0.9, 0.999 rates as well (or maybe rephrasing the description to match the ablation).
- In figure 5, when comparing SWAP with “own-task Q”, it’s interesting to see the difference between shorter-horizon rollouts and longer ones. How do the authors explain this difference?
3 comments
Cont.
- SWAP vs own-task Q
This ablation isolates the benefit of cross-task proposals. At where vanilla chunking plateaus at 77.7% (Table 7), selecting among 50 own-task chunks barely moved success to 78.5% (Table 10), whereas cross-task proposals (SWAP) reached 88.9% (+10.4 pts; Table 10). At and , mean gains were smaller (0.4 and 0.7 pts; Table 10) because high baseline success rates mean -maximization is merely refining mostly high return action proposals rather than discovering missing behaviors, making cross-task policy switching less necessary. Furthermore, SWAP reduced 95% confidence interval widths by over , ensuring more stable convergence (Table 10)
Please also see point 2(c) in the response to Reviewer QLwF
Thank you for taking the time to write a detailed rebuttal and responding to my questions. Adding these explanations and clarifications where appropriate in the paper would strengthen it considerably. In particular, the discussion on the limitations of the method (effective in domains with shared morphologies and action spaces) is significant.
As most of my concerns have been resolved, I will maintain my positive evaluation of the paper.
SWAP is designed for massively parallel multi-task settings where tasks share common structure and motor primitives, which MTBench (MT50) directly provides. In benchmarks with completely dissimilar tasks, disjoint state/action spaces, or different embodiments, direct action-level policy switching would naturally offer little benefit without a shared latent action space, and representation-learning methods would be more appropriate.
SWAP always bootstraps using fresh chunks sampled from the target task's own policy A′∼πθ(⋅∣s,zτ), and never bootstraps with cross-task donor chunks. In Step TD(λ), querying Qϕ on freshly sampled own-task chunks at mid-chunk states does introduce replanning mismatch, which Option TD(λ) eliminates by bootstrapping only at chunk boundaries.
We clarify how SWAP handles potential out-of-distribution (OOD) actions in both step target bootstrapping (1) and cross-task selection (2):
When we matched the bootstrap policy on switched rollouts to the switching behavior (QPS) in Step mode, SWAP at T=8 rose to match PPO and CPO, confirming that continuation mismatch at short horizons is a primary driver. (Figure 10/Table 14). Intuitively, this makes sense as at such a short horizon, the 1 and 2 step returns truncate after only 1 or 2 chunks. This heavily weights bootstrap error in the critic target.
We roll out E environments for T timesteps to create a batch of E×T rows, where policy evaluation and policy improvement steps use that data. An unfinished episode continues into the next rollout. If an episode ends during collection, that environment resets and continues filling the same batch i.e environments do not have to be synced at the same timestep within an episode.
At short rollout lengths, we accept the mid-chunk replanning bias in step TD(λ) in exchange of retaining enough critic targets (supervision) given the number of targets decreases as the rollout window length decreases. Option targets avoid this bias by bootstrapping at chunk boundaries.
Even with replanning handled this way, there is a separate question of continuation mismatch. As we report in our response to Reviewer QLwF (Point 1(a)), we tested TD(0), which uses one executed chunk and then bootstraps from the task’s own policy. This completely removes the continuation-policy mismatch, but this performed substanially worse across the rollout lengths tested. We therefore retain multi-chunk returns to propagate observed rewards farther before relying on the critic, and acknowledge the resulting bias with respect to the task-policy value.
The actor is 1 block of a SimbaV2 MLP. It predicts the parameters of a tanh-squashed Gaussian over all N actions. We sample the chunk once and execute its actions open-loop.
The reviewer is correct that there are O(KT/N) candidate evaluations per switching environment over a T-step rollout with T/N chunk boundaries. Our use of “forward passes” in Line 320 was imprecise. We have corrected the complexity in the revised manuscript.
At each chunk boundary, policy-switching uses one batched actor call to propose one chunk from each of the K task policies and one batched critic call to score those K chunks under the environment’s task. This gives O(K) candidate evaluations per switching environment per boundary, i.e., O(KT/N) evaluations over a rollout of T/N complete chunks.
We divide the 24,576 environments as evenly as possible across K=50 tasks. For MT50, 26 tasks receive 492 environments each and the remaining 24 receive 491 each.
We agree and will remove “monotonically.” Among the partial-switching fractions we tested, larger fractions generally gave higher task success.