acceptodds
Under review as a conference paper at ICLR 2027

SWAP: Switched Action Chunking Policies with Massively Parallelized Multi-Task Reinforcement Learning

Abstract

In massively parallel on-policy Reinforcement Learning (RL), performance saturates beyond a certain environment count because additional environments yield redundant trajectories due to executing similar actions sampled around the policy mean. Standard entropy regularization fails to address this, leading prior single-task methods to artificially partition environments across follower policies to force behavioral diversity with separate entropy bonuses. We propose that scaling *multi-task* reinforcement learning in this regime is unique in that policies inherently provide behavioral diversity by virtue of solving multiple tasks. The key insight is that agents can execute cross-task or *switched* actions during any task's rollout, using the otherwise excess rollout capacity of massively parallel simulation. However, utilizing this diversity to induce coherent exploration requires temporal commitment through action chunking. During a rollout, in a designated subset of parallel environments per task, policies replan by sampling action chunks across all task policies and executing the candidate that maximizes the value under the true task. By relying on pathwise policy gradients that differentiate directly through the critic, the policy absorbs this off-policy exploration without importance sampling correction. We call this framework Switched Action Chunking Policies (SWAP), which only incurs inference overhead during training that scales linearly with the number of tasks. In Meta-World, SWAP outperforms recent methods addressing performance saturation in the massively parallelized regime.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, multi-task, action chunking, massive parallelization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_YCeFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper describes SWAP, a method for improving the efficiency of on-policy massively parallel training in multi-task settings. Following the observation that adding more parallel environments has diminishing returns in single-task scenarios, even when diversity is added manually, SWAP leverages multi-task training to obtain naturally diverse policies and make better use of the available environments. To alleviate the high variance of using different policies at every time-step, SWAP follows the recently-popular chunked action paradigm.

Critical strengths

The idea of using excess environment capacity to train multi-task policies is intriguing - in effect, instead of lowering the parallelism rate and lengthening training, SWAP uses the available resources more effectively and therefore gains training efficiency. The fact that sample efficiency is gained as well is an additional strength.

The paper is clearly written and explains its motivations and methods well.

Critical weaknesses

The central weakness I see in this paper is the low diversity in the experiment section. While the experiments do support most of the claims made in the paper, it would be beneficial to show experimentation on additional environment domains.

In addition, I would have liked to see more discussion of the effects of bootstrapping with out-of-distribution actions - while these can assist with exploration, this is effective because the MT50 tasks are quite similar to each other. In other domains this behavior could be limiting and possibly slow convergence. This is another reason why additional experimentation could strengthen the paper.

Other concerns are outlined in the questions section below.

Questions for authors
  1. Why does SWAP underperform at low T? The authors mention this is “…most likely due to the relative difficulty of learning a Q function versus a value function”. Why is this more pronounced in short horizons?
  2. Rollouts seem to be of length T≤64 throughout the paper, but it seems like the episode is longer (Fig. 6, Appendix E). Could tha authors provide a clearer explanation of where the rollout length is used in the method?
  3. Sec 4.2 mentions the estimator for step TD(\lambda) is biased - how is that rectified?
  4. What is the architecture of the chunked policy networks? Do they predict the chunks as a single block or autoregressively?
  5. “The inference overhead incurred at chunk boundaries is minimal since it scales as O(K/N) forward passes” (Line 320): do you mean O(KT/N)? One (or actually two) forward pass per each of the K tasks at each T/N decision boundary?
  6. In the reported environments, what is the value of K (number of environments per task)? I may be missing this but can’t seem to find it in the experiments section.
  7. “Increasing the ratio \rho improves performance monotonically up until all environments are switching” - I don’t think this ablation study fully shows this as described, to test this claim I’d expect 0.9, 0.999 rates as well (or maybe rephrasing the description to match the ablation).
  8. In figure 5, when comparing SWAP with “own-task Q”, it’s interesting to see the difference between shorter-horizon rollouts and longer ones. How do the authors explain this difference?
3 comments

1(a). Experimental Diversity

SWAP is designed for massively parallel multi-task settings where tasks share common structure and motor primitives, which MTBench (MT50) directly provides. In benchmarks with completely dissimilar tasks, disjoint state/action spaces, or different embodiments, direct action-level policy switching would naturally offer little benefit without a shared latent action space, and representation-learning methods would be more appropriate.

1(b). Bootstrapping with OOD actions

SWAP always bootstraps using fresh chunks sampled from the target task's own policy A′∼πθ(⋅∣s,zτ)A' \sim \pi_\theta(\cdot \mid s, z_\tau), and never bootstraps with cross-task donor chunks. In Step TD(λ)\text{TD}(\lambda), querying QϕQ_\phi on freshly sampled own-task chunks at mid-chunk states does introduce replanning mismatch, which Option TD(λ)\text{TD}(\lambda) eliminates by bootstrapping only at chunk boundaries.

We clarify how SWAP handles potential out-of-distribution (OOD) actions in both step target bootstrapping (1) and cross-task selection (2):

  1. We query QϕQ_\phi on freshly sampled own-task chunks at mid-chunk states in Step TD(λ)\text{TD}(\lambda), which is the replanning bias. In settings where tasks have disjoint state/action spaces or different embodiments, direct action-level policy switching would naturally be less applicable without a shared latent action space.
  2. Unlike offline RL, if the critic erroneously assigns high Q value to a switched action, the policy will simply try it out and recieve low reward. The critic will see this and correct itself.

1(c) Why does SWAP underperform at low T?

When we matched the bootstrap policy on switched rollouts to the switching behavior (QPSQ^{\text{PS}}) in Step mode, SWAP at T=8T=8 rose to match PPO and CPO, confirming that continuation mismatch at short horizons is a primary driver. (Figure 10/Table 14). Intuitively, this makes sense as at such a short horizon, the 1 and 2 step returns truncate after only 1 or 2 chunks. This heavily weights bootstrap error in the critic target.

  1. Rollout and Episode Length

We roll out EE environments for TT timesteps to create a batch of E×TE \times T rows, where policy evaluation and policy improvement steps use that data. An unfinished episode continues into the next rollout. If an episode ends during collection, that environment resets and continues filling the same batch i.e environments do not have to be synced at the same timestep within an episode.

  1. Step TD(\lambda) bias

At short rollout lengths, we accept the mid-chunk replanning bias in step TD(λ)TD(\lambda) in exchange of retaining enough critic targets (supervision) given the number of targets decreases as the rollout window length decreases. Option targets avoid this bias by bootstrapping at chunk boundaries.

Even with replanning handled this way, there is a separate question of continuation mismatch. As we report in our response to Reviewer QLwF (Point 1(a)), we tested TD(0)TD(0), which uses one executed chunk and then bootstraps from the task’s own policy. This completely removes the continuation-policy mismatch, but this performed substanially worse across the rollout lengths tested. We therefore retain multi-chunk returns to propagate observed rewards farther before relying on the critic, and acknowledge the resulting bias with respect to the task-policy value.

  1. Actor Parameterization

The actor is 1 block of a SimbaV2 MLP. It predicts the parameters of a tanh-squashed Gaussian over all NN actions. We sample the chunk once and execute its actions open-loop.

  1. Inference overhead

The reviewer is correct that there are O(KT/N)O(KT/N) candidate evaluations per switching environment over a TT-step rollout with T/NT/N chunk boundaries. Our use of “forward passes” in Line 320 was imprecise. We have corrected the complexity in the revised manuscript.

At each chunk boundary, policy-switching uses one batched actor call to propose one chunk from each of the KK task policies and one batched critic call to score those KK chunks under the environment’s task. This gives O(K)O(K) candidate evaluations per switching environment per boundary, i.e., O(KT/N)O(KT/N) evaluations over a rollout of T/NT/N complete chunks.

  1. Number of environments per task

We divide the 24,576 environments as evenly as possible across K=50K=50 tasks. For MT50, 26 tasks receive 492 environments each and the remaining 24 receive 491 each.

  1. “Increasing the ratio ρ\rho improves performance monotonically up until all environments are switching”

We agree and will remove “monotonically.” Among the partial-switching fractions we tested, larger fractions generally gave higher task success.

  1. SWAP vs own-task Q

This ablation isolates the benefit of cross-task proposals. At T=16T=16 where vanilla chunking plateaus at 77.7% (Table 7), selecting among 50 own-task chunks barely moved success to 78.5% (Table 10), whereas cross-task proposals (SWAP) reached 88.9% (+10.4 pts; Table 10). At T=32T=32 and T=64T=64, mean gains were smaller (0.4 and 0.7 pts; Table 10) because high baseline success rates mean QQ-maximization is merely refining mostly high return action proposals rather than discovering missing behaviors, making cross-task policy switching less necessary. Furthermore, SWAP reduced 95% confidence interval widths by over 4×4\times, ensuring more stable convergence (Table 10)

Please also see point 2(c) in the response to Reviewer QLwF

Thank you for taking the time to write a detailed rebuttal and responding to my questions. Adding these explanations and clarifications where appropriate in the paper would strengthen it considerably. In particular, the discussion on the limitations of the method (effective in domains with shared morphologies and action spaces) is significant.

As most of my concerns have been resolved, I will maintain my positive evaluation of the paper.

Reviewer_XLpeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses performance saturation in massively parallel multi-task reinforcement learning by proposing SWAP, which combines cross-task policy switching with action chunking. SWAP selects candidate action chunks based on their Q-values to promote more diverse and temporally coherent exploration. Experiments on the 50 robotic manipulation tasks in MTBench show that SWAP outperforms existing baselines in most settings, achieving higher task success rates and greater sample efficiency.

Critical strengths
  1. The problem is clearly formulated, and the motivation is well justified and compelling. The paper attributes the diminishing performance gains from increasing the number of parallel environments to insufficient behavioral diversity: a high proportion of redundant trajectories reduces learning efficiency. I find this motivation and the accompanying analysis valuable.
  2. The proposed method substantially outperforms existing methods, and the paper provides extensive experiments analyzing the effects of its hyperparameters.
Critical weaknesses
  1. The experiments do not directly demonstrate that SWAP alleviates performance saturation in massively parallel training. They fix the number of parallel environments at 24,576 while varying the rollout horizon and other parameters. These results show that SWAP achieves higher task success rates and better learning efficiency under a fixed large-scale configuration, but they do not establish whether SWAP is less susceptible to performance saturation than other methods as the number of parallel environments increases.
  2. Stronger baselines from the same class of methods are missing. The main experiments do not include comparisons with the more closely related cross-task behavior-sharing methods discussed in the related work.
  3. The paper argues that insufficient behavioral diversity is a major reason why increasing the number of parallel environments does not lead to continued performance improvements in massively parallel RL. However, the experiments do not provide direct evidence that SWAP improves behavioral diversity. Although the appendix analyzes how frequently different task policies are used and how behaviors are borrowed across tasks, these results do not directly establish that SWAP produces more diverse exploratory trajectories or substantially reduces redundant data.
Questions for authors
  1. Could the authors include a sweep over the number of parallel environments, plotting the performance of SWAP and the strongest baseline as this number increases?
  2. Could the authors add a comparison with at least one cross-task behavior-sharing method?
  3. Could the authors provide more detailed evidence of increased behavioral diversity and reduced trajectory redundancy, preferably through quantitative analysis?
1 comment
  1. The experiments do not directly demonstrate that SWAP alleviates performance saturation in massively parallel training.

By performance saturation, we follow the definition in prior work (e.g., SAPG). That is, beyond some environment count, adding more parallel environments yields little further improvement in PPO’s task success given indefinite training. Then, a comparison of task success of a proposed method using any environment count from the plateaued region answers if that method (e.g SWAP) resolves saturation. SAPG uses this reasoning in its Figure 2, where it shows PPO leveling off across batch sizes, which is exactly why we included results at different rollout horizons.

Continuous performance improvement as the number of environments scales is a strictly stronger requirement that we did not claim in the paper.

Nevertheless, we ran the requested environment-count sweep for PPO and SWAP in Figure 6. At the higher counts tested, task success plateaued for both methods, with SWAP reaching a higher plateau than PPO. In general, we believe expecting continuous performance improvement with respect to environment count is beyond the scope of the paper.

  1. Stronger baselines from the same class of methods are missing.

To address the reviewer's concern regarding stronger baselines and cross-task behavior-sharing methods, we include:

  1. Closed-loop policy commitment (CTPG-like): Instead of executing an action chunk, we follow CTPG [1] in committing to a policy for NN closed-loop steps, where each step samples a new action from that committed policy. In CTPG, a separate network is used to guide the selection of such a policy. To avoid deviating so much from our method and reduce the overhead of training an extra network, we relied on our criterion of maximizing QQ for selecting which policy to execute closed-loop. Please see the results of this method in Figure 7.

  2. an exploration method, RLE [2], that is well-suited to massive parallelization, built on top of PPO, and was also applied to continuous control robotic tasks (see Fig. 1).

PPO-RLE performs similarly to PPO across horizons (74.9%74.9\% / 81.2%81.2\% / 87.7%87.7\% / 88.6%88.6\% vs. PPO's 78.5%78.5\% / 82.0%82.0\% / 86.2%86.2\% / 91.1%91.1\%) (Figure 1). Commit (N=4N = 4) strongly outperforms PPO, CPO, and SAPG at T≥16T \ge 16 (87.2%87.2\% / 95.1%95.1\% / 94.9%94.9\%), confirming the benefit of temporally extended, cross-task behavior sharing (Table 12). At the same time, SWAP (N=4N = 4) outperforms matched Commit (N=4N = 4) across all four horizons (73.7%73.7\% vs. 71.4%71.4\% at T=8T = 8; 91.0%91.0\% vs. 87.2%87.2\% at T=16T = 16; 96.0%96.0\% vs. 95.1%95.1\% at T=32T = 32; 98.2%98.2\% vs. 94.9%94.9\% at T=64T = 64), as well as every other tested commitment length N∈{2,8,16}N \in \{2, 8, 16\}, with markedly tighter confidence intervals (especially at T=16T = 16 and T=64T = 64).

  1. Trajectory Diversity

Our appendix shows which donors are selected, but different donors can still visit similar states. To compare the diversity of states visited by SWAP and PPO, we adapt the reconstruction analyses in Figures 7 and 8 from SAPG. Specifically, we run this on the top-20 tasks where vanilla PPO has the biggest delta in success rate with SWAP. Then, if SWAP visits a more diverse state distribution, its reconstruction error should be higher. For each task, we select checkpoints every 50M frames from the 50M to 450M frames, draw 100k rollout states from each checkpoint, and aggregate those states separately for SWAP and PPO.

We fit PCA and MLP reconstructions to a training split of each aggregated batch for each task. When most state variation lies along a few directions, a small number of principal components suffices for accurate reconstruction. Similarly, an autoencoding MLP tests whether the same states can be compressed using a nonlinear representation, where a higher reconstruction error signals more diverse data.

For each task and method, we compute the mean squared L2 reconstruction error summed across all D = 39 state dimensions using an 80/20 train/test split, and average across the 20/50 tasks.

In Figures 14 and 15, we see that the average test MSE decreases slower for SWAP than it does for PPO as the number of components or hidden layer width increases.

[1] Cross-Task Policy Guidance (Multi-Task RL): He, J., Li, K., Zang, Y., Fu, H., Fu, Q., Xing, J., & Cheng, J. (2025). Efficient multi-task reinforcement learning with cross-task policy guidance. 38th Conference on Neural Information Processing Systems (NeurIPS 2024). https://doi.org/10.48550/arxiv.2507.06615

[2] Mahankali, S., Hong, Z.-W., Sekhari, A., Rakhlin, A., & Agrawal, P. (2024). Random latent exploration for deep reinforcement learning. arXiv. https://doi.org/10.48550/arxiv.2407.13755

Comments on the paper

7 comments

1(a). The critic targets and their interpretation need a clearer justification.

We thank the reviewer for pointing out the imprecision of stating that the option target incurs "no bias." Removing the replanning timing mismatch is separate from removing the continuation policy mismatch.

Both step and option targets use rewards from the rollouts in policy-switched environments and bootstrap under the target task policy. As the reviewer notes, Qπ(st,At,zτ)Q^\pi(s_t,A_t,z_\tau) conditioning on AtA_t accounts for the reward in the option target, but subsequent rewards depend on intermediate actions selected by the PS-induced behavior policy, rather than the target policy πθ(⋅∣s,zτ)\pi_\theta(\cdot\mid s,z_\tau) indicated in Eq. (6), creating an off-policy target.

We draw the analogy that this same mismatch occurs with the use of uncorrected, n-step returns in off-policy, replay-based methods, where intermediate behavior-policy rewards are included before bootstrapping under the target policy. Our option target has this same structure at the chunk level.

The distinction in the bias incurred by both estimators lies in the replanning alignment. We will revise the text to note that

  1. Option TD(λ)TD(\lambda) targets incur no mid-chunk replanning bias, while acknowledging that for λ>0\lambda > 0 and ρ>0\rho > 0, multi-chunk targets for PS rollouts intentionally retain bias incurred by continuation-policy mismatch.
  2. Step TD(λ)TD(\lambda) targets additionally incur mid-chunk replanning bias where fresh-chunk bootstraps are drawn at states where the rollout remains committed to an existing chunk.

Removing Continuation Mismatch via 1-Chunk TD(0)\text{TD}(0)

To test the reviewer's observation directly, we evaluated the 1-chunk TD(0)\text{TD}(0) target, which accumulates only the discounted reward within the single recorded chunk AtA_t (which Qϕ(st,At,zτ)Q_\phi(s_t, A_t, z_\tau) is explicitly conditioned on) followed immediately by an own-task policy bootstrap at st+Ns_{t+N}:

GtTD(0)=Rτ(st,At)+γNV^(st+N,zτ)G_t^{\text{TD}(0)} = R_\tau(s_t, A_t) + \gamma^N \hat{V}(s_{t+N}, z_\tau)

This retains using policy switching during data collection while avoiding the inclusion of subsequent PS-selected chunks in the reward prefix. In Figure 11, we see it performs substantially worse than using the multi-chunk estimator across all horizons.

Therefore, our decision to use uncorrected returns with this continuation-policy mismatch is confirmed by empirical results as well as the analogy to nn-step returns.


Matching the Bootstrap's Policy to the PS Behavior Policy (QPSQ^{\text{PS}}):

Separately, the reviewer's point of continuation policy mismatch prompted us to try to match the continuation policy to that of the PS-induced behavior policy. Specifically, let policy switched environment rollouts bootstrap using the Q maximizer of all candidate action chunks proposed from each task policy, while non-policy switched environments rollouts retain task-policy bootstrapping. Both rollout types use the same critic network.

Critically, Q must now represent QPSQ^{\text{PS}}, the expected discounted return when continuing under the PS-induced policy as well as QtaskQ^{\text{task}}, the expected discounted return when continuing under the task's own policy. While theoretically problematic to model this without any conditioning information, in practice, we approximate both with a single, shared network.

Compared with self-policy bootstrapping, this variant improves step mode performance at T=8T=8 to be competitive with PPO/CPO, while performance remains within confidence intervals at the other tested rollout horizons. Please see Figure 10.

This does not remove step mode's mid chunk timing mismatch.

1(b) Clarifying the Step Target

In Section 4.2 (line 272), the continuation term in Rτ(st,At)+γNGt+NλR_\tau(s_t, A_t) + \gamma^N G_{t+N}^\lambda was indeed incorrect and should instead use the continuation term from the per-step TD(λ)\mathrm{TD}(\lambda) recursion:

Gtstep,N=Rτ(st,At)+γN−1[Gt+N−1λ−rτ(st+N−1,at+N−1)].G_t^{\mathrm{step},N} = R_\tau(s_t, A_t) + \gamma^{N-1}\left[G_{t+N-1}^\lambda - r_\tau(s_{t+N-1}, a_{t+N-1})\right].

We will correct the main text and include the explanation below to derive the step target and its N=1N=1 reduction.

First, we run REPPO's per-step TD(λ)\mathrm{TD}(\lambda) backward recursion over the rollout. Let hk+1=−αlog⁡πθ,0(ak+1,0′∣sk+1,zτ)h_{k+1} = -\alpha \log \pi_{\theta,0}(a'_{k+1,0} \mid s_{k+1}, z_\tau) be the entropy bonus for the first action of the chunk:

Gkλ=rτ(sk,ak)+γ[hk+1+(1−λ)V^(sk+1,zτ)+λGk+1λ⏟Vλ(sk+1,zτ)],G_k^\lambda = r_\tau(s_k, a_k) + \gamma \left[\underbrace{h_{k+1} + (1-\lambda)\hat{V}(s_{k+1}, z_\tau) + \lambda G_{k+1}^\lambda}_{V^\lambda(s_{k+1}, z_\tau)}\right],

where V^(sk+1,zτ)=Qϕ(sk+1,A′,zτ)\hat{V}(s_{k+1}, z_\tau) = Q_\phi(s_{k+1}, A', z_\tau) with A′∼πθ(⋅∣sk+1,zτ)A' \sim \pi_\theta(\cdot \mid s_{k+1}, z_\tau) is calculated as a single-sample estimate from QϕQ_\phi.

This gives the array of per-step targets G0λ,G1λ,…G_0^\lambda, G_1^\lambda, \ldots, where the bracketed continuation term is:

Vλ(sk+1,zτ):=hk+1+(1−λ)V^(sk+1,zτ)+λGk+1λ.V^\lambda(s_{k+1}, z_\tau) := h_{k+1} + (1-\lambda)\hat{V}(s_{k+1}, z_\tau) + \lambda G_{k+1}^\lambda.

At step k=t+N−1k = t+N-1, rearranging the recursion gives:

γVλ(st+N,zτ)=Gt+N−1λ−rτ(st+N−1,at+N−1).\gamma V^\lambda(s_{t+N}, z_\tau) = G_{t+N-1}^\lambda - r_\tau(s_{t+N-1}, a_{t+N-1}).

We then form the step target Gtstep,NG_t^{\mathrm{step},N} with chunk size NN just like a 1-chunk TD target, except instead of truncating credit assignment at st+Ns_{t+N} with V^(st+N,zτ)\hat{V}(s_{t+N}, z_\tau), we substitute Vλ(st+N,zτ)V^\lambda(s_{t+N}, z_\tau) extracted from the per-step TD(λ)\mathrm{TD}(\lambda) sweep:

Gtstep,N=Rτ(st,At)+γNVλ(st+N,zτ)=Rτ(st,At)+γN−1(Gt+N−1λ−rτ(st+N−1,at+N−1)),G_t^{\mathrm{step},N} = R_\tau(s_t, A_t) + \gamma^N V^\lambda(s_{t+N}, z_\tau) = R_\tau(s_t, A_t) + \gamma^{N-1}\left(G_{t+N-1}^\lambda - r_\tau(s_{t+N-1}, a_{t+N-1})\right),

which expands to:

Gtstep,N=Rτ(st,At)+γN[ht+N+(1−λ)V^(st+N,zτ)+λGt+Nλ].G_t^{\mathrm{step},N} = R_\tau(s_t, A_t) + \gamma^N\left[h_{t+N} + (1-\lambda)\hat{V}(s_{t+N}, z_\tau) + \lambda G_{t+N}^\lambda\right].

At N=1N=1, since Rτ(st,At)=rτ(st,at)R_\tau(s_t, A_t) = r_\tau(s_t, a_t), this reduces identically to the standard TD(λ)\mathrm{TD}(\lambda) recursion:

Gtstep,1=rτ(st,at)+γ[ht+1+(1−λ)V^(st+1,zτ)+λGt+1λ]=Gtλ.G_t^{\mathrm{step},1} = r_\tau(s_t, a_t) + \gamma \left[h_{t+1} + (1-\lambda)\hat{V}(s_{t+1}, z_\tau) + \lambda G_{t+1}^\lambda\right] = G_t^\lambda.

What value function do the multi-chunk targets estimate when later chunks are selected by the switching behavior?

As described above, the multi-chunk target on PS rollouts is generally a biased surrogate for the task-policy value in Eq. (6).

Exact step target pseudocode

# Step TD(λ), original self-policy bootstrap (exact match to implementation).
# T = rollout length, N = chunk length.
# At step k in {0, ..., T-1}:
#   r[k]       = stored task reward (before entropy adjustment)
#   timeout[k] = True iff episode hit time-limit truncation at step k
#   done[k]    = terminated[k] or timeout[k] (IsaacGym episode-reset flag;
#                in MT50, episodes only end via 250-step timeouts, so done == timeout)
#   s[k+1]     = stored next observation (post-auto-reset state if done[k] is True,
#                following REPPO's IsaacGym proxy-bootstrap convention)

# 1. Compute per-step soft TD(λ) returns backward over the T-step rollout.
for k = T - 1 down to 0:
    u = k + 1
    A = sample_chunk(pi_theta, s[u], z_tau)           # fresh N-action chunk from own-task policy
    q = Q_phi(s[u], A, z_tau)                         # plain chunk Q (no entropy subtracted)
    h = -alpha * log_pi_first(A[0] | s[u], z_tau)     # first-action entropy reward shift

    if timeout[k] or k == T - 1:                      # time-limit truncation or rollout boundary:
        v = q                                         # cut λ-trace (λ = 0) and keep bootstrap q
    else if done[k]:                                  # true absorbing termination (done and not timeout):
        v = 0                                         # zero future bootstrap value
    else:
        v = (1 - lambda) * q + lambda * G[u]

    G[k] = r[k] + gamma * (h + v)

# 2. Build the critic target for each sliding N-action window A_t = a[t : t+N].
for t = 0 to T - N:                                   # W = T - N + 1 windows per env
    e = t + N - 1                                     # last step in the window

    # Discounted raw reward prefix, cut after any episode reset (done or timeout).
    R = 0
    for k = t to e:
        R += gamma**(k - t) * r[k]
        if done[k] or timeout[k]:
            break                                     # include boundary reward r[k], drop post-reset steps

    # Keep the boundary bootstrap iff:
    #   (a) no reset (neither termination nor timeout) occurred strictly inside t..e-1
    #       (otherwise s[e+1] belongs to a new post-reset episode), AND
    #   (b) step e did not hit a termination (a timeout at e still bootstraps).
    no_mid_reset = (not done[k] for all k in t, ..., e - 1)
    survive_end  = (not done[e]) or timeout[e]
    keep = 1.0 if (no_mid_reset and survive_end) else 0.0

    # G[e] - r[e] == gamma * (h + v) at step e, contributing gamma**N * V^λ(s[t+N]).
    G_step[t] = R + keep * gamma**(N - 1) * (G[e] - r[e])

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.