- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Bayesian Task Relabelling (BTR) for offline meta-RL. It builds on the Bayesian linear task model from GLiBRL, samples reward parameters from an approximation to the posterior, and generates new rewards along recorded trajectories while retaining their states and transitions. Sequential Bayesian updates produce additional task representations for policy training. The actor uses posterior-mean features, while an asymmetric critic receives extra task information. Experiments compare online adaptation and offline-context evaluation on eight benchmarks, with additional goal-extrapolation tests and ablations of relabelling, task sampling, and critic inputs.
Critical strengths
-
The method makes practical use of the Bayesian model: one sampled reward function generates a consistent task across each set of history segments, and the same model updates the task representation. Retaining recorded next states avoids adding dynamics-model errors. The paper is clear about which parts come from GLiBRL and about the approximation used for task sampling.
-
The results are promising across both evaluation protocols, with the highest mean return on all eight standard-split online tests and seven offline-context tests. Some gains are substantial: Ant-Dir OOD return reaches 308 versus 58 for the best external baseline, and standard-split Walker-Param-Sparse reaches 103 versus 83 (Tables 1–2). The shared datasets and three-seed comparisons provide useful evidence for the complete method.
Critical weaknesses
-
The no-relabelling ablation changes more than the rewards. Original tuples use the same context belief at both ends of a transition, whereas relabelled tuples receive sequential belief updates (Section 3.3; Appendix A.4). Relabelling also adds training tuples, and the two branches handle terminal flags differently. Removing that branch therefore removes several training changes at once. In particular, Walker-Param improves from 216 to 450 even though the environment's reward function is shared across tasks (Table 5). This could come from training on evolving dynamics beliefs, as the authors themselves suggest. A control using original rewards with the same sequential updates, initialisation, sample count, and terminal handling is needed to establish the additional value of imaginary tasks.
-
The existing-task comparison does not isolate the benefit of sampling beyond training tasks. BTR samples around the source task's posterior, while the alternative selects a different training task's posterior mean (Appendix C). Meanwhile, half of the relabelled segments start from a belief inferred using the source task's original rewards (Appendix A.4). This can give the alternative a less appropriate starting belief, alongside changing the reward model's centre and its sampling variance. A comparison with the source task's own posterior mean, retaining the same reward noise, would isolate parameter sampling more directly. Comparing both samplers from the prior would also remove the initialisation concern. Without these controls, I find it difficult to attribute the large gains in Table 9 specifically to posterior task sampling.
Questions for authors
-
Additional control: How does BTR compare with training on the same history segments using original rewards and sequential belief updates, with the same initial beliefs, number of tuples, and terminal handling? This would help separate task relabelling from the change in policy-training protocol.
-
Task-sampling ablation: What happens when the sampled reward parameters are replaced by the source task's posterior mean, keeping the reward-noise model unchanged? Does the advantage over the existing-task variant persist when both start exclusively from the prior?
-
Implementation clarification: What is the training context size k, and how are original and relabelled losses weighted? Appendix A.4 retains terminal flags for original tuples but uses non-terminal targets for relabelled segments. Are genuine environment terminations excluded from those segments, or are they bootstrapped through?
Remark: I'm willing to adjust my score if the authors can properly address my concerns above.
2 comments
As the rebuttal period is ending in a couple of days, could we kindly ask if the response has addressed your concerns? We are happy to provide further clarification.
We thank the reviewer for the insightful feedback. We will take the chance to clarify as follows:
W.1 The no-relabelling ablation removes sequential beliefs, extra tuples and the terminal handling together with the rewards, so Table 5 cannot credit Walker-Param to imaginary tasks. and Q.1 How does BTR compare with the same segments, original rewards and sequential updates, with matched initial beliefs, tuple count and terminal handling?
BTR has two components: (1) the first offline BLTM-based OMRL (2) easy belief relabelling. Running the first component alone under the same experiment setting will test the control you ask for. It keeps everything in BTR (segments, starting beliefs, step-by-step updates, etc) with the only difference that the recorded reward was used and nothing is relabelled.
It is correct that the gain in Walker-Param is indeed from the offline OMRL rather than the relabelling. We thank the reviewer for pointing this out. Nonetheless, the relabelling is still improving most of the standard split / OOD performance as expected. In the revised paper, we will include this table to identify the gain introduced by difference components of BTR more clearly.
W.2 The existing-task comparison changes the sampling centre, the variance and the starting belief at once, so Table 9 does not isolate sampling beyond training tasks. and Q.2 What happens with the source task's posterior mean instead of a sample, keeping the reward noise? Does the advantage over the existing-task variant hold when both start from the prior?
We ran the settings that vary one factor at a time (sampling centre, sampling procedure, segment start).
The first row is Table 9, and the last row is the full BTR. This shows the performance of different factors combined together. The setting BTR by default uses is the best overall.
For the first subquestion of Q2 (row 4 against row 5): using posterior mean rather than using samples make the result worse.
For the second subquestion of Q2 (row 2 against row 3): yes, though by a smaller margin.
Q.3 Context size k, loss weighting, and terminals inside relabelled segments.
k is drawn uniformly from [0, 200]. There is no separate weights, and every transition, original or relabelled, has equal weight in one shared actor and critic loss.
Regarding termination flags: the seven MuJoCo datasets contain no terminal flags, and Point-Robot-Sparse terminates at the goal. So for MuJoCo envs, both original and relabelled bootstrap through episode ends. For the Point-Robot-Sparse, the original does not bootstrap through, while the relabelled does on purpose because we have no termination model for the sampled goal.