- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MORSE, a latent multi-task offline RL framework with mask-empowered reward learning for urban decision-making. The problem setting is multi-task offline RL where tasks share state and action spaces but differ in reward functions; for a target task, only a small labeled dataset is available, while large unlabeled trajectories from related tasks can be used. MORSE consists of three components: (1) Urban State Compression, which trains an autoencoder with a reconstruction loss plus KL regularization to map high-dimensional urban states (e.g., 125-dimensional spatial-temporal features) into a compact latent space; (2) Mask-Empowered Reward Learning, which jointly trains a reward ensemble and a trajectory masked autoencoder (TrajMAE) on labeled and unlabeled trajectories, where unlabeled rewards are annotated by the reward ensemble and a random proportion of state tokens are masked; and (3) Multi-Task Offline RL with Adaptive Knowledge Fusion, which trains two IQL policies, one on labeled data and one on relabeled data, and periodically fuses the relabeled policy into the labeled policy via an EMA update. Experiments on taxi trajectory datasets from Shenzhen and Chengdu evaluate policy performance against single-task offline RL baselines (IQL, CQL, BCQ, BEAR) and multi-task data-sharing methods (CDS, MODA, UDS, SRTD, Lador, HarmoDT), together with reward prediction error analysis, ablation studies, data-quality studies, and hyperparameter sensitivity analyses.
Critical strengths
-
The problem setting is well motivated and practically relevant. The paper targets multi-task offline RL in urban environments where target-task reward labels are sparse and rewards are implicit, which is a realistic and underexplored setting. The three identified challenges—data sparsity, implicit rewards, and high-dimensional state spaces—are concrete and align well with the proposed components. This makes the paper more than a generic offline RL method paper and gives it a clear application-driven narrative.
-
The proposed framework is technically coherent and the empirical validation is broad. The combination of state compression, masked trajectory modeling for reward learning, and EMA-based knowledge fusion is reasonable and mutually reinforcing. The experiments cover two real-world cities, multiple driver expertise levels, reward prediction metrics, ablations, data-quality variations, and hyperparameter sensitivity. The consistent improvements over both single-task and multi-task baselines, especially on expert drivers, provide non-trivial evidence for the effectiveness of the approach.
Critical weaknesses
-
The evaluation relies entirely on a learned simulator, which raises concerns about the reliability of the reported policy performance. The passenger-seeking simulator is an ensemble of seven dynamics models with an OOD termination rule, trained on data from all drivers. While the authors state that the simulator is trained separately from policy learning and used only at evaluation time, the reported returns ultimately depend on the fidelity of this learned simulator. There is no comparison against any ground-truth or held-out real trajectory outcome, nor a validation of whether the simulator preserves the relative ordering of policies. Since the central claim is that MORSE improves policy performance, the lack of direct real-world or high-confidence evaluation weakens the empirical support. This is consequential because simulator bias could favor methods that exploit the learned dynamics in ways that do not transfer.
-
The relationship between the target task and the unlabeled source tasks is assumed to be benign, but this assumption is not sufficiently examined. The method relabels unlabeled trajectories from other tasks using the target-task reward ensemble, and then trains a policy on this relabeled data. If source-task trajectories are generated by drivers with very different reward functions or behavioral preferences, the relabeled rewards may be systematically misleading. The paper does include a data-quality study in Table 4, which shows performance degradation when unlabeled data quality drops. However, it does not analyze task relatedness quantitatively, nor does it test cases where source tasks conflict with the target task. The limitation is acknowledged in the conclusion, but the core method still relies on this assumption. This is consequential because in real urban multi-task settings, negative transfer is a central risk, and the paper's claims about robustness would be stronger with a more explicit characterization of when the method fails.
Questions for authors
-
Simulator validation. Can the authors provide any validation that the learned passenger-seeking simulator preserves the relative performance ordering of policies? For example, were any policies evaluated on real held-out trajectories or on a small set of human-interpretable outcomes, and do the simulator returns correlate with those outcomes? This is a clarification question because it directly affects how much weight the main results can carry.
-
Task relatedness. How is task relatedness quantified or controlled across drivers? Are the source tasks selected based on some similarity measure, or are all other drivers used indiscriminately? If all other drivers are used, can the authors report performance as a function of source-task similarity to the target task? This is a clarification question because it determines whether the method's success depends on a hidden selection mechanism.
-
Reward ensemble calibration. Table 2 reports MAE and RMSE for reward prediction, but it is unclear whether the reward ensemble is calibrated. Can the authors provide calibration plots or uncertainty estimates for the predicted rewards, especially for relabeled unlabeled transitions? This is a clarification question because the policy training on relabeled data relies on the reliability of these predictions.
-
EMA fusion mechanism. The EMA update in Eq. (8) fuses the relabeled policy into the labeled policy. Is the fusion applied only to the policy network, or also to the Q and value networks? If only the policy network is fused, how is the value function kept consistent with the updated policy? This is a clarification question because it affects the stability and correctness of the knowledge fusion procedure.
-
Comparison to stronger multi-task baselines. HarmoDT is included, but it is a Decision Transformer-based method. Were other recent multi-task offline RL methods designed for reward-free or sparsely labeled settings considered, such as methods that learn task representations without reward supervision? This is a suggestion for future work, but it also helps contextualize the novelty of MORSE.
5 comments
W2 & Q2: Task relatedness
There is no hidden selection mechanism. For each target driver, the trajectories of all other drivers are used as the unlabeled pool, without any similarity measure (Appendix A.2 and Algorithm 1).
Our definition of tasks follows the common paradigm of multi-task offline RL and offline meta-RL. In the standard benchmarks of these areas (e.g., Mitchell et al., 2021), the same HalfCheetah robot is rewarded for running forward in one task and backward in the other (Cheetah-Dir), each task rewards a different target velocity (Cheetah-Vel), or each task rewards moving in a different direction (Ant-Dir). The tasks of each benchmark share the state space, the action space and the dynamics, and differ only in the direction or the target velocity specified by the reward function, yet they are treated as different tasks. Our setting, in which drivers share the urban dynamics and differ in their reward functions (Section 2), follows the same intuition.
We agree that task relatedness, as raised in Q2, is a very good point. To our knowledge, there is no unified way of quantifying it in multi-task offline RL or offline meta-RL: different tasks may have similar reward functions over the same state and action spaces, as in the benchmarks above, or may differ entirely, as in cross-domain settings where even the state and action spaces are different. In our setting, the rewards of the source tasks never enter MORSE. An unlabeled transition is relabeled by the reward model of the target task, and since the dynamics are shared by all tasks, a transition generated by any driver is a valid transition of the target MDP, for which the target reward is well defined. A source driver with different preferences therefore changes which transitions the pool contains, i.e., its coverage and behavior quality, and not the meaning of the labels assigned to them.
The effect of this factor is reported in Table 4. A pool generated by a random policy is the extreme case, since its behavior follows no driver's objective. When the pool changes from expert to random, the return of MORSE decreases (724.18 to 625.85, 616.39 to 523.39 and 543.68 to 422.37), but it remains above No Sharing in all three cases (by 19.0%, 8.3% and 9.6%), so no negative transfer occurs, whereas CDS falls below No Sharing in two of these cases. The remaining limit is the one stated in the conclusion: tasks with a different objective, beyond the passenger-seeking tasks studied here, are left for future work.
Q3: Reward ensemble calibration
MORSE does not rely on uncertainty estimates of the ensemble. Relabeling uses only the point prediction given by the trimmed mean of Eq. (2), so what matters for the current method is the accuracy of this prediction, which is what Table 2 measures. Against held-out true rewards, the MAE is 7.76/8.44/7.68 (Expert/Medium/Random), compared with 11.36/13.95/16.48 for the plain ensemble and with 51.58/39.09/43.98 when the rewards are set to zero as in UDS. The trimmed mean further discards the highest and the lowest member, which removes the most extreme predictions.
For the relabeled transitions, the true rewards of the target task are not available by the definition of the setting, so their reliability is reflected in the policy trained on them: replacing the plain ensemble by our reward model for relabeling raises the return from 622.59/546.85/420.09 to 687.37/597.22/469.67 (Table 3). The errors that remain are handled by knowledge fusion, which was designed for this risk (Section 3.3): the returned policy belongs to the learner trained on the true labels, and only the policy of the relabeled learner is fused into it.
We agree that the disagreement of the ensemble is a useful signal. A calibrated uncertainty would be required to adapt the fusion to the reliability of the predicted rewards, which we consider a promising direction for future work.
We thank the reviewer for the supportive assessment and for the clearly stated questions, which we answer in order.
W1 & Q1: Simulator validation
Evaluating policies in a simulator built from data is the established protocol in this line of work, since a learned policy cannot be executed in the real city for safety and practical reasons. Our simulator follows MODA (Zhao et al., 2024) and D2TSC (Li et al., 2023), and its construction, an ensemble of dynamics models with termination under model disagreement, is the one used in model-based offline RL (Kidambi et al., 2020; Yu et al., 2020).
Following the reviewer's suggestion, we validated the simulator against real held-out outcomes. A learned policy cannot be executed on logged data, so we replayed the real trajectories of the test split, which is never seen by the simulator, and compared the simulated return of each trajectory with the return realized in the data (Table R4). We also report the prediction error of the simulator on this split (Table R3). Both results have been added to Appendix A.7.
Table R3. Prediction error of the simulator on the held-out test split (mean absolute error over the state dimensions, each normalized by its standard deviation). The k-step predictions start from a real state and follow the logged actions, with each prediction fed back as the next input.
Table R4. Held-out real trajectories replayed in the simulator: agreement between the simulated return and the return realized in the data.
The simulated returns are strongly correlated with the realized returns in both cities, so the simulator preserves the ordering of real outcomes on held-out data. Regarding the ordering of policies, the simulator also reproduces every ordering that is known in advance: Expert > Medium > Random for all twelve methods in both cities (Table 1), and returns that are monotone in the quality of both the labeled and the unlabeled data for all five methods of Table 4. Moreover, the simulators of the two cities are trained independently on different data, and the rankings of the twelve methods in the two cities have a Spearman correlation of 0.85. Finally, no method can exploit the learned dynamics: the simulator is trained separately from policy learning and is used only for evaluation, MORSE is model-free and never queries a dynamics model, and a rollout is terminated as soon as the seven dynamics models disagree beyond a threshold (Appendix A.7).