acceptodds
Under review as a conference paper at ICLR 2027

Urban-Centric Latent Multi-Task Offline Reinforcement Learning with Mask-Empowered Reward Learning

Abstract

In urban environments, human decision-making strategies—such as commuting routes, delivery logistics, and public transit planning—are often shaped by individual preferences and localized constraints, leading to suboptimal outcomes and leaving substantial room for improvement. Offline reinforcement learning (RL) offers a promising paradigm for improving urban decision-making by optimizing policies from pre-collected datasets, without requiring real-time interaction that could raise safety concerns. However, despite its potential, the application of offline RL to urban settings, particularly to realistic multi-task urban environments, remains largely underexplored due to the unique challenges inherent to urban decision-making. In this paper, we propose MORSE—Latent Multi-Task Offline RL with Mask-Empowered Reward Learning, a novel framework for optimizing diverse human strategies in urban environments. MORSE operates in a latent state space through Urban State Compression, improving learning efficiency and scalability. Moreover, it introduces a Mask-Empowered Reward Learning framework that integrates a novel trajectory masked autoencoder (TrajMAE) with a reward ensemble network, jointly trained in a semi-supervised manner, leveraging sparse labeled data from the target task alongside abundant unlabeled data from related tasks. The learned reward function is then used for data relabeling and augmentation. To mitigate the risk of over-reliance on relabeled data, MORSE further employs an Adaptive Knowledge Fusion strategy, enabling robust and generalizable policy learning in complex urban environments. Extensive experiments on real-world datasets demonstrate that MORSE significantly outperforms state-of-the-art baselines in learning diverse urban human strategies.

Primary area:
reinforcement learning
Keywords:
Multi-Task Offline Reinforcement Learning, Reward Learning, Semi-Supervised Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_u5RZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MORSE, a latent multi-task offline RL framework with mask-empowered reward learning for urban decision-making. The problem setting is multi-task offline RL where tasks share state and action spaces but differ in reward functions; for a target task, only a small labeled dataset is available, while large unlabeled trajectories from related tasks can be used. MORSE consists of three components: (1) Urban State Compression, which trains an autoencoder with a reconstruction loss plus KL regularization to map high-dimensional urban states (e.g., 125-dimensional spatial-temporal features) into a compact latent space; (2) Mask-Empowered Reward Learning, which jointly trains a reward ensemble and a trajectory masked autoencoder (TrajMAE) on labeled and unlabeled trajectories, where unlabeled rewards are annotated by the reward ensemble and a random proportion of state tokens are masked; and (3) Multi-Task Offline RL with Adaptive Knowledge Fusion, which trains two IQL policies, one on labeled data and one on relabeled data, and periodically fuses the relabeled policy into the labeled policy via an EMA update. Experiments on taxi trajectory datasets from Shenzhen and Chengdu evaluate policy performance against single-task offline RL baselines (IQL, CQL, BCQ, BEAR) and multi-task data-sharing methods (CDS, MODA, UDS, SRTD, Lador, HarmoDT), together with reward prediction error analysis, ablation studies, data-quality studies, and hyperparameter sensitivity analyses.

Critical strengths
  1. The problem setting is well motivated and practically relevant. The paper targets multi-task offline RL in urban environments where target-task reward labels are sparse and rewards are implicit, which is a realistic and underexplored setting. The three identified challenges—data sparsity, implicit rewards, and high-dimensional state spaces—are concrete and align well with the proposed components. This makes the paper more than a generic offline RL method paper and gives it a clear application-driven narrative.

  2. The proposed framework is technically coherent and the empirical validation is broad. The combination of state compression, masked trajectory modeling for reward learning, and EMA-based knowledge fusion is reasonable and mutually reinforcing. The experiments cover two real-world cities, multiple driver expertise levels, reward prediction metrics, ablations, data-quality variations, and hyperparameter sensitivity. The consistent improvements over both single-task and multi-task baselines, especially on expert drivers, provide non-trivial evidence for the effectiveness of the approach.

Critical weaknesses
  1. The evaluation relies entirely on a learned simulator, which raises concerns about the reliability of the reported policy performance. The passenger-seeking simulator is an ensemble of seven dynamics models with an OOD termination rule, trained on data from all drivers. While the authors state that the simulator is trained separately from policy learning and used only at evaluation time, the reported returns ultimately depend on the fidelity of this learned simulator. There is no comparison against any ground-truth or held-out real trajectory outcome, nor a validation of whether the simulator preserves the relative ordering of policies. Since the central claim is that MORSE improves policy performance, the lack of direct real-world or high-confidence evaluation weakens the empirical support. This is consequential because simulator bias could favor methods that exploit the learned dynamics in ways that do not transfer.

  2. The relationship between the target task and the unlabeled source tasks is assumed to be benign, but this assumption is not sufficiently examined. The method relabels unlabeled trajectories from other tasks using the target-task reward ensemble, and then trains a policy on this relabeled data. If source-task trajectories are generated by drivers with very different reward functions or behavioral preferences, the relabeled rewards may be systematically misleading. The paper does include a data-quality study in Table 4, which shows performance degradation when unlabeled data quality drops. However, it does not analyze task relatedness quantitatively, nor does it test cases where source tasks conflict with the target task. The limitation is acknowledged in the conclusion, but the core method still relies on this assumption. This is consequential because in real urban multi-task settings, negative transfer is a central risk, and the paper's claims about robustness would be stronger with a more explicit characterization of when the method fails.

Questions for authors
  1. Simulator validation. Can the authors provide any validation that the learned passenger-seeking simulator preserves the relative performance ordering of policies? For example, were any policies evaluated on real held-out trajectories or on a small set of human-interpretable outcomes, and do the simulator returns correlate with those outcomes? This is a clarification question because it directly affects how much weight the main results can carry.

  2. Task relatedness. How is task relatedness quantified or controlled across drivers? Are the source tasks selected based on some similarity measure, or are all other drivers used indiscriminately? If all other drivers are used, can the authors report performance as a function of source-task similarity to the target task? This is a clarification question because it determines whether the method's success depends on a hidden selection mechanism.

  3. Reward ensemble calibration. Table 2 reports MAE and RMSE for reward prediction, but it is unclear whether the reward ensemble is calibrated. Can the authors provide calibration plots or uncertainty estimates for the predicted rewards, especially for relabeled unlabeled transitions? This is a clarification question because the policy training on relabeled data relies on the reliability of these predictions.

  4. EMA fusion mechanism. The EMA update in Eq. (8) fuses the relabeled policy into the labeled policy. Is the fusion applied only to the policy network, or also to the Q and value networks? If only the policy network is fused, how is the value function kept consistent with the updated policy? This is a clarification question because it affects the stability and correctness of the knowledge fusion procedure.

  5. Comparison to stronger multi-task baselines. HarmoDT is included, but it is a Decision Transformer-based method. Were other recent multi-task offline RL methods designed for reward-free or sparsely labeled settings considered, such as methods that learn task representations without reward supervision? This is a suggestion for future work, but it also helps contextualize the novelty of MORSE.

5 comments

We thank the reviewer for the supportive assessment and for the clearly stated questions, which we answer in order.

W1 & Q1: Simulator validation

Evaluating policies in a simulator built from data is the established protocol in this line of work, since a learned policy cannot be executed in the real city for safety and practical reasons. Our simulator follows MODA (Zhao et al., 2024) and D2TSC (Li et al., 2023), and its construction, an ensemble of dynamics models with termination under model disagreement, is the one used in model-based offline RL (Kidambi et al., 2020; Yu et al., 2020).

Following the reviewer's suggestion, we validated the simulator against real held-out outcomes. A learned policy cannot be executed on logged data, so we replayed the real trajectories of the test split, which is never seen by the simulator, and compared the simulated return of each trajectory with the return realized in the data (Table R4). We also report the prediction error of the simulator on this split (Table R3). Both results have been added to Appendix A.7.

Table R3. Prediction error of the simulator on the held-out test split (mean absolute error over the state dimensions, each normalized by its standard deviation). The kk-step predictions start from a real state and follow the logged actions, with each prediction fed back as the next input.

City1-step5-step
Shenzhen0.030.07
Chengdu0.050.11

Table R4. Held-out real trajectories replayed in the simulator: agreement between the simulated return and the return realized in the data.

CityPearsonSpearmanMean abs. error of return
Shenzhen0.920.8652.3
Chengdu0.870.8059.6

The simulated returns are strongly correlated with the realized returns in both cities, so the simulator preserves the ordering of real outcomes on held-out data. Regarding the ordering of policies, the simulator also reproduces every ordering that is known in advance: Expert > Medium > Random for all twelve methods in both cities (Table 1), and returns that are monotone in the quality of both the labeled and the unlabeled data for all five methods of Table 4. Moreover, the simulators of the two cities are trained independently on different data, and the rankings of the twelve methods in the two cities have a Spearman correlation of 0.85. Finally, no method can exploit the learned dynamics: the simulator is trained separately from policy learning and is used only for evaluation, MORSE is model-free and never queries a dynamics model, and a rollout is terminated as soon as the seven dynamics models disagree beyond a threshold (Appendix A.7).

W2 & Q2: Task relatedness

There is no hidden selection mechanism. For each target driver, the trajectories of all other drivers are used as the unlabeled pool, without any similarity measure (Appendix A.2 and Algorithm 1).

Our definition of tasks follows the common paradigm of multi-task offline RL and offline meta-RL. In the standard benchmarks of these areas (e.g., Mitchell et al., 2021), the same HalfCheetah robot is rewarded for running forward in one task and backward in the other (Cheetah-Dir), each task rewards a different target velocity (Cheetah-Vel), or each task rewards moving in a different direction (Ant-Dir). The tasks of each benchmark share the state space, the action space and the dynamics, and differ only in the direction or the target velocity specified by the reward function, yet they are treated as different tasks. Our setting, in which drivers share the urban dynamics and differ in their reward functions (Section 2), follows the same intuition.

We agree that task relatedness, as raised in Q2, is a very good point. To our knowledge, there is no unified way of quantifying it in multi-task offline RL or offline meta-RL: different tasks may have similar reward functions over the same state and action spaces, as in the benchmarks above, or may differ entirely, as in cross-domain settings where even the state and action spaces are different. In our setting, the rewards of the source tasks never enter MORSE. An unlabeled transition is relabeled by the reward model of the target task, and since the dynamics are shared by all tasks, a transition generated by any driver is a valid transition of the target MDP, for which the target reward is well defined. A source driver with different preferences therefore changes which transitions the pool contains, i.e., its coverage and behavior quality, and not the meaning of the labels assigned to them.

The effect of this factor is reported in Table 4. A pool generated by a random policy is the extreme case, since its behavior follows no driver's objective. When the pool changes from expert to random, the return of MORSE decreases (724.18 to 625.85, 616.39 to 523.39 and 543.68 to 422.37), but it remains above No Sharing in all three cases (by 19.0%, 8.3% and 9.6%), so no negative transfer occurs, whereas CDS falls below No Sharing in two of these cases. The remaining limit is the one stated in the conclusion: tasks with a different objective, beyond the passenger-seeking tasks studied here, are left for future work.

Q3: Reward ensemble calibration

MORSE does not rely on uncertainty estimates of the ensemble. Relabeling uses only the point prediction given by the trimmed mean of Eq. (2), so what matters for the current method is the accuracy of this prediction, which is what Table 2 measures. Against held-out true rewards, the MAE is 7.76/8.44/7.68 (Expert/Medium/Random), compared with 11.36/13.95/16.48 for the plain ensemble and with 51.58/39.09/43.98 when the rewards are set to zero as in UDS. The trimmed mean further discards the highest and the lowest member, which removes the most extreme predictions.

For the relabeled transitions, the true rewards of the target task are not available by the definition of the setting, so their reliability is reflected in the policy trained on them: replacing the plain ensemble by our reward model for relabeling raises the return from 622.59/546.85/420.09 to 687.37/597.22/469.67 (Table 3). The errors that remain are handled by knowledge fusion, which was designed for this risk (Section 3.3): the returned policy belongs to the learner trained on the true labels, and only the policy of the relabeled learner is fused into it.

We agree that the disagreement of the ensemble is a useful signal. A calibrated uncertainty would be required to adapt the fusion to the reliability of the predicted rewards, which we consider a promising direction for future work.

Reviewer_3Kr9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MORSE, a multi-task offline reinforcement-learning framework for urban decision-making with sparse reward supervision. It combines latent state compression and semi-supervised reward learning through a trajectory masked autoencoder and a reward ensemble. It also introduces EMA-based policy fusion to use unlabeled cross-task data. Experiments on taxi-trajectory datasets from Shenzhen and Chengdu report consistent improvements over existing offline RL baselines.

Critical strengths
  1. The paper addresses a meaningful problem. Learning individual decision-making policies for urban offline RL when reward signals are sparse, while benefiting from trajectories collected from related tasks.
  2. The proposed components are coherent and complementary. State compression, semi-supervised reward estimation, and policy fusion form an integrated framework for exploiting unlabeled cross-task trajectories while accounting for potentially imperfect reward predictions.
  3. The empirical results are promising. MORSE consistently outperforms the baselines across both cities and different driver-expertise levels. The improvements in reward-prediction accuracy further support the effectiveness of the proposed reward-learning framework.
Critical weaknesses
  1. The reward construction raises concerns about the validity of the problem formulation. Driver-specific rewards are constructed retrospectively from completed trajectory outcomes and estimated preferences. Because these rewards depend on trajectory-level information, it is unclear whether they can be represented consistently by the assumed transition-level function Ri(s,a,s′)R_i (s, a, s').
  2. The mechanism underlying Mask-Empowered Reward Learning is insufficiently justified. In Eq. (6), predicted rewards appear in both the input trajectories and the reconstruction targets, while only state tokens are masked. The reconstruction objective may therefore allow shortcut solutions that reproduce the supplied reward predictions without improving reward estimation. The paper does not explain what prevents this behavior or why the resulting gradients should improve reward generalization.
  3. The evaluation relies heavily on a learned simulator trained on trajectories from all available drivers. The paper provides insufficient quantitative evidence of simulator fidelity, particularly for long-horizon policy evaluation and out-of-distribution actions. In addition, the main evaluation covers only three selected target drivers per city, limiting the strength of the generalization claims.
  4. The methodological novelty is somewhat limited. The framework mainly integrates established techniques, including autoencoder-based representation learning, masked trajectory reconstruction, reward ensembles, IQL, and EMA parameter updates. In particular, Adaptive Knowledge Fusion uses fixed fusion coefficients and intervals rather than adapting them to reward uncertainty or task relationships.
Questions for authors
  1. Given that driver-specific rewards are constructed retrospectively from completed trajectory outcomes and estimated preferences, why can they be represented by a transition-level reward function Ri(s,a,s′)R_i (s, a, s')?
  2. Because reward tokens remain unmasked, what prevents TrajMAE from simply reproducing the supplied reward predictions? Could the authors provide further evidence that the unlabeled objective contributes meaningful reward supervision?
  3. Could the authors report prediction errors on held-out simulator data? Evaluations on additional target drivers or with complementary offline policy-evaluation methods would also help establish whether the improvements extend beyond the selected learned simulator.
7 comments

We thank the reviewer for the positive assessment of the problem and of the framework, and for the precise questions, which we answer below.

W1 & Q1: Whether the constructed rewards can be represented by a transition-level function

In our formulation, the reward of each task is defined on transitions. As described in Appendix A.8, the outcome of a passenger-seeking segment is distributed over the transitions of the segment by Eq. (10), so that every transition in the data carries its own reward and these rewards sum to the outcome of the segment. The datasets therefore have the standard form {(s,a,s′,r)}\{(s,a,s',r)\} of offline RL, and maximizing the cumulative transition reward maximizes the trajectory-level outcome. Assigning a delayed, trajectory-level outcome to individual steps in this way is the established practice of reward redistribution (Arjona-Medina et al., 2019; Ren et al., 2022).

The transition reward is not a deterministic function of (s,a,s′)(s,a,s'), since quantities such as the length of the trip that follows and the stochastic component zz in Eq. (9) are not contained in a single transition. As in any MDP with stochastic rewards, the reward function is then the conditional expectation Ri(s,a,s′)=E[r∣s,a,s′]R_i(s,a,s')=\mathbb{E}[r\mid s,a,s'], which is the quantity estimated by the squared-error loss of Eq. (3). The construction keeps this expectation informative, because the reward is tied to where and when a transition takes place. The preference score is computed from the regions and time periods in which the driver operates (Appendix A.8), and the state contains the grid position, the time slot and the day of week together with the demand, waiting time, traffic volume and speed of the 5×55\times5 neighborhood (Appendix A.3). In addition, Eq. (10) assigns the largest share of the outcome to the transition at which the passenger is found, whose (s,a,s′)(s,a,s') identifies this place and time directly.

Table 2 measures how well a transition-level function represents these rewards. Using only (s,a,s′)(s,a,s'), the reward model predicts the rewards of held-out transitions with an MAE of 7.76/8.44/7.68 (Expert/Medium/Random), against 51.58/39.09/43.98 when the rewards are set to zero as in UDS, which reflects their magnitude. Finally, all methods are trained on the same transition-level labels of the target driver and are evaluated in the same simulator under the same driver-specific reward (Appendix A.3), so the construction does not favor any method. We have added this clarification to Appendix A.8.

W2 & Q2: What prevents TrajMAE from reproducing the supplied rewards

Reproducing the visible tokens. The reviewer is right that the reward tokens are visible and that TrajMAE can reproduce them easily. This does not give a shortcut for the objective, for two reasons. First, even in the worst case, in which a reward is reproduced exactly, it contributes zero loss and therefore no gradient to the reward ensemble, so this term at worst neither improves nor distorts the reward predictions. Second, the loss of Eq. (6) cannot be minimized by reproducing the visible tokens alone, because the masked states still have to be reconstructed. They cannot be copied and have to be inferred from the visible states, actions and rewards of the trajectory, and it is this part of the objective that carries the learning signal. The terms on the visible tokens serve to keep the three token types in a common representation.

Why the gradients improve reward estimation. On labeled trajectories, TrajMAE receives the true rewards and learns how the reward of the target task relates to the states along a trajectory. On unlabeled trajectories, the same TrajMAE receives the rewards predicted by the ensemble, and the masked states are reconstructed well only if these predictions are consistent with the relation learned from the true labels. The gradient of LU\mathcal{L}^{\mathrm{U}} with respect to r^t\hat{r}_t therefore acts as a consistency signal on the large unlabeled set. In every update, it is combined with the supervised loss Lensemble\mathcal{L}_{\mathrm{ensemble}} on the true labels (Eq. (7); Algorithm 1, lines 16-17), which keeps the ensemble fitted to the true rewards.

Further evidence. (i) In Table 2, R-Ensemble w/ S.C. and MORSE differ only in the joint training with TrajMAE. If TrajMAE merely reproduced the supplied predictions, the two would have the same error. Instead, the MAE on held-out true rewards decreases from 11.36/13.95/16.48 to 7.76/8.44/7.68 (Expert/Medium/Random), i.e., by 32%, 39% and 53%. (ii) The gain carries over to the policy: without TrajMAE, the return drops from 687.37/597.22/469.67 to 622.59/546.85/420.09 (Table 3). (iii) When only the masked states are reconstructed, so that rewards are never targets, the return on the Medium driver of Shenzhen is 556.68 ± 34.60, against 597.22 ± 26.31 for Eqs. (5)-(6) (new ablation in Appendix A.11). Keeping the visible tokens among the targets is therefore not harmful, and is beneficial in our experiments.

W3 & Q3: Simulator fidelity and target drivers

Evaluating policies in a simulator built from data is the established protocol in this line of work. Our simulator follows MODA (Zhao et al., 2024), which evaluates passenger-seeking policies in an ensemble of dynamics models learned from the taxi trajectories, and D2TSC (Li et al, 2023) likewise evaluate offline RL for traffic signal control in a simulation environment built from the historical data of a real intersection. Its construction, an ensemble of dynamics models with termination under model disagreement, is the one used in model-based offline RL (Kidambi et al., 2020; Yu et al., 2020). In urban applications, a learned policy cannot be executed in the real city for safety and practical reasons, so a simulator is the reasonable way to evaluate it. More generally, the standard benchmarks of RL, such as the MuJoCo tasks, are themselves simulators.

Fidelity of the simulator. Following the reviewer's request, we validated the simulator in two ways on the test split of the data, which is reserved before training and is never seen by the simulator. Table R3 reports its prediction error, and Table R4 compares the simulated and the realized returns of held-out real trajectories replayed in the simulator.

Table R3. Prediction error of the simulator on the held-out test split. The error is the mean absolute error over the state dimensions, each normalized by its standard deviation. The kk-step predictions start from a real state and follow the logged actions, with each prediction fed back as the next input.

City1-step5-step
Shenzhen0.030.07
Chengdu0.050.11

Table R4. Held-out real trajectories replayed in the simulator. Starting from the first real state of a trajectory, the logged actions are executed in the simulator, and the simulated return is compared with the return realized in the data.

CityPearsonSpearmanMean abs. error of return
Shenzhen0.920.8652.3
Chengdu0.870.8059.6

The one-step error is 0.03 and 0.05 of a standard deviation per state dimension and remains at 0.07 and 0.11 after five steps. On complete held-out trajectories, the simulated return is strongly correlated with the realized return (Pearson 0.92/0.87 and Spearman 0.86/0.80), so the simulator preserves the ordering of real outcomes. Both results have been added to Appendix A.7. The simulator is trained on the training data of all drivers (50 in Shenzhen and 30 in Chengdu), several times the data available to any compared method, and it is trained separately from policy learning, used only for evaluation and shared by all methods (Appendices A.3 and A.7).

Additional target drivers. The three target drivers of each city were selected at random within the three levels of expertise. Following the reviewer's suggestion, we compared MORSE with the strongest baseline, MODA, on five further target drivers.

Table R2. Return on five further target drivers, under the protocol of Table 1 (three seeds, 20 rollouts each).

CityDriverExpertiseMORSEMODA
Shenzhen1Expert610.05 ± 22.70550.19 ± 33.56
Shenzhen2Medium537.20 ± 34.17486.45 ± 29.91
Shenzhen3Random474.91 ± 35.48455.63 ± 40.26
Chengdu4Expert563.15 ± 27.64513.84 ± 41.28
Chengdu5Medium488.35 ± 31.43476.29 ± 37.62

Together with Table 1, MORSE obtains the higher return on all 11 target drivers, with an average improvement over MODA of 10.6% on Expert, 7.0% on Medium and 4.6% on Random drivers.

Reviewer_mWr7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper explores the problem of offline policy learning in urban mobility scenarios, a task complicated by the extreme scarcity of reward labels. The authors treat each driver as an individual task; while these tasks share the same environment dynamics, they possess distinct reward functions. The proposed method consists of three components: first, a Variational Autoencoder is employed to compress states into low-dimensional representations. Second, a reward model is constructed by training a reward ensemble network in conjunction with TrajMAE Autoencoder, which is then used to assign rewards to the trajectories of other drivers. Finally, two Implicit Q-Learning models are trained—one using only labeled data and the other using re-labeled data—with their parameters periodically fused during the training process. Experiments based on ride-hailing data from Shenzhen and Chengdu demonstrate that MORSE outperforms both single-task and multi-task offline reinforcement learning baselines.

Critical strengths
  1. In practical applications, it is very common for a target task to have only a small amount of data with reward labels, while a large volume of trajectory data—lacking such labels—is available from other tasks. This paper addresses the challenges posed by both insufficient data coverage and the difficulty of obtaining reward annotations.
  2. The overall approach is sound, effectively integrating representation learning, reward relabeling, and offline policy optimization. By training on both the original labeled data and the relabeled data, the method successfully mitigates reward prediction errors.
Critical weaknesses
  1. There are discrepancies between the paper and the code regarding the methods used. The paper states that TrajMAE reconstructs states, actions, and rewards (Equations 5–6), whereas the code (in train_semi_mae.py and models.py) reconstructs only the masked states. The paper mentions that only the policy network is subjected to fusion (Appendix A.6, Algorithm 1), while the code (in train_iql_ema.py) performs fusion on the Q, value and policy network. Furthermore, the paper specifies that fusion occurs every η training steps, whereas the code performs it every η epochs. It is unclear which of these approaches was actually used to produce the final results.

  2. TrajMAE, the core innovation of this paper, suffers from two issues. First, the model cannot distinguish which specific time step is being masked. Its token embeddings consist solely of "type + value" (Appendix A.5.1 and models.py) and lack positional encoding; all masked states are replaced by the same zero vector. Consequently, at the input stage, masked states within the same trajectory become identical, and if Dropout is disabled, their outputs will also be identical. The model cannot reconstruct different states at different time steps. Second, the state reconstruction loss, computed using unlabeled data, influences the reward model through the reward prediction component. However, the paper fails to demonstrate whether this optimization actually improves the accuracy of reward prediction, or if it merely steers the predicted values ​​toward numerical features that happen to facilitate state reconstruction. According to Equations (5) and (6), actions and rewards are never masked; reconstructing them is effectively equivalent to simple copying. The authors need to explain why this training approach leads to more accurate reward predictions.

  3. The results cannot be reproduced, and some figures are inconsistent. The config.py and utils.py files imported in the code are missing from the repository; furthermore, the reward formulation, simulator training and evaluation procedures, baseline implementations, and datasets are not provided. Additionally, the peak values ​​across the subplots in Fig. 2 differ noticeably and do not align with the data presented in Table 1. Moreover, for each city, results are reported for only 3 out of 20 drivers, and experiments were conducted using only three random seeds. In some cases, the performance differences are marginal, which is insufficient to demonstrate a comprehensive advantage.

Questions for authors
  1. Which version produced the reported results?
  2. The paper never specifies what rewards are used for Sharing All, CDS, and MODA on the source data. If any of these methods relabel the source data using the true target rewards, then the experimental setup across the various baselines becomes inconsistent.
  3. The authors need to clarify the innovation of the approach. Compared to UDS, PDS, MTM (Wu et al., 2023), and Ludor, the novelty of MORSE lies primarily in its integration of existing research; however, its ablation studies are limited to the removal of entire components. They fail to consider simpler configurations—such as applying PDS within the same IQL backbone or employing the same fusion scheme for reward regression.
9 comments

We sincerely thank the reviewer for the careful reading.

W1 & Q1: Which version produced the reported results

During the development of MORSE, we implemented several variants of TrajMAE and of the fusion step. The version adopted in the paper is the one described in the text: TrajMAE reconstructs the states, actions and rewards of a trajectory (Eqs. (5)-(6)), and only the policy network is fused (Appendix A.6 and Algorithm 1, line 28). The results in the tables of the paper were produced by this version. The pervious files released belonged to another variant, in which only the masked states are reconstructed and the Q, value and policy networks are all fused. We apologize for this oversight, and the anonymous repository now contains the version used in the paper. Regarding the fusion interval, η\eta is counted in epochs (η=3\eta=3 with 1000 epochs), which is also the unit of Fig. 2(d), and the wording "training step" in Section 3.3 and Algorithm 1 has been corrected.

To measure the effect of the two choices in which these variants differ, we evaluated all four combinations on the Medium driver of Shenzhen with three seeds, following the protocol of Table 1.

Reconstruction target of TrajMAEFused networksReturn
States, actions and rewardsPolicy (MORSE, as in the paper)597.22 ± 26.31
Masked states onlyPolicy556.68 ± 34.60
Masked states onlyQ, value and policy523.51 ± 24.25
States, actions and rewardsQ, value and policy534.07 ± 19.44

The configuration described in the paper obtains the highest return. Reconstructing only the masked states lowers it by 40.54 (6.8%), and fusing the Q and value networks together with the policy lowers it by 63.15 (10.6%). The variant examined by the reviewer, which differs in both choices, is the lowest and is 73.71 (12.3%) below. Each choice is also preferable irrespective of the other: fusing only the policy gives the higher return under both reconstruction targets (by 63.15 and 33.17), and reconstructing states, actions and rewards gives the higher return under both fusion schemes (by 40.54 and 10.56). The fusion scheme has the larger effect. Both observations are consistent with the design of the method.

Reconstruction target. On unlabeled trajectories, the reward ensemble is trained through the reward tokens that it supplies to TrajMAE (Eq. (6)), so the benefit of the joint training depends on how much the reconstruction relies on these tokens. When only the masked states are reconstructed, nothing constrains how the action and reward tokens are represented. Since consecutive states of a passenger-seeking segment are adjacent grid cells in consecutive time slots, a masked state can largely be recovered from its neighbouring states, and TrajMAE can reach a low loss while giving little weight to the rewards, in which case little signal reaches the ensemble. Under Eqs. (5)-(6), the action and the reward of every step must remain decodable from the same representations that are used to recover the masked states. Reconstructing these visible tokens is not difficult in itself; its role is to keep the three token types in a common representation, which encourages the masked states, which cannot be copied, to be recovered from states, actions and rewards jointly, as described in Section 3.2.

Fusion. In IQL, the Q and value networks are learned from the dataset by temporal-difference and expectile regression and do not depend on the policy (Kostrikov et al., 2021). The critic of the labeled learner is thus determined by the true rewards of DL\mathcal{D}_L, whereas the critic of the relabeled learner is fitted to predicted rewards. Fusing the critics carries the errors of the predicted rewards into the value estimates of the labeled learner, from which they enter the advantage weights of all its subsequent policy updates. When only the policy is fused, the labeled learner receives the broader coverage of πrelabeled\pi^{\text{relabeled}}, while its critic remains supervised by true rewards alone and continues to correct the fused policy on DL\mathcal{D}_L. This preserves the reliability of πlabeled\pi^{\text{labeled}} that Section 3.3 aims for.

This comparison has been added to the revised paper as an ablation study in Appendix A.11.

W2 (first issue): Positional encoding and masked states

We have checked the final version of the code, which is the one now in the anonymous repository (see W1 & Q1), and it does include a timestep embedding. Each token is embedded as the sum of three terms: a type embedding (state, action or reward), a value embedding, and a learnable timestep embedding indexed by the timestep tt and shared by the state, action and reward tokens of that timestep (time_embed in models.py). For a masked state, the value embedding is replaced by a learnable mask token, while the type and timestep embeddings are kept. The issue raised by the reviewer therefore does not arise. Masked states of the same trajectory are not identical at the input, since each of them carries the embedding of its own timestep, so their outputs differ with or without dropout, and the model can reconstruct different states at different timesteps. The shared timestep embedding also indicates which action and reward belong to which state. This is consistent with the method described in Section 3.2, where a trajectory is an ordered sequence and TrajMAE reconstructs the state, action and reward of every timestep (Eqs. (5)-(6)). The description of the token embedding in Appendix A.5.1, which omitted the timestep embedding, has been corrected, together with the sentence on masking in Section 3.2.

W2 (second issue): Does the unlabeled objective improve reward prediction

We agree that the action and reward tokens are visible, so their reconstruction terms in Eqs. (5)-(6) are easy to satisfy and are not the source of the learning signal. The signal comes from the masked states. A masked state sts_t cannot be copied and has to be inferred from the visible tokens, including the rewards of the transitions it belongs to, which are informative about sts_t because the reward is a function of (s,a,s′)(s, a, s'). On labeled trajectories, TrajMAE thus learns how the true rewards of the target task relate to the states and actions. On unlabeled trajectories, the same TrajMAE receives the predicted rewards, and the masked states are recovered well only if these predictions follow the relation learned from the true rewards. The gradient of LU\mathcal{L}^{\mathrm{U}} with respect to r^t\hat{r}_t carries this relation to the unlabeled transitions, while Lensemble\mathcal{L}_{\text{ensemble}}, optimized jointly in Eq. (7), keeps the ensemble fitted to the true rewards.

Table 2 measures the effect against the true rewards of held-out transitions: joint training with TrajMAE reduces the MAE of the same reward ensemble (R-Ensemble w/ S.C.) from 11.36/13.95/16.48 to 7.76/8.44/7.68. If the objective only made the predictions easier to reconstruct, this error would not decrease. Figure 2(b) shows that the gain comes from masking: the return rises from p=0p=0, where only visible tokens are reconstructed, to its peak at p=30%p=30\%.

W3: Reproducibility, Figure 2, drivers and seeds

The repository now includes config.py and utils.py. The reward formulation, the training and evaluation of the simulator, and the baselines are described in Appendices A.8, A.7 and A.9, respectively. Each baseline is implemented strictly following the original code of the corresponding paper and is applied to our data under the same protocol, which ensures a fair comparison. The complete dataset will be released upon acceptance.

Regarding Fig. 2, its purpose is to show the trend with respect to each hyperparameter. Each panel was run separately and each point is a single run, so the absolute values are subject to the variation across seeds (standard deviations of 25 to 38 in Table 1) and are not directly comparable with the three-seed averages of Table 1. Within each panel, all points share the same setting except for the varied factor, so the comparison is valid and consistent.

Regarding the drivers and seeds, the three target drivers of each city were selected at random within the three levels of expertise, and three random seeds are used following previous work such as MODA (Zhao et al., 2024). To extend the evaluation, we compared MORSE with the strongest baseline, MODA, on five further target drivers.

Table R2. Return on five further target drivers, under the protocol of Table 1 (three seeds, 20 rollouts each).

CityDriverExpertiseMORSEMODA
Shenzhen1Expert610.05 ± 22.70550.19 ± 33.56
Shenzhen2Medium537.20 ± 34.17486.45 ± 29.91
Shenzhen3Random474.91 ± 35.48455.63 ± 40.26
Chengdu4Expert563.15 ± 27.64513.84 ± 41.28
Chengdu5Medium488.35 ± 31.43476.29 ± 37.62

Together with Table 1, MORSE obtains the higher return on all 11 target drivers, with an average improvement over MODA of 10.6% on Expert, 7.0% on Medium and 4.6% on Random drivers, which shows a consistent advantage across drivers and cities. The margin is small in some cases because the advantage of MORSE is largest when the labeled data is of high quality and decreases with the quality of the supervision, since the quality of the relabeled rewards depends directly on the labeled data from which the reward model is learned. This behavior is consistent with the design of the method and is discussed in Section 4.3 and in the conclusion. Improving the performance when the labeled data is of low quality is a meaningful direction for future work.

Q2: Rewards used by the baselines on the source data

No method uses true target rewards on the source data. Sharing All, CDS and MODA are given the same data as MORSE: the labeled data of the target driver and the pool of the other drivers without rewards. These methods have no mechanism of their own to obtain rewards for the shared transitions and could not use the pool otherwise, and assigning zero reward to it, as UDS does, would put them at a disadvantage. For these three methods, the shared transitions are therefore labeled by a reward model trained on the labeled data of the target driver. Their reward supervision is thus the same as that of MORSE, namely the sparse labels of the target task, and the setting is consistent.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.