- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SMART-HOLD, a scalable MARL solution for dynamic bus holding control. Experiments on six real-world bus routes illustrate the efficiency of the proposed method.
Critical strengths
-
The motivation of the whole design is clear and direct.
-
The structure and writing of this paper are good.
Critical weaknesses
- The innovation of this paper is weak. Even though the task-oriented design is proper and good, almost all modules come from mature techniques.
- The competing approaches are very naive, without comparison to recent SOTA solutions.
- The paper only considers a single route, neglecting the inter-route impact.
Questions for authors
The RTDC solution is similar to the role-play MARL methods. What are their core differences, and what are the benefits of RTDC?
4 comments
Response to Reviewer UyYe cnt.
W2. Insufficient comparison with recent SOTA methods
Our existing evaluation includes CAAC, a specialized asynchronous MARL method for bus holding control, together with established MARL methods MADDPG and MAPPO. SMART-HOLD achieves the lowest mean Bunching, Headway CV, and EWT across all six routes. To further strengthen the evaluation, we introduce an additional baseline following HASAC (Liu et al., ICLR 2024), a recent maximum-entropy MARL method.
(1) New baseline: Heterogeneous SAC. We adapt the heterogeneous-policy learning idea of HASAC to our asynchronous BHC setting, where buses make decisions at different times and the number of operating buses varies. Specifically, we train stochastic actors with shared twin centralized critics. Each bus trip is assigned one actor in rotation, which controls the bus throughout that trip. The actors are trained using maximum-entropy SAC, without knowledge distillation.
We refer to these variants as Heterogeneous SAC-k3 and SAC-k5. Both variants are trained on R1 for 100 episodes and evaluated using the same simulator and evaluation procedure as the other learning-based methods.
(2) Experimental results. The following table compares SMART-HOLD with the two new baselines across all six routes. Results are reported as mean ± standard deviation. Lower values are better.
| Route | Method | Bunching | Headway CV | EWT (min) |
|---|---|---|---|---|
| R1 | SMART-HOLD | 1.45 ± 0.91 | 0.21 ± 0.02 | 0.35 ± 0.07 |
| SAC-k3 | 1.99 ± 1.69 | 0.24 ± 0.01 | 0.43 ± 0.05 | |
| SAC-k5 | 2.36 ± 2.42 | 0.25 ± 0.01 | 0.46 ± 0.05 | |
| R3 | SMART-HOLD | 5.97 ± 3.64 | 0.24 ± 0.03 | 0.49 ± 0.10 |
| SAC-k3 | 5.37 ± 2.44 | 0.27 ± 0.02 | 0.59 ± 0.07 | |
| SAC-k5 | 7.28 ± 3.50 | 0.29 ± 0.02 | 0.65 ± 0.07 | |
| M1 | SMART-HOLD | 0.00 ± 0.00 | 0.23 ± 0.03 | 0.55 ± 0.15 |
| SAC-k3 | 0.08 ± 0.63 | 0.25 ± 0.02 | 0.66 ± 0.12 | |
| SAC-k5 | 0.18 ± 0.97 | 0.25 ± 0.03 | 0.64 ± 0.16 | |
| M2 | SMART-HOLD | 2.25 ± 5.77 | 0.27 ± 0.03 | 0.41 ± 0.08 |
| SAC-k3 | 2.10 ± 2.37 | 0.29 ± 0.02 | 0.44 ± 0.05 | |
| SAC-k5 | 4.47 ± 3.90 | 0.29 ± 0.01 | 0.44 ± 0.04 | |
| M3 | SMART-HOLD | 0.27 ± 0.72 | 0.26 ± 0.02 | 0.42 ± 0.07 |
| SAC-k3 | 8.02 ± 6.00 | 0.31 ± 0.02 | 0.59 ± 0.07 | |
| SAC-k5 | 12.06 ± 6.74 | 0.31 ± 0.02 | 0.61 ± 0.06 | |
| M7 | SMART-HOLD | 1.59 ± 3.11 | 0.25 ± 0.02 | 0.30 ± 0.03 |
| SAC-k3 | 20.82 ± 7.57 | 0.33 ± 0.02 | 0.50 ± 0.05 | |
| SAC-k5 | 28.62 ± 8.59 | 0.33 ± 0.01 | 0.52 ± 0.04 |
SMART-HOLD achieves lower Headway CV and EWT than both heterogeneous SAC variants on all six routes, including five unseen routes. Averaged across routes, its relative EWT reductions are approximately 21% compared with SAC-k3 and 24% compared with SAC-k5. It also achieves fewer bunching events than SAC-k5 on every route and SAC-k3 on four of the six routes, with SAC-k3 achieving marginally fewer bunching events on R3 and M2.
(3) Implications for policy sharing. These results provide further evidence for the effectiveness of SMART-HOLD's shared-policy design. Unlike the heterogeneous SAC variants, which maintain multiple policies during execution, SMART-HOLD uses regional teachers during training and consolidates their knowledge into a single student policy. This enables coordinated decision-making across buses without maintaining separate execution policies.
The consistent improvements on unseen routes support the effectiveness of this design under varying operating conditions and fleet sizes. We have included the new baseline results and implementation details in the revised paper.
Response to Reviewer UyYe cnt.
W3. Lack of inter-route interactions
Our study focuses on within-route coordination, rather than simultaneous control of interacting bus routes. However, our evaluation is not limited to a single route. SMART-HOLD is trained exclusively on R1 and evaluated without retraining on six routes, including five unseen routes with different lengths, service frequencies, and fleet sizes (Table 1, Section 5.1, page 8).
As shown in Table 2 (page 9), SMART-HOLD achieves the lowest mean Bunching, Headway CV, and EWT across all six routes, demonstrating its ability to generalize to different operating conditions.
We agree that inter-route coordination is a separate challenge, involving additional interactions such as shared-stop congestion and passenger transfers. These interactions are beyond the scope of our current study. We have revised the Future Work discussion to explicitly acknowledge this limitation and identify multi-route coordination as an important direction for future research.
Response to Reviewer UyYe cnt.
Q1. Differences between RTDC and role-based MARL methods
Thank you for this question. Role-based MARL differentiates agents by their behavioral roles, whereas RTDC trains policies for different route regions and combines their knowledge into a single shared policy.
(1) Specialization. Role-based MARL methods, such as ROMA [1] and Role Play [2], learn agent behaviors through role representations. In contrast, RTDC does not assign behavioral roles to agents. Instead, it trains regional teacher policies using experiences collected from different parts of the bus route. All teachers share the same action space and learning objective, while each learns from its regional operating conditions.
(2) Coordination. RTDC periodically distills knowledge from regional teachers into a shared student policy, which is then transferred back to all teachers for further learning. This design enables knowledge sharing and inter-agent coordination across regions, rather than allowing regional policies to learn independently. Only the shared student policy is used during execution.
(3) Empirical benefits. To directly evaluate RTDC, we conducted an additional ablation that retains three regional teachers but removes distillation and policy synchronization. Under the same training interaction and DDPG-update budgets, RTDC reduces EWT by 44.4% on R1 and 41.0% on the unseen R3 route. It also achieves lower Bunching, Headway CV, and EWT across all six routes.
These results demonstrate that RTDC improves regional coordination through knowledge sharing while producing a single policy that generalizes across different operating conditions.
References
[1] Wang et al. ROMA: Multi-Agent Reinforcement Learning with Emergent Roles. ICML, 2020. https://proceedings.mlr.press/v119/wang20f.html
[2] Long et al. Role Play: Learning Adaptive Role-Specific Strategies in Multi-Agent Interactions. Knowledge-Based Systems, 2025. https://doi.org/10.1016/j.knosys.2025.113811
Response to Reviewer UyYe
W1. Limited novelty
We thank the reviewer for this comment. While SMART-HOLD builds on established MARL techniques, its novelty lies in explicitly modeling the spatial, temporal, and asymmetric interactions of asynchronous bus control through three complementary designs.
(1) Fleet-wide and temporal state representations. Unlike existing BHC methods such as CAAC, which use local actor observations and focus on intervening decision events, FSR provides fleet-wide information for action selection, while TSR incorporates both intervening events and passive buses into critic learning. Together, they capture complementary spatial and temporal information about fleet interactions.
(2) Role-separated cross-attention. Rather than treating other buses as a single group, SMART-HOLD explicitly distinguishes upstream and downstream buses in the actor, and active upstream, active downstream, and passive buses in the critic. This allows the networks to model their different influences on holding decisions and value estimation.
(3) Regionalized training with distilled coordination (RTDC). RTDC enables regional teachers to learn from different operating conditions while periodically consolidating their knowledge into a single shared student policy. This promotes coordination across regions without requiring separate regional policies during execution.
Empirical evidence. The ablation results in Table 7 (Appendix H.3, page 24) demonstrate the benefits of these designs. Compared with unified attention, SMART-HOLD reduces EWT by 36.7% on R1 and 40.0% on R3. Further ablations confirm the benefits of fleet-wide observations and temporally augmentated state representation. Our additional RTDC ablation also shows 44.4% and 41.0% lower EWT on R1 and R3, respectively, compared with independently trained regional policies.
These results demonstrate that the proposed designs provide measurable benefits beyond simply applying established MARL techniques. We have clarified these specific contributions in the revised manuscript.