acceptodds
Under review as a conference paper at ICLR 2027

Learning to Hold: Fleet-Aware and Temporally Augmented Multi-Agent Reinforcement Learning for Bus Control

Abstract

Effective bus holding control (BHC) is a real-time multi-agent decision-making problem where buses must coordinate under directional, time-varying, and partially observable fleet interactions. Existing BHC methods often rely on local observations, limited temporal context, or centralized training schemes that scale poorly with fleet size. We propose SMART-HOLD, a scalable multi-agent reinforcement learning approach for dynamic BHC. SMART-HOLD introduces a Fleet-wide State Representation that allows each bus agent to reason over fleet-level spatial structure, and a critic-only Temporally-Augmented State Representation that captures short-term system evolution for more accurate value estimation without leaking future information to the policy. A cross-attention actor-critic architecture selectively models subject, active, and passive agents to capture directional and role-dependent interactions. For scalable coordinated training, a Regionalized Training with Distilled Coordination strategy is developed to train multiple regional teacher policies in parallel and distills them into a shared actor for decentralized execution. Experiments on six real-world bus routes, including unseen routes with different service frequencies, fleet sizes, and temporal variability, show that SMART-HOLD substantially reduces bunching and improves service reliability over state-of-the-art BHC and multi-agent learning baselines.

Primary area:
optimization
Keywords:
MARL, bus holding control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UyYeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SMART-HOLD, a scalable MARL solution for dynamic bus holding control. Experiments on six real-world bus routes illustrate the efficiency of the proposed method.

Critical strengths
  1. The motivation of the whole design is clear and direct.

  2. The structure and writing of this paper are good.

Critical weaknesses
  1. The innovation of this paper is weak. Even though the task-oriented design is proper and good, almost all modules come from mature techniques.
  2. The competing approaches are very naive, without comparison to recent SOTA solutions.
  3. The paper only considers a single route, neglecting the inter-route impact.
Questions for authors

The RTDC solution is similar to the role-play MARL methods. What are their core differences, and what are the benefits of RTDC?

4 comments
AuthorsOpenReview ↗

Response to Reviewer UyYe

W1. Limited novelty

We thank the reviewer for this comment. While SMART-HOLD builds on established MARL techniques, its novelty lies in explicitly modeling the spatial, temporal, and asymmetric interactions of asynchronous bus control through three complementary designs.

(1) Fleet-wide and temporal state representations. Unlike existing BHC methods such as CAAC, which use local actor observations and focus on intervening decision events, FSR provides fleet-wide information for action selection, while TSR incorporates both intervening events and passive buses into critic learning. Together, they capture complementary spatial and temporal information about fleet interactions.

(2) Role-separated cross-attention. Rather than treating other buses as a single group, SMART-HOLD explicitly distinguishes upstream and downstream buses in the actor, and active upstream, active downstream, and passive buses in the critic. This allows the networks to model their different influences on holding decisions and value estimation.

(3) Regionalized training with distilled coordination (RTDC). RTDC enables regional teachers to learn from different operating conditions while periodically consolidating their knowledge into a single shared student policy. This promotes coordination across regions without requiring separate regional policies during execution.

Empirical evidence. The ablation results in Table 7 (Appendix H.3, page 24) demonstrate the benefits of these designs. Compared with unified attention, SMART-HOLD reduces EWT by 36.7% on R1 and 40.0% on R3. Further ablations confirm the benefits of fleet-wide observations and temporally augmentated state representation. Our additional RTDC ablation also shows 44.4% and 41.0% lower EWT on R1 and R3, respectively, compared with independently trained regional policies.

These results demonstrate that the proposed designs provide measurable benefits beyond simply applying established MARL techniques. We have clarified these specific contributions in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer UyYe cnt.

W2. Insufficient comparison with recent SOTA methods

Our existing evaluation includes CAAC, a specialized asynchronous MARL method for bus holding control, together with established MARL methods MADDPG and MAPPO. SMART-HOLD achieves the lowest mean Bunching, Headway CV, and EWT across all six routes. To further strengthen the evaluation, we introduce an additional baseline following HASAC (Liu et al., ICLR 2024), a recent maximum-entropy MARL method.

(1) New baseline: Heterogeneous SAC. We adapt the heterogeneous-policy learning idea of HASAC to our asynchronous BHC setting, where buses make decisions at different times and the number of operating buses varies. Specifically, we train k∈{3,5}k\in\{3,5\} stochastic actors with shared twin centralized critics. Each bus trip is assigned one actor in rotation, which controls the bus throughout that trip. The actors are trained using maximum-entropy SAC, without knowledge distillation.

We refer to these variants as Heterogeneous SAC-k3 and SAC-k5. Both variants are trained on R1 for 100 episodes and evaluated using the same simulator and evaluation procedure as the other learning-based methods.

(2) Experimental results. The following table compares SMART-HOLD with the two new baselines across all six routes. Results are reported as mean ± standard deviation. Lower values are better.

RouteMethodBunchingHeadway CVEWT (min)
R1SMART-HOLD1.45 ± 0.910.21 ± 0.020.35 ± 0.07
SAC-k31.99 ± 1.690.24 ± 0.010.43 ± 0.05
SAC-k52.36 ± 2.420.25 ± 0.010.46 ± 0.05
R3SMART-HOLD5.97 ± 3.640.24 ± 0.030.49 ± 0.10
SAC-k35.37 ± 2.440.27 ± 0.020.59 ± 0.07
SAC-k57.28 ± 3.500.29 ± 0.020.65 ± 0.07
M1SMART-HOLD0.00 ± 0.000.23 ± 0.030.55 ± 0.15
SAC-k30.08 ± 0.630.25 ± 0.020.66 ± 0.12
SAC-k50.18 ± 0.970.25 ± 0.030.64 ± 0.16
M2SMART-HOLD2.25 ± 5.770.27 ± 0.030.41 ± 0.08
SAC-k32.10 ± 2.370.29 ± 0.020.44 ± 0.05
SAC-k54.47 ± 3.900.29 ± 0.010.44 ± 0.04
M3SMART-HOLD0.27 ± 0.720.26 ± 0.020.42 ± 0.07
SAC-k38.02 ± 6.000.31 ± 0.020.59 ± 0.07
SAC-k512.06 ± 6.740.31 ± 0.020.61 ± 0.06
M7SMART-HOLD1.59 ± 3.110.25 ± 0.020.30 ± 0.03
SAC-k320.82 ± 7.570.33 ± 0.020.50 ± 0.05
SAC-k528.62 ± 8.590.33 ± 0.010.52 ± 0.04

SMART-HOLD achieves lower Headway CV and EWT than both heterogeneous SAC variants on all six routes, including five unseen routes. Averaged across routes, its relative EWT reductions are approximately 21% compared with SAC-k3 and 24% compared with SAC-k5. It also achieves fewer bunching events than SAC-k5 on every route and SAC-k3 on four of the six routes, with SAC-k3 achieving marginally fewer bunching events on R3 and M2.

(3) Implications for policy sharing. These results provide further evidence for the effectiveness of SMART-HOLD's shared-policy design. Unlike the heterogeneous SAC variants, which maintain multiple policies during execution, SMART-HOLD uses regional teachers during training and consolidates their knowledge into a single student policy. This enables coordinated decision-making across buses without maintaining separate execution policies.

The consistent improvements on unseen routes support the effectiveness of this design under varying operating conditions and fleet sizes. We have included the new baseline results and implementation details in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer UyYe cnt.

W3. Lack of inter-route interactions

Our study focuses on within-route coordination, rather than simultaneous control of interacting bus routes. However, our evaluation is not limited to a single route. SMART-HOLD is trained exclusively on R1 and evaluated without retraining on six routes, including five unseen routes with different lengths, service frequencies, and fleet sizes (Table 1, Section 5.1, page 8).

As shown in Table 2 (page 9), SMART-HOLD achieves the lowest mean Bunching, Headway CV, and EWT across all six routes, demonstrating its ability to generalize to different operating conditions.

We agree that inter-route coordination is a separate challenge, involving additional interactions such as shared-stop congestion and passenger transfers. These interactions are beyond the scope of our current study. We have revised the Future Work discussion to explicitly acknowledge this limitation and identify multi-route coordination as an important direction for future research.

AuthorsOpenReview ↗

Response to Reviewer UyYe cnt.

Q1. Differences between RTDC and role-based MARL methods

Thank you for this question. Role-based MARL differentiates agents by their behavioral roles, whereas RTDC trains policies for different route regions and combines their knowledge into a single shared policy.

(1) Specialization. Role-based MARL methods, such as ROMA [1] and Role Play [2], learn agent behaviors through role representations. In contrast, RTDC does not assign behavioral roles to agents. Instead, it trains regional teacher policies using experiences collected from different parts of the bus route. All teachers share the same action space and learning objective, while each learns from its regional operating conditions.

(2) Coordination. RTDC periodically distills knowledge from regional teachers into a shared student policy, which is then transferred back to all teachers for further learning. This design enables knowledge sharing and inter-agent coordination across regions, rather than allowing regional policies to learn independently. Only the shared student policy is used during execution.

(3) Empirical benefits. To directly evaluate RTDC, we conducted an additional ablation that retains three regional teachers but removes distillation and policy synchronization. Under the same training interaction and DDPG-update budgets, RTDC reduces EWT by 44.4% on R1 and 41.0% on the unseen R3 route. It also achieves lower Bunching, Headway CV, and EWT across all six routes.

These results demonstrate that RTDC improves regional coordination through knowledge sharing while producing a single policy that generalizes across different operating conditions.

References

[1] Wang et al. ROMA: Multi-Agent Reinforcement Learning with Emergent Roles. ICML, 2020. https://proceedings.mlr.press/v119/wang20f.html

[2] Long et al. Role Play: Learning Adaptive Role-Specific Strategies in Multi-Agent Interactions. Knowledge-Based Systems, 2025. https://doi.org/10.1016/j.knosys.2025.113811

Reviewer_SA8wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SMART-HOLD for continuous, asynchronous bus holding control. It combines fleet-wide actor observations, a temporally augmented critic, role-separated cross-attention, and regional teacher policies periodically distilled into a shared student. Simulation experiments using six real-world route configurations report improved headway regularity, excess waiting time, and bunching performance, including transfer from one training route to five unseen routes.

Critical strengths
  1. The framework combines a fleet-aware actor with a critic that explicitly separates subject, active, and passive buses and their directional interactions. This task-informed architecture addresses the asynchronous structure of bus holding, while regional-to-student distillation provides a mechanism for consolidating regional policies into one deployable actor.

  2. The TSR design in Section 4.1.2 lacks sufficient justification for its use in policy optimization. Future events can depend on the subject’s holding action, including which buses become active. However, the described actor update replaces the subject action while retaining the recorded future features. It is unclear why this produces an appropriate action-value gradient. Keeping future information out of the actor ensures causal execution, but does not by itself resolve the training concern associated with action-dependent hindsight conditioning.Training on one route and evaluating on five unseen routes tests transfer across different route lengths, service frequencies, and fleet sizes. The reported results show the lowest mean headway CV and excess waiting time across all six routes. Component ablations also support the usefulness of fleet-wide actor information and direction-separated attention.

Critical weaknesses
  1. The TSR design in Section 4.1.2 lacks sufficient justification for its use in policy optimization. Future events can depend on the subject’s holding action, including which buses become active. However, the described actor update replaces the subject action while retaining the recorded future features. It is unclear why this produces an appropriate action-value gradient. Keeping future information out of the actor ensures causal execution, but does not by itself resolve the training concern associated with action-dependent hindsight conditioning.

  2. The RTDC procedure in Section 4.3 differs materially from Appendix F. The former collects experience using regional teachers and distills recorded actions, whereas the latter collects experience using the student and distills current teacher outputs. These choices change exploration and the supervised targets. The authors should reconcile the descriptions and identify the protocol used for the reported results, particularly the distillation ablation.

Questions for authors
  1. During TSR-based actor updates, are future features and active/passive memberships recomputed for the candidate holding action? If they remain fixed, what assumptions or argument justify the resulting action gradient?

  2. Which RTDC protocol generated the main and ablation results: teacher or student rollouts, and recorded replay actions or current teacher outputs as distillation labels? Please reconcile the two algorithm descriptions.

4 comments
AuthorsOpenReview ↗

Response to Reviewer SA8w

W1 & Q1. Justification of TSR-based policy optimization

We thank the reviewer for this important question. We clarify how TSR is used during actor updates and why the resulting gradient is an approximation.

(1) Actor update. During training, the critic uses recorded future states and actions to capture asynchronous fleet interactions. When updating the actor, we replace the subject bus's recorded action with the action predicted by the current actor, while keeping other buses' subsequent states, actions, and roles fixed. These are not recomputed for the new holding action.

(2) Justification and limitations. The gradient guides the actor to improve its holding decision based on the recorded future events. However, changing the holding action may also affect these events, including which buses make subsequent decisions. The resulting gradient therefore approximates the effect of changing the holding action, rather than capturing its full impact on future events. This approximation is more reasonable when small changes in holding time have limited effects on subsequent fleet interactions. We acknowledge that this condition is not always guaranteed and that the resulting gradient may be biased.

Importantly, the actor uses only information available at the current decision time, while TSR is used exclusively by the critic during training. This ensures causal execution but does not eliminate potential gradient bias.

(3) Empirical evidence. The ablation results in Table 7 (Appendix H.3, page 24) support the practical benefits of TSR. Compared with the single-snapshot critic, TSR reduces Headway CV from 0.2372 to 0.2118 and EWT from 0.4527 to 0.3493 minutes on R1. Similar improvements are observed on the unseen R3 route, where Headway CV decreases from 0.2529 to 0.2436 and EWT from 0.5389 to 0.4912 minutes.

These results demonstrate the empirical benefits of temporal augmentation, although they do not establish an unbiased policy gradient. We have clarified the approximation and its limitations in Section 4.1.2 and Appendix F.

AuthorsOpenReview ↗

Response to Reviewer SA8w cnt.

W2 & Q2. Inconsistent RTDC training protocols

Thank you for identifying these inconsistencies. We clarify the RTDC protocol used in our main experiments and ablation studies.

(1) Experience collection and teacher updates. At each decision step, the bus selects its holding action using the teacher actor assigned to its current region. The resulting transition is stored in the corresponding regional replay buffer. Each teacher has its own actor and target actor, while all teachers share a centralized critic and its target network. The teachers are trained using DDPG, with 128 updates in total per episode. The student actor does not collect training experience.

(2) Knowledge distillation. Distillation is performed every five episodes, starting at episode 5. Observations are sampled approximately equally from the regional replay buffers. Each teacher generates target actions for observations from its region using its current policy, rather than using recorded replay actions. The student is trained for 25 gradient steps and then copied to all regional teachers and their target actors. The best-performing distilled student is retained for final evaluation.

(3) Evaluation and ablation results. The main results and ablations retaining RTDC are evaluated using the distilled student policy. In contrast, the training curves in Figure 6 (Appendix I, page 26) report the performance of the policies used to collect training experience, corresponding to regional teacher policies when k>1k>1.

We acknowledge the inconsistencies in the original descriptions. We have revised Section 4.3, Equation (11), Algorithm 1, and Algorithm 2 in Appendix F to reflect the actual training procedure. In particular, Algorithm 2 has been corrected to specify 128 DDPG updates in total per episode, with distillation beginning at episode 5, rather than episode 25.

Thank you for your detailed response. All my concerns have been adequately addressed.

Thank you for your prompt response and thoughtful feedback.

Reviewer_nW79OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SMART-HOLD, a multi-agent reinforcement learning framework for bus holding control (BHC). It combines a fleet-wide state representation for the actor, a temporally augmented state representation for the critic, role- and direction-aware cross-attention, and a regional teacher-to-student training strategy called Regionalized Training with Distilled Coordination (RTDC). The method is trained on route R1 and evaluated on R1 and five other routes, including four routes introduced from the Metlink dataset. Across six routes, the authors report improvements in bus bunching, headway coefficient of variation, and excess waiting time over heuristic and MARL baselines.

Critical strengths

The paper addresses a practically important problem. Reliable bus headways affect both passenger waiting times and the stability of public transit operations, and the proposed approach is evaluated across routes with varied lengths, service frequencies, and fleet sizes. The reported improvements across several service metrics provide encouraging evidence that SMART-HOLD can be useful beyond a single route configuration. The method is evaluated with a substantial set of experiments and the separation of subject, active, and passive buses also gives the model a clear structure for representing interactions that differ by timing and direction.

Critical weaknesses
  1. The paper does not establish its positioning and motivation clearly enough. The related-work discussion characterizes prior methods broadly as falling short of real-time BHC, but it does not sufficiently distinguish their capabilities or identify the precise limitation that SMART-HOLD resolves. The claim that BHC is "not a conventional MARL problem" is also not fully substantiated: asynchronous decisions and partial observations are relevant characteristics, but the paper should explain more concretely why they require a different formulation from established MARL settings. Although the proposed representations and training strategy address fleet context, interaction structure, and scaling, their connection to the stated real-time deployment challenge remains unclear.
  2. Important details needed to assess and reproduce the experiments are either unclear or inconsistent. The paper gives limited information about how the simulation environment models travel times, passenger demand, dispatching, and disturbances, and how the four Metlink routes were collected and converted into simulator inputs. In addition, the training descriptions conflict: Appendix C.1 reports 128 gradient updates shared across three teachers and distillation beginning at episode 5, whereas Algorithm 2 specifies 256 updates per region and distillation beginning after episode 20 (with the first distillation at episode 25). These differences could materially affect training cost and results. Until the experimental protocol and the role of RTDC are made more precise, it is difficult to judge how much of the reported gain is attributable to the proposed coordination strategy and how readily the evaluation can be reproduced.
Questions for authors
  1. What simulator and operational assumptions were used for training and evaluation? Please specify how travel times, passenger arrivals or demand, dispatch schedules, route geometry, and stochastic variation are represented. Were the same environment, training budget, and tuning procedure used for all learning-based baselines?

  2. Could the authors clarify the exact RTDC procedure and reconcile the differences between Appendix C.1 and Algorithm 2 regarding the number of gradient updates and the first distillation episode? In particular, are the critics shared across regional teachers, and are the student targets taken from actions stored in replay buffers or from current teacher-policy outputs? A concise account of how RTDC improves coordination over independently trained regional policies would also help.

  3. How were the four newly introduced routes (M1, M2, M3, and M7) collected and processed into the simulation? Please clarify whether their preprocessing, route representation, service schedules, and operating assumptions follow the same protocol as R1 and R3, and whether the resulting data or conversion procedure will be made available.

6 comments
AuthorsOpenReview ↗

Response to Reviewer nW79

We thank the reviewer for this constructive comment. We clarify the positioning and motivation of SMART-HOLD from three perspectives.

W1. Unclear positioning, motivation, and connection to real-time BHC

(1) Distinction from prior work. While CAAC already considers asynchronous decisions and directional interactions, SMART-HOLD makes three key advances: (i) FSR provides fleet-wide observations for actor decisions rather than relying solely on local information; (ii) TSR incorporates intervening decisions and passive buses into critic learning, capturing both temporal and fleet-level interactions; and (iii) RTDC combines regional policy specialization with knowledge distillation to enable coordinated learning and a shared policy for execution. Role-separated cross-attention further captures the distinct influences of different bus groups.

(2) Motivation from event-driven MARL. Our design addresses two complementary challenges:

  • Spatial and temporal information: Modern real-time tracking systems [1] make fleet-level information increasingly accessible. However, a snapshot alone cannot capture subsequent decisions by other buses before the subject bus acts again. FSR provides fleet-wide information for action selection, while TSR incorporates intervening events into critic learning. This addresses the spatial and temporal characteristics of asynchronous BHC without requiring a fundamentally different MARL formulation.

  • Coordinated learning: BHC naturally involves homogeneous agents sharing the same action space and control objective, but operating under different conditions. Following the principle of parameter sharing [2], RTDC trains region-specific teachers and distills their knowledge into a single shared policy. This enables regional specialization during training while supporting coordinated execution across varying fleet configurations, without maintaining separate policies for individual buses.

(3) Connection to real-time deployment. SMART-HOLD executes only the shared student actor, using fleet information available at decision time. Neither future information from TSR nor regional teacher policies are required during execution. The measured inference time of 3.49 ms per decision demonstrates its computational feasibility for real-time BHC under our experimental conditions.

We have revised the Introduction and Related Work to clarify these distinctions and their connection to real-time BHC.

References

[1] Metlink. Metlink real-time information system set for overhaul. 2023. https://www.metlink.org.nz/news-and-updates/news/metlink-real-time-information-system-set-for-overhaul

[2] Terry et al. Revisiting Parameter Sharing in Multi-Agent Deep Reinforcement Learning. 2020. https://arxiv.org/abs/2005.13625

AuthorsOpenReview ↗

Response to Reviewer nW79 cnt.

W2 & Q1. Simulator and operational assumptions

All methods use the same event-driven Rust simulator, modeling one-directional bus operations along a looped route. Buses depart from the origin, serve all stops sequentially, and return to the origin to complete each trip. Each episode simulates a complete operating day. Training is conducted exclusively on R1 (46 stops, 17.4 km, 06:00–22:00, 58 scheduled services), followed by zero-shot evaluation on R1 and five unseen routes (R3, M1, M2, M3, and M7).

(1) Route geometry. Each route is represented as an ordered one-dimensional loop, with buses traveling in a fixed direction without overtaking. Bus and stop positions are normalized to [0,1][0,1], preserving their relative spacing. Physical segment distances are obtained using the route's total length. Detailed street-network geometry is not modeled.

(2) Travel times and stochastic variation. Each route has three speed parameters: mean speed μr\mu_r, trip-level standard deviation σrtrip\sigma_r^{\mathrm{trip}}, and segment-level standard deviation σrseg\sigma_r^{\mathrm{seg}}. At the beginning of each trip, a speed multiplier is sampled as:

mi∼N(1,σrtripμr),m_i \sim \mathcal{N}\left(1,\frac{\sigma_r^{\mathrm{trip}}}{\mu_r}\right), where the second parameter denotes the standard deviation, and mim_i is clipped to [0.7,1.3][0.7,1.3]. For each segment between consecutive stops, a temporary speed is independently sampled as:

v~i,s∼N(μr,σrseg)\tilde{v}_{i,s}\sim\mathcal{N}(\mu_r,\sigma_r^{\mathrm{seg}}).

The resulting segment speed is vi,s=miv~i,sv_{i,s}=m_i\tilde{v}_{i,s}, and travel time is computed from the segment distance and speed. Trip-level variation captures persistent speed differences between trips, while segment-level variation captures short-term fluctuations.

(3) Passenger service. The simulator approximates passenger service through a stochastic dwell time at each stop:

dpassenger∼Uniform(0,2) minutes.d_{\mathrm{passenger}}\sim\mathrm{Uniform}(0,2)\text{ minutes}.

The controller additionally selects a holding time in [0,2][0,2] minutes. The simulator does not explicitly model passenger arrivals, boarding, alighting, or vehicle capacity constraints. Instead, passenger service is approximated by a randomly sampled dwell time at each stop. Excess Waiting Time (EWT) is calculated from observed arrival headways as:

EWT=Var(h)2E[h],\mathrm{EWT}=\frac{\mathrm{Var}(h)}{2\mathbb{E}[h]}, assuming random passenger arrivals. It therefore estimates additional waiting due to irregular headways rather than directly simulating passenger waiting times.

(4) Dispatch schedules. Each route uses a fixed timetable specifying dispatch times in HH:MM format. At each scheduled time, a bus enters the origin, receives its stochastic passenger dwell time, and becomes eligible for a holding decision.

(5) Experimental consistency and fairness. All methods are trained and evaluated in the same simulation environment, using consistent reward functions, action limits, and random seeds. Each learning method is trained on R1 for 100 full-day episodes with 10 independent training seeds. Each trained policy is then evaluated over 10 full-day episodes using the same evaluation seeds, yielding 100 evaluation runs per method per route. Exploration noise is disabled during evaluation.

For off-policy methods, we use a replay capacity of 20,000 transitions per buffer, a batch size of 256, γ=0.99\gamma=0.99, τ=0.001\tau=0.001, and actor and critic learning rates of 10−410^{-4} and 3×10−43\times10^{-4}, respectively. During training, Gaussian exploration noise is added to the holding actions, with its standard deviation decreasing from an initial value of 0.30 minutes to a minimum of 0.15 minutes.

Training exposure is matched by episode count, while optimization budgets remain algorithm-specific. SMART-HOLD, CAAC, IAC (DDPG-MLP), and MADDPG perform 128 optimization iterations per episode; for SMART-HOLD, these are distributed across the regional teachers. MAPPO instead performs up to 10 PPO epochs per episode, with minibatches of 256 and KL-based early stopping. SMART-HOLD additionally performs 25 behavioral-cloning updates every five episodes.

For each method, we use the same hyperparameter settings across all 10 training seeds. All policies are trained exclusively on R1, with no further training or hyperparameter tuning on the five unseen test routes.

AuthorsOpenReview ↗

Response to Reviewer nW79 cnt.

W2 & Q2. RTDC training protocol and its contribution

Thank you for identifying these inconsistencies. We clarify the training protocol used in our experiments and provide additional ablation results to demonstrate the contribution of RTDC.

(1) RTDC training protocol. RTDC uses three regional teacher actors, each with its own target actor and replay buffer. All teachers share one centralized critic and its target network. At each decision event, the bus selects an action using the teacher assigned to its current region. After each episode, 128 DDPG update steps are performed in total across the three teachers.

Knowledge distillation occurs every five episodes, starting at episode 5. During distillation, observations are sampled approximately equally from the regional replay buffers. Each teacher generates target actions for observations from its region using its current policy, rather than using recorded replay actions. The student is trained for 25 gradient steps, evaluated without exploration, and then copied to all regional teachers and their target actors. The best-performing student is retained for deployment.

We will correct the inconsistent descriptions in Section 4.3 and Appendix F. Specifically, regional teachers collect training experience. During distillation, their current policies generate the target actions for the student. Training involves 128 DDPG updates in total per episode, with distillation starting at episode 5. We will revise Section 4.3, Equation (11), Algorithm 1, and Algorithm 2 in Appendix F to ensure that they consistently describe the training procedure used in experiments.

(2) Contribution of RTDC. To directly evaluate the benefits of knowledge distillation and policy synchronization, we conducted an additional ablation that retains the same three regional teachers and shared critic but disables distillation and synchronization. Each regional teacher continues learning without periodic policy sharing, and the three regional policies are used directly during execution.

Training performance. Both variants are evaluated on R1 every five episodes using the same three validation seeds. At episode 50, RTDC reduces Bunching from 13.03 to 1.30 and EWT from 1.04 to 0.64 minutes. By episode 100, RTDC achieves 1.40 versus 8.47 Bunching events and 0.55 versus 0.69 minutes EWT. These results show that distillation improves learning performance and coordination across regions.

Testing performance. The following results are reported as mean ± standard deviation over 10 independently trained policies, each evaluated using 10 common evaluation seeds. Lower values are better.

RouteMethodBunchingHeadway CVEWT (min)
R1SMART-k31.45 ± 0.910.21 ± 0.020.35 ± 0.07
Independent regional7.17 ± 14.020.27 ± 0.090.63 ± 0.45
R3SMART-k35.97 ± 3.640.24 ± 0.030.49 ± 0.10
Independent regional15.80 ± 28.200.31 ± 0.100.83 ± 0.61
M1SMART-k30.00 ± 0.000.23 ± 0.030.55 ± 0.15
Independent regional2.28 ± 6.170.33 ± 0.091.17 ± 0.66
M2SMART-k32.25 ± 5.770.27 ± 0.030.41 ± 0.08
Independent regional17.60 ± 41.260.32 ± 0.090.60 ± 0.42
M3SMART-k30.27 ± 0.720.26 ± 0.020.42 ± 0.07
Independent regional10.39 ± 21.670.33 ± 0.110.74 ± 0.52
M7SMART-k31.59 ± 3.110.25 ± 0.020.30 ± 0.03
Independent regional30.04 ± 25.440.32 ± 0.090.53 ± 0.32

SMART-HOLD achieves lower Bunching, Headway CV, and EWT on all six routes, including five unseen routes. For example, compared with regional training without distillation, RTDC reduces EWT by 44.4% on R1 and 41.0% on R3. These results demonstrate the benefit of periodically sharing knowledge across regional policies rather than training them without policy synchronization.

(3) Training budget and computational cost. Both variants train for 100 R1 days with similar interactions and 12,800 DDPG updates, using the same checkpoint criterion. RTDC adds 500 replay-based distillation updates with no new training interactions; SMART-k3 adds 20 evaluation rollouts for checkpoint selection.

Training budgetSMART-k3Independent regional
Training episodes100100
Training interactions258,539 ± 612258,671 ± 370
DDPG updates12,80012,800
Distillation updates5000
Routine validation rollouts6060
Additional student-selection rollouts200
Training time (min)24.3 ± 2.022.4 ± 1.3

RTDC improves performance across all six routes with approximately 8% additional computational cost, demonstrating effective coordination of regional learning through periodic knowledge distillation at a modest additional training cost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.