acceptodds
Withdrawn from ICLR 2027

Trajectory-Conditioned Objective for Multi-Intersection Traffic Signal Control

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-intersection traffic signal control requires balancing feedback relevance with spatiotemporal coverage. Local, region-shared, and global rewards aggregate costs over spatial sets whose contributing vehicles change over time. Motivated by the observation that vehicle trajectories link feedback for the same vehicles over time and across intersections, we propose a Trajectory-Conditioned Objective (TCO). TCO aggregates queueing costs along the realized trajectories of vehicles currently observed at an intersection within a bounded spatiotemporal horizon. Experiments on real-world road networks show that TCO, as a simple plug-and-play objective, consistently improves representative baselines without otherwise changing their settings and remains robust across traffic loads and network scales.

Primary area:
reinforcement learning
Keywords:
Traffic Signal Control, Multi-Agent Reinforcement Learning, Reward Design, Intelligent Transportation Systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NpfrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TCO, which constructs an intersection’s reward from the subsequent queueing costs of vehicles currently observed there. Vehicle identities determine whose costs are included, while temporal and intersection-hop limits bound the trajectory window. Rewards are computed retrospectively and used with existing learners. Experiments report improvements across three controllers and eight traffic settings.

Critical strengths

1.Table 1 uses ten seeds and reports improvements in all 24 controller–scenario combinations. The paired-run results make this stronger than a comparison of means alone. 2.The method is simple, and the reward-scaling control and partial-coverage experiments address useful practical questions.

Critical weaknesses
  1. The trajectory-based rewards depend on subsequent actions, while the training procedure on page 5 retains the original TD/PPO updates. Please clarify the training objective and how these updates relate to the objective defined in Eq. (6).
  2. Table 6 reports lower ATT for K=3 than K=0 in seven of eight settings, but does not specify seed counts or uncertainty estimates. Reporting paired ATT differences across seeds, with confidence intervals, would help assess whether these gains are consistent across runs.
  3. As stated in Appendix A.2, training and evaluation use the same traffic flow file. Evaluation on independently generated arrival patterns or routes is not reported, leaving generalization to unseen traffic realizations unclear.
Questions for authors
  1. Could you clarify how the TD targets and PPO advantages are computed with TCO rewards, and how these updates relate to the objective in Eq. (6)?
  2. How many seeds were used for Table 6? Could you report the paired ATT differences between K=0 and K=3, and check whether the gains persist under the main H=24 setting?
  3. Do TCO’s gains persist when the trained models are evaluated on independently generated traffic flows with the same demand level but different arrival times or routes?
Reviewer_uwAqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Trajectory-Conditioned Objective (TCO), a training-time reward construction for multi-intersection traffic signal control. For each intersection and control step, TCO takes the vehicles currently on the incoming lanes. It sums their discounted queueing costs along their realized trajectories. The trajectory window is bounded by a maximum number of vehicle records and a maximum number of intersection hops, with a separate trajectory discount. Rewards are computed at episode end from logged vehicle identities and stored with the transitions. The controller architecture, observations, and update rule stay unchanged, and execution remains decentralized.

The theoretical content is a single proposition. It shows that the sum of TCO discounted returns over all agents equals a coverage-weighted discounted queueing cost, where the weight counts overlapping trajectory windows.

Experiments use CityFlow on the Jinan, Hangzhou, and Manhattan road networks under several demand levels. TCO is added to Advanced-MPLight, G2P-MPLight, and CoordLight. Under the main configuration, mean average travel time decreases for every backbone in every traffic setting. Additional experiments compare TCO with the Future-Region, Scaled Local, and gamma-Reward constructions. They also vary the trajectory horizon, hop limit, and trajectory discount, test partial trajectory coverage, vary network size on a controlled grid, and report fixed-policy mutual-information diagnostics.

Critical strengths

S1. The method is clearly designed. TCO changes only the reward labels written to replay or used for advantages. It keeps the backbone architecture, observations, and decentralized execution, and its training overhead is small, so it is easy to adopt in existing traffic signal control codebases. The construction is defined precisely, its degenerate cases connect it directly to the local reward, and the proposition is stated with its assumptions and a correct derivation. The paper is also clearly written, with notation defined once and reused.

S2. The experimental results are strong and consistent in current exp setting. Under the main configuration, TCO lowers mean travel time for every backbone in every traffic setting. The backbones cover different learning paradigms: independent Q-learning and PPO with a neighbor-aware critic. Results use multiple seeds with paired counts. The paper also controls for several obvious confounds, including the replay insertion schedule, reward magnitude, downstream feedback, and a fixed-region future reward, and compares against a re-implementation of gamma-Reward. Gains remain when only a small fraction of vehicles provide trajectory feedback.

Critical weaknesses

W1. The paper does not clearly explain where the improvement comes from, or which design component supports its central hypothesis. This is the main concern of the whole paper. The hypothesis is that following vehicles across intersections links each controller to the downstream costs its decisions cause. The ablations point elsewhere. When costs are restricted to the source intersection (hop limit of zero), TCO already recovers most of the gain over the local-reward control, about 55% to 85% in five of six Jinan and Hangzhou settings, and all of it on Jinan Moderate (See on tables 6 and 7). Cross-intersection tracing adds only a smaller increment on top. The evidence therefore suggests that the main benefit comes from extending the local objective over time, not from downstream tracing. The baselines needed to separate these effects are missing: a standard n-step or lam-return target on the local reward, and a future-local reward over all vehicles at the source intersection. The theory does not fill this gap. The main single proposition (Prop1) is an accounting identity over all agents, while each learner optimizes only its own return, and the coverage weights depend on the policy. So, I think the paper also does not analyze how a reward built from future actions interacts with bootstrapping and replay. Maybe some step-by-step decomposition would make the source of the gain explicit clearer.

W2. The experimental design needs improvement, and the interpretation of results should be more careful. On design: training and evaluation use the same flow file, the selection of the main hyperparameters is not described, and the ablations run on the same scenarios with different configurations and without standard deviations. Average travel time is the only metric, so hwo about worst situation, longest waiting time, throughput? Several flows appear to be custom-scaled rather than standard files, and the Manhattan Moderate flow may cover only a small subset of the network. Maybe these exp details should diccuss. On interpretation: the robustness claim holds only for the main configuration, since three lightweight pairs do not improve. This is not enough to build robustness of the method. The drop at the largest trajectory discount is attributed to equal weighting, but that setting also enlarges the reward scale by roughly an order of magnitude. The mutual-information diagnostic uses an estimator designed for continuous variables on a discrete action, under a deterministic policy, so it cannot be read as evidence of action and reward dependence. Comparisons with spatially discounted rewards (MA2C) and difference rewards are also missing.

Questions for authors
  1. Where does the gain come from? The current results suggest that a large part of the improvement may already come from extending the local objective over time at the source intersection. The paper does not yet make clear how much vehicle identity and cross-intersection tracing add on top of this. I hope the authors can design experiments that separate the contribution of each component, as this would make the central claim much more convincing.

  2. How does the reward interact with bootstrapping and replay? The reward already contains future costs that depend on later actions, and the learning target adds a bootstrapped value on top. Did you try bootstrapping from the end of the trajectory window, or limiting replay age? Beyond the accounting identity over all agents, is there any argument that each agent optimizing its own TCO return improves the network objective?

  3. How robust is the method beyond average travel time on the training flows? Can you report worst-case or tail results, the longest waiting time, and throughput? I think this is the common metrics in related area. How were the main hyperparameters chosen?

  4. How should the ablations be interpreted? The largest trajectory discount also enlarges the reward scale by roughly an order of magnitude, so is its drop due to equal weighting or to scale? Three lightweight pairs do not improve, so what supports the robustness claim beyond the main configuration? For the mutual-information diagnostic, which estimator handles the discrete action variable, and is there a shuffled-action null baseline?

If the authors can address the main concern, namely a clear explanation of where the gain comes from, I would be willing to raise my score.

Reviewer_NtqUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a trajectory-conditioned objective (TCO) for the RL in the multi-intersection traffic signal control. Instead of using the sum of local and global reward, the trajectory of each car unifies the local and global reward. On the other hand, their trajectory modeling method considers both spatial and temporal limits: the impact will decrease when the car is far aways from the start intersection over time. Using the plug-an-play objective, the authors conducted solid experiments on three datasets. The results show that the TCO improves representative baselines and remains robust across traffic loads and network scales.

Critical strengths
  1. The presentation is good. The authors have clarified each component of the reward function well.
  2. The experiments are solid. The authors have conducted comprehensive experiments for TCO in both the main paper and the appendix.
Critical weaknesses
  1. The novelty is limited. The paper "Multi-agent reinforcement learning for traffic light control, Marco Wiering, ICML 2000" uses each car's expected waiting time at this light and all later lights. But the authors neither cite nor compare against this work.

  2. The paper doesn't isolate what drives TCO's gain. In table 6, there is little difference between K=0 and K=3. When K=3, the result improves a little compared with K=0. So the "across intersection" idea seems to contribute little.

  3. Evaluation settings. The test uses the same traffic as training. Most Table 10 flows, all Hangzhou, Jinan Heavy/Extreme, both Manhattan, match none of the released benchmark files. The authors also do not show how the heavy, extreme are generated.

Questions for authors
  1. How do the authors choose the H,K and γτ? They are important parameters but none of the sweeps of table 6,7,13 includes main configuration.
  2. Can the authors show learning curves in appendix? CoordLight trains its variants for 10000 episodes in its own paper. But this paper trains only 500. This will show whether TCO converges to better result or mainly learns faster.
  3. Can the authors publish the code for reproducibility?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.