- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the multi-objective traffic signal control problem and proposes a Pareto-Consistent Preference Learning framework to address the representation collapse issue inherent in traditional scalar reinforcement learning. It introduces a two-stage approach: first, utilizing LLMs for offline preference annotation to train a structured multi-head representation network; second, applying Proximal Policy Optimization (PPO) with a mixed reward function to optimize policies across multiple objectives. Experiments demonstrate that the method achieves competitive performance compared to existing baselines.
Critical strengths
- It proposes a structured multi-head representation that effectively addresses the representation collapse issue prevalent in traditional scalar reinforcement learning methods.
- Comprehensive ablation experiments are provided to verify the effectiveness of individual modules and loss functions.
- It features a decoupled architecture that separates offline representation learning from online RL training, successfully transferring LLM knowledge into the policy network without incurring online inference overhead.
Critical weaknesses
- The chosen traditional DRL baselines are relatively outdated, dating back four years or more (such as PressLight and CoLight from 2019-2020). The authors fail to compare their model against contemporary state-of-the-art DRL frameworks—especially recent top-tier conference baselines like CoSLight (KDD 2024) or X-Light (IJCAI 2024)—which severely weakens the persuasiveness of the experimental results.
- The efficiency evaluation only fixes the PPO step count while completely ignoring significant upstream costs, including LLM API expenses and offline representation training overhead.
- Compressing current traffic states into pure text for LLM queries leads to the loss of critical microscopic details. Furthermore, general-purpose LLMs lack formal knowledge of traffic flow theory and physical simulation capabilities, making their preference judgments questionable.
Questions for authors
- How does the proposed method compare against contemporary state-of-the-art TSC baselines from recent AI venues (e.g., CoSLight from KDD 2024 or X-Light from IJCAI 2024), rather than relying solely on 4-year-old traditional models like PressLight or CoLight?
- Can the authors provide a total cost analysis including LLM API expenses and offline pre-training overhead?
- How does the text-based state compression affect microscopic traffic dynamics?
2 comments
I have carefully reviewed the additional results in Tables 11–13 and 16. These experiments provide useful clarification but do not fully convince me of the method's practical advantages. I therefore cannot recommend acceptance and will maintain my original score of 2.
Response to Reviewer pAmv
1. Recent traffic controllers
CoSLight and X-Light are relevant comparisons, and neither appears in Table 3. Advanced-CoLight is a different method. We therefore make no claim of outperforming CoSLight or X-Light. Against Weighted-RL, Table 16 shows where the practical differences lie. In Hangzhou, fairness rises from 0.7913 to 0.8064 and queue oscillation falls from 0.8694 to 0.8426. Both measures also favor our controller in Jinan and New York. The mean travel-time differences are small, and all three paired intervals include zero.
2. Cost of the offline pipeline
The PPO step count leaves out the cost of constructing the traffic-quality signal. In the main Hybrid setting, we sent 16,500 distinct preference comparisons to the LLM three times each, giving 49,500 requests. Voting and filtering retained 16,000 LLM labels. The remaining 32,000 objective-specific labels came from rules and required no API calls (Appendix A.7). Collecting and preprocessing the states and training the representation also incur costs before PPO starts.
Table 12 quantifies what the annotation budget buys. On Jinan, increasing the budget from 500 to 1,000 pairs per scenario doubled the requests from 24,750 to 49,500 at the same policy-training budget. Preference accuracy rose from 0.873 to 0.890 and hypervolume from 0.701 to 0.720. For policy training, the shared controller used one million environment steps. Weighted-RL used one million for each of five policies. We can quantify this saving in simulator interactions. The submission gives neither an itemized API bill nor total preprocessing and training time, so we withdraw any claim of lower end-to-end cost. At deployment the controller selects actions without calling the LLM.
3. Information lost in the traffic summaries
The loss of detail occurs when the observation is reduced to a set of fields. Two vehicle configurations can have the same queue length and TTC percentile while differing in which vehicles are approaching one another. The text supplied to the LLM preserves the reported fields, including signal phase, elapsed phase time, per-approach queues, throughput and braking indicators. The simulator advances the traffic dynamics. The LLM makes an offline comparison between two recorded observations.
We examined how much the judgments depend on those fields. Removing the safety fields lowers preference accuracy from 0.890 to 0.858. Adding noise at 10% of each field's training range lowers it to 0.844 (Table 13). For the labels themselves, two annotators independently judged a held-out audit set and a third resolved disagreements. The LLM agrees with the human judgment on 162/180 preference-conditioned comparisons and 101/120 reversal-heavy comparisons. The corresponding rule counts are 152/180 and 91/120. On objective-specific comparisons the rules do slightly better, 172/180 versus 170/180 (Table 11). This is why we use rules for the individual objectives and LLM judgments for trade-offs.
Each retained preference label requires agreement from at least two of three queries, including for a Tie. An A/B/Tie split is discarded. The 19 disagreements on the 120 reversal-heavy audit cases show where that procedure still makes mistakes. Because the annotators see the same summaries, their agreement measures the reliability of a judgment given those fields. It cannot detect a hazard omitted from the summary altogether.