acceptodds
Under review as a conference paper at ICLR 2027

Pareto-Consistent Preference Learning for Traffic Signal Control

Abstract

Urban traffic signal control is inherently multi-objective, yet traffic quality is not directly available as a well-defined reward vector. Instead, it must be inferred from competing criteria such as efficiency, safety, fairness, and stability, and the preferred ordering of traffic states can reverse as preferences change. Methods based on preference-independent scalar rewards or learned scores reduce these competing criteria to a single implicit compromise. We formulate traffic signal control as multi-objective preference learning under the risk of representation collapse. Our method learns a structured vector-valued traffic-quality representation and a unified preference-conditioned policy family, rather than a preference-independent scalar score and a fixed controller. To supervise this representation, we construct deterministic, schema-constrained traffic abstractions and combine rule-based objective-specific comparisons with preference-conditioned comparisons generated offline by an LLM. We further encourage dominance consistency to preserve Pareto-relevant relational structure. Based on the learned representation, we train a single controller that can realize diverse preference-conditioned trade-offs without retraining for each preference or querying the LLM during deployment. Results on CityFlow benchmarks across multiple urban networks, unseen preferences, and shifted traffic conditions indicate competitive traffic performance and improvements in ordering preservation, preference alignment, and empirical trade-off quality within the evaluated settings.

Primary area:
applications to robotics, autonomy, planning
Keywords:
preference learning, multi-objective reinforcement learning, traffic signal control, representation learning, Pareto consistency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pAmvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the multi-objective traffic signal control problem and proposes a Pareto-Consistent Preference Learning framework to address the representation collapse issue inherent in traditional scalar reinforcement learning. It introduces a two-stage approach: first, utilizing LLMs for offline preference annotation to train a structured multi-head representation network; second, applying Proximal Policy Optimization (PPO) with a mixed reward function to optimize policies across multiple objectives. Experiments demonstrate that the method achieves competitive performance compared to existing baselines.

Critical strengths
  1. It proposes a structured multi-head representation that effectively addresses the representation collapse issue prevalent in traditional scalar reinforcement learning methods.
  2. Comprehensive ablation experiments are provided to verify the effectiveness of individual modules and loss functions.
  3. It features a decoupled architecture that separates offline representation learning from online RL training, successfully transferring LLM knowledge into the policy network without incurring online inference overhead.
Critical weaknesses
  1. The chosen traditional DRL baselines are relatively outdated, dating back four years or more (such as PressLight and CoLight from 2019-2020). The authors fail to compare their model against contemporary state-of-the-art DRL frameworks—especially recent top-tier conference baselines like CoSLight (KDD 2024) or X-Light (IJCAI 2024)—which severely weakens the persuasiveness of the experimental results.
  2. The efficiency evaluation only fixes the PPO step count while completely ignoring significant upstream costs, including LLM API expenses and offline representation training overhead.
  3. Compressing current traffic states into pure text for LLM queries leads to the loss of critical microscopic details. Furthermore, general-purpose LLMs lack formal knowledge of traffic flow theory and physical simulation capabilities, making their preference judgments questionable.
Questions for authors
  1. How does the proposed method compare against contemporary state-of-the-art TSC baselines from recent AI venues (e.g., CoSLight from KDD 2024 or X-Light from IJCAI 2024), rather than relying solely on 4-year-old traditional models like PressLight or CoLight?
  2. Can the authors provide a total cost analysis including LLM API expenses and offline pre-training overhead?
  3. How does the text-based state compression affect microscopic traffic dynamics?
2 comments
AuthorsOpenReview ↗

Response to Reviewer pAmv

1. Recent traffic controllers

CoSLight and X-Light are relevant comparisons, and neither appears in Table 3. Advanced-CoLight is a different method. We therefore make no claim of outperforming CoSLight or X-Light. Against Weighted-RL, Table 16 shows where the practical differences lie. In Hangzhou, fairness rises from 0.7913 to 0.8064 and queue oscillation falls from 0.8694 to 0.8426. Both measures also favor our controller in Jinan and New York. The mean travel-time differences are small, and all three paired intervals include zero.

2. Cost of the offline pipeline

The PPO step count leaves out the cost of constructing the traffic-quality signal. In the main Hybrid setting, we sent 16,500 distinct preference comparisons to the LLM three times each, giving 49,500 requests. Voting and filtering retained 16,000 LLM labels. The remaining 32,000 objective-specific labels came from rules and required no API calls (Appendix A.7). Collecting and preprocessing the states and training the representation also incur costs before PPO starts.

Table 12 quantifies what the annotation budget buys. On Jinan, increasing the budget from 500 to 1,000 pairs per scenario doubled the requests from 24,750 to 49,500 at the same policy-training budget. Preference accuracy rose from 0.873 to 0.890 and hypervolume from 0.701 to 0.720. For policy training, the shared controller used one million environment steps. Weighted-RL used one million for each of five policies. We can quantify this saving in simulator interactions. The submission gives neither an itemized API bill nor total preprocessing and training time, so we withdraw any claim of lower end-to-end cost. At deployment the controller selects actions without calling the LLM.

3. Information lost in the traffic summaries

The loss of detail occurs when the observation is reduced to a set of fields. Two vehicle configurations can have the same queue length and TTC percentile while differing in which vehicles are approaching one another. The text supplied to the LLM preserves the reported fields, including signal phase, elapsed phase time, per-approach queues, throughput and braking indicators. The simulator advances the traffic dynamics. The LLM makes an offline comparison between two recorded observations.

We examined how much the judgments depend on those fields. Removing the safety fields lowers preference accuracy from 0.890 to 0.858. Adding noise at 10% of each field's training range lowers it to 0.844 (Table 13). For the labels themselves, two annotators independently judged a held-out audit set and a third resolved disagreements. The LLM agrees with the human judgment on 162/180 preference-conditioned comparisons and 101/120 reversal-heavy comparisons. The corresponding rule counts are 152/180 and 91/120. On objective-specific comparisons the rules do slightly better, 172/180 versus 170/180 (Table 11). This is why we use rules for the individual objectives and LLM judgments for trade-offs.

Each retained preference label requires agreement from at least two of three queries, including for a Tie. An A/B/Tie split is discarded. The 19 disagreements on the 120 reversal-heavy audit cases show where that procedure still makes mistakes. Because the annotators see the same summaries, their agreement measures the reliability of a judgment given those fields. It cannot detect a hazard omitted from the summary altogether.

I have carefully reviewed the additional results in Tables 11–13 and 16. These experiments provide useful clarification but do not fully convince me of the method's practical advantages. I therefore cannot recommend acceptance and will maintain my original score of 2.

Reviewer_eo9aOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Pareto-consistent preference learning method for traffic signal control problem, using rule-based objective-specific comparisons and offline LLM-generated preference-conditioned comparisons to learn a multi-objective quality representation and train a preference-conditioned controller. Experiments on three real-world datasets report competitive traffic performance and improved trade-off quality across efficiency, safety, fairness, and stability under different preferences.

Critical strengths
  1. The paper combines structured multi-objective representation learning with Pareto- consistency constraints for traffic signal control. This provides a useful way to preserve competing objectives and support different preferences within a single controller.

  2. The paper includes extensive experiments on three traffic benchmarks, with ablation studies and tests on unseen preferences and changing traffic conditions. These experiments provide broad support for the proposed method.

Critical weaknesses
  1. The proposed framework mainly combines existing techniques, including multi-objective preference learning, vector-valued reward representations, Pareto dominance regularization, and preference-conditioned RL. Mu et al. [1] have already studied learning multi-objective reward models from preference feedback. Moreover, multi-objective optimization has been widely studied in TSC, including flexible Pareto-based control [2], network-wide multi-objective coordination [3], and joint optimization of safety and efficiency [4]. However, these closely related studies are not discussed in the paper. The proposed LLM-based supervision and dominance regularization appear to be relatively straightforward extensions, and the methodological novelty over existing approaches is unclear.

In addition, the practical motivation is not sufficiently convincing. The paper does not clearly explain when and why TSC requires dynamically adjusting the trade-off between safety and efficiency, particularly since basic traffic signal safety requirements are typically enforced through traffic rules and control constraints.

  1. The paper claims to address preference-dependent representation collapse, but does not report strict paired ranking-reversal accuracy to directly validate this claim. Although several representation diagnostics and ablations are provided, comparisons with preference-conditioned scalar reward models using the same supervision and training settings are still missing. Therefore, it is unclear whether the improvements mainly come from the proposed representation or additional supervision.

The preference alignment metric in Eq. (34) is also questionable. Preference weights represent the relative importance of different objectives, rather than the expected proportions of achieved objective values. In fact, this metric can assign a better score to a Pareto-dominated solution than to a dominating one. Therefore, the reported improvements in preference alignment do not necessarily indicate better preference satisfaction.

Finally, while the metrics of HV, Coverage, Fairness, and Oscillation improve, the ATT improvements over Weighted-RL are not statistically significant across the three networks, and TTC-P10 improves significantly only on Jinan. The results therefore provide limited evidence of additional practical benefits, and do not clearly establish that the multi-objective improvements are due to the proposed Pareto-consistent representation.

[1] Mu N, Luan Y, Jia Q S. Preference-based multi-objective reinforcement learning[J]. IEEE Transactions on Automation Science and Engineering, 2025, 22: 18737-18749. [2] Saiki T, Arai S. Flexible traffic signal control via multi-objective reinforcement learning[J]. IEEE Access, 2023, 11: 75875-75883. [3] Fang J, You Y, Xu M, et al. Multi-objective traffic signal control using network-wide agent coordinated reinforcement learning[J]. Expert Systems with Applications, 2023, 229: 120535. [4] Zhang G, Chang F, Jin J, et al. Multi-objective deep reinforcement learning approach for adaptive traffic signal control system with concurrent optimization of safety, efficiency, and decarbonization at intersections[J]. Accident Analysis & Prevention, 2024, 199: 107451.

Questions for authors
  1. What is the main methodological novelty compared with existing preference-based MORL methods, particularly Mu et al. [1], and previous multi-objective TSC studies [2–4]? How does the proposed structured representation differ from existing multi-objective reward learning approaches?

  2. Can the authors report strict paired ranking-reversal accuracy and compare with a preference-conditioned scalar reward model using the same preference supervision, model capacity, and RL training setup? This would verify whether the proposed representation is necessary to address representation collapse.

  3. How do the authors justify the preference alignment metric in Eq. (34), given that preference weights represent objective importance rather than the expected proportions of achieved objective values? Since this metric can favor Pareto-dominated solutions, would the conclusions change under other preference-satisfaction metrics?

  4. In what practical TSC scenarios is it necessary to dynamically adjust the preference weights between safety and efficiency? What advantages does the proposed formulation offer over existing multi-objective or safety-constrained TSC approaches?

[1] Mu N, Luan Y, Jia Q S. Preference-based multi-objective reinforcement learning[J]. IEEE Transactions on Automation Science and Engineering, 2025, 22: 18737-18749. [2] Saiki T, Arai S. Flexible traffic signal control via multi-objective reinforcement learning[J]. IEEE Access, 2023, 11: 75875-75883. [3] Fang J, You Y, Xu M, et al. Multi-objective traffic signal control using network-wide agent coordinated reinforcement learning[J]. Expert Systems with Applications, 2023, 229: 120535. [4] Zhang G, Chang F, Jin J, et al. Multi-objective deep reinforcement learning approach for adaptive traffic signal control system with concurrent optimization of safety, efficiency, and decarbonization at intersections[J]. Accident Analysis & Prevention, 2024, 199: 107451.

2 comments
AuthorsOpenReview ↗

Response to Reviewer eo9a

1. Contribution and relation to prior work

We agree that the advance over prior preference-based MORL is incremental. Mu et al. (2025) already learn a vector-valued reward from preferences over trajectory segments. Multi-objective signal control also has a substantial history. Saiki and Arai (2023) study policies for different traffic-flow ratios, Fang et al. (2023) study network-wide coordination, and Zhang et al. (2024) jointly optimize safety, efficiency, and emissions. These papers belong in our related work.

The traffic problem we address is how to learn from two kinds of judgment about the same pair of states. Queue and delay measurements can tell us which state scores better on efficiency. They leave open how to rank the pair when the faster state also has shorter times to collision. We use objective-specific comparisons to train four coordinates, preference comparisons to supervise their weighted sum, and a dominance loss on pairs where the objective labels favor one state on at least one coordinate and rate it at least as highly on every other coordinate. This gives the controller a common score for a specified preference while leaving each objective open to inspection.

Table 18 holds the architecture fixed. The objective-only model scores 0.929 on objective accuracy, slightly above the full model's 0.924, yet scores 0.818 on preference accuracy versus 0.890 for the full model. Table 1 separates the two added terms. Removing preference supervision lowers preference accuracy from 0.8897 to 0.8126. Removing the dominance loss lowers the fraction of held-out dominance relations preserved from 0.9148 to 0.8365. The benefit we can attribute to the design is that each type of supervision improves the corresponding judgment within this architecture.

2. Conditioned scalars and ranking reversals

A deterministic score of the state alone must give the same ordering under every preference. A scalar that also receives the preference can reverse that ordering. Our own readout, w⊤q(o)w^\top q(o), illustrates the point. To isolate the value of explicit coordinates, a scalar f(o,w)f(o,w) would need the same objective labels through f(o,ek)f(o,e_k), the same preference and dominance labels, and matched capacity and training budgets. Table 1 includes Cond-Scalar-RL, but it does not isolate architecture under all of these controls.

Strict reversal accuracy gives credit only when both predictions for the same held-out pair agree with its two opposite targets. Table 17 measures how often the targets reverse. The paper reports no aggregate strict model accuracy or matched advantage on that measure. We therefore limit our claim to the supervision effects above and the ranking behavior illustrated in Table 20. We withdraw the suggestion that explicit coordinates are necessary for a preference-conditioned model to reverse a ranking.

3. Preference satisfaction and Eq. (34)

Your objection to Align is correct. With equal weights, an outcome of (0.4,0.4,0.4,0.4)(0.4,0.4,0.4,0.4) has zero Align error. Raising its first coordinate to 0.8 improves that objective and leaves all others unchanged, yet the error rises to 0.30. We withdraw the preference-satisfaction claims based on Align, including the corresponding interpretation of calibration error.

Under our linear preference model, Eq. (33) scores the realized traffic outcome by w⊤zw^\top z, using common scales across methods within each city. It values the improvement in the example at 0.50 versus 0.40. Table 2 reports mean utility of 0.7288 for our method and 0.7057 for Weighted-RL. That is a descriptive difference. Table 15 addresses a separate question about the set of operating points. Across five paired seeds, the 95% intervals for improvement over Weighted-RL are [0.0117, 0.0647] for hypervolume and [0.0221, 0.1073] for Coverage.

4. Operating priorities and practical gains

An operator might emphasize clearing accumulated queues during a demand surge and later give greater weight to short time-to-collision events when selecting a plan. Conflicting movements must remain prohibited and clearance requirements still apply in both modes. The practical benefit is being able to select an operating point from one trained controller. Our evaluation covers different choices of preference across runs. It does not test abrupt switches within a run or establish a safety guarantee.

We agree with your reading of Table 16. Against Weighted-RL, the paired travel-time intervals include zero in all three cities. The improvement in the 10th percentile of time to collision (TTC-P10) reaches significance only in Jinan. The more consistent gains are in waiting-time fairness and queue oscillation, for which the metric-specific paired comparisons favor our method in all three cities.

I appreciate the authors’ detailed responses and clarifications. However, my main concerns remain unresolved. In particular, the lack of a controlled comparison with preference-conditioned scalar models leaves the advantage of the proposed structured representation unclear. The authors also acknowledge the flaw in the original preference alignment metric and withdraw the corresponding claims.

Overall, while the method shows some empirical benefits, I still find the methodological novelty and evidence supporting the central claims insufficient to meet the acceptance standards of this conference. Therefore, I must maintain my original assessment.

Reviewer_sPPeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies traffic signal control under changing preferences over efficiency, safety, fairness, and stability. It learns a four-dimensional traffic-quality representation using objective-specific comparisons, offline LLM-generated preference comparisons, and dominance regularization. A preference-conditioned controller is subsequently trained using this representation. Experiments on CityFlow evaluate conventional traffic performance, representation quality, preference response, and empirical trade-offs. The appendix additionally provides ablations, a human-label audit, and selected statistical analyses.

Critical strengths

(1) The paper addresses a meaningful practical problem that adapts traffic signal control to competing objectives and changing preferences.

(2) The representation diagnostics, targeted ablations, and held-out human-label audit provide useful evidence about the roles of different supervision components.

Critical weaknesses

(1) Although the appendix includes additional traffic controllers, preference-aware comparisons mainly use generic scalarization baselines and separately trained policies. Comparisons with established preference-controllable methods, such as the cited PCRL, would better establish the proposed method’s advantage.

(2) The central preference-preservation claim needs more direct validation. Strict paired-reversal accuracy is defined but not reported. Moreover, the Align metric assumes that realized objective proportions should match preference weights, an assumption that is insufficiently justified.

(3) Although the overall training objectives and PPO settings are provided, the concrete pairwise loss and external reward remain insufficiently specified. With no code available for review, these gaps limit reproducibility and independent verification.

(4) The problem formulation, motivation, and methodological design is insufficiently clear. Although formal definitions and component-wise ablations are provided, the limitation of a fixed scalar score does not by itself establish the need for the proposed design over existing preference-conditioned approaches. A concrete example tracing how the representation and supervision change ranking predictions and downstream control would help clarify the contribution of each design choice.

Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer sPPe

1. Comparison with PCRL

We agree that PCRL would test whether the resulting controller compares favorably with an established preference-controllable method. We have no head-to-head result with PCRL. The submitted policy comparison includes Cond-Scalar-RL and Weighted-RL. In the preference sweep, Table 7 measures how often the change in realized behavior follows the requested direction. The monotonicity scores are 0.9531 for our controller, 0.7837 for Cond-Scalar-RL, and 0.8468 for Weighted-RL. Our comparative claim is limited to these evaluated methods. A direct PCRL comparison would require a common interaction budget, evaluation preferences, and externally normalized traffic outcomes.

2. Direct reversal evidence and Align

You ask whether the model follows a change in preference on the same state pair. If the target favors A under the efficiency preference and B under the safety preference, Eq. (28) requires the model to predict A and then B. A tie or a switch in the wrong direction earns no credit. On held-out pairs, the targets reverse in 14% of cases in Jinan, 18% in Hangzhou, and 21% in New York (Table 17). The submission does not report the fraction of those reversals that the model gets right on both queries. The two training-pool examples below illustrate the behavior and one failure case.

We accept your criticism of Align. The weights specify how much the operator values each objective. They do not prescribe the proportions in which the outcomes should be achieved. We withdraw claims of preference satisfaction based on Eq. (34) or on calibration to those proportions. The weighted utility in Eq. (33) gives a direct evaluation under the linear preference assumption. Table 2 reports 0.7288 for our controller and 0.7057 for Weighted-RL. The ablation results on preference accuracy and dominance preservation (Table 1) and downstream hypervolume (Table 9) are computed independently of Align.

3. Comparison loss and external reward

For observations A and B, the signed label in Eqs. (10)–(11) is y=+1y=+1 for A, −1-1 for B, and 00 for a tie. Eq. (17) uses these signed objective labels to form dominance pairs. For the binary cross-entropy loss we use the target t=(y+1)/2t=(y+1)/2, so a retained tie has target 1/21/2. The score difference is Δk=qk(oA)−qk(oB)\Delta_k=q_k(o_A)-q_k(o_B) for objective kk and Δw=w⊤[q(oA)−q(oB)]\Delta_w=w^\top[q(o_A)-q(o_B)] for preference ww. Given either difference Δ\Delta, the loss is −tlog⁡p−(1−t)log⁡(1−p)-t\log p-(1-t)\log(1-p) with p=σ(Δ/τ)p=\sigma(\Delta/\tau). The dominance term in Eq. (18) is max⁡(0,m−min⁡kΔk)\max(0,m-\min_k\Delta_k). A positive margin also encourages separation on coordinates labeled as ties. This is a regularizer on the labeled pairs and gives no guarantee about the entire Pareto frontier.

During PPO the representation is frozen. Eq. (22) adds λ(t)w⊤q(ot+1)\lambda(t)w^\top q(o_{t+1}) to the external traffic reward rext(ot,at,ot+1)r_{\mathrm{ext}}(o_t,a_t,o_{t+1}). The preference changes the learned term. Its coefficient increases linearly from 0 to 0.4 over the first 30% of the one-million-step training budget (Appendix A.8). The submission leaves the external reward's exact formula and scale, the numerical temperature, and the dominance margin unspecified. Reproducing both training stages requires these details. Eq. (33) defines the evaluation utility separately.

4. How supervision changes a ranking and enters control

Pair S in Tables 19–20 makes the conflict concrete. Average travel time favors A (49.2 versus 53.8 seconds), while TTC-P10 favors B (2.31 versus 1.82 seconds). The efficiency-focused target favors A and the safety-focused target favors B. The fixed scalar predicts B twice. The objective-only multi-head model predicts A twice. The full model predicts A and then B. On Pair F the full model misses the efficiency-focused ranking.

The link to control is the next observation's contribution to the PPO reward. Under the efficiency preference, an action leading to a state with a higher efficiency score receives more credit from that coordinate. Table 9 tests the downstream effect across runs. Removing preference supervision reduces policy hypervolume from 0.7198 to 0.6448. Pair S illustrates how the learned ordering can change with the preference. The ablation measures how training with those comparisons affects the controller's set of outcomes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.