acceptodds
Under review as a conference paper at ICLR 2027

VLALIGHT: A Vision-Language-Action Model for Traffic Signal Control

Abstract

Traffic signal control (TSC) plays an important role in managing urban traffic flow and reducing congestion. In recent years, roadside cameras have become widely deployed at signalized intersections, providing continuous visual observations of traffic conditions. However, the potential of using these visual observations directly for traffic signal control remains largely unexplored. Most reinforcement learning and large language model (LLM)-based controllers still rely on complete, structured traffic states provided by simulators, which offer an idealized description of traffic conditions. Recent vision-language models (VLMs) introduce visual scene understanding, but commonly use VLMs as perception modules or high-level planners while delegating signal control to separate RL or language-model components. To bridge this gap, we present VLALight, an end-to-end vision-language-action (VLA) model for traffic signal control. Specifically, VLALight extracts traffic information from physical visual observations and reasons over dynamic scenes with multiple vehicles, lanes, and directions. It then incorporates information from neighbouring intersections to coordinate signal decisions across the road network. Besides, we introduce a cooperative reinforcement learning framework that leverages local and network-level traffic feedback to train a VLA agent that coordinates multi-intersection signal control and adapts its reasoning depth to traffic complexity. Extensive experiments on both synthetic and real-world traffic datasets demonstrate the effectiveness, scalability, and robustness of VLALight across diverse traffic scenarios.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language-Action Models, Traffic Signal Control, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RsceOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VLALight builds on Qwen3.5-4B for video-based traffic signal control. It converts roadside videos into structured traffic descriptions, combines them with neighboring-intersection information, and selects signal phases. Supervised initialization is followed by cooperative reinforcement learning, with Fast/Slow reasoning intended to balance control quality and inference cost. Evaluation covers seven traffic traces across three simulated urban networks.

Critical strengths
  1. The approach addresses a relevant perception-to-control problem. Combining roadside video observations with neighboring traffic information is well motivated, since local camera views alone may miss traffic approaching from connected intersections.
  2. The reported network-level results are promising. Experiments include a 196-intersection network, and the ablations support the usefulness of cooperative information and reinforcement learning beyond supervised initialization. The inference-time breakdown also provides a useful account of the cost of explicit reasoning.
Critical weaknesses
  1. The additional benefits of VLA decision-making and adaptive reasoning are not sufficiently established. The illustrated decisions prioritize demand, waiting duration, and approaching traffic, but the paper does not establish how the VLA improves on rules or a small policy network using these same factors. Its advantage over existing MARL controllers is similarly unclear: VLALight also relies on a shared policy, neighboring traffic information, and cooperative rewards, while the comparisons do not isolate language-based reasoning from differences in perception, supervised initialization, and training configuration. Furthermore, always-Fast and always-Slow experiments demonstrate differences in control quality and latency, but do not establish that learned switching outperforms simple rules or random routing. Its additional benefit at comparable measured inference cost remains unclear.
  2. The core RL mechanism is incompletely specified, and experimental stability is insufficiently documented. Fast/Slow prefixes are externally assigned during rollouts, but the loss and target construction used to update mode selection are not explicitly defined. Because candidates change actions at multiple intersections, an intersection’s Fast/Slow utility comparison is affected by decisions elsewhere. The paper does not explain how this dependence is accounted for when constructing mode-selection targets. Moreover, SFT dataset sizes and network splits, quality checks for teacher-generated Slow-mode demonstrations, and independent training and evaluation repeats are also inadequately reported.
Questions for authors
  1. With perception and cooperative inputs held fixed, how much does the VLA decision stage improve over a lightweight controller?
  2. What are the exact mode-selection loss and targets? How are local and network rewards used when other intersections’ actions also vary across candidates?
  3. Which networks and traces enter each SFT stage, how are teacher demonstrations screened, and how many independent training and evaluation runs underlie the main results?
4 comments

We thank the reviewer for recognizing the motivation for video-based cooperative control, the network-scale evaluation, and the inference-time analysis. Below, we clarify the evidence in our original submission and provide additional results addressing the remaining concerns.

[W1, Q1] What supports VLA decision-making over rules, lightweight policies, and MARL?

The research gap described in our original Introduction concerns both traffic perception and decision-making. Conventional rules use predefined priorities. Lightweight RL and MARL controllers learn task-specific mappings from structured states to signal actions. Language-based controllers introduce pretrained semantic and relational reasoning, but generally still assume that structured traffic states are available.

VLALight extends this approach by learning traffic perception directly from four-direction videos and generating coordinated signal actions with the same VLA backbone. The policy compares local queues, queue changes, service history, and upstream traffic when selecting a phase. Cooperative RL adapts these comparisons using actual local and network traffic outcomes. Adaptive Fast/Slow mode selection further controls when explicit reasoning is used.

Related work supports this motivation. LLMLight [1] studies language-based signal decisions and cross-network transfer. CoLLMLight [2] incorporates relations between connected intersections and trains with environmental feedback. Traffic-R1 [3] reports improved zero-shot control after RL adaptation of the same Qwen2.5 backbone. RT-2 [4] demonstrates the value of vision-language pretraining for action generalization. VLALight brings traffic perception, cooperative signal learning, and adaptive reasoning together for multi-intersection control.

[W1] Does learned routing improve over random or simple rules?

We have added Random and Rule-based mode selection to Table 4 of the revised manuscript. Random selects Fast or Slow randomly, while Rule-based selects the mode using predefined thresholds on the gap between the two largest candidate-phase queues. For a fair comparison, both baselines use the same VLALight signal decision policy and approximately match its Fast/Slow mode proportions. This holds the policy fixed and keeps the overall allocation of reasoning modes comparable, allowing us to evaluate how modes are assigned to traffic states. VLALight's adaptive mode selection improves all six metrics over both baselines. The table below combines the Pure Slow and VLALight results from our original submission with the Random and Rule-based results added in the revised manuscript:

Mode-selection strategyJinan 1 ATT (s)AQLAWT (s)Hangzhou 2 ATT (s)AQLAWT (s)
Pure Slow263.48154.9772.86295.2897.0959.90
Random268.73157.6574.62297.15100.1661.89
Rule-based267.83156.6273.97297.02100.0461.65
VLALight267.05154.7473.31296.2999.0360.91

This supports state-dependent allocation beyond matching mode proportions, while retaining control quality close to Pure Slow. Table 5 in our original submission reports Fast-mode and Slow-mode latency of 2.866/4.507 s and an overall sampled mean of 3.936 s, 12.7% below the Slow-mode latency.

References:

[1] Lai, S., Xu, Z., Zhang, W., Liu, H., and Xiong, H. LLMLight: Large Language Models as Traffic Signal Control Agents. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2335–2346, 2025.

[2] Yuan, Z., Lai, S., and Liu, H. CoLLMLight: Cooperative Large Language Model Agents for Network-Wide Traffic Signal Control. International Conference on Learning Representations, 2026.

[3] Zou, X., Yang, Y., Chen, Z., Hao, X., Chen, Y., Huang, C., and Liang, Y. Traffic-R1: Reinforced LLMs Bring Human-Like Reasoning to Traffic Signal Control Systems. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, pp. 21823–21838, 2026.

[4] Brohan, A., et al. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. Proceedings of the Conference on Robot Learning, PMLR 229:2165–2183, 2023.

[W2, Q2] How is mode selection learned when rollout prefixes are externally assigned?

The original submission described balanced rollouts and mode comparison in Section 3.2 and Appendix A.6, but omitted some optimization details for brevity. We now make these details explicit, including the mode utility and soft target in Appendix A.6, Eqs. (11)–(12) of the revised manuscript, the weighted mode loss in Eq. (13) of the revised manuscript, and the combined objective in Eq. (14) of the revised manuscript. The same appendix gives sample-eligibility rules. Appendix A.7 of the revised manuscript expands the rollout procedure. Assigned prefixes balance exploration. The mode-selection objective trains the model's own selection probabilities.

Six network-level candidates start from the same snapshot, sharing observations, traffic perception outputs, and cooperative contexts. Two complementary assignment maps each generate three candidates, giving every intersection three Fast and three Slow trials. After joint execution, local reward, network reward, and reasoning cost are normalized separately within each intersection's candidate group, as specified in Appendix A.6, Eq. (10) of the revised manuscript. The utility and soft-target definitions are given in Appendix A.6, Eqs. (11)–(12) of the revised manuscript: higher net benefit from Slow increases its target probability, while insufficient benefit relative to cost favors Fast.

At the mode-token prediction position, the existing language-model logits for Fast and Slow are converted to a two-class probability distribution. The weighted soft-target cross-entropy in Appendix A.6, Eq. (13) of the revised manuscript directly optimizes these probabilities without adding a classification head. The subsequent reasoning and signal tokens use the clipped decision objective described in Section 3.2, Eq. (9) of both the original submission and revised manuscript. traffic perception supervision maintains traffic perception, and Appendix A.6, Eq. (14) of the revised manuscript specifies the combined optimization. Thus, mode learning has its own optimization path. At deployment, the model generates its mode autonomously from traffic context. Complete targets, sample weighting, eligibility rules, and losses are provided in Appendix A.6, Eqs. (10)–(14) of the revised manuscript.

[W2, Q2] How are local and network utility defined when other intersections' actions also change?

Our training environment contains multiple connected intersections. A signal action changes outgoing traffic and therefore affects queues and arrivals at other intersections. Candidates consequently evaluate jointly executed signal actions from a common network snapshot, as described in original Appendix A.6 and Algorithm 1.

The reward explicitly includes both local and network-level traffic objectives. Local reward compares the target intersection's queue before and after the first control transition. Network reward compares the total network queue at the initial snapshot and after three transitions. Each candidate's network reward is shared across its intersections. This encourages the shared policy to coordinate signal decisions for network-wide improvement, rather than optimizing each intersection independently. The definitions and time horizons are clarified in revised Appendices A.6–A.7.

[W2, Q3] What data, demonstration checks, and repetitions support reproducibility?

We have added Appendix A.5, Table 7 in the revised manuscript to report the two-stage data distribution:

Networktraffic perception traintraffic perception validationDecision trainDecision validation
Hangzhou17001881696192
Jinan12601481261144
New York244026411
Total54006002958337

The stages are sequential: the traffic perception adapter is merged before cooperative decision training. Traffic perception samples cover Jinan, Hangzhou, and New York. Decision supervision primarily covers Jinan and Hangzhou. Initial sampling preserves each city's Fast/Slow mode proportions and retains four-direction videos and available coordination views.

Decision examples require same-step records for the target and all actual topological neighbours. Splitting by city–time-step prevents shared cooperative context crossing the split. Initial sampling selects individual intersection–timestep records, so complete same-step neighbourhoods are sparse in the larger New York network. This criterion leaves two New York decision targets. We have added teacher-generation parameters, screening criteria, and dataset-splitting details in revised Appendix A.5. Empty responses are filtered. Signal tags must be unique and valid. Slow demonstrations require a nonempty reasoning block. Failed requests are retried. The queue-gap rule constructs Fast demonstrations with direct signal targets and Slow demonstrations with teacher reasoning and actions. These checks retain structurally usable demonstrations with legal signal actions.

Online RL uses only Jinan 1 and Hangzhou 1, with two differently seeded instances per network at each step and 50 training steps (Appendix A.5 of our original submission). These settings are retained in Appendix A.6 of the revised manuscript. The appendix distinguishes training steps, parallel environment instances, rollout seeds, and repeated evaluations.

[W2, Q3] What do the training curves and repeated evaluations show?

VLALight (Qwen3.5-4B) reaches a relatively stable reward level during training. We have added its 50-step training curve with a five-step moving average in Appendix A.13, Figure 9 of the revised manuscript. The improvement over its initial reward level supports the effectiveness of our training procedure.

To address your specific concern about dependence on a single backbone, we additionally trained VLALight with Qwen3.5-2B and Gemma 4 E2B. Figure 9 also presents their training curves, which show reward improvements across the evaluated backbones. Their control results are reported in Appendix A.13, Table 11 of the revised manuscript and reproduced below. All three variants are compared on Jinan 1 and Hangzhou 1 after the same number of training steps.

ScenarioVLALight variantATT (s)AQLAWT (s)
Jinan 1VLALight (Qwen3.5-4B)267.05154.7473.31
Jinan 1Qwen3.5-2B264.44156.9674.01
Jinan 1Gemma 4 E2B265.14158.9175.13
Hangzhou 1VLALight (Qwen3.5-4B)294.9055.9452.05
Hangzhou 1Qwen3.5-2B298.1858.1854.76
Hangzhou 1Gemma 4 E2B300.8960.0256.81

VLALight (Qwen3.5-4B) achieves the lowest AQL and AWT on both scenarios, while the policy trained with the Qwen3.5-2B backbone achieves the lowest ATT on Jinan 1. Together, Figure 9 and Table 11 in the revised manuscript document training progress and control performance across the evaluated backbones. The repeated evaluations below assess the stability of the same trained policy under different evaluation seeds.

To assess evaluation stability, we conducted three-seed evaluations of the same trained checkpoint on Jinan 1, Hangzhou 1, and New York 1. We have added these results to Appendix A.14, Table 12 of the revised manuscript (mean ± sample standard deviation):

ScenarioATT (s)AQLAWT (s)
Jinan 1267.34 ± 0.42155.41 ± 0.7673.63 ± 0.28
Hangzhou 1294.64 ± 0.8555.58 ± 0.5751.58 ± 0.60
New York 1978.44 ± 9.971580.81 ± 27.76570.90 ± 10.79

The ATT standard deviation is approximately 1% of its mean or less across the three scenarios. AQL and AWT also show relatively small variation. These are repeated evaluations of the same policy, while Figure 9 documents the separate backbone training runs.

Reviewer_kiUdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VLALight, a Vision-Language-Action (VLA) model for end-to-end traffic signal control (TSC). The model directly maps multi-view roadside video to coordinated signal actions via spatiotemporal reasoning and topology-aware cooperative perception. A two-stage training pipeline is introduced: supervised cold-start followed by cooperative agentic reinforcement learning. The paper also contributes an adaptive fast-slow reasoning mechanism to trade off decision quality and inference cost.

Critical strengths

[S1]This work is the first to explore VLA models for end-to-end traffic signal control, bridging the gap between physical visual observations and control decisions. [S2]Experiments span seven traffic-flow datasets across three urban networks, with ablation studies validating key components.

Critical weaknesses

[W1]. Unclear Motivation and Incremental Value over VLM-based TSC.​ The core motivation, why ACTION must be integrated into a VLM for TSC, is not well justified. Prior work such as VLMLight already demonstrates strong performance using VLM-based visual perception without end-to-end action generation. The paper does not convincingly argue why a full VLA paradigm yields meaningful gains over traditional or VLM-based methods, especially considering that network-wide perception, control, and efficiency optimizations can already be achieved by existing approaches. Given the substantially higher computational cost of VLA models, the benefit of introducing visual modalities for this task remains debatable.

[W2] Insufficient Validation Across Different Base Models.​ The experiments are conducted on a limited set of backbone (base) VLA/VLM models. To demonstrate generalizability, the method should be evaluated with multiple different base models (e.g., varying scales or architectures) to verify that improvements are not tied to a specific pretrained checkpoint.

[W3] Inadequate Efficiency Comparison.​ The efficiency analysis is self-contained and lacks comparison against relevant baselines, particularly VLMLight and other VLM-based or RL-based TSC methods. Inference latency, throughput, and decision quality under different reasoning modes should be benchmarked against these methods across all three urban networks to substantiate the claimed efficiency–quality trade-off.

[W4] Limited Technical Novelty.​ Beyond being the first VLA application to TSC, the individual components (two-stage training, cooperative RL, fast-slow reasoning) largely build upon existing techniques from other domains. The paper reads more as an engineering combination of known methods rather than a conceptually novel contribution, which weakens its significance for a top-tier venue.

Questions for authors

Check the weakness.

5 comments

We thank the reviewer for the comments on motivation, backbone validation, efficiency, and technical contributions. Below, we clarify the evidence in our original submission and provide additional results addressing the remaining concerns.

[W1] What does VLA add over existing VLM-based TSC, particularly VLMLight?

VLMLight uses a meta-controller to route signal decisions to a separate deep RL policy or language-model reasoning module. In contrast, VLALight learns a shared VLA policy for network-wide cooperative control. The main differences are:

  • Joint signal and mode learning. Fast directly generates a signal, while Slow generates reasoning before the signal. Both are generated by the same VLA policy. Local traffic feedback, network reward, and reasoning cost guide learning toward high decision quality and low inference cost.
  • Multi-intersection cooperation. VLALight exchanges relevant traffic information along road connections and optimizes jointly executed signal decisions using local and network rewards.

As reported in Tables 1 and 2 of our original submission, VLALight improves control performance over VLMLight and its SFT initialization:

MethodNew York 1 ATT (s)New York 2 ATT (s)
VLMLight1271.801743.75
VLALight SFT1171.081439.73
VLALight975.801240.11

Beyond control performance, visual input brings VLALight closer to practical deployment by using observations from roadside cameras. This reduces reliance on predefined structured traffic states and dedicated sensing devices. We further evaluate visual-input reliability and traffic perception accuracy in Appendices A.10–A.11 of the revised manuscript.

[W2] Does the framework extend to other base models?

The training curve in Appendix A.13, Figure 9 of the revised manuscript shows that VLALight (Qwen3.5-4B) reaches a relatively stable reward level during training. The improvement over its initial reward level supports the effectiveness of our training procedure.

To address your specific concern about dependence on a single backbone, we additionally trained VLALight with Qwen3.5-2B and Gemma 4 E2B. Figure 9 shows reward improvements for both variants, providing further evidence that the training procedure is effective across the evaluated backbones. Their control performance is reported in Appendix A.13, Table 11 of the revised manuscript. All rows below represent trained VLALight variants.

ScenarioBackboneATT (s)AQLAWT (s)
Jinan 1VLALight (Qwen3.5-4B)267.05154.7473.31
Jinan 1Qwen3.5-2B264.44156.9674.01
Jinan 1Gemma 4 E2B265.14158.9175.13
Hangzhou 1VLALight (Qwen3.5-4B)294.9055.9452.05
Hangzhou 1Qwen3.5-2B298.1858.1854.76
Hangzhou 1Gemma 4 E2B300.8960.0256.81

VLALight (Qwen3.5-4B) achieves the lowest AQL and AWT on both scenarios. The other trained variants also obtain competitive results, supporting applicability across the evaluated model sizes and families. The trained model is available in our repository.

[W3] How does efficiency compare with relevant baselines?

We added cross-method output-token comparisons in Figure 3 and estimated API costs in Appendix A.12 of the revised manuscript. The following table reproduces the cost and ATT values from Figure 8. Costs include input and output tokens at the model-specific prices listed in Table 10, in USD per 1,000 intersection decisions.

MethodNew York 1 Cost (USD/1,000 decisions)ATT (s) ↓New York 2 Cost (USD/1,000 decisions)ATT (s) ↓
LLMLight0.15261051.990.15281347.41
CoLLMLight0.92081009.340.95911263.85
VLMLight2.99151271.803.35821743.75
VLALight0.4353975.800.43411240.11

VLALight achieves the lowest ATT on both traces among these methods. Its estimated API cost is 53–55% lower than CoLLMLight and 85–87% lower than VLMLight. LLMLight has a lower estimated cost, while VLALight provides better control performance and processes video inputs. These results support a favorable cost–quality balance under the listed prices. Revised Table 5 additionally reports measured latency for LLMLight, CoLLMLight, and VLALight.

[W4] What are the technical contributions?

As stated in the Introduction of our original submission, we introduce VLALight, an end-to-end VLA model connecting roadside videos with traffic reasoning and coordinated signal control. To our knowledge, it is the first VLA model for traffic signal control. Its technical contributions address two connected problems.

First, adaptive reasoning mode selection. The policy learns both signal decisions and whether explicit reasoning is worthwhile. Balanced Fast/Slow rollouts compare traffic outcomes from a common initial state. Traffic feedback and reasoning cost then train mode selection to balance control quality and inference cost.

Second, network-wide cooperative control from videos. The shared policy extracts movement-level traffic information and exchanges relevant observations along road connections. Cooperative RL combines immediate local queue improvement with network-level queue improvement. This trains intersections to coordinate their signal actions toward network-wide traffic efficiency.

Reviewer_kiUdOpenReview ↗

Reply

Thanks for your rebuttal. Most of my concerns have been addressed. However, I cannot find much interesting method from this paper due to the lack of the novelty. I would like to keep my score.

Reviewer_TomROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VLALight, a VLA framework for TSC using roadside videos. It combines structured traffic perception, topology-aware cooperative information, and RL for coordinated TSC. An adaptive Fast/Slow reasoning mechanism is introduced to balance control performance and inference efficiency.

Critical strengths
  1. The paper explores an interesting direction by introducing a VLA framework for TSC, integrating visual traffic perception, cooperative information, and signal decision-making within a unified framework.
  2. The experiments cover seven traffic-flow datasets across three urban networks, with comparisons against various baselines.
Critical weaknesses
  1. Although VLALight is presented as an end-to-end VLA model, it first converts visual observations into structured traffic states, such as vehicle counts and queue lengths, before generating signal actions. This intermediate representation may discard useful visual information, making the framework closer to a conventional perception–decision pipeline.
  2. Eq. 2 extracts vehicle counts and queue lengths from images using a VLM. However, these traffic states could also be obtained using conventional detection models such as YOLO, followed by RL-based signal control. The paper lacks comparisons with such pipelines to demonstrate the advantages of VLM/VAL-based traffic perception.
  3. The Fast/Slow mode labels are assigned using a handcrafted queue-gap threshold during SFT. Why not simply use an if–else rule to select the reasoning mode?
  4. Fig 6 does not clearly indicate lane functions. How does the VLM distinguish different traffic movements, such as through and left-turn movements, when extracting movement-level traffic information? Moreover, Fig 6 and 7 show a considerable visual gap between the simulated traffic scenes and real-world roadside imagery, raising concerns about sim-to-real generalization.
  5. The manuscript is difficult to follow in several places. First, abbreviations and full terms are used inconsistently. For example, traffic signal control is abbreviated as TSC in Line 037 but appears in full in Lines 043, 047, and 068; similar inconsistencies occur for reinforcement learning (Lines 051, 101, and 209) and vision-language-action (Lines 066 and 080). Second, the repeated description of VLALight as an end-to-end VLA model is not well aligned with its explicitly separated perception and decision stages.
Questions for authors
  1. What are the key advantages of the proposed VLA framework over a conventional perception–decision pipeline, such as YOLO-based traffic perception followed by RL-based control?
  2. Why is learned Fast/Slow mode selection necessary instead of using a simple rule-based strategy, such as an if–else condition based on queue differences?
  3. How does the VLM identify lane functions and distinguish different traffic movements when such information is not explicitly available in the visual observations?
12 comments

We thank the reviewer for the detailed comments on the VLA formulation, mode selection, and traffic perception. Below, we clarify the evidence in our original submission and provide additional results addressing the remaining concerns.

[W1–W2, Q1] How does the VLA formulation differ from YOLO + RL, and does the structured representation retain useful visual information?

VLALight unifies traffic perception, cooperative reasoning, and signal decision generation within a single vision-language backbone rather than using the VLM as a detector frontend. As detailed in Sections 3.1–3.2 of our original submission, sequential SFT establishes both traffic perception and signal-decision capabilities, and cooperative RL then refines the control policy using local and network rewards while retaining traffic perception supervision. Consequently, the shared backbone jointly learns visual representations and signal actions through an inspectable structured interface.

Section 3.2 (Eq. 5) of our original submission specifies both perception and decision supervision and notes that the perception loss is retained during RL. By optimizing traffic perception and signal-decision losses jointly over shared parameters, VLALight couples traffic perception directly with action learning and control feedback.

Explicit intermediate representations are compatible with vision-to-action modeling: for example, EMMA [1] predicts scene descriptions and object behavior before driving trajectories, and ECoT [2] generates spatial representations (bounding boxes and end-effector positions) before robot actions. In VLALight, the structured traffic state organizes movement-level demand, queues, temporal evolution, and topology-routed cooperative information into an inspectable semantic interface. Crucially, it enables compact cooperative information to be exchanged along road connectivity without repeatedly processing every neighbouring intersection's full video. This design performs control-oriented spatiotemporal traffic perception rather than merely converting static detections into text.

By contrast, a YOLO + RL pipeline constructs control inputs through a multi-stage chain of detection, tracking, lane association, and state aggregation. Object detectors provide object classes and bounding boxes. Queue states, temporal changes, and information from connected intersections require further processing. Missed detections and association errors can therefore lead to error accumulation in downstream control. Our RL framework jointly optimizes traffic perception and signal decisions within a shared backbone. Retaining traffic perception supervision while optimizing decisions with traffic feedback helps mitigate potential error accumulation.

To evaluate traffic perception, we conducted an additional offline evaluation on 500 test samples. The results are summarized below and reported in Appendix A.11, Table 9 of the revised manuscript:

FieldExact matchWithin ±1
Vehicle count95.25%99.05%
Queue length97.30%98.75%
Vehicle-count change84.40%94.15%
Queue-length change87.85%94.45%
Estimated potential arrivals from neighbouring intersections90.63%94.93%

The results demonstrate reliable extraction of both static traffic quantities and temporal and cooperative information. The representation is designed to retain control-relevant traffic semantics, while the field-level test directly measures their extraction quality. The main experiments and ablations further evaluate the resulting closed-loop policy.

References:

[1] Hwang, J.-J., et al. EMMA: End-to-End Multimodal Model for Autonomous Driving. Transactions on Machine Learning Research, 2025.

[2] Zawalski, M., Chen, W., Pertsch, K., Mees, O., Finn, C., and Levine, S. Robotic Control via Embodied Chain-of-Thought Reasoning. Proceedings of the Conference on Robot Learning, PMLR 270:3157–3181, 2025.

[W3, Q2] Why learn Fast/Slow selection rather than use a queue-gap rule?

A fixed queue-gap rule cannot account for differences in traffic conditions across intersections and decision times. The same queue gap may require different decisions when upstream arrivals, queue trends, or time since last service differ. VLALight learns when additional reasoning improves traffic efficiency enough to justify its cost.

To learn when additional reasoning is worthwhile, we compare Fast and Slow decisions under the same initial traffic conditions during cooperative RL. Specifically, we generate six joint candidates from the same network snapshot. Each intersection has three Fast trials that directly generate a signal and three Slow trials that generate reasoning before the signal. We compare their local queue improvement, network queue improvement, and reasoning length. If Slow provides greater traffic benefit after accounting for reasoning cost, its target selection probability increases. Otherwise, the target favors Fast. A soft-target cross-entropy loss trains the model's Fast/Slow probabilities. The reasoning cost penalty discourages longer outputs unless they bring sufficient control benefit. These details are specified in Appendix A.6, Eqs. (11)–(13) of the revised manuscript. After post-training, VLALight automatically selects a reasoning mode to balance traffic efficiency and inference cost.

We added Random and Rule-based mode selection to Table 4 of the revised manuscript. For a fair comparison, both baselines use the same signal decision policy as VLALight and approximately match its Fast/Slow mode proportions. Random selects the reasoning mode randomly. Rule-based selects the reasoning mode using predefined thresholds on the gap between the two largest candidate-phase queues. This comparison evaluates how modes are assigned to traffic states while keeping the policy and overall mode proportions comparable.

Mode-selection strategyJinan 1 ATT (s)AQLAWT (s)Hangzhou 2 ATT (s)AQLAWT (s)
Random268.73157.6574.62297.15100.1661.89
Rule-based267.83156.6273.97297.02100.0461.65
VLALight267.05154.7473.31296.2999.0360.91

VLALight improves all six metrics over both baselines. In addition, Table 4 of our original submission shows that VLALight's adaptive policy achieves control performance close to Pure Slow, which uses explicit reasoning for every decision. Table 5 of our original submission reports a mean total latency of 3.936 s across 500 sampled VLALight decisions, compared with 4.507 s for VLALight's Slow decisions, a reduction of 12.7%.

[W4, Q3] How does the VLM distinguish lane functions and traffic movements without turn arrows?

Movement-level traffic perception is supervised during Stage 1 SFT, and this supervision is retained during RL. As described in Section 3.2 of our original submission, four-direction videos are paired with movement-specific traffic labels. These labels follow each intersection's lane positions and permitted movements for ETWT, NTST, ELWL, and NLSL. Also, revised Appendix A.5 details this supervision.

[W4] How are the visual differences between simulation and real roadside scenes addressed?

Simulation provides controlled, reproducible conditions for studying visual control, multi-intersection cooperation, and reasoning mode selection. Prior VLM-based TSC work, such as VLMLight, is evaluated in SUMO-based visual simulation. Following this established setup, our evaluation covers seven real-world traffic-flow traces across three urban networks (Section 4.1 and Appendix A.2 of our original submission). For real-road transfer, evaluating on annotated roadside videos, adapting traffic perception across viewpoints, lighting, and occlusions, and conducting field experiments remain important future directions, which we describe in the Conclusion of the revised manuscript.

We added visual-input failure tests on Jinan 1 in Appendix A.10, Figure 7 of the revised manuscript. Selected cameras are affected during 1,200–1,800 s, after which normal observations resume. We test two types of failure: frame dropping removes a specified number of frames from each six-frame observation of the affected cameras, while camera blackout makes their visual observations unavailable throughout the fault period. The table below reports representative results for both frame dropping and camera blackout. ATT, AQL, and AWT are evaluated over the full simulation, and their percentage increases are calculated relative to the same setting without visual failures.

Visual failureAffected camerasATT increaseAQL increaseAWT increase
Two of six frames dropped30%0.28%0.86%0.91%
Two of six frames dropped80%1.48%3.88%4.65%
Camera blackout30%0.39%1.13%1.10%
Camera blackout90%11.85%33.11%41.51%

When two of six frames are dropped, all three metrics increase by less than 1% at 30% camera coverage and less than 5% at 80% coverage. At 30% camera blackout, increases remain below 1.2%, whereas 90% blackout causes substantially greater degradation. These results indicate tolerance to partial observation loss and highlight the importance of sufficient visual coverage.

[W5] How have terminology and architecture descriptions been clarified?

We thank the reviewer for identifying these presentation inconsistencies. We have standardized the use of abbreviations in our revised manuscript, including traffic signal control (TSC), reinforcement learning (RL), and vision-language-action (VLA). Each term is introduced in its full form at its first occurrence in the Abstract and again at its first occurrence in the main text, followed by consistent abbreviation use throughout the corresponding sections.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Request for Reviewer Engagement

Dear Area Chair,

With only two days remaining in the discussion period, could you kindly encourage the reviewers to review our responses and participate in the discussion? We would greatly appreciate the opportunity to clarify any remaining concerns.

Thank you very much for your time and help.

Best regards, The Authors of Submission 31196

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.