acceptodds
Under review as a conference paper at ICLR 2027

VLALight: Lightweight Vision-Language-Action Models for Emergency-Aware Traffic Signal Control

10 views

Abstract

Traffic signal control (TSC) is essential for mitigating urban congestion. Recent advances in vision-language models (VLMs) enable richer interpretation of intersection scenes, opening new opportunities for visual-context-aware TSC. However, existing VLM-based controllers follow a cascaded perceive–reason–control pipeline whose vision-to-text conversion loses fine-grained spatial detail and whose sequential multi-agent inference adds latency that is problematic for real-time control. We propose VLALight, a lightweight vision-language-action (VLA) framework that maps a stitched multi-directional observation of an intersection, together with a textual description of its phase configuration, directly to discrete signal-phase actions in a single forward pass, avoiding intermediate image-to-text descriptions and handcrafted traffic-state representations. A simulation-grounded pipeline anchored to real intersection topologies provides expert demonstrations, and a compact 0.5B-parameter multimodal backbone, adapted with LoRA and a small set of learnable phase queries, answers each decision in 0.22 s on a local GPU. In closed-loop simulation on six replicas of real intersections, VLALight attains the strongest emergency-vehicle service among the compared controllers, reducing pooled emergency waiting time by 21.1% over the cascaded cloud-scale VLMLight, and generalizes to unseen intersection topologies and traffic-flow.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Traffic Signal Control, Intelligent Transportation Systems, VLA, Emergency, Lightweight

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CCcwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VLALight uses a vision-language model with a 0.5B language backbone to select traffic signal phases. The model takes stitched camera views and a textual description of the phase configuration, then predicts an action through learnable queries and an action head. It is trained on simulated demonstrations from MaxPressure and an emergency-priority labeling rule. On the pooled regular-flow evaluation, it achieves the lowest emergency waiting time among the compared controllers, but its overall waiting time is worse than that of several baselines. The paper also evaluates inference latency, unseen traffic flows, held-out intersections, and the effects of image stitching and minimum green time.

Critical strengths

-Overall well-written and easy to follow.

-The latency result is a clear practical advantage over the cascaded VLM controller. A 224 ms average forward pass leaves substantial room within the five-second decision interval.

-The paper reports the cost of emergency prioritization to other traffic rather than showing only the favorable emergency metrics. The examples involving simultaneous emergencies and missed vehicles are also helpful for understanding the model's limitations.

Critical weaknesses

-I would like to see a simpler emergency-aware baseline. VLMLight is a relevant comparison, but a detector followed by MaxPressure with emergency preemption is also a natural design. Appendix F dismisses such a baseline as artificial, which I do not find convincing. An oracle version using simulator emergency identity would be useful even before adding a detector.

-The trade-off is quite large: emergency waiting falls from 12.5 to 9.84 seconds relative to VLMLight, while overall waiting rises from 11.2 to 19.2 seconds. Some penalty is expected when prioritizing emergency vehicles, but the current experiments do not show whether this much is necessary. It could also reflect errors in routine control.

-The experiments do not establish which parts of the architecture are needed. Much of the model comes from VLA-Adapter, and stitching changes resolution and token count as well as layout. A vision-only policy and a simpler classifier trained on the same data would help determine whether the language input and query mechanism contribute meaningfully.

-The latency and generalization claims need tighter support. The cloud timing includes overhead excluded from VLALight's forward-pass measurement, and it is unclear how inference time affects the simulation. The reported variation also comes from four event sets with one checkpoint, while the separation between training and evaluation emergency configurations is not fully explained.

Questions for authors

-How does the method compare with MaxPressure plus emergency preemption, first with oracle emergency identity and then with a visual detector?

-How well does VLALight perform when there are no emergency vehicles, and can it achieve a better routine-versus-emergency trade-off by changing the priority setting?

-What happens with a matched vision-only model or a simpler action head?

-How are inference delays incorporated into the simulator, which emergency configurations are held out, and how stable are the results across training seeds?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer CCcw (Part 1)

Comment:

Below we report the preemption sweep and the no-event comparison. The sweep treats the sensing distance as the only free parameter; the no-event table runs every method with the injection sets disabled, on the same six intersections, with three seeds.

[W1, Q1] How does the method compare with MaxPressure plus emergency preemption, first with oracle emergency identity and then with a visual detector?

The preemption rule has one free parameter, the sensing distance: how many metres before the stop line an emergency vehicle becomes visible to the controller. At zero the rule never triggers, so the row reduces to plain MaxPressure and serves as the no-preemption reference.

Each distance is averaged over six intersections and four injection sets on the increasing-demand flow, 24 runs per distance.

Preempt distance (m)EMG_WAIT (s)ALL_WAIT (s)
506.10 ± 4.1714.96 ± 5.15
406.94 ± 5.2514.77 ± 4.92
307.18 ± 5.0913.87 ± 4.31
207.30 ± 5.0313.59 ± 4.16
107.99 ± 5.6513.71 ± 4.46
013.07 ± 8.8511.56 ± 3.69
VLALight10.33 ± 7.8717.06 ± 6.05

Emergency waiting falls from 13.07 s without preemption to 6.1 s at 50 m, and routine waiting rises from 11.56 s to 14.96 s over the same range. The rule buys emergency clearance with routine throughput.

VLALight reaches 10.33 s from four images, with no detector and no radius to tune. It beats MaxPressure without emergency awareness at 13.07 s. It trails MaxPressure with preemption at 10 m, which reaches 7.99 s. The rule reads emergency identity from the simulator state, so it is an upper bound rather than a deployable baseline.

[W2, Q2] The trade-off is large: emergency waiting falls from 12.5 to 9.84 seconds while overall waiting rises from 11.2 to 19.2 seconds. How does VLALight perform without emergency vehicles, and can a better trade-off be obtained by changing the priority setting?

The first group below contains methods that perceive an emergency vehicle. The second contains methods that cannot.

Emergency-aware methods

MethodAWT (s)
VLMLight (3B)27.03 ± 10.8
VLALight (0.5B)19.15 ± 5.8

Non-emergency-aware methods

MethodAWT (s)
MaxPressure12.10 ± 3.9
Webster13.59 ± 4.5
FixTime13.60 ± 4.5
SOTL14.56 ± 4.3
IntelliLight12.72 ± 6.0
AttendLight21.07 ± 12.6
PressLight25.52 ± 16.1
CoLight26.29 ± 14.8
Advanced-MPLight31.39 ± 19.5
LLMLight (Qwen2.5-3B)27.96 ± 14.9
LLMLight (Llama-2-13B)23.43 ± 10.4
Traffic-R123.99 ± 10.6

VLALight's routine waiting is 19.15 s. It is above the four rule-based controllers and IntelliLight, which lie between 12.10 and 14.56 s, and that gap is the green time reserved for emergency vehicles. It is below the remaining eight learned controllers.

VLALight's routine waiting is 19.15 s. It is above the four rule-based controllers and IntelliLight, which lie between 12.10 and 14.56 s. That gap is the green time it holds back for emergency vehicles, so it is the price of the emergency capability rather than a loss on routine traffic. Against the learned controllers, which are the relevant reference for a policy trained from data, VLALight is the lowest of all of them, from AttendLight at 21.07 s down to Advanced-MPLight at 31.39 s. It also has the smallest spread in that group, 5.8 s.

Priority is expressed inside the instruction, not as a coefficient. There is no separate weight that could be turned up or down. How this priority should be adjusted, and how to improve the trade-off itself, is a direction our future work will focus on.

AuthorsOpenReview ↗

Reply to Reviewer CCcw (Part 2)

Comment:

Below we address the component ablation, then latency inside the simulator, held-out configurations and stability across seeds.

[W3, Q3] Which parts of the architecture are needed? Would a vision-only policy or a simpler classifier trained on the same data perform as well?

We freeze the backbone and train a small classifier on its last-layer features. The features are the action-query states of the last LLM layer, mean-pooled into 896 dimensions. The probe uses 1,500 frames of the test split, drawn from the six intersections and split 80/20 by stratified random sampling into 1,200 fitting and 300 held-out frames. It tests whether the phase decision is already carried by the visual representation.

Logistic regression reaches 84.00% and a two-layer MLP 86.33%, while always predicting the most frequent phase of the fitting half reaches 30.00%. The generative head is therefore not the source of the capability. The phase is largely linearly decodable from the representation, and a simpler head suffices. Distilling the policy into such a classifier is the most concrete route to a cheaper deployment. What kind of action head signal control actually needs, and how far a lighter head can be pushed before it stops reacting to emergencies, is a direction our future work will focus on.

[W4, Q4] How are inference delays incorporated into the simulator, which emergency configurations are held out, and how stable are the results across training seeds?

The simulator is configured to take one decision every five seconds. While a decision is being computed its clock is frozen, and time starts running again only when the returned decision is executed. The reported metrics therefore contain no explicit delay term. A VLALight decision takes 224 ms on one RTX 4090, which is 4.5% of the interval, so inserting the real delay would change the numbers very little.

The freeze is where the comparison stops being realistic for the larger controllers. A frozen clock lets a slow method hide its cost, because the intersection does not move while it thinks. In reality the intersection keeps moving. VLMLight needs 10.9 s with its 32B backbone and 4.2 s with its 3B backbone against a 5 s interval, so an unfrozen loop would either skip decisions or execute a phase chosen for an observation one or two intervals old. That is why we report latency for every method rather than assuming the freeze: it flatters any controller whose decision time exceeds the interval, and none of ours does.

Emergency configurations are held out in a weaker sense than flows or topologies. Training uses the emergency frames present in the rendered scenes, with a randomised phase-to-view mapping. Testing uses four fixed configurations of six emergency vehicles each, with departure times and routes fixed in advance. The policy therefore meets emergency vehicles at positions and timings it never trained on, while the vehicle types themselves stay fixed.

Stability across seeds follows from the sampling regime. The training set contains about 16k decision frames, 1,462 emergency frames plus routine frames from 20 flows per intersection. Training runs 5,000 steps with an effective batch size of 128, which draws about 640k samples, or about 40 traversals of the set. Every frame is therefore seen about 40 times, and sampling noise from a different seed is averaged out. We report mean and standard deviation over four emergency sets with a single checkpoint, and the variation in our tables comes from the emergency configuration rather than from retraining.

Reviewer_BS5gOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows a lightweight VLA model for traffic signal control. It takes a traffic camera image of an intersection and text instructions, then directly selects signal phases for direct signal control. The model is fine-tuned from a pretrained VLM using data collected from simulations based on real-world traffic data. In control, it needs to respond to emergency vehicles while maintaining routine traffic. Its end-to-end image-to-action inference avoids the delays and enables transferability to unfamiliar traffic patterns and regions.

Critical strengths
  • Their end-to-end VLA design maps traffic camera observations directly to traffic signal actions. They successfully avoid expensive LLM inference proposed in recent literature and enable responses to emergency vehicles in routine control.
  • Their evaluation tests various traffic conditions and networks. Their method balances the trade-off between emergency vehicles and routine traffic efficiency.
Critical weaknesses
  • Their evaluations only include single intersection scenarios, and multiple strong baselines are ignored, including RL-based (e.g., CityLight, CoLight, AdvancedXLight, etc.) and LLM-based methods (e.g., LLMLight, CoLLMlight, Traffic-R1). They mentioned these related works but ignored them in the evaluation. While the effectiveness is validated in single-intersection scenarios, its effectiveness in region-level and city-level traffic networks remains a concern. While most previous methods have already focus on city-level control and are based on multi-intersection scenarios, this is a serious weakness of this paper.

  • The experiments rely on synthetic images. The transferability to real-world scenarios is not validated. Moreover, the challenging scenarios (e.g., conflicts of vehicles, ambiguity of images, congestion) are examined only qualitatively. The failure-rate analysis and validation with real-world traffic camera images are required to substantiate the method's reliability. Also, they only use Maxpressure as the golden label for routine control training, while this method is commonly used as a relatively naive baseline. The training method is somewhat limited.

Questions for authors
  • How much gain is obtained from VLALight’s emergency response compared to the baseline?

  • Is inference latency considered and included in the evaluation? Please clarify how delayed actions could affect traffic efficiency.

  • How is the performance in region-level and city-level road networks? Can the improvement for emergency vehicles be transferred or even improved to large road networks?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer BS5g (Part 1)

Comment:

We thank the reviewer for the comments on baselines and on what a simulator bounds. Below we report the added baselines first, then the two ways in which rendering and label provenance bound the result.

[W1, Q1] The evaluation only covers single intersections and omits strong RL and LLM baselines. How much gain does VLALight's emergency response obtain over them?

We add five baselines: CoLight, Advanced-MPLight, LLMLight with Qwen2.5-3B, LLMLight with Llama-2-13B, and Traffic-R1. All run under the same protocol as VLALight. Each method enters 72 runs, six intersections times three flows times four sets, and the table averages over them.

CoLight and Advanced-MPLight are RL controllers. CoLight's graph attention reduces to a single-agent value network at one intersection. Advanced-MPLight is the FRAP phase-competition variant of Advanced-XLight and needs no inter-intersection communication. Both are therefore faithful single-intersection instances of their methods, and each is trained per intersection for 3x10^5 steps on the movement-pressure state of our simulator.

The three LLM controllers run behind vLLM at the same interface. LLMLight prompts with Qwen2.5-3B. LLMLight prompts with the Llama-2-13B model released with the original work. Traffic-R1 uses its own reasoning template.

MethodATT (s)AWT (s)ETT (s)EWT (s)
CoLight54.19±16.719.44±15.354.64±24.918.85±23.1
LLMLight (Qwen2.5-3B)58.00±16.822.69±15.859.15±30.722.01±26.0
LLMLight (Llama-2-13B)53.75±16.716.46±12.358.41±30.219.32±21.6
Traffic-R154.02±16.716.77±12.358.59±30.819.50±22.1
Advanced-MPLight58.17±19.523.44±18.860.73±36.625.36±33.9
VLALight (0.5B)49.99±5.617.06±5.645.22±11.010.88±9.7

Emergency waiting is 10.88 s for VLALight, against 18.85 s to 25.36 s for the five baselines. Emergency travel time is 45.22 s, against 54.64 s to 60.73 s. The margin holds by flow. Emergency waiting of the baselines is 6.6 to 11.4 s under low demand, 17.6 to 26.4 s under rush hour and 27.6 to 38.3 s under random perturbation, against 3.4, 7.4 and 21.6 s.

VLALight has the lowest ATT, ETT and EWT of the six controllers. Routine waiting is 17.06 s, within 0.6 s of the two LLM baselines that read structured detector states and below every RL controller in the table. The 0.5B model therefore keeps the emergency advantage without giving up ground to the learned baselines on the routine side.

[W2] Rendering and labels both come from a simulator, and the labels come from max-pressure. How far does this bound the conclusions?

The images are rendered in Blender from the real intersection geometry. Roads and lane counts come from OpenStreetMap, and camera placement, signage and lane layout are calibrated against satellite and street-view imagery of each of the six sites. Geometry is therefore real. Lighting variation, weather, sensor noise, compression artefacts and the visual variety of an operational camera are not reproduced. Our results hold for this rendering distribution, and the sim-to-real gap is not measured here.

The label for routine traffic is the phase max-pressure would serve. Max-pressure is a strong classical controller but not an oracle, so it bounds the policy quality reachable on routine frames: the model can at best reproduce it. The main table follows this pattern, with routine waiting above the rule-based controllers and emergency waiting below every learned baseline. Emergency frames are labelled differently. The simulator identifies the emergency vehicle and the label is the phase its approach requires, so emergency behaviour is not bounded by max-pressure.

One way past this ceiling is to let the model learn in the loop rather than from recorded labels. Both limits above come from the imitation setting, where the target is whatever the simulator or max-pressure produced. Training the policy inside the simulator with reinforcement learning, so that it is rewarded on waiting time rather than fitted to a fixed label, is a direction our future work will focus on. It would let the model go beyond the max-pressure ceiling on routine frames and remove the dependence on recorded emergency labels.

AuthorsOpenReview ↗

Reply to Reviewer BS5g (Part 2)

Comment:

Below we address the latency question and the network-scale question.

[Q2] Is inference latency considered and included in the evaluation? How could delayed actions affect traffic efficiency?

A VLALight decision takes 224 ms on one RTX 4090, averaged over all decisions of the six intersections. That is 4.5% of the five-second interval, so the policy finishes well before the next decision point even with the clock running. The simulator does freeze its clock while a decision is computed, which isolates inference cost from the metrics we report. For VLALight the freeze changes little, because the decision is over before the interval ends.

The latency problem belongs to the VLM and LLM controllers. VLMLight needs 10.9 s with its 32B backbone and 4.2 s with its 3B backbone, the first twice the interval and the second close to it. A controller in that range cannot sit inside a five-second loop. Its numbers therefore rest on the freeze: the reported metrics are those of a policy that always answers the observation it was given. Once the clock runs, the same policy answers a state that has already moved.

A delayed action is not a small perturbation. In five seconds the queues in the served direction drain and the opposing queues grow, so the phase that was optimal at observation time can be the wrong phase by the time it is applied. An emergency vehicle can cross from two blocks away to the stop line in the same window. The wrong phase is then held for at least the minimum green, so one stale decision costs several seconds of the next one as well. This is why we treat latency as part of the controller rather than as an implementation detail of the evaluation.

[W1, Q3] How does the method perform in region-level and city-level road networks?

We have not built a multi-intersection network environment, and we could not add one within this revision. Rendering cost also grows with the number of intersections, since each one needs its own stitched view and its own forward pass. The deployment we have in mind treats every intersection as an independent agent. Each agent runs VLALight on its own four views and its own instruction, which already names that site and its phase mapping. Coordination between agents is the next step.

Thank you for your clarification. However, the sim-to-real gap seems to be left as a problem to be addressed in the future.

I do not fully agree that the simulations from Blender can completely and faithfully simulate real-world scenarios, like weather and signal interruptions. The author should provide the calibration results or an environment fidelity report to address this limitation.

Secondly, it seems like the scalability experiment is left out because the resource cost is too high. This is also a serious limitation in the real-world gap, as the authors ignored the spread of the vehicle flow. This is an important factor affecting real-world traffic. However, many existing works, including RL-based and LLM-based methods, have studied this effect.

In summary, I tend to keep the score for now.

Reviewer_jcvFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

VLALight proposes a compact vision-language-action controller that selects traffic signal phases directly from stitched multi-view images and textual phase descriptions. In closed-loop simulations of six real-world intersection layouts, it reports a mean local inference time of 224 ms and a 21.1% reduction in emergency-vehicle waiting time relative to the 32B VLMLight baseline. These emergency-service gains are accompanied by longer travel and waiting times for ordinary traffic.

Critical strengths
  1. The single-pass design addresses the latency of cascaded controllers, and the evaluation reports both emergency response and routine traffic efficiency across several intersection layouts and traffic flows.
Critical weaknesses
  1. The contribution of language is not clearly isolated. Since phase descriptions are generated from each intersection’s configuration, ablations of the instruction content would help establish what the language component adds to visual phase selection.
  2. Training and evaluation rely on rendered scenes and simulation. Without validation on real camera footage, it is difficult to assess whether the reported emergency response transfers to operational conditions.
Questions for authors
  1. How does the policy respond to changes in the wording or content of the phase instructions, particularly at unseen intersections? A targeted ablation would help clarify the language component’s contribution and generalization.
  2. Could the authors evaluate the model on real intersection footage, even in an offline setting? This would help assess whether the gains observed with rendered images persist under real visual conditions.
1 comment
AuthorsOpenReview ↗

Reply to Reviewer jcvF

Comment:

We thank the reviewer for the comments on the language input and on evaluation beyond rendered scenes. Below we clarify what our experiments cover and what they do not.

[W1, Q1] The contribution of language is not isolated; how does the policy respond to changes in the instruction content, particularly at unseen intersections?

The instruction follows a single template. It names the intersection and lists which phase corresponds to which image. The wording is therefore similar but not identical across the six intersections, because the name and the number of phases differ. An unseen intersection enters through the same template with a new name and a new mapping, and the zero-shot topology experiments vary exactly those two fields.

We have not run the ablation that isolates the instruction content, so we cannot report it here. Our next step is to make the instruction content a first-class input of the model and to re-test it with varied wording at unseen intersections. How to make the instruction content carry signal is a direction our future work will focus on.

[W2, Q2] Without validation on real camera footage, it is difficult to assess whether the emergency response transfers to operational conditions. Could the model be evaluated on real intersection footage, even offline?

Real intersection footage is not available to us. The cameras record pedestrians and license plates, and we have no permission to collect or redistribute such recordings. We render the six intersections from their real topology, geometry and lane structure instead, so the layout is real and the pixels are synthetic. Training and testing both take place in these scenes.

Every result, including the comparison with the five added baselines, is measured on rendered observations, under identical conditions for all methods. A different appearance or camera placement would change what the model sees. Rendering the same scenes photorealistically keeps the labels and the control interface unchanged, and validation on a small real-footage set is a direction we intend to pursue once data access is resolved.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.