- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates test-time adaptation for reinforcement learning-based traffic signal control. The authors propose Credible Residual Calibration (CRC), a framework that freezes a pretrained RL controller and trains a lightweight residual calibration network to dynamically adjust its Q-values based on test-time interaction feedback. The proposed method incorporates Bellman inconsistency-based drift estimation, action-wise credibility, calibration-guided action routing, and dynamic replay updates.
The method is evaluated on five traffic signal control benchmarks with four representative RL controllers. Experimental results demonstrate improvements in average travel time and waiting time under different traffic demand shifts, suggesting the potential of the proposed approach for test-time adaptation.
Critical strengths
- The paper addresses an important problem in reinforcement learning: how to adapt a pretrained policy to changing environments during testing without retraining the entire network. The proposed residual calibration mechanism is appealing because it preserves the original controller while introducing a lightweight trainable module to adjust Q-values using online feedback.
2.The authors evaluate CRC with multiple RL-based traffic signal controllers under different traffic demand shifts. The experimental results demonstrate consistent improvements in traffic efficiency, suggesting that the proposed method can serve as a flexible plug-in adaptation mechanism for pretrained Q-value-based controllers.
Critical weaknesses
-
Although this paper focuses on traffic signal control, the proposed method seems applicable to other RL tasks, as its main components are not traffic-specific. I think the paper would be more convincing if the authors evaluated CRC on other RL benchmarks to demonstrate its generalization across different tasks and environments.
-
If the authors intend to focus on traffic signal control, I would expect more experiments on safety and robustness rather than just traffic efficiency. In real-world traffic systems, unexpected events such as accidents, sensor failures, and road closures can have serious consequences.
I think CRC could be particularly useful in these situations, where quickly adapting to unexpected changes matters more than small improvements in average travel time. It would be valuable to see whether CRC can improve safety and help the controller recover from such events.
- Only focus on traffic signal control will reduce the contribution, because the policy model size is small, we can update the model in-time, if the model is a LLM the method will be attractive.
Questions for authors
-
Could the authors evaluate CRC on other RL benchmarks to better demonstrate its general applicability under different environments and distribution shifts?
-
How does CRC perform under unexpected events such as traffic accidents, road closures, or severe congestion? Can it recover faster or reduce safety risks compared with the original controller?
-
Could the authors include safety-related metrics, such as traffic conflicts, queue spillback, worst-case waiting time, and recovery time, rather than focusing only on average traffic efficiency?
-
How robust is CRC to noisy, missing, or delayed observations? Could unreliable feedback lead to incorrect Q-value corrections and worse control decisions?
-
Could the authors compare CRC with controllers trained using data augmentation or domain randomization? This would help clarify the benefits of test-time calibration over stronger offline training.
14 comments
Author Response (Part 1): W2/W3/Q2/Q3
W2/W3/Q2/Q3. Safety and Congestion Recovery under Unexpected Traffic Shifts
Our benchmarks simulate temporally localized traffic-demand surges rather than uniform changes in average traffic volume. Specifically, HZ1-Peak and HZ2-Valley increase vehicle arrivals by ×1.2, ×1.6, and ×2.0 within designated time windows. These sudden demand changes can cause queue buildup and persistent congestion, allowing us to evaluate congestion-related risks and post-peak dissipation.
To assess robustness beyond average traffic efficiency, we introduce five metrics. Queue Spillback Risk Rate (QSRR) measures the proportion of lane–decision-step observations in which queues approach lane storage capacity. Worst-Case Waiting Time (WCWT) records the maximum observed cumulative waiting time across vehicles, while W95 reports its 95th percentile to capture tail delays. Peak Queue measures the maximum network-average queue length during the demand-shift window and an allowed lag period. Post-Peak Dissipation (PPD) measures the difference between each controller's peak queue and its mean queue length over the following 400 s, characterizing subsequent congestion dissipation.
We compare frozen CoLight and FRAP with their CRC-enhanced counterparts on HZ1 and HZ2. Lower values indicate better performance for all metrics except PPD. CRC results are reported as mean ± standard deviation. The better result in each comparison is highlighted in bold.
CoLight - Safety and recovery metrics
| Traffic shift | Method | QSRR ↓ | WCWT ↓ | W95 ↓ | PPD ↑ | Peak Queue ↓ |
|---|---|---|---|---|---|---|
| HZ1-Peak ×1.2 | Frozen Base | 0.00 | 1058.00 | 168.10 | 2.48 | 5.75 |
| HZ1-Peak ×1.2 | Base + CRC | 0.00 ± 0.00 | 1007.50 ± 108.79 | 134.14 ± 4.79 | 2.55 ± 0.40 | 5.69 ± 0.38 |
| HZ1-Peak ×1.6 | Frozen Base | 0.00 | 1097.00 | 189.90 | 4.17 | 8.88 |
| HZ1-Peak ×1.6 | Base + CRC | 0.00 ± 0.00 | 1087.25 ± 53.37 | 163.88 ± 7.67 | 4.95 ± 1.00 | 7.20 ± 0.31 |
| HZ1-Peak ×2.0 | Frozen Base | 0.00 | 1123.00 | 223.00 | 5.37 | 13.50 |
| HZ1-Peak ×2.0 | Base + CRC | 0.00 ± 0.00 | 963.75 ± 165.13 | 197.31 ± 10.66 | 6.51 ± 0.63 | 11.94 ± 0.77 |
| HZ2-Valley ×1.2 | Frozen Base | 0.41 | 2695.00 | 503.50 | 0.31 | 3.00 |
| HZ2-Valley ×1.2 | Base + CRC | 0.31 ± 0.05 | 2029.75 ± 614.42 | 429.09 ± 19.59 | 0.43 ± 0.22 | 2.72 ± 0.15 |
| HZ2-Valley ×1.6 | Frozen Base | 0.38 | 2698.00 | 459.65 | 0.60 | 3.62 |
| HZ2-Valley ×1.6 | Base + CRC | 0.30 ± 0.06 | 2093.50 ± 575.99 | 416.85 ± 17.16 | 0.93 ± 0.45 | 3.55 ± 0.45 |
| HZ2-Valley ×2.0 | Frozen Base | 0.41 | 2691.00 | 484.00 | 1.14 | 4.94 |
| HZ2-Valley ×2.0 | Base + CRC | 0.32 ± 0.03 | 2228.50 ± 629.71 | 432.43 ± 7.20 | 1.55 ± 0.24 | 4.52 ± 0.36 |
Author Response (Part 2): W2/W3/Q2/Q3
FRAP - Safety and recovery metrics
| Traffic shift | Method | QSRR ↓ | WCWT ↓ | W95 ↓ | PPD ↑ | Peak Queue ↓ |
|---|---|---|---|---|---|---|
| HZ1-Peak ×1.2 | Frozen Base | 0.00 | 1536.00 | 262.10 | 2.82 | 7.50 |
| HZ1-Peak ×1.2 | Base + CRC | 0.00 ± 0.00 | 926.25 ± 124.22 | 149.28 ± 8.39 | 2.97 ± 0.50 | 6.16 ± 0.53 |
| HZ1-Peak ×1.6 | Frozen Base | 0.00 | 1554.00 | 305.90 | 3.96 | 10.94 |
| HZ1-Peak ×1.6 | Base + CRC | 0.00 ± 0.00 | 892.75 ± 72.63 | 175.45 ± 5.49 | 4.76 ± 0.30 | 9.75 ± 0.73 |
| HZ1-Peak ×2.0 | Frozen Base | 0.00 | 1566.00 | 334.00 | 5.11 | 15.25 |
| HZ1-Peak ×2.0 | Base + CRC | 0.00 ± 0.00 | 884.25 ± 217.31 | 243.00 ± 9.63 | 6.57 ± 1.40 | 14.31 ± 0.96 |
| HZ2-Valley ×1.2 | Frozen Base | 0.61 | 3373.00 | 777.20 | 0.04 | 4.50 |
| HZ2-Valley ×1.2 | Base + CRC | 0.44 ± 0.06 | 3036.50 ± 404.79 | 538.75 ± 18.77 | 1.54 ± 0.84 | 3.12 ± 0.76 |
| HZ2-Valley ×1.6 | Frozen Base | 0.57 | 3373.00 | 757.05 | 0.13 | 5.69 |
| HZ2-Valley ×1.6 | Base + CRC | 0.49 ± 0.05 | 3323.50 ± 1.00 | 572.42 ± 66.14 | 0.44 ± 0.72 | 4.33 ± 0.14 |
| HZ2-Valley ×2.0 | Frozen Base | 0.63 | 3373.00 | 797.10 | 0.96 | 7.44 |
| HZ2-Valley ×2.0 | Base + CRC | 0.48 ± 0.04 | 3032.00 ± 393.75 | 545.29 ± 61.47 | 1.70 ± 0.54 | 4.55 ± 0.57 |
Across all 12 comparisons, CRC reduces WCWT and W95, alleviating extreme vehicle delays, and lowers Peak Queue, limiting maximum queue accumulation. On HZ2, it also reduces QSRR in all six settings, indicating lower near-capacity queue risk. Moreover, CRC increases PPD in all 12 settings, indicating greater post-peak queue dissipation despite the lower peak queues. Overall, these results show that CRC mitigates queue buildup and extreme waiting, reduces spillback risk where present, and improves congestion dissipation under sudden demand surges.
All revisions are highlighted in red in the latest submitted version of the paper for ease of review.
Author Response: W1/Q1
W1/Q1. Scope of Generalization Beyond TSC
Our work focuses on test-time adaptation for traffic signal control (TSC), and we do not claim that CRC has been validated as a general-purpose method for all RL tasks. Within TSC, we evaluate CRC with four pretrained RL controllers across five traffic benchmarks and multiple traffic shifts. These results demonstrate its applicability across different TSC controllers and traffic conditions, which is the intended scope of our contribution. We have clarified this scope in the paper and consider evaluation on broader RL benchmarks an interesting direction for future work.