acceptodds
Under review as a conference paper at ICLR 2027

CRC: Credible Residual Calibration for Traffic Signal Control under Test-Time Drift

Abstract

Traffic signal control (TSC) plays a critical role in improving urban traffic efficiency. Although reinforcement learning (RL) has achieved remarkable progress in TSC, RL-based controllers are typically trained under specific traffic conditions and remain frozen during testing. In practice, environmental feedback under dynamic traffic conditions may change after a signal action is applied, leading to performance degradation. Existing solutions mainly rely on adaptation capabilities acquired before testing and remain limited in exploiting feedback changes that emerge at test time. In this paper, we introduce Action-Conditioned Feedback Distribution (ACFD) drift to characterize changes in the feedback distribution over next states and rewards conditioned on the current state and action, and propose Credible Residual Calibration CRC), a plug-in test-time adaptation framework for TSC. CRC freezes the pretrained base controller and attaches a residual calibrator to its outputs, enabling adaptation through test-time feedback. CRC uses the test-time Bellman inconsistency of the frozen base controller as an observable value-relevant signal of ACFD drift and maintains calibration-state statistics from stepwise feedback. Based on the calibration state, the residual calibrator learns action-wise candidate residual corrections, while action-wise credibility controls their release for decision making. CRC further uses calibration-state-guided action routing to adjust the decision mode and dynamic replay to update the residual calibrator from informative test-time transitions. Extensive experiments on real-world datasets and multiple base TSC controllers demonstrate that CRC consistently improves test-time control performance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Traffic Signal Control, Test-Time Adaptation, Credible Calibration, Intelligent Transportation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wT5nOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates test-time adaptation for reinforcement learning-based traffic signal control. The authors propose Credible Residual Calibration (CRC), a framework that freezes a pretrained RL controller and trains a lightweight residual calibration network to dynamically adjust its Q-values based on test-time interaction feedback. The proposed method incorporates Bellman inconsistency-based drift estimation, action-wise credibility, calibration-guided action routing, and dynamic replay updates.

The method is evaluated on five traffic signal control benchmarks with four representative RL controllers. Experimental results demonstrate improvements in average travel time and waiting time under different traffic demand shifts, suggesting the potential of the proposed approach for test-time adaptation.

Critical strengths
  1. The paper addresses an important problem in reinforcement learning: how to adapt a pretrained policy to changing environments during testing without retraining the entire network. The proposed residual calibration mechanism is appealing because it preserves the original controller while introducing a lightweight trainable module to adjust Q-values using online feedback.

2.The authors evaluate CRC with multiple RL-based traffic signal controllers under different traffic demand shifts. The experimental results demonstrate consistent improvements in traffic efficiency, suggesting that the proposed method can serve as a flexible plug-in adaptation mechanism for pretrained Q-value-based controllers.

Critical weaknesses
  1. Although this paper focuses on traffic signal control, the proposed method seems applicable to other RL tasks, as its main components are not traffic-specific. I think the paper would be more convincing if the authors evaluated CRC on other RL benchmarks to demonstrate its generalization across different tasks and environments.

  2. If the authors intend to focus on traffic signal control, I would expect more experiments on safety and robustness rather than just traffic efficiency. In real-world traffic systems, unexpected events such as accidents, sensor failures, and road closures can have serious consequences.

I think CRC could be particularly useful in these situations, where quickly adapting to unexpected changes matters more than small improvements in average travel time. It would be valuable to see whether CRC can improve safety and help the controller recover from such events.

  1. Only focus on traffic signal control will reduce the contribution, because the policy model size is small, we can update the model in-time, if the model is a LLM the method will be attractive.
Questions for authors
  1. Could the authors evaluate CRC on other RL benchmarks to better demonstrate its general applicability under different environments and distribution shifts?

  2. How does CRC perform under unexpected events such as traffic accidents, road closures, or severe congestion? Can it recover faster or reduce safety risks compared with the original controller?

  3. Could the authors include safety-related metrics, such as traffic conflicts, queue spillback, worst-case waiting time, and recovery time, rather than focusing only on average traffic efficiency?

  4. How robust is CRC to noisy, missing, or delayed observations? Could unreliable feedback lead to incorrect Q-value corrections and worse control decisions?

  5. Could the authors compare CRC with controllers trained using data augmentation or domain randomization? This would help clarify the benefits of test-time calibration over stronger offline training.

14 comments
AuthorsOpenReview ↗

Author Response: W1/Q1

W1/Q1. Scope of Generalization Beyond TSC

Our work focuses on test-time adaptation for traffic signal control (TSC), and we do not claim that CRC has been validated as a general-purpose method for all RL tasks. Within TSC, we evaluate CRC with four pretrained RL controllers across five traffic benchmarks and multiple traffic shifts. These results demonstrate its applicability across different TSC controllers and traffic conditions, which is the intended scope of our contribution. We have clarified this scope in the paper and consider evaluation on broader RL benchmarks an interesting direction for future work.

AuthorsOpenReview ↗

Author Response (Part 1): W2/W3/Q2/Q3

W2/W3/Q2/Q3. Safety and Congestion Recovery under Unexpected Traffic Shifts

Our benchmarks simulate temporally localized traffic-demand surges rather than uniform changes in average traffic volume. Specifically, HZ1-Peak and HZ2-Valley increase vehicle arrivals by ×1.2, ×1.6, and ×2.0 within designated time windows. These sudden demand changes can cause queue buildup and persistent congestion, allowing us to evaluate congestion-related risks and post-peak dissipation.

To assess robustness beyond average traffic efficiency, we introduce five metrics. Queue Spillback Risk Rate (QSRR) measures the proportion of lane–decision-step observations in which queues approach lane storage capacity. Worst-Case Waiting Time (WCWT) records the maximum observed cumulative waiting time across vehicles, while W95 reports its 95th percentile to capture tail delays. Peak Queue measures the maximum network-average queue length during the demand-shift window and an allowed lag period. Post-Peak Dissipation (PPD) measures the difference between each controller's peak queue and its mean queue length over the following 400 s, characterizing subsequent congestion dissipation.

We compare frozen CoLight and FRAP with their CRC-enhanced counterparts on HZ1 and HZ2. Lower values indicate better performance for all metrics except PPD. CRC results are reported as mean ± standard deviation. The better result in each comparison is highlighted in bold.

CoLight - Safety and recovery metrics

Traffic shiftMethodQSRR ↓WCWT ↓W95 ↓PPD ↑Peak Queue ↓
HZ1-Peak ×1.2Frozen Base0.001058.00168.102.485.75
HZ1-Peak ×1.2Base + CRC0.00 ± 0.001007.50 ± 108.79134.14 ± 4.792.55 ± 0.405.69 ± 0.38
HZ1-Peak ×1.6Frozen Base0.001097.00189.904.178.88
HZ1-Peak ×1.6Base + CRC0.00 ± 0.001087.25 ± 53.37163.88 ± 7.674.95 ± 1.007.20 ± 0.31
HZ1-Peak ×2.0Frozen Base0.001123.00223.005.3713.50
HZ1-Peak ×2.0Base + CRC0.00 ± 0.00963.75 ± 165.13197.31 ± 10.666.51 ± 0.6311.94 ± 0.77
HZ2-Valley ×1.2Frozen Base0.412695.00503.500.313.00
HZ2-Valley ×1.2Base + CRC0.31 ± 0.052029.75 ± 614.42429.09 ± 19.590.43 ± 0.222.72 ± 0.15
HZ2-Valley ×1.6Frozen Base0.382698.00459.650.603.62
HZ2-Valley ×1.6Base + CRC0.30 ± 0.062093.50 ± 575.99416.85 ± 17.160.93 ± 0.453.55 ± 0.45
HZ2-Valley ×2.0Frozen Base0.412691.00484.001.144.94
HZ2-Valley ×2.0Base + CRC0.32 ± 0.032228.50 ± 629.71432.43 ± 7.201.55 ± 0.244.52 ± 0.36
AuthorsOpenReview ↗

Author Response (Part 2): W2/W3/Q2/Q3

FRAP - Safety and recovery metrics

Traffic shiftMethodQSRR ↓WCWT ↓W95 ↓PPD ↑Peak Queue ↓
HZ1-Peak ×1.2Frozen Base0.001536.00262.102.827.50
HZ1-Peak ×1.2Base + CRC0.00 ± 0.00926.25 ± 124.22149.28 ± 8.392.97 ± 0.506.16 ± 0.53
HZ1-Peak ×1.6Frozen Base0.001554.00305.903.9610.94
HZ1-Peak ×1.6Base + CRC0.00 ± 0.00892.75 ± 72.63175.45 ± 5.494.76 ± 0.309.75 ± 0.73
HZ1-Peak ×2.0Frozen Base0.001566.00334.005.1115.25
HZ1-Peak ×2.0Base + CRC0.00 ± 0.00884.25 ± 217.31243.00 ± 9.636.57 ± 1.4014.31 ± 0.96
HZ2-Valley ×1.2Frozen Base0.613373.00777.200.044.50
HZ2-Valley ×1.2Base + CRC0.44 ± 0.063036.50 ± 404.79538.75 ± 18.771.54 ± 0.843.12 ± 0.76
HZ2-Valley ×1.6Frozen Base0.573373.00757.050.135.69
HZ2-Valley ×1.6Base + CRC0.49 ± 0.053323.50 ± 1.00572.42 ± 66.140.44 ± 0.724.33 ± 0.14
HZ2-Valley ×2.0Frozen Base0.633373.00797.100.967.44
HZ2-Valley ×2.0Base + CRC0.48 ± 0.043032.00 ± 393.75545.29 ± 61.471.70 ± 0.544.55 ± 0.57

Across all 12 comparisons, CRC reduces WCWT and W95, alleviating extreme vehicle delays, and lowers Peak Queue, limiting maximum queue accumulation. On HZ2, it also reduces QSRR in all six settings, indicating lower near-capacity queue risk. Moreover, CRC increases PPD in all 12 settings, indicating greater post-peak queue dissipation despite the lower peak queues. Overall, these results show that CRC mitigates queue buildup and extreme waiting, reduces spillback risk where present, and improves congestion dissipation under sudden demand surges.

All revisions are highlighted in red in the latest submitted version of the paper for ease of review.

Reviewer_2xmLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CRC to improve traffic signal control when traffic conditions change at test time. CRC keeps the base controller fixed and learns corrections from new feedback. It uses Bellman errors to detect changes and action-wise credibility to decide how much correction to use. It also switches to a queue-based rule when calibration errors are high. The results show lower travel time and waiting time on several traffic benchmarks. Although the authors have made good efforts, some issues still need to be addressed.

Critical strengths
  1. The paper studies a useful problem in traffic signal control. A controller trained on past traffic may perform poorly when traffic conditions change. CRC uses new feedback to adjust its decisions and can be added to different base controllers.

  2. The experiments cover four base controllers and five traffic benchmarks. The authors test different levels and periods of traffic changes. They also report travel time, waiting time, and ablation results, which help show the effectiveness of the full method.

Critical weaknesses
  1. It is not clear whether the Bellman error can separate traffic changes from noise and base-model errors. Appendix C.1 explains the expected signed error, but CRC uses the absolute error from each feedback sample. The experiments do not fully show whether this signal can reliably detect traffic changes.

2.It is not clear how much of the improvement comes from calibration. CRC also switches to a queue-based rule, which may improve control on its own. The current ablations are useful, but they do not fully separate the gains from learned corrections and rule switching.

Questions for authors
  1. In Section 4.1 and Appendix C.1, the authors should test the drift signal under unchanged traffic with different noise levels. They should compare these results with settings where traffic changes. This would help show whether large errors come from traffic changes or noisy feedback.

  2. In Section 5.3, the authors should show whether higher credibility and lower calibration error lead to lower waiting time or shorter queues. A smaller Bellman error alone does not show that the action is better.

  3. In Section 5.4, the authors should add a baseline that keeps the same queue-based switching rule but removes learned corrections. This would help show how much improvement comes from calibration.

  4. The authors should further compare the parts of the calibration method under the same switching rules and update budget. For example, they could compare learned credibility with a fixed weight, and dynamic replay with standard prioritized replay. This would help explain which parts produce the gains.

6 comments
AuthorsOpenReview ↗

Author Response: W1/Q1

W1/Q1. Bellman Inconsistency under Traffic Drift and Observation Noise

To examine how traffic-demand shifts and observation noise affect Bellman inconsistency, we conduct additional experiments using a frozen CoLight controller on HZ1 and HZ2. At each decision step, we randomly select 5%, 10%, 20%, or 30% of network lanes and add zero-mean Gaussian noise to their vehicle-count observations. We evaluate the unchanged flow and three demand-shifted flows (×1.2, ×1.6, and ×2.0), without updating the base controller.

HZ1 - Mean absolute base Bellman error (mean ± standard deviation)

Noise ratioUnchanged×1.2×1.6×2.0
5%0.0677 ± 0.00130.0686 ± 0.00300.0710 ± 0.00110.0723 ± 0.0032
10%0.1108 ± 0.00330.1131 ± 0.00260.1146 ± 0.00130.1204 ± 0.0006
20%0.1888 ± 0.00500.1897 ± 0.00400.1973 ± 0.00470.1938 ± 0.0036
30%0.2569 ± 0.00200.2575 ± 0.00260.2581 ± 0.00450.2701 ± 0.0016

HZ2 - Mean absolute base Bellman error (mean ± standard deviation)

Noise ratioUnchanged×1.2×1.6×2.0
5%0.1246 ± 0.00090.1272 ± 0.00160.1275 ± 0.00200.1296 ± 0.0039
10%0.1945 ± 0.00350.1970 ± 0.00390.1972 ± 0.00190.2031 ± 0.0085
20%0.3201 ± 0.01000.3220 ± 0.00450.3275 ± 0.00260.3268 ± 0.0078
30%0.4233 ± 0.00590.4263 ± 0.00570.4244 ± 0.00440.4326 ± 0.0041

The mean absolute Bellman error increases with the noise ratio under both unchanged and shifted traffic flows. Even on the unchanged flows, the mean absolute error rises from 0.0677 to 0.2569 on HZ1 and from 0.1246 to 0.4233 on HZ2 as the noise ratio increases from 5% to 30%. This is expected because corrupted observations affect the frozen Q-value estimates and bootstrapped Bellman targets. When rewards are computed from corrupted queue observations, the reward term is also affected.

To examine whether traffic-demand shifts introduce additional inconsistency at the same noise level, we compare each shifted flow with the corresponding unchanged flow: Δdrift(λ,p)=∣δbase∣‾λ,p−∣δbase∣‾Base,p,\Delta_{\mathrm{drift}}(\lambda,p) = \overline{|\delta^{\mathrm{base}}|}_{\lambda,p} - \overline{|\delta^{\mathrm{base}}|}_{\mathrm{Base},p}, where λ\lambda is the demand multiplier and pp is the proportion of noise-corrupted lanes.

Additional mean absolute Bellman error relative to the unchanged flow (matched noise ratio)

Noise ratioHZ1 ×1.2HZ1 ×1.6HZ1 ×2.0HZ2 ×1.2HZ2 ×1.6HZ2 ×2.0
5%+0.0009+0.0033+0.0046+0.0026+0.0029+0.0050
10%+0.0023+0.0038+0.0096+0.0025+0.0027+0.0086
20%+0.0009+0.0085+0.0050+0.0019+0.0074+0.0067
30%+0.0006+0.0012+0.0132+0.0030+0.0011+0.0093

All 24 matched comparisons yield positive differences, indicating that traffic-demand shifts can introduce additional Bellman inconsistency even in the presence of observation noise. Thus, Bellman inconsistency captures mismatches between the frozen controller's value estimates and observed feedback, rather than uniquely identifying the source of those mismatches.

These results also clarify the scope of our study. CRC currently constructs Bellman inconsistency from test-time feedback under the assumption of reliable observations. Addressing corrupted observations requires explicit noise-aware treatment, which is a separate problem from the traffic-demand drift studied here. We consider noise-robust test-time calibration an important direction for future work.

AuthorsOpenReview ↗

Author Response: W2/Q3

W2/Q3. Contributions of Calibration and Queue-Based Routing

To distinguish the contributions of learned calibration and the queue-based rule, we introduce a Queue-Only baseline that applies the rule throughout testing without residual calibration. We compare it with CRC w/o Physical Mode (calibration without the queue-based action mode), CRC w/o Route (calibration without Physical or Release-Boost Mode), and Full CRC. The tables report ATT (lower is better).

CoLight - ATT:

Traffic shiftQueue-OnlyCRC w/o Physical ModeCRC w/o RouteFull CRC
HZ1-Peak ×1.2337.66335.42±1.29339.93±0.49334.47±0.55
HZ1-Peak ×1.6343.53343.34±1.50346.81±0.53342.49±1.22
HZ1-Peak ×2.0358.15356.99±2.73359.55±2.26356.12±0.76
HZ2-Valley ×1.2422.98418.84±1.76421.78±0.48417.55±1.84
HZ2-Valley ×1.6426.99420.65±1.72422.36±0.57417.17±1.59
HZ2-Valley ×2.0426.43420.51±2.25422.05±0.88418.88±1.59

FRAP - ATT:

Traffic shiftQueue-OnlyCRC w/o Physical ModeCRC w/o RouteFull CRC
HZ1-Peak ×1.2337.66342.10±3.31350.59±0.82338.32±0.95
HZ1-Peak ×1.6343.53371.00±23.01360.09±2.38348.03±2.39
HZ1-Peak ×2.0358.15415.51±2.52394.85±7.16365.89±1.68
HZ2-Valley ×1.2422.98434.35±3.15448.02±6.53425.85±1.47
HZ2-Valley ×1.6426.99433.85±1.36449.66±1.19426.27±2.19
HZ2-Valley ×2.0426.43439.90±7.00448.51±2.50427.93±2.51

The results reveal two findings. First, the gains are not solely attributable to the queue-based rule. With CoLight, CRC w/o Physical Mode outperforms Queue-Only in all six traffic shifts despite excluding the rule-based mode (e.g., 420.65 versus 426.99 on HZ2-Valley ×1.6). Second, Full CRC achieves lower ATT than CRC w/o Physical Mode in every setting for both CoLight and FRAP, indicating that the queue-based mode offers an additional benefit when combined with learned calibration.

We also observe that Queue-Only remains competitive with or outperforms Full CRC in several FRAP settings. More importantly, the queue-based rule may also support test-time calibration, beyond its direct control performance. When calibration is unreliable, repeatedly selecting the action with the highest corrected Q-value may reinforce inaccurate action preferences and restrict the feedback available for subsequent updates. Physical Mode provides a conservative, queue-aware alternative that may diversify state–action–feedback samples while maintaining reasonable control performance. This is related to the exploration–exploitation trade-off, but random ϵ\epsilon-greedy exploration is unsuitable for test-time TSC because each action has real consequences and cannot be undone or repeatedly retried. Accordingly, our three-mode routing combines calibrated action selection with rule-guided decisions when credibility is low.

AuthorsOpenReview ↗

Author Response: Q2

Q2. Bellman Error and Traffic Control Performance

We agree that a lower Bellman error does not necessarily imply a better control performance. To clarify this point, we have extended Fig. 5 to include the time-aligned waiting-time and queue-length curves alongside the Bellman inconsistency, calibration error, and credibility signals.

The curves show no consistent stepwise relationship between these signals and traffic performance. This is not contradictory, as traffic congestion also depends on demand and accumulated queues, whose effects may persist despite more reliable value estimates.

Importantly, we do not claim that Bellman error or credibility directly measures control performance. Bellman inconsistency reflects disagreement between the frozen controller's value estimates and observed feedback, while credibility determines how strongly CRC relies on learned corrections. The quality of control decisions is evaluated separately using traffic-efficiency metrics such as ATT and AWT. We have clarified this distinction and added the corresponding curves to Fig. 5.

All revisions are highlighted in red in the latest submitted version of the paper for ease of review.

Reviewer_EVmpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper tries to solve the problem for reinforcement learning based traffic signal control failing to really adapt to the real enviroments and thus having degraded performance. This paper focuses on test time solution:

  1. "Action-Conditioned Feedback Distribution (ACFD) drift" to characterize changes in the feedback distribution over next states and rewards conditioned on the current state and action
  2. "Credible Residual Calibration (CRC)" a plug-in test-time adaptation: it freezes the pretrained base controller and attaches a residual calibrator to its outputs, enabling adaptation through test-time feedback and uses the test-time Bellman inconsistency of the frozen base controller as an observable value-relevant signal
Critical strengths

S1: The author proposed to use a test-time solution to help RL-based TSC handle real/dynamic environments.

S2: The experiments are comprehensive, covering effectiveness, adaptation behaviour, mechanism analysis (ablation / sensitivity studies),, adapability, and efficiency

Critical weaknesses

W1. There is a track of "generalized RL agents for TSC" being missing in related work / baseline comparision.

W2: The empirical gains look like they come from the queue heuristic, not the residual calibrator. The ablation (Table 11): FRAP w/o Route gets ATT 368.51 on HZ1, which is worse than the frozen base (363.38). CoLight w/o Route is essentially the same as Base (348.76 vs 348.96). So the calibrator alone, with credibility and replay but no routing, contributes nothing or hurts.

W3: The signal cannot separate ACFD drift from state shift Eq. 8 assumes Q_θ ≈ T_train Q_θ. This holds at best for states seen during training. With pure state shift, where p(s) changes but p(s′, r | s, a) does not, the agent visits new states where Q_θ is inaccurate. Then δ_base is also large. So Bellman inconsistency cannot tell ACFD drift apart from state shift. But this difference is the main idea of the paper (Introduction). CRC changes the actions, so it visits new states, which increases δ_base. Part of the "drift" signal may be caused by CRC itself. The state only contains partial information (lane counts and phase), so the problem is partially observed (a POMDP). A change in "p(s′ | s, a)" can then come from hidden variables, not from real dynamics change.

Questions for authors

see W1

4 comments
AuthorsOpenReview ↗

Author Response: W1

W1. Generalized RL-Based TSC Baselines

In the RL-based TSC Methods subsection of Related Work, we have already discussed meta-RL-based TSC methods, which represent a class of generalized RL-based TSC methods. These methods are particularly relevant to our study, as they aim to improve generalization across traffic scenarios and adaptability to test-time changes. We have also included them in our experimental comparisons (Fig. 4), with detailed results reported in Tables 8-12 of Appendix D.5.

To make this coverage more explicit, we have revised the Related Work and Baselines sections to clearly identify these methods as generalized RL-based TSC baselines.

All revisions are highlighted in red in the latest submitted version of the paper for ease of review.

AuthorsOpenReview ↗

Author Response: W2

W2. Analysis of the Queue Heuristic

The w/o Route ablation removes both Physical Mode and Release-Boost Mode, not merely the queue heuristic. Thus, the Table 13 establishes the importance of the complete Action Routing mechanism but does not show that the queue heuristic alone accounts for CRC's gains. In addition, w/o Route improves upon the frozen base controller on HZ2 in the original evaluation, indicating that residual calibration can help even without routing.

To disentangle these effects, we add four settings: Queue-Only (rule-based control without calibration), CRC w/o Physical Mode (calibration with two-mode routing, excluding the queue heuristic), CRC w/o Route (calibration without either routing mode), and Full CRC. The following tables report ATT (lower is better), with mean ± standard deviation where available.

CoLight:

Traffic shiftQueue-OnlyCRC w/o Physical ModeCRC w/o RouteFull CRC
HZ1-Peak ×1.2337.66335.42±1.29339.93±0.49334.47±0.55
HZ1-Peak ×1.6343.53343.34±1.50346.81±0.53342.49±1.22
HZ1-Peak ×2.0358.15356.99±2.73359.55±2.26356.12±0.76
HZ2-Valley ×1.2422.98418.84±1.76421.78±0.48417.55±1.84
HZ2-Valley ×1.6426.99420.65±1.72422.36±0.57417.17±1.59
HZ2-Valley ×2.0426.43420.51±2.25422.05±0.88418.88±1.59

FRAP:

Traffic shiftQueue-OnlyCRC w/o Physical ModeCRC w/o RouteFull CRC
HZ1-Peak ×1.2337.66342.10±3.31350.59±0.82338.32±0.95
HZ1-Peak ×1.6343.53371.00±23.01360.09±2.38348.03±2.39
HZ1-Peak ×2.0358.15415.51±2.52394.85±7.16365.89±1.68
HZ2-Valley ×1.2422.98434.35±3.15448.02±6.53425.85±1.47
HZ2-Valley ×1.6426.99433.85±1.36449.66±1.19426.27±2.19
HZ2-Valley ×2.0426.43439.90±7.00448.51±2.50427.93±2.51

The results distinguish the contributions of calibration and routing. For CoLight, both CRC w/o Physical Mode and Full CRC achieve lower mean ATT than Queue-Only in all six traffic shifts, directly showing that the gains are not solely due to the queue heuristic (e.g., HZ2-Valley ×1.6: 420.65 and 417.17 versus 426.99). Meanwhile, Full CRC achieves lower mean ATT than CRC w/o Physical Mode in all six settings for both CoLight and FRAP, demonstrating the complementary contribution of Physical Mode. We also acknowledge that Queue-Only is highly competitive for FRAP in several settings. This suggests that the benefits vary with the base controller and traffic condition.

More importantly, Physical Mode is not simply an additional queue heuristic. It may also support the optimization of the residual calibrator through test-time feedback. Without rule-guided action selection, the controller predominantly exploits the actions with the highest calibrated Q-values. When traffic conditions shift and calibration credibility is low, these value estimates may be inaccurate. Repeatedly acting greedily on such estimates can restrict the feedback collected to the calibrator's current action preferences, making it difficult to correct erroneous rankings. Physical Mode instead introduces a queue-aware alternative that may diversify the observed state–action–feedback samples while maintaining reasonable control performance.

This reflects the exploration–exploitation trade-off, but conventional ϵ\epsilon-greedy exploration is not well suited to test-time TSC. Randomly selected phases may worsen congestion, and each real-time decision has consequences that cannot be undone or repeatedly retried. Our three-mode Action Routing therefore balances exploitation of calibrated Q-values with credibility-aware, rule-guided decisions when calibration is uncertain. In this way, Physical Mode can potentially provide more informative feedback for subsequent CRC updates without resorting to random trial and error.

AuthorsOpenReview ↗

Author Response: W3

W3. Bellman Inconsistency, ACFD Drift, and State Shift

ACFD drift can induce value-estimation mismatch, which is reflected by base Bellman inconsistency. Specifically, changes in the action-conditioned reward and next-state distributions can alter the expected Bellman target constructed from test-time feedback. Since the pretrained controller QθQ_\theta remains frozen, its original action-value estimates may no longer match this target.

To clarify the relationship between state shifts and value-estimation mismatch, we conducted an additional experiment on a single intersection. We trained FRAP on an initially uniform traffic flow and then froze its parameters. We constructed 11 shifted flows, each containing approximately the same total number of vehicles as the original flow. Three flows redistribute arrivals over time (early, late, or middle peaks), and eight redistribute demand across traffic phases or directions. We evaluated frozen FRAP on all 12 flows without test-time updates.

Evaluation flowATT ↓AWT ↓Mean abs. δbase\delta^{\mathrm{base}}Mean DbaseD^{\mathrm{base}}
Original166.5937.760.14660.0578
Temporal: early peak454.21171.380.54170.0711
Temporal: late peak229.5098.190.38410.0915
Temporal: middle peak1043.13421.552.20300.1028
Phase: P1637.14461.191.76540.0997
Phase: P2191.6267.830.17070.0477
Phase: P3631.14461.191.76520.0993
Phase: P4192.6163.940.16630.0483
Phase: through176.8648.200.11610.0421
Phase: left177.9349.310.12470.0444
Phase: east–west181.9351.260.12340.0421
Phase: north–south165.1338.680.11160.0478

The results show that state shifts do not necessarily lead to control degradation. For example, the north–south redistribution achieves an ATT of 165.13, close to 166.59 on the original flow, whereas the middle-peak shift increases ATT to 1,043.13. Moreover, greater performance degradation is generally associated with stronger Bellman inconsistency. The middle-peak shift increases mean absolute Bellman error from 0.1466 to 2.2030 and normalized inconsistency from 0.0578 to 0.1028. Spearman's rank correlation measures the strength of a monotonic association between two variables [1]. Across the 12 flows, its coefficients between ATT and these two inconsistency measures are 0.94 and 0.77, respectively. These results indicate that Bellman inconsistency reflects value-relevant feedback mismatch, whereas the occurrence of a state shift alone does not establish that the controller has become unreliable.

Regarding CRC-induced action changes, the base Bellman error is computed as δtbase=[rt+γmax⁡a′Qθ(st+1,a′)]−Qθ(st,at),\delta_t^{\mathrm{base}} = \left[r_t+\gamma\max_{a'}Q_\theta(s_{t+1},a')\right] - Q_\theta(s_t,a_t), where ata_t is the actually executed action. This error compares the frozen base controller's value estimate for the executed action with the Bellman target obtained from the reward and next-state feedback generated by that same action. Neither the residual correction nor an indicator of whether CRC changed the original action enters this calculation. Therefore, changing the selected action does not inherently increase the base Bellman inconsistency.

In summary, ACFD describes the observable action-conditioned feedback, and Bellman inconsistency captures its value-relevant mismatch with the frozen controller. We have clarified this relationship and added the above experimental analysis to the ACFD Drift discussion in the paper.

All revisions are highlighted in red in the latest submitted version of the paper for ease of review.

Reference

[1] Spearman C. The proof and measurement of association between two things. 1961.

Dear Reviewer,

We would like to follow up on our rebuttal. When you have a chance, we would appreciate hearing whether our responses have addressed your concerns and whether there are any points you would like us to clarify. Thank you again for your time and feedback.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Updates and Additional Evidence

Dear Reviewers and Area Chair,

We sincerely thank the reviewers for recognizing the significance of our work and its comprehensive empirical results. The constructive feedback has helped us clarify our contributions and strengthen the experimental evidence. Below, we summarize the key updates in our rebuttal addressing the reviewers’ shared concerns.

  1. Methodological Clarifications and Validation: We further clarify the design of CRC, particularly Bellman-inconsistency-based drift detection and calibration routing. We provide additional experiments and analyses on Bellman inconsistency under traffic drift and observation noise, as well as the role of the queue heuristic, to validate these designs.
  2. Additional Comparisons and Ablation Studies: We conduct further experiments to validate the effectiveness of learned action credibility and dynamic replay, along with additional comparisons against data augmentation and domain randomization.
  3. Comprehensive Safety and Robustness Evaluation: We introduce five congestion-related safety metrics to assess queue spillback risk, extreme waiting times, and congestion recovery. We further evaluate CRC under missing and noisy observations combined with traffic-demand shifts, including additional comparisons with RobustLight (ICML 2025) and other advanced TSC methods.

We believe these clarifications, additional experiments, and analyses further strengthen our work and provide more comprehensive evidence supporting CRC's effectiveness.

All new results and clarifications have been detailed in our individual responses below.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.