acceptodds
Under review as a conference paper at ICLR 2027

Do Reinforcement Learning Controllers Use Live State? A Diagnostic for Metro Operations

Abstract

A learned transit controller that beats a timetable has not shown that it uses the day's observations: recurring demand and an adaptive planning layer can supply the gain. We introduce a diagnostic framework that attributes a learned controller's gain to the calendar, to the planner's own feedback and to the learned layer's use of live state. It reconstructs each layer from calendar information alone, checks every run against its archived trajectory by exact replay, values single decisions by branching from identical states, asks which information predicts those values, and certifies its own power under declared demand perturbations; a proposition states what the reconstruction gap measures. Applied to 18 residual proximal policy optimization (PPO) checkpoints on a Hangzhou metro simulation driven by real demand, the framework finds three things. The best gain transfers but the feedback does not: on four sealed dates opened once after every method was frozen, the best checkpoint retains a s/passenger advantage over the planner (3.42 on development dates) but none over its calendar reconstruction, which is better than the live policy in six of eight cases. The instrument is not blind: the planner given live queues, a hand-written rule and an analytic dwell rule all move under the perturbations; the Hangzhou residual learners do not. We pre-registered and tested three alternative explanations for this null result: insufficient action authority, a training distribution that never departs from the calendar, and an instrument too insensitive to detect state use. Widening authority makes decisions state-dependent, and perturbation training does not make that dependence useful; neither yields a policy that outperforms the planner or rule it modifies. On six Beijing metro lines, where the learned layer holds trains at the queue it observes, the learners beat their same-day calendar tables by 0.3-1.2 units of the objective yet fall below the analytic rule in all 630 paired cases; on nine Shanghai lines the certificate reads within 0.01 of zero for the rule itself and the protocol declines to rule. The framework needs a learned layer on an adaptive planner or rule in a setting with recurring context; all three instances here are metro control.

Primary area:
reinforcement learning
Keywords:
reinforcement learning evaluation, open-loop baselines, counterfactual evaluation, public transit control

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_yEkpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a framework to determine whether reinforcement learning controllers in metro operations genuinely use live state information, known as the online feedback. The framework comprises four operations: calendar-conditioned reconstruction (P1), counterfactual rollouts from identical simulator states (P2), a comparison of the information sets that predict those counterfactual values (P3), and a power certificate under declared demand perturbations (P4). Experimetns applied method to three metro control examples, Hangzhou, Beijing and Shanghai.

Critical strengths

Extensive experiments on 3 metro systems, including Beijing, Shanghai and Hangzhou. Provide a lot of information for whether the method works or not.

Critical weaknesses

While applying RL algorithms to metro systems sounds new, I find little theoretical contribution in this paper. It looks more or less like an application of PPO to metro system. Few baselines are used, only PPO. Newer baselines can be compared.

I find it is very hard to follow the author's writing. For example, Table 2 and 3 contains too much information, usually the paper should pass the main message to the readers, instead of contain everything they did during experiment. Using tools to help revising texts is fine, but I think it would be better if the author have a clear idea of what message they want to deliver.

Questions for authors

Is it possible to compare with more recnet baselines like GRPO, DAPO? If not, please provide reasoning.

Reviewer_8ujbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work is a study of applying reinforcement learning (RL) on a local metro rail operations. The topic of this study is whether an RL-based controller leverages the live state for performing effective decision-making or if it ignores the important live variables.

Critical strengths

Although there are some strengths to this work, I am afraid to say I found none critical enough to mention here.

Critical weaknesses
  1. The scope of the work is too narrow. I am not sure how the diagnostics designed in this work are generally applicable to other RL controllers.
  2. The writing does not synthesize the idea and instead asserts every sentence. The draft does not read any natural and it was extremely hard to read.
  3. The paper uses obscure terminology for describing standard RL terms, e.g., "discrete decision callback", "learned residual πθ\pi_\theta" not clear to me what these mean.
  4. AI usage: The authors do not disclose if AI was used to perform literature survey, developing code, ideation, generating results. It seems to me this paper is heavily AI written and without proper disclosure, ICLR guidelines provide enough ground to reject such a paper.
Questions for authors

Please refer to critical weaknesses listed above to improve the paper.

In general, the paper does not demonstrate any considerable novelty, expositional clarity or wider applicability. These are sufficient reasons for me to clearly reject this work.

Reviewer_rw4yOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper diagnoses whether metro RL controllers benefit from live state through calendar-based reconstruction and counterfactual, information-set, and perturbation tests. Replacing the learned layer preserves the best policies gains while retaining the planner.

Critical strengths

Replay checks and positive controls strengthen the Hangzhou comparison within the tested setting.

Critical weaknesses

Limited novelty. This is the main weakness. The framework combines established diagnostic ideas. Its calendar-based, layer-wise adaptation does not, in my view, provide a substantial methodological advance. Proposition 1 gives an upper bound, and same-day refitting can leave reconstruction error. The Beijing gap establishes an advantage over the fitted table. An advantage over the optimal calendar policy is unclear for me.

Questions for authors
  1. What substantive methodological advance does the combined protocol provide over existing feedback-ablation and replay analyses?
  2. What supports the representativeness of these short evaluation windows and the stability of the findings under seasonal, holiday, and other calendar-driven demand changes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.