acceptodds
Under review as a conference paper at ICLR 2027

LongBusBench: Benchmarking LLM Agents for Long-Horizon Bus Control

Abstract

Large language model (LLM) agents are increasingly capable of long-horizon control, yet their ability to operate continuously in real-world dynamic systems remains largely unexplored. Public transit provides a challenging setting in which control actions have delayed and interacting consequences: for example, bus holding changes vehicle headways and passenger queues, thereby altering the conditions encountered by subsequent vehicles and the future validity of past experience. We introduce LongBusBench, a data-grounded benchmark for evaluating LLM agents in continual and asynchronous bus control under evolving operating conditions. We further propose MEMBus, a memory-augmented LLM agent that structures experience into episodic and procedural memories and selectively retrieves, consolidates, revises, and forgets information towards efficient transit control. Experiments across various operating protocols compare MEMBus with memory-free LLM agents, alternative memory designs, and classical control baselines. Results show that operational memory can improve public transit efficiency, while diagnostic analyses reveal that its effectiveness depends critically on the relevance and validity of accumulated experience. These findings establish long-horizon public-transit control as a practical and promising testbed for studying continual LLM agents in persistent real-world operational systems.

Primary area:
datasets and benchmarks
Keywords:
long-horizon control, LLM agents, agent benchmarks, bus bunching

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ResBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

LongBusBench tests LLM agents on bus control in simulation. The agent sees vehicle states, passenger queues, and pending commands, and decides how long each bus holds at a stop so that buses do not bunch together. The authors want to test whether an agent can stop using outdated experience when operating conditions change, and use it again when the earlier conditions come back. They also propose MEMBus, which stores past control actions and their outcomes so the agent can look them up, and can update, archive, and reactivate these records. Experiments use 10 real routes. In a 30-minute setting, the LLM controls only the first 10 minutes. In an eight-hour setting, it controls the whole run. MEMBus reduces mean passenger waiting time by about 30 s and 72 s compared with the same LLM given only the current state. A simple control that adds no extra holding has the shortest waiting time in the main table. Rules that regulate the spacing between buses make the spacing more even, but passengers wait longer.

Critical strengths

Strength 1: Broad baseline coverage

The baselines make it possible to test whether MEMBus is more effective than simple policies and existing control methods. Tables 2 and 10 include other LLM agents, rule-based controllers, and RL methods, as well as a simple control that adds no extra holding.

Critical weaknesses

Weakness 1: The experiments do not test the long-term experience management that the paper emphasizes

The introduction (lines 66–69) states:

The benchmark is designed to evaluate not only service-level control performance, but also whether an agent can retain useful operational experience, adapt to changing operating regimes, recover knowledge when earlier regimes recur, and avoid harmful reuse of stale or superficially similar memories.

The experiments, however, report only final passenger waiting time, headway regularity, and served fraction. These outcomes cannot show whether the differences observed for MEMBus come from its assessment of the current state, the additional control knowledge it receives, or its management of past experience. The test scenarios, the main comparison, and the ablations each fall short of testing this capability.

First, the test scenarios never produce experience that becomes invalid and later applies again. According to Appendix C.3 (lines 925–927), passenger demand does not vary over time and includes no surges. Vehicle capacity is reduced only during 720–1440 s and 2160–2880 s after simulation start, which falls within warmup and the early control period. Conditions remain unchanged for the rest of the eight-hour run. The only retirement of experience in MEMBus is a scheduled step that removes the initial source conditions from active presentation 300 s after control begins, and the authors state that this removal does not establish that the associated actions have become harmful (Appendix C.5, lines 986–987). The paper therefore never shows an agent detecting that a piece of experience has become invalid, suspending it, and reactivating it once the earlier conditions return. This limitation becomes apparent only in the appendices, and the main text may lead readers to overestimate what is actually tested.

Second, MEMBus outperforms the current-observation LLM in the main comparison, but this gain does not show that it manages experience well. Simpler LLM methods in Table 2 reach a similar level. DiLu only retrieves three similar development examples, and ReAct uses no memory. The mean waiting times of DiLu, ReAct, and MEMBus are 607.67 s, 609.97 s, and 609.53 s in the 1.5h window, and 2679.40 s, 2658.68 s, and 2672.54 s in the 8h window. MEMBus is not clearly better than either method in waiting time, headway regularity, or served fraction. Its only clear advantage is fewer bunching events. MEMBus's mechanisms for updating, suspending, and reactivating experience therefore show no additional benefit on the main service metrics.

The gain over the current-observation LLM may come mainly from records that prompt the model to hold less. Compared with the current-observation control, the memory condition additionally receives development-stage evidence records, recent operating history, and procedural cards. All six external evidence records in Appendix C.5 compare shortening or cancelling one vehicle's hold against keeping the existing plan. In Table 10, the zero-additional-holding control outperforms MEMBus in both mean waiting time and served fraction (mean waiting of 599.20 s vs. 609.53 s in the 1.5h window, and 2539.86 s vs. 2672.54 s in the 8h window). If so, the gain comes from the content of the records themselves and is unrelated to updating, suspending, or reactivating experience.

Third, the ablations do not isolate the experience-management mechanisms. All three configurations in Appendix D keep online procedure formation, archival, and re-entry enabled, and the authors acknowledge that these contrasts cannot separate the effects of these mechanisms (lines 1058–1059). It therefore remains unclear what they add beyond simple retrieval.

I suggest starting from a simple retrieval method such as DiLu, adding MEMBus's updating, suspension, and reactivation mechanisms one at a time, and reporting the benefit of each. The benchmark should also include scenarios in which earlier experience genuinely becomes invalid and later applies again, and report which experience each agent selects, suspends, and reactivates in each phase, along with the resulting differences in control outcomes.

In addition, the paper describes the control problem as asynchronous (line 75), yet the LLM issues fleet-level plans on a fixed decision schedule and the simulation pauses during inference (line 844), so inference latency never affects vehicle operation. Asynchronous control is therefore not tested either.

Section 6.2 (lines 375–387) uses a weighted ranking that places MEMBus first among the six LLM controllers. The main text does not say that these weights were chosen after inspecting the results. This is stated only in Appendix E.7 (lines 1288–1290). The ranking also excludes conventional baselines such as CQL, so it does not show that MEMBus performs best overall.

Bus control is a suitable setting for studying the limitations of LLM agents, but the current experiments do not connect control performance to the capabilities the authors claim to measure. This is the main reason for my rejection recommendation. Addressing it requires targeted scenarios and controls. Adding routes, seeds, or models alone would not resolve this concern.

Weakness 2: Information required for reproduction is incomplete

Appendix E.4 describes the algorithms used by the RL baselines, but for CQL and other controllers it omits the network architecture, training data size, number of training steps, and full hyperparameters, so readers cannot reconstruct these baselines. The full LLM prompts and state input format are also not shown. Appendix C.4 states only that these configurations are retained with the run records. The paper provides no accessible link to code or environment resources. This makes it difficult for other researchers to reproduce the results or to compare new methods under the same conditions. As a benchmark, the submission needs to provide a runnable environment, evaluation configurations, and baseline implementations.

Questions for authors
  1. (Related to Weakness 1) Do you have experiments in which a piece of experience changes from helpful to harmful, and later becomes applicable again when the original conditions return? How do different agents select, suspend, and reactivate experience across these phases, and how do their control outcomes differ?

  2. (Related to Weakness 1) DiLu and ReAct achieve service metrics close to those of MEMBus in Table 2. How much additional benefit do MEMBus's updating, archival, and reactivation mechanisms provide over a simple retrieval method such as DiLu? Do DiLu, ReAct, and MEMBus use the same model?

4 comments

Thank you for the comments.

First of all, we would like to address your major concern: "Bus control is a suitable setting for studying the limitations of LLM agents, but the current experiments do not connect control performance to the capabilities the authors claim to measure." We agree that the original version does not establish the agent capabilities discussed in the paper. The revised manuscripts therefore connect service outcomes to controlled comparisons and observable agent behaviour:

  • Using operational experience. With the same LLM, action interface, and decision schedule, removing historical observations and external memory increases eight-hour waiting from 425 to 879 s (Section 6.3).
  • Adapting to changed conditions and recovering relevant experience. In an A–B–A episode with unannounced regime changes, MEMBus leaves the previously selected programme earlier than the same-interface ExpeL-style agent (2.47 versus 3.75 h). (Section 6.5).
  • Identifying which components produce these behaviours. Section 6.4 and Appendix K.6 include a mechanism ladder, removal ablations, and a full factorial analysis.

In the following, we would like to address your concerns in more detail point by point:

1. Does experience changes from helpful to harmful, and later becomes applicable again? The revised protocol runs A for 1 h, B for 4 h, and A again for 3 h, carrying forward fleet state, queues, and memory. Transition times are withheld from the agents. In B, the programme stored for A produces at least 10% more waiting than the best alternative, then becomes applicable again when A returns. It still outperforms zero additional holding in B, so we define staleness relative to the available alternative. (Appendix K.6.3).

2.How does the agents select across phases. On 25 routes × 5 seeds, MEMBus reduces waiting relative to a same-interface ExpeL-style agent (Table 18). The phase analysis shows similar waiting before the change and a 2.2% reduction during B. In the post hoc traces, MEMBus leaves the stored programme after 2.47 h against 3.75 h for the ExpeL-style agent, and returns to it in A2 in 98% of cases against 12% (Section 6.5). Figure 5 and Appendix K.6.5 show the selection, holding, and waiting trajectories.

3. Benefit beyond retrieval and the original ablations. We agree that the original B1/B12/B123 contrasts did not isolate experience management, because these mechanisms remained enabled in every arm. We therefore added a ladder from a DiLu-style agent, leave-one-out comparisons, and a matched management-on versus management-off comparison (Table 3; Appendix K.6.3).

We additionally examine mechanism interactions through a full factorial analysis using a scripted online learner (Appendix K.6.4). The revised analyses therefore provide a more targeted assessment of experience management.

4. DiLu, ReAct, zero holding, and model identity. The original table mixed backbones. Appendix K.1 now discloses this and provides a same-model re-estimate on the original testbed. It retains the close results for MEMBus, ReAct, and DiLu, with zero holding numerically ahead at eight hours.

All revised main-table LLM arms use GPT-6-Luna. On the rebuilt testbed, eight-hour waiting is 425 s for MEMBus, 1,066 s for zero holding, and 424 s for tuned headway feedback. The testbed was rebuilt after inspecting the original results, and we report that limitation explicitly.

5. Asynchronous execution. Bus-level holding opportunities arise at different stop-arrival times, a characteristic discussed by Wang and Sun (2021). LongBusBench issues fleet plans periodically and applies individual commands at eligible stops.

The main runs pause simulated time during API calls. Appendices K.7 and K.12 now evaluate measured response delays with the simulator advancing under the previous programme. MEMBus waiting increases from 424.9 to 437.9 s.

A separate controlled test applies each answer one 300 s tick late. Recovery-phase waiting remains 7.95% lower than same-interface ExpeL on 25 routes, with reductions of 10.1–12.3% on three additional backbones. This extends the latency analysis to the memory-recovery task.

  • Wang, J., & Sun, L. (2021). Reducing Bus Bunching with Asynchronous Multi-Agent Reinforcement Learning. IJCAI, 426–433.

6. Weighted ranking. We have withdrawn the LLM-only ranking in the revised manuscript. The revision reports raw outcomes and paired tests, with CQL and other conventional controllers shown directly (per-metric tests: Section 6.2, Eq. (5); Appendix K.5, Table 15).

7. Reproducibility. Appendices K.2–K.4 now provide more details; Figure 2 traces one logged review. Appendix K.8 specifies the relative architecture, and training details. Archived responses support replay, while fresh API calls can yield different decisions. Independent execution requires access to the runnable package, which should remain internal during review.

AuthorsOpenReview ↗

Update manuscript

We have updated the manuscript to incorporate your suggestions into the additional experiments and bencharmk dicussion. Thank you again for your constructive feedback. We respectfully ask you to reconsider your assessment in light of these updates.

Thanks for your response. But the revision changes the method entirely not just addd experiments or specific scenario which i mentioned in w1. For example, you change the testbed and the action space in your main experiment. So the new results come from a different setting from the one I reviewed and they can't answer my quesions. The abstract is also unchanged and no longer matches the revised experiments. Even worse, you compare different methods with different models in you original submisisons. Although you have revise them,there are still many things make me worry about how you manage your experiments and how you ensure your results are reliable. For example, ReAct was described as using tools, but its prompt said "no tools" and the training scripts of four RL baselines cannot be recovered.

So I will keep my score.

Thank you for your time. We would like to brietly clarify the following points since there is not sufficient time left for reviewer-author discussion:

  1. Changed testbed and the action space. The new testbed and action space are updated to introduce more real-world constraints to enhace reliability. We rerun all the models and comparisons and all the conclusions are maintained after new setting. In addition, we continue to improve the detail setting with in our method after original submission and the method revised can outperform most LLM baselines. The reason we do not modify abstract is that we would like to position this study as a bechmark study and we suggest highlight the performance of revised method is not an urgent objective.

  2. ReAct issue. The revised manuscript has already evaluated ReAct as a tool-used variant, shown in Appendix E.2, line 1223. In addition, we have evaluate both tool-used and no-tool version. The appendix has shown that there is no significant difference among these variants.

  3. Training scripts of four RL baselines. We will opensource the whole benchamark.

Reviewer_unjvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper builds an agent evaluation benchmark for bus dispatching as a long-horizon task. The benchmark data comes from 95 public transit lines across four major Chinese cities, enriched using prior work such as CPTOND, TransitLM, and SUMO, and is exposed to agents through an API so that they can be evaluated via interaction with the environment. On top of the benchmark, the authors design a memory-augmented bus-control agent that organizes historical records into experience-operation cards which a downstream decision-and-control agent can later retrieve, and show that it outperforms several conventional baselines on the benchmark.

Critical strengths
  1. Bus dispatching is a good long-horizon decision-making task for agents, and this work makes a tangible contribution in constructing the underlying task data.
Critical weaknesses
  1. The set of evaluated agents is very limited. Although the authors compare 15 different baselines, it should be emphasized that, for an agent-oriented benchmark, at least two base models together with their external harnesses should be compared. The experiments in this paper are run on a single base model (gpt-5.6-luna) with a subset of harness-construction methods (e.g., ReAct, Self-Refine), which may limit the practical reference value of the benchmark.
  2. The complicated formal definitions impose a high cost on operability and comprehension. Although the authors try, within the limited space, to formally define the task itself and the various parameters the evaluation environment relies on, these definitions appear incomplete, which easily leads readers to doubt the feasibility and rigor of the work. For readers who wish to reuse the work, I suggest grounding the formal definitions in the main text in operability: concretely, spell out the actual inputs and outputs, ideally with a worked example (e.g., around Figure 1), rather than giving only a schematic description of the pipeline.
Questions for authors

I would encourage the authors to focus primarily on the two key issues raised in the Weaknesses. The minor points below are meant to help polish the exposition; the questions that materially affect my assessment are those in the Weaknesses section.

  1. What is the difference between the latest available observation o and r? Can o be understood as r_new? Given that r is already defined, why is it necessary to additionally define o?
  2. How exactly are the weights of the weighted evaluation obtained? Would assigning identical weights to all metrics change the evaluation results?
  3. What is the actual role of Eq. (7)?
  4. Line 127 mentions multiple controllers, whereas the later description (e.g., line 292) seems to involve a single controller. Should the controllers at line 127 be understood as multiple instantiations of one controller?
4 comments

Thank you for identifying two priorities for the revision. We expanded the backbone and harness comparisons and added a worked example that connects the task definitions to an actual controller decision.

Weakness 1. Backbone and harness coverage

The revised main table evaluates both harnesses you highlighted, ReAct and Self-Refine, alongside DiLu and MEMBus on the same GPT-6-Luna backbone. ReAct uses a reasoning and tool-use loop, while Self-Refine uses a draft–feedback–revision cycle. We also added RAG, ExpeL, Generative Agents, and a MemGPT-style agent to broaden coverage of memory-based harnesses.

We then evaluated four harnesses and MEMBus across multiple backbones. Eight-hour mean passenger waiting, 17 routes × 5 seeds (Appendix K.10, Table 23):

BackboneReActDiLuExpeLRAGMEMBus
GPT-6-Luna (main)1,261.4 s1,264.7 s875.6 s1,033.1 s512.9 s
GLM 5.3 Flash851.5 s998.0 s623.7 s612.5 s512.9 s
Nemotron 3.5 Super VL1,254.1 s1,259.8 s1,047.1 s1,252.8 s512.9 s
Nemotron 3 Ultra1,130.3 s1,272.4 s872.7 s965.3 s512.9 s

MEMBus has the same value on every backbone. This is not a copying error: every model accepted the memory layer's recommendation at every review, and the executor and simulator are deterministic for a given seed. The model matters when the recommendation is wrong: replaced by the lightest-holding programme, five of seven backbones override it in 85–100% of reviews, while two copy it (Appendix K.10).

We also controlled for the action interface. Appendix J.4 (Table 12) gives ReAct, RAG, ExpeL, and Reflexion the same programme interface as MEMBus. ReAct’s waiting time drops to 676.0 s with that interface. ExpeL, given the same interface and development evidence, reaches 429.8 s, close to MEMBus’s 424.9 s. This comparison shows why we report the interface effect separately from the memory comparison.

Weakness 2. Concrete inputs and outputs

Section 5.2 describes what the controller receives and returns. Following your suggestion, we added Figure 2 in Section 6.1, which traces a logged decision on route 484 and connects each stage to the notation in Section 3, including oto_t, Ht\mathcal H_t, Mt\mathcal M_t, atpropa_t^{\mathrm{prop}}, and atexeca_t^{\mathrm{exec}}. Appendix K.2 provides the field definitions.

  1. Input. At simulation time 2,370 s, the controller receives vehicle states and headways, stop queues, pending holds, feasible actions, recent history, and programme cards.
  2. Agent output. The LLM returns a structured SELECT response identifying a headway-balancing programme.
  3. Execution. The executor converts the programme into CANCEL, SET 15.52 s, and SET 90 s for the three buses, validates the commands, and applies them at eligible stops. At the next two 300-second ticks, it recomputes commands from the updated state using the same programme, without another LLM call.

Appendix K.3 also provides captured requests and responses for four harnesses.

Answers to the remaining questions

  1. oto_t versus rjr_j. A record rjr_j can be an observation, message, prior action, or execution receipt. oto_t is the latest observation-type record available at time tt . If the newest record is a command receipt, oto_t still refers to the preceding observation. We name oto_t separately to identify the snapshot used by the current-observation baseline.
  2. Metric weights. The original composite assigned 30% each to waiting and bunching, 15% each to headway variability and served fraction, and 10% to token use. The short and long windows received 30% and 70%, respectively. These weights were chosen after inspecting the results. With equal weights, DiLu ranks first at 2.30 versus 2.32 for MEMBus, with lower scores preferred.On the rebuilt testbed, MEMBus ranks first among the original-interface LLM agents under all 286 weight settings we tried (Appendix K.10). We withdrew the original composite ranking as evidence and now report individual metrics with paired route-level tests (Section 6.2; Appendix K.5).
  3. Eq. (7). Eq. (7) only names the nested ablation differences, Δupdate=YB12−YB1\Delta_{\mathrm{update}}=Y_{B12}-Y_{B1} and Δexit=YB123−YB12\Delta_{\mathrm{exit}}=Y_{B123}-Y_{B12}; it carries no mechanism of its own. The revision keeps it (now Eq. (6) in Section 6.4), adds Δmgmt=Yfull−Ystatic\Delta_{\mathrm{mgmt}}=Y_{\mathrm{full}}-Y_{\mathrm{static}} for the same agent with management switched off, and reports all contrasts in Table 3.
  4. Number of controllers. The index ii accommodates multiple controllers in the general formulation. The experiments use one centralized fleet controller. The LLM selects a programme every 900 seconds, and a deterministic executor issues bus-level commands every 300 seconds.
AuthorsOpenReview ↗

Update manuscript

We have updated the manuscript to incorporate your suggestions into the additional experiments and user friendly workflow introduction. Thank you again for your constructive feedback. We respectfully ask you to reconsider your assessment in light of these updates.

I thank the authors for their detailed response, which resolved my concerns. I also read their responses to the other two reviewers. After considering the rebuttal as a whole, I have decided to keep my original score. Overall, I consider this a strong paper.

Thank you for your time and acknowledgement of our work.

Reviewer_UgknOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a benchmark for evaluating LLM agents on bus holding control. Because bus scheduling actions can cause delayed and propagating consequences across vehicle headways and passenger queues, agents cannot misapply stale policies. The authors propose a memory framework that structures experience into episodic and procedural memory, supporting selective retrieval, evidence updates, suspension, and reactivation based on real-time and downstream utility. Across two control protocols, incorporating this structured operational memory reduces passenger waiting times and improves service responsiveness compared with memory-free agents.

Critical strengths
  • They propose to build operational experience into episodic and procedural memory to address data shifts, evidence validity, and stale memory reuse.

  • The evaluation spans two time-span horizons against LLMs, RL, and classical baselines. They also include memory ablations to validate their effectiveness of their memory mechanisms.

Critical weaknesses
  • As a benchmark paper, the authors only evaluate a single small LLM (i.e., GPT-5.6-Luna). Moreover, general memory agents (e.g., MemGPT, Voyager) are cited but omitted from the baselines. This strongly limits their generalizability across different LLMs and agents.

  • Although the authors emphasize their contributions to "real-world operational control", the experiments rely on simulations with simplified assumptions. No real-world tests or sim-to-real fidelity and transfer validation are included.

Questions for authors
  • Why is the evaluation limited to one LLM and ignoring other memory agent frameworks? Please clarify the scope and provide additional comparisons or justify their omission.

  • Could you provide the evidence supporting transfer to actual transit operations? Please clarify the benchmark’s claims and discuss sim-to-real limitations.

4 comments

Thank you for highlighting the need for broader agent coverage and a clearer account of what the simulator establishes. We addressed both points in the revision.

1. LLM backbones and memory-agent baselines. The revised main comparison uses the same backbone, gpt-6-luna, across all LLM arms. Appendix K.10 (Tables 22–24) crosses MEMBus, ReAct, and DiLu with additional backbones, including MiniMax, DeepSeek, GLM, and Nemotron models. It reports an eight-route comparison and a 17-route extension, with reduced coverage and format or context failures. Figures 3 and 4 in the main text summarise seven backbones on 17 routes: MEMBus waits less than ReAct and DiLu on all seven.

To include more memory agents as you suggested, we also added RAG, ExpeL, Generative Agents, and a MemGPT-style baseline to the main comparison, and a Voyager-style baseline in Appendix K.9. On the main table the MemGPT-style agent waits 1,068 s against 425 s for MEMBus; against a strengthened Voyager-style agent, MEMBus waits 3.53% less on 25 routes. Both are transit adaptations; Appendix K.9 documents the simplifications and how each agent obtains its programme library.

The recurring-regime experiment provides the direct comparison of memory management. MEMBus reduces recovery-phase waiting by 5.88% relative to same-interface ExpeL on 25 routes, Appendix K.6.3). The lifecycle comparison also shows recovery-phase reductions of 9.6–13.1% on three additional backbones (Figure 4b, Appendix K.6.7).

2. Simulation fidelity and transfer to actual operations

LongBusBench fills a gap in long-horizon agent evaluation for transit control. It evaluates fleet management over extended service periods, where holding one bus affects later headways, queues, and passenger waiting. Its inputs and outputs closely match real-world transit dispatch. Inputs include vehicle positions, stop arrivals, headways, passenger loads or queue estimates, and pending commands, which can be populated from operational data feeds where available. Outputs specify a bus and a bounded holding instruction at an eligible stop, providing a practical interface for integration with dispatch systems.

Holding control is a bounded dispatch intervention with field precedent. Transit operators have implemented real-time holding on high-frequency routes (Berrebi et al., 2018). LongBusBench checks vehicle eligibility, pending-command rules, and holding bounds, with requested holds capped at 90 seconds. In deployment, these instructions would adjust departure times while drivers retain control of vehicle movement.

LongBusBench is grounded in curated real-world route data. We manually reviewed and cleaned a subset of source records and combined this work with automated map matching and execution checks. The catalogue covers 95 routes across four cities, including 92 road-network bindings passing strict validation and three reconstructed bindings documented separately. Based on the real-world data, we use SUMO, a well-established microscopic traffic simulator, to model individual vehicle movements (Alvarez Lopez et al., 2018), which is widely used in traffic engineering and control simulation task to evaluate traffic control policy (Vinitsky et al., 2018, Ault and Sharon, 2021).

We also quantify the remaining simulation gap. Appendix K.11 compares simulated supply with timetable and trip-planner estimates and tests a timetable-driven supply variant. Across 22 routes with independent planner estimates, simulated one-way running times were a median 7.2% shorter. These checks support the benchmark’s operational grounding, while passenger demand and service dynamics remain partly synthetic. Establishing field performance would require calibration against observed vehicle trajectories and passenger counts, followed by an operator-supervised pilot.

References

  • Alvarez Lopez, P., et al. (2018). Microscopic Traffic Simulation using SUMO. IEEE ITSC, 2575–2582.
  • Berrebi, S. J., Crudden, S. Ó., & Watkins, K. E. (2018). Translating research to practice: Implementing real-time control on high-frequency transit routes. Transportation Research Part A, 111, 213–226.
  • Eclipse SUMO.. Public Transport and TraCI: Change Vehicle State. Eclipse SUMO Documentation.
  • Vinitsky, E., et al. (2018). Benchmarks for Reinforcement Learning in Mixed-Autonomy Traffic. In Proceedings of the 2nd Conference on Robot Learning, PMLR 87, pp. 399–409.
  • Ault, J., & Sharon, G. (2021). Reinforcement Learning Benchmarks for Traffic Signal Control. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Vol. 1.
AuthorsOpenReview ↗

Update manuscript

We have updated the manuscript to incorporate your suggestions into the additional experiments and sim-to-real discussion. Thank you again for your constructive feedback. We respectfully ask you to reconsider your assessment in light of these updates.

Reviewer_UgknOpenReview ↗

Response

Thank you for the response. However, as a benchmark paper, this work is primarily based on simulations. The evaluation of the sim-to-real gap, fidelity calibration, or real-world experiments are not included. Based on this evaluation. I tend to keep my score.

Thank you for your time. We agree the sim-to-real gap is general and a challenge for any bechmark.

However, in transit control domain, we believe this is a significant step to close the sim-to-real gap for benchamrking LLM agent control by collecting, organzing and manually reviewing the real-world data for a well-calibrated simulation, as well as evaluating on various LLM agents including the proposed MemBus.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.