- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
LongBusBench tests LLM agents on bus control in simulation. The agent sees vehicle states, passenger queues, and pending commands, and decides how long each bus holds at a stop so that buses do not bunch together. The authors want to test whether an agent can stop using outdated experience when operating conditions change, and use it again when the earlier conditions come back. They also propose MEMBus, which stores past control actions and their outcomes so the agent can look them up, and can update, archive, and reactivate these records. Experiments use 10 real routes. In a 30-minute setting, the LLM controls only the first 10 minutes. In an eight-hour setting, it controls the whole run. MEMBus reduces mean passenger waiting time by about 30 s and 72 s compared with the same LLM given only the current state. A simple control that adds no extra holding has the shortest waiting time in the main table. Rules that regulate the spacing between buses make the spacing more even, but passengers wait longer.
Critical strengths
Strength 1: Broad baseline coverage
The baselines make it possible to test whether MEMBus is more effective than simple policies and existing control methods. Tables 2 and 10 include other LLM agents, rule-based controllers, and RL methods, as well as a simple control that adds no extra holding.
Critical weaknesses
Weakness 1: The experiments do not test the long-term experience management that the paper emphasizes
The introduction (lines 66–69) states:
The benchmark is designed to evaluate not only service-level control performance, but also whether an agent can retain useful operational experience, adapt to changing operating regimes, recover knowledge when earlier regimes recur, and avoid harmful reuse of stale or superficially similar memories.
The experiments, however, report only final passenger waiting time, headway regularity, and served fraction. These outcomes cannot show whether the differences observed for MEMBus come from its assessment of the current state, the additional control knowledge it receives, or its management of past experience. The test scenarios, the main comparison, and the ablations each fall short of testing this capability.
First, the test scenarios never produce experience that becomes invalid and later applies again. According to Appendix C.3 (lines 925–927), passenger demand does not vary over time and includes no surges. Vehicle capacity is reduced only during 720–1440 s and 2160–2880 s after simulation start, which falls within warmup and the early control period. Conditions remain unchanged for the rest of the eight-hour run. The only retirement of experience in MEMBus is a scheduled step that removes the initial source conditions from active presentation 300 s after control begins, and the authors state that this removal does not establish that the associated actions have become harmful (Appendix C.5, lines 986–987). The paper therefore never shows an agent detecting that a piece of experience has become invalid, suspending it, and reactivating it once the earlier conditions return. This limitation becomes apparent only in the appendices, and the main text may lead readers to overestimate what is actually tested.
Second, MEMBus outperforms the current-observation LLM in the main comparison, but this gain does not show that it manages experience well. Simpler LLM methods in Table 2 reach a similar level. DiLu only retrieves three similar development examples, and ReAct uses no memory. The mean waiting times of DiLu, ReAct, and MEMBus are 607.67 s, 609.97 s, and 609.53 s in the 1.5h window, and 2679.40 s, 2658.68 s, and 2672.54 s in the 8h window. MEMBus is not clearly better than either method in waiting time, headway regularity, or served fraction. Its only clear advantage is fewer bunching events. MEMBus's mechanisms for updating, suspending, and reactivating experience therefore show no additional benefit on the main service metrics.
The gain over the current-observation LLM may come mainly from records that prompt the model to hold less. Compared with the current-observation control, the memory condition additionally receives development-stage evidence records, recent operating history, and procedural cards. All six external evidence records in Appendix C.5 compare shortening or cancelling one vehicle's hold against keeping the existing plan. In Table 10, the zero-additional-holding control outperforms MEMBus in both mean waiting time and served fraction (mean waiting of 599.20 s vs. 609.53 s in the 1.5h window, and 2539.86 s vs. 2672.54 s in the 8h window). If so, the gain comes from the content of the records themselves and is unrelated to updating, suspending, or reactivating experience.
Third, the ablations do not isolate the experience-management mechanisms. All three configurations in Appendix D keep online procedure formation, archival, and re-entry enabled, and the authors acknowledge that these contrasts cannot separate the effects of these mechanisms (lines 1058–1059). It therefore remains unclear what they add beyond simple retrieval.
I suggest starting from a simple retrieval method such as DiLu, adding MEMBus's updating, suspension, and reactivation mechanisms one at a time, and reporting the benefit of each. The benchmark should also include scenarios in which earlier experience genuinely becomes invalid and later applies again, and report which experience each agent selects, suspends, and reactivates in each phase, along with the resulting differences in control outcomes.
In addition, the paper describes the control problem as asynchronous (line 75), yet the LLM issues fleet-level plans on a fixed decision schedule and the simulation pauses during inference (line 844), so inference latency never affects vehicle operation. Asynchronous control is therefore not tested either.
Section 6.2 (lines 375–387) uses a weighted ranking that places MEMBus first among the six LLM controllers. The main text does not say that these weights were chosen after inspecting the results. This is stated only in Appendix E.7 (lines 1288–1290). The ranking also excludes conventional baselines such as CQL, so it does not show that MEMBus performs best overall.
Bus control is a suitable setting for studying the limitations of LLM agents, but the current experiments do not connect control performance to the capabilities the authors claim to measure. This is the main reason for my rejection recommendation. Addressing it requires targeted scenarios and controls. Adding routes, seeds, or models alone would not resolve this concern.
Weakness 2: Information required for reproduction is incomplete
Appendix E.4 describes the algorithms used by the RL baselines, but for CQL and other controllers it omits the network architecture, training data size, number of training steps, and full hyperparameters, so readers cannot reconstruct these baselines. The full LLM prompts and state input format are also not shown. Appendix C.4 states only that these configurations are retained with the run records. The paper provides no accessible link to code or environment resources. This makes it difficult for other researchers to reproduce the results or to compare new methods under the same conditions. As a benchmark, the submission needs to provide a runnable environment, evaluation configurations, and baseline implementations.
Questions for authors
-
(Related to Weakness 1) Do you have experiments in which a piece of experience changes from helpful to harmful, and later becomes applicable again when the original conditions return? How do different agents select, suspend, and reactivate experience across these phases, and how do their control outcomes differ?
-
(Related to Weakness 1) DiLu and ReAct achieve service metrics close to those of MEMBus in Table 2. How much additional benefit do MEMBus's updating, archival, and reactivation mechanisms provide over a simple retrieval method such as DiLu? Do DiLu, ReAct, and MEMBus use the same model?
4 comments
Update manuscript
We have updated the manuscript to incorporate your suggestions into the additional experiments and bencharmk dicussion. Thank you again for your constructive feedback. We respectfully ask you to reconsider your assessment in light of these updates.
Thanks for your response. But the revision changes the method entirely not just addd experiments or specific scenario which i mentioned in w1. For example, you change the testbed and the action space in your main experiment. So the new results come from a different setting from the one I reviewed and they can't answer my quesions. The abstract is also unchanged and no longer matches the revised experiments. Even worse, you compare different methods with different models in you original submisisons. Although you have revise them,there are still many things make me worry about how you manage your experiments and how you ensure your results are reliable. For example, ReAct was described as using tools, but its prompt said "no tools" and the training scripts of four RL baselines cannot be recovered.
So I will keep my score.
Thank you for your time. We would like to brietly clarify the following points since there is not sufficient time left for reviewer-author discussion:
-
Changed testbed and the action space. The new testbed and action space are updated to introduce more real-world constraints to enhace reliability. We rerun all the models and comparisons and all the conclusions are maintained after new setting. In addition, we continue to improve the detail setting with in our method after original submission and the method revised can outperform most LLM baselines. The reason we do not modify abstract is that we would like to position this study as a bechmark study and we suggest highlight the performance of revised method is not an urgent objective.
-
ReAct issue. The revised manuscript has already evaluated ReAct as a tool-used variant, shown in Appendix E.2, line 1223. In addition, we have evaluate both tool-used and no-tool version. The appendix has shown that there is no significant difference among these variants.
-
Training scripts of four RL baselines. We will opensource the whole benchamark.
Thank you for the comments.
First of all, we would like to address your major concern: "Bus control is a suitable setting for studying the limitations of LLM agents, but the current experiments do not connect control performance to the capabilities the authors claim to measure." We agree that the original version does not establish the agent capabilities discussed in the paper. The revised manuscripts therefore connect service outcomes to controlled comparisons and observable agent behaviour:
In the following, we would like to address your concerns in more detail point by point:
1. Does experience changes from helpful to harmful, and later becomes applicable again? The revised protocol runs A for 1 h, B for 4 h, and A again for 3 h, carrying forward fleet state, queues, and memory. Transition times are withheld from the agents. In B, the programme stored for A produces at least 10% more waiting than the best alternative, then becomes applicable again when A returns. It still outperforms zero additional holding in B, so we define staleness relative to the available alternative. (Appendix K.6.3).
2.How does the agents select across phases. On 25 routes × 5 seeds, MEMBus reduces waiting relative to a same-interface ExpeL-style agent (Table 18). The phase analysis shows similar waiting before the change and a 2.2% reduction during B. In the post hoc traces, MEMBus leaves the stored programme after 2.47 h against 3.75 h for the ExpeL-style agent, and returns to it in A2 in 98% of cases against 12% (Section 6.5). Figure 5 and Appendix K.6.5 show the selection, holding, and waiting trajectories.
3. Benefit beyond retrieval and the original ablations. We agree that the original B1/B12/B123 contrasts did not isolate experience management, because these mechanisms remained enabled in every arm. We therefore added a ladder from a DiLu-style agent, leave-one-out comparisons, and a matched management-on versus management-off comparison (Table 3; Appendix K.6.3).
We additionally examine mechanism interactions through a full factorial analysis using a scripted online learner (Appendix K.6.4). The revised analyses therefore provide a more targeted assessment of experience management.
4. DiLu, ReAct, zero holding, and model identity. The original table mixed backbones. Appendix K.1 now discloses this and provides a same-model re-estimate on the original testbed. It retains the close results for MEMBus, ReAct, and DiLu, with zero holding numerically ahead at eight hours.
All revised main-table LLM arms use GPT-6-Luna. On the rebuilt testbed, eight-hour waiting is 425 s for MEMBus, 1,066 s for zero holding, and 424 s for tuned headway feedback. The testbed was rebuilt after inspecting the original results, and we report that limitation explicitly.
5. Asynchronous execution. Bus-level holding opportunities arise at different stop-arrival times, a characteristic discussed by Wang and Sun (2021). LongBusBench issues fleet plans periodically and applies individual commands at eligible stops.
The main runs pause simulated time during API calls. Appendices K.7 and K.12 now evaluate measured response delays with the simulator advancing under the previous programme. MEMBus waiting increases from 424.9 to 437.9 s.
A separate controlled test applies each answer one 300 s tick late. Recovery-phase waiting remains 7.95% lower than same-interface ExpeL on 25 routes, with reductions of 10.1–12.3% on three additional backbones. This extends the latency analysis to the memory-recovery task.
6. Weighted ranking. We have withdrawn the LLM-only ranking in the revised manuscript. The revision reports raw outcomes and paired tests, with CQL and other conventional controllers shown directly (per-metric tests: Section 6.2, Eq. (5); Appendix K.5, Table 15).
7. Reproducibility. Appendices K.2–K.4 now provide more details; Figure 2 traces one logged review. Appendix K.8 specifies the relative architecture, and training details. Archived responses support replay, while fresh API calls can yield different decisions. Independent execution requires access to the runnable package, which should remain internal during review.