- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AMHSTrafficLab, an industrially motivated benchmark for evaluating routing and regional traffic-control algorithms in AMHS, particularly OHT networks used in semiconductor manufacturing. The benchmark models large-scale directed transport networks with persistent vehicle fleets, dynamically arriving pickup-and-delivery tasks, vehicle-following dynamics, shared-resource competition, intersection access coordination, and regional traffic-control mechanisms. A central motivation is that routing algorithms cannot be evaluated independently of the traffic-control policies governing access to congested regions and shared resources. To support reproducible evaluation, AMHSTrafficLab provides configurable simulation scenarios, standardized policy interfaces, and implementations of classical, heuristic, and learning-based routing algorithms. The main results show that throughput collapse occurs frequently without regional control. Across 910 runs, the number of collapse cases decreases from 265 without regional control to 9 under regional admission control, and further to 3 when empty-vehicle rebalancing is added.
Critical strengths
Strength 1
The paper addresses an important limitation of existing routing evaluations: the behavior of a routing policy depends not only on its path-selection strategy but also on the traffic-control mechanisms governing access to shared resources. AMHSTrafficLab provides a unified simulation environment. It allows researchers to study routing policies under realistic forms of resource competition and evaluate system-level failures that may not be adequately reflected by conventional shortest-path or travel-time metrics.
Strength 2
The paper evaluates 13 routing algorithms across seven scenarios, three regional-control settings, and ten evaluation seeds per configuration. The benchmark covers different network layouts, fleet sizes, traffic loads, and routing strategies, including classical shortest-path algorithms, congestion-aware heuristics, and reinforcement-learning-based approaches. The experiments use common task streams and initial vehicle states within matched scenario–seed configurations, improving the comparability of the results. An important finding is that learning-based methods do not consistently outperform congestion-aware heuristics.
Critical weaknesses
Weakness 1
The experimental evidence does not sufficiently establish how closely the simulated system reproduces the behavior of real industrial AMHS deployments. In particular, the paper does not provide a direct comparison between simulated and observed operational measurements, such as vehicle travel times, queue lengths, congestion patterns, task cycle-time distributions, or deadlock frequencies. The benchmark also relies on workloads generated from fixed From–To demand matrices. Although tasks arrive dynamically, the underlying spatial demand distributions remain fixed within each scenario. In actual semiconductor manufacturing environments, transport demand can vary substantially with production schedules, equipment availability, and process disruptions.
Weakness 2
The paper evaluates routing performance using several complementary metrics, which is a strength. However, the primary efficiency comparisons are based on averages computed only over runs that do not experience throughput collapse. A routing policy that performs well in its successful runs but occasionally experiences severe congestion or collapse may appear competitive in terms of mean cycle time, even though its overall operational reliability is inferior to that of a consistently stable policy. This issue is especially relevant in industrial transport systems, where rare but severe failures can be more consequential than modest improvements in average travel efficiency. The detailed appendices are valuable, but documentation alone does not fully substitute for an independently executable benchmark.
Questions for authors
Q1
The paper introduces a comprehensive industrial simulation benchmark for routing and regional traffic control in automated material handling systems. While the benchmark appears practically valuable, its primary contributions seem more closely related to industrial simulation, transportation optimization, and operations research than to machine learning methodology. Could the authors clarify the intended contribution of AMHSTrafficLab to the broader machine learning community and, specifically, its relevance to ICLR? What novel machine learning research questions does this benchmark enable that cannot be adequately investigated using existing multi-agent reinforcement learning or transportation benchmarks?
Q2
The benchmark focuses on routing and regional traffic control in industrial transport networks, a problem closely related to established operations research and combinatorial optimization. This raises a fundamental question regarding the motivation for introducing this problem as a machine learning benchmark. In particular, what unique machine learning challenges does AMHSTrafficLab introduce that cannot be adequately addressed by existing optimization or control methods? Without demonstrating such opportunities, the contribution appears primarily to be an industrial simulation and algorithm-comparison platform rather than a benchmark that addresses a compelling machine learning research problem.
Q3
The benchmark incorporates layouts and workloads derived from semiconductor manufacturing environments, but the paper does not present a direct comparison with observed operational measurements. Could the authors clarify whether any simulation components have been calibrated or validated against real AMHS operational logs?
Q4
The paper shows that admission control substantially reduces throughput collapse, but the baseline relies on predefined regional water-level thresholds and region configurations. How sensitive are the reported improvements to these choices?
4 comments
Check in
Hi Reviewer 1Uh6,
We are grateful for your thorough feedback, which helped us better articulate the paper's relevance to the ML community and strengthen our real-world validation and reliability evaluation. Before the discussion period concludes, we wanted to check in on whether our responses have resolved your concerns. Please let us know if there is anything else we can clarify—we would be happy to follow up. Thank you again for your valuable input.
Thank you for your detailed rebuttal and supplementary experiments.
However, I still do not believe that the proposed benchmark makes a sufficiently compelling contribution to the machine learning community.
Despite the additional experiments, the study remains primarily focused on the fields of industrial simulation and traffic optimization, and lacks sufficient evidence to demonstrate that it addresses unique machine learning challenges or offers methodological insights relevant to ICLR.
Therefore, I regret to say that I am upholding my original score.
Dear Reviewer 1Uh6,
Thank you for reading our rebuttal and for your candid feedback. We understand and respect your decision, and we truly appreciate your time throughout the review process.
Best regards, The Authors
We thank the reviewer for recognizing the value of our work and for the insightful feedbacks. We explain the reviewer's concerns and answer the questions below.
1. Relevance to ICLR and the learning problem (Q1, Q2)
Our first contribution is that we expose the AMHS scene, a scene of great practical importance, as a complex joint-action traffic MDP, and provides a platform for publicly learning and fairly evaluating policies targeting this scene. In the MDP, vehicle routers choose paths and regional controllers regulate entry or rebalance idle vehicles; both are agents whose decisions determine fleet movement and task completion.
This MDP is a complex one: even with a fixed regional policy, two routes for each of m vehicles give 2m route assignments. A vehicle's travel time depends on the routes and entry permissions of others. A queue outside one region can extend through a merge and delay vehicles whose routes never hit that region. Targeting routers in traffic systems of such complexity, we raise a question: With regional control fixed, which parts of the traffic state should a router observe, what history should it retain, and how should it represent value or cost to improve fleet performance?
Under admission control, exponentially averaged delay routing (EDR) had the lowest mean task cycle time among collapse-free methods in all seven scenarios (Table 3). Its decision process uses two designs:
We added a matched DQN experiment (Section 5.2). DQN learns destination-conditioned next-hop values from traffic features. We trained and evaluated DQN with or without QEDR as an additional feature for each candidate next hop. Independently, we either froze DQN's weights during evaluation or updated them online from new traversal times. Below, each cell reports throughput (tasks/h) / mean task cycle time (s) over ten matched seeds (0–9), with 20,000 simulated seconds per run under CA. All 120 runs were collapse-free.
Results show that under regional control, DQN achieves shorter task cycles in all comparisons when supplying fleet-wide traversal history as a shared, scalable cost, motivating future machine learning algorithms to utilize or exploit these two aspects.
2. Industrial calibration (Q3, W1)
In earlier collaboration with our industrial partners, we anonymized facility maps, calibrated From–To demand from industrial process routes, and set vehicle-motion parameters using manufacturer specifications. The operational data and manufacturers' identities cannot be disclosed for confidentiality. However, certain aspects we adopted in layouts, transport demand, and vehicle motion already appear in relevant literature. We added Appendix H.1–H.3 to the paper for explanation.
The general idea of admission control comes from industry, but the control regions and water-level thresholds are specifically engineered for our benchmark topologies. See the answer to Q4 below.
3. Region and threshold sensitivity (Q4)
We amended Appendix H.4 to explain the design rationale of regional control and to exhibit a sensitivity analysis using 10 alternative regional configurations in LARGE 300. Without regional control, Dijkstra, EDR, and QLBWR experienced throughput collapse in 10/10, 6/10, and 9/10 runs, respectively; across the 10 alternative admission configurations, the highest collapse counts for the same methods were 4/10, 0/10, and 3/10, respectively (Tables 51 and 55). The reduction in collapse therefore persists across the tested regional configurations with From–To demand held fixed.
4. Interpreting the metrics (W2)
We thank the reviewer for pointing this out, and we admit that the original paper did not make it clear. Because sustained service is essential, collapse counts over all seeds take precedence. The reported cycle-time and throughput means should be fairly comparable only among policies with zero observed collapses on the same scenario and seeds.