- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors designed DeliveryGym to train and evaluate agents on continuous courier shifts, in which there are long-term dependencies such as time, energy and money that needs to be considered beyond the immediate task. It builds on top of DeliveryBench, and adds features such as the state transitions and reward feedback to convert it into an RL environment for long-term planning. The RL framework optimizes the agents’ policy based on their overall profit (pay - expenses). The safety variant also considers red-light violations and obstacle collisions. The authors performed evaluation of six foundation models, along with a GRPO-trained Qwen3-VL-4B model.
Critical strengths
- The environment is complex and the tasks well-designed, with city navigation, traffic avoidance, resource / cost management, and potential interaction with other actors in the scene.
- This paper demonstrates an insightful use-case of the DeliveryBench environment for complex long-term planning tasks, and provides useful baselines / benchmarks in an RL-friendly setting.
Critical weaknesses
- The paper describes the DeliveryGym environment that the authors created but the paper at a high-level, but does not go into technical depth in terms of their development and a lot of heavy-lifting seems to be done by the DeliveryBench backbone. The paper should state clearly which elements of the environment and evaluation were already present in the DeliveryBench work and what has been added / modified.
- The main hypothesis of the paper LL: 077-079 “long-horizon embodied planning and reasoning can be improved through RL when operationally structured tasks are coupled with executable world dynamics and verifiable trajectory rewards” is not a compelling hypothesis since it is almost evident that it is true (RL can improve it given enough training epochs). Also, there may be other non-RL (reward-free) ways of learning a better strategy, such as imitation learning and learning better thinking traces. The hypothesis should be more specific.
- The evaluation performed in this work seems to be in a setting that is significantly simpler than the original DeliveryBench work (for instance, the number of food types have been reduced from 22 to 4, and the types and business of traffic have also been reduced). The original DeliveryBench supported multi-agent interaction that the DelieveryGym dropped.
- DeliveryBench already reported results for RL evaluation, so the added contribution of making it into an RL environment is less significant than claimed.
- Contrary to the claim of long-term planning evaluation, the evaluation is capped at 60 turns.
- The increase in income or improvements in traffic compliance / obstacle avoidance are very monotonic improvements, rather than step change improvements that you might expect from improvements in long-term planning strategies (because these are difficult to discover and will result in boost in performance). Hence, while the motivation for the benchmark is for long-term planning, it seems that the improvement via RL is more due to optimizing for short-term goals (e.g. traffic compliance and obstacle avoidance).
- The RL evaluation with Qwen3-VL-4B is only performed on waypoint navigation, not on any-point navigation, which is the more interesting and challenging setting.
- There are some paragraphs which have strong characteristics of being generated by AI, e.g. LL: 039-046, especially LL: 045-046. They tend to sound impressive but do not carry meaningful information and are more confusing for the reader than helpful, or are verbose and repetitive. Overuse of colons and semi-colons.
Questions for authors
- DeliveryBench already provides a set of RL experiments where the RL rewards are given as the “trajectory-level net profit increase”. What exactly does DeliveryGym contribute over DeliveryBench which is more than a convenient wrapper around DeliveryBench?
- How does a greedy rule-based policy (as in DeliveryBench) perform in DeliveryGym, and how does the trained policy compare?
- Can you showcase the “long-term planning” capabilities delivered by RL via metrics that a greedy / short-horizon strategy would not be able to maximize?
- Does the RL policy parallelize orders (e.g., a time-efficiency metric as in DeliveryBench), or has it learned the greedy single-order routine DeliveryBench observed?
- Why were car/bus transport, multi-agent play, special customer requests, and the human baseline removed, and do the findings depend on those removals?
- L079: “Verifiable trajectory rewards” - Is this reward given as a single scalar per trajectory? Why this design choice over a reward for every successful delivery/failed action?
- L125: It is unclear whether the invalid actions have some kind of penalty, and if they also update the resources and safety (are they considered executed actions? If so “All valid/invalid actions also update…” will be less ambiguous)
- L156: Unclear what a “persistent notebook” means (or what it contains).
6 comments
Response to Reviewer Javn (Part 2)
W3: Is DeliveryGym simpler than DeliveryBench?
Thank you for raising this. We'd like to clarify that DeliveryGym is harder, not simpler.
Food types. DeliveryBench's 22 foods differ only in a few handling attributes: temperature class, motion sensitivity, and time sensitivity. DeliveryGym keeps each of these handling classes (Table 4), so the decisions an agent must make are the same. To check this, we re-ran the benchmark with all 22 food types (130 shifts, Waypoint, income in USD):
| Model | 4 food types (paper, Table 1) | 22 food types (new) |
|---|---|---|
| GPT-5.6 Sol | 18.3 | 18.1 |
| Claude Fable 5 | 18.1 | 17.8 |
| Qwen3-VL-4B | 7.6 | 7.5 |
These results show that adding food types barely changes income (≤0.3), so it does not make the task harder.
Traffic. We'd like to clarify that DeliveryBench does not simulate traffic: its paper describes no traffic lights, congestion, or obstacles, and its only continuously moving vehicle is one bus on a fixed loop. DeliveryGym adds traffic lights and obstacles, so traffic is added, not reduced.
What makes DeliveryGym harder. Traffic lights and obstacles with safety penalties, step-by-step movement from the camera view (at most 80 m per turn), a pixel-level Any-Point mode, and real-city maps. Even the strongest model earns 24% less than the oracle (L277), and Any-Point lowers income by 50–90% (L303–304). We explain why multi-agent play is left out in Q5.
W4: DeliveryBench already reported RL results
Thank you for checking this. We'd like to clarify that DeliveryBench does not report any RL training. Its only learning experiments are context engineering (ACE, Dynamic Cheatsheet) and SFT of LLaVA-OneVision-8B on 9 human trajectories (Sec. 5.5). RL appears only as future work in its Appendix A: "Although DeliveryBench currently serves primarily as an evaluation benchmark, ... Such data can be used to study how different learning paradigms, including reinforcement learning, ... adapt to our long-horizon delivery task." Net profit is its evaluation metric, not a training reward, and the phrase "trajectory-level net profit increase" does not appear in the paper. If we have missed an experiment, we would be grateful for a pointer.
W5: The 60-turn cap
Thank you for this comment. We'd like to note that 60 turns is already long for agent RL. Common agent-RL environments use similar or shorter episodes, e.g., WebShop (15 steps), OSWorld (15 steps), and ALFWorld (50 steps). Each of our turns is also a macro action of up to 80 m, and within 60 turns the agent must already plan its resources: the starting battery and stamina cover at most 3.125 km, so recharging or resting must be planned ahead (App. D).
We chose 60 turns because of compute. For the rebuttal, we borrowed additional GPUs and trained with 100 turns under the same protocol (80 evaluation shifts, step 100, 3 runs, income in USD):
| Turn cap | Initialization | RL (+ traffic + obstacle) | Gain |
|---|---|---|---|
| 60 (paper) | 8.1 | 12.0 | +48% |
| 100 (new) | 12.6 | 19.4 | +54% |
These results show that RL works equally well with longer shifts, with an even larger gain. We will add this result to Section 3.3.
Response to Reviewer Javn (Part 3)
W6: Does RL learn long-term planning or only short-term goals?
Thank you for this question. We'd like to clarify that the gains come from long-term planning, not only from safety:
- Smooth curves are expected. Each point in Figures 4–5 averages 3 runs over 80 shifts, every 20 updates, with a small learning rate (5×10⁻⁷). This smooths out any jumps within one run.
- Earnings-only RL, with no safety reward, also raises income by 30% (Sec. 3.3).
- Planning across orders matters. The same RL budget on isolated orders reaches 9.1 USD, versus 12.0 on full shifts (App. D). From identical states with execution fixed, the trained policy's regret falls from 3.5 to 2.6 USD (App. D). It also plans resources ahead: replenishment rises from 18.5% to 40.0% of shifts and empty batteries fall from 27.7% to 9.2% (Table 8).
We give a greedy baseline, a concrete example, and parallel-delivery metrics in our answers to Q2–Q4 below.
W7: RL with Any-Point control
Thank you for this suggestion. When we submitted, Any-Point was too hard for current models (Table 1), and it needs live rendering, which cannot use our cached waypoint images and is much slower (Table 11), so we trained with Waypoint. For the rebuttal, we ran a preliminary Any-Point RL training of Qwen3-VL-4B using the additional GPUs (80 evaluation shifts):
| Any-Point | Income (USD) ↑ | Delivered ↑ | Red light ↓ | Obstacle ↓ |
|---|---|---|---|---|
| Initialization | 0.9 | 0.1 | 4.1 | 6.3 |
| Any-Point RL | 3.6 | 0.6 | 2.6 | 4.2 |
These results show that RL also improves low-level Any-Point control, and DeliveryGym supports training at this level. We will add the full training run to Section 3.3.
W8: Writing style
Thank you for pointing this out. As disclosed in our AI Use Statement (L488–492), we used AI tools only to polish the wording, which the ICLR 2027 AI Policy permits; all ideas, experiments, and results are our own. We will rewrite L039–046 to be shorter and more concrete, e.g.: "In long-horizon embodied tasks, one action changes what the agent can do next: a detour uses battery, and accepting one order can make another one late. To learn this, an agent needs a simulator that carries these consequences forward."
Questions
Q1. What does DeliveryGym contribute beyond a wrapper? We'd like to clarify that DeliveryBench has no RL experiments, and the phrase "trajectory-level net profit increase" does not appear in its paper (see W4). DeliveryGym is more than a wrapper: it changes the task (traffic lights and obstacles, step-by-step movement, real-city maps, train/test splits), adds the reward, rollout, and curriculum infrastructure that RL needs (W1), and shows how these environment designs change RL gains (W2).
Q2. Greedy rule-based policy. Thank you for this suggestion. We'd like to note that, as far as we can tell, DeliveryBench does not include a greedy or rule-based policy; its baselines are VLM agents and three human players (its Sec. 5.1). We agree this is a useful reference, so we implemented one (it accepts the highest pay-per-distance order, delivers in arrival order, and moves along scripted shortest paths). We also report DeliveryBench's TimeEff metric (>1 means orders are handled in parallel), on the 60 unseen-city test shifts:
| Policy | Income (USD) | TimeEff | Delivered while carrying another order (%) |
|---|---|---|---|
| Greedy (rule-based) | 9.4 | 0.82 | 6.3 |
| Qwen3-VL-4B (initialization) | 7.3 | 0.71 | 4.8 |
| Safety RL | 11.1 | 1.24 | 27.5 |
| Oracle | 22.8 | 1.86 | 48.2 |
These results show that the trained policy earns 18% more than the greedy policy.
Response to Reviewer Javn (Part 1)
Thank you for the detailed review, and for recognizing the complex, well-designed tasks and the value of an RL-friendly setting with useful baselines.
W1: What is reused from DeliveryBench, and what is new?
Thank you for this question. We'd like to note that the paper states this relationship in three places:
We will make it explicit in Section 2 as follows.
Reused. We reuse DeliveryBench's courier task and its rules for orders, food, and resources (e.g., speeds, stamina and battery drain, recharge price, hospital rescue; Table 4).
Task: more realistic and harder. We changed the task so that it is closer to real courier work:
Infrastructure: making RL possible. DeliveryBench is built for evaluation, and its released gym-style wrapper is a placeholder (the reward is always 0 and episodes never end). We add the three pieces RL needs:
In short, DeliveryGym keeps DeliveryBench's task rules, makes the task more realistic and harder, and adds the reward, rollout, and curriculum infrastructure needed to train agents with RL.
W2: The hypothesis seems evident
Thank you for this comment. We'd like to clarify that our question is not whether RL can improve a policy given enough epochs, but how an RL environment should be built so that RL training is effective and scalable. Each design choice in DeliveryGym is compared against a simpler alternative under the same training budget (evaluation income in USD; Table 5, App. D):
These results show that how the environment is built changes how much RL improves the agent: each design choice adds 1.4–3.9 USD, and the gains keep growing as we add more tasks and maps. This also answers the reviewer's point on non-RL alternatives: imitation of a stronger teacher learns less than RL in the same environment. We will rewrite L077–079 to state these specific claims.