acceptodds
Under review as a conference paper at ICLR 2027

DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive Curriculum

Abstract

Executable environments enable LLM agents to learn from the consequences of their actions. For embodied agents, those consequences extend beyond whether the current task succeeds: completing a delivery can consume the time, energy, or money needed for later work. Learning to plan therefore requires environments that preserve these dependencies and turn them into feedback across a complete trajectory. We introduce DeliveryGym, a 3D environment for evaluating and training agents on continuous courier shifts. It couples multimodal tool interaction with persistent world dynamics and computes trajectory rewards from simulator events, making the costs of an agent’s decisions available for reinforcement learning (RL). The environment also adapts future training shifts to the policy’s observed weaknesses while keeping evaluation fixed. Across six models and 13 city maps, evaluation exposes a gap between reliably executing assigned deliveries and choosing and sequencing work over a shift. On the unseen-city test set, RL improves Qwen3-VL-4B’s net income by 54.3%, showing that learning from complete shifts improves performance under these coupled constraints. Adapting the training environment improves evaluation income by 18% over uniform sampling at 100 updates, indicating that which situations an agent practices also matters. DeliveryGym provides an executable setting for studying how agents learn to coordinate deliveries and preserve resources for later orders within an episode.

Primary area:
reinforcement learning
Keywords:
RL Environment, Embodied Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JavnOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors designed DeliveryGym to train and evaluate agents on continuous courier shifts, in which there are long-term dependencies such as time, energy and money that needs to be considered beyond the immediate task. It builds on top of DeliveryBench, and adds features such as the state transitions and reward feedback to convert it into an RL environment for long-term planning. The RL framework optimizes the agents’ policy based on their overall profit (pay - expenses). The safety variant also considers red-light violations and obstacle collisions. The authors performed evaluation of six foundation models, along with a GRPO-trained Qwen3-VL-4B model.

Critical strengths
  • The environment is complex and the tasks well-designed, with city navigation, traffic avoidance, resource / cost management, and potential interaction with other actors in the scene.
  • This paper demonstrates an insightful use-case of the DeliveryBench environment for complex long-term planning tasks, and provides useful baselines / benchmarks in an RL-friendly setting.
Critical weaknesses
  • The paper describes the DeliveryGym environment that the authors created but the paper at a high-level, but does not go into technical depth in terms of their development and a lot of heavy-lifting seems to be done by the DeliveryBench backbone. The paper should state clearly which elements of the environment and evaluation were already present in the DeliveryBench work and what has been added / modified.
  • The main hypothesis of the paper LL: 077-079 “long-horizon embodied planning and reasoning can be improved through RL when operationally structured tasks are coupled with executable world dynamics and verifiable trajectory rewards” is not a compelling hypothesis since it is almost evident that it is true (RL can improve it given enough training epochs). Also, there may be other non-RL (reward-free) ways of learning a better strategy, such as imitation learning and learning better thinking traces. The hypothesis should be more specific.
  • The evaluation performed in this work seems to be in a setting that is significantly simpler than the original DeliveryBench work (for instance, the number of food types have been reduced from 22 to 4, and the types and business of traffic have also been reduced). The original DeliveryBench supported multi-agent interaction that the DelieveryGym dropped.
  • DeliveryBench already reported results for RL evaluation, so the added contribution of making it into an RL environment is less significant than claimed.
  • Contrary to the claim of long-term planning evaluation, the evaluation is capped at 60 turns.
  • The increase in income or improvements in traffic compliance / obstacle avoidance are very monotonic improvements, rather than step change improvements that you might expect from improvements in long-term planning strategies (because these are difficult to discover and will result in boost in performance). Hence, while the motivation for the benchmark is for long-term planning, it seems that the improvement via RL is more due to optimizing for short-term goals (e.g. traffic compliance and obstacle avoidance).
  • The RL evaluation with Qwen3-VL-4B is only performed on waypoint navigation, not on any-point navigation, which is the more interesting and challenging setting.
  • There are some paragraphs which have strong characteristics of being generated by AI, e.g. LL: 039-046, especially LL: 045-046. They tend to sound impressive but do not carry meaningful information and are more confusing for the reader than helpful, or are verbose and repetitive. Overuse of colons and semi-colons.
Questions for authors
  • DeliveryBench already provides a set of RL experiments where the RL rewards are given as the “trajectory-level net profit increase”. What exactly does DeliveryGym contribute over DeliveryBench which is more than a convenient wrapper around DeliveryBench?
  • How does a greedy rule-based policy (as in DeliveryBench) perform in DeliveryGym, and how does the trained policy compare?
  • Can you showcase the “long-term planning” capabilities delivered by RL via metrics that a greedy / short-horizon strategy would not be able to maximize?
  • Does the RL policy parallelize orders (e.g., a time-efficiency metric as in DeliveryBench), or has it learned the greedy single-order routine DeliveryBench observed?
  • Why were car/bus transport, multi-agent play, special customer requests, and the human baseline removed, and do the findings depend on those removals?
  • L079: “Verifiable trajectory rewards” - Is this reward given as a single scalar per trajectory? Why this design choice over a reward for every successful delivery/failed action?
  • L125: It is unclear whether the invalid actions have some kind of penalty, and if they also update the resources and safety (are they considered executed actions? If so “All valid/invalid actions also update…” will be less ambiguous)
  • L156: Unclear what a “persistent notebook” means (or what it contains).
6 comments
AuthorsOpenReview ↗

Response to Reviewer Javn (Part 1)

Thank you for the detailed review, and for recognizing the complex, well-designed tasks and the value of an RL-friendly setting with useful baselines.

W1: What is reused from DeliveryBench, and what is new?

Thank you for this question. We'd like to note that the paper states this relationship in three places:

  • "Building on the delivery task and city simulation of DeliveryBench (Mao et al., 2025), DeliveryGym shifts the focus from evaluating agents to improving their policies through online RL." (L081–083)
  • "DeliveryGym adds online RL training and adaptive task configuration to this delivery setting." (L132–133)
  • "DeliveryBench already evaluates continuous delivery in such a city (Mao et al., 2025), but none offers an LM/VLM-oriented RL training interface." (L444–446)

We will make it explicit in Section 2 as follows.

Reused. We reuse DeliveryBench's courier task and its rules for orders, food, and resources (e.g., speeds, stamina and battery drain, recharge price, hospital rescue; Table 4).

Task: more realistic and harder. We changed the task so that it is closer to real courier work:

  • Maps. DeliveryBench's 10 procedurally generated cities become 13 fixed maps, including the real street layouts of Paris and New York (L131).
  • Safety. We add traffic lights and obstacles, and penalize red-light violations and collisions (L145, Eq. 2), because a real courier must obey traffic.
  • Movement. In DeliveryBench, one action takes the agent to any given coordinate, however far away. In DeliveryGym, the agent walks step by step: each turn it picks its next target from the camera view and moves at most 80 m (L159–161, Table 4). This is closer to real deployment, where the agent must keep reading the street along the way.
  • Evaluation. DeliveryBench is only a test benchmark, with no training and test split. DeliveryGym fixes 130 replayable shifts, trains on disjoint seeds, and holds out three unseen cities (L192–195), so that we can test whether training generalizes.

Infrastructure: making RL possible. DeliveryBench is built for evaluation, and its released gym-style wrapper is a placeholder (the reward is always 0 and episodes never end). We add the three pieces RL needs:

  • Reward. A verifiable reward for the whole shift, computed from simulator events (Eqs. 1–2, L208–215).
  • Rollouts. A parallel, asynchronous training stack that raises throughput from 2.1 to 18.4 shifts/h (L216–231).
  • Curriculum. Adaptive task generation with a feasibility solver (L233–249, App. C).

In short, DeliveryGym keeps DeliveryBench's task rules, makes the task more realistic and harder, and adds the reward, rollout, and curriculum infrastructure needed to train agents with RL.

W2: The hypothesis seems evident

Thank you for this comment. We'd like to clarify that our question is not whether RL can improve a policy given enough epochs, but how an RL environment should be built so that RL training is effective and scalable. Each design choice in DeliveryGym is compared against a simpler alternative under the same training budget (evaluation income in USD; Table 5, App. D):

Environment designSimpler alternativeOurs
Keep consequences across orders (full shifts)Reset after each order: 9.112.0
Reward includes safety eventsEarnings only: 10.512.0
Learn from rewards in the environmentImitate a stronger teacher (SFT): 10.612.0
Adapt tasks to the policy's weaknessesUniform sampling: 12.214.4
Scale training maps1 map: 9.48 maps: 13.3

These results show that how the environment is built changes how much RL improves the agent: each design choice adds 1.4–3.9 USD, and the gains keep growing as we add more tasks and maps. This also answers the reviewer's point on non-RL alternatives: imitation of a stronger teacher learns less than RL in the same environment. We will rewrite L077–079 to state these specific claims.

AuthorsOpenReview ↗

Response to Reviewer Javn (Part 2)

W3: Is DeliveryGym simpler than DeliveryBench?

Thank you for raising this. We'd like to clarify that DeliveryGym is harder, not simpler.

Food types. DeliveryBench's 22 foods differ only in a few handling attributes: temperature class, motion sensitivity, and time sensitivity. DeliveryGym keeps each of these handling classes (Table 4), so the decisions an agent must make are the same. To check this, we re-ran the benchmark with all 22 food types (130 shifts, Waypoint, income in USD):

Model4 food types (paper, Table 1)22 food types (new)
GPT-5.6 Sol18.318.1
Claude Fable 518.117.8
Qwen3-VL-4B7.67.5

These results show that adding food types barely changes income (≤0.3), so it does not make the task harder.

Traffic. We'd like to clarify that DeliveryBench does not simulate traffic: its paper describes no traffic lights, congestion, or obstacles, and its only continuously moving vehicle is one bus on a fixed loop. DeliveryGym adds traffic lights and obstacles, so traffic is added, not reduced.

What makes DeliveryGym harder. Traffic lights and obstacles with safety penalties, step-by-step movement from the camera view (at most 80 m per turn), a pixel-level Any-Point mode, and real-city maps. Even the strongest model earns 24% less than the oracle (L277), and Any-Point lowers income by 50–90% (L303–304). We explain why multi-agent play is left out in Q5.

W4: DeliveryBench already reported RL results

Thank you for checking this. We'd like to clarify that DeliveryBench does not report any RL training. Its only learning experiments are context engineering (ACE, Dynamic Cheatsheet) and SFT of LLaVA-OneVision-8B on 9 human trajectories (Sec. 5.5). RL appears only as future work in its Appendix A: "Although DeliveryBench currently serves primarily as an evaluation benchmark, ... Such data can be used to study how different learning paradigms, including reinforcement learning, ... adapt to our long-horizon delivery task." Net profit is its evaluation metric, not a training reward, and the phrase "trajectory-level net profit increase" does not appear in the paper. If we have missed an experiment, we would be grateful for a pointer.

W5: The 60-turn cap

Thank you for this comment. We'd like to note that 60 turns is already long for agent RL. Common agent-RL environments use similar or shorter episodes, e.g., WebShop (15 steps), OSWorld (15 steps), and ALFWorld (50 steps). Each of our turns is also a macro action of up to 80 m, and within 60 turns the agent must already plan its resources: the starting battery and stamina cover at most 3.125 km, so recharging or resting must be planned ahead (App. D).

We chose 60 turns because of compute. For the rebuttal, we borrowed additional GPUs and trained with 100 turns under the same protocol (80 evaluation shifts, step 100, 3 runs, income in USD):

Turn capInitializationRL (+ traffic + obstacle)Gain
60 (paper)8.112.0+48%
100 (new)12.619.4+54%

These results show that RL works equally well with longer shifts, with an even larger gain. We will add this result to Section 3.3.

AuthorsOpenReview ↗

Response to Reviewer Javn (Part 3)

W6: Does RL learn long-term planning or only short-term goals?

Thank you for this question. We'd like to clarify that the gains come from long-term planning, not only from safety:

  • Smooth curves are expected. Each point in Figures 4–5 averages 3 runs over 80 shifts, every 20 updates, with a small learning rate (5×10⁻⁷). This smooths out any jumps within one run.
  • Earnings-only RL, with no safety reward, also raises income by 30% (Sec. 3.3).
  • Planning across orders matters. The same RL budget on isolated orders reaches 9.1 USD, versus 12.0 on full shifts (App. D). From identical states with execution fixed, the trained policy's regret falls from 3.5 to 2.6 USD (App. D). It also plans resources ahead: replenishment rises from 18.5% to 40.0% of shifts and empty batteries fall from 27.7% to 9.2% (Table 8).

We give a greedy baseline, a concrete example, and parallel-delivery metrics in our answers to Q2–Q4 below.

W7: RL with Any-Point control

Thank you for this suggestion. When we submitted, Any-Point was too hard for current models (Table 1), and it needs live rendering, which cannot use our cached waypoint images and is much slower (Table 11), so we trained with Waypoint. For the rebuttal, we ran a preliminary Any-Point RL training of Qwen3-VL-4B using the additional GPUs (80 evaluation shifts):

Any-PointIncome (USD) ↑Delivered ↑Red light ↓Obstacle ↓
Initialization0.90.14.16.3
Any-Point RL3.60.62.64.2

These results show that RL also improves low-level Any-Point control, and DeliveryGym supports training at this level. We will add the full training run to Section 3.3.

W8: Writing style

Thank you for pointing this out. As disclosed in our AI Use Statement (L488–492), we used AI tools only to polish the wording, which the ICLR 2027 AI Policy permits; all ideas, experiments, and results are our own. We will rewrite L039–046 to be shorter and more concrete, e.g.: "In long-horizon embodied tasks, one action changes what the agent can do next: a detour uses battery, and accepting one order can make another one late. To learn this, an agent needs a simulator that carries these consequences forward."

Questions

Q1. What does DeliveryGym contribute beyond a wrapper? We'd like to clarify that DeliveryBench has no RL experiments, and the phrase "trajectory-level net profit increase" does not appear in its paper (see W4). DeliveryGym is more than a wrapper: it changes the task (traffic lights and obstacles, step-by-step movement, real-city maps, train/test splits), adds the reward, rollout, and curriculum infrastructure that RL needs (W1), and shows how these environment designs change RL gains (W2).

Q2. Greedy rule-based policy. Thank you for this suggestion. We'd like to note that, as far as we can tell, DeliveryBench does not include a greedy or rule-based policy; its baselines are VLM agents and three human players (its Sec. 5.1). We agree this is a useful reference, so we implemented one (it accepts the highest pay-per-distance order, delivers in arrival order, and moves along scripted shortest paths). We also report DeliveryBench's TimeEff metric (>1 means orders are handled in parallel), on the 60 unseen-city test shifts:

PolicyIncome (USD)TimeEffDelivered while carrying another order (%)
Greedy (rule-based)9.40.826.3
Qwen3-VL-4B (initialization)7.30.714.8
Safety RL11.11.2427.5
Oracle22.81.8648.2

These results show that the trained policy earns 18% more than the greedy policy.

Reviewer_uLJkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DeliveryGym, which is a follow-up work of DeliveryBench. The focus is more on an RL training infra rather than benchmarking, and on validating the hypothesis that " long-horizon embodied planning and reasoning can be improved through RL when operationally structured tasks are coupled with executable world dynamics and verifiable trajectory rewards". It provides nice support, such as reward functions and adaptive curriculums. Improvements were shown for the planning agent by using the infra provided.

Critical strengths
  • Interesting and practically useful problem setting, I like this type of domain that is driven by the real need instead of made-up tasks.
  • Complete infra support, especially the harness, the adaptive generator, held out test sets (may consider releasing only the val set and hide the test sets).
  • Studies are well controlled, which leads to convincing results, especially the benefits of adaptive curriculum as shown in Fig 5 and the appendix, although the evaluation seems to be on in-distribution set only?
Critical weaknesses
  • Figure 5(A): the performance is still improving; keep training and conducting an analysis of the performance ceiling (if <optimal oracle) would be helpful
  • Waypoint control seems to be too high-level, which is understandable in the current setting but should consider more low-level controls
  • Table 2 related work: some of the metrics are either vague/ungrounded or arbitrary, e.g., what is visual realism here? Is it quantifiable? In the "Multi-turn" definition, what is the notion of a single "task"? The list of relevant benchmarks is a bit outdated as well. Consider having a more comprehensive table and moving it to the appendix if needed.
  • It is useful to report more simulation speed / computational cost-related stats, such as FPS, profiling in the RL training process, etc, in addition to what was reported in appendix F
Questions for authors

see comments above

4 comments
AuthorsOpenReview ↗

Response to Reviewer uLJk (Part 1)

Thank you for the positive and constructive review, and for recognizing the real-need-driven task, the complete infrastructure, and our well-controlled studies.

W1: Does performance keep improving after 100 steps?

Thank you for this suggestion. We continued all four curriculum runs to 150 and 200 steps under the same protocol, and ran the search oracle on the same 80 evaluation shifts (net income in USD, mean ± SD over 3 runs):

SamplingStep 100 (Table 5)Step 150Step 200
Uniform12.2 ± 0.4612.7 ± 0.3812.8 ± 0.35
Random targeting12.6 ± 0.4413.0 ± 0.4112.9 ± 0.34
Static curriculum13.1 ± 0.3513.6 ± 0.3313.5 ± 0.31
Adaptive curriculum14.4 ± 0.4614.8 ± 0.3614.9 ± 0.32
Oracle (search, same shifts)24.424.424.4

These results show that training has converged: from step 150 to 200, every method changes by at most 0.1, well within run-to-run variation (SD 0.31–0.46), and the adaptive curriculum stays ahead of uniform sampling (+2.1 at step 200). The adaptive policy reaches 61% of oracle income. Much of the remaining gap is in order planning: an online planner that only chooses orders adds 2.7 USD to Qwen3-VL-4B (Table 9). We will add these curves to Figure 5(A).

W2: Waypoint control is too high-level

Thank you for raising this; we considered the same issue. This is why DeliveryGym also provides Any-Point control, a lower-level interface where the agent picks a pixel in its camera view and moves to that point (L159–161). It is much harder: when we submitted, even the best model lost about half its income with Any-Point (Table 1), so we trained RL with Waypoint, where models could already make progress.

Newer models are now much stronger at Any-Point. We recently evaluated three of them under the same protocol (130 shifts, 60 turns, Any-Point):

Any-Point movementNet income (USD) ↑Delivered ↑On time (%) ↑Red light ↓Obstacle ↓
GPT-5.6 Sol (Table 1)9.21.758.21.73.3
Claude Fable 5 (Table 1)8.61.554.42.03.7
GPT-6 Astra (new)14.83.085.20.71.3
Claude Opus 5.5 (new)15.13.186.70.61.1
Claude Fable 5.2 (new)16.03.390.40.40.8
Oracle (search, Table 1)17.53.798.20.00.0

These results show that frontier models are closing the gap on low-level control: the best model now reaches 91% of oracle income with Any-Point, up from 53%. This makes Any-Point a practical next target for RL training in DeliveryGym. We will add these results to Table 1.

AuthorsOpenReview ↗

Response to Reviewer uLJk (Part 2)

W3: Table 2 criteria and coverage

Thank you for this careful reading. We'd like to clarify what each column means:

  • Visual realism. It rates how observations are rendered, from 1 to 3 plus signs: one for text, 2D, or stylized game graphics; two for 3D simulators with simpler rendering (e.g., AI2-THOR, Habitat); three for photorealistic engine rendering (e.g., Unreal Engine 5, NVIDIA Omniverse). We will add the renderer of each environment to the table so every rating can be checked.
  • What a "task" is, and Multi-turn. A task is one goal with its own success check, e.g., one ALFRED instruction or one delivery order. Multi-turn (✓) means one episode contains many tasks that share the same world state, so each choice changes which later tasks remain feasible; ✗ means the episode ends once its single goal is reached. This is the property DeliveryGym is built to study: a shift is one episode with many orders, and taking one order uses time and energy needed for the next (Section 2.1). We will state this definition in the caption.
  • RL interface. ✓ means the environment provides an interface for training LM/VLM policies with RL, demonstrated in its paper.

Below are recent environments released before July 17, 2026, the contemporaneous-work cutoff in the ICLR 2027 Reviewer Guide. We will add them to an expanded Table 2 in Appendix H and we are happy to add more related works if we missed any:

EnvironmentVisual realism (renderer)Multi-turnVerifiable rewardRL interfaceAdaptive curriculum
RAGEN (2025)+ (text gridworld)✗✓✓✗
VAGEN (2025)+ / ++ (2D grid; AI2-THOR, ManiSkill)✗✓✓✗
AgentGym-RL (2025)+ (text / 2D games)✗✓✓✗
FindingDory (2025)++ (Habitat)✗✓✓✗
Embodied Arena (2025)++ (3D game engine)✗✓✗✗
SimWorld-Robotics (2025)+++ (UE5)✗✓✗✗
DeliveryBench (2025)+++ (UE5)✓✓✗✗
EcoGym (2026)+ (text)✓✓✗✗
NativeEmbodied (2026)++ (AI2-THOR)✗✓✗✗
AsgardBench (2026)++ (AI2-THOR)✗✓✗✗
WorldLines (2026)++ (Habitat)✓✗✗✗
DeliveryGym+++ (UE5)✓✓✓✓

These comparisons show that recent environments offer at most two of photorealism, multi-task episodes, and RL training; DeliveryGym is the only one that combines all of them with an adaptive curriculum.

W4: Simulation speed and profiling

Thank you for this suggestion. We profiled one agent turn with the full training stack (Waypoint, 8 UE workers, 1 RTX 4090, 2 A100-80GB; same setup as Table 11):

Stage (per agent turn)Time (s)Share
View rendering (81% cache hits; a miss takes 3.1 s)0.62%
Simulator step (movement, collisions, events)4.216%
Observation transfer0.31%
Batched policy inference17.567%
Waiting in queue3.513%
Total26.1

UE renders the 1024-px view at about 30 FPS, and the simulator runs movement at about 10× real time. With 8 workers, this gives 18.4 shifts/h (Table 11), so one 2,400-trajectory run takes 5.4 days, or about 130 RTX 4090-hours and 260 A100-hours. These results show that after caching and asynchronous execution, the simulator is no longer the bottleneck: policy inference takes two-thirds of the time. We will add this table to Appendix F.

Other comments in the review

  • "Consider releasing only the val set and hiding the test sets." Thank you, we will do exactly this: we will release the evaluation set and keep the unseen-city test set behind an evaluation server.
  • "The evaluation seems to be on in-distribution set only?" We'd like to note that the paper also evaluates on three unseen cities that are never used in training (Appendix B). There, safety RL improves income by 52.1% (7.3 → 11.1 USD).

Dear Reviewer,

We hope our responses have addressed your concerns. We would appreciate it if you could take another look and let us know if you have any further questions.

Thank you!

Dear Reviewer,

As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal. If any questions or concerns remain, we would be happy to provide further clarification.

Best regards, The Authors

Reviewer_isJYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes DeliveryGym, a multimodal 3D environment for training an LLM agent on courier shifts. DeliveryGym allows an agent to interact with the environment through multimodal tool interactions and computes a reward for RL training at the end of the episode. DeliveryGym also supports an adaptive curriculum, where the environment adapts future training shifts to the weaker parts of the policy from skill failures to guide future task generation. The paper evaluates 6 models on DeliveryGym. DeliveryGym closes the loop from tool interaction, grounded consequence, verifiable reward, policy update, and targeted future episode generation, and the paper demonstrates this through Qwen3-VL-4B.

Critical strengths
  1. Strong delivery success alone does not ensure that an agent chooses and sequences work well across a shift for profit. The paper demonstrates that DeliveryGym enables RL training to improve this.
  2. The adaptive curriculum approach of using policy weakness diagnosis and targeted task generation has been shown to be more effective than uniform sampling at 100 updates.
Critical weaknesses
  1. While the paper acknowledges that DeliveryGym builds on DeliveryBench, it could benefit from explaining its contributions more explicitly. Are the maps and tasks adapted from DeliveryBench? Does DeliveryBench already provide interactive environments that make actions, state transitions, and task progress available? What is reused and what is not from DeliveryBench?

  2. The Policy Weakness Diagnosis seems to require human effort to design and can be domain-specific. It may not generalize easily to new tasks/domains.

Questions for authors
  1. As the paper primarily concerns RL, I think there could be a table similar to Table 1 to compare the performance of the base model, SFT, and different RL settings side by side.
  2. There is no supplementary material attached. Would the authors consider making DeliveryGym available in the future?
4 comments
AuthorsOpenReview ↗

Response to Reviewer isJY (Part 1)

Thank you for the thoughtful review, and for recognizing that delivery success alone does not ensure good shift-level planning, that DeliveryGym's RL training improves it, and that our adaptive curriculum beats uniform sampling.

W1: What is reused from DeliveryBench, and what is new?

Thank you for this question. We'd like to note that the paper states this relationship in two places:

  • "DeliveryGym adds online RL training and adaptive task configuration to this delivery setting." (L132–133)
  • "DeliveryBench already evaluates continuous delivery in such a city (Mao et al., 2025), but none offers an LM/VLM-oriented RL training interface." (L444–446)

We will make it explicit in Section 2 as follows.

Reused. Yes, most of the maps and tasks are adapted from DeliveryBench. We reuse its courier task and its rules for orders, food, and resources (e.g., speeds, stamina and battery drain, recharge price, hospital rescue; Table 4).

Task: more realistic and harder. We upgrade the task itself:

  • Maps. DeliveryBench's 10 procedurally generated cities become 13 fixed maps, including Paris and New York (L131).
  • Safety. We add traffic lights and obstacles. Red-light violations and collisions are tracked and penalized (L145, Eq. 2).
  • Movement. In DeliveryBench, one action takes the agent to any given coordinate, however far away, so a whole trip is a single step. In DeliveryGym, the agent walks there step by step: at each turn it picks its next target from the camera view, either a pixel (Any-Point) or a visible waypoint (L159–161), and moves at most 80 m (Table 4). It must keep reading the street, traffic lights, and obstacles along the way, which is much closer to how a real delivery agent is deployed.
  • Evaluation. DeliveryBench is only a test benchmark: it evaluates agents on its city maps, with no training data and so no separation between training and test. DeliveryGym fixes 130 replayable shifts (start state, orders, deadlines, hazards, and seed; L192–193), trains on disjoint seeds, and holds out three unseen cities to test whether training generalizes (L194–195).

These changes make the task clearly harder: even the strongest model earns 24% less than the oracle (L277), and moving by pixel lowers income by 50–90% (L303–304).

Infrastructure: making RL possible. DeliveryBench is interactive, so an agent can act, see the world change, and track its orders. But it is built for evaluation, not RL. In its official code, the gym-style wrapper is a placeholder: the reward is always 0, episodes never end, and only one environment/sandbox runs at a time, on a GUI thread. We add the three pieces RL needs:

  • Reward. A verifiable reward for the whole shift, computed from simulator events (Eqs. 1–2, L208–215).
  • Rollouts. A parallel, asynchronous training stack that raises throughput from 2.1 to 18.4 shifts/h (L216–231).
  • Curriculum. Adaptive task generation with a feasibility solver (L233–249, App. C).

In short, DeliveryGym keeps DeliveryBench's task rules, makes the task more realistic and harder, and adds the reward, rollout, and curriculum infrastructure needed to train agents with RL.

W2: Does weakness diagnosis require human design?

Thank you for this question. We'd like to clarify that only the skill families need to be designed, and they can be designed by an LLM instead of a human. Everything else is domain-agnostic: probe scoring, the smoothed deficit, the sampling weights, the 70/30 mixture, and the feasibility filter (App. C). Each family is just three binary checks over simulator events (Table 7).

Experiment (new). We give an LLM the simulator's event list and 50 failed rollouts of the initial policy, and ask it to propose three skill families, each with three binary checks and a task-parameter range. Everything else is unchanged (eval. income at step 100, mean ± SD over 3 runs):

Who designs the skill familiesNet income (USD)
None (uniform sampling, Table 5)12.2 ± 0.46
Qwen3-VL-4B (the policy itself)13.0 ± 0.41
Claude Opus 5.513.8 ± 0.38
Human (paper, Table 7)14.4 ± 0.46

These results show that human-designed families work best, but LLM-designed families also beat uniform sampling, even when the policy designs its own (+0.8), and a stronger designer closes most of the gap (+1.6). We will add this experiment to Section 3.4.

AuthorsOpenReview ↗

Response to Reviewer isJY (Part 2)

Q1: Base vs. SFT vs. RL side by side

Thank you for this helpful suggestion. Below is the same view as Table 1 for Qwen3-VL-4B on the 80 evaluation shifts (step-100 checkpoints, mean over 3 runs). Net income is from Table 5 and Appendix B; the other columns are computed from the same evaluation runs.

Qwen3-VL-4BNet income (USD) ↑Delivered ↑On time (%) ↑Red light ↓Obstacle ↓
Base (initialization)8.11.556.32.43.6
SFT (10k teacher trajectories)10.62.271.51.42.3
RL, earnings only10.52.370.21.82.8
RL, + traffic11.02.473.40.82.6
RL, + traffic + obstacle12.02.777.60.71.3
RL, + adaptive curriculum14.43.181.90.51.0

These results show that RL on full shifts beats SFT once the reward includes safety (12.0 vs. 10.6 USD), with fewer violations, and the adaptive curriculum adds the largest further gain (14.4 USD). We will add this table to Section 3.3.

Q2: Will DeliveryGym be released?

Thank you for asking. Yes, we will open-source DeliveryGym upon publication, including the maps, agent harness, task generator and feasibility filter, verifier, asynchronous RL stack, training configs, and evaluation suite.

Dear Reviewer,

We hope our responses have addressed your concerns. We would appreciate it if you could take another look and let us know if you have any further questions.

Thank you!

Dear Reviewer,

As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal. If any questions or concerns remain, we would be happy to provide further clarification.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers and the AC for their time and constructive feedback.

What DeliveryGym offers. Long-horizon embodied agents must plan over consequences that carry across many tasks, but realistic embodied environments are built for evaluation, not RL training. DeliveryGym brings together photorealistic city scenes, multi-task episodes, verifiable rewards, an RL training interface, and an adaptive curriculum in one environment (Table 2, App. H). With it:

  • RL improves income on unseen cities by 52.1%;
  • the adaptive curriculum adds 18% over uniform sampling;
  • each environment design choice measurably increases what RL learns under a fixed budget (point 2 below).

Reviewers recognized each part:

  • The problem matters: it is driven by a real need rather than made-up tasks (uLJk), and delivery success alone does not ensure good shift-level planning (isJY).
  • The environment is well built: the tasks are complex and well designed (Javn), with complete infrastructure (uLJk) and useful baselines in an RL-friendly setting (Javn).
  • The results are convincing: RL in DeliveryGym improves shift-level planning (isJY), the studies are well controlled (uLJk), and the adaptive curriculum beats uniform sampling (isJY, uLJk).

Main concerns and how we addressed them

1. Contribution beyond DeliveryBench (isJY, Javn). We reuse DeliveryBench's courier task and its order, food, and resource rules, and add what RL needs:

  • A harder, more realistic task: 13 fixed maps including Paris and New York, traffic lights and obstacles with safety penalties, step-by-step movement from the camera view (at most 80 m per turn), and three held-out cities. Using all 22 DeliveryBench food types changes income by at most 0.3 USD.
  • RL infrastructure: DeliveryBench reports no RL training. We add a verifiable shift-level reward, a parallel asynchronous rollout stack (2.1 → 18.4 shifts/h), and an adaptive curriculum.

(App. A.)

2. What the environment design contributes (Javn, isJY). We compare each design choice with a simpler alternative under the same budget:

Design choiceSimpler alternativeGain (USD)
Full shiftsReset after each order+2.9
Safety-aware rewardEarnings only+1.5
Adaptive curriculumUniform sampling+2.2
8 training maps1 map+3.9

We also add a side-by-side table of the base model, SFT, and RL variants (isJY). (App. D.)

3. Long-horizon planning (Javn).

  • Longer shifts: with 100-turn shifts, RL gains +54% (12.6 → 19.4 USD), vs. +48% at 60 turns.
  • Planning, not only safety: earnings-only RL already gains 30%; full-shift training beats isolated orders (12.0 vs. 9.1 USD); regret from identical states falls from 3.5 to 2.6 USD; planned replenishment rises from 18.5% to 40.0% of shifts.
  • Beyond greedy: the trained policy beats a greedy rule-based policy (11.1 vs. 9.4 USD) and handles orders in parallel (TimeEff 1.24 vs. 0.82).

(App. D.)

4. Low-level control (uLJk, Javn). DeliveryGym already provides pixel-level Any-Point control. Newer models now reach 91% of oracle income with it (up from 53% at submission), and a preliminary Any-Point RL run raises income from 0.9 to 3.6 USD. (Table 1; App. D.)

5. Convergence (uLJk). From step 150 to 200, every method changes by at most 0.1 USD; the adaptive curriculum stays 2.1 USD above uniform sampling and reaches 61% of oracle income. (App. C.)

6. Generality of the curriculum (isJY). Only the skill families need design, and an LLM can design them. LLM-designed families also beat uniform sampling (12.2 USD): 13.0 USD when the policy designs its own, 13.8 with Claude Opus 5.5, and 14.4 with human-designed families. (§3.4; App. C.)

7. Related work and systems (uLJk). Table 2 now defines each criterion and lists every renderer. A new table adds 11 recent environments; among the environments we compare, none combines photorealism, multi-task episodes, and RL training. A per-turn profile shows that policy inference, not the simulator, takes 67% of the time. (Table 2; App. F, H.)

8. Release (isJY, uLJk). We will open-source DeliveryGym upon publication, release the evaluation set, and keep the unseen-city test set behind an evaluation server.

Revised PDF

We have uploaded a revised PDF. All changes are coloured by reviewer: isJY in blue, uLJk in green, Javn in orange, and changes for several reviewers in purple. To stay within the page limit, new results are in Appendices A–D, F, and H, with pointers from the main text (§2.1, §3.3, §3.4, Table 1, Figure 5).

Discussion status

  • Reviewer isJY: no reply yet.
  • Reviewer uLJk: no reply yet.
  • Reviewer Javn: no reply yet.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.