- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies why world-model controllers like TD-MPC2 and DreamerV3 struggle with network delays, packet loss, and out-of-order messages when controlling real robots.
The key insight is that each model fails differently. TD-MPC2 plans a sequence of actions, so delays can make its plan inconsistent with what the robot actually executes. DreamerV3 relies on action-observation pairs to update its internal state, so attributing an observation to the wrong action can throw off its state tracking.
The proposed fixes are simple and don't require retraining. For TD-MPC2, send the remaining action sequence to the actuator. For DreamerV3, use feedback on the action actually executed to update the model's state.
Experiments across multiple benchmarks, real network traces, and a two-process UDP setup show that both fixes improve robustness to network issues. However, one interesting finding is that these fixes can sometimes hurt models already trained to handle fixed delays, suggesting a mismatch with what they learned during training.
Critical strengths
-
Rather than treating network issues as generic delays, the paper identifies why TD-MPC2 and DreamerV3 fail for different reasons. TD-MPC2 relies on planned action sequences that may not match actual execution, while DreamerV3 can lose track of state when the executed action differs from the one used for its update. The authors isolate these failure modes through controlled experiments, making the diagnosis a strong contribution beyond simply proposing a robustness technique.
-
The proposed fixes directly address each model's failure mode: TD-MPC2 sends the remaining action sequence to the actuator, while DreamerV3 uses feedback on the action actually executed. Both changes are lightweight and don't require retraining the model. Validation across benchmarks and a real two-process UDP setup also strengthens the practical relevance of the approach.
Critical weaknesses
-
Table 2 shows that Future-Sequence improves performance for clean-trained models but can hurt models trained with fixed delays, with little benefit for random-delay training. This limits its practical value, especially when delay-aware training is already used. The paper acknowledges this issue but offers no guidance on when the fix helps or hurts, weakening the claim that it is a general solution.
-
In Table 1, DATS and ACDA perform poorly even under clean execution, scoring 0.266 and 0.407 compared with the normalized baseline of 1.0. This raises questions about implementation quality, tuning, or evaluation fairness. The paper does not adequately explain this gap, making its claims of superiority over existing delay-aware methods less convincing.
Questions for authors
-
Which checkpoints (command-trained or applied-action-trained) are used for each arm in Figure 6? Figure 4 shows benefits only for applied-action-trained models, raising questions about the “no retraining” claim. How do you isolate the interface effect from the training effect?
-
Section 4 discusses results across TD-MPC, MetaWorld, ManiSkill2, MyoSuite, measured network traces, task success, and overhead, but provides no quantitative results. Where are these results reported, and which network traces were used?
-
Table 4 shows Future-Sequence underperforming Single-Action at Fixed-2 and Fixed-3, yet Table 1 excludes fixed delays and Figure 7 tests only a one-period delay. Why were these conditions omitted, and how should the claim of robustness across adverse conditions be interpreted?
-
Future-Sequence performs similarly to Receding Buffer in three of four Table 1 settings, while the packet-loss difference may be explained by the fallback policy (zero versus hold-last). Can you ablate the fallback policy to establish whether the gains come from future-sequence execution itself?
-
Why does Oracle score lower than Future-Sequence under reordering (0.892 vs. 0.942), despite Finding 1? Can you also test the hypothesis that Fixed-2-trained models fail under adverse conditions because they effectively compensate for delay twice?
8 comments
Response to W2
Thank you for raising this important concern about baseline fairness. To address the relatively low performance of DATS and ACDA in the original Table 1, we carefully revisited their training and evaluation settings and conducted extensive additional experiments.
We found that both methods can achieve strong control performance on certain tasks. After further training and evaluation, DATS achieves a clean normalized return of 0.982 on Walker, while ACDA achieves 0.981 and 0.988 on Walker and Cheetah, respectively. In contrast, Hopper and Dog remain challenging, contributing to lower aggregate performance. We therefore conduct task-matched comparisons under identical network conditions on tasks where both methods achieve strong baseline control, while further investigating the difficulties encountered on the weaker tasks.
(1) ACDA comparison. On Walker and Cheetah, we use matched initial states and the same 0–3-step network perturbations.
| Method / Task | Clean | Random | Bursty | Loss | Reorder | Adverse avg. |
|---|---|---|---|---|---|---|
| ACDA / Walker | 0.981 | 0.894 | 0.834 | 0.841 | 0.876 | 0.861 |
| ACDA / Cheetah | 0.988 | 0.692 | 0.703 | 0.681 | 0.711 | 0.697 |
| ACDA / Avg. | 0.985 | 0.793 | 0.768 | 0.761 | 0.793 | 0.779 |
| Future-Sequence / Walker | 0.996 | 0.881 | 0.623 | 0.683 | 0.960 | 0.787 |
| Future-Sequence / Cheetah | 1.086 | 0.832 | 0.818 | 0.639 | 1.036 | 0.831 |
| Future-Sequence / Avg. | 1.041 | 0.856 | 0.721 | 0.661 | 0.998 | 0.809 |
Both controllers perform well without perturbations. Under this evaluation setup, Future-Sequence has a slightly higher adverse average (0.809 vs. 0.779), with a larger advantage on Cheetah. ACDA is stronger in some individual conditions, so we do not claim uniform superiority.
(2) DATS comparison. On Walker, both methods have similarly strong clean performance and are tested under the same network conditions.
| Method / Task | Clean | Random | Bursty | Loss | Reorder | Adverse avg. |
|---|---|---|---|---|---|---|
| DATS / Walker | 0.982 | 0.260 | 0.289 | 0.269 | 0.287 | 0.276 |
| Future-Sequence / Walker | 0.996 | 0.881 | 0.623 | 0.683 | 0.960 | 0.787 |
Despite comparable clean performance (0.982 vs. 0.996), Future-Sequence achieves higher returns under all four perturbations (0.787 vs. 0.276 on average). Thus, its gains in this comparison cannot be explained simply by DATS failing to control Walker under clean execution.
(3) Additional training and failure analysis. We completed 21 additional training or continuation runs (12 DATS, 9 ACDA), totaling 11.4 million environment interaction steps. We examined planning horizons, candidate counts, batch sizes, action-history handling, and target-entropy settings, together with prediction and behavior diagnostics.
For Hopper, effective training signals are scarce: the agent must reach a standing-height threshold before receiving meaningful forward-motion rewards. Consequently, 83–90% of ACDA's rewards and about 99.8% of DATS's rewards are zero. This makes it harder for ACDA to learn useful value estimates and for DATS to rank action sequences using predicted rewards. We tested three ACDA configurations with three seeds each and one million steps per seed, without obtaining consistently strong control.
For Dog, high-dimensional coordination (223 observation dimensions; 38 action dimensions) makes accurate prediction difficult. ACDA's two-step pelvis-height MAE is 6.29 cm, versus 2.15 cm for a constant-height predictor. DATS's reward-prediction RMSE is 0.084, exceeding the mean observed reward of 0.043. Such errors undermine the reliability of planning and action selection.
For Cheetah, we evaluated 17 DATS training/planning candidates, including 400,000 steps of continuation training. The best validation score reached 0.742, but the independent clean-test score remained 0.590.
These diagnostics explain why some external baselines have low aggregate scores despite strong clean control on other tasks. The revised Table 1 therefore reports matched-task comparisons rather than relying only on aggregate figures. Future-Sequence adapts existing pretrained world models through a lightweight execution interface, without extra delay-aware training, reducing the cost of asynchronous deployment. Our central contribution rests on controlled experiments with frozen world models and the diagnosis of action-semantic mismatch, not on weak baseline scores. We thank the reviewer for prompting these clarifications.
Response to Q1 and Q2
Response to Q1: Checkpoint Usage and the No-Retraining Claim
Thank you for pointing out this ambiguity. For each task in Figure 6, CMD, RCP, and APP all use the same command-trained checkpoint, which was trained under zero-delay execution. Under these conditions, commanded and applied actions coincide. All model parameters remain frozen during evaluation.
Importantly, APP denotes the feedback interface used during evaluation, not a switch to an applied-action-trained checkpoint. When comparing CMD and APP, we change only the previous-action input to the recurrent state update, replacing the issued command with the action actually applied. This isolates the effect of the execution interface from the effect of training.
Figure 4 serves a different purpose. It uses separate checkpoints trained under random delays, with either commanded or applied actions as training inputs, to examine how training-time action semantics affect feedback correction. Thus, the improvements in Figure 6 require neither additional training nor a change of checkpoint.
We have clarified the training conditions, action inputs, and checkpoint usage in the figure captions and experimental details of the revised manuscript.
Response to Q2: Quantitative Results and Network Traces
Thank you for pointing out this omission. We have added quantitative results to Table 3 in Section 4.2 of the revised manuscript and documented the network-trace sources, replay settings, and overhead measurements in Appendix E.3.
For clarity, the table below reports raw episodic returns and task success rates. Table 3 in the manuscript instead normalizes performance by the clean Single-Action performance for the same task and checkpoint.
| Evaluation and metric | Single-Action | Future-Sequence |
|---|---|---|
| Original TD-MPC: Walker return | 474.4 | 910.3 |
| Original TD-MPC: Hopper return | 17.1 | 210.3 |
| Original TD-MPC: Cup return | 800.1 | 958.3 |
| MetaWorld success rate | 86.7% | 100.0% |
| ManiSkill2 success rate | 50.0% | 93.3% |
| MyoSuite success rate | 90.0% | 100.0% |
| Measured-latency replay: Dog return | 318.2 | 488.9 |
| Measured-latency replay: MetaWorld Pick-place success rate | 60.0% | 95.0% |
The original TD-MPC and cross-domain results are averaged over three conditions: random delay, packet loss, and packet reordering, with five evaluation seeds per condition. The measured-latency replay results are averaged over four traces, with five evaluation seeds per trace. Across the three cross-domain benchmarks, the mean success rate improves by 22.2 percentage points.
The four measured network traces comprise office and library recordings from ACDA's public evaluation artifact, together with rural n8 and urban n78 road recordings from CICV5G. We replay these recorded latencies in simulation. The source links, original filenames, and replay settings are provided in Appendix E.3.
We also report the following deployment overhead measurements:
| Overhead measurement | Result |
|---|---|
| Additional runtime per step (NVIDIA L40S) | 0.284 ms |
| Additional communication per observation: ECT-Full | 261.8 bytes |
| Additional communication per observation: ECT-ID | 181.4 bytes |
The runtime overhead was measured on an exclusively used NVIDIA L40S. Communication overhead represents the additional serialized JSON bytes introduced by execution-receipt fields and excludes UDP/IP headers. Detailed measurement procedures are provided in Appendix E.3.
These results provide quantitative evidence of the effectiveness of Future-Sequence across the evaluated controllers, task domains, and measured network-delay conditions. The improvements on MetaWorld, ManiSkill2, and MyoSuite are reflected in task success rates. We appreciate the reviewer's suggestion, which helped us improve the clarity and completeness of the experimental reporting.
Response to W1
Thank you for highlighting the applicability of Future-Sequence. Our method primarily targets existing world models pretrained under delay-free conditions. Unlike approaches that require delay-aware training, Future-Sequence improves robustness to asynchronous execution through a lightweight interface modification, without retraining or updating model parameters, thereby avoiding substantial additional training costs.
Models already trained under delayed execution have learned to accommodate their training-time action delays and may not require the same correction. Our goal is therefore not to further improve models that have already adapted to delay, but to enable efficient, low-cost asynchronous deployment of existing pretrained world models.