acceptodds
Under review as a conference paper at ICLR 2027

HETEROGENEOUS ACTION CACHING: FREEING THE ACTION SOLVER OF WORLD ACTION MODELS

Abstract

World Action Models (WAMs) are markedly more robust under distribution shift than present-only policies, yet they are costly to run. The common practice of storing the future representation once lets a single video pass serve every action step, but it leaves the dominant cost untouched: once the future context exists, the action solver absorbs 84% of the runtime (165.27 of 195.29 ms on FASTER-WAM), against 25.60 ms for the visual branch and 6.04 ms for VAE encoding. The solver has been neglected as a caching target at the granularity that matters: latent-space and block-level caches act on spatially resolved features, while action-policy caches reuse whole chunks or demand an offline schedule or a learned gate. We present HAC (Heterogeneous Action Caching), which carries curvature-style heterogeneous token caching into action space. It treats each of the 32 chunk positions as a token, derives a scale-free curvature of its denoising path from the three most recent full evaluations, and forecasts each skipped step with a token-specific rule—copy for stable tokens, first-order extrapolation for linear tokens, and a smoothstep-damped blend for chaotic tokens—while a chaotic-only drift budget decides when a genuine evaluation is required. HAC trains nothing, searches nothing offline, and adds no learned parameters. It removes six of ten solver evaluations, speeding up the pipeline by 1.63× (1.86×on the solver) over FASTER-WAM at η=12: +0.09 points on RoboTwin 2.0 and +0.35 on LIBERO (seed 42), and −0.22 points on the matched LIBERO-Plus OOD set. The drift budget η acts as a single speed–quality knob, lowering latency monotonically from 8 to 12 while leaving in-distribution success within noise.

Primary area:
generative models
Keywords:
world action models, action denoising, training-free acceleration, curvature-aware forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3eywOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Heterogeneous Action Caching (HAC), a training-free framework for accelerating action generation in world action models. Using three recent full solver evaluations, HAC groups action-chunk positions by estimated trajectory curvature and predicts skipped outputs through copying, linear extrapolation, or a damped update. Accumulated drift on high-curvature positions triggers subsequent full evaluations, while the existing future-conditioning cache remains unchanged. Validation includes Faster-WAM and Fast-WAM on LIBERO and RoboTwin, a matched LIBERO-Plus subset, and paired GPU latency measurements. The authors report reducing full solver evaluations from ten to four while retaining similar in-distribution success rates; OOD performance is evaluated with a more conservative caching threshold.

Critical strengths
  1. HAC focuses on repeated action-solver evaluations after future conditioning has been cached. Its output-level wrapper requires no additional training or learned parameters and preserves the underlying checkpoint. This makes the approach appealing for reducing the deployment cost of existing world action models.
  2. The experiments cover Faster-WAM and Fast-WAM on LIBERO and RoboTwin, include distribution-shift evaluation on a matched LIBERO-Plus subset, and measure latency through paired runs on the same hardware. This combination addresses both computational savings and their consequences for task execution, providing relevant evidence of practical utility.
Critical weaknesses
  1. The curvature-based grouping, heterogeneous predictors, and drift-triggered refresh closely follow WorldCache. However, Table A2 lacks comparisons with uniform prediction, fixed refresh schedules, and reduced-step solvers under equal network-evaluation budgets. Consequently, the experiments do not establish how much benefit comes specifically from heterogeneous prediction and adaptive refreshing, weakening the methodological contribution beyond transferring an existing approach to action space.
  2. Table 4 implies approximately 1.82× overall and 2.23× solver speedups, inconsistent with the stated 1.63× and 1.86×. The reported LIBERO baseline of 98.65% is also incompatible with integer success counts over the stated 1,200 episodes under standard equal-weight aggregation. Furthermore, Appendix A.4 says the headline configuration drops the final FULL evaluation, conflicting with the default algorithm. These discrepancies require clarification of the actual configurations and raw counts before the claimed performance preservation can be reliably assessed.
Questions for authors
  1. What are the actual episode totals and integer success counts for each method in Tables 1–2? In particular, how is the LIBERO baseline of 98.65% obtained under the stated 1,200-episode protocol? Were the baseline and HAC evaluated with identical episode budgets and aggregation rules?
  2. Please specify the threshold and actual FULL/CACHE schedules corresponding to Tables 4 and A2. How do the reported latencies reconcile with the stated 1.63× overall and 1.86× solver speedups? Does default HAC retain the final FULL evaluation at η=12, as the algorithm specifies, or omit it, as Appendix A.4 states?
  3. Which algorithmic changes, beyond applying the framework to action outputs, distinguish HAC from WorldCache? Do any existing comparisons isolate the benefits of heterogeneous prediction and adaptive refreshing against simpler strategies with equal network-evaluation budgets? Please distinguish what the current experiments establish from design choices whose benefits remain untested.
Reviewer_xSFjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces HAC, which a method that reduces action solver evaluations in World Action Models without additional training. It applies WorldCache’s curvature rules to positions within an action chunk, predicts skipped outputs from cached history, and uses accumulated drift to decide when to recompute. The cached future context is left unchanged.

Critical strengths
  1. HAC reduces computation in the action solver, which is the main inference bottleneck, without retraining or changing the backbone.

  2. Tests on LIBERO, RoboTwin, and LIBERO-Plus assess whether caching preserves task success, including under distribution shifts.

Critical weaknesses
  1. HAC largely reuses WorldCache’s rules without showing an advantage over simpler action-caching baselines. Its four FULL evaluations may follow entirely from the mandatory opening and closing steps, leaving the adaptive trigger’s contribution unclear.

  2. The reported numbers are not internally consistent. Success rates conflict with the stated trial counts, speedups disagree with Table 4, and descriptions of the final FULL step contradict each other, making the speed–quality trade-off difficult to assess.

Questions for authors

At η=12, does HAC outperform a fixed schedule with FULL evaluations at steps 0, 1, 2, and 9, using the same token-wise predictors?

Reviewer_9kNaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Heterogeneous Action Caching (HAC) to reduce the inference latency of the action solver in World Action Models (WAMs). The method partitions the 32 action chunk positions into three dynamic sets (stable, linear, and chaotic) based on a scale-free finite-difference curvature derived from recent full solver evaluations. Skipped steps are predicted using position-specific rules, and recomputation is triggered by a scale-free drift budget accumulated over the chaotic tokens. The approach aims to reduce solver evaluations without retraining, demonstrating speedups on RoboTwin 2.0 and LIBERO benchmarks.

Critical strengths

Method Novelties and Experiment Rigor

  • Identifies a Practical Bottleneck: The paper correctly isolates the action solver, rather than the visual backbone, as the primary source of latency in WAM inference once the future context is cached.

  • The curvature-based, scale-free difficulty proxy and chaotic-only drift budget provide a principled, training-free, online control signal that generalizes across action dimensions and trajectories. Overall, while these designs are heuristic, they make sense to some extent.

  • The framework is evaluated across extensive simulation suites (3,000 RoboTwin 2.0 episodes, 1,200 LIBERO episodes, and 9,305 OOD LIBERO-Plus tasks), demonstrating robust latency reduction with negligible impact on task success.

Critical weaknesses

However, the reviewer found the following key issues:

  • Poor Readability and Missing Intuition: The writing heavily focuses on mathematical formulations while neglecting the physical intuition necessary to understand the method's relevance to the robot execution phase. It is difficult to conceptualize how the three dynamic sets (stable, linear, chaotic) meaningfully translate to physical robotic control. The over-indexing on formula details leaves the reader lost. Furthermore, the overall scheme in Figure 1 is just a flow of computation, and does not provide good enough intuition beyond design choices such as Stable, Linear, and Chaotic.

  • Inadequate Result Presentation: Tables 1 and 2 report only success rates, whereas a trade-off comparison between success rate and FLOPS (or latency) would be significantly more informative. Because the success rate improvements are minor, it is confusing whether the primary goal is enhancing performance or saving computational costs, leading to a lack of compelling evidence for the method's utility. Furthermore, the figure captions introduce several methodological terms without prior explanation of the insights obtained from the results, making them difficult to interpret independently.

  • (Most important) Hallucinated and Incorrect Reference The paper contains severe citation errors and hallucinations that violate ICLR rules. For instance:

  • In line 528, the first author of "MagCache: Fast video generation with magnitude-aware cache" is incorrectly listed as "Yuan Cai et al." The correct first author is Zehong Ma.

  • In lines 679-681, the authors for "Accelerating diffusion transformers with token-wise feature caching" are listed as "Chang Zou, Xinyin Liu, Ting Liu, Shichen Huang, and Linfeng Zhang". Two names are altered; the correct author list is Chang Zou, Xuyang Liu, Ting Liu, Siteng Huang, Linfeng Zhang.

  • In lines 594-596, the last two authors for the paper "Timestep embedding tells: It's time to cache for video diffusion model" are given as "Raghav Pandey, and Hang Xu". This is a complete hallucination. The actual last two authors are Qixiang Ye and Fang Wan.

There are also other issues about using Irrelevant Citations. For instance, the paper cites "DistriFusion" (Li et al., 2024) to support the statement: "layer/block caches reuse internal U-Net or DiT features on fixed or learned schedules." This is factually incorrect. DistriFusion enables running diffusion models in parallel across multiple devices to reduce latency. It is a distributed, parallel inference framework, not a feature-caching mechanism for skipping layers.

Questions for authors

The author can address my main concerns regarding the weaknesses noted above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.