acceptodds
Under review as a conference paper at ICLR 2027

ActWeave: Efficient Action Generation for Vision-Language-Action Models via Hierarchical Autoregression and Wavefront Decoding

Abstract

Autoregressive vision-language-action (VLA) models have demonstrated strong robotic manipulation capabilities. However, fine-grained action tokenization can produce long sequences, making token-wise decoding through large vision-language models (VLMs) expensive. We introduce **ActWeave**, a hierarchical autoregressive framework that shifts VLM computation from tokens to structured action patches. A deterministic discrete cosine transform (DCT) tokenizer retains all quantized coefficients in an action-dimension–frequency grid without a learned codebook. The VLM-based Thinker models inter-patch dependencies, while a lightweight Weaver performs **wavefront decoding**, predicting each anti-diagonal in parallel conditioned on earlier wavefronts. Direct VLM fine-tuning without robot-data pretraining yields competitive LIBERO and SimplerEnv performance. Controlled LIBERO comparisons show a 17.9× full-chunk speedup over flat global autoregression. Wavefront decoding reduces latency by 36.0% relative to raster local decoding at comparable success rates. VLABench reconstruction-and-replay experiments show that the tokenizer preserves control-relevant information under the evaluated distribution shift. These results demonstrate efficient autoregressive action generation without shortening the action-token sequence.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, robot learning, autoregressive modeling, action tokenization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6G8YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ActWeave is a hierarchical autoregressive VLA that reduces the cost of generating discrete action tokens. A VLM predicts one action patch at a time, while a lightweight Weaver generates the tokens within each patch using wavefront decoding. The paper evaluates task success on LIBERO, SimplerEnv, and VLABench, and uses architectural ablations to measure inference latency and the effects of its design choices.

Critical strengths

The controlled comparison shows a substantial reduction in full-chunk latency relative to token-by-token VLM decoding while retaining the same number of action tokens. This directly supports the paper’s central efficiency claim.

ActWeave achieves competitive success rates on both LIBERO and SimplerEnv without robot-data pretraining. These results show that the faster generation design can support effective closed-loop control.

Critical weaknesses
  1. The paper’s main contribution is inference efficiency, but it does not directly compare ActWeave’s latency with publicly available efficient VLA baselines under the same hardware and timing protocol. The large speedup over token-by-token VLM decoding therefore does not establish whether ActWeave offers a better success-rate–latency tradeoff than existing methods.

  2. Table 2 evaluates architectural variants after only 10k training steps, while the main LIBERO result uses an 80k-step checkpoint. Different architectures may converge at different rates, so the observed success-rate differences cannot establish that the Patch Encoder, Weaver, or wavefront decoding improve performance after sufficient training. The latency differences remain informative, but the success-rate attribution is unresolved.

Questions for authors
  1. Please provide a direct latency comparison with representative publicly available efficient VLA baselines under the same hardware and timing protocol.

  2. Please report whether the Table 2 variants retain their success-rate differences after longer training, using the same training settings as the main LIBERO experiment.

  3. Please provide a real-robot evaluation.

Reviewer_K1wmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ActWeave, a hierarchical autoregressive framework designed to accelerate discrete action generation in Vision-Language-Action (VLA) models without shortening the tokenized action sequence. Instead of compressing action chunks via learned codebooks (e.g., VQ-VAE/RVQ) or byte-pair encoding (as in FAST), ActWeave partitions an HH-step action chunk into N=⌈H/P⌉N = \lceil H/P \rceil non-overlapping temporal patches of length PP and applies a deterministic Fixed-Grid Discrete Cosine Transform (FG-DCT) tokenizer. FG-DCT retains all quantized coefficient positions to form a fixed Da×PD_a \times P action-dimension–by–frequency token grid per patch. ActWeave achieves competitive results on simulation benchmarks such as Libero, Simpler.

Critical strengths
  1. A practical problem framed cleanly, with competitive results from a general VLM. The paper separates two quantities that are usually conflated: how fine-grained the action tokens are, and how many sequential VLM evaluations are needed. It keeps the full, uncompressed DCT grid (no lossy BPE or learned codebook) while reducing VLM evaluations per chunk from 70 to 2. Starting from Qwen3-VL-4B without robot-data pretraining, it reaches 95.9% on LIBERO and 84.38% (405/480) on SimplerEnv-WidowX, at a measured deployment latency of about 115 ms per chunk on LIBERO.
  2. Transparent reporting. The paper gives raw success counts, task-cluster bootstrap CIs, exact McNemar tests, a detailed latency protocol (multiple launches, preprocessing included) and memory use. It also reports results that cut against its own design: the non-monotonic patch/horizon sweep, the Patch Encoder probe in F.2, and the selection history of the VLABench decoding rule. This level of disclosure is what makes a careful assessment possible, and it should be the norm.
Critical weaknesses
  1. ActWeave is a tunable sub-chunk with a small autoregressive action head, which departs from the AR-VLA premise the paper invokes. Its mechanism is not explained.
    • Each patch is simply a shorter action chunk. The VLM advances once per sub-chunk and never emits action tokens itself. All token-level generation is done by a separate 4-layer Weaver that runs its own autoregression. This is an action-expert design with an AR head, not VLM autoregression.
    • This conflicts with the motivation in the introduction. AR-VLAs are attractive because action tokens are produced by the pretrained VLM's own sequence modelling, which "can accelerate policy adaptation and help preserve pretrained knowledge" (Driess et al., 2025). In ActWeave the VLM only supplies N = 2 hidden states per chunk, so none of these benefits is shown to carry over.
    • None of the experiments that would establish these AR-VLA advantages appear in the paper. There is no test of language following or instruction generalization, no convergence or adaptation-speed comparison (e.g., success vs. training steps), and no measurement of retained VLM capability. There is also no controlled comparison with a standard AR-VLA in which the VLM generates the action tokens itself. The only same-backbone AR reference is variant F (flat AR over uncompressed DCT tokens, 10k steps, 160 episodes); the AR rows in Table 1 (OpenVLA, π0-FAST) are source-reported numbers with different backbones and training data.
    • Is the hierarchy needed? In Table 11 (1k steps), (H, P) = (10, 10) uses a single patch: no inter-patch Thinker autoregression and no use of the Patch Encoder. It scores 87.0% at 73.5 ms vs 85.0% at 106.7 ms for the default (10, 5), and (20, 10) reaches 90.0%. This happens even though the sweep favours the default: every configuration restores the same 60k-step LIBERO backbone trained at (10, 5) and only re-initializes the action modules for 400/1k adaptation steps. Probe F.2 adds that the structured Patch Encoder is worse than a plain MLP (0.299 vs 0.148 MSE). The effective method may reduce to "one VLM prefill plus a small AR head over a DCT grid".
    • Why anti-diagonals? The schedule seems dictated by the head design rather than by the data: each frequency appears at most once per anti-diagonal, so one hidden state with P frequency-specific heads suffices. The stated motivation (lower before higher frequencies within a dimension) is better served by column-major parallel decoding: all dimensions per frequency, P = 5 steps instead of 11, fixed group size without ragged masking, and conditioning on the lower frequencies of all dimensions. Row-major parallel decoding (D_a = 7 steps) is the other obvious control. The only dependency unique to the diagonal is on the arbitrary action-dimension index (d conditions on d − 1 at the same frequency). The audio delay-pattern rationale (a long streaming time axis and an ordered RVQ-level axis) does not apply to a fixed 7 × 5 grid.
  2. The efficiency claims rest on weak reference points, and the alternatives named in the paper are never compared.
    • The 17.9× speedup is measured against flat AR over 70 uncompressed DCT tokens through the 4B VLM, which no practical system would deploy.
    • In the paper's own controlled timing (Table 7), single-action global AR takes 168.6 ms vs 58.7 ms for ActWeave (≈2.9×). The parallel continuous head (38.9 ms) and the 10-step flow-matching head (53.8 ms) are both faster than ActWeave.
    • The introduction itself names block-wise and parallel decoding (FASTer, PD-VLA) and compressed tokenizers (FAST/BPE, learned VQ) as the competing routes, yet none is evaluated on the same backbone. Within ActWeave, row-major and column-major parallel schedules are also missing; Table 2 only compares fully serial raster (ordering unspecified), wavefront and fully parallel decoding.
  3. The experimental protocol is underpowered and partly tuned on evaluation data.
    • The Bridge→VLABench experiment reports a single number (42.8%) with no reference point: no VLABench policy trained directly from the VLM (so the benefit of the 10k-step Bridge initialization is unquantified), no baseline architecture and no latency. Bridge (real WidowX, one camera) and VLABench (simulated Franka, three cameras, Euler deltas) differ in embodiment, viewpoint and action space. The number is also fragile: the decoding rule was switched on overlapping evaluation scenes (36.0→42.8), and continuing to 13k steps gives 34.2%. What claim is this experiment meant to support?
    • There are no real-robot experiments. All results are in simulation, which matters especially for a paper whose main claim is inference efficiency: latency is measured offline with simulation excluded, and the paper itself notes that it "does not measure real-time deadline failures". Whether the speedup translates into higher achievable control frequency or better closed-loop behaviour on hardware is not shown.
Questions for authors

See above.

Reviewer_nETQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes ACTWEAVE, a hierarchical autoregressive framework for efficient action generation in VLAs. It introduces a fixed-grid DCT tokenizer that preserves all quantized coefficients, a VLM-based Thinker for patch-level autoregression, and a lightweight Weaver for parallel wavefront decoding. The paper reports competitive success rates on LIBERO and SimplerEnv, along with substantial inference speedups over flat autoregressive decoding.

Critical strengths
  • Improving autoregressive VLA inference efficiency is a relevant and practically important problem.
  • Separating expensive VLM computation from lightweight local action generation is a reasonable design. Wavefront decoding also provides an intuitive way to reduce sequential decoding steps.
  • The paper includes evaluations across multiple benchmarks and several architectural ablations.
Critical weaknesses
  • The paper is difficult to follow, particularly in the methodology section. Multiple components, including FG-DCT tokenization, the Patch Encoder, hierarchical autoregression, and wavefront decoding, are introduced without clearly explaining their individual motivations or how they address the central problem. The presentation emphasizes implementation details rather than the key ideas, making it difficult to identify the actual contribution.

  • The proposed framework introduces too many changes simultaneously. Rather than accelerating token generation within an existing action tokenization framework, ACTWEAVE introduces a different tokenizer, a hierarchical architecture, and additional prediction modules. This substantially complicates the comparison with existing methods, as improvements may arise from changes in action representation, model architecture, or decoding strategy. Consequently, it is difficult to isolate the contribution of wavefront decoding itself.

  • Despite inference acceleration being the central motivation, Table 1 primarily compares success rates across methods with different backbones, training procedures, and action representations. Such comparisons provide limited evidence of the proposed method's effectiveness. The main controlled efficiency comparison instead focuses on Flat Global AR, a relatively inefficient baseline. More relevant comparisons against existing efficient generation approaches, such as FASTer and other block-wise or parallel decoding methods, are missing under matched settings. Notably, Appendix C.2 reports that parallel continuous and flow-matching heads can achieve lower latency than ACTWEAVE, further questioning its practical efficiency advantage.

  • The claimed acceleration is not sufficiently connected to practical control efficiency. The reported speedup comes from an architectural comparison rather than an end-to-end comparison against competitive VLA systems. Moreover, the main LIBERO evaluation executes only one action per prediction, and Appendix C.1 reports approximately 115 ms of inference latency per executed action. It remains unclear whether the proposed method offers a meaningful practical advantage in closed-loop robot execution.

  • The motivation regarding action compression and reconstruction fidelity is not well justified. The introduction highlights the limitations of existing tokenizers such as FAST, particularly the need to preserve control-relevant information under distribution shift. However, it does not clearly explain how the proposed design fundamentally addresses these limitations. FG-DCT simply retains all quantized DCT coefficients without compression, making improved reconstruction fidelity less surprising. The reconstruction comparisons also involve different action-space adapters and lack matched token budgets, making it difficult to attribute the improvements to the proposed representation itself.

  • No real-robot experiments.

Questions for authors
  • How sensitive is the proposed framework to the patch and frequency loss weights? Are these hyperparameters transferable across different benchmarks and action representations?

  • What is the additional training cost and parameter overhead introduced by the Patch Encoder and Weaver compared with conventional lightweight action heads?

  • How sensitive is performance to the different token-selection and postprocessing strategies used across benchmarks? Would a unified decoding configuration yield similar results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.