acceptodds
Under review as a conference paper at ICLR 2027

ActionCodec2: A Streaming Action Tokenizer

Abstract

In autoregressive vision-language-action (AR-VLA) models, action tokenizers map continuous actions to discrete tokens and back to executable actions. Most existing tokenizers are chunkwise, allowing the robot to react only at chunk boundaries. AR models can instead support streaming tokenization, where each action is decoded as soon as its tokens are available. Existing streaming designs, however, either require many tokens or lack distribution-independent reconstruction guarantees, while small errors in incremental actions can accumulate into trajectory drift. We therefore seek the minimum token rate subject to bounded reconstruction error and a user-specified trajectory tolerance. Under the group structure of the action space, this becomes a covering problem. We prove fixed motion primitives optimal among one-label-per-step decoders, with an exact closed-form solution for translation and constant-factor-tight bounds for orientation. We further compress the primitive stream losslessly with Set-BPE, a byte-pair encoding over unordered sets. The resulting ActionCodec2 requires no gradient training. Fitted on k hours from embodiments, it encodes one second of motion in – tokens on average. Using the same VLA backbone, ActionCodec2 outperforms the compared tokenizers on LIBERO (), LIBERO-Plus (), and SimplerEnv (), and reaches task progress across four zero-shot Franka tasks. More results are available at  https://actioncodec2-anon.pages.dev.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, Action Tokenizer, Discrete VLA, Vision Language Action models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WzRNOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ActionCodec2, a streaming action tokenizer for autoregressive VLAs. Unlike existing chunk-based tokenizers, ActionCodec2 allows actions to be decoded and executed as tokens are generated. The method uses error-feedback quantization to keep trajectory reconstruction errors within a given tolerance, and formulates the minimum required action vocabulary as a geometric covering problem. It also introduces Set-BPE to compress action tokens across dimensions and time steps. Extensive experiments show strong performance across simulation benchmarks and real-world robotic tasks. The authors further demonstrate how action tokens can serve as a control interface for VLM agents, allowing them to build and reuse motion macros.

Critical strengths
  1. The paper is well organized and easy to follow. The figures and tables are also clear and well designed.

  2. I find the motivation behind streaming tokenization and bounded tracking error interesting. The geometric covering formulation makes sense, and the theoretical results directly guide the method design. I appreciate this approach, as it is more convincing than simply stacking additional neural network modules.

  3. The experiments are solid, covering multiple simulation benchmarks and real-world robot experiments. The results demonstrate the advantages of ActionCodec2 in imitation learning, reinforcement learning, and streaming execution.

  4. I find the Agent Interface experiments particularly interesting. The authors show that frontier LLMs can compose action tokens into reusable macros and gradually improve them across attempts. This is a novel and interesting way to use action tokens beyond conventional VLA training, and I think it opens up some promising possibilities for future research.

Critical weaknesses
  1. Some parts of the paper are difficult to understand, especially the theoretical sections. There are many definitions and proofs, but not enough intuitive explanations or examples. I think the authors could make the theory much easier to follow with a few simple examples.

  2. The authors claim that streaming tokenization enables closed-loop feedback within an action chunk, allowing the policy to adjust its actions during execution. This makes sense, but the paper lacks experiments that directly demonstrate this capability. I would like to see how ActionCodec2 performs when the environment changes or unexpected disturbances occur during chunk execution, and whether streaming feedback actually improves the policy's ability to react to these changes.

  3. Although the authors provide a pretrained universal version of ActionCodec2, its vocabulary size and tracking-error tolerance are fixed. Different robots and tasks may require very different levels of precision and control frequencies. I am not sure whether a single fixed configuration can meet all these requirements in practice.

Questions for authors
  1. Could the authors provide a simple worked example illustrating how the covering formulation determines the primitive set and prevents trajectory drift? Please also clarify the scope of the optimality guarantee, particularly the role of the full tolerance requirement.

  2. Does the evaluated streaming policy incorporate new observations and revise subsequent actions during execution, or does it only release actions earlier from an otherwise fixed prediction? What experimental evidence demonstrates improved reactions to disturbances introduced during chunk execution, beyond the latency benefits shown in Section 7.6?

  3. How should the pretrained universal tokenizer be adapted to robots or tasks requiring different tracking tolerances and control frequencies? Would this require rebuilding the primitive set, refitting Set-BPE, or retraining the VLA? Please clarify the trade-offs among precision, token rate, and compatibility with the pretrained vocabulary.

Reviewer_PoL3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces ActionCodec2, an action tokenizer for autoregressive VLAs that enables action decoding in a streaming manner from compressed tokens. They achieve this by framing tokenization as a worst-case covering problem, ensuring that reconstruction error is bounded and by then compressing primitives with Set-BPE. The authors show that ActionCodec2:

  • outperforms FAST, OAT, X-Tokenizer and ActionCodec on LIBERO, LIBERO-Plus, SimplerEnv and zero-shot real-robot Franka tasks, using the same G0.5 backbone;
  • improves reactiveness under a 20 Hz real-time control loop, since actions execute as soon as their tokens are decoded instead of waiting for whole chunks;
  • converges faster and higher under PPO fine-tuning than the chunk tokenizers;
  • can serve as a control interface for VLM agents, which build reusable token macros across episodes.
Critical strengths
  • Controlled, broad evaluation: same VLA backbone and training recipe for all tokenizers, fair baseline handling, and consistent gains across LIBERO, LIBERO-Plus, SimplerEnv, PPO fine-tuning and real-robot experiments (largest in OOD settings).
  • A single universal tokenizer with a per-step worst-case error bound, fitted without gradient training and transferring across embodiments.
  • Well motivated through replay-drift analysis.
Critical weaknesses
  • Streaming alone is not new: per-dimension binning already streams, and time-ordered local-support spline tokenizers (e.g., BEAST, whose control-point tokens are interleaved in temporal order) could, in principle, decode progressively after a few control points. The paper groups BEAST with chunk-only tokenizers without discussing this or comparing against it.
  • The streaming advantage is only weakly supported. It is isolated in a single experiment (Table 2, 20 Hz LIBERO), where ACodec2+ streaming (91.0) beats the best chunked baseline, OAT_aligned (87.2), by only 3.8 points. The large gap is to ACodec2+'s own chunked variant (73.1). Chunked baselines send zero motion when their queue empties, and the chunk-overlap schedule is unspecified.
  • Table 1 pauses the simulation during decoding, so its gains cannot be attributed to streaming execution, but instead support the representation itself. While pretraining and the second-order variant are attributed by the ACodec2 variants, it is not clear which property of the base ActionCodec2 (e.g., bounded error, per-step targets, or token rate) drives its gains over the baselines beyond the Set-BPE ablation.
  • The real-robot comparison conflates streaming with the tokenizer and reports only inference latency, not completion or idle time. The paper includes no experiment on reactiveness to mid-execution changes as illustrated in Fig. 1(3). Also, Fig. 1(2) overstates the locality of errors: an incorrect incremental token can shift all subsequent reconstructed poses. Overall, the main streaming contribution needs either reframing or stronger evidence.
Questions for authors
  • How relevant is episode-level replay drift (Fig. 2) when deployed policies re-anchor at every chunk? Replay success does not predict policy success (e.g., ActionCodec 61% replay vs 95.1% LIBERO; FAST 90% vs 92.8%). Does the decoder re-initialize from proprioception at each chunk?
  • The project website shows in-chunk perturbation suites (random motion/physics/force; e.g., 46.2% vs 27.6% vs 9.2% grasp rate), but this experiment is not in the paper and it should be included. Does the closed-loop variant receive privileged object state that the baselines lack, and how do chunked baselines compare when replanning more often (fewer than 20 executed actions per chunk, which is kind of a large action chunk for libero)?
Reviewer_JW3mOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes ActionCodec2, a streaming action tokenizer for autoregressive VLAs. It formulates action quantization as a covering problem with bounded reconstruction error, derives optimal or near-optimal motion primitives under specific assumptions, and introduces Set-BPE to compress action tokens across spatial dimensions and time. Experiments report competitive performance on simulation benchmarks and real-robot tasks.

Critical strengths
  • The problem of efficient streaming action tokenization is relevant, particularly for reducing inference latency in autoregressive VLAs.
  • The combination of error-aware quantization and Set-BPE is technically interesting and achieves promising compression rates.
  • The paper includes extensive simulation experiments, real-robot evaluations, and several ablation studies.
Critical weaknesses
  • The paper is unnecessarily difficult to follow. Numerous technical concepts are introduced without sufficient definitions, intuition, or motivation. Terms such as "full tolerance requirement," "composition decoder," "one-sided perturbation," and various Set-BPE constructs are presented with little explanation of their practical meaning. The mathematical development is dominated by formal details, while the central ideas and their relationships remain obscure. The transitions between the covering formulation, optimality results, second-order extension, and actual tokenizer are particularly abrupt. As a result, it is difficult to understand what each theoretical result establishes and why it is needed.

  • Section 2 defines streaming as requiring at least one additional token for every action step. However, ActionCodec2 reports only 8.8–10.8 tokens per second for 15 Hz actions, violating this definition. Moreover, Theorem 2 establishes optimality for one-label-per-step decoders under the full tolerance requirement, whereas the actual tokenizer uses variable-length spatial and temporal Set-BPE merges. The paper does not clearly explain how the theoretical optimality extends to the implemented system or its average token rate.

  • The reconstruction guarantee assumes a known bounded motion set and an encoder that observes the true next action and corrects previous quantization errors. Neither condition directly applies to autoregressive VLA predictions. Furthermore, the practical motion bounds are estimated using q99.9 statistics rather than guaranteed physical limits. Therefore, the claimed distribution-independent guarantee does not directly establish robustness to OOD predictions, accumulated policy errors, or actual robot execution.

  • Proposition 9 requires relaxed velocity and acceleration-like bounds for the reconstructed trajectory to maintain bounded error. If the original bounds represent physical constraints, the reconstructed commands may not be executable. The paper provides insufficient justification or empirical validation for these additional margins, further weakening the connection between its mathematical guarantees and robot control.

  • Several important claims are overstated. For example, Figure 1 suggests that an incorrect streaming token affects only one action step. This is generally not true for incremental actions, where errors can propagate through subsequent states. Temporal Set-BPE merges can also associate one token with multiple steps. Similarly, the worst-case covering arguments in Appendix G are used to criticize learned chunkwise tokenizers, but the analysis ignores the strong temporal correlations and low-dimensional structure of realistic robot trajectories.

  • The real-time comparison primarily evaluates streaming against chunkwise decoding that waits for complete chunks, without a sufficiently strong asynchronous or receding-horizon chunk baseline. The results therefore do not establish that streaming tokenization is inherently superior to practical chunkwise execution. Moreover, improvements from reconstruction accuracy, compression, streaming, and second-order modeling are not sufficiently disentangled. A simple error-feedback scalar quantizer with standard BPE would be an important baseline.

Questions for authors
  • How does ActionCodec2 handle invalid or incomplete token sequences generated by an autoregressive VLA? Are decoding failures measured, particularly given the paper's criticism of malformed FAST outputs?

  • How are coordinate frames, action scales, control frequencies, and action conventions standardized across different embodiments when fitting and applying the universal tokenizer?

  • What are the distributions of tokens per action step and actual action-release latency, rather than just their averages? In particular, how frequently do temporal merges introduce delayed or multi-step decoding?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.