acceptodds
Under review as a conference paper at ICLR 2027

HoloBrain-Discrete: Shared Action Tokenization for Cross-Embodiment Robot Learning

Abstract

Learning a single policy across robot embodiments requires an action representation that retains joint-level motion while accommodating different robot structures. Joint-graph trajectories, which record each joint's angle and link pose, expose this structure but create many correlated prediction targets. We present HoloBrain-Discrete, which couples a virtual-joint action tokenizer with a code/cue delayed autoregressive policy. Virtual joints serve as latent bottleneck nodes positioned on the robot graph, compressing trajectories into discrete codes with one encoder and decoder shared across embodiments. The policy generates these codes in groups along temporal, virtual-joint, and coarse-to-fine refinement axes, advancing one group per step rather than one code at a time. A separate prediction cue for each code gathers its context from earlier groups. Compact, structured action targets support efficient cross-embodiment pretraining and few-shot transfer. Pretraining on approximately 10,000 hours spanning 18 embodiments yields higher success than diffusion-based HoloBrain at every measured checkpoint across five evaluation settings. Cross-embodiment demonstrations also boost few-shot learning: co-training two UR5 demonstrations per task with Aloha data raises success from 6% to 39%, versus 7% to 17% for HoloBrain. Model inference is 1.9–2.1 faster than HoloBrain on LIBERO and RoboTwin.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation, Action Tokenizer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bECnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HoloBrain-Discrete, which builds on HoloBrain’s joint-centric action representation and graph attention, introducing a shared virtual-joint tokenizer and a delayed autoregressive policy that predicts discrete action codes, which are subsequently decoded into continuous-valued joint and link-pose trajectories. Its main contributions are a shared virtual-joint tokenizer that compresses joint and link-pose trajectories across embodiments, and a code/cue delayed autoregressive policy that predicts token groups across temporal, virtual-joint, and residual-refinement axes. Evaluation uses approximately 10,000 hours of data spanning 18 embodiments, simulation benchmarks, limited-demonstration co-training, and seven real-world tasks. The method achieves stronger performance at measured pretraining checkpoints, improved cross-embodiment co-training results, and approximately 1.9–2.1× faster model inference than HoloBrain. However, post-training gains are mixed, with lower average real-world progress.

Critical strengths

Seems useful approach to compressing structured, heterogeneous action spaces. The virtual-joint bottleneck preserves access to robot connectivity while sharing tokenizer parameters across embodiments.

Critical weaknesses
  1. The comparison omits a particularly relevant competing method, FASTer. Although acknowledged in Section 2, FASTer is absent from the representation and policy evaluations. Its learned residual tokenizer and blockwise autoregressive generation address closely related problems, making it especially relevant for assessing the additional value of virtual-joint tokenization and code/cue delayed generation.

  2. The stronger pretraining results do not consistently translate into better deployed policies. Post-training gains remain uneven, with lower real-world manipulation scores.

Questions for authors

Table 10 shows substantial real-world progress deficits on Piper relative to HoloBrain for towel folding (−29.4 points) and bottle disposal (−19.5 points). Is it possible to do targeted failure analysis or ablations to identify whether these deficits primarily originate in perception, token prediction, tokenizer reconstruction, or control?

Reviewer_41aJOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

HoloBrain-Discrete aims to make cross-embodiment robot learning more scalable. It reads robot structure as a joint graph and compresses trajectories into discrete tokens through virtual nodes placed on that graph. The virtual joints keep each robot’s physical connectivity and joint information, while producing tokens that one model can share across robots. To avoid the slow inference of generating tokens one by one, the policy predicts groups of codes in parallel, which makes inference 1.9–2.1× faster than diffusion-based HoloBrain. Pretrained on about 10,000 hours from 18 embodiments, it outperforms HoloBrain at every pretraining checkpoint and in few-shot co-training, while results after post-training are mixed and real-robot scores are lower.

Critical strengths
  • The paper offers a basis for policy learning that is not tied to one robot. A single encoder and decoder handle robots with different joint counts, and the structure comes from the robot’s own geometry. This is a sensible direction for cross-embodiment generalization.
  • Inference is fast. Grouped generation is 3.49× faster than full autoregressive decoding on RoboTwin in Table 2, and 1.9–2.1× faster than diffusion-based HoloBrain. Fast inference is meaningful for real deployment.
  • Training with data from another robot makes the policy work well even with very few demonstrations of the target robot. With two UR5 demonstrations per task, co-training with Aloha raises success from 6.00% to 39.38%, compared with 7.13% to 16.69% for HoloBrain (Fig. 6). This is a real advantage on physical robots, where collecting demonstrations is expensive.
Critical weaknesses

W1. It is unclear where the pretraining gain over HoloBrain comes from.

  • The paper presents compression and discretization as the way to speed up learning (Sec. 1). But HoloBrain-Discrete differs from HoloBrain in at least four ways at once. It compresses each RoboTwin chunk (7,168 values into 192 codes), it uses discrete codes, it generates with delayed autoregression instead of diffusion, and it adds training tricks such as code corruption and label smoothing (App. A.3). The current comparison cannot tell which of these matters.
  • HoloBrain is not included in the controlled mixed-pool studies (Tables 1 and 2). The compression sweep (Fig. 7) shows that fewer codes help inside the discrete model (672 → 192 codes: Aloha 59.75 → 73.25, UR5 33.25 → 57.50). It still does not separate discretization from the generation method.
  • The HoloBrain setup is not described. It is unclear whether both models use the same pretraining data, VLM backbone, batch size, and number of updates.
  • At 150k updates, both models are still improving. HoloBrain rises on Aloha (17.5 → 30.9 → 39.6) and UR5 (14.3 → 21.7 → 28.9). On LIBERO and LIBERO-Plus it rises and then drops (e.g., 5.45 → 50.65 → 39.75). This is not discussed.
  • After post-training, HoloBrain is better on clean and randomized Aloha, LIBERO, and the real robots. HoloBrain-Discrete is better only on LIBERO-Plus (Table 3). The abstract reports only the pretraining checkpoints. Overall, the results support faster early learning, not a better final policy.

W2. The joint graph is not tested on robots with different structures.

  • The paper motivates the joint graph by robot heterogeneity: EE commands drop joint motion, and padded vectors ignore geometry (Sec. 1).
  • However, all four dual-arm robots used for evaluation share one structure: two arms, each with six joints and a gripper (J=14, V=4, Table 4). Aloha and UR5 differ in link geometry, but their graphs and virtual-joint layouts are identical. LIBERO uses a single EE node (J=1, V=1), so the graph and virtual-joint placement are trivial there. Robots with other graphs, such as dual-arm Franka (J=16, App. A.1), are in the training corpus but are not evaluated.
  • Some of the largest gains in Table 1 appear on LIBERO, where the graph plays no role: +20.8 points over OAT and +34.3 over FAST on LIBERO, and +51.7 over FAST on LIBERO-Plus. There, the method is essentially patch-wise residual FSQ, a state-conditioned decoder, and the code/cue policy. A large part of the gain may therefore come from parts other than the graph.
  • The experiment that directly tests the motivation is missing: keep the tokenizer and policy fixed, and replace only the input with a padded joint vector or an EE pose. Table 1 cannot do this, because FAST and OAT also change the tokenizer.
  • No robot unseen by the tokenizer is evaluated. The few-shot study uses a tokenizer that was already trained on UR5 motion (App. B.1).

W3. Two stated contributions are not directly tested.

  • Virtual-joint placement. Contribution 1 relies on placing virtual joints on the robot graph. All virtual joints start from the same learned vector, so graph distance is the only thing that tells them apart (Fig. 3). The paper varies how many virtual joints there are (Fig. 7), but not where they are placed, for example random positions or OAT-style registers without graph distances. The OAT comparison in Table 1 also changes the code count, tokenizer budget, and compression unit (Table 12), so it does not isolate placement.
  • Three-axis delay. Contribution 2 claims delay along time, virtual joints, and residual stages. Table 2 removes two of these axes, and only the residual axis helps. Removing residual delay lowers Aloha by 3.25 and LIBERO by 8.15 points. Removing virtual-joint delay gives 74.00% vs. 73.25% on Aloha at the same latency, and on LIBERO it changes nothing because V=1 (App. B.5). The authors themselves only conclude that residual-stage dependencies should be kept (Sec. 4.4). Unless the virtual-joint delay helps somewhere, the contribution is better described as two-axis delay.

W4. The real-robot gap may be linked to the chosen compression level.

  • On the real robots, HoloBrain-Discrete scores 74.29 vs. 85.30 for HoloBrain. The largest gaps are towel folding (−29.4), bottle disposal (−19.5), and block stacking (−17.7) (App. B.3). The authors suggest that compression loses detail needed for fine manipulation (Sec. 4.5).
  • The default V=4 was chosen from simulation, where it is best on Aloha, UR5, and LIBERO-Plus (Table 14). On the real seven-task data, however, it has the worst reconstruction of all five settings, worse even than V=2 (MAE 0.040 vs. 0.025–0.031, Table 15). This fits the authors' own hypothesis but is not discussed.
  • The sweep tokenizers are trained on the mixed pool, not the full-corpus tokenizer used for deployment, so this link is indirect. Still, the claim that reconstruction does not predict policy quality (Sec. 4.4) rests only on simulation, since the sweep has no real-robot rollouts (App. B.4). It remains open whether a less compressed setting would close the real-robot gap.
Questions for authors
  1. Was HoloBrain trained with the same pretraining data, VLM backbone, batch size, and number of updates as HoloBrain-Discrete, in both pretraining and post-training? How does HoloBrain behave beyond 150k updates, and why does its LIBERO and LIBERO-Plus success drop between 100k and 150k?
  2. Can you separate compression from discretization? One option is a diffusion policy that generates the same continuous virtual-joint features, compared with the discrete policy under the same setup.
  3. On the mixed pool, how does the method perform with the same tokenizer and policy, but with a padded joint vector or an EE pose as input?
  4. Are there results on a robot with a different graph (e.g., dual-arm Franka, J=16), or on a robot held out from tokenizer training?
  5. How does performance change with random virtual-joint placement, or with registers that use distinct learned initial vectors instead of graph distances?
Reviewer_4jrxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes HoloBrain-Discrete, a shared action tokenizer and policy for learning across robot embodiments. A graph-conditioned encoder compresses joint trajectories through virtual latent nodes and residual quantization, while a decoder reconstructs actions using the robot’s structure and current state. The policy predicts these codes through grouped autoregressive generation with separate prediction cues. Experiments evaluate pretraining progress, transfer from demonstrations collected on other robots, compression and generation choices, and downstream simulation and real-world performance.

Critical strengths
  • This paper provides a useful framework for shared action representation across embodiments.The paper combines robot structure with compact action tokenization, enabling a shared encoder and decoder to accommodate different joint configurations while retaining joint-level motion information. This addresses an important challenge in learning policies from heterogeneous robot demonstrations.
  • The paper provides comprehensive results to validate the proposed method and its design choices. For example, Figure 7 shows that moderate compression improves policy performance despite worse reconstruction accuracy, providing useful insight into representation design. Table 2 demonstrates that grouped generation substantially reduces inference latency while maintaining or improving success with the tokenizer held fixed.
Critical weaknesses
  • Despite pretraining across many embodiments, the controlled transfer study focuses on adding Aloha (Piper) demonstrations to UR5 (PiperX) policy training under low-data conditions, and the demonstrated embodiment gap is relatively small (all are bimanual tabletop arms with grippers although Figure 1 shows a bimanual mobile/humanoid arms with dexterous hands).
  • The learning advantages do not consistently translate into stronger final manipulation performance. Table 3 shows that, after benchmark-specific fine-tuning, HoloBrain-Discrete slightly underperforms HoloBrain on Aloha and LIBERO and achieves substantially lower real-world task progress. Similar results can be found on Table 8 and Table 10 in the appendix. Thus, the stronger pretraining results and low-data transfer gains do not establish a consistent advantage in final deployment performance compared to HoloBrain.
Questions for authors
  • Compared with OAT, how much of the improvement comes from graph-positioned virtual joints rather than differences in code budget and training setup? Is there a matched comparison that separates these factors?
  • In the real-robot experiments, have you tried to use the end-effector 6D poses and the gripper actions as the actions, which might alleviate the issue of joint mismatch between Piper and PiperX shown in the attached video?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.