- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces Physical Rank Consistency (PRC), a metric intended to measure whether an action tokenizer preserves local ordering of physical distances between action chunks after encoding and decoding. It proposes ActionPiece, an RVQ-based action tokenizer trained with a physical rank preservation loss over representation distances and a quantization regularization loss over codeword-assignment distributions, and evaluates it on LIBERO, LIBERO-Plus, SimplerEnv, VLA-Arena, and real-robot insertion tasks.
Critical strengths
The paper identifies a useful limitation of reconstruction-only tokenizer evaluation.
PRC is a sensible, decoder-agnostic evaluation concept. Measuring rank preservation in decoded action space rather than in each tokenizer's latent space makes the comparison meaningful across tokenizers with different vocabularies and sequence lengths. The use of physical translation, SO(3) geodesic rotation, and gripper discrepancies in Equations 4-5 is substantially more appropriate for this setting than a raw Euclidean distance over heterogeneous action coordinates.
The backbone, policy data, prompt, global batch size, action horizon, and policy training steps are held fixed. Under this setup, ActionPiece achieves 94.8% on LIBERO and 68.8% on LIBERO-Plus, outperforming the strongest reported baselines by 1.1 and 4.5 points, respectively. The larger gain under the out-of-distribution LIBERO-Plus shifts is consistent with the paper's hypothesis that preserving fine action distinctions can matter under perturbation.
The implementation details are relatively thorough. The manuscript specifies the action chunk length, number of tokens, codebook size, objective weights, physical-distance normalization, hard-forward/soft-backward codeword estimator, and training protocols.
Critical weaknesses
The central causal claim is not yet fully disentangled from improved reconstruction quality and other representation improvements.
The relation between the training objective and the reported PRC metric is weaker than the terminology suggests. PRP in Equation 8 is an extreme-pair triplet-style objective: for each anchor it selects one batch-local action at the 5th percentile and one at the 95th percentile, and enforces QR in Equation 10 imposes the analogous constraint on Jensen-Shannon divergence. In contrast, PRC@32 evaluates the rank ordering among the 32 nearest original neighbors. Enforcing the order of one near and one far point does not directly supervise the ordering among local neighbors, nor does it guarantee local neighborhood preservation. This distinction matters because the paper frames PRP and QR as physical “rank preservation,” while they supervise only sparse, batch-dependent ordinal constraints. The paper should evaluate PRC over several values of k report neighborhood-overlap or retrieval metrics, and compare against objectives that directly optimize local ordinal consistency or a differentiable rank surrogate.
The physical metric introduces consequential embodiment- and parameterization-dependent assumptions that are not stress-tested.
Since VLA fine-tuning can exhibit nontrivial seed sensitivity, the reliability of the main ranking is difficult to assess. At least three policy seeds for the matched LIBERO and LIBERO-Plus comparisons, accompanied by confidence intervals over evaluation rollouts, would substantially strengthen the evidence.
Questions for authors
Can the authors provide multi-seed results and confidence intervals for the matched Table 1 and Table 4 comparisons? In particular, is the 1.1-point LIBERO difference from ActionCodec statistically distinguishable from training and evaluation variation?
Were any LIBERO-Plus, SimplerEnv, VLA-Arena, or real-world evaluation results used for model selection or hyperparameter selection?
Can the authors provide a capacity- and reconstruction-matched analysis? For example, compare standard RVQ, PRP, QR, and ActionPiece at matched MSE, matched token count, and matched codebook utilization, then test whether PRC remains predictive of downstream success. How sensitive are PRC and ActionPiece to the action-distance design? Please report results under alternative translation/rotation/gripper weights, alternative rotational normalizations.
Given that the paper motivates physical-rank preservation for precise closed-loop behavior, can the authors provide failure analyses showing cases where two nearby demonstrations are reconstructed with reversed or attenuated adjustments by standard RVQ and correctly separated by ActionPiece? Such examples would make the mechanism more tangible than aggregate PRC alone.
5 comments
Response to Reviewer 66PF (2/3)
| Q1 Run-to-Run Variability and Robustness Checks
Thank you for asking whether the LIBERO gain is robust to training and evaluation variability. Our LIBERO evaluation uses repeated rollouts across different initial layouts for each task. We repeated the matched LIBERO comparison three times under the same protocol. The table reports success rates (%) as mean ± sample standard deviation across runs. Overall is the equal-weight mean of the four suites within each run.
| Tokenizer | Spatial | Object | Goal | Long | Overall |
|---|---|---|---|---|---|
| ActionCodec | 93.27 ± 0.61 | 99.00 ± 0.40 | 92.40 ± 0.60 | 89.80 ± 0.60 | 93.62 ± 0.35 |
| ActionPiece | 96.13 ± 0.70 | 98.60 ± 0.40 | 93.67 ± 0.46 | 90.67 ± 0.50 | 94.77 ± 0.10 |
The average Overall gain is 1.15 percentage points. The lowest ActionPiece Overall score (94.65%) exceeded the highest ActionCodec score (93.95%), so the advantage was consistent across the three runs.
LIBERO is already approaching saturation in this comparison, so the remaining headroom is limited. The gains are larger on LIBERO-Plus and SimplerEnv WidowX: ActionPiece achieves 68.8% versus 64.2% for ActionCodec on LIBERO-Plus (+4.6 percentage points, Table 1), and 71.9% versus 65.2% for ActionCodec-BAR on SimplerEnv WidowX (+6.7 points, Table 2).
Appendix G (Table 10) reports sensitivity to margins, loss weights, encoder–quantized mixing, and near–far sampling. We additionally report matched tokenizer comparisons on Gemma-4 E2B and Qwen2.5-4B below. ActionPiece improves LIBERO and LIBERO-Plus success over Plain RVQ with each backbone.
| Backbone | Tokenizer | LIBERO (%) | LIBERO-Plus (%) |
|---|---|---|---|
| Gemma-4 E2B | Plain RVQ | 90.4 | 61.3 |
| Gemma-4 E2B | ActionPiece | 93.2 | 65.1 |
| Qwen2.5-4B | Plain RVQ | 88.7 | 58.2 |
| Qwen2.5-4B | ActionPiece | 92.5 | 63.6 |
We did not use LIBERO-Plus, SimplerEnv, VLA-Arena, or real-world evaluation results for model selection or hyperparameter tuning. The LIBERO-Plus results in Table 10 are reported as additional evaluations of the hyperparameter settings.
|Q2 Matched Comparisons and the Physical Distance Definition
Appendix H (Table 11) separates depth from token budget. At a fixed budget of 16 tokens per chunk, we compare one level with 16 slots against two levels with eight slots, for both Plain RVQ and ActionPiece. We also report the 32-token setting that keeps all 16 slots at two levels. Tables 6 and 10 provide comparisons at similar reconstruction MSE, including the comparison discussed in W1. These comparisons control the token budget or compare similar MSE.
ActionPiece is also effective beyond EEF action parameterizations. On RoboCasa-GR1, we tokenize joint commands directly and use Euclidean distance between normalized joint-action chunks to select near–far pairs. With the same Qwen3-VL-4B backbone, ActionPiece improves mean success over FAST and Plain VQ by 5.17 and 6.25 percentage points, respectively (Appendix F, Table 9).
| Tokenizer (Qwen3-VL-4B, 100K steps) | RoboCasa-GR1 mean success across 24 tasks (%) |
|---|---|
| Plain VQ | 46.25 |
| FAST | 47.33 |
| ActionPiece | 52.50 |
Response to Reviewer 66PF (3/3)
|Q3 Preserving Action Differences Between Nearby Demonstrations
Thank you for suggesting a direct reconstruction analysis. Appendix I (Table 12) reports results over the full LIBERO tokenizer validation split. We recomputed the physical top-32 neighbors and report 14,638 same-task, cross-episode pairs and 181,039 cross-task pairs.
For a pair of chunks , we measure how much the reconstructed eight-step translation difference deviates from the original difference , relative to its magnitude: .
ActionPiece lowers median error by 54.2% on same-task, cross-episode pairs and by 53.5% on cross-task pairs. It also lowers the local sign-reversal rate for every translation and relative rotation component within the same-task group. This directly tests whether the decoded commands preserve differences between nearby demonstrations.
(a) Relative error in reconstructed translation differences
| Metric | Same task, different episodes | Cross-task neighbors |
|---|---|---|
| Number of pairs | 14,638 | 181,039 |
| Plain RVQ median relative error | 0.299 | 0.297 |
| ActionPiece median relative error | 0.137 | 0.138 |
| Relative reduction in median error | 54.2% | 53.5% |
(b) Local sign reversals within same-task, cross-episode pairs
| Component | Plain RVQ | ActionPiece |
|---|---|---|
| Translation | 5.22% | 1.36% |
| Translation | 5.46% | 1.97% |
| Translation | 4.71% | 0.89% |
| Relative rotation | 1.63% | 0.34% |
| Relative rotation | 2.01% | 0.48% |
| Relative rotation | 2.09% | 0.46% |
Compared with Plain RVQ, ActionPiece more than halves the median relative error in action differences for both groups and reduces sign reversals in all six translation and relative rotation components. These results show that ActionPiece more faithfully preserves the magnitude and direction of the adjustments between nearby demonstrations.
Response to Reviewer 66PF (1/3)
Thank you for your careful review and constructive suggestions. We appreciate your recognition of the limitation of reconstruction-only evaluation, the value of measuring physical ordering in decoded action space, and the controlled comparisons and implementation details.
| W1 Reconstruction Fidelity, Physical Ordering, and Policy Success
Our claim is that physical-order supervision improves tokenizer quality and downstream policy performance, while PRC provides complementary information beyond mean reconstruction MSE. The randomized-label control in Table 10 and the comparisons at similar MSE provide evidence for these two points, respectively.
We retain the original pairs and randomly reverse half of the near–far labels, randomizing the ordering supervision. Using the original physical labels raises PRC@32 from 0.871 to 0.953 and LIBERO-Plus success from 61.7% to 68.8%. At similar reconstruction MSE, the encoder/quantized weighting of 0.50/0.50 achieves MSE 3.47, PRC 0.955, and LIBERO/LIBERO-Plus success of 94.3%/68.9%, compared with MSE 3.50, PRC 0.871, and 91.2%/61.7% for the randomized-label control. At approximately the same mean reconstruction error, randomizing physical-order supervision substantially reduces PRC and downstream success. This shows that mean MSE alone is insufficient to explain the performance difference and supports the interpretation that preserving physical relationships benefits downstream control.
The similar-MSE comparison answers a different question: what does PRC tell us beyond reconstruction error? As discussed in Sections 3.1–3.2, similar reconstruction error can coexist with different levels of local physical-order preservation. In Tables 6 and 10, SIGReg and the QR-margin 0.05 variant both have an MSE of 3.77 at the reported precision, yet differ in PRC (0.950 vs. 0.932) and LIBERO-Plus success (66.5% vs. 64.3%).
| W2 Local Ordering and Neighborhood Diagnostics
We tested more direct local supervision by sampling 1, 8, or 16 pairs per anchor from its 32 nearest training-set neighbors. The margins, loss weights, and training budget are fixed (100K steps, seed 42), and all settings use the same validation samples and metric implementation. The results are shown below.
Eight pairs improve PRC and neighbor overlap over one pair at similar MSE, while sixteen pairs do not sustain that improvement. We also report complementary neighborhood metrics below. Appendix I (Table 12) adds a direct analysis of the differences between nearby reconstructed action commands.