acceptodds
Under review as a conference paper at ICLR 2027

How Should Vision-Language-Action Models Use Proprioceptive

Abstract

Recent Vision–Language–Action (VLA) models almost universally take robot proprioceptive state as input, yet wire it in incompatible ways—serialized into text prompts, projected into the vision–language prefix, or fed directly to the action expert—and almost always as a single current frame. Three questions remain open: (1) whether, and on which tasks, current state actually improves closed-loop control; (2) how much state history helps, and whether its benefit reflects genuine temporal variation rather than added conditioning capacity; and (3) where state should enter the model—the vision–language backbone or the action-generation module. We answer these questions through controlled experiments on a flow-matching VLA, fixing the backbone, training data, action representation, and evaluation protocol throughout. We implement five representative interfaces—discrete state prompt, VLM prefix, action prefix, state expert, and feature modulation—under matched implementation details, and evaluate them on 45 atomic tasks spanning three task families plus 20 composite tasks; we then sweep the state-history length from 1 to 96 frames to examine how historical state information affects model performance. The experiments yield systematic answers to all three questions, distilled into testable design principles for state-aware VLAs.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embeded AI, Robotics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_asdgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how proprioceptive state should be represented and integrated into vision-language-action models. Using a shared \pi_{0.5} scaffold, the authors compare five state interfaces and investigate the effects of current-state conditioning, state-history length, and injection into the VLM versus the action expert. Experiments cover 45 atomic tasks and 20 composite tasks in RoboCasa365. The main finding is an interaction between interface choice and history length: on composite tasks, the VLM-prefix interface performs better with a single state frame, while the action-prefix interface performs better with a short state history.

Critical strengths
  1. The paper addresses an important foundational question. Proprioception is widely used in robot policies, yet its representation, temporal context, and integration with pretrained VLMs are often treated as implementation choices. Investigating these decisions directly is valuable for understanding and designing VLAs.

  2. The experimental framework enables more informative comparisons than comparisons across unrelated models. The interfaces share a backbone, visual inputs, action representation, and evaluation protocol. Reporting results separately across task families also reveals trade-offs that would be obscured by a single aggregate score. The principal comparisons include multiple independent training seeds.

  3. The interface–history interaction is interesting, and the repeat-current control strengthens the evidence. The improvement from AP1 to AP8, together with the limited change from VP1 to VP8, suggests that temporal context and interface choice should be considered jointly. Training a slot-matched repeat-current baseline is a useful step toward distinguishing the benefit of temporal information from the effect of increasing the number of conditioning tokens.

Critical weaknesses
  1. I appreciate the focus on a fundamental and practically important question. However, prior work such as ThinkProprio, which the paper cites, already includes controlled comparisons of proprioceptive encoding and entry points. The authors should more explicitly distinguish their findings from these existing studies. The main incremental contribution appears to be the interaction between state history and the conditioning interface, supported by the history-length sweep and repeat-current control. While this interaction is interesting, its validation remains concentrated in one scaffold and one simulation benchmark. I am therefore not yet convinced that the current evidence establishes sufficiently general design principles beyond a useful empirical comparison within the evaluated configuration.

  2. The central interface–history interaction needs a deeper explanation. The most compelling result is that adding eight-frame state history improves AP from 28.6% to 38.7%, whereas VP remains essentially unchanged at 34.1% versus 34.0%. However, the paper does not sufficiently explain why the same numerical history benefits the action-side interface but not the VLM-side interface. It does not identify which temporal information is useful or why the interfaces exploit it differently.

  3. The atomic K=8 values combine three-seed K=1 means with single-checkpoint history increments, rather than reporting directly measured multi-seed endpoints. Although this construction is disclosed, presenting these values alongside directly measured means makes the comparison harder to interpret.

Questions for authors

see weaknesses

5 comments
AuthorsOpenReview ↗

Regarding Q1

We are grateful to you for the insightful and constructive feedback. In response, we have carefully considered each comment, conducted additional experiments, and revised the manuscript accordingly. We would be pleased to discuss these revisions in greater detail with you.

Q1: Broader transfer beyond the original scaffold

Response to Q1. Thank you for this valuable comment, which has been very helpful in improving the quality of our paper. We now clarify our relation to ThinkProprio in Related Work (L108--111). Beyond RoboCasa365, we added two complementary evaluations: LIBERO-Plus (L353--356, Table 3) and a real-robot study (Sec. 5.3, L449--466, Table 4). Both show that short history improves control: on LIBERO-Plus by +2.78 pp for AP and +2.38 pp for VP, and on the real robot from 59.0% to 79.0% for AP and from 60.0% to 72.0% for VP.

Additional results (paper Table 3): LIBERO-Plus success rate (percent)

\text{Policy} & \text{Spatial} & \text{Object} & \text{Goal} & \text{Long} & \text{Average} \\\\ \hline \text{AP1} & 85.3 & 84.9 & 77.4 & 76.3 & 81.0 \\\\ \text{AP8} & 90.0 & 88.2 & 79.2 & 77.6 & 83.8 \\\\ \text{VP1} & 87.2 & 85.8 & 79.5 & 76.8 & 82.3 \\\\ \text{VP8} & 88.6 & 91.4 & 81.8 & 77.1 & 84.7 \end{array}$$ **Additional results (paper Table 4): real-robot success rate (percent)** $$\begin{array}{lrrrrrr} \text{Configuration} & \text{Medication sorting} & \text{Handover} & \text{Can Spot} & \text{Bowls} & \text{Barcode} & \text{Average} \\\\ \hline \text{NS} & 30 & 70 & 100 & 70 & 10 & 56.0 \\\\ \text{SP1} & 20 & 70 & 100 & 80 & 10 & 56.0 (+0.0) \\\\ \text{AP1} & 35 & 80 & 100 & 60 & 20 & 59.0 (+3.0) \\\\ \text{AP8} & 75 & 80 & 100 & 100 & 40 & 79.0 (+23.0) \\\\ \text{VP1} & 50 & 60 & 100 & 80 & 10 & 60.0 (+4.0) \\\\ \text{VP8} & 60 & 80 & 100 & 80 & 40 & 72.0 (+16.0) \end{array}$$ **Changes in the manuscript:** We clarified the distinction from ThinkProprio in Related Work (L108--111) and added the LIBERO-Plus and real-robot evaluations in Table 3 (L353--356) and Sec. 5.3/Table 4 (L449--466).
AuthorsOpenReview ↗

Regarding Q2 and Q3

Q2: Interface-history interaction

**Response to Q2.**Thank you for highlighting this interaction. The EEF-pose result alone does not imply that history is ineffective for VP. In the joint-angle setting, VP improves from 33.9% to 35.5% with eight-frame history (+1.6 pp; Table~2). Consistent gains appear on LIBERO-Plus (82.3%$\rightarrow$84.7%, +2.38 pp), the real robot (60.0%$\rightarrow$72.0%), and the atomic depth sweep, which also shows a positive short-window effect.

Our probes provide a possible explanation for the route difference. VLM-side history changes the shared multimodal prefix, with attention redistribution increasing from 17.3% for VP1 to 22.0% for VP8, whereas action-side history directly conditions the action expert throughout denoising and increases state-conditioned correction alignment from 0.079 to 0.270 (task-bootstrap 95% interval for the increase: [+0.143, +0.239]). Although these probes do not isolate a single sufficient temporal statistic, they suggest that ordered recent kinematic changes are more directly usable on the action side.

Changes in the manuscript: We added the overall VP1--VP8 task-bootstrap check and clarified the route-level probe interpretation in Sec. 5.2.3 (L390--431) and Appendix F.2.

Q3: Derived atomic K=8K=8 values

Response to Q3. We agree that the derived atomic K=8K=8 values should be visually separated from directly measured means. In the revised Figure 4, they are now drawn as hatched bars and labeled as derived values (K=1K=1 three-seed mean plus single-checkpoint increment), and the routing conclusion continues to rest on the directly measured composite endpoints (Table 2).

Changes in the manuscript: We revised the Figure 4 caption and Appendix F.1 to label atomic K=8K=8 values as derived and to base the routing conclusion on directly measured composite endpoints.

We thank you again for the constructive feedback. We hope our revisions address the concerns, and we welcome any further questions or comments during the discussion.

Thank you for the additional experiments and clarifications. I appreciate the expanded evaluation on LIBERO-Plus and the real robot. However, my main concern regarding the scientific contribution remains. In particular, the LIBERO-Plus results show nearly identical history gains for AP and VP, with an interaction effect of only 0.40 pp. Thus, while the benefit of history transfers, the central claim that the preferred injection location changes with history length is not convincingly validated on this independent benchmark. The real-robot results are encouraging, but the comparison between VP and AP is still affected by differences in their native widths and computation paths. The proposed width-controlled experiment also remains incomplete. Overall, I appreciate the substantial additional effort, but I still view the contribution primarily as a useful empirical comparison rather than a sufficiently established general design principle.

Reviewer_T3ZmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares five ways of incorporating proprioceptive state into π0.5 on RoboCasa365, including both single-frame inputs and state histories. It finds that the benefits depend on the task and interface. On composite tasks, VLM-prefix conditioning performs better with a single frame, while action-prefix conditioning performs better with a short history. A control that repeats the current state across the same input slots suggests that the history gains come from temporal information.

Critical strengths
  1. The paper compares five state interfaces within the same π0.5 setup, keeping the backbone, data, and evaluation protocol largely fixed. This makes the comparison easier to interpret than results from different VLA systems. The VP–AP ranking reversal is particularly interesting, as it shows that the preferred interface can change when state history is available.

  2. The repeat-current baseline is a useful control. With the same number of state slots, real history performs substantially better than repeated current state, supporting the value of temporal information.

Critical weaknesses
  1. The experiments show that short state history helps, but offer limited insight into why. The repeat-current control rules out extra input slots as the sole explanation, without identifying which information in the history matters. The stage-transition example and model probes suggest possible explanations, but do not establish what drives the success-rate gains. I find the comparison useful, but it mainly tells us which configuration works better in this setup. It offers less insight into what temporal information the policy uses or why the preferred interface changes.

  2. It is unclear how well the VP–AP ranking reversal extends beyond the tested setup. All main experiments use π0.5 and RoboCasa365, and VP and AP differ in hidden width and computation path as well as injection location. The joint-angle results offer some support, but the eight-frame gap is much smaller. The real-robot experiments compare only no-state and AP8, so they do not test the reversal. The authors acknowledge these limitations. Even within the stated scope, I find the contribution too narrow to support acceptance.

Questions for authors
  1. Beyond the comparison of complete interfaces at their native widths, do the authors have evidence that the VP–AP ranking reversal persists when hidden width is controlled? This would help me judge whether the result reflects injection location or the particular interface implementations.

  2. Is the VP–AP ranking reversal seen across most composite tasks, or driven by a few tasks? Please provide per-task VP–AP comparisons at both history lengths, with uncertainty estimates where available.

4 comments
AuthorsOpenReview ↗

Regarding W1

We sincerely thank the reviewer for the careful and detailed review and for the constructive suggestions. Based on your comments, we have made several revisions and added new experiments to the paper. We hope these additions will be helpful as you reconsider our manuscript.

W1: Understanding the role of temporal information

Response to W1 To better characterize the role of temporal information, we conducted a real-robot barcode-scanning case study; details are provided in the revised manuscript. The barcode case study gives a concrete view of where history helps. The task has two stages separated by a pause. Single-frame policies fail at these boundaries: AP1 picks up the scanner and then puts it down again, while SP1 stalls after scanning and does not return it. With eight frames, AP8 completes both stages, and VP8 more often resumes after the pause and returns the scanner. This suggests that recent ordered state changes help disambiguate stage transitions, while the case study remains descriptive rather than causal.

Changes in the manuscript: We expanded Sec. 5.2.3 (L390--431), added the barcode case study in Sec. 5.3 (Fig. 6; L449--466), and added its detailed per-stage counts in Appendix H (Table 7).

AuthorsOpenReview ↗

Regarding Q1 and Q2

Q1: Native width and injection location

Response to Q1. This is a very insightful question. We have implemented a prefix-visible 1024-wide VP variant, VP@1024, and are currently running VP@1024-1 and VP@1024-8. We will report the measurements once the runs finish.

And our current design rationale is that, in a pretrained VLA, the choice of where state enters also determines the host width, computation path, and training and inference cost: VLM-side tokens pass through the 2048-wide backbone, whereas action-side tokens pass through the 1024-wide action expert at every denoising step.

Q2: Task-level consistency

Response to Q2. Appendix J, Table 10 now reports VP1, VP8, AP1, and AP8 for all 20 composite tasks on seed42. VP leads AP on 13/20 tasks at K=1K=1; AP leads VP on 11/20 at K=8K=8; and the VP--AP difference shifts toward AP on 15/20 tasks, with six strict sign reversals. These one-seed, 25-episode-per-task estimates are descriptive, so uncertainty is reported for the paired aggregate comparisons in Table 2. We also revised Figure 4 (hatched derived atomic K=8K=8 bars and caption) so the routing conclusion relies only on directly measured composite endpoints.

Additional results (paper Table 10): per-task composite success rate (percent)

\text{Task} & \text{AP1} & \text{AP8} & \text{VP1} & \text{VP8} & \Delta_{K=1} & \Delta_{K=8} \\\\ \hline \text{AddMarshmallow} & 12 & 36 & 28 & 32 & +16 & -4 \\\\ \text{ChooseMeasuringCup} & 20 & 24 & 24 & 20 & +4 & -4 \\\\ \text{ClearCuttingBoard} & 44 & 64 & 60 & 24 & +16 & -40 \\\\ \text{KettleBoiling} & 8 & 32 & 20 & 16 & +12 & -16 \\\\ \text{LoadDishwasher} & 40 & 48 & 64 & 56 & +24 & +8 \\\\ \text{OrganizeCleaningSupplies} & 28 & 44 & 8 & 24 & -20 & -20 \\\\ \text{OrganizeCondiments} & 20 & 28 & 20 & 16 & 0 & -12 \\\\ \text{PreSoakPan} & 20 & 20 & 44 & 24 & +24 & +4 \\\\ \text{PrepForSanitizing} & 28 & 68 & 48 & 40 & +20 & -28 \\\\ \text{PrepareCoffee} & 4 & 4 & 12 & 4 & +8 & 0 \\\\ \text{PrepareToast} & 48 & 48 & 28 & 64 & -20 & +16 \\\\ \text{RecycleStackedYogurt} & 20 & 32 & 40 & 36 & +20 & +4 \\\\ \text{RinseSinkBasin} & 32 & 48 & 24 & 24 & -8 & -24 \\\\ \text{ScrubCuttingBoard} & 56 & 52 & 40 & 56 & -16 & +4 \\\\ \text{SetUpCuttingStation} & 20 & 20 & 8 & 16 & -12 & -4 \\\\ \text{StackBowlsCabinet} & 32 & 48 & 52 & 64 & +20 & +16 \\\\ \text{StrainerSetup} & 60 & 68 & 60 & 56 & 0 & -12 \\\\ \text{TurnOffSimmeredSauceHeat} & 8 & 4 & 16 & 28 & +8 & +24 \\\\ \text{VeggieDipPrep} & 32 & 64 & 56 & 48 & +24 & -16 \\\\ \text{WashLettuce} & 32 & 28 & 36 & 28 & +4 & 0 \\\\ \text{Macro} & 28.2 & 39.0 & 34.4 & 33.8 & +6.2 & -5.2 \end{array}$$ Here, $\Delta_{K=1}$ and $\Delta_{K=8}$ denote VP minus AP at the corresponding history length. **Changes in the manuscript:** We added per-task VP/AP results in Appendix J (Table 10) . Thank you again for your thoughtful suggestions, which helped us clarify these design choices. We would be happy to continue the discussion, and please feel free to raise any remaining questions or concerns.

Thank you for the additional experiments. I appreciate the authors' efforts, but my main concern remains unresolved. The LIBERO-Plus results show nearly identical history gains for AP and VP (2.78 vs. 2.38 pp), providing limited evidence for the proposed routing crossover. The real-robot comparison is also confounded by differences in hidden width and computation paths, while the width-controlled experiment remains unavailable. Overall, the revisions strengthen the empirical evaluation, but do not sufficiently establish the claimed generalizable design principles. I therefore maintain my original rating.

AuthorsOpenReview ↗

Follow-up: width-matched VP (VP@1024)

Response (follow-up). Thank you for the continued discussion. As promised, we built a width-matched variant of VP, VP@1024, to separate injection location from hidden width. VP@1024 keeps VP's attention pattern: the state tokens join the image–language prefix block, so every image and language token can attend to them, as in VP. The only change is where the state is processed. Instead of the 2048-wide VLM backbone, the state tokens pass through a 1024-wide stream with the same geometry as the action expert.

Success rate (%) on 500 paired composite scenes

\text{Interface} & K=1 & K=8 & \text{history gain (pp) [95\% CI]} \\\\ \hline \text{AP} & 28.8 & 37.2 & +8.4\ [+3.8, +13.2] \\\\ \text{VP} & 28.4 & 32.2 & +3.8\ [-0.4, +8.0] \\\\ \text{VP@1024} & 26.6 & 27.6 & +1.0\ [-3.2, +5.2] \end{array}$$ **On your concern.** We appreciate this point. In the native interfaces, injection location changes together with hidden width and computation path, so it is reasonable to ask which of them drives the observed difference. Separating these factors is important for turning an empirical finding into a reliable design guideline, and it directly motivated the width-matched VP@1024 experiment above. **We would value your advice.** We are sincerely grateful for the time and care you have put into them. For the community, we hope this line of work to lead to be a stronger and more reliable design principle because we think that how proprioceptive state should enter a VLA is a truly important and fundamental question. **If you have any suggestions on how we could further strengthen it, we would be very grateful to hear them.**
Reviewer_tGuuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how proprioceptive state should be incorporated into Vision-Language-Action (VLA) models, focusing on state representation, history length, and injection location. Using a unified π0.5 framework, the authors compare five state interfaces on 45 atomic tasks and 20 composite tasks from RoboCasa365. They also investigate state history lengths ranging from 1 to 96 frames and conduct two real-robot experiments. The results suggest that the benefits of proprioceptive state depend on task characteristics, short state histories can improve control performance, and the preferred injection location may change with history length.

Critical strengths

Important and practically relevant research question. The paper investigates how proprioceptive state should be incorporated into VLA models, addressing an important yet relatively underexplored design question. Understanding the effects of state representation, temporal information, and injection location could provide useful guidance for future VLA architectures.

Critical weaknesses
  1. Insufficient validation of generalizable design principles. The paper investigates a general question about how proprioceptive state should be incorporated into VLA models. However, all the main controlled experiments are conducted using π0.5 on RoboCasa365. Although the evaluation covers multiple tasks, experiments within a single benchmark may not be sufficient to establish generalizable design principles. In particular, the routing crossover between VLM-side and action-side state injection is one of the paper's central findings, and validation on additional independent benchmarks would significantly strengthen this conclusion. Furthermore, the real-robot experiments only compare AP8 against a no-state baseline on two closely related medication-sorting tasks, with limited trials and no statistically significant differences. They therefore do not directly validate the benefits of state history or the routing crossover in real-world settings.
  2. Limited investigation of proprioceptive state representations. The paper primarily considers position- and pose-based proprioceptive states, while velocity is another important component of robot proprioception. A more comprehensive investigation of state representations would strengthen the study. In particular, the reported benefits of state history may partly arise from the model implicitly estimating velocity or other local motion information from consecutive states. It would therefore be valuable to compare current-state conditioning, current state with explicit velocity information, and multi-frame state history. Such a comparison could help clarify whether the improvements require a full history window or can be achieved using a more compact representation of the robot's motion state.
Questions for authors
  1. Generalizability across benchmarks. Is there additional evidence that the observed routing crossover between VLM-side and action-side state injection holds across different benchmarks or task distributions? A focused comparison of VP1, VP8, AP1, and AP8 on an independent benchmark would be particularly informative.
  2. Role of velocity information. Have the authors investigated using explicit velocity information alongside the current proprioceptive state? How does such a representation compare with multi-frame state history, and to what extent could the reported history benefits be explained by implicit velocity estimation?
  3. Real-world validation. Are there additional real-robot results comparing different state history lengths or injection interfaces, such as AP1 versus AP8 or VP8 versus AP8? Such comparisons would help determine whether the main findings from simulation also hold in real-world manipulation.
3 comments
AuthorsOpenReview ↗

Regarding Q1 and Q3

We sincerely thank you for the thoughtful review and for recognizing the importance of this design question. Your suggestions were very helpful: the independent benchmark, explicit-velocity experiment, and expanded real-robot interface comparisons have all been added to the revision. We also value the opportunity to address your inquiries and engage in this discussion!

Q1 and Q3: Generalization beyond RoboCasa365 and additional real-robot validation

Response to Q1 and Q3. We added two independent validations. On LIBERO-Plus (10,030 paired episodes; L353--356, Table 3), history improves both AP (+2.78 pp) and VP (+2.38 pp). VP8 is slightly ahead of AP8 overall (84.7% vs. 83.8%), while the crossover appears on the Spatial and Long suites. We also extended the real-robot study to six configurations (NS, SP1, AP1, AP8, VP1, VP8) on five tasks and 360 trials (Sec. 5.3, Table 4, App. H). VP1 and AP1 are close at K=1K=1 (60.0% vs. 59.0%), while AP8 leads VP8 at K=8K=8 (79.0% vs. 72.0%). A barcode-scanning case study further shows where history is useful at stage transitions. We revised the conclusion to present action-side injection as a reasonable default when history is used.

Additional results (paper Table 3): LIBERO-Plus success rate (percent)

\text{Policy} & \text{Spatial} & \text{Object} & \text{Goal} & \text{Long} & \text{Average} \\\\ \hline \text{AP1} & 85.3 & 84.9 & 77.4 & 76.3 & 81.0 \\\\ \text{AP8} & 90.0 & 88.2 & 79.2 & 77.6 & 83.8 \\\\ \text{VP1} & 87.2 & 85.8 & 79.5 & 76.8 & 82.3 \\\\ \text{VP8} & 88.6 & 91.4 & 81.8 & 77.1 & 84.7 \end{array}$$ **Additional results (paper Table 4): real-robot success rate (percent)** $$\begin{array}{lrrrrrr} \text{Configuration} & \text{Medication sorting} & \text{Handover} & \text{Can Spot} & \text{Bowls} & \text{Barcode} & \text{Average} \\\\ \hline \text{NS} & 30 & 70 & 100 & 70 & 10 & 56.0 \\\\ \text{SP1} & 20 & 70 & 100 & 80 & 10 & 56.0 (+0.0) \\\\ \text{AP1} & 35 & 80 & 100 & 60 & 20 & 59.0 (+3.0) \\\\ \text{AP8} & 75 & 80 & 100 & 100 & 40 & 79.0 (+23.0) \\\\ \text{VP1} & 50 & 60 & 100 & 80 & 10 & 60.0 (+4.0) \\\\ \text{VP8} & 60 & 80 & 100 & 80 & 40 & 72.0 (+16.0) \end{array}$$ **Additional results (paper Table 7): barcode stage breakdown (percent)** $$\begin{array}{lrrrrrr} \text{Stage} & \text{NS} & \text{SP1} & \text{AP1} & \text{AP8} & \text{VP1} & \text{VP8} \\\\ \hline \text{Stage 1: pick up and scan} & 30 & 60 & 20 & 90 & 80 & 70 \\\\ \text{Stage 2 given stage 1} & 33\\,(1/3) & 17\\,(1/6) & 100\\,(2/2) & 44\\,(4/9) & 13\\,(1/8) & 57\\,(4/7) \\\\ \text{Full task} & 10 & 10 & 20 & 40 & 10 & 40 \end{array}$$ **Changes in the manuscript:** We revised Sec. 5.2.2 (L353--356, Table 3), Sec. 5.3 (L449--466, Table 4), and the Conclusion (L512--515), added the six-configuration protocol and barcode stage analysis in Appendix H, and included supporting details in Appendices I and H.
AuthorsOpenReview ↗

Regarding Q2

Q2: Role of explicit velocity

Response to Q2. We added an explicit-velocity variant of VP (App. K): a single state token holds the current state plus a 14-D motion vector (end-effector, base, and gripper rates), computed either from two frames (local2) or as a trend over eight frames (trend8). We are currently running the corresponding AP-side experiment and will report its results in a later version.

Additional results (paper Table 11): explicit-velocity VP variants

\text{Task} & \text{local2} & \text{trend8} & \text{Task} & \text{local2} & \text{trend8} \\\\ \hline \text{AddMarshmallow} & 32 & 32 & \text{PrepareToast} & 44 & 60 \\\\ \text{ChooseMeasuringCup} & 16 & 24 & \text{RecycleStackedYogurt} & 48 & 40 \\\\ \text{ClearCuttingBoard} & 36 & 56 & \text{RinseSinkBasin} & 32 & 32 \\\\ \text{KettleBoiling} & 32 & 28 & \text{ScrubCuttingBoard} & 48 & 56 \\\\ \text{LoadDishwasher} & 40 & 68 & \text{SetUpCuttingStation} & 24 & 16 \\\\ \text{OrganizeCleaningSupplies} & 36 & 44 & \text{StackBowlsCabinet} & 48 & 60 \\\\ \text{OrganizeCondiments} & 40 & 32 & \text{StrainerSetup} & 68 & 52 \\\\ \text{PreSoakPan} & 32 & 36 & \text{TurnOffSimmeredSauceHeat} & 12 & 4 \\\\ \text{PrepForSanitizing} & 48 & 40 & \text{VeggieDipPrep} & 72 & 44 \\\\ \text{PrepareCoffee} & 4 & 8 & \text{WashLettuce} & 24 & 36 \\\\ \text{Mean over 20 tasks} & 36.8 & 38.4 & & & \end{array}$$ **Changes in the manuscript:** We added the explicit-velocity experiment and its per-task results in Appendix K (Table 11, L1350--1403) and clarified the role of state history. Thank you again for these constructive suggestions. If you have any further questions or concerns, we would be glad to discuss them.
AuthorsOpenReview ↗

Follow-up on Q2

We thank you again for this question. It led us to an experiment that we found informative. As promised in our earlier response, we have now completed the AP-side counterpart of the explicit-velocity study.

Q2 (follow-up): AP-side explicit velocity

Response to Q2 (follow-up). For explicit-velocity experiment on the action side (AP), everything else is the same as for the VP variants in our earlier response: the same state token (current state plus a 14-D motion vector), the same two ways of computing velocity (local2 from the last two frames, trend8 from an eight-frame trend), the same training budget (3.84M samples, seed 42), and the same 500 paired composite scenes.

Success rate (%) on composite tasks

\text{Route} & \text{current state (K=1)} & \text{+ local2 velocity} & \text{+ trend8 velocity} \\\\ \hline \text{AP} & 28.2 & 38.2 & 38.8 \\\\ \text{VP} & 34.4 & 36.8 & 38.4 \end{array}$$ The current-state column is the seed-42 checkpoint. We see two things: 1. **Velocity information helps.** Adding velocity to the current state improves both routes. AP rises from 28.2% to 38.2% / 38.8% (about +10 points), and VP rises from 34.4% to 36.8% / 38.4% (+2.4 / +4.0 points). 2. **Velocity computed over a short history helps a bit more.** On both routes, the eight-frame trend (trend8) is better than the two-frame estimate (local2): AP 38.8% vs. 38.2% (+0.6 pp, 95% CI [−4.0, +5.2]), VP 38.4% vs. 36.8% (+1.6 pp, [−3.2, +6.4]). These are single-seed results, and the intervals only reflect variation across scenes. **Per-task results of the AP variants (%, 25 scenes per task)** $$\begin{array}{lrr|lrr} \text{Task} & \text{local2} & \text{trend8} & \text{Task} & \text{local2} & \text{trend8} \\\\ \hline \text{AddMarshmallow} & 44 & 40 & \text{PrepareToast} & 40 & 60 \\\\ \text{ChooseMeasuringCup} & 32 & 24 & \text{RecycleStackedYogurt} & 24 & 32 \\\\ \text{ClearCuttingBoard} & 40 & 48 & \text{RinseSinkBasin} & 36 & 40 \\\\ \text{KettleBoiling} & 24 & 28 & \text{ScrubCuttingBoard} & 60 & 40 \\\\ \text{LoadDishwasher} & 64 & 48 & \text{SetUpCuttingStation} & 20 & 56 \\\\ \text{OrganizeCleaningSupplies} & 32 & 32 & \text{StackBowlsCabinet} & 72 & 56 \\\\ \text{OrganizeCondiments} & 32 & 16 & \text{StrainerSetup} & 56 & 60 \\\\ \text{PreSoakPan} & 40 & 32 & \text{TurnOffSimmeredSauceHeat} & 8 & 16 \\\\ \text{PrepForSanitizing} & 36 & 44 & \text{VeggieDipPrep} & 56 & 68 \\\\ \text{PrepareCoffee} & 8 & 8 & \text{WashLettuce} & 40 & 28 \\\\ \text{Mean over 20 tasks} & 38.2 & 38.8 & & & \end{array}$$ We think this is a very good suggestion. In these preliminary experiments, explicit velocity gives a large improvemen. Within the short rebuttal period, we did our best to complete these experiments, and we have placed the preliminary results in the appendix. We will continue to explore velocity information in our future work. If you have any further questions, we would be glad to discuss them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.