acceptodds
Under review as a conference paper at ICLR 2027

Do You Need Proprioceptive States for Spatial Generalization in Visuomotor Policies?

Abstract

Visuomotor policies commonly combine images with proprioceptive states, yet such states can limit transfer to unseen spatial layouts. We empirically investigate State-free Policies, which remove proprioceptive state inputs, in the setting of spatial generalization from demonstrations collected under constrained spatial layouts. We study how state-input choices, action representations, and task-relevant visual coverage affect generalization. We identify that State-free Policies consistently outperforms the evaluated state-based alternatives, suggesting that reliance on state cues associated with training layouts can limit spatial generalization. This benefit depends on action representation, with relative end-effector (EEF) actions emerging as a key enabling condition. With this action representation, sufficient visual coverage for capturing task-relevant regions further supports State-free Policies' spatial generalization. Under these conditions, State-free Policies shows an increase in average success from 0% to 86% for height generalization and from 6% to 64% for horizontal generalization, compared with matched state-based policies. These findings characterize when state removal supports spatial generalization from demonstrations with constrained spatial coverage, and how action representation and visual coverage shape this benefit. Real-world robot videos are in supplementary materials.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot manipulation, Imitation learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MH42OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This study empirically investigates the spatial generalization of visuomotor policies that combine images and proprioceptive states. The authors find that State-free policies consistently outperform the State-based policies in both real-world and simulation environments. They suggest that such benefits are derived from relative end-effector(EEF) action representation and visual coverage.

Critical strengths
  • The paper studies an important and practical question of visuomotor policies: whether proprioceptive state helps or hurts spatial generalization of visuomotor policies. The height and horizontal shift experiments are controlled and easy to interpret, and they isolate a real failure mode of absolute state under spatial distribution shift.

  • The evaluation spans real-world tasks, simulations, ablations, and several policy architectures, with a substantial number of real-world trials. Comparisons with prior methods are also well-constructed.

Critical weaknesses
  • The title overclaims the scope of the work. Even though the paper narrows the claim to spatial generalization, it does not remove the risk of misunderstanding. Proprioception can still be useful when vision changes substantially, for example under a large background-color shift.

  • Figure 7 shows that at a data scale of n=300 the effect of state on policy performance is no longer clear, and the gap between state-free and state-based policies essentially disappears. The claim of spatial-generalization is therefore supported mainly in the small-data regime. Given the background-change case above, removing state at large scale may limit robustness rather than improve it.

Questions for authors
  • For tasks with partial occlusion or background-color shift, will the generalization of State-free policies remain strong, since no visual coverage could be reliably guaranteed? And how about State-based policies?
3 comments
AuthorsOpenReview ↗

Response to Reviewer MH42

Thank you for recognizing the practical importance of our question, the controlled experimental design, and the breadth of our evaluations and baseline comparisons. Regarding your concerns about the title scope, data scale, and visual robustness, we respond point by point below.

  • Weakness 1: The title may overclaim. We clarified the intended scope of our conclusions and highlighted supporting evidence from existing experiments.

    Thank you for your thoughtful consideration of the central question of our study. We agree that proprioception can be useful when visual conditions change within an in-domain spatial layout. We appreciate the opportunity to clarify our intended scope. Our title asks whether proprioceptive inputs are needed for spatial generalization, and our answer is conditional. The relevant conditions are relative EEF actions and sufficient task-relevant visual information, as examined in Sections 3.2-3.3 and Tables 4-5. Supporting results are reported in Figure 5, with further comparisons against alternative state representations and training strategies in Table 3.

    We understand that your concern about the title may be related to the points raised in Weakness 2 and Question 1, which we address in detail below. Figure 7 reports in-domain rather than spatial-generalization performance, and State-based Policies still failed when occlusion was combined with unseen layouts. These points support our conditional answer.

  • Weakness 2: Concern of Figure 7. We clarified the interpretation of Figure 7 and explained its relationship to the existing spatial generalization results.

    We clarify that Figure 7 reports the In-Domain performance, rather than generalization performance, as the number of demonstrations increases to 300 episodes. It shows that State-based Policies need more data to avoid overfitting even in the in-domain setting, as discussed in Appendix G. It also identifies 300 episodes as the scale at which both policies achieve 100% in-domain success.

    Our main experiments use this same data scale of 300 episodes in Pick & Place tasks, as specified in Appendix A, yet show a substantial spatial generalization gap in Figure 5. Thus, the spatial-generalization advantage persists even when both policies achieve equally perfect in-domain performance.

  • Question 1: What happens under occlusion or background-color shifts? We provided further analyses of real-robot observations and conducted additional simulation experiments on visual robustness.

    Occlusion. In our real-robot observations, occlusion adversely affected State-free Policies in both in-domain and unseen spatial layouts, with qualitatively similar effects. This is consistent with the need for task-relevant visual information discussed in Section 3.3 and supported by Table 5: both occlusion and reduced camera coverage can limit access to this information. For State-based Policies, however, the effect depended on the spatial layout. Under occlusion in in-domain layouts, they performed better than State-free Policies, with behavior consistent with following trajectories learned during training. However, they also failed when occlusion was combined with unseen layouts. The robot's own state does not directly reveal the target's new location. Thus, retaining state can support execution under occlusion in familiar layouts, but this benefit did not extend to the spatial-generalization settings we tested.

    Background color. The moderate lighting changes we tested had only a limited impact on both State-free and State-based Policies. These observations suggest that the tested lighting variations were less disruptive than occlusions that obscured task-relevant visual information.

    At the same time, during this discussion period, we added LIBERO-10 simulation experiments to quantitatively evaluate the effects of occlusion and background-color changes. We have included these results in Section R1.1 on page 10 of the revised manuscript. The results are similar to our real-robot observations described above.

Thank you again for your feedback. We hope these clarifications address your concerns and encourage you to reconsider the overall contribution of this paper. We will update the manuscript again after the discussion, incorporating all of the feedback received.

The authors' response has addressed my concerns regarding overclaim and visual disturbance. I will therefore raise my score to Weak acceptance.

AuthorsOpenReview ↗

Thank Reviewer MH42 for raising the rating

Thank you for your positive feedback and for raising your rating. We appreciate your thoughtful comments and are glad that our clarifications and additional experiments have addressed your concerns.

Reviewer_EC52OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies spatial generalization when visuomotor policies are trained on demonstrations with restricted spatial layouts. Removing proprioceptive policy inputs improves performance across five real-robot tasks while preserving in-distribution success. Further experiments investigate action representations, camera coverage, alternative state inputs, different architectures, and cross-embodiment adaptation.

Critical strengths
  1. The substantial improvements across five real-robot tasks are practically relevant.
  2. Matched camera comparisons and the state-correction diagnostic provide useful experimental controls.
Critical weaknesses
  1. Limited differentiation from prior findings. GAP already reports benefits from removing proprioception under spatial shifts, while UMI combines relative actions with wide wrist-camera coverage. The paper adds useful experiments, but should clarify which findings substantially advance our understanding beyond these observations.
  2. Coordinate mismatch remains insufficiently separated from state removal. In Table 3, correcting state coordinates achieves 96.7%, close to the state-free policy’s 98.3%. Comparisons restricted to Pick Pen height shifts, with limited state augmentation and a PALM setting outside its stated planar scope, do not fully establish benefits beyond coordinate alignment.
Questions for authors

Additional Limitations 3. Mechanistic ablations concentrate on one task. The key action-representation and camera-configuration experiments are largely conducted on Pick Pen. It remains unclear whether the same factors explain improvements across the other tasks. 4. Reproduction details remain incomplete. The manuscript does not fully specify the checkpoints, optimization settings, action chunks, normalization, and controller configuration. More explicit descriptions of the UMI-style and PALM adaptations would also help readers assess the comparisons. 5. Limited uncertainty across training runs. Three training seeds are reported for Pick Pen height generalization, but comparable results are absent for other tasks and most ablations. The consistency of the broader findings across training randomness therefore remains unclear. 6. Cross-embodiment adaptation contains a coordinate confound. The adaptation experiment changes from table-frame to robot-frame state coordinates. A coordinate-aligned state baseline is needed to distinguish the benefit of removing state from avoiding this mismatch. 7. The supplementary simulation evidence is narrow. The RoboCasa evaluation covers six selected tasks, with insufficient detail about task selection, spatial splits, and evaluation counts. This limits reproducibility and the strength of broader simulation-generalization conclusions.

Questions For Authors

  1. What specific new conclusion beyond GAP and UMI is established by these experiments?
  2. Does the advantage persist with aligned coordinates, selectively retained state, and augmentation covering the test shifts?
  3. Can the authors provide complete reproduction settings and broader training-seed results?
6 comments
AuthorsOpenReview ↗

Response to Reviewer EC52, Part 1

Thank you for recognizing the practical relevance of our real-robot experiments and the value of our matched camera comparisons and state-correction diagnostic. Regarding your concerns about novelty, coordinate effects, experimental coverage and reliability, and reproducibility, we respond point by point below.

  • Weakness 1 & Question 1: Contributions beyond GAP and UMI. We further explained our contributions relative to prior work and highlighted supporting evidence from existing experiments.

    Thank you for encouraging us to clarify our contribution beyond prior work. As Reviewer MH42 notes, our controlled spatial generalization experiments "isolate a real failure mode of absolute state under spatial distribution shift"; below, we explain in detail the specific insights our study.

    We agree with GAP's findings on the limitations of state inputs and UMI's contribution to relative state representations. Our focus is to systematically investigate, through controlled experiments, the conditions needed for State-free Policies to work well in practical spatial-generalization settings. Specifically, spatial generalization is jointly influenced by the use of state inputs, visual information, and action representations.

    GAP shows that simply removing state yields suboptimal performance. Building on this observation, we investigate these conditions systematically and in depth. Table 5 shows that sufficient visual coverage is an important factor for State-free Policies to work well in spatial-generalization settings. The comparison with GAP in Table 3 further highlights the value of understanding the conditions under which State-free Policies work well. Furthermore, Table 3 shows a generalization gap between UMI-style relative state and State-free Policies. This indicates that reducing absolute-coordinate information alone does not eliminate the gap, consistent with continued reliance on correlations associated with training trajectories. We hope these clarifications address your concerns about novelty.

  • Weakness 2 & Question 2: Coordinate alignment and state removal. We provided further explanations of the experimental design and existing comparisons, and conducted additional experiments on coordinate correction and state augmentation.

    (1) State correction. The state-hacked policy (96.7% success) uses the known difference between training and test table heights to correct its state input at evaluation. However, this hack requires prior knowledge of the test shift an therefore does not demonstrate the spatial generalization. Its recovery in Table 3 and the sensitivity analysis in Section 4.3 indicate reliance on state coordinates associated with the training layout. State-free Policies achieve comparable success without this correction.

    At the same time, during this discussion period, we added additional LIBERO-10 simulation experiments to evaluate the coordinate correction. We have included these results in Section R1 on page 10 of the revised manuscript. Here, the State-free Policies still outperformed the oracle-corrected policy. The tested translation correction narrowed the gap but left an advantage w/o state (see Figure R1(a) in Section R1 on page 10 of the revised manuscript).

    (2) Choice of height shifts. As explained in Section 4.3, height changes provide controlled spatial perturbations. They allow us to quantify the shift and systematically vary state correction and noise augmentation along a single positional state dimension. Compared with our two-dimensional horizontal evaluations, this provides a simpler setting for investigating state sensitivity.

    (3) State augmentation. Our original [−5, 5] cm augmentation examines whether a State-based Policy can extrapolate to height shifts beyond its augmentation range. This is an important setting because the magnitude of shifts in unseen layouts may not be known in advance. As shown in Table 3, this augmentation improves success to 63.3% but does not close the gap to the State-free Policy’s 98.3%.

    To directly assess augmentation that covers the test shifts, we additionally trained a State-based Policy with uniform noise in [−10, 10] cm applied to the state’s z coordinate. Its success rate increased to 91.7%, compared with 63.3% under the original [−5, 5] cm augmentation. The wider noise range spans much of the task-relevant vertical workspace, making the z coordinate an unreliable cue during training. The resulting improvement is consistent with our finding that reducing reliance on layout-specific state coordinates supports spatial generalization.

--- Response to Weakness 2 and Question 2 continues in Part 2. ---

AuthorsOpenReview ↗

Response to Reviewer EC52, Part 2

  • Weakness 2 & Question 2 (continued).

    (4) PALM. As explained in Section 4.3, we include it without additional mechanism to investigate whether aligned state that retains height transfers to unseen table heights. This examines transfer beyond its original planar setting. Additionally, PALM uses camera intrinsics and extrinsics for visual and proprioceptive alignment. In contrast, State-free Policies offer a more direct alternative by avoiding this explicit alignment step.

    (5) Selectively retained state. Table 3 includes two relevant state-retaining alternatives. The UMI-style Relative State baseline removes absolute EEF-pose coordinates while retaining relative pose history. GAP uses estimated motion-transition phases to modulate gradients in the proprioceptive input branch during fine-tuning. State-free Policies outperform both alternatives in comparisons.

  • Weakness 3: Scope of mechanistic ablations. We explained the selection of representative tasks, highlighted supporting evidence from existing experiments, and conducted additional real-robot ablation experiments.

    We chose Pick Pen as a representative pick-and-place task, as it is a common category of robotic manipulation. It requires visual access to the pen and pen holder at the relevant stages, which makes it suitable for a fine-grained analysis of visual coverage. And Figure 5 also extends the camera comparisons to other tasks.

    For action representations, in this discussion period, we additionally evaluated relative joint actions on Fold Shirt under the same spatial-generalization setting as Table 2. The relative-joint policy achieved 0% success, compared with 83.3% for the relative-EEF policy reported in Table 2.

    Table 4 uses the representative task Pick Pen to show that all other representations fail in generalization evaluation. And in Figure 5 and Table 2, the relative EEF representation also shows the generalization ability in our main experiments with different tasks, supporting its broader applicability. Given the cost of extensive real-robot ablations and the limited rebuttal period, we will extend selected ablations to more tasks in the camera-ready revision.

  • Weakness 4 & Question 3: Reproduction details. We provided additional implementation details.

    We provide the implementation details below. (1) Checkpoints and optimization: We use checkpoints trained for 20k steps, or 60k steps for Fold Shirt. Training uses 8 NVIDIA A100 GPUs, a global batch size of 16, and a learning rate decaying from 2.5e-5 to 2.5e-6. (3) Action chunks: We use 30 FPS data with a chunk size of 60 steps (2 seconds). (4) Normalization: We apply min-max normalization. (5) Controller configuration: Control commands are interpolated from 60 Hz to 120 Hz.

    For UMI-style, we use two-frame state input with a temporal interval of 66.7 ms. Relative positions are computed by directly subtracting XYZ coordinates, while relative rotations are computed after converting the rotation vectors into rotation matrices.

    For PALM, we use our camera's calibrated extrinsics (translation: [0, -0.07214, 0.14402] m; rotation: 22° around the x-axis) to transform EEF poses from the torso frame to the camera frame, while retaining gripper states.

AuthorsOpenReview ↗

Response to Reviewer EC52, Part 3

  • Weakness 5 & Question 3: Variability across training runs. We highlighted supporting evidence from existing multi-seed experiments and conducted additional training-seed validation.

    Our existing seed study directly examines the central comparison between State-based and State-free Policies. As reported in Appendix B, we independently train both settings with three random seeds on a representative task. These results support the reproducibility of the observed gap on this task. Beyond this seed study, the same direction of improvement appears across five real-robot tasks in Figure 5 and Table 2 and four policy architectures in Table 10. Thus, our main finding is supported by both repeated training on the representative task and consistent improvements across varied tasks and architectures.

    In this discussion period, we have additionally trained the relative-joint-action policy (a key analysis experiment) with a new random seed. It again achieved 0% success on Pick Pen height generalization, consistent with the result in Table 4. This provides an additional check on the relative-joint-action ablation.

    We agree that broader seed coverage would strengthen this evidence. Given the limited rebuttal period and the cost of training and real-robot evaluation, we plan to add multi-seed results for key ablations in the camera-ready revision.

  • Weakness 6: Coordinate Confound in Cross-Embodiment Adaptation. We provided additional details on coordinate alignment across embodiments and further explained the remaining challenges of cross-embodiment adaptation.

    We align the coordinate axes across embodiments using a common right-handed convention (x: left, y: backward, z: upward, relative to the overhead camera), as well as the zero-rotation conventions. But the coordinate origins remain different. This is a common phenomenon in cross-embodiment data, which is precisely the challenge that State-based Policies need to address for effective cross-embodiment adaptation and State-free Policies naturally avoid.

    Although the coordinate origins differ, aligning them alone would not eliminate the cross-embodiment state shift: differences in robot kinematics and reachable workspaces can still lead to substantial differences in the distributions of feasible EEF poses. State-based Policies must adapt to these shifts in their proprioceptive inputs, whereas State-free Policies avoid explicit dependence on EEF-pose coordinates.

  • Weakness 7 & Question 3: Reproducibility of Simulation Evaluations. We provided additional details on simulation settings and evaluation protocols, and conducted new spatial generalization experiments in simulation.

    We use the π0.5 backbone, trained on all RoboCasa training tasks for 20k steps using 8 NVIDIA A100 GPUs. We select six tasks from spatial splits of the official RoboCasa spatial generalization evaluation set by choosing one or two representative tasks from each group of tasks with similar action patterns: OpenDrawer, OpenStandMixerHead, CloseFridge, NavigateKitchen, CloseToasterOvenDoor, and TurnOnSinkFaucet. To support real-time execution, we implement RTC in simulation with a chunk size of 50, executing 20 timesteps per chunk and simulating a 100 ms inference delay (2 timesteps). Each task is evaluated over 50 trials with random seed 42.

    At the same time, during this discussion period, we added additional LIBERO-10 simulation experiments to evaluate policy generalization under progressively increasing spatial shifts. We have included these results in Section R1 on page 10 of the revised manuscript. We used 30 matched initial scenes per task, resulting in 300 paired rollouts per condition. Both π0 variants (state-free and state-based) were fine-tuned for 30,000 steps with a batch size of 32.

Thank you again for your feedback. We hope these clarifications address your concerns and encourage you to reconsider the overall contribution of this paper. We will update the manuscript again after the discussion, incorporating all of the feedback received.

Reviewer_FteLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies whether removing explicit proprioceptive inputs improves spatial generalization in visuomotor policies trained on demonstrations with limited spatial coverage. It finds substantial benefits from state removal, particularly with relative end-effector actions and sufficient task-relevant visual coverage. Simulation and real-world experiments support these findings, with large improvements in real-world height and horizontal generalization.

Critical strengths
  • The paper presents a simple, practically useful finding, supported by real-world experiments across several tasks and policy architectures.
Critical weaknesses
  • The generalization range is relatively narrow. Real-world height evaluation uses two unseen heights, while most horizontal evaluations use shifts of 5 cm and 10 cm. These demonstrate transfer beyond the training layouts, but do not establish how performance changes with increasing displacement or combined height and horizontal shifts.
  • The state-offset intervention supports sensitivity to training coordinates, but the main state-representation and action-space comparisons are restricted to Pick Pen height generalization. The conclusion that relative EEF actions are a key enabling condition would be more convincing across additional tasks and shifts.
  • The evaluated state input is a world-frame EEF pose, rather than a broader set of proprioceptive signals. The results therefore most directly establish the benefit of removing this representation under constrained training layouts. Success rates clearly demonstrate the performance gap, but reveal less about execution quality or potential costs of state removal, such as positioning error, corrective motions, or completion time.
Questions for authors
  • How does performance vary over a denser range of unseen heights and horizontal displacements? Does the advantage persist when these shifts are combined?
  • How does the benefit of state removal change as the spatial coverage of training demonstrations increases? This would help identify when removing state is preferable to retaining it.
  • Do the action-representation results hold beyond Pick Pen height generalization, particularly for horizontal shifts or long-horizon tasks? The current evidence seems insufficient to generalize the failure of joint-space actions.
  • Can the authors provide quantitative execution-level measures, such as grasp or placement error and completion time, to distinguish improved spatial alignment from other causes of the success-rate gains?
5 comments
AuthorsOpenReview ↗

Response to Reviewer FteL, Part 1

Thank you for recognizing the practical value of our findings and the breadth of our real-world experiments. We appreciate your constructive suggestions regarding the generalization range, action representations, and execution-level analysis, and address them point by point below.

  • Weakness 1 & Question 1: Generalization range. We provided additional details and analyses of existing experimental results, explained the observed generalization limitations, and conducted new simulation experiments.

    For height shifts, we observed no noticeable degradation in state-based policies within approximately ±3 cm of the 80 cm training height. It is due to the gripper's friction, which allows it to lift the pen even with a shallow grasp, as well as small variations in object lifting height during demonstration collection. Beyond this range, success declined rapidly: the gripper failed to reach the pen on lower tables, while the pen collided with the holder during placement on higher tables. In comparison, State-free Policies achieved nearly 100% success at both unseen heights of 72 and 90 cm, averaging 98.3%. However, their success rate also declined as the table height increased beyond 95 cm, reaching 0% at 100 cm, as reported in Table 6, due to substantial changes in the overhead observation.

    For horizontal shifts, we acknowledge that Figure 5 reports the average over 5 and 10 cm shifts, and we provide the separate results here. For example, in Pick Pen, State-free Policies achieved 100% success at 5 cm and 16.7% at 10 cm. Moreover, Table 6 reports 0% success at a larger 20 cm shift with the overhead camera retained. In contrast, state-based policies achieved 0% at all three displacements, failing to align the pen with the pen holder during placement. When height and horizontal shifts were combined, we observed that the overall success rate was primarily determined by horizontal generalization, as performance under height shifts was stronger.

    Additionally, we discuss the spatial-generalization limits of State-free Policies in Section 4.5. We attribute the observed failures under very large layout changes to substantial shifts in overhead-camera observations, as visualized in Figure 6.

    At the same time, we add additional experiments in the controlled LIBERO-10 simulation environment to investigate how policy performance changes as spatial shifts increase, including both individual horizontal and vertical shifts and their combinations. We have added these experimental results to page 10 of the revised manuscript. The success rate of State-free Policies declined more gradually as horizontal and combined shifts increased (Figure R1(a), Section R1 of the revised manuscript). Under height shifts, the fitted curves were close near zero and separated most clearly when the table height was lowered. We will include further success-rate analyses under varying spatial shifts across the full LIBERO suite and real-world experiments in the camera-ready version.

  • Weakness 2 & Question 3: Scope of action-representation results. We provided further explanations, highlighted supporting evidence from existing experiments, and conducted additional real-robot experiments.

    Section 3.2, Eqs. (3)-(5), provides a local kinematic explanation: when similar visual observations lead to similar action predictions, the same joint increment can produce different EEF motions across robot configurations. Relative EEF actions directly specify the desired local motion, while the controller uses the current robot state to execute it. This reasoning applies beyond height shifts and explains why joint-space actions can work in-domain but may produce inconsistent EEF motions under spatial shifts.

    Table 4 provides a controlled test on Pick Pen: all four representations achieve 100% in-domain success, while only relative EEF actions achieve successful height generalization. Figure 5 and Table 2 further demonstrate horizontal generalization and successful long-horizon execution using relative EEF actions across different tasks. Together, these results and the kinematic analysis support relative EEF actions as a key enabling condition in our studied settings.

    At the same time, we additionally evaluated relative joint actions on Fold Shirt under the same horizontal-generalization setting as Table 2. The relative-joint policy achieved 0% success, compared with 83.3% for the relative-EEF policy reported in Table 2. Given the cost of training and real-robot evaluation, and the limited rebuttal period, we plan to extend selected more action-representation ablations to additional tasks in the camera-ready revision.

AuthorsOpenReview ↗

Response to Reviewer FteL, Part 2

  • Weakness 3 & Question 4: State inputs and execution quality. We conducted additional state-representation experiments, provided further quantitative and qualitative analyses of real-robot execution, and added execution-level evaluations in simulation.

    State inputs. To broaden the evaluation beyond EEF-state inputs, we additionally tested a state-based policy with joint-state inputs and relative-joint actions on Pick Pen. It achieved 0% success in both height and horizontal generalization, consistent with the state-based results in Figure 5. Together, EEF and joint states cover the two commonly used forms of state inputs[1]. And Table 3 further examines whether retaining state in relative representations alleviates the same problem; the evaluated alternatives improve performance to varying degrees but do not close the gap to the state-free policy.

    Execution quality. As illustrated by supplementary real-robot videos covering all tasks, we observed no noticeable deterioration in motion quality, smoothness, or apparent execution speed after removing state. In contrast, State-based Policies exhibited systematic spatial misalignment under unseen layouts: grasping above objects on lower tables, insufficient lifting during placement, and missing the shirt or refrigerator handle. Their grasping and release positions remained associated with the training layouts. For a quantitative example, on Pick Pen, when the table height was reduced from the training height of 80 cm to 72 cm, the gripper closed approximately 5 cm above the pen. When the pen holder was shifted 5 cm from its training position, the actual release position was approximately 2 cm from the edge of the pen holder.

    For retries, successful State-free executions of the simple tasks required no retries, whereas repeated State-based attempts did not resolve the observed alignment failures. On Fold Shirt, we observed up to three retries with State-free Policies when attempting to grasp a corner of the shirt. State-based trials exceeding five retries were marked as failures. These behaviors can be clearly observed in the supplementary video.

    At the same time, in the additional LIBERO-10 simulation experiments added to page 10 of the revised manuscript, we also analyzed execution details. As shown in Table R2 of the revised manuscript, State-free Policies achieved higher first-attempt grasp success rates under horizontal, vertical, and combined spatial shifts. Among episodes completed by both policies, they also required 6-11 fewer control steps. Because the control timestep is fixed, this corresponds to shorter execution time in these episodes.

  • Question 2: Spatial coverage of training demonstrations. We explained the experimental design, conducted additional experiments on training-data coverage, and provided supporting analyses from new simulation experiments.

    Thank you for this excellent question. We explicitly considered training-layout coverage when designing our experiments. However, to cleanly isolate how state inputs affect the native spatial generalization, we deliberately restricted the layouts used for demonstration collection, as described in Section 4.1.

    In this discussion period, we additionally explored expanding data coverage in two ways. First, we collected additional demonstrations with small variations in table height (77 cm to 83 cm). Second, we mixed in demonstrations from a bottle-cap grasping task that covered the height range encountered in Pick Pen at a 90 cm table height. Neither attempt improved spatial generalization in the tested settings.

    At the same time, in the additional LIBERO-10 simulation experiments added to page 10 of the revised manuscript, we co-train all LIBERO-10 tasks and evaluate their spatial generalization. As shown in Figure R1(a), State-free Policies still retain their advantage over state-based policies as spatial shifts increase. These exploratory observations suggest that modestly expanding data coverage does not necessarily resolve the observed generalization failures.

Thank you again for your feedback. We hope these clarifications address your concerns and encourage you to reconsider the overall contribution of this paper. We will update the manuscript again after the discussion, incorporating all of the feedback received.

[1] Khazatsky, Alexander, et al. "Droid: A large-scale in-the-wild robot manipulation dataset." arXiv preprint arXiv:2403.12945 (2024).

The authors have addressed most of my concerns in their response. However, I noticed that many of the newly added descriptions, results, and conclusions have not yet been incorporated into the revised manuscript. In addition, any further analysis the authors plan to include in the camera-ready version should also be reviewed at this stage. I will therefore keep my current score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Response Summary

We sincerely thank all reviewers for their thoughtful and constructive feedback, as well as their recognition of the value of our work. We are particularly encouraged that all reviewers acknowledge the practical importance of studying spatial generalization in visuomotor policies and the substantial improvements demonstrated in our real-world experiments.

The reviewers also provided valuable constructive feedback, primarily regarding:

  • the novelty and scope of our findings relative to prior work;
  • the underlying mechanisms of spatial generalization, including coordinate alignment, action representations, and visual coverage;
  • the breadth and robustness of experimental evaluations, including larger spatial shifts, training-data coverage, visual perturbations, and cross-embodiment adaptation;
  • experimental reproducibility and additional quantitative analyses.

We have carefully addressed this feedback through further clarifications, explanations, and detailed analyses. Beyond these clarifications, we have made additional efforts to strengthen the paper, including:

  • Expanded spatial generalization evaluations: Conducted additional LIBERO-10 experiments with progressively increasing horizontal, vertical, and combined spatial shifts.
  • Additional action and state representation experiments: Evaluated relative-joint actions on Fold Shirt and joint-state inputs with relative-joint actions on Pick Pen.
  • Further coordinate correction and state augmentation experiments: Evaluated oracle state correction in LIBERO-10 and expanded the state augmentation range in real-robot experiments.
  • Additional training-data coverage experiments: Investigated the effects of expanding demonstration coverage through spatial variations and additional task data.
  • Quantitative execution-level analyses: Evaluated grasp success, corrective behaviors, and execution efficiency under spatial shifts.
  • Additional visual robustness experiments: Evaluated the effects of partial occlusion and background-color changes in LIBERO-10, complementing our real-robot observations.
  • Additional training-seed validation: Repeated a key action-representation experiment with a different random seed to assess result consistency.

We have incorporated the additional LIBERO-10 spatial generalization experiments into page 10 of the revised manuscript.

We sincerely thank the reviewers again for their valuable feedback and constructive suggestions, which have helped us further strengthen our work. We greatly appreciate the time and effort devoted to reviewing our paper. We hope our responses adequately address the reviewers' concerns and encourage them to reconsider the overall contribution of our paper. We will carefully revise the manuscript again after receiving further feedback from the reviewers.


Summary of Updates Before the Discussion Deadline

Following our discussions with the reviewers, we have revised the manuscript to incorporate their feedback. We sincerely thank all reviewers again for their constructive comments and suggestions. The main updates are summarized below.

  1. Added Section R1: Conducted controlled spatial generalization experiments on LIBERO-10, analyzing performance under increasing spatial shifts, visual perturbations, and execution quality.
  2. Revised Section 4.3: Extended state-noise augmentation to ±10 cm to examine generalization under heavily corrupted state inputs.
  3. Revised Section 4.4: Added relative joint-action results on the Fold Shirt task to further validate the importance of action representation.
  4. Revised Appendix B: Added joint-state generalization results, independent repetition of the relative joint-action ablation, and detailed training, controller, and baseline implementations.
  5. Revised Appendix D.2: Clarified the implementation and evaluation details on RoboCasa365.
  6. Revised Appendix G: Expanded the discussion of coordinate alignment and proprioceptive distribution shifts in cross-embodiment adaptation.
  7. Added Appendix H: Analyzed real-robot performance under varying spatial shifts, along with execution quality and retry behavior.
  8. Added Appendix I: Investigated whether expanding training-data coverage improves spatial generalization.

The updates above are highlighted in violet in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.