acceptodds
Withdrawn from ICLR 2027

ARIA: Assistive Teleoperation for Humanoid Loco-Manipulation via Residual Interaction Refinement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Whole-body teleoperation enables an operator to drive a humanoid as a physical avatar, but this intuitive body-level mapping has a blind spot: it cannot close the loop at the contact interface with objects. Kinematic mismatches between human and robot cause grasps to fail and objects to slip, yet the operator — limited by a restricted view of the contact and the cognitive load of coordinating locomotion with manipulation — cannot compensate in real time. Therefore, we introduce ARIA (Assistive Residual Interaction for Avatars), a co-pilot that closes this loop on the operator's behalf: it augments a frozen teleoperation policy with bounded, perception-gated corrections to task-relevant joints conditioned on egocentric object geometry, and reverts to pure teleoperation when perception is unavailable. In simulation, on three loco-manipulation tasks (picking up a box from a platform or from the ground, and carrying a table), ARIA achieves an average success rate of 84.2%, where all whole-body teleoperation baselines fail, and reduces the object error by 36–59% relative to the best baseline while keeping the root-relative tracking error within the range of the baselines. Deployed on a Unitree G1 with PICO VR or a motion-capture suit and ArUco-based onboard perception, ARIA enables assisted box pickup where unassisted teleoperation fails.

Primary area:
applications to robotics, autonomy, planning
Keywords:
assistive teleoperation, humanoid loco-manipulation, residual policy, shared autonomy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WW24OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ARIA (Assistive Residual Interaction for Avatars), an assistive teleoperation framework for humanoid loco-manipulation on the Unitree G1 robot. To address failures of whole-body motion-tracking controllers at object-contact interfaces due to kinematic mismatches and limited operator perception, ARIA augments a frozen base teleoperation policy (SONIC) with a perception-gated, bounded reinforcement-learning residual policy. The residual acts on 14 arm joints and is conditioned on a 15D egocentric object-geometry representation.

The authors evaluate ARIA in Isaac Gym across three loco-manipulation tasks—Platform Pickup, Ground Pickup, and Table Carry—using replayed OMOMO motions. They also provide qualitative real-world demonstrations using PICO VR and an inertial motion-capture suit with ArUco-marker perception.

The design is modular and practical, and the real-robot integration demonstrates substantial engineering effort. However, the evaluation raises significant concerns about baseline fairness and realism: all three baselines achieve 0.0% success across every simulated task, while the evaluations use open-loop replay without a live human operator. The paper lacks quantitative real-world trials and human-in-the-loop user studies. Generalization evidence is also limited: some tasks use only a few source motions, all simulated objects have a fixed mass of 0.5 kg, and the real-world demonstrations cover only Platform Pickup. Fine-tuned SONIC outperforms ARIA on Ground Pickup, suggesting that restricting residual corrections to the 14 arm joints can limit whole-body coordination. Further concerns include the mismatch between training and hardware safety limits, the lack of evaluation under the proposed stochastic perception-error model, and manual re-arming after perception dropout.

Critical strengths
  1. Well-Motivated Practical Problem: Addressing the “contact blind spot” in humanoid whole-body teleoperation via a bounded, perception-gated residual co-pilot is intuitive and modular, and preserves the stability of the frozen base locomotion controller.

  2. Safe Compositional Design: Restricting bounded residuals (tanh⁡\tanh saturation) to upper-body joints while gating activation by hand-to-object proximity and perception timeout (vtv_t) provides a clean fallback mechanism for hardware deployment.

  3. System Integration: Successfully integrating the pipeline onto a real Unitree G1 with both PICO VR and IMU-suit interfaces demonstrates solid engineering effort.

Critical weaknesses
  1. Baseline Fairness and Evaluation Realism Are Not Adequately Established. All three competitive baselines—SONIC, HoloMotion, and TeleopIt—achieve 0.0% success across every task in Table 1, while ARIA achieves 100.0%, 52.5%, and 100.0%. However, the simulation uses open-loop replay of recorded human motions without a human operator in the loop, potentially disadvantaging methods that depend on closed-loop human visual correction. The paper also lacks quantitative real-world comparisons, real human-in-the-loop user studies, and comparisons against object-aware alternatives such as IK-based or impedance-based assistance.

  2. Generalization and the Limitations of the Arm-Only Residual Are Insufficiently Evaluated. According to Appendix Table 6, Platform Pickup uses only two training source motions from one subject and two test source motions from one subject, while Table Carry uses only six training source motions and two test source motions. All simulated objects have a fixed mass of 0.5 kg. Fine-tuned SONIC also outperforms ARIA on Ground Pickup, where whole-body coordination may be necessary. Moreover, the proposed stochastic perception-error model is not used in the main simulation evaluations, and the training residual limits differ from the hardware safety limits.

Questions for authors
  1. Questionable Fairness of the 0.0% Success Rate Across All Baselines (Table 1). All three competitive baselines—SONIC, HoloMotion, and TeleopIt—achieve 0.0% success across every task, whereas ARIA achieves 100.0%, 52.5%, and 100.0%. Although the authors note that the baselines are object-agnostic trackers, a 0.0% baseline across the board raises serious questions about experimental setup and fairness.

    • In Section 4.1 (line 347), simulation evaluation uses open-loop replay of recorded human motions without a human operator in the loop. In real teleoperation, an operator can visually observe the robot’s hands approaching the box and adjust arm squeezing and lifting posture in closed loop to compensate for kinematic gaps. Does evaluating the baselines through open-loop offline replay remove this feedback and artificially drive their success rate to zero?
    • HoloMotion and TeleopIt track retargeted robot trajectories generated by OmniRetarget, which explicitly preserves hand-object contact relations. Why does tracking a contact-preserving retargeted trajectory still result in a 0.0% hold rate (Table 7), even on the simple Platform Pickup task? Were the PD gains, wrist/arm compliance, and reference penetration margins tuned fairly for grasping in these baselines?
    • The paper does not compare against shared-autonomy or object-aware residual baselines, such as adapting ResMimic (Zhao et al., 2025a) to the teleoperation command, or using a simple IK-based or heuristic impedance attraction toward the perceived box surfaces. Can the authors provide such comparisons? Comparing an object-aware policy solely against open-loop, object-agnostic motion trackers does not validate the specific design of ARIA’s RL residual.
  2. Lack of Quantitative Real-World Experiments and Baselines. Section 4.3 (“Real-World Demonstration”) is purely qualitative and shows only a single illustrative sequence in Figure 4, without quantitative metrics. For a robotics paper claiming reliable assistive teleoperation and introducing a carefully designed stochastic perception-error model in Section 3.3.2, systematic real-world evaluation is needed.

    • What is the real-world success rate over N≥20N\geq20 trials across different initial box positions and orientations for ARIA versus unassisted SONIC with a real human operator in the loop?
    • Real-world deployment demonstrates only Platform Pickup. How does ARIA perform on Ground Pickup and Table Carry on the real Unitree G1?
  3. Absence of Real Human-in-the-Loop User Studies. All simulation experiments use recorded OMOMO clips as a “proxy operator” (line 346), so there is no evaluation of how real human operators interact with the residual co-pilot. When πres\pi_{\text{res}} overrides arm joints by up to 0.3750.375 rad (approximately 21.5∘21.5^\circ at the elbows, Table 3), does this create visual-proprioceptive conflict or counterproductive human adaptation when a live operator is in the loop? Could the authors provide closed-loop experiments with human subjects, evaluating task completion time, success rate, operator cognitive load, and correction effort?

  4. Extremely Limited Dataset Diversity and Overfitting Risk (Table 6). The experimental setup raises substantial concerns about whether the residual policy learns generalizable interaction refinement or overfits to a narrow set of scripted trajectories and object configurations.

    • According to Appendix Table 6, Platform Pickup uses only two source motions from one training subject and two source motions from one test subject, with augmentation limited to planar rigid-body shifts. Table Carry uses only six training source motions and two test source motions.
    • All simulated objects have a fixed mass of 0.5 kg (Appendix B.3, line 825), and the evaluated tasks are largely variations of symmetric bimanual lifting with simplified rubber hands.
    • Could the authors provide more convincing generalization experiments involving unseen subjects and motion patterns, diverse object geometries (e.g., cylinders, spheres, asymmetric or smaller objects), and substantially different masses and inertias (e.g., 1–3 kg)?
    • It would also be valuable to evaluate task types beyond lifting, such as precise object placement, pushing or sliding objects, and handover or insertion. These experiments would help establish whether ARIA is a broadly applicable framework for dexterous loco-manipulation or primarily improves performance on a small set of narrowly defined tasks.
  5. Inconsistency in the Fine-Tuned SONIC Ablation (Table 2). On the hardest task, Ground Pickup, Fine-tuned SONIC achieves a 66.3% success rate and 63.2 cm object error, outperforming ARIA (52.5% success rate and 68.0 cm error) by a substantial margin. This contradicts the claim that freezing the base policy and restricting corrections to the 14 arm joints is universally superior. Deep squatting requires whole-body coordination involving hip, knee, ankle, and waist adjustments to reach the box cleanly. Freezing the lower body and waist prevents ARIA from correcting whole-body reach discrepancies, causing it to fail to lift the object in 46.2% of episodes (Table 7). How do the authors reconcile this limitation of the 14-arm-joint mask during whole-body loco-manipulation?

  6. Discrepancy Between Training Caps and Hardware Safety Caps. In Appendix Table 3, the residual caps cic_i during training are 0.25, 0.375, and 0.16 rad for the shoulder, elbow, and wrist, respectively. However, Appendix E.4 (line 991) states that on real hardware the correction is hard-clipped to 0.15 rad per joint and rate-limited to 0.4 rad/s. Why was the policy not trained with the actual hardware bounds, and how does this mismatch affect sim-to-real performance?

  7. Sim-to-Real Perception Model Ablation. Section 3.3.2 introduces a five-component stochastic perception-error model for sim-to-real transfer, but Tables 1 and 2 evaluate ARIA entirely using noise-free ground-truth object poses. Can the authors provide quantitative simulation results using the proposed stochastic perception-error model, varying noise, delay, and dropout rates, to verify the claimed robustness?

  8. Manual Re-Arming Upon Perception Dropout. Appendix E.4 (line 999) notes that if marker perception drops out for more than 100 ms, assistance fades out and is not automatically re-enabled when perception recovers; an explicit manual console request is required. Does this requirement undermine the goal of seamless perception-gated activation (ηt\eta_t) during dynamic manipulation, where brief self-occlusions are frequent?

Reviewer_XJkSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ARIA, a perception-gated residual policy that builds on a frozen SONIC controller to improve humanoid teleoperation by correcting arm motions based on object geometry. The residual policy is trained with PPO while keeping the base motion-tracking policy frozen. The simulation experiments are conducted under an open-loop setting, where prerecorded human motion trajectories are replayed as operator commands without real-time human feedback or adjustments. Under this setting, ARIA demonstrates improved manipulation success rates over motion-tracking baselines across three simulated loco-manipulation tasks. The authors further provide real-world box-pickup demonstrations on a Unitree G1 using live human teleoperation.

Critical strengths
  1. The paper studies an important and practical research problem.

  2. The paper tries to address the gap between accurate whole-body motion tracking and successful object interaction.

  3. The idea of modular residual design is intuitive and allows the original teleoperation controller to remain frozen.

Critical weaknesses
  1. Limited Generalization and Practical Applicability Due to Object-Specific Modeling and Marker-Based Tracking.

The proposed method relies heavily on prior knowledge of object geometry and accurate object pose estimation, which significantly limits its generalizability and practical applicability. Specifically, ARIA requires a predefined object geometry model to compute hand–object spatial relationships, while its real-world implementation relies on attaching ArUco markers to the manipulated object for continuous pose tracking. Such assumptions are restrictive in real-world teleoperation scenarios, where robots are expected to interact with diverse, previously unseen objects that cannot be modeled or instrumented in advance. Moreover, the current experiments only consider two object types (a box and a table) in simulation and a marker-equipped box in real-world deployment, providing insufficient evidence of generalization to novel object geometries and unstructured environments. To demonstrate the practical value and generalizability of ARIA, I strongly encourage the authors to evaluate the method using general-purpose, marker-free object tracking techniques, ideally without requiring preconstructed object models. Experiments on previously unseen objects with varying geometries, together with quantitative evaluations under realistic perception uncertainty, would provide much stronger evidence of the method's applicability beyond carefully prepared experimental settings.

  1. Unfair and insufficiently controlled experimental comparisons.

The current comparisons do not adequately isolate the contributions of object-aware observations, residual learning, and arm-only corrections. Moreover, prior works such as ResMimic (Zhao et al., 2025) and GRAIL (Xie et al., 2026) have already demonstrated that motion tracking alone is insufficient for reliable manipulation and that object-aware corrections can significantly improve task success. Comparisons against object-unaware motion trackers therefore provide limited evidence of ARIA's specific algorithmic advantages.

Unfair comparison regarding object observations. ARIA explicitly uses object geometry, whereas the motion-tracking baselines do not. It is also unclear whether Fine-tuned SONIC receives object observations, as Appendix C only describes decoder fine-tuning. The authors should clarify this and include an object-aware baseline, such as SONIC + Object-aware Wrist IK, using the same object observations to determine whether learned residual corrections outperform simpler geometry-based methods.

Lack of closed-loop human teleoperation evaluation. The simulation experiments replay prerecorded human motions without allowing operators to react to grasping failures. While useful for controlled evaluation, this does not establish ARIA's advantages over realistic closed-loop human teleoperation, where operators can actively correct their movements. The authors should conduct real-time human-in-the-loop comparisons and report task success rates, completion time, operator correction effort, and deviations from commanded motions.

Different action spaces in Fine-tuned SONIC. Fine-tuned SONIC modifies the whole-body decoder, whereas ARIA restricts corrections to 14 arm joints. The larger tracking errors of Fine-tuned SONIC may therefore result from its broader action modifications. An arm-only Fine-tuned SONIC baseline, ideally with the same object observations, is needed to isolate the benefits of the residual architecture.

Questions for authors
  1. Concerns about the real-world and simulation comparisons.

In the real-world videos, SONIC seems to fail mainly because only the robot's fingertips touch the box, causing the box to slide out of its hands. In contrast, ARIA appears to rotate the wrists to increase the contact area with the box, allowing the robot to hold it more securely. However, the human operator does not seem to make any wrist adjustments in either video. This is somewhat unnatural, as a human would normally adjust their wrist orientation to get a better grip and prevent the box from sliding. I would like to see whether ARIA still offers a clear advantage when operators actively adjust their motions based on what they observe, as they would in natural teleoperation. I also have concerns about the fairness of the simulation comparisons. In the Ground Pickup task, SONIC appears to kick the box away before its hands even reach the object, which leads to failure. Since ARIA only modifies arm joints, it is unclear how its arm corrections could address this issue. In the Table Carry task, SONIC and ARIA appear to bend forward by different amounts, causing SONIC's hands to fail to reach the table. Given that ARIA uses the same frozen SONIC policy and only modifies arm joint commands, why do the two methods show such different body motions? I would appreciate clarification on what causes these differences and whether both methods are evaluated under the same conditions.

  1. Insufficient discussion and comparison with closely related work.

The related work discussion appears incomplete, particularly regarding several highly relevant studies, including GRAIL (Xie et al., 2026), TASC (Fu et al., 2026), and OKAMI (Li et al., CoRL 2025), which are neither adequately discussed nor experimentally compared. Among these, GRAIL is especially relevant, as it adopts a very similar technical framework: it builds upon a frozen SONIC controller and introduces an object-aware latent adaptor that learns residual corrections to improve humanoid manipulation. Although ARIA applies bounded residual corrections in joint space rather than latent space and targets real-time human teleoperation, the underlying idea of combining a frozen SONIC controller with object-aware residual learning has already been explored. Therefore, this combination alone should not be presented as a fundamentally new technical contribution. I would like the authors to explicitly discuss the differences between ARIA and GRAIL's object-aware adaptor and, ideally, provide a direct comparison with a GRAIL-style latent residual baseline trained using the same motion data, object observations, and evaluation conditions. Such a comparison would help establish whether ARIA's particular residual design provides meaningful advantages beyond existing approaches. Furthermore, TASC investigates object-aware shared control for teleoperated robotic manipulation, while OKAMI explores object-aware retargeting for humanoid manipulation. Although these methods differ in their overall task settings, they address closely related problems of improving object interaction through perception-based motion correction. Importantly, since ARIA's learned corrections are restricted to the 14 arm joints while the remaining whole-body control is handled by frozen SONIC, the use of a humanoid platform alone should not justify excluding relevant robotic-arm shared-autonomy methods from consideration. I would therefore appreciate a more comprehensive discussion and, where technically feasible, appropriately adapted comparisons with these approaches. In its current form, ARIA appears closer to a practically valuable extension and integration of existing object-aware residual control techniques into real-time humanoid teleoperation, rather than a fundamentally new paradigm of object-aware residual learning. The authors should clarify which aspects of the proposed framework constitute genuinely new algorithmic contributions and provide stronger evidence to support these claims.

References

  1. GRAIL — (Xie et al., 2026) Xie, T., Zhang, H., Park, J., Wang, Z., et al. (2026). GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors. Conference on Robot Learning (CoRL 2026). arXiv:2606.05160.
  2. TASC — (Fu et al., 2026) Fu, Z., Song, P., Hu, Y., & Detry, R. (2026). TASC: Task-Aware Shared Control for Teleoperated Manipulation. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2026). arXiv:2509.10416.
  3. OKAMI — (Li et al., 2025) Li, J., Zhu, Y., Xie, Y., Jiang, Z., Seo, M., Pavlakos, G., & Zhu, Y. (2025). OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation. Proceedings of the 8th Conference on Robot Learning (CoRL), Proceedings of Machine Learning Research, 270, 299–317.
Reviewer_ajAHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses object-interaction failures in whole-body humanoid teleoperation, where human–robot kinematic mismatch causes grasps to fail even when body motion is tracked well. ARIA keeps a pretrained whole-body tracker (SONIC) frozen and adds an RL-trained residual that applies bounded corrections to the 14 arm joints, conditioned on a 15-D egocentric object state and switched on by a proximity- and perception-based gate. The residual is trained in Isaac Gym on OMOMO demonstrations retargeted to the Unitree G1. On three simulated tasks evaluated by replaying held-out-subject demonstrations, ARIA reaches 100.0 / 52.5 / 100.0% success while three whole-body trackers reach 0%; ablations cover object geometry, the gate, and fine-tuning the base policy instead, and the system is demonstrated qualitatively on a real G1.

Critical strengths
  • A careful, safety-conscious system that transfers from simulation to a real humanoid. The correction is limited to arm joints and capped by tanh saturation (Eq. 7), and it reverts to pure teleoperation whenever the object pose is invalid. A five-part perception error model (noise, delay, dropout and burst occlusion, calibration bias, visibility) bridges the sim-to-real gap, and the same policies run with both PICO VR and a motion-capture suit. Subject-level splits, exact success criteria, and a per-condition breakdown (Table 7) make the results easy to interpret; Table 7 also shows that every tracker stays upright yet almost never lifts the object, placing the failure at the contact interface.
  • The ablations isolate each design choice. Zeroing the object state drops success to 50.0 / 25.0 / 27.0% with root-relative error (E_L) within 0.4 cm of ARIA, so the geometry determines how to correct rather than degrading tracking. Removing the gate drops success to 85.0 / 42.5 / 51.0%. Compared with fine-tuning the base policy on the same data and rewards, the residual gives higher success on two of three tasks and lower E_L on all three (5.4 / 6.5 / 8.2 vs 11.4 / 8.9 / 10.8 cm), and the authors openly note that fine-tuning wins on Ground Pickup (66.3 vs 52.5%).
Critical weaknesses
  • Assistance to a human operator is the paper's main claim, but it is not evaluated. What sets ARIA apart from autonomous object-aware residuals such as ResMimic and AdaptManip is that the correction runs inside the teleoperation loop, with claims that the operator retains control and that "the assistive layer is transparent to the operator". However, all quantitative results come from replaying recorded demonstrations: "each test clip plays the role of the operator: its recorded motion is replayed open-loop as the command, without reacting to the robot." The residual is also trained to track the demonstration's own robot and object reference for the same task, so at test time it completes a demonstrated pickup whose goal is known, rather than assisting an operator whose intent may differ. There is no user study, no measure of operator effort or of how often the correction overrides the operator, and the real-robot results (Fig. 4) are qualitative without trial counts or success rates. The only stand-in for operator variability is the held-out subject, which comes down to two source motions for Platform Pickup and Table Carry (Table 6). As it stands, the evidence supports "an object-aware residual completes demonstrated pickups" more than "ARIA helps operators". A small closed-loop study with real operators (success with and without ARIA, even with a few operators and tens of trials on the G1) would address this; otherwise, the assistive framing should be narrowed.
Questions for authors
  1. [Clarification] Does fine-tuned SONIC (Table 2) observe the object state? Only its action decoder is updated, with encoders frozen (App. C). If it does not see the object, the comparison mixes "residual vs. fine-tuning" with "object observation vs. none"; results for fine-tuned SONIC with object observation would make it cleaner.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.