acceptodds
Under review as a conference paper at ICLR 2027

TRACK: Object Keypoints and End-Effector Waypoints as Geometric Task Representations

Abstract

Executing a new manipulation task requires translating a task specification into actions suited to the robot. Geometric specifications obtained from demonstrations or generated by planners can describe the intended behavior, but their hand or end-effector trajectories generally cannot be replayed directly to accomplish the task. We propose TRACK, a framework for specifying and executing manipulation tasks through object keypoint trajectories, abstract grasp flags, and approximate hand or end-effector waypoints. Object trajectories and grasp flags specify desired object motion and grasping interactions, while hand waypoints provide soft guidance that allows deviations needed for successful manipulation. This shared representation accommodates specifications from scripted experts, motion planners, and vision-language models. We train a policy across multiple tasks using reinforcement learning, with reference-derived rewards for object motion, grasping, and end-effector tracking within a positional tolerance, without task-specific success rewards. A state-based progress tracker aligns the policy’s reference window and training reward with the robot’s execution progress. At test time, the policy receives a single task specification and executes the task without additional training. Experiments on single-arm and bimanual manipulation demonstrate generalization to unseen tasks from a single task specification. TRACK outperforms reference replay with approximate end-effector trajectories and imitation learning-based baselines, and also executes specifications generated by a vision-language model.

Primary area:
applications to robotics, autonomy, planning
Keywords:
trajectory-conditioned policies, robot manipulation, multi-task reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_efAPOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TRACK, a reference-conditioned reinforcement learning framework that represents manipulation tasks using object keypoint trajectories, grasp flags, and approximate end-effector waypoints. A progress tracker aligns policy execution with the reference. The authors evaluate a single policy on seen and unseen tasks in Meta-World and RoboTwin, including corrupted scripted references and VLM-generated plans.

Critical strengths
  • The geometric task representation and state-based progress tracker are technically meaningful.
  • The tracker improves robustness to temporal misalignment and generally outperforms clock-based tracking.
  • The framework supports single-arm and bimanual manipulation without task-specific success rewards.
Critical weaknesses

The main experimental setting does not provide convincing evidence of generalization to unseen tasks. The reference and test episode are initialized from the same simulator state. Moreover, only the end-effector trajectory is corrupted, whereas the object keypoint trajectory and grasp flags remain exact. The applied corruption is relatively limited, consisting of a smooth displacement of at most 2 cm per axis and Gaussian jitter with a standard deviation of 3 mm. The corruption procedure is also shared between training and evaluation.

Under this protocol, the policy can largely solve the task by reconstructing the exact reference trajectory from a mildly corrupted version. The experiment therefore resembles reference denoising more closely than one-shot execution of multiple unseen tasks. A stronger evaluation should provide the task specification from a different initial state. A reference collected from another robot embodiment would provide more direct evidence for the claimed embodiment independence, but no such experiment is included.

The VLM-plan experiment reinforces this concern. Even on training tasks, one-shot VLM plans achieve only 0.60 success on ML10 and 0.52 on ML45 with the tracker, despite providing an example plan from the same task. These results are substantially below those obtained with scripted references. This suggests that the method depends strongly on having a reference closely aligned with the exact execution instance. Although TRACK outperforms direct reference replay in the aggregate VLM results, the experiment does not establish robust task execution from an independently generated task specification.

On unseen ML45 tasks, TRACK achieves only 25% success with one-shot plans, whereas DCRL[1] reports 48% with one demonstration. Furthermore, DCRL evaluates each episode with independently sampled demonstration and execution initial conditions, providing a stronger test of task-level generalization.

[1] Christopher R. Dance, Julien Perez, and Th ́eo Cachet. Demonstration-conditioned reinforcement learning for few-shot imitation. In International Conference on Machine Learning (ICML), volume 139 of PMLR, pp. 2376–2387, 2021.

Questions for authors
  • Can the authors evaluate TRACK using references collected from initial states different from those of the execution episodes?

  • How does TRACK perform when the corruption type or magnitude differs between training and testing?

  • Can the authors provide a genuine cross-embodiment experiment using references collected from a different robot?

  • How does TRACK compare with DCRL under a common ML45 unseen-task protocol, particularly when the reference and execution initial states are sampled independently?

9 comments

We thank the reviewer for the careful and critical review. The new evaluations are in purple.

Q1. References from initial states other than the execution's

We added this evaluation, together with a simple adaptation of the reference to the new scene. In short, a reference recorded in another initial state no longer solves the new scene for any method, but the adapted reference brings TRACK back close to its matched performance and above replay.

We evaluate each reference in three settings (ML10, App. C.14). Matched is Table 1's protocol, where the episode starts from the initial state in which the reference was recorded. In another initial state, the same reference runs from the initial state of another demonstration of the same task. Adapted uses that initial state with the reference first adapted to it, by shifting the end-effector path and the object tracks onto the scene's object and moving the end of the carry to the scene's goal. Replay receives the same adapted references, and the goal is given to the adaptation, not to the policy. Matched replay and clock values are those of Table 1:

ML10trainheld-outone-shot
Matched
Replay0.830.800.61
TRACK (clock)0.990.880.67
TRACK (tracker)0.960.890.77
Another initial state
Replay0.390.230.39
TRACK (clock)0.370.210.37
TRACK (tracker)0.340.190.37
Adapted
Replay0.790.790.60
TRACK (clock)0.940.880.65
TRACK (tracker)0.910.860.71

Every method drops with a reference from another initial state, mostly because that reference no longer fits the scene. Its object sits a median 6.9 cm away, and moving an object by 5 cm already makes every method fail on most episodes. Where the goal is placed independently of the object, the reference also ends at the wrong goal. Adapted to the scene, the reference recovers at least 84% of TRACK's drop, and TRACK is again above replay on every split.

So TRACK tolerates references that are approximate but made for its scene, such as corrupted end-effector paths and grasp flags (Q2), errors in the perceived object position (vYK8, Q4, App. C.13) or demonstrations from another robot (Q3). A demonstration from another scene must first be adapted to it.

Q2. Different corruption type or magnitude in training and testing

We tested the paper's policies at other corruption magnitudes and on corruption types they never saw, and trained policies at other magnitudes. TRACK is above replay under every one of these test corruptions on ML10, at every magnitude from exact references to 4× the training corruption, and under every corruption type we tested, on training, held-out and unseen-task references. The tracker policy is above replay in every case, and the clock policy in every case but one, where it is level (one-shot under a constant offset).

Magnitude (ML10, App. C.15). Cleaner references than in training cost TRACK nothing. At twice the training magnitude it loses about as much as replay, and at three and four times the magnitude its lead over replay is larger than at the training magnitude. Training at the test magnitude, exact or twice as large, helps by at most 0.07, so TRACK does not rely on seeing the test corruption in training.

Types not seen in training. Time warps and grasp flags shifted by 10–20 reference indices cost TRACK at most 0.05, except for the tracker policy with flags 20 indices early, while replay loses up to 0.35 when the flags shift. Replay closes the gripper on the flags, whereas TRACK's grasp timing depends little on them. A constant offset costs TRACK the most overall, at most 0.10, and leaves replay unaffected, because replay subtracts the offset of the first waypoint. Jitter of 1 cm per index, more than three times the 3 mm in training, costs TRACK at most 0.05 and replay up to 0.07. Noisy object tracks, at half the hand's corruption (0.7 cm on average), cost at most 0.01.

On "reference denoising". Three observations argue against the policy simply reconstructing the clean reference. Among the episodes it wins and replay loses on ML10, its path leans toward the uncorrupted track in only 19–41%, while a grasp correction occurs in 66–92% (vYK8, Q1). Corruption types never seen in training cost it little, and training on exact or 2× references helps by at most 0.07 (above).

On the VLM results, and 25% vs. DCRL's 48%

The VLM experiment tests references that come from a planner rather than from the simulator, and the two numbers are measured under different protocols.

The plans are made for the episode's own scene, and in a plan no channel is the simulator's record, so they test the case the reviewer raises. There the planner is the bottleneck (on ML10's training tasks, replay scores 0.81 on expert references but 0.40 on one-shot plans of the same scenes). Our planners are open models of at most 27B parameters, far smaller than current state-of-the-art models, and going from an 8B to a 27B planner already improves the plans (App. C.18). Better plans help TRACK more than replay, and with expert references, a stand-in for a near-perfect planner, TRACK reaches 0.95 on its training tasks (App. C.18). The 25% is TRACK on ML45's unseen tasks executed from VLM plans rather than from a demonstration. Conditioned on one demonstration of the same tasks, TRACK reaches 0.74 (Table 1). DCRL's 48% uses Meta-World v1, the observed goal, task rewards and 10 folds over 46 tasks, which leave out two of ML45's test tasks. DCRL's own supplement (Sec. 4.7) reports 45% for a policy that receives no demonstration, against 48% with one. Most of the 48% therefore comes from the observed goal and the task reward, not from the demonstration, and our DCRL-style policy shows the same pattern (App. C.7). For a comparison under one protocol, we trained a DCRL-style policy in our framework (Q4 below, and the table in our reply to R25b, Q1, App. C.7). TRACK is well above it when the reference matches the scene, and, with the reference adapted to the scene, above both DCRL-style arms when demonstration and initial state are drawn randomly and independently.

Q3. References from another robot

We added an experiment in which policies trained on one robot execute demonstrations recorded with another. In short, TRACK executes the other robot's demonstrations and, on average, is above replaying them, on unseen tasks and on the tasks it was trained on.

Setup (App. C.16). A simulated Franka Panda (system-identified controller and gripper) with Meta-World's own objects in its workspace. TRACK's keypoint recipe (K = 4, clock and tracker, 5 seeds each) is trained on the Panda's own demonstrations of ten tasks. A controller turns the wrist to the reference's orientation, and the policy commands the translation and the gripper. At test time, the policies are conditioned on the Sawyer's original Meta-World demonstrations, mapped into the Panda's frame, on the training tasks and on five Meta-World tasks they were never trained on. The two robots differ in hand geometry (the Panda's hand is about 20 cm long).

Panda, Sawyer demonstrationstrainheld-outone-shot
Replay0.560.560.48
TRACK (clock)0.600.620.55
TRACK (tracker)0.590.610.60

TRACK's lead comes from tasks on which replaying the Sawyer's path fails on the Panda. On the training tasks and on window-close, its successes come from following the Sawyer reference, since none remain when the reference is frozen at its first index.

In a second experiment (App. C.17, details in our reply to vYK8, Q1), a simulated KUKA iiwa arm with a five-finger hand is trained and evaluated with the Sawyer's demonstrations as references, which contain no finger motion. TRACK learns the finger motion and is well above replay in distribution and on the five unseen tasks (one-shot 0.65 vs. 0.40 with the tracker). These are two pairs of robots and few unseen tasks, so we report them as first results. For a real robot, including five human demonstrations, see our reply to vYK8, Q4.

Q4. DCRL under a common protocol with random initial states

We trained a DCRL-style policy in our framework, with demonstrations and initial states drawn randomly and independently, as suggested in the review (see our reply to R25b, Q1 and App. C.7). In short, TRACK is well above it when the reference matches the scene. With random initial states, TRACK is at replay's level, but adapting the reference to the scene puts it well above both DCRL-style arms on unseen tasks (0.57 / 0.69 vs. 0.16 / 0.27).

Thank you for the extensive additional experiments. The results for Q2 and Q3 show robustness to inaccurate end-effector paths and to references from another robot. They also clarify what TRACK requires, and this raises my main concern about its intended use.

Generalization

With a reference from another initial state, TRACK drops to 0.34/0.19/0.37 and does not outperform replay. Performance recovers only after the reference is adapted using the current object position and goal. In the main protocol, the object track is the exact object trajectory of the execution episode. In the VLM experiment, the planner receives the goal coordinates. In every working setting, therefore, the object and goal geometry of the current scene is resolved outside TRACK, by the simulator, a planner, or a goal-aware adaptation. TRACK corrects the end-effector path, grasp timing, and embodiment differences of a reference that is already exact for the scene. This makes it a closed-loop executor of scene-specific object plans. It does not show generalization to unseen tasks from an independently obtained demonstration.

the claims and comparisons.

  • The absence of task-specific rewards is a property of training. It does not reduce the information TRACK needs at test time. As the rebuttal itself argues for DCRL's 48%, goal information explains much of the performance, and TRACK also receives the goal in every setting. The comparison with DCRL therefore does not establish better task-level generalization.
  • If TRACK is an executor, the relevant baselines are other methods that execute the same input, such as trajectory-tracking controllers or policies conditioned on object trajectories. Replay is not sufficient.
  • The rebuttal states that the planner is the bottleneck (replay 0.81 with expert references vs. 0.40 with one-shot plans). It also states that moving an object by 5 cm makes most episodes fail. Practical use thus depends on accurate upstream estimation. However, no results are reported with realistic errors in the object and goal positions used to adapt the reference.

I ask the authors to restrict the claims to execution of scene-aligned references, to state explicitly that the goal is available in all settings, and to evaluate adaptation under realistic object and goal estimation errors.

Given the additional evidence on corruption robustness and cross-embodiment transfer, I raise my rating from 1 to 2. The paper remains below acceptance because its central claim of executing tasks from independently obtained references still depends on scene-exact object and goal information.

Reviewer_R25bOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRACK, a framework for executing manipulation tasks from geometric task specifications consisting of end-effector waypoints, grasp flags, and object keypoint trajectories. It trains a multi-task reinforcement learning policy using task-independent tracking rewards and introduces a state-based progress tracker to align reference trajectories with actual robot execution, including synchronization for bimanual manipulation. Once trained, the policy can execute unseen tasks from a single reference without further training. Experiments on Meta-World and RoboTwin 2.0 demonstrate improvements over reference replay and imitation learning baselines, particularly on unseen tasks. The paper also evaluates the execution of VLM-generated task specifications.

Critical strengths
  1. Meaningful and practical framework. TRACK combines geometric task representations, multi-task RL, and closed-loop progress tracking to enable unseen-task execution from a single reference without additional training.

  2. Promising experimental results. The RoboTwin experiments show that the progress tracker improves unseen-task success from 59% to 69% under the same 300-step budget, providing evidence of its effectiveness.

Critical weaknesses
  1. Unclear technical novelty and insufficient comparisons. The proposed task representation, multi-task RL, and closed-loop tracking build on closely related ideas. The paper does not sufficiently distinguish its contributions from prior work, and the main comparisons focus on reference replay and imitation learning baselines with different inputs or training conditions. Comparisons with more closely related reference-conditioned RL methods are needed.

  2. Unfair evaluation settings and incomplete ablations. On Meta-World, TRACK with the progress tracker is evaluated for 500 steps on unseen tasks, while other methods mainly use 200 steps, making it difficult to isolate the tracker's benefit. Furthermore, RoboTwin lacks a "Tracker + Zero Residual Replay" baseline, leaving the contributions of progress tracking and learned corrections insufficiently disentangled.

Questions for authors
  1. Technical novelty and related work. What are the key contributions of TRACK beyond existing reference-conditioned RL and trajectory-tracking methods, such as DCRL and Scaffolding? Can the authors provide more closely related comparisons under matched input and training conditions?

  2. Fair evaluation. On Meta-World, TRACK with the progress tracker uses 500 evaluation steps, while other methods mainly use 200. Can the authors report results under matched execution budgets to clarify how much improvement comes from the tracker itself?

  3. Progress tracker ablations. On RoboTwin, how does "Tracker + Zero Residual Replay" perform compared with the full method? Additionally, how does the proposed tracker compare with a simpler feedback-based waypoint advancement mechanism? These comparisons would help isolate the contributions of learned corrections and the specific tracking design.

  4. Task representation. The end-effector reference ablation removes both policy conditioning and reward supervision. Can the authors disentangle their individual contributions? Also, how is the wrist orientation used by the RoboTwin controller consistent with the proposed task representation?

4 comments

We thank the reviewer for the thoughtful questions. The new comparisons are in green in the revision.

Q1. Novelty relative to DCRL and Scaffolding

TRACK's contributions relative to DCRL, Scaffolding and trajectory-tracking methods are:

  1. A task specification without robot actions or task identity. End-effector waypoints, object keypoint tracks and grasp flags, which a planner, a VLM, a teleoperator, a human or another robot can provide (Table 3, Sec. 4.5, App. C.16).
  2. One task-agnostic reward computed only from the reference. A single policy trains across tasks without task rewards or success labels, and executes unseen tasks from one reference without further training.
  3. A state-based progress tracker that re-indexes both the conditioning and the reward from the observed task state, objects included, and synchronizes two arms at grasp events.
  4. Evidence on single-arm and bimanual benchmarks that our learned controller outperforms executing the reference directly when the reference is approximate (vYK8, Q1).

DCRL instead conditions on demonstrations of the same robot, observes the goal and trains on each task's own reward. Scaffolding trains one policy per task. To compare under one protocol, we reimplemented these defining choices in our framework, as DCRL-style and Scaffolding-style variants.

Scaffolding-style specialists (RoboTwin, App. C.6) are per-task policies in our stack, without the end-effector reward. On the unseen tasks of Table 2, a single TRACK policy that never saw them is above specialists trained on that very task (0.59 / 0.69 for clock / tracker vs. 0.52 / 0.57), and the specialists are at or below replay (0.57). On their own training tasks the specialists are, as expected, better at their best checkpoints (0.79 / 0.85 vs. 0.73), but they train five times as long per task. With the same training per task as TRACK they are level with it (0.72 / 0.70), but every new task needs its own specialist.

DCRL-style arm (App. C.7), on the standard ML45 split, with the task reward, the goal observed, and demonstration and initial state drawn randomly and independently, plus DCRL's no-demonstration control, at 400 steps on training and held-out references and 500 one-shot, except that with matched initial states replay and TRACK (clock) are as in Table 1:

ML45, train / held-out / one-shotmatched initial staterandom initial state
TRACK (clock, Table 1)0.88 / 0.84 / 0.550.45 / 0.31 / 0.21
TRACK (clock), reference adapted to the scene's object and goal—0.84 / 0.80 / 0.57
TRACK (tracker)0.90 / 0.89 / 0.740.46 / 0.31 / 0.30
TRACK (tracker), reference adapted to the scene's object and goal—0.88 / 0.84 / 0.69
DCRL-style (task reward, goal observed)0.53 / 0.47 / 0.130.49 / 0.42 / 0.16
DCRL-style, no demonstration0.60 / 0.52 / 0.220.59 / 0.54 / 0.27
Replay0.72 / 0.70 / 0.680.45 / 0.29 / 0.25

The DCRL-style arm is below its no-demonstration control on every split. With matched initial states TRACK is 0.35–0.61 above it. With random initial states, a reference recorded in another scene no longer matches the scene TRACK acts in (efAP Q1), but adapting the reference to the scene's object and goal restores TRACK above both DCRL-style arms. The remaining differences from DCRL are listed in App. C.7.

Q2. Matched execution budgets

We re-scored ML10 with replay and both TRACK variants at 200 and at 400 / 500 steps (App. C.1). On ML10 the longer horizon matters only for the tracker. Replay and the clock policy advance through the reference at a fixed pace and then hold its final pose, so extra steps change their success by at most 0.01. The tracker can slow the reference down, so it needs time to finish it. We therefore report replay and the clock policy as in Table 1 and the tracker at 400 / 500 steps. One-shot on ML10, the horizon does not matter (no success occurs after step 200), so the tracker's 0.77 against replay's 0.61 is a matched-budget result. In distribution, the tracker's deficit to the clock at 200 steps (0.87 vs. 0.99 on training references) is the progress floor truncating the reference. At 400 steps the tracker policy reaches 0.96 vs. 0.99. On ML45 the tracker does need the longer horizon one-shot (0.57 at 200 steps, 0.74 at 500, App. C.1), and replay gains 0.04 there (0.68 to 0.72). It is below replay at 200 steps and level with it at 500, which is why Sec. 4.2 reports the two as similar. Table 3 scored the tracker at 500 steps and the other methods at 200. With every method at the same horizon, 200 or 500 steps, the tracker is still above replay on ML10's VLM plans, and giving the clock policy 500 steps leaves the tracker's lead over it about as in Table 3.

Q3. Replay with the tracker, and a simpler rule

On RoboTwin (App. C.3), we separate the two parts and also test a simpler progress rule. The reach rule advances to the farthest of the next 5 waypoints that lies within 2 or 4 cm of both hands, and keeps the tracker's floor and grasp barriers:

RoboTwintrainheld-outone-shot
Replay with clock0.680.640.57
Replay with tracker0.640.670.53
TRACK with clock0.700.730.59
TRACK with tracker0.700.730.69
TRACK with reach rule, 2 cm / 4 cm0.68 / 0.650.77 / 0.740.47 / 0.61

The gain needs both the learned correction and the tracker. The tracker alone, without the learned correction, does not beat replay on average and is below it one-shot (0.53 vs. 0.57), and the correction alone, with the clock, adds only +0.02 one-shot. Replacing the tracker with the reach rule costs TRACK one-shot success on every seed (−0.22 / −0.09), because the rule checks only the hands, whose path is corrupted. Our tracker also matches the object keypoints, and without them it loses its one-shot gain as well (−0.17 matching on the end-effectors alone, −0.18 with the grasp flags added). With uncorrupted end-effector references, the simpler rules are level with the tracker, so they lose because the hand path is wrong.

Q4. Observation vs. reward, and wrist orientation

Splitting the end-effector ablation into observation and reward (ML10, App. C.4):

hand observedhand rewardedML10 clockML10 tracker
yesyes (full method)0.99 / 0.88 / 0.670.96 / 0.89 / 0.77
noyes0.83 / 0.75 / 0.460.86 / 0.76 / 0.53
yesno0.33 / 0.30 / 0.100.25 / 0.23 / 0.06
nono (paper's ablation)0.36 / 0.33 / 0.170.33 / 0.30 / 0.18

The reward carries most of the effect (59–84% of the gap to the full method). Observing the path helps only when it is also rewarded. Without the reward it changes success by −0.12 to −0.02. Observing the rewarded path adds +0.21 / +0.24 one-shot (clock / tracker), mostly on drawer-open and lever-pull. On RoboTwin, references also carry the wrist orientation, which only the controller uses and which is not corrupted (App. C.5). Replay uses the same controller, so the orientation helps both methods equally. The text after Eq. 1 now states that a reference may carry the orientation as well.

Thanks for the detailed rebuttal and additional experiments. I think most of my original concerns have been addressed, especially the comparisons with related methods, matched evaluation budgets, and the progress tracker ablations.

I still have some concerns about the technical novelty and the reliance on scene-aligned references. However, the new experiments provide stronger evidence for the effectiveness of the proposed framework.

Overall, I am more positive about this work after the rebuttal, and I will raise my rating from 2 to 3.

We thank the reviewer for the positive assessment and for raising the rating. On the two remaining concerns, the newest revision and our discussion with Reviewer efAP add the following.

  • Scene-aligned references. The revision states this setting explicitly and measures how exact the alignment has to be. When a reference is adapted with object and goal positions that are 1 cm off, TRACK loses at most 0.06 on ML10 (App. C.14).
  • Technical novelty. Our latest reply to Reviewer efAP positions TRACK against related work on trajectory-conditioned execution and residual RL.

We thank the reviewer again for the constructive discussion.

Reviewer_vYK8OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes TRACK, a general RL-based tracking controller for robotic manipulation. It uses end-effector waypoints, object keypoint trajectories, and grasp flags as task representations, and trains a shared policy across multiple manipulation tasks using a unified tracking reward. The method also introduces a progress tracker to align the reference trajectory with the robot's execution progress. Experiments on Meta-World and RoboTwin demonstrate improvements over reference replay and imitation learning baselines, including on unseen tasks.

Critical strengths

The formulation of manipulation tasks as a unified RL tracking problem is good. Instead of designing task-specific rewards, the authors use object keypoints, end-effector waypoints, and grasp flags to train a shared tracking policy across different tasks. The idea of learning a general RL controller from multiple reference trajectories has some novelty and could potentially benefit from larger-scale training. Moreover, an RL controller trained through interaction in simulation has the potential to provide stronger closed-loop control, better failure recovery, and more effective handling of contact dynamics, potentially achieving performance beyond conventional imitation learning methods.

Critical weaknesses

Insufficient motivation and evidence for the necessity of an RL-based low-level tracking controller in gripper manipulation.

My main concern is whether learning a separate low-level RL tracking controller addresses a sufficiently important problem in the parallel-jaw gripper manipulation setting considered in this paper.

For dexterous hand manipulation, learning-based tracking controllers are well motivated because directly replaying reference motions often fails to reproduce complex contact dynamics, in-hand manipulation, and fine-grained finger coordination. However, the situation is somewhat different for parallel-jaw grippers. For many grasp-and-transport tasks, once a stable grasp has been established, the remaining manipulation can often be accomplished through conventional end-effector trajectory following.

Indeed, the reference replay baseline already achieves relatively strong performance in several settings, despite the artificially corrupted end-effector trajectories. While TRACK further improves success rates, the paper does not sufficiently explain what additional manipulation capabilities are learned by the RL controller or why they justify the additional learning and system complexity.

One possible explanation is that the learned controller acquires robustness to imperfect grasps, contact errors, or execution failures. For example, it might correct an inaccurate grasp or compensate for object motion that deviates from the reference. Such capabilities would be useful, but the current experiments do not isolate them from improvements due to temporal alignment or reference-following corrections.

This concern is particularly relevant to RoboTwin, where the learned policy only predicts translation residuals on top of an existing reference replay controller, while wrist orientation and gripper control are still largely determined by the reference and heuristic controllers.

I would therefore like to see a more detailed analysis of where and why TRACK improves over replay. In particular, comparisons with stronger non-learning reference-following controllers, including replay with the same progress tracker, would help establish the actual contribution of the learned low-level policy.

Limited demonstration of the role of TRACK in a complete manipulation system and its practical relevance.

The paper primarily studies the execution of already available task-solving reference trajectories. However, it remains unclear how the proposed low-level controller would be integrated into a complete and scalable manipulation system.

In the main experiments, the references are generated by scripted experts or planners in simulation. The paper also explores VLM-generated references, which is a valuable step, but this evaluation does not yet demonstrate a reliable high-level planning and low-level execution pipeline for challenging unseen manipulation tasks.

This raises an important question about the overall benefit of the proposed hierarchical approach. If a high-level policy is already capable of producing detailed end-effector trajectories and grasping instructions, to what extent does a separately learned RL controller improve performance over directly executing these plans with conventional controllers, or over a task-conditioned visuomotor policy?

I am also concerned about the practical assumptions required for deployment. The method relies on reference trajectories and online object keypoint observations, both of which are readily available from simulation but more difficult to obtain reliably in the real world. The paper does not investigate sim-to-real transfer, perception errors, or the robustness of the learned controller to discrepancies between simulated and real contact dynamics.

Finally, the comparison with Diffusion Policy does not fully resolve these questions. The imitation-learning baseline uses a different conditioning interface and learning objective from TRACK, making it difficult to attribute the performance differences solely to the benefit of RL-based low-level tracking. A stronger comparison would place the methods in a common high-level planning and execution setting, with comparable information available at inference time.

Overall, although TRACK demonstrates improvements on the selected benchmarks, I am not yet convinced that the experiments establish the broader necessity, scalability, or practical advantage of the proposed low-level RL tracking formulation.

Questions for authors

1. What specific capabilities does the learned RL controller acquire beyond reference replay?

Could the authors provide a more detailed analysis of the failure modes in which TRACK outperforms replay? Are the improvements primarily due to grasp correction, recovery from execution errors, robustness to reference perturbations, or adaptive temporal alignment?

In particular, could the authors evaluate reference replay equipped with the same progress tracker to separate the contribution of the learned controller from that of reference alignment?

I would also suggest evaluating TRACK on more challenging manipulation tasks where simple trajectory replay is likely to fail, and where closed-loop control and recovery behaviors are genuinely necessary. For example, tasks involving dynamic object interactions, unstable grasps, or pick-and-place with slippery objects could better highlight the advantages of an RL-based controller. These settings would help demonstrate whether TRACK can learn meaningful contact-aware corrections, adapt to unexpected object motion, and recover from execution failures, rather than primarily improving the tracking of slightly perturbed reference trajectories.

A more detailed analysis of these behaviors, including comparisons of end-effector trajectories, object tracking errors, grasp stability, and recovery success rates, would provide stronger evidence for the necessity and practical value of the proposed low-level controller.

2. How should TRACK be integrated with a high-level manipulation policy?

What is the intended role of TRACK in a complete hierarchical manipulation system? How would the high-level policy generate reference trajectories for previously unseen tasks, and what information would it need to provide?

I would be particularly interested in whether a high-level planner combined with TRACK could outperform the same planner combined with conventional reference execution, or a task-conditioned Diffusion Policy, under comparable inference inputs and evaluation conditions.

Even if a full experimental evaluation is beyond the current scope, a concrete discussion of the intended system architecture and its advantages would help clarify the significance of the work.

3. How does the benefit of the learned tracking controller scale with task diversity and complexity?

Does increasing the number and diversity of training tasks improve the generality and robustness of the low-level controller? Is there evidence that the learned controller develops transferable manipulation capabilities rather than primarily learning task-distribution-specific trajectory corrections?

It would also be useful to understand which manipulation regimes most benefit from the proposed formulation, particularly tasks involving complex contacts or significant deviations from reference trajectories.

4. What is the intended pathway to real-world deployment?

How do the authors envision obtaining reliable object keypoint observations and task reference trajectories in real environments? Given that the low-level controller is trained in simulation, what are the anticipated challenges in transferring learned contact and grasp correction behaviors to physical robots?

A concrete discussion of the sim-to-real strategy, and the assumptions required for successful deployment, would strengthen the practical motivation.

7 comments

We thank the reviewer for the detailed feedback. The requested analyses are in blue in the revision.

W1 (motivation). We agree that replay is strong when the reference is exact. Our learned controller matters when the reference cannot simply be executed, for example with approximate end-effector paths, planner outputs that miss the grasp, or a different hand geometry. These are realistic use cases, because a reference from a planner, a VLM, a teleoperator, a human or another robot is rarely exact for the robot that executes it. Our new experiments support the advantages the reviewer anticipates for an RL controller trained in simulation, in particular closed-loop handling of grasps and contacts. Its gain over replay comes from the tasks that replay cannot execute, and mostly from grasps that replay fails to form (Q1). The review also notes that learned tracking is best motivated for dexterous hands. We added such an experiment, in which TRACK is well above replay (Q1, App. C.17). We also ran a TRACK policy trained only in simulation on a real Franka Panda, where it completed cup stacking in five of five rollouts from human demonstrations retargeted from RGB-D videos (Q4, App. D). The revision states this scope.

Q1. What our learned controller adds beyond replay

Replay with the same tracker (App. C.2). All rows use the paper's corrupted references. Replay and the clock policy are as in Table 1, and the two tracker rows use 400 steps on training and held-out references and 500 steps one-shot, which the tracker needs to finish the reference (R25b, Q2):

ML10trainheld-outone-shot
Replay with clock (Table 1's baseline)0.830.800.61
Replay with tracker0.820.810.61
TRACK with clock0.990.880.67
TRACK with tracker0.960.890.77

Under the same tracker, our learned policy is ahead on every split, on 5/5 seeds. A stronger replay that aims one waypoint ahead does engage the tracker, and TRACK with the tracker is still above it by +0.15 / +0.06 / +0.09 (App. C.2). On average, the tracker alone does not help replay. On RoboTwin, replay with the tracker scores 0.64 / 0.67 / 0.53, against TRACK's 0.70 / 0.73 / 0.69.

Where and how the controller wins (App. C.8). TRACK's gain comes from exactly the tasks that replay cannot execute. On every ML10 task where replay often fails, such as the grasp-and-carry tasks basketball and sweep, TRACK succeeds in most episodes, and on the tasks that replay already solves it stays high. The deciding behaviour is the grasp. In the large majority of the episodes that TRACK wins and replay loses on training tasks, and in about two thirds on unseen tasks, TRACK forms a stable grasp that carries the object and replay does not. A repaired path occurs in far fewer of these wins. Timing alone does not explain them either. For the clock policy a shifted timing occurs in a minority of wins, and replay driven by the same tracker gains nothing (table above). On training tasks TRACK follows the end-effector path about as closely as replay, but keeps the object far closer to its reference track (2.3–2.9 vs. 10.2 cm), forms a stable transporting grasp more often (0.94–0.97 vs. 0.76 of episodes) and drops the object less often (App. C.8). Robustness to reference perturbations is in our reply to efAP, Q2.

Q1 (continued)

A dexterous hand (App. C.17). We ported the ML10 setup to a simulated KUKA iiwa arm with a five-finger hand, where the policy commands 7 arm and 22 hand joints. The references are the Sawyer's Meta-World demonstrations, so they contain no finger motion. Because of this high-dimensional action space, we train with REPPO instead of PPO. Replay closes the hand into a fixed power grasp on the grasp flags.

iiwa + five-finger hand, 5 seedstrainheld-outone-shot
Replay with clock0.310.310.40
Replay with tracker0.310.310.39
TRACK with clock0.620.620.59
TRACK with tracker0.610.620.65

TRACK with the tracker is above replay on every split and seed. Here too the gain comes from the grasp. Replay's fixed grasp fails every grasp-and-carry training task with this hand, while TRACK learns finger motions that carry the object. On the unseen tasks, much of the lead comes from the learned finger motion, for example on drawer-open, where the hand hooks the handle that replay misses.

Injected errors and harder conditions (App. C.9, C.10). After an identical push or forced release, TRACK keeps more of its successes than replay in every setting on training and held-out references. With the gripper–object friction cut to a tenth, TRACK stays above replay, and the tracker policy loses less success than replay. When the object is moved once mid-approach, TRACK keeps clearly more of its successes than replay on unseen tasks (+0.26 / +0.35 at 2 cm and +0.16 / +0.17 at 4 cm, clock / tracker). On training and held-out references it has no advantage over replay, and we state a displaced object as a limitation (App. C.9, C.14).

The RoboTwin residual (App. C.5). On RoboTwin, TRACK's policy does not move the arms on its own. A controller follows the reference's end-effector path and orientation and opens and closes the grippers on the grasp flags, and the policy adds a position correction of up to 2 cm per axis. Replay is the same controller without the correction, so Table 2 measures exactly what the learned correction and the progress tracker add. One-shot success rises from 0.57 to 0.69. The orientation is not corrupted and replay executes it in the same way, so it gives TRACK no advantage over replay. The revision states this design explicitly.

Q2. TRACK in a hierarchical system

A planner supplies the task specification of Eq. 1, end-effector waypoints, grasp flags and object keypoint tracks, and TRACK executes it in closed loop. Our results suggest that the end-effector path may be approximate and the object tracks noisy (efAP Q2), but the object tracks should still roughly match the scene (efAP Q1). On ML10's VLM plans (Table 3), TRACK with the tracker is above replay in all four settings at matched horizons, and also above replay with the same tracker (App. C.1).

A task-conditioned Diffusion Policy. We agree that our Diffusion Policy comparison compared implemented systems. We therefore ran three new experiments with it (5 seeds each, App. C.11). They are the paper's Diffusion Policy executing the same VLM plans as replay and TRACK, the same Diffusion Policy retrained with our progress tracker, and a version that also receives TRACK's object keypoint inputs. For the ML10 policies on the VLM plans, at 500 steps (one-shot plans, zero-shot plans in brackets):

ML10, VLM planstraining tasksunseen tasks
Replay0.40 (0.27)0.34 (0.19)
Diffusion Policy0.34 (0.17)0.21 (0.15)
Diffusion Policy + tracker0.53 (0.29)0.35 (0.19)
TRACK (tracker)0.60 (0.35)0.41 (0.24)

The paper's Diffusion Policy is below replay in every setting. With our tracker it improves in all four settings but stays below TRACK. TRACK's object inputs barely change it, on the plans and on expert references, and it stays below replay and TRACK.

Q3. Scaling with task diversity

We trained nested pools of 1, 3, 5 and 10 of ML10's training tasks with the same budget per task and scored them on ML10's five unseen tasks (App. C.12). One-shot success rises from 0.12 / 0.07 with one task (clock / tracker) to 0.67 / 0.77 with ten (the policies of Table 1). Ten tasks still beat one or three by +0.39 to +0.64 when the smaller pools get the same total training, and a single-task policy that solves its own task transfers only 0.00–0.33. On ML45 we extended ML10's ten tasks by 10 and then 20 tasks drawn at random, and trained on all 45. There, transfer to ML45's five unseen tasks does not grow from 10 to 30 tasks (clock 0.63, 0.52, 0.53 and tracker 0.58, 0.59, 0.56) and rises only for the tracker at 45 tasks (0.74). Each ML45 pool size is a single draw of tasks, so we cannot tell whether the rise at 45 comes from the number of tasks or from the tasks added last. So transfer grows steeply up to ten tasks and holds at that level as the pool grows to 30 tasks. The largest gains are on grasp-and-carry tasks (Q1) and with larger reference errors, where TRACK's lead over replay grows (efAP, Q2).

Q4. Pathway to real-world deployment

We have since evaluated a TRACK policy on a real Franka Panda, on cup stacking. The policy is trained only in simulation and is not fine-tuned on the robot (Sec. 4.5 and App. D). Its training pool includes a simulated cup scene, so this tests transfer from simulation to the real system. The transfer rests on a simulator of the same arm (controller fitted to real-arm logs, the real gripper's latency, physics randomization), the same action and observation interfaces as on hardware, and inputs that the real system provides. The reference is a teleoperated demonstration recorded on the real arm, and the four keypoints are clicked by an operator on the cup and tracked online by CoTracker3 in an external RGB-D camera. In first viability tests the system ran in closed loop at about 20 Hz and completed the task (grasp, transport and nesting).

We have also conditioned the same policy, without retraining, on human demonstrations extracted from RGB-D videos. Hand and cup points are tracked offline, grasp and release are labelled by hand, and the path is shifted by one measured offset between the thumb and the gripper's contact point. The robot completed the task in five of five rollouts, one per demonstration and each on the first attempt, with the cup at a different position in every demonstration (App. D, with a video of one of them in the supplementary material). Five rollouts are a first test, not a quantitative evaluation.

In simulation, TRACK is robust to errors in the perceived object position (App. C.13). Errors of up to 2 cm cost it at most 0.03, and even a constant 4 cm offset costs at most 0.10 in distribution. Training with such errors is not needed and does not help. For contact dynamics that differ from the simulator, TRACK stays above replay with the gripper–object friction cut to a tenth or a tenfold object mass (Q1, App. C.10).

Comments on the paper

1 comment

We thank the reviewers for careful and constructive reviews. We ran new experiments for every question and added them to the revised PDF, where changes are colored by reviewer, blue for vYK8, green for R25b, purple for efAP, and orange where a change answers several reviewers.

The main new results:

  1. Robustness to the reference (efAP, vYK8, App. C.13–C.15). On ML10, TRACK with the tracker is above replay under every corruption we tested, from exact references to four times the training magnitude and including types not seen in training. Errors of up to 2 cm in the perceived object position cost it at most 0.03. A reference recorded in another initial state no longer fits the scene and drops TRACK to replay's level, but adapting it to the scene recovers most of the loss.
  2. Other embodiments (efAP, vYK8, App. C.16, C.17). Executing Meta-World's Sawyer demonstrations, TRACK is above replay on average on a simulated Franka Panda, and well above it on a simulated KUKA iiwa arm with a five-finger hand, where the demonstrations contain no finger motion.
  3. Further baselines and ablations (R25b, vYK8, App. C.4, C.6, C.7, C.11, C.12). TRACK is well above a DCRL-style policy trained in our stack when the reference matches the scene, and, with the reference adapted to the scene, above both DCRL-style arms with random initial states. On unseen tasks it is also above per-task Scaffolding-style specialists. Diffusion Policy variants stay below TRACK on the VLM plans. Transfer to unseen tasks grows with the number of training tasks up to ten, and the end-effector path matters mainly as a reward, while observing it adds most on unseen tasks.
  4. Sim-to-real deployment and retargeting from human demonstrations (vYK8, efAP, Sec. 4.5, App. D). A TRACK policy trained only in simulation completed cup stacking on a real Franka Panda, conditioned on a teleoperated demonstration with camera-tracked keypoints. Without retraining, the same policy also completed the task in five of five rollouts from human demonstrations retargeted from RGB-D videos, one per demonstration, each recorded with the cup at a different position.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.