- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes T3DP, a framework designed to generalize robotic manipulation to unseen behavioral specifications within known task families, such as target positions, displacements, and articulation extents. It learns behavior representations through reward prediction, aligns a CLIP text encoder with frozen behavior features using global contrastive learning and a bidirectional late-interaction objective, and conditions a point-cloud diffusion policy on the aligned language representations to generate actions for novel behavioral specifications.
Critical strengths
- A well-controlled comparison with global alignment. The problem setting is relevant, particularly when similar observations require different actions depending on the instruction. The comparison with T2DA is convincing. According to Appendix A.5, both methods share the frozen behavior encoder, text initialization, alignment data, downstream policy, and training budget. The main difference is the local alignment objective, which brings consistent improvements across all 15 task families in the separate-family experiments.
- Useful analyses beyond downstream success rates. The ablation study clearly separates the effects of language conditioning, global alignment, and the proposed local alignment. I also appreciate the representation and same-state action probes, which examine whether the learned language features retain information relevant to different specifications.
Critical weaknesses
- The evidence for token-level behavior grounding remains indirect. The results support the benefit of local alignment, but do not directly establish correspondences between specific linguistic elements and the behavioral components they affect. A direct correspondence analysis or a controlled experiment replacing local behavior features with repeated pooled features would help clarify whether the improvements come from meaningful local grounding or additional contrastive supervision.
- The real-robot training details and evaluation uncertainty need clarification. Section 4.2 describes reward prediction as the behavior-learning objective, and Appendix F identifies reward-annotated trajectories as a requirement. However, Appendix E.4 does not explain how reward labels are obtained from the physical demonstrations. In addition, each real-robot task is evaluated on only 20 held-out targets, without reported uncertainty estimates. This makes it difficult to assess the reliability of the observed improvements.
Questions for authors
- Did the real-robot experiments use the reward-prediction objective described in Section 4.2? How were the rewards obtained, and did T2DA and T3DP use the same supervision?
- Can the authors provide direct evidence of token-level language–behavior correspondence? For example, how does the alignment change when the target position or displacement changes?
- Can the authors further isolate the effect of local alignment? For example, what happens when local behavior features are replaced with repeated pooled features, or when only state or action tokens are used?
- Why are recent VLA models such as pi_0 and pi_0.5 not included, especially on RoboTwin?
- Do the improvements over global alignment remain consistent across different training and test specification splits, rather than only the fixed splits used in the main experiments?
- Were the 20 real-robot trials in Figure 5 obtained from a single trained policy or multiple seeds? Could the authors provide uncertainty estimates or per-target results?
4 comments
Author Response (Part 2/3)
W2(a) & Q1: Section 4.2 describes reward prediction as the behavior-learning objective, but Appendix E.4 does not explain how reward labels are obtained from physical demonstrations. Did the real-robot experiments use this objective, how were the rewards obtained, and did T2DA and T3DP use the same supervision?
A2(a). The original real-robot experiments use the reward-prediction objective in Section 4.2. We collect demonstrations by replaying simulated expert trajectories on the physical robot and retaining successful executions that satisfy the requested specifications. Real observations, robot states, and executed actions are recorded during replay. The simulation provides the reference motion, while behavior learning uses the physical execution records.
Per-step reward labels are computed offline from these physical records using task-specific geometric rules. Following the simulation reward design, we estimate the geometric quantities required by the reward rubric from recorded robot motion. Gripper poses at grasping and placement events, expressed in a common robot coordinate frame, provide approximate object and target locations. For example, in Move Pill Bottle, the bottle and target positions are estimated using gripper positions, and these estimates and recorded robot states are used to compute per-step rewards. The labels are recomputed from physical execution rather than copied from the simulated reference.
T2DA and T3DP use identical reward labels and the same frozen behavior-encoder checkpoint. This keeps behavior supervision fixed while comparing global and fine-grained language–behavior alignment. Reward labels are used during representation learning and are not required at deployment. Next-state prediction, with reward inputs removed, is evaluated separately as a reward-free alternative; it is not the objective used for the original real-robot results. We will document the reward rules, information sources, and geometric assumptions in Appendix E.4.
W2(b) & Q6: Each real-robot task is evaluated on only 20 held-out targets without reported uncertainty estimates, making the reliability of the improvements difficult to assess. Were the Figure 5 results obtained from a single trained policy or multiple seeds, and can the authors provide uncertainty estimates or per-target results?
A2(b). Figure 5 originally reports one training seed; we extend the evaluation to three seeds per method. We train and evaluate T3DP and T2DA with two additional random seeds under the same real-robot protocol. With 20 held-out target specifications per task and one execution per target for each seed, this gives 60 executions per method and task on the same 20 targets. A trial succeeds when the final object position lies within 3 cm of the instructed target in the tabletop plane.
T3DP achieves higher mean success on both tasks across the three training seeds. The table reports success rates (%) as the mean and standard deviation of the three per-seed success rates. T3DP improves mean success by 20.0 percentage points on Move Pill Bottle and 11.7 points on Move Stapler.
| Task | T2DA | T3DP |
|---|---|---|
| Move Pill Bottle | 43.3 ± 7.6 | 63.3 ± 2.9 |
| Move Stapler | 50.0 ± 0.0 | 61.7 ± 2.9 |
The reported uncertainty quantifies training-seed variability on the fixed target set. The 60 executions are repeated evaluations of 20 distinct specifications, and the standard deviations summarize variation among the three trained policies. Together, the higher means and observed across-seed variability support the consistency of the gains under this evaluation protocol. We will update Figure 5 and Appendix E.5 with the multi-seed results and evaluation details.
Author Response (Part 3/3)
Q4: Why are recent VLA models such as pi_0 and pi_0.5 not included, especially on RoboTwin?
A4. Our original baselines prioritize isolating the alignment contribution, and the added VLA comparison assesses practical performance. In the limited-demonstration setting, our alignment controls share the behavior encoder, demonstrations, downstream architecture, and training budget while varying the alignment objective. Following your suggestion, we additionally evaluate on RoboTwin using the same downstream demonstrations, instructions, held-out specification splits, and success criteria.
The adapted achieves 43.0% average success, compared with 41.8% for T3DP. We start from a -base checkpoint and evaluate it before and after further fine-tuning on our specification-conditioned demonstrations with 100 epochs. T3DP achieves higher mean success on Move Pill Bottle (39.3% versus 28.3%) and Move Playing Card (52.3% versus 40.0%), while the adapted VLA performs better on the other three tasks. We will document pretraining and input-modality differences alongside the matched downstream protocol.
| RoboTwin | DP3 | T3DP | -base | (fine-tuning) |
|---|---|---|---|---|
| Open Microwave | 15.7 ± 8.6 | 52.0 ± 13.5 | 18.3 ± 16.1 | 60.0 ± 8.7 |
| Open Cabinet Drawer | 5.0 ± 1.3 | 37.3 ± 0.6 | 21.7 ± 2.9 | 41.7 ± 2.9 |
| Move Pill Bottle | 3.7 ± 1.3 | 39.3 ± 0.5 | 0.0 ± 0.0 | 28.3 ± 5.8 |
| Move Playing Card | 9.0 ± 2.2 | 52.3 ± 1.5 | 0.0 ± 0.0 | 40.0 ± 8.7 |
| Move Stapler | 2.7 ± 2.4 | 28.0 ± 1.6 | 0.0 ± 0.0 | 45.0 ± 8.7 |
| Avg. | 7.2 | 41.8 | 8.0 | 43.0 |
T3DP approaches the adapted VLA's mean success with approximately one-tenth of its deployment parameters. T3DP deploys only its aligned text encoder and diffusion policy (323.12M parameters), compared with 3.35B for (10.4× fewer). Its behavior encoder and reward decoder are used only during training. For , we freeze the vision and language backbones and fine-tune the action branch (430.10M trainable parameters). We will include the detailed protocol and parameter breakdown in Appendix D.8 of the revised manuscript.
Q5: Do the improvements over global alignment remain consistent across different training and test specification splits, rather than only the fixed splits used in the main experiments?
A5. We test robustness to the specification partition using two additional independently sampled train–test splits. We compare T3DP with global-only alignment (T2DA) on Reach, Push, and Door Open under the original split and two new splits. Training and test set sizes and evaluation conditions are kept fixed, and both methods use the same data within each split. Each partition is applied consistently to behavior learning, language alignment, and policy training.
T3DP achieves higher mean success in all nine task–split comparisons. The gains range from 5.4 to 22.4 percentage points across the three tasks and three splits. Although absolute success rates vary with the partition, the direction of the improvement is consistent across every evaluated task–split pair. The table reports the results separately for each split.
| Task | T2DA (Original) | T3DP (Original) | T2DA (Split 1) | T3DP (Split 1) | T2DA (Split 2) | T3DP (Split 2) |
|---|---|---|---|---|---|---|
| Push | 45.7 ± 8.6 | 64.7 ± 17.9 | 53.3 ± 2.9 | 66.7 ± 2.3 | 40.0 ± 0.0 | 57.7 ± 6.8 |
| Reach | 24.3 ± 4.0 | 38.3 ± 2.9 | 18.3 ± 3.0 | 40.7 ± 3.0 | 19.0 ± 2.6 | 31.7 ± 2.9 |
| Door Open | 41.3 ± 8.3 | 59.7 ± 4.2 | 46.3 ± 2.5 | 51.7 ± 5.5 | 35.3 ± 18.2 | 57.0 ± 24.6 |
The training-coverage study provides complementary evidence. With 10, 20, and 40 training specifications per task and a fixed test set, T3DP improves over global alignment by 8.5, 14.2, and 7.2 points, respectively (Appendix D.1.1, Table 6). Together with the new split results, these findings support robustness across specification coverage levels and train–test partitions. We will include the new results and protocol in Appendix D.1.2.
Summary Response
Thank you for recognizing the controlled comparison and representation analyses. We add a coordinate-token pairing analysis and matched-objective controls to assess specification-sensitive matching and isolate the benefit of distinct local features beyond additional contrastive supervision (W1, Q2–3). We clarify real-robot reward construction and shared supervision, and report three-seed results with gains on both tasks (W2, Q1, Q6). The added RoboTwin comparison shows T3DP approaching adapted ’s average success with approximately one-tenth of its deployment parameters (Q4). Across three specification splits, T3DP improves mean success in all nine task–split comparisons (Q5). We will incorporate the additional evidence and detailed protocols into the revision and would appreciate your reconsideration.
Looking forward to further discussions
Dear Reviewer VUJQ,
Thank you for your continued effort in reviewing our work. We were wondering if our response and revision have resolved your concerns. We have addressed your initial questions through our rebuttal and are eager to clarify any further points you might raise. Please feel free to provide additional feedback. We greatly appreciate your continued engagement.
Best regards,
Authors
Author Response (Part 1/3)
W1(a) & Q2: The evidence for token-level behavior grounding remains indirect. Can the authors provide more direct evidence of correspondence between specific linguistic elements and robot behavior, and show how the alignment changes when the target position or displacement changes?A1(a). A controlled pairing analysis tests whether coordinate-token matching responds to a change in the target specification. We compare two pick-and-place demonstrations starting from the same initial state. Both instructions ask the robot to “pick up the puck and place its center” at a specified location; their targets are A: (0.06,0.83,0.13) m and B: (−0.09,0.83,0.13) m. Only the x-coordinate changes. We encode each full instruction and then select its contextualized x-coordinate token representations Ex for the diagnostic comparison.
Both methods prefer the matched trajectory, with T3DP showing greater separation in this example. For Ex and the projected trajectory state–action features Z, we use the directional log-mean-exp scores from Section 4.3 with β=20 (matching temperature 0.05), and report ssym(Z,Ex)=21[Sτ→l(Z,Ex)+Sl→τ(Z,Ex)]. Unlike training, this diagnostic uses only the selected coordinate tokens on the language side. The table reports the resulting scores and the matched-minus-mismatched margin for each instruction; bold scores identify the matched pairs. Changing the coordinate changes the preferred trajectory for both methods, while T3DP exhibits larger matching margins.
This case provides evidence of specification-sensitive token–trajectory matching, complemented by representation visualization. The score aggregates features across the complete trajectory, so its interpretation is a target-dependent preference among trajectories. A within-task drawer-opening t-SNE visualization provides complementary qualitative evidence of how text representations vary with the desired opening percentage. We will include the pairing analysis and the supporting t-SNE visualization in Section 5.4 of the revised manuscript (Figure 5 for the visualization).
W1(b) & Q3: Do the improvements come from meaningful local behavior grounding or additional contrastive supervision? Can the authors isolate the effect of local alignment by replacing local behavior features with repeated pooled features, or by using only state- or action-token outputs?A1(b). Matched-objective controls separate the effect of local behavior features from additional contrastive supervision. We compare global-only alignment with repeated-pooled, state-token-output-only, action-token-output-only, and full T3DP variants. The repeated-pooled variant retains individual instruction tokens and the local contrastive objective but replaces each local behavior feature with the same mean-pooled trajectory representation. The state/action variants select the corresponding contextualized outputs from the jointly encoded trajectory rather than removing the other input modality.
Distinct local behavior features improve mean success beyond the repeated-pooled control. Repeated-pooled features already outperform global-only alignment on all three tasks, supporting a contribution from the added token-level objective. Full T3DP further improves mean success by 3.0, 6.7, and 7.7 percentage points on Reach, Door Open, and Push, respectively. These comparisons support an additional benefit from retaining local behavior differences under the local contrastive objective. The table also includes the state/action-output controls.
Retaining both contextualized state and action outputs yields the highest mean success across these controls. The state-output-only and action-output-only variants each improve over global-only alignment but remain below full T3DP in mean success on all three tasks. This supports using both output types to capture physical configurations and the actions associated with them. Together, the controls distinguish the benefit of the added objective from the further gains associated with distinct local features and joint state–action output selection. We will document the control definitions and training settings in the revised manuscript.