acceptodds
Under review as a conference paper at ICLR 2027

Text-to-3D Policy: Fine-Grained Language-Behavior Alignment for Unseen Specification Generalization

Abstract

3D visuomotor policies provide a strong foundation for spatially precise manipulation, yet current text-to-3D policies struggle to follow unseen fine-grained behavioral specifications beyond those covered by demonstrations. We study this challenge as *unseen specification generalization*, where language specifies behaviorally significant variations, such as target position, displacement, or articulated state, that are absent from policy training. We find that pretrained language representations and conventional global behavior-language alignment capture coarse task semantics but often blur nearby specifications that require distinct behaviors. We introduce **T3DP**, a **T**ext-to-**3D** **P**olicy framework for fine-grained language-behavior alignment. Rather than compressing each instruction and demonstration into a single global embedding, T3DP preserves their local structures and establishes bidirectional token-level correspondence between linguistic elements and behavioral segments. This directly grounds subtle linguistic variations in the behavior components they affect, preventing closely related specifications from collapsing in the representation space. The resulting specification-sensitive language representation conditions a point-cloud-based 3D diffusion policy, enabling more precise control over unseen behavioral specifications without modifying the underlying policy architecture. Across Meta-World, ManiSkill, and RoboTwin, T3DP improves average held-out-specification success over global language-behavior alignment by **+11.0-14.2 points**, with gains on all 15 task families; on real-robot tasks, it further raises average success from 47.5% to 65.0% (**+17.5 points**). Representation and action-probe analyses show that fine-grained alignment better preserves specification geometry and action-relevant variation, linking local behavior grounding to downstream control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Language-Conditioned Robot Manipulation, Language-Behavior Alignment, 3D Policy, Diffusion Policy

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VUJQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes T3DP, a framework designed to generalize robotic manipulation to unseen behavioral specifications within known task families, such as target positions, displacements, and articulation extents. It learns behavior representations through reward prediction, aligns a CLIP text encoder with frozen behavior features using global contrastive learning and a bidirectional late-interaction objective, and conditions a point-cloud diffusion policy on the aligned language representations to generate actions for novel behavioral specifications.

Critical strengths
  1. A well-controlled comparison with global alignment. The problem setting is relevant, particularly when similar observations require different actions depending on the instruction. The comparison with T2DA is convincing. According to Appendix A.5, both methods share the frozen behavior encoder, text initialization, alignment data, downstream policy, and training budget. The main difference is the local alignment objective, which brings consistent improvements across all 15 task families in the separate-family experiments.
  2. Useful analyses beyond downstream success rates. The ablation study clearly separates the effects of language conditioning, global alignment, and the proposed local alignment. I also appreciate the representation and same-state action probes, which examine whether the learned language features retain information relevant to different specifications.
Critical weaknesses
  1. The evidence for token-level behavior grounding remains indirect. The results support the benefit of local alignment, but do not directly establish correspondences between specific linguistic elements and the behavioral components they affect. A direct correspondence analysis or a controlled experiment replacing local behavior features with repeated pooled features would help clarify whether the improvements come from meaningful local grounding or additional contrastive supervision.
  2. The real-robot training details and evaluation uncertainty need clarification. Section 4.2 describes reward prediction as the behavior-learning objective, and Appendix F identifies reward-annotated trajectories as a requirement. However, Appendix E.4 does not explain how reward labels are obtained from the physical demonstrations. In addition, each real-robot task is evaluated on only 20 held-out targets, without reported uncertainty estimates. This makes it difficult to assess the reliability of the observed improvements.
Questions for authors
  1. Did the real-robot experiments use the reward-prediction objective described in Section 4.2? How were the rewards obtained, and did T2DA and T3DP use the same supervision?
  2. Can the authors provide direct evidence of token-level language–behavior correspondence? For example, how does the alignment change when the target position or displacement changes?
  3. Can the authors further isolate the effect of local alignment? For example, what happens when local behavior features are replaced with repeated pooled features, or when only state or action tokens are used?
  4. Why are recent VLA models such as pi_0 and pi_0.5 not included, especially on RoboTwin?
  5. Do the improvements over global alignment remain consistent across different training and test specification splits, rather than only the fixed splits used in the main experiments?
  6. Were the 20 real-robot trials in Figure 5 obtained from a single trained policy or multiple seeds? Could the authors provide uncertainty estimates or per-target results?
4 comments
AuthorsOpenReview ↗

Author Response (Part 1/3)


W1(a) & Q2: The evidence for token-level behavior grounding remains indirect. Can the authors provide more direct evidence of correspondence between specific linguistic elements and robot behavior, and show how the alignment changes when the target position or displacement changes?

A1(a). A controlled pairing analysis tests whether coordinate-token matching responds to a change in the target specification. We compare two pick-and-place demonstrations starting from the same initial state. Both instructions ask the robot to “pick up the puck and place its center” at a specified location; their targets are A: (0.06,0.83,0.13)(0.06,0.83,0.13) m and B: (−0.09,0.83,0.13)(-0.09,0.83,0.13) m. Only the x-coordinate changes. We encode each full instruction and then select its contextualized x-coordinate token representations ExE_x for the diagnostic comparison.

Both methods prefer the matched trajectory, with T3DP showing greater separation in this example. For ExE_x and the projected trajectory state–action features ZZ, we use the directional log-mean-exp scores from Section 4.3 with β=20\beta=20 (matching temperature 0.05), and report ssym(Z,Ex)=12[Sτ→l(Z,Ex)+Sl→τ(Z,Ex)]s_{\mathrm{sym}}(Z,E_x)=\frac{1}{2}[S_{\tau\rightarrow l}(Z,E_x)+S_{l\rightarrow\tau}(Z,E_x)]. Unlike training, this diagnostic uses only the selected coordinate tokens on the language side. The table reports the resulting scores and the matched-minus-mismatched margin for each instruction; bold scores identify the matched pairs. Changing the coordinate changes the preferred trajectory for both methods, while T3DP exhibits larger matching margins.

MethodSelected instruction tokensA-behaviorB-behaviorMatching margin
T2DAA-coordinate0.02450.02200.0025
T2DAB-coordinate0.01950.02370.0042
T3DPA-coordinate0.1388−0.12860.2674
T3DPB-coordinate−0.08400.19280.2768

This case provides evidence of specification-sensitive token–trajectory matching, complemented by representation visualization. The score aggregates features across the complete trajectory, so its interpretation is a target-dependent preference among trajectories. A within-task drawer-opening t-SNE visualization provides complementary qualitative evidence of how text representations vary with the desired opening percentage. We will include the pairing analysis and the supporting t-SNE visualization in Section 5.4 of the revised manuscript (Figure 5 for the visualization).


W1(b) & Q3: Do the improvements come from meaningful local behavior grounding or additional contrastive supervision? Can the authors isolate the effect of local alignment by replacing local behavior features with repeated pooled features, or by using only state- or action-token outputs?

A1(b). Matched-objective controls separate the effect of local behavior features from additional contrastive supervision. We compare global-only alignment with repeated-pooled, state-token-output-only, action-token-output-only, and full T3DP variants. The repeated-pooled variant retains individual instruction tokens and the local contrastive objective but replaces each local behavior feature with the same mean-pooled trajectory representation. The state/action variants select the corresponding contextualized outputs from the jointly encoded trajectory rather than removing the other input modality.

Distinct local behavior features improve mean success beyond the repeated-pooled control. Repeated-pooled features already outperform global-only alignment on all three tasks, supporting a contribution from the added token-level objective. Full T3DP further improves mean success by 3.0, 6.7, and 7.7 percentage points on Reach, Door Open, and Push, respectively. These comparisons support an additional benefit from retaining local behavior differences under the local contrastive objective. The table also includes the state/action-output controls.

TaskT2DA (global-only)State-token outputsAction-token outputsRepeated pooled featuresT3DP
Reach24.3 ± 4.029.7 ± 5.030.0 ± 10.035.3 ± 0.638.3 ± 2.9
Door Open41.3 ± 8.356.0 ± 6.251.0 ± 5.353.0 ± 2.659.7 ± 4.2
Push45.7 ± 8.659.7 ± 0.658.7 ± 15.357.0 ± 2.664.7 ± 17.9

Retaining both contextualized state and action outputs yields the highest mean success across these controls. The state-output-only and action-output-only variants each improve over global-only alignment but remain below full T3DP in mean success on all three tasks. This supports using both output types to capture physical configurations and the actions associated with them. Together, the controls distinguish the benefit of the added objective from the further gains associated with distinct local features and joint state–action output selection. We will document the control definitions and training settings in the revised manuscript.

AuthorsOpenReview ↗

Author Response (Part 2/3)


W2(a) & Q1: Section 4.2 describes reward prediction as the behavior-learning objective, but Appendix E.4 does not explain how reward labels are obtained from physical demonstrations. Did the real-robot experiments use this objective, how were the rewards obtained, and did T2DA and T3DP use the same supervision?

A2(a). The original real-robot experiments use the reward-prediction objective in Section 4.2. We collect demonstrations by replaying simulated expert trajectories on the physical robot and retaining successful executions that satisfy the requested specifications. Real observations, robot states, and executed actions are recorded during replay. The simulation provides the reference motion, while behavior learning uses the physical execution records.

Per-step reward labels are computed offline from these physical records using task-specific geometric rules. Following the simulation reward design, we estimate the geometric quantities required by the reward rubric from recorded robot motion. Gripper poses at grasping and placement events, expressed in a common robot coordinate frame, provide approximate object and target locations. For example, in Move Pill Bottle, the bottle and target positions are estimated using gripper positions, and these estimates and recorded robot states are used to compute per-step rewards. The labels are recomputed from physical execution rather than copied from the simulated reference.

T2DA and T3DP use identical reward labels and the same frozen behavior-encoder checkpoint. This keeps behavior supervision fixed while comparing global and fine-grained language–behavior alignment. Reward labels are used during representation learning and are not required at deployment. Next-state prediction, with reward inputs removed, is evaluated separately as a reward-free alternative; it is not the objective used for the original real-robot results. We will document the reward rules, information sources, and geometric assumptions in Appendix E.4.


W2(b) & Q6: Each real-robot task is evaluated on only 20 held-out targets without reported uncertainty estimates, making the reliability of the improvements difficult to assess. Were the Figure 5 results obtained from a single trained policy or multiple seeds, and can the authors provide uncertainty estimates or per-target results?

A2(b). Figure 5 originally reports one training seed; we extend the evaluation to three seeds per method. We train and evaluate T3DP and T2DA with two additional random seeds under the same real-robot protocol. With 20 held-out target specifications per task and one execution per target for each seed, this gives 60 executions per method and task on the same 20 targets. A trial succeeds when the final object position lies within 3 cm of the instructed target in the tabletop plane.

T3DP achieves higher mean success on both tasks across the three training seeds. The table reports success rates (%) as the mean and standard deviation of the three per-seed success rates. T3DP improves mean success by 20.0 percentage points on Move Pill Bottle and 11.7 points on Move Stapler.

TaskT2DAT3DP
Move Pill Bottle43.3 ± 7.663.3 ± 2.9
Move Stapler50.0 ± 0.061.7 ± 2.9

The reported uncertainty quantifies training-seed variability on the fixed target set. The 60 executions are repeated evaluations of 20 distinct specifications, and the standard deviations summarize variation among the three trained policies. Together, the higher means and observed across-seed variability support the consistency of the gains under this evaluation protocol. We will update Figure 5 and Appendix E.5 with the multi-seed results and evaluation details.

AuthorsOpenReview ↗

Author Response (Part 3/3)


Q4: Why are recent VLA models such as pi_0 and pi_0.5 not included, especially on RoboTwin?

A4. Our original baselines prioritize isolating the alignment contribution, and the added VLA comparison assesses practical performance. In the limited-demonstration setting, our alignment controls share the behavior encoder, demonstrations, downstream architecture, and training budget while varying the alignment objective. Following your suggestion, we additionally evaluate π0.5\pi_{0.5} on RoboTwin using the same downstream demonstrations, instructions, held-out specification splits, and success criteria.

The adapted π0.5\pi_{0.5} achieves 43.0% average success, compared with 41.8% for T3DP. We start from a π0.5\pi_{0.5}-base checkpoint and evaluate it before and after further fine-tuning on our specification-conditioned demonstrations with 100 epochs. T3DP achieves higher mean success on Move Pill Bottle (39.3% versus 28.3%) and Move Playing Card (52.3% versus 40.0%), while the adapted VLA performs better on the other three tasks. We will document pretraining and input-modality differences alongside the matched downstream protocol.

RoboTwinDP3T3DPπ0.5\pi_{0.5}-baseπ0.5\pi_{0.5} (fine-tuning)
Open Microwave15.7 ± 8.652.0 ± 13.518.3 ± 16.160.0 ± 8.7
Open Cabinet Drawer5.0 ± 1.337.3 ± 0.621.7 ± 2.941.7 ± 2.9
Move Pill Bottle3.7 ± 1.339.3 ± 0.50.0 ± 0.028.3 ± 5.8
Move Playing Card9.0 ± 2.252.3 ± 1.50.0 ± 0.040.0 ± 8.7
Move Stapler2.7 ± 2.428.0 ± 1.60.0 ± 0.045.0 ± 8.7
Avg.7.241.88.043.0

T3DP approaches the adapted VLA's mean success with approximately one-tenth of its deployment parameters. T3DP deploys only its aligned text encoder and diffusion policy (323.12M parameters), compared with 3.35B for π0.5\pi_{0.5} (10.4× fewer). Its behavior encoder and reward decoder are used only during training. For π0.5\pi_{0.5}, we freeze the vision and language backbones and fine-tune the action branch (430.10M trainable parameters). We will include the detailed protocol and parameter breakdown in Appendix D.8 of the revised manuscript.


Q5: Do the improvements over global alignment remain consistent across different training and test specification splits, rather than only the fixed splits used in the main experiments?

A5. We test robustness to the specification partition using two additional independently sampled train–test splits. We compare T3DP with global-only alignment (T2DA) on Reach, Push, and Door Open under the original split and two new splits. Training and test set sizes and evaluation conditions are kept fixed, and both methods use the same data within each split. Each partition is applied consistently to behavior learning, language alignment, and policy training.

T3DP achieves higher mean success in all nine task–split comparisons. The gains range from 5.4 to 22.4 percentage points across the three tasks and three splits. Although absolute success rates vary with the partition, the direction of the improvement is consistent across every evaluated task–split pair. The table reports the results separately for each split.

TaskT2DA (Original)T3DP (Original)T2DA (Split 1)T3DP (Split 1)T2DA (Split 2)T3DP (Split 2)
Push45.7 ± 8.664.7 ± 17.953.3 ± 2.966.7 ± 2.340.0 ± 0.057.7 ± 6.8
Reach24.3 ± 4.038.3 ± 2.918.3 ± 3.040.7 ± 3.019.0 ± 2.631.7 ± 2.9
Door Open41.3 ± 8.359.7 ± 4.246.3 ± 2.551.7 ± 5.535.3 ± 18.257.0 ± 24.6

The training-coverage study provides complementary evidence. With 10, 20, and 40 training specifications per task and a fixed test set, T3DP improves over global alignment by 8.5, 14.2, and 7.2 points, respectively (Appendix D.1.1, Table 6). Together with the new split results, these findings support robustness across specification coverage levels and train–test partitions. We will include the new results and protocol in Appendix D.1.2.


Summary Response

Thank you for recognizing the controlled comparison and representation analyses. We add a coordinate-token pairing analysis and matched-objective controls to assess specification-sensitive matching and isolate the benefit of distinct local features beyond additional contrastive supervision (W1, Q2–3). We clarify real-robot reward construction and shared supervision, and report three-seed results with gains on both tasks (W2, Q1, Q6). The added RoboTwin comparison shows T3DP approaching adapted π0.5\pi_{0.5}’s average success with approximately one-tenth of its deployment parameters (Q4). Across three specification splits, T3DP improves mean success in all nine task–split comparisons (Q5). We will incorporate the additional evidence and detailed protocols into the revision and would appreciate your reconsideration.

AuthorsOpenReview ↗

Looking forward to further discussions

Dear Reviewer VUJQ,

Thank you for your continued effort in reviewing our work. We were wondering if our response and revision have resolved your concerns. We have addressed your initial questions through our rebuttal and are eager to clarify any further points you might raise. Please feel free to provide additional feedback. We greatly appreciate your continued engagement.

Best regards,

Authors

Reviewer_B1qhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes T3DP, which uses fine-grained language-behavior alignment to improve generalization to unseen behavior specifications. It shows improvements on Meta-World, ManiSkill, RoboTwin, and real robots.

Critical strengths

1.Meaningful problem. It focuses on generalization within the same task, such as different positions and distances.​

  1. Good experiments. The method is evaluated on multiple benchmarks and real robots, with consistent improvements.​

  2. Good ablations. The paper compares different alignment methods and analyzes learned representations.

Critical weaknesses
  1. Limited novelty. Token-level contrastive alignment has been studied before. The main contribution is applying it to robot behaviors.

​2. Limited language generalization. All instructions use fixed templates, so it mainly tests numerical specifications rather than real language generalization.​

  1. Reward dependency. The behavior encoder needs reward supervision, which may limit its use on real-world robot data.​

  2. Limited generalization. All task categories appear during training. Unseen tasks and compositional generalization are not tested.​

  3. Missing ablations. It is unclear whether the improvement comes from fine-grained alignment or additional contrastive learning.

Questions for authors

1.What is the main novelty compared with existing token-level alignment methods like FILIP?​

  1. Can the authors test free-form language or paraphrased instructions?

​3. What happens if reward supervision is removed?

​4. Can the authors compare token-level, segment-level, and trajectory-level alignment?​

  1. Can the method generalize to unseen task categories or multiple changing factors?
6 comments
AuthorsOpenReview ↗

Author Response (Part 1/3)


W1 & Q1: What does T3DP contribute beyond existing token-level alignment methods such as FILIP?

A1. T3DP studies a specific control challenge: preserving behavioral distinctions between closely related instructions. In unseen specification generalization, similar observations and task descriptions can require different actions because the target, displacement, or desired state changes. Our preliminary study identifies strong reliance on visual target cues and incomplete preservation of specification geometry under global alignment (Table 1). This motivates studying which language-behavior correspondences are needed to execute held-out specifications, beyond recognizing coarse task semantics.

The contribution is a behavior-grounded representation design for specification-sensitive 3D control. We build on the token-level late-interaction principle established by FILIP, which we cited in Related Work and we do not claim this matching primitive as new. T3DP learns contextualized state-action representations from demonstrations and aligns instruction tokens with these local behavior features alongside global alignment. This supervision encourages specification-defining language to retain information about the physical configurations and actions through which a specification is realized. The resulting pooled language representation then conditions the 3D policy, connecting local training supervision to downstream action generation.

T3DP achieves the highest benchmark-average success among all evaluated baselines. Across Meta-World, ManiSkill, and RoboTwin, it exceeds the strongest baseline in each benchmark by 14.2, 13.5, and 11.0 points, respectively, with further benefits on real-robot tasks. Our representation controls isolate the alignment design while sharing the behavior encoder, data, policy architecture, and training budget. The repeated-pooled control retains the local contrastive objective while removing local behavior differences. Although this control already improves over global-only alignment, distinct local behavior features provide a further 3.0-7.7-point gain in mean success across three tested tasks. State/action-token controls, representation analyses, and same-state action probes provide complementary evidence.

The design improves instruction conditioning while preserving the existing policy interface. Local behavior matching is used during representation learning and alignment. At deployment, T3DP uses the aligned text encoder and 3D diffusion policy with the current observation and instruction, without demonstrations, the behavior encoder, or online trajectory matching. Its simplicity therefore provides a practical way to improve specification-sensitive control without redesigning the downstream policy.


W2 & Q2: Can T3DP generalize beyond fixed instruction templates to unseen paraphrases or free-form instructions?

A2. We extend the specification-generalization evaluation with a direct test of unseen linguistic expressions. Our main experiments use templated instructions to isolate generalization to held-out behavioral specifications. Following your suggestion, we additionally evaluate the existing policies on semantically equivalent unseen paraphrases without retraining or adaptation, keeping the requested specifications and model parameters fixed. This tests whether the learned language condition remains effective when the wording changes but the intended behavior does not.

T3DP retains a clear advantage under unseen paraphrases and shows a smaller average performance drop. It outperforms T2DA by 19.4, 24.5, and 18.9 percentage points on Reach, Door Open, and Push, respectively. Averaging equally across the three tasks, success decreases by 4.6 points for T3DP, compared with 8.4 points for T2DA. Performance is particularly well preserved on Reach and Door Open, supporting the robustness of fine-grained alignment to the tested linguistic variations.

TaskT2DA (original instruction)T3DP (original instruction)T2DA (unseen paraphrases)T3DP (unseen paraphrases)
Reach24.3 ± 4.038.3 ± 2.916.7 ± 1.736.1 ± 2.6
Door Open41.3 ± 8.359.7 ± 4.234.4 ± 9.558.9 ± 5.9
Push45.7 ± 8.664.7 ± 17.935.0 ± 8.753.9 ± 13.9

The evaluation tests whether the learned language condition remains effective across equivalent expressions. The original evaluation uses one instruction wording × three training seeds, and the paraphrase evaluation uses three wordings × three seeds, reporting mean ± std across all nine runs. We will include actual original-paraphrase examples and the full evaluation protocol in the revision. These results show that T3DP preserves its performance advantage when the same behavioral specifications are expressed through unseen, semantically equivalent instructions.

AuthorsOpenReview ↗

Author Response (Part 2/3)


W3 & Q3: Can the behavior encoder be trained without reward annotations, and how does removing reward supervision affect performance?

A3. Behavior learning can use reward or transition supervision. Our original reward-prediction objective requires reward annotations, but alternative objectives can supply behavior representations for language-behavior alignment. The factorization pσ(s′,r∣s,a)=P(s′∣s,a) pσ(r∣s,a,s′)p_\sigma(s',r\mid s,a)=P(s'\mid s,a)\,p_\sigma(r\mid s,a,s') distinguishes physical transition information from task-dependent reward information. Available rewards provide direct supervision about task objectives; without them, recorded state transitions remain a source of self-supervision.

Next-state prediction provides a concrete reward-free alternative. We replace reward prediction with prediction of a recorded subsequent state and remove reward tokens from the encoder input. This requires temporally aligned state–action transitions and paired instructions for language alignment, but no reward labels. We retain the subsequent alignment and policy-learning stages and apply the same supervision choice to T3DP and T2DA.

Additional experiments show that the advantage of fine-grained alignment persists without rewards. Across three seeds, reward-free T3DP achieves 48.3% success on both Move Pill Bottle and Move Stapler, exceeding T2DA by 11.6 and 8.3 percentage points. It remains 15.0 and 13.4 points below reward-supervised T3DP, respectively. Since specifications share physical dynamics, next-state prediction need not preserve the same task information as reward prediction. These results support a feasible reward-free option on the two evaluated tasks, while favoring task-specific reward supervision when available.

PolicyBehavior-encoder supervisionMove Pill BottleMove Stapler
T2DALabeled reward43.3 ± 7.650.0 ± 0.0
T3DPLabeled reward63.3 ± 2.961.7 ± 2.9
T2DANext-state (reward-free)36.7 ± 2.940.0 ± 0.0
T3DPNext-state (reward-free)48.3 ± 2.948.3 ± 2.9

W5 & Q4: Do the gains come from additional contrastive supervision or fine-grained structure, and how do trajectory-, segment-, and token-level alignment compare?

A4. We distinguish the contribution of the added contrastive objective from that of distinct local behavior features. Global alignment matches pooled instruction–trajectory embeddings, whereas T3DP retains the global loss and adds a local contrastive objective. The original ablation achieves 19.1%, 36.1%, and 50.3% average success with unaligned language, global alignment, and T3DP, respectively. The 14.2-point gain establishes the overall value of the added local objective, but does not by itself separate additional supervision from local behavior structure. We therefore add the following controls.

A repeated-pooled control isolates the value of distinct local behavior features while retaining the local contrastive objective. We retain individual instruction tokens but replace every local behavior feature with the same pooled trajectory representation, keeping the remaining training settings fixed. This control improves over global-only alignment, confirming that extra token-level supervision contributes. Full T3DP further improves mean success by 3.0, 6.7, and 7.7 points on Reach, Door Open, and Push, respectively. Thus, distinct local behavior features provide additional value beyond the extra contrastive objective.

TaskT2DA (global-only)Repeated pooled behaviorT3DP (local behavior)
Reach24.3 ± 4.035.3 ± 0.638.3 ± 2.9
Door Open41.3 ± 8.353.0 ± 2.659.7 ± 4.2
Push45.7 ± 8.657.0 ± 2.664.7 ± 17.9

Token-level language alignment also outperforms an intermediate semantic-span alternative. We manually divide instructions into meaningful spans and average token representations within each span before alignment. Here, “segment-level” refers to language spans, while the trajectory-level baseline pools both the instruction and the trajectory. Token-level T3DP exceeds the span-level variant by 4.4–9.7 points across the three tasks below, with a 6.8-point average gain. We will document the segmentation rules and contrastive objectives in the revision.

TaskT2DA (trajectory-level)Language-span-levelT3DP (language-token-level)
Pick Place46.0 ± 1.748.3 ± 2.958.0 ± 2.6
Door Open41.3 ± 8.353.3 ± 5.959.7 ± 4.2
Push45.7 ± 8.660.3 ± 3.164.7 ± 17.9
AuthorsOpenReview ↗

Author Response (Part 3/3)


W4 & Q5: Can T3DP generalize across multiple changing factors and previously unseen task categories?

A5. Our main evaluation follows within-task generalization practice while explicitly holding out behavioral specifications. A common practice in robot manipulation is to assess unseen instances of known task categories, such as different initial scene configurations or object placements, as in DP3 [1] and RoboTwin [2]. Within this paradigm, T3DP focuses on language-specified target positions, displacements, or articulated states absent from policy training, testing whether instructions can guide the required behavioral changes. On Meta-World Push, both direction and movement magnitude vary, and T3DP raises mean success from 45.7% to 64.7% (+19.0 points).

The benefit also persists when a single policy is jointly trained across multiple task families. In Appendix D.2, we combine demonstrations from all five Meta-World task families into one shared policy while retaining the original train–test specification splits. T3DP achieves 40.7% average success on held-out specifications, compared with 35.5% for global alignment (+5.2 points). This shows that the average advantage extends to a policy handling heterogeneous task semantics and specification spaces.

We additionally assess transfer of the jointly trained policy to unseen task categories. These tasks are excluded from behavior-encoder training, language–behavior alignment, and the initial policy training. These held-out tasks require task-specific contact/actuation patterns and object geometries not seen during training. We evaluate direct zero-shot transfer and one-shot adaptation: the former deploys the jointly trained policy unchanged, while the latter fine-tunes only the policy using one example from the new task, keeping the behavior and language encoders fixed. The table reports mean success (%) ± standard deviation over three seeds. After one-shot adaptation, T3DP outperforms T2DA by 3.4 and 8.3 points on Button Press and Faucet Open, respectively. We will document the task-level holdout protocol and results in the revised manuscript, distinguishing this evaluation from held-out specifications within the joint-training task families.

Held-out taskT2DA (zero-shot)T3DP (zero-shot)T2DA (1-shot)T3DP (1-shot)
Button Press8.3 ± 14.43.3 ± 2.928.3 ± 7.631.7 ± 2.9
Faucet Open0.0 ± 0.00.0 ± 0.061.7 ± 28.470.0 ± 32.8

[1] Yanjie Ze et al., 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations, arXiv 2024.

[2] Tianxing Chen et al., Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation, ICML 2026.

Summary Response

Thank you for recognizing the importance of the problem, the broad evaluation, and the ablation studies. We clarify T3DP’s behavior-grounded representation design for specification-sensitive control (W1, Q1). New experiments support robustness to unseen paraphrases and a reward-free training alternative on the evaluated tasks (W2–3, Q2–3). Repeated-pooled and language-span controls further distinguish the benefits of local behavior structure and language granularity from additional contrastive supervision (W5, Q4). We also clarify the generalization setting, report a 5.2-point gain under joint multi-task training, and add transfer results on unseen task categories with zero-shot and one-shot evaluation (W4, Q5). We will incorporate these results and detailed protocols into the revision. We would appreciate your reconsideration in light of these clarifications and new evidence.

Reviewer_XiAaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies generalization to unseen behavioral specifications within known robot manipulation task families, where language specifies target positions, displacements, or articulated states not encountered during policy training. It proposes T3DP, a Text-to-3D Policy framework for fine-grained language–behavior alignment. Experiments across three simulation benchmarks and two real-robot placement tasks show that T3DP improves average success over global alignment by 11.0–14.2 and 17.5 percentage points, respectively.

Critical strengths

The controlled comparison with global alignment shows non-trivial improvements, providing good empirical support for the proposed fine-grained alignment objective.

Critical weaknesses

As authors mentioned in the Limitations section, a key weakness is that the behavior encoder’s reward-prediction objective requires reward-annotated expert demonstrations. Such annotations are often unavailable or costly to obtain in real-world settings, limiting the method’s applicability to existing robot datasets that contain only state–action sequences.

Questions for authors

The following questions would help clarify my assessment:

  1. How are reward labels obtained for the expert demonstrations, particularly in the real-robot experiments? Please specify the reward functions and the information required to compute them.
  2. What additional supervision or assumptions would be needed to apply T3DP to existing robot datasets containing only state–action sequences? Could the authors discuss feasible ways to obtain the required reward signals?
3 comments
AuthorsOpenReview ↗

Author Response


W1(a) & Q1: How are reward labels computed in simulation and on the real robot?

A1. Simulation rewards are computed automatically from task-specific reward functions. We use the reward-prediction objective in Section 4.2, with per-step rewards stored in the expert demonstrations. For example, our Meta-World Reach setup uses rt=10exp⁡[−ln⁡(2)(dt/0.01)2]r_t=10\exp[-\ln(2)(d_t/0.01)^2], where dtd_t is the TCP-to-target distance in meters. Both positions are available from the simulator, and 0.01 m controls the reward scale.

Real-robot rewards are computed offline from recorded physical executions. We replay simulated expert trajectories on the robot and retain successful executions satisfying the requested specifications. The same reward-prediction objective uses per-step labels computed by a geometric rubric following the simulation reward design. Recorded robot motion and gripper poses at grasping and placement events provide geometric estimates, and rewards are recomputed from physical execution rather than copied from simulation. For example, in the real-robot Move Pill Bottle task, we replay expert trajectories, estimate bottle and target positions using the gripper position, and compute per-step rewards as in simulation using these estimates and recorded robot states.

Reward supervision is shared across methods and required only during representation learning. T2DA and T3DP use identical reward labels and the same frozen behavior encoder, isolating the alignment objective. Deployment requires no rewards. Label generation requires task-specific rules and geometric information, whose sources and assumptions will be documented in Appendix E.4. W1(b) & Q2 below addresses datasets without reward labels through a reward-free alternative.


W1(b) & Q2: Does T3DP require reward-annotated demonstrations, and can it be applied to existing datasets containing only state-action sequences?

A2. Behavior learning can use reward or transition supervision. Our original reward-prediction objective requires reward annotations, but alternative objectives can supply behavior representations for language-behavior alignment. The factorization pσ(s′,r∣s,a)=P(s′∣s,a) pσ(r∣s,a,s′)p_\sigma(s',r\mid s,a)=P(s'\mid s,a)\,p_\sigma(r\mid s,a,s') distinguishes physical transition information from task-dependent reward information. Available rewards provide direct supervision about task objectives. Without them, task-specific rubrics can estimate proxy rewards from recorded states or visual observations using state-based heuristics or instruction-conditioned VLMs to assess task progress, while recorded state transitions offer a reward-free source of self-supervision.

Next-state prediction provides a concrete reward-free alternative. We replace reward prediction with prediction of a recorded subsequent state and remove reward tokens from the encoder input. This requires temporally aligned state–action transitions and paired instructions for language alignment, but no reward labels. We retain the subsequent alignment and policy-learning stages and apply the same supervision choice to T3DP and T2DA.

Additional experiments show that the advantage of fine-grained alignment persists without rewards. Across three seeds, reward-free T3DP achieves 48.3% success on both Move Pill Bottle and Move Stapler, exceeding T2DA by 11.6 and 8.3 percentage points. It remains 15.0 and 13.4 points below reward-supervised T3DP, respectively. Since specifications share physical dynamics, next-state prediction need not preserve the same task information as reward prediction. These results support a feasible reward-free option on the two evaluated tasks, while favoring task-specific reward supervision when available.

PolicyBehavior-encoder supervisionMove Pill BottleMove Stapler
T2DALabeled reward43.3 ± 7.650.0 ± 0.0
T3DPLabeled reward63.3 ± 2.961.7 ± 2.9
T2DANext-state (reward-free)36.7 ± 2.940.0 ± 0.0
T3DPNext-state (reward-free)48.3 ± 2.948.3 ± 2.9

Summary Response

Thank you for recognizing our controlled comparison and empirical evidence for fine-grained alignment. We clarify reward acquisition and shared supervision in simulation and real-robot experiments (W1(a), Q1). We also evaluate a reward-free next-state objective, under which T3DP outperforms T2DA by 11.6 and 8.3 percentage points on the two tested tasks (W1(b), Q2). We will document the reward definitions, geometric assumptions, and additional experiments in the revision. We would appreciate your reconsideration in light of these clarifications and new evidence.

Thank you for your response! I noticed that you have added Appendix D.8 that compares with pi0.5 model. I'm wondering can the proposed method be applied to improve the pi0.5 model?

AuthorsOpenReview ↗

Author Response

Thank you for this follow-up, which motivates testing whether language–behavior alignment can complement pretrained VLAs. We have implemented an initial adaptation to π0.5\pi_{0.5} and evaluated it on three RoboTwin tasks. We organize the response around the adaptation, the empirical evidence, and a resulting research direction: connecting alignment targets to the representations used for action generation.


Adaptation: Extending language–behavior alignment to $\pi_{0.5}$

We retain T3DP’s three-stage structure of behavior learning, alignment, and policy learning. First, we replace point-cloud inputs in the trajectory encoder and reward decoder with RGB observations, using a pretrained ResNet image encoder while retaining state–action information and reward supervision. Second, we freeze the action expert and align the VLM’s final hidden states with the learned trajectory representations. Third, we freeze the aligned VLM and fine-tune the action expert following Appendix D.8. This adapts the language–behavior alignment principle to the VLA while keeping the action expert’s architecture unchanged.


Evidence: A small average gain with task-dependent effects

Using the same demonstrations and evaluation protocol, average success across three RoboTwin tasks increases from 37.8% to 38.9% (+1.1 percentage points). Move Stapler improves from 45.0% to 51.7%, Move Playing Card remains at 40.0%, and Move Pill Bottle decreases from 28.3% to 25.0%. The current results therefore show a modest average gain with mixed task-level effects. The table also includes standalone T3DP on these same three tasks for reference.

RoboTwinπ0.5\pi_{0.5} (fine-tuning)π0.5\pi_{0.5} (fine-tuning) + T3DPT3DP
Move Pill Bottle28.3 ± 5.825.0 ± 5.039.3 ± 0.5
Move Playing Card40.0 ± 8.740.0 ± 10.052.3 ± 1.5
Move Stapler45.0 ± 8.751.7 ± 2.928.0 ± 1.6
Avg.37.838.939.9

Insight: Connecting alignment targets to action conditioning

A key design question is which representations receive behavior supervision and how they condition action generation. T3DP supplies aligned text features directly to its policy. In π0.5\pi_{0.5}, the VLM jointly processes observations and instructions. Our current adaptation supervises its final 2048-dimensional hidden states, while the action expert attends to layerwise visual–language keys and values. The alignment objective updates the VLM parameters without directly supervising these layerwise conditioning features. This motivates aligning the representations used by the action expert as a concrete next step. With only one demonstration per training specification, the interaction between alignment targets and demonstration coverage also merits study. Testing these choices could clarify how to make behavior alignment more effective for pretrained VLAs.


Summary Response

Our initial study provides a concrete adaptation of T3DP’s alignment principle to π0.5\pi_{0.5}, yielding a small average gain with task-dependent effects. The follow-up motivates a focused research direction: aligning the representations that the action expert uses for control and examining how the benefits depend on demonstration coverage. This offers a path toward combining pretrained VLA capabilities with more precise specification following. We thank the reviewer for motivating this investigation, and look forward to further discussions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author Responses


Summary of Author Responses

We thank all reviewers for their constructive feedback. In the rebuttal, we addressed all weaknesses and questions through new experiments and clarifications, all of which are included in the revised manuscript (marked in blue).

New experiments. A reward-free variant of the behavior encoder addresses the concern about reward dependency (B1qh, XiAa), and T3DP keeps its advantage without reward labels. Matched-objective controls (repeated-pooled features, state- or action-only features, span-level language) show that local behavior grounding adds gains beyond the extra contrastive supervision (B1qh, VUJQ). A token–behavior pairing analysis gives direct evidence of specification-sensitive grounding (VUJQ). Evaluations on unseen paraphrases (B1qh), unseen task categories (B1qh), and additional train–test splits (VUJQ) show that the advantage extends beyond the original setting. Multi-seed real-robot results confirm the reliability of the real-world gains (VUJQ). A comparison with π0.5 shows that T3DP reaches comparable performance with a much smaller model (VUJQ), and an adaptation of T3DP to π0.5 answers XiAa's follow-up question.

Clarifications. We clarified the novelty of T3DP relative to FILIP: it grounds instruction tokens in local state–action behavior for specification-sensitive 3D control (B1qh). We explained how reward labels are obtained in simulation and on the real robot, and that both T3DP and the global-alignment baseline use the same labels (XiAa, VUJQ). We also clarified the generalization setting, including variation in multiple factors and joint training across task families (B1qh).

Recognized strengths. Reviewers found the problem meaningful (B1qh, VUJQ), the comparison with global alignment well controlled with consistent improvements (B1qh, VUJQ, XiAa), the ablations and representation analyses informative (B1qh, VUJQ), and the paper clear and easy to follow (B1qh, VUJQ). We sincerely thank the reviewers for this recognition and for their valuable suggestions, which have helped us strengthen the paper. We hope the additional evidence addresses the remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.