- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FuncBridge studies transfer of hitting, sweeping, and hooking behaviors to tools not represented in the action-labeled training set. It uses a two-stage architecture: a flow-matching predictor learns future trajectories of functional, target, and tool keypoints from action-free observations, and a second flow-matching policy converts predicted trajectories, images, and proprioception into robot actions. Tool masks and additional keypoints are obtained with SAM2 and farthest-point sampling, and trajectories are extracted using CoTracker. A provided-reference study compares keypoints with affordance images, human-video prompts, functional videos, and object masks. The main simulation study trains the execution policy on four tools and evaluates on three action-held-out tools, obtaining average success rates of 0.36, 0.45, and 0.23 for hitting, sweeping, and hooking. Real-world experiments and extensions using human videos and VLM-selected functional points are also reported.
Critical strengths
-
The intermediate representation fits the geometric requirements of tool use. Including the designated functional region, target, and additional tool points gives the execution policy a compact description of contact and movement. Supplying the same initial functional and target information to the baselines is a useful control (Section 4.1). The method improves all three simulation function averages in Table 1, rather than relying on a single successful task.
-
The paper examines the representation choice and demonstrates physical execution.** The provided-reference comparison in Table 3 separates several types of conditioning under a shared execution policy, and Table 2 reports real-world improvements over FM across all tested patterns.
Critical weaknesses
-
The data access and comparison design leave the source of generalization insufficiently isolated. Section 4.1 excludes evaluation tools from action-labeled training, but does not establish whether those tools are excluded from the action-free predictor's data. Appendix A lists reference/keypoint data covering all ten simulation tools and seven real tools, while Appendix A.3 describes independently collected action-free pretraining without an explicit tool-by-stage split. This could be a legitimate action-label-transfer setting, but it differs from encountering a tool absent from all task-specific training. In addition, ATM uses a generic LIBERO-90-pretrained predictor, whereas FuncBridge receives function-specific pretraining for 240,000 steps. Without adapting a matched predictor on the same action-free corpus, the comparison cannot cleanly attribute the gain to functional reasoning rather than data coverage.
-
The broader representation and generalization claims exceed the scope of the controlled evidence. The representation comparison supplies pre-extracted reference conditions, including future execution information, and is limited to hitting; it establishes utility when such references are available, not the relative performance of learned predictors for each representation. Its keypoint score is 0.49, compared with 0.36 for the learned full pipeline (Tables 1 and 3). Physical evaluation compares only against FM, with ten trials per pattern, and tests sweeping/hooking on only one held-out tool. These are useful demonstrations, but leave uncertainty about robustness and the advantage over a strong, data-matched keypoint method. The modest 0.36-to-0.39 human-video improvement is also reported without training-seed uncertainty.
Questions for authors
-
Please provide a tool-by-dataset split for System-2 pretraining, System-1 training, reference extraction, and final evaluation, separately for simulation and hardware. Are scoop, shoe, book, and the real-world evaluation tools absent from System-2 training, or only from action-labeled training? Are evaluation trajectories or matching reference clips excluded from all learned-model training?
-
How does ATM, or an otherwise matched generic keypoint predictor, perform after training on the same action-free data and budget as FuncBridge? This is important for determining whether the proposed function-aware treatment adds value beyond task-specific trajectory pretraining.
-
What is the precise information supplied at test time in Table 3, including reference selection and temporal alignment? Can the authors quantify the gap between reference trajectories and predicted trajectories, including point-tracking failures and execution errors, on the same held-out episodes?
4 comments
Response to Reviewer NPc1 (2/4)
Responses to Weaknesses and Questions
Question 1 (CW1, Q1): Unclear Data Splits and the Scope of Unseen-Tool Generalization
Answer: We thank the reviewer for pointing out the unclear split of evaluation tools. In the original setting, System-2 is pretrained on action-free keypoint trajectories from all original benchmark tools (10 simulation and 7 real-world, Fig. 3), while System-1 uses action-labeled data only from seen tools. Thus, the original evaluation is System-1-unseen, not full-unseen.
To evaluate full-unseen generalization, we add Canoe, Flask, and Tablet in simulation (Figs. 14–16, App. B.2) and Tiltaxes in the real world (Fig. 10, App. B.1). None are used in System-2 pretraining or System-1 fine-tuning (Tab. A). Additionally, training and evaluation use distinct initial configurations. The main evaluations (Tabs. 1–4) do not use ground-truth future trajectories or reference clips at test time.
Table A. Training and Evaluation Data Splits. Tool and trajectory statistics for System-2 pretraining, System-1 fine-tuning, and evaluation. Reported per function.
| Domain | System-2 Pretraining | System-1 Fine-tuning | System-1-Unseen Evaluation | Full-Unseen Evaluation |
|---|---|---|---|---|
| Simulation | 10 tools (900 traj.) | Hammer, Mustard, Tobject, Pickaxe (360 traj.) | Scoop, Shoe, Book (within the 10 tools) | Canoe, Flask, Tablet (outside the 10 tools) |
| Real-World | 7 tools (280 traj.) | Hammer, Mustard, Pickaxe (120 traj.) | Tobject, Book, Shoe (within the 7 tools) | Tiltaxes (outside the 7 tools) |
As shown in Tabs. 2 and 4, FuncBridge achieves higher average SRs on full-unseen tools in both simulation and the real world, demonstrating that function-aware keypoint trajectories support functional generalization even to full-unseen tools.
Question 2 (CW1, Q2): Lack of comparison with ATM pretrained on the same data
Answer: Thanks for highlighting the mismatch in pretraining data in the ATM comparison. To ensure a fair comparison, we pretrain ATM's tracker on the same action-free keypoint trajectories and fine-tune its downstream policy on the same action-labeled data as FuncBridge, using matched training budgets (App. A.3). We denote this baseline as ATM (Pretrained).
Table B (App. B.3, Tab. 9). Real-World Hitting Results on System-1-Unseen and Full-Unseen Tools. Average success rate (SR) across two functional patterns, with 10 rollouts per pattern.
| System-1-Unseen | Full-Unseen | |||
|---|---|---|---|---|
| Method | Tobject | Book | Shoe | Tiltaxes |
| FM | 0.30 | 0.25 | 0.10 | 0.20 |
| ATM (Pretrained) | 0.55 | 0.60 | 0.35 | 0.50 |
| FuncBridge | 0.70 | 0.80 | 0.40 | 0.60 |
As shown in Tabs. 1–2 and Tab. B (App. B.3, Tab. 9), FuncBridge outperforms ATM (Pretrained) in average SR across all three functions in both simulation settings and on real-world hitting. Compared with ATM, FuncBridge uses flow matching to predict functional keypoint trajectories and introduces random keypoint perturbations during System-1 training to improve robustness to prediction errors (Sec. 3.3). These designs combine function-aware motion prediction with grounded execution across novel tools. Qualitative comparisons are provided in Figs. 14–16 (App. B.2).
Response to Reviewer NPc1 (3/4)
Question 3 (CW2, Q1, Q3): Reference Data Independence, Test-Time Information, and Temporal Alignment
Answer: Thanks for comments on the provided-reference setting. Tab. 5 (originally Tab. 3) evaluates the action groundability of different intermediate representations using provided references, without System-2 prediction. We extract these references from held-out demonstrations and evaluate System-1 under matching initial configurations. The reference demonstrations are excluded from both training stages, ensuring no trajectory-level overlap with the training data.
As described in Sec. 3.2 and App. A.1, we use the following reference selection and temporal alignment strategies:
- Affordance images: A static tool-specific affordance image is used throughout execution.
- Human video prompts: A Position 01 video is selected by matching the tool, setting, and sample. Temporal average pooling produces a time-invariant condition.
- Functional videos, object masks, and keypoint trajectories: Time-varying representations are extracted from the corresponding held-out demonstrations and provided at the matching action timesteps.
During rollout, the policy uses the reference representation at the corresponding timestep, without dynamic temporal re-alignment.
Question 4 (CW2, Q3): Representation Generalization and the Gap Between Reference and Predicted Trajectories
Answer: We thank the reviewer for the suggestions on representation generalization and prediction errors. The original representation study (Tab. 5) is conducted on hitting only. We further extend the provided-reference comparison between object masks and keypoint trajectories to sweeping and hooking, using the same execution-policy evaluation protocol.
Table C (App. B.4, Tab. 10). Functional Representation Comparison Across Functions. Success rate (SR) on System-1-unseen tools in simulation when provided-reference object masks or keypoint trajectories condition the same execution policy.
| Function | Object Masks | Keypoint Trajectories |
|---|---|---|
| Hitting | 0.32 | 0.49 |
| Sweeping | 0.36 | 0.53 |
| Hooking | 0.12 | 0.28 |
As shown in Tab. C, keypoint trajectories consistently outperform object masks across all three functions, demonstrating stronger action groundability under matched execution-policy settings.
Compared with provided-reference guidance, the complete FuncBridge pipeline using System-2-predicted trajectories achieves lower SRs on hitting (0.36 vs. 0.49), sweeping (0.45 vs. 0.53), and hooking (0.23 vs. 0.28) (Tabs. 1 and C). This reveals a performance gap between reference-guided execution and the complete prediction-and-execution pipeline.
To quantify System-2 prediction errors, we compute the mean 2D Euclidean distance between predicted and reference trajectories across seven keypoints on the same evaluation demonstrations.
Table D. System-2 Keypoint Prediction Errors Across Functions. Mean 2D Euclidean distance (pixels) between predicted and reference keypoint trajectories, averaged over seven keypoints.
| Function | System-1-Unseen Tools (Scoop/Shoe/Book) | Full-Unseen Tools (Canoe/Flask/Tablet) |
|---|---|---|
| Hitting | 7.8 | 11.6 |
| Sweeping | 3.5 | 9.8 |
| Hooking | 6.4 | 13.9 |
As shown in Tab. D, prediction errors are higher on full-unseen tools, indicating greater difficulty in predicting functional motion for tools excluded from both training stages. These errors also suggest that imperfect System-2 predictions contribute to the performance gap between provided-reference guidance and the complete pipeline (Tabs. 1 and C). To improve robustness to prediction errors, we apply random 1–5 pixel perturbations to keypoints during System-1 training (Sec. 3.3).
Regarding learned predictors, our goal is to identify a representation that is both effective for action grounding and practical to predict. Functional generalization primarily depends on where functional contact occurs and how these points move relative to the target. Keypoint trajectories capture this essential information in a compact, low-dimensional form. In contrast, predicting dense videos or object masks requires modeling additional visual or geometric details that are not essential for functional execution, making prediction more challenging. Together with the results in Tab. C, it motivates our choice of keypoint trajectories as the prediction target for System-2.
Response to Reviewer NPc1 (4/4)
Question 5 (CW2): Limited Real-World Evaluation and Stronger Baseline Comparison
Answer: We thank the reviewer for the suggestion on real-world evaluation. To address it, we extend the real-world evaluation of sweeping and hooking from Book to three System-1-unseen tools (Tobject, Book, and Shoe), and additionally evaluate a full-unseen tool (Tiltaxes).
As shown in Tabs. 3–4, FuncBridge consistently outperforms FM across all three functions in the expanded real-world evaluations. We also add a real-world comparison with ATM (Pretrained) on hitting in Tab. B (App. B.3, Tab. 9). These results provide further evidence of functional generalization across diverse tools and against a data-matched keypoint baseline.
We provide representative qualitative comparisons in Figs. 7–10 (App. B.1).
Question 6 (CW2): Robustness to Background Clutter and Viewpoint Changes
Answer: To further evaluate FuncBridge's robustness, we add experiments on hitting, sweeping, and hooking under background clutter and camera viewpoint changes. As shown in Fig. 17 (App. B.5), we evaluate two separate perturbations: adding irrelevant tools to the scene and rotating the camera by 10°.
Table E (App. B.5, Fig. 17). Robustness Under Background Clutter and Viewpoint Changes. Success rate (SR) of FuncBridge across three functions in simulation under the standard setting, background clutter, and a 10° camera rotation.
| Function | Standard | Background Clutter | Camera Rotation (10°) |
|---|---|---|---|
| Hitting | 0.36 | 0.26 | 0.31 |
| Sweeping | 0.45 | 0.36 | 0.38 |
| Hooking | 0.23 | 0.15 | 0.18 |
As shown in Tab. E, FuncBridge maintains its ability to perform all three functions under both visual perturbations without additional training, showing the robustness of function-aware keypoint guidance.
Question 7 (CW2): Training-Seed Variability and Human-Video Pretraining Gains
Answer: We thank the reviewer for the suggestion on experimental stability. For human-video pretraining, we fix the System-2 checkpoint pretrained with an additional 525 human videos and fine-tune System-1 using two new random seeds. The additional runs achieve success rates of 0.39 and 0.40, close to the original result of 0.39. These results indicate that the improvement from human-video pretraining is reasonably stable.
For the main experiments, FuncBridge consistently outperforms the baselines across all three functions (Tabs. 1–4), with substantial performance gains. We therefore believe these improvements mainly stem from the effectiveness of our method rather than variations caused by training randomness.
Response to Reviewer NPc1 (1/4)
We thank the reviewer for the insightful comments and for recognizing the value of our functional keypoint representation, controlled comparisons, and real-world validation. To address your concerns, we report additional experiments on full-unseen generalization, data-matched ATM comparisons, representation comparisons across functions, expanded real-world evaluation, and visual robustness. We also clarify the data splits, provided-reference evaluation protocol, reference-versus-predicted performance gap, and training-seed variability.
We have incorporated additional experiments and corresponding revisions into the revised manuscript. Revised text is highlighted in red, while new figures and tables are highlighted in blue. Parenthetical references indicate locations in the revised manuscript (e.g., Tab. B → App. B.3, Tab. 9).
Additional Experimental Results
Table 1. Simulation Results on System-1-Unseen Tools. Success rate (SR) on three tools across hitting, sweeping, and hooking. Each tool is evaluated under 3 initial settings with 30 rollouts per setting.
Table 2. Simulation Results on Full-Unseen Tools. Success rate (SR) on three tools excluded from both System-2 pretraining and System-1 fine-tuning. Each tool is evaluated under 3 initial settings with 30 rollouts per setting.
Table 3. Real-World Results on System-1-Unseen Tools. Success rate (SR) on three tools across hitting, sweeping, and hooking. Each tool is evaluated under two functional patterns, with 10 rollouts per pattern.
Hitting
Sweeping
Hooking
Table 4. Real-World Results on Full-Unseen Tools. Success rate (SR) on Tiltaxes, with 10 rollouts per pattern.