acceptodds
Under review as a conference paper at ICLR 2027

FuncBridge: Functional Tool-Use Generalization via Keypoint Trajectory Reasoning

Abstract

While humans readily repurpose a book, a stone, or a shoe to drive a nail, robots trained on specific tools fail to transfer the same function to novel ones – a gap we formalize as functional generalization. Functionally equivalent tools share visually recognizable functional intent, such as where contact can occur and how a contact region should move to the target. However, this perceptual similarity does not directly carry over to action space, where each tool demands a different motor pattern to realize the function. To bridge this gap, we explore intermediate representations including affordance images, human video prompts, functional videos and object masks, and 2D keypoint trajectories, finding that keypoint trajectories best balance functional expressiveness and action groundability. Building on this, we present FuncBridge, a two-stage framework that decouples functional reasoning from action execution: learning to predict generalizable keypoint trajectories from action-free data, then grounding them into robot actions with limited demonstrations. Across a benchmark spanning ten tools and three functions, including hitting, sweeping, and hooking, FuncBridge consistently outperforms state-of-the-art methods on unseen tools in both simulation and the real world.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Functional Generalization, Robotic Tool Use, Keypoint Trajectories, Imitation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NPc1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FuncBridge studies transfer of hitting, sweeping, and hooking behaviors to tools not represented in the action-labeled training set. It uses a two-stage architecture: a flow-matching predictor learns future trajectories of functional, target, and tool keypoints from action-free observations, and a second flow-matching policy converts predicted trajectories, images, and proprioception into robot actions. Tool masks and additional keypoints are obtained with SAM2 and farthest-point sampling, and trajectories are extracted using CoTracker. A provided-reference study compares keypoints with affordance images, human-video prompts, functional videos, and object masks. The main simulation study trains the execution policy on four tools and evaluates on three action-held-out tools, obtaining average success rates of 0.36, 0.45, and 0.23 for hitting, sweeping, and hooking. Real-world experiments and extensions using human videos and VLM-selected functional points are also reported.

Critical strengths
  1. The intermediate representation fits the geometric requirements of tool use. Including the designated functional region, target, and additional tool points gives the execution policy a compact description of contact and movement. Supplying the same initial functional and target information to the baselines is a useful control (Section 4.1). The method improves all three simulation function averages in Table 1, rather than relying on a single successful task.

  2. The paper examines the representation choice and demonstrates physical execution.** The provided-reference comparison in Table 3 separates several types of conditioning under a shared execution policy, and Table 2 reports real-world improvements over FM across all tested patterns.

Critical weaknesses
  1. The data access and comparison design leave the source of generalization insufficiently isolated. Section 4.1 excludes evaluation tools from action-labeled training, but does not establish whether those tools are excluded from the action-free predictor's data. Appendix A lists reference/keypoint data covering all ten simulation tools and seven real tools, while Appendix A.3 describes independently collected action-free pretraining without an explicit tool-by-stage split. This could be a legitimate action-label-transfer setting, but it differs from encountering a tool absent from all task-specific training. In addition, ATM uses a generic LIBERO-90-pretrained predictor, whereas FuncBridge receives function-specific pretraining for 240,000 steps. Without adapting a matched predictor on the same action-free corpus, the comparison cannot cleanly attribute the gain to functional reasoning rather than data coverage.

  2. The broader representation and generalization claims exceed the scope of the controlled evidence. The representation comparison supplies pre-extracted reference conditions, including future execution information, and is limited to hitting; it establishes utility when such references are available, not the relative performance of learned predictors for each representation. Its keypoint score is 0.49, compared with 0.36 for the learned full pipeline (Tables 1 and 3). Physical evaluation compares only against FM, with ten trials per pattern, and tests sweeping/hooking on only one held-out tool. These are useful demonstrations, but leave uncertainty about robustness and the advantage over a strong, data-matched keypoint method. The modest 0.36-to-0.39 human-video improvement is also reported without training-seed uncertainty.

Questions for authors
  1. Please provide a tool-by-dataset split for System-2 pretraining, System-1 training, reference extraction, and final evaluation, separately for simulation and hardware. Are scoop, shoe, book, and the real-world evaluation tools absent from System-2 training, or only from action-labeled training? Are evaluation trajectories or matching reference clips excluded from all learned-model training?

  2. How does ATM, or an otherwise matched generic keypoint predictor, perform after training on the same action-free data and budget as FuncBridge? This is important for determining whether the proposed function-aware treatment adds value beyond task-specific trajectory pretraining.

  3. What is the precise information supplied at test time in Table 3, including reference selection and temporal alignment? Can the authors quantify the gap between reference trajectories and predicted trajectories, including point-tracking failures and execution errors, on the same held-out episodes?

4 comments
AuthorsOpenReview ↗

Response to Reviewer NPc1 (1/4)

We thank the reviewer for the insightful comments and for recognizing the value of our functional keypoint representation, controlled comparisons, and real-world validation. To address your concerns, we report additional experiments on full-unseen generalization, data-matched ATM comparisons, representation comparisons across functions, expanded real-world evaluation, and visual robustness. We also clarify the data splits, provided-reference evaluation protocol, reference-versus-predicted performance gap, and training-seed variability.

We have incorporated additional experiments and corresponding revisions into the revised manuscript. Revised text is highlighted in red, while new figures and tables are highlighted in blue. Parenthetical references indicate locations in the revised manuscript (e.g., Tab. B → App. B.3, Tab. 9).


Additional Experimental Results

Table 1. Simulation Results on System-1-Unseen Tools. Success rate (SR) on three tools across hitting, sweeping, and hooking. Each tool is evaluated under 3 initial settings with 30 rollouts per setting.

MethodHittingSweepingHooking
ScoopShoeBookAvg.ScoopShoeBookAvg.ScoopShoeBookAvg.
FM0.070.030.160.090.490.230.190.300.220.060.140.14
DP0.180.080.260.170.490.180.130.270.120.060.080.09
ATM0.120.060.060.080.410.160.190.250.100.070.240.14
ATM (Pretrained)0.330.210.310.280.510.170.310.330.150.130.240.17
FuncBridge0.420.230.420.360.710.290.340.450.290.170.240.23

Table 2. Simulation Results on Full-Unseen Tools. Success rate (SR) on three tools excluded from both System-2 pretraining and System-1 fine-tuning. Each tool is evaluated under 3 initial settings with 30 rollouts per setting.

MethodHittingSweepingHooking
CanoeFlaskTabletAvg.CanoeFlaskTabletAvg.CanoeFlaskTabletAvg.
FM0.090.170.130.130.190.170.270.210.040.180.130.12
DP0.110.280.180.190.140.280.320.250.020.220.160.13
ATM0.010.140.070.070.160.220.170.180.040.160.190.13
ATM (Pretrained)0.130.300.190.210.310.280.330.310.060.210.240.17
FuncBridge0.210.360.370.310.390.400.420.400.030.270.240.18

Table 3. Real-World Results on System-1-Unseen Tools. Success rate (SR) on three tools across hitting, sweeping, and hooking. Each tool is evaluated under two functional patterns, with 10 rollouts per pattern.

Hitting

MethodTobject P01Tobject P02Tobject Avg.Book P01Book P02Book Avg.Shoe P01Shoe P02Shoe Avg.
FM0.000.600.300.500.000.250.000.200.10
FuncBridge0.600.800.700.601.000.800.200.600.40

Sweeping

MethodTobject P01Tobject P02Tobject Avg.Book P01Book P02Book Avg.Shoe P01Shoe P02Shoe Avg.
FM0.000.200.100.300.000.150.100.200.15
FuncBridge0.800.400.600.800.500.650.400.600.50

Hooking

MethodTobject P01Tobject P02Tobject Avg.Book P01Book P02Book Avg.Shoe P01Shoe P02Shoe Avg.
FM0.000.300.150.200.000.100.000.000.00
FuncBridge1.000.800.900.800.400.600.500.800.65

Table 4. Real-World Results on Full-Unseen Tools. Success rate (SR) on Tiltaxes, with 10 rollouts per pattern.

MethodHittingSweepingHooking
TiltaxesTiltaxesTiltaxes
P01P02Avg.P01P02Avg.P01P02Avg.
FM0.000.400.200.100.300.200.000.200.10
FuncBridge0.201.000.600.700.500.600.600.600.60
AuthorsOpenReview ↗

Response to Reviewer NPc1 (2/4)

Responses to Weaknesses and Questions


Question 1 (CW1, Q1): Unclear Data Splits and the Scope of Unseen-Tool Generalization

Answer: We thank the reviewer for pointing out the unclear split of evaluation tools. In the original setting, System-2 is pretrained on action-free keypoint trajectories from all original benchmark tools (10 simulation and 7 real-world, Fig. 3), while System-1 uses action-labeled data only from seen tools. Thus, the original evaluation is System-1-unseen, not full-unseen.

To evaluate full-unseen generalization, we add Canoe, Flask, and Tablet in simulation (Figs. 14–16, App. B.2) and Tiltaxes in the real world (Fig. 10, App. B.1). None are used in System-2 pretraining or System-1 fine-tuning (Tab. A). Additionally, training and evaluation use distinct initial configurations. The main evaluations (Tabs. 1–4) do not use ground-truth future trajectories or reference clips at test time.

Table A. Training and Evaluation Data Splits. Tool and trajectory statistics for System-2 pretraining, System-1 fine-tuning, and evaluation. Reported per function.

DomainSystem-2 PretrainingSystem-1 Fine-tuningSystem-1-Unseen EvaluationFull-Unseen Evaluation
Simulation10 tools (900 traj.)Hammer, Mustard, Tobject, Pickaxe (360 traj.)Scoop, Shoe, Book (within the 10 tools)Canoe, Flask, Tablet (outside the 10 tools)
Real-World7 tools (280 traj.)Hammer, Mustard, Pickaxe (120 traj.)Tobject, Book, Shoe (within the 7 tools)Tiltaxes (outside the 7 tools)

As shown in Tabs. 2 and 4, FuncBridge achieves higher average SRs on full-unseen tools in both simulation and the real world, demonstrating that function-aware keypoint trajectories support functional generalization even to full-unseen tools.


Question 2 (CW1, Q2): Lack of comparison with ATM pretrained on the same data

Answer: Thanks for highlighting the mismatch in pretraining data in the ATM comparison. To ensure a fair comparison, we pretrain ATM's tracker on the same action-free keypoint trajectories and fine-tune its downstream policy on the same action-labeled data as FuncBridge, using matched training budgets (App. A.3). We denote this baseline as ATM (Pretrained).

Table B (App. B.3, Tab. 9). Real-World Hitting Results on System-1-Unseen and Full-Unseen Tools. Average success rate (SR) across two functional patterns, with 10 rollouts per pattern.

System-1-UnseenFull-Unseen
MethodTobjectBookShoeTiltaxes
FM0.300.250.100.20
ATM (Pretrained)0.550.600.350.50
FuncBridge0.700.800.400.60

As shown in Tabs. 1–2 and Tab. B (App. B.3, Tab. 9), FuncBridge outperforms ATM (Pretrained) in average SR across all three functions in both simulation settings and on real-world hitting. Compared with ATM, FuncBridge uses flow matching to predict functional keypoint trajectories and introduces random keypoint perturbations during System-1 training to improve robustness to prediction errors (Sec. 3.3). These designs combine function-aware motion prediction with grounded execution across novel tools. Qualitative comparisons are provided in Figs. 14–16 (App. B.2).

AuthorsOpenReview ↗

Response to Reviewer NPc1 (3/4)

Question 3 (CW2, Q1, Q3): Reference Data Independence, Test-Time Information, and Temporal Alignment

Answer: Thanks for comments on the provided-reference setting. Tab. 5 (originally Tab. 3) evaluates the action groundability of different intermediate representations using provided references, without System-2 prediction. We extract these references from held-out demonstrations and evaluate System-1 under matching initial configurations. The reference demonstrations are excluded from both training stages, ensuring no trajectory-level overlap with the training data.

As described in Sec. 3.2 and App. A.1, we use the following reference selection and temporal alignment strategies:

  1. Affordance images: A static tool-specific affordance image is used throughout execution.
  2. Human video prompts: A Position 01 video is selected by matching the tool, setting, and sample. Temporal average pooling produces a time-invariant condition.
  3. Functional videos, object masks, and keypoint trajectories: Time-varying representations are extracted from the corresponding held-out demonstrations and provided at the matching action timesteps.

During rollout, the policy uses the reference representation at the corresponding timestep, without dynamic temporal re-alignment.


Question 4 (CW2, Q3): Representation Generalization and the Gap Between Reference and Predicted Trajectories

Answer: We thank the reviewer for the suggestions on representation generalization and prediction errors. The original representation study (Tab. 5) is conducted on hitting only. We further extend the provided-reference comparison between object masks and keypoint trajectories to sweeping and hooking, using the same execution-policy evaluation protocol.

Table C (App. B.4, Tab. 10). Functional Representation Comparison Across Functions. Success rate (SR) on System-1-unseen tools in simulation when provided-reference object masks or keypoint trajectories condition the same execution policy.

FunctionObject MasksKeypoint Trajectories
Hitting0.320.49
Sweeping0.360.53
Hooking0.120.28

As shown in Tab. C, keypoint trajectories consistently outperform object masks across all three functions, demonstrating stronger action groundability under matched execution-policy settings.

Compared with provided-reference guidance, the complete FuncBridge pipeline using System-2-predicted trajectories achieves lower SRs on hitting (0.36 vs. 0.49), sweeping (0.45 vs. 0.53), and hooking (0.23 vs. 0.28) (Tabs. 1 and C). This reveals a performance gap between reference-guided execution and the complete prediction-and-execution pipeline.

To quantify System-2 prediction errors, we compute the mean 2D Euclidean distance between predicted and reference trajectories across seven keypoints on the same evaluation demonstrations.

Table D. System-2 Keypoint Prediction Errors Across Functions. Mean 2D Euclidean distance (pixels) between predicted and reference keypoint trajectories, averaged over seven keypoints.

FunctionSystem-1-Unseen Tools (Scoop/Shoe/Book)Full-Unseen Tools (Canoe/Flask/Tablet)
Hitting7.811.6
Sweeping3.59.8
Hooking6.413.9

As shown in Tab. D, prediction errors are higher on full-unseen tools, indicating greater difficulty in predicting functional motion for tools excluded from both training stages. These errors also suggest that imperfect System-2 predictions contribute to the performance gap between provided-reference guidance and the complete pipeline (Tabs. 1 and C). To improve robustness to prediction errors, we apply random 1–5 pixel perturbations to keypoints during System-1 training (Sec. 3.3).

Regarding learned predictors, our goal is to identify a representation that is both effective for action grounding and practical to predict. Functional generalization primarily depends on where functional contact occurs and how these points move relative to the target. Keypoint trajectories capture this essential information in a compact, low-dimensional form. In contrast, predicting dense videos or object masks requires modeling additional visual or geometric details that are not essential for functional execution, making prediction more challenging. Together with the results in Tab. C, it motivates our choice of keypoint trajectories as the prediction target for System-2.

AuthorsOpenReview ↗

Response to Reviewer NPc1 (4/4)

Question 5 (CW2): Limited Real-World Evaluation and Stronger Baseline Comparison

Answer: We thank the reviewer for the suggestion on real-world evaluation. To address it, we extend the real-world evaluation of sweeping and hooking from Book to three System-1-unseen tools (Tobject, Book, and Shoe), and additionally evaluate a full-unseen tool (Tiltaxes).

As shown in Tabs. 3–4, FuncBridge consistently outperforms FM across all three functions in the expanded real-world evaluations. We also add a real-world comparison with ATM (Pretrained) on hitting in Tab. B (App. B.3, Tab. 9). These results provide further evidence of functional generalization across diverse tools and against a data-matched keypoint baseline.

We provide representative qualitative comparisons in Figs. 7–10 (App. B.1).


Question 6 (CW2): Robustness to Background Clutter and Viewpoint Changes

Answer: To further evaluate FuncBridge's robustness, we add experiments on hitting, sweeping, and hooking under background clutter and camera viewpoint changes. As shown in Fig. 17 (App. B.5), we evaluate two separate perturbations: adding irrelevant tools to the scene and rotating the camera by 10°.

Table E (App. B.5, Fig. 17). Robustness Under Background Clutter and Viewpoint Changes. Success rate (SR) of FuncBridge across three functions in simulation under the standard setting, background clutter, and a 10° camera rotation.

FunctionStandardBackground ClutterCamera Rotation (10°)
Hitting0.360.260.31
Sweeping0.450.360.38
Hooking0.230.150.18

As shown in Tab. E, FuncBridge maintains its ability to perform all three functions under both visual perturbations without additional training, showing the robustness of function-aware keypoint guidance.


Question 7 (CW2): Training-Seed Variability and Human-Video Pretraining Gains

Answer: We thank the reviewer for the suggestion on experimental stability. For human-video pretraining, we fix the System-2 checkpoint pretrained with an additional 525 human videos and fine-tune System-1 using two new random seeds. The additional runs achieve success rates of 0.39 and 0.40, close to the original result of 0.39. These results indicate that the improvement from human-video pretraining is reasonably stable.

For the main experiments, FuncBridge consistently outperforms the baselines across all three functions (Tabs. 1–4), with substantial performance gains. We therefore believe these improvements mainly stem from the effectiveness of our method rather than variations caused by training randomness.

Reviewer_tWSQOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper formalizes the problem of functional generalization in robotic tool use: the ability to accomplish the same task function (hitting, sweeping, hooking) using novel tools with distinct geometries. The authors construct a simulation benchmark with 10 tools across 3 functions, plus a real-world benchmark on a Franka robot. Experiments show that FUNCBRIDGE consistently outperforms end-to-end baselines (Flow-Matching, Diffusion Policy) and a generic keypoint baseline (ATM) on unseen tools. They further demonstrate scalability by incorporating human video pretraining and VLM-based functional point selection.

Critical strengths
  • Functional generalization is a highly relevant capability for open-world robotics and has been systematically underexplored relative to scene-level and category-level generalization. The paper clearly delineates the problem boundaries (Figure 1) and articulates the core perception-to-action mismatch, providing a useful conceptual framework for the community.
  • The head-to-head comparison of five representation types (static affordance images, pooled human video prompts, per-frame videos, per-frame object masks, and keypoint trajectories) is methodologically sound and directly answers the paper’s central research question. This ablation is a strong, evidence-based justification for the core design choice, rather than an ad-hoc architectural decision.
  • The decoupling of action-free functional reasoning from action-labeled execution matches the practical asymmetry: visual tool-use data is abundant, but robot action demonstrations are expensive and scarce. The framework naturally supports scaling with heterogeneous data sources (simulation, human videos), as validated in Section 4.4.
Critical weaknesses
  • The core technical components are largely adapted from prior work: two-stage visuomotor pipelines, flow matching for trajectory generation, SAM2 for segmentation, and CoTracker for tracking. The primary contribution lies in problem formulation, systematic representation benchmarking, and integration of existing components into a functional tool-use framework. While the integration is effective and well-executed, the algorithmic novelty is incremental relative to recent works on keypoint-guided manipulation and diffusion/flow-matching policies.
  • In the default setting, the method requires manually specified functional points and target points. While the VLM extension is a step toward autonomy, it achieves only 0.31 average SR (vs. 0.36 with manual points), and errors in point selection directly cause task failure. The method’s autonomy in truly unconstrained scenes remains limited.
  • Regarding the related work on robot manipulation generalization, I recommend the authors cite and briefly discuss [1], [2], and [3]. Including this reference will help position your work more comprehensively within the broader landscape of generalization paradigms in robot learning. [1] Spatially Generalizable Mobile Manipulation via Adaptive Experience Selection and Dynamic Imagination. IJCAI 2026 [2] R2RGEN: Real-to-Real 3D Data Generation for Spatially Generalized Manipulation. RSS 2026 [3] Generalizable Coarse-to-Fine Robot Manipulation via Language-Aligned 3D Keypoints. ICLR 2026
  • Real-world sweeping and hooking are tested on only one unseen tool (book), with 10 rollouts per pattern. The limited tool diversity and small sample size reduce the statistical reliability of the real-world conclusions for these two tasks.
Questions for authors
  • Figure 2(b) label "Kep. Position" should be "Kpt. Position" (keypoint position).
  • The three contribution bullets in the introduction frame the representation finding as a core contribution. While the empirical comparison is valuable, it is a conclusion rather than a technical invention; the contribution statement could be phrased more precisely.
  • The related work section could more sharply distinguish functional generalization from existing affordance generalization and category-level generalization work, rather than just listing them separately.
  • If the authors resolve all my concerns, I will consider raising the final score.
3 comments
AuthorsOpenReview ↗

Response to Reviewer tWSQ (1/3)

We thank the reviewer for the constructive comments and for recognizing the value of our problem formulation, systematic representation benchmarking, and functional tool-use framework. To address your concerns, we have conducted additional experiments on expanded real-world evaluations, VLM-based functional point selection across functions, and robustness under visual perturbations. We have also clarified our contributions and distinctions from related work.

We have incorporated additional experiments and corresponding revisions into the revised manuscript. Revised text is highlighted in red, while new figures and tables are highlighted in blue. Parenthetical references indicate locations in the revised manuscript (e.g., Tab. B → App. B.6, Tab. 11).

Responses to Weaknesses and Questions


Question 1 (CW1, Q2, Q3): Contributions and distinctions from prior generalization work

Answer: We thank the reviewer for recognizing our contributions. The primary contributions of FuncBridge lie in functional generalization problem formulation, systematic representation analysis, and a unified perception-to-action framework. Following your suggestions, we have clarified these contributions in the Introduction more precisely:

  1. Problem Formulation. We formalize functional generalization as performing the same function with tools of different geometries and motion requirements, and identify the perception-to-action gap as a central challenge.
  2. Representation Study and Framework. We systematically investigate functional intermediate representations and, based on our findings, propose FuncBridge, a two-stage framework that learns functional keypoint motion from action-free data and grounds it into robot actions.
  3. Benchmark and Evaluation: We construct simulation and real-world benchmarks covering hitting, sweeping, and hooking, and demonstrate improved generalization to state-of-the-art methods on unseen tools.

We further clarify the distinctions between functional generalization, category-level generalization, and affordance generalization, as summarized in Tab. A.

Table A. Comparison of Generalization Settings. Differences in generalization objectives and research focus.

SettingGeneralization TargetResearch Focus
Category-level GeneralizationUnseen object instances within the same semantic category.Transfer manipulation skills across objects with different geometries and appearances, without explicitly focusing on motion adaptation.
Affordance GeneralizationTransferable interaction affordances across objects.Learn transferable contact regions and interaction behaviors.
Functional Generalization (Ours)Geometrically diverse tools, potentially across semantic categories.Adapt functional contact points and motion trajectories to accomplish the same function with different tools.

These generalization settings are complementary. Category-level generalization typically focuses on unseen object instances, while affordance generalization studies transferable interaction regions and behaviors. FuncBridge specifically focuses on adapting functional contact points and motion trajectories across diverse tools to accomplish the same function.

Thanks again for your suggestions, we have revised the contributions in Sec. 1 (Introduction) and clarified these distinctions in Sec. 2 (Related Work).

AuthorsOpenReview ↗

Response to Reviewer tWSQ (2/3)

Question 2 (CW2): Limited Evaluation of VLM-Based Point Selection and Limited Autonomy in Unconstrained Scenes

Answer: Thanks for the suggestions on system autonomy. To address this concern, we extend VLM-based functional point selection to all three functions and further evaluate FuncBridge in more challenging scenes with visual perturbations.

Table B (App. B.6, Tab. 11). VLM-Based Functional Point Selection Across Functions. Success rate (SR) with manually specified and VLM-selected functional and target points in simulation.

FunctionManual PointsVLM-Selected Points
Hitting0.360.31
Sweeping0.450.42
Hooking0.230.23

As shown in Tab. B, VLM-based point selection achieves comparable performance versus manually specified points across all three functions. This indicates that the autonomy of FuncBridge can be extended across multiple functions through VLM-based point selection, reducing the need for human intervention.

Furthermore, to alleviate the concern about unconstrained scenes, we add experiments on hitting, sweeping, and hooking under background clutter and camera viewpoint changes. As shown in Fig. 17 (App. B.5), we evaluate two separate perturbations: adding irrelevant tools to the scene and rotating the camera by 10°.

Table C (App. B.5, Fig. 17). Robustness Under Background Clutter and Viewpoint Changes. Success rate (SR) of FuncBridge across three functions under the standard setting, background clutter, and a 10° camera rotation.

FunctionStandardBackground ClutterCamera Rotation (10°)
Hitting0.360.260.31
Sweeping0.450.360.38
Hooking0.230.150.18

As shown in Tab. C, FuncBridge maintains satisfactory success rates across functions under different perturbations, demonstrating considerable robustness of keypoints-based functional representations.

In summary, these results suggest that FuncBridge can support autonomy across multiple functions while adapting to certain visual variations, providing a step toward autonomous tool use in more unconstrained scenes.


Question 3 (CW3, Q1): More Discussion of Recent Works on Manipulation Generalization

Answer: We thank the reviewer for recommending these relevant works. Tab. D summarizes their generalization objectives and key differences from FuncBridge.

Table D. Comparison with Recent Works on Manipulation Generalization.

MethodMain FocusKey Difference from FuncBridge
AES [1]Spatial generalization through adaptive experience selection and imagination.AES targets spatial variations, while FuncBridge targets functional motion adaptation across diverse tools.
R2RGen [2]Spatial generalization through generated 3D observation-action demonstrations.R2RGen augments spatial demonstration data, while FuncBridge learns functional keypoint motion from action-free trajectories.
CLAP [3]Coarse-to-fine manipulation guided by language-aligned 3D keypoints.CLAP uses 3D keypoints for step-wise manipulation, while FuncBridge models function-specific motion with temporal 2D keypoint trajectories.

We have added these references and the corresponding discussion to Sec. 2 (Related Work) and have corrected "Kep. Position" to "Kpt. Position" in Fig. 2.

[1] Zhong et al. Spatially Generalizable Mobile Manipulation via Adaptive Experience Selection and Dynamic Imagination. IJCAI 2026.

[2] Xu et al. R2RGen: Real-to-Real 3D Data Generation for Spatially Generalized Manipulation. RSS 2026.

[3] Hu et al. Generalizable Coarse-to-Fine Robot Manipulation via Language-Aligned 3D Keypoints. ICLR 2026.

AuthorsOpenReview ↗

Response to Reviewer tWSQ (3/3)

Question 4 (W4): Limited Real-World Evaluation

Answer: We thank the reviewer for the suggestion on real-world evaluation. To address this issue, we have extended the evaluation of sweeping and hooking from Book to three System-1-unseen tools (Tobject, Book, and Shoe), and further introduced a full-unseen tool (Tiltaxes). System-1-unseen tools are excluded from action-labeled fine-tuning for execution policy (System 1), while full-unseen tools are excluded from both pretraining of keypoint motion predictor (System 2) and fine-tuning of execution policy (System 1).

Table 3. Real-World Results on System-1-Unseen Tools. Success rate (SR) on three tools across hitting, sweeping, and hooking. Each tool is evaluated under two functional patterns, with 10 rollouts per pattern.

Hitting

MethodTobject P01Tobject P02Tobject Avg.Book P01Book P02Book Avg.Shoe P01Shoe P02Shoe Avg.
FM0.000.600.300.500.000.250.000.200.10
FuncBridge0.600.800.700.601.000.800.200.600.40

Sweeping

MethodTobject P01Tobject P02Tobject Avg.Book P01Book P02Book Avg.Shoe P01Shoe P02Shoe Avg.
FM0.000.200.100.300.000.150.100.200.15
FuncBridge0.800.400.600.800.500.650.400.600.50

Hooking

MethodTobject P01Tobject P02Tobject Avg.Book P01Book P02Book Avg.Shoe P01Shoe P02Shoe Avg.
FM0.000.300.150.200.000.100.000.000.00
FuncBridge1.000.800.900.800.400.600.500.800.65

Table 4. Real-World Results on Full-Unseen Tools. Success rate (SR) on Tiltaxes, with 10 rollouts per pattern.

MethodHittingSweepingHooking
TiltaxesTiltaxesTiltaxes
P01P02Avg.P01P02Avg.P01P02Avg.
FM0.000.400.200.100.300.200.000.200.10
FuncBridge0.201.000.600.700.500.600.600.600.60

As shown in Tabs. 3–4, FuncBridge consistently outperforms the FM baseline across the expanded real-world evaluations. These results provide further evidence of functional generalization across diverse tools. We provide representative qualitative comparisons in Figs. 8–10 (App. B.1).

Reviewer_HtrUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies functional generalization in robotic tool use. FuncBridge uses a two-stage framework in which a flow-matching predictor learns future 2D trajectories of functional, target, and tool keypoints from action-free data, and a second flow-matching policy grounds the predicted trajectories into robot actions using limited action-labeled demonstrations. The paper compares intermediate representations and constructs a simulation benchmark spanning ten tools and hitting, sweeping, and hooking, with additional Franka experiments. The paper also reports quantitative real-world results, a human-video pretraining extension, and VLM-based functional-point selection.

Critical strengths
  1. A relevant problem with quantitative evidence across several functions

Transferring a tool-use function across geometrically different objects is practically meaningful. Tables 1 and 2 show improvements across hitting, sweeping, and hooking in both simulation and real-world experiments. The real-world evaluation includes three held-out tools for hitting and two functional patterns per evaluated tool/function combination.

  1. Useful analysis of structured intermediate representations

Table 3 includes temporally varying video and object-mask conditions, which makes the representation study more informative than a comparison against static or pooled conditions alone.

Critical weaknesses
  1. The evaluation does not clearly establish the stated scope of unseen-tool generalization. Section 4.1 explicitly holds evaluation tools out of action-labeled training, while Table 5 and Appendix A.1 describe keypoint data covering all ten simulation tools and all seven real-world tools. Appendix A.3 states that action-free data are independently collected, but does not identify which tools are included in predictor training or how training and evaluation trajectories are separated. Thus, it is unclear whether the test tools are unseen by the complete system or only by the action-grounding stage. Learning to execute with tools seen in action-free videos is a useful setting, but it supports a different claim from generalization to tools unseen throughout training. The paper should provide an explicit split for each stage and align its claims accordingly.

The evaluation also supports a narrower robustness claim than the broad framing suggests. The three functions are a meaningful improvement in coverage, but the default setting still provides functional points, real-world sweeping and hooking each test only the book, and the VLM experiment covers only simulated hitting. There is no quantitative analysis of viewpoint changes, depth ambiguity, or occlusion in the supplied manuscript, despite the reliance on 2D trajectories. I therefore view the evidence as promising transfer under controlled conditions, with the robustness of the representation still unresolved.

  1. The comparisons do not isolate a convincing methodological advance over existing trajectory-guided policies. Learning point trajectories from action-free videos and using them to guide a policy trained with limited action labels is already central to ATM; a related separation of visual flow generation and action execution is used by Im2Flow2Act. My concern is not the use of established components, but that the incremental contribution of functional-point modeling is not cleanly demonstrated. In Section 4.1, ATM uses a predictor pretrained on LIBERO-90, whereas FuncBridge receives pretraining on the tool-use data. Since ATM also receives the designated functional and target points, the difference in performance may reflect the pretraining-domain mismatch rather than superior functional reasoning. A comparison with ATM adapted to the same action-free data, or an equivalent generic-point predictor trained under matched conditions, is needed to support that interpretation.

  2. Table 3 provides useful evidence about action grounding when reference representations are supplied, but it evaluates only hitting and does not compare the predictability of the alternative representations. It therefore supports the advantage of the tested keypoint condition in that setting, rather than fully establishing the broader claim that keypoints best balance functional expressiveness and action groundability across functions. The real-world comparison is also limited to FM, leaving the advantage over another trajectory-guided method untested on hardware.

Questions for authors
  1. Training and evaluation splits: For each function and domain, which tools and how many trajectories are used in the action-free predictor training, action-labeled policy training, and evaluation? Do the action-free simulation, real-world, or human-video datasets contain the evaluation tools? Are evaluation trajectories and initial configurations held out independently?

  2. Matched baseline: Was the ATM predictor adapted to the same action-free tool-use data? If so, please provide the protocol and results; if not, what evidence separates the effect of in-domain pretraining from the proposed function-aware modeling? A matched ATM or generic-point predictor would be a particularly consequential control.

  3. Function conditioning and inference: Are separate predictors and execution policies trained for hitting, sweeping, and hooking, or does one model handle all three? If a shared model is used, how is the requested function specified, given that Equations (2)-(4) do not include an explicit task variable? How are current keypoints updated and trajectories replanned during execution?

  4. Robustness and uncertainty: Do you have results under viewpoint changes, partial occlusion, or noisy functional-point inputs? Please also state the number of independent training runs and provide uncertainty estimates for the main comparisons and the 0.36-to-0.39 human-video gain. These would help distinguish stable improvements from variation in training and rollout outcomes.

3 comments
AuthorsOpenReview ↗

Response to Reviewer HtrU (1/3)

We thank the reviewer for the valuable comments and for recognizing the importance of functional generalization and the value of our cross-function evaluations and representation analysis. To address your concerns, we have conducted additional experiments on full-unseen generalization, data-matched ATM comparisons, representation analysis across functions, robustness to visual perturbations, and expanded real-world and VLM-based point-selection evaluations.

We have incorporated additional experiments and corresponding revisions into the revised manuscript. Revised text is highlighted in red, while new figures and tables are highlighted in blue. Parenthetical references indicate locations in the revised manuscript (e.g., Tab. B → App. B.3, Tab. 9).


Additional Experimental Results

Table 1. Simulation Results on System-1-Unseen Tools. Success rate (SR) on three tools across hitting, sweeping, and hooking. Each tool is evaluated under 3 initial settings with 30 rollouts per setting.

MethodHittingSweepingHooking
ScoopShoeBookAvg.ScoopShoeBookAvg.ScoopShoeBookAvg.
FM0.070.030.160.090.490.230.190.300.220.060.140.14
DP0.180.080.260.170.490.180.130.270.120.060.080.09
ATM0.120.060.060.080.410.160.190.250.100.070.240.14
ATM (Pretrained)0.330.210.310.280.510.170.310.330.150.130.240.17
FuncBridge0.420.230.420.360.710.290.340.450.290.170.240.23

Table 2. Simulation Results on Full-Unseen Tools. Success rate (SR) on three tools excluded from both System-2 pretraining and System-1 fine-tuning. Each tool is evaluated under 3 initial settings with 30 rollouts per setting.

MethodHittingSweepingHooking
CanoeFlaskTabletAvg.CanoeFlaskTabletAvg.CanoeFlaskTabletAvg.
FM0.090.170.130.130.190.170.270.210.040.180.130.12
DP0.110.280.180.190.140.280.320.250.020.220.160.13
ATM0.010.140.070.070.160.220.170.180.040.160.190.13
ATM (Pretrained)0.130.300.190.210.310.280.330.310.060.210.240.17
FuncBridge0.210.360.370.310.390.400.420.400.030.270.240.18

Table 3. Real-World Results on System-1-Unseen Tools. Success rate (SR) on three tools across hitting, sweeping, and hooking. Each tool is evaluated under two functional patterns, with 10 rollouts per pattern.

Hitting

MethodTobject P01Tobject P02Tobject Avg.Book P01Book P02Book Avg.Shoe P01Shoe P02Shoe Avg.
FM0.000.600.300.500.000.250.000.200.10
FuncBridge0.600.800.700.601.000.800.200.600.40

Sweeping

MethodTobject P01Tobject P02Tobject Avg.Book P01Book P02Book Avg.Shoe P01Shoe P02Shoe Avg.
FM0.000.200.100.300.000.150.100.200.15
FuncBridge0.800.400.600.800.500.650.400.600.50

Hooking

MethodTobject P01Tobject P02Tobject Avg.Book P01Book P02Book Avg.Shoe P01Shoe P02Shoe Avg.
FM0.000.300.150.200.000.100.000.000.00
FuncBridge1.000.800.900.800.400.600.500.800.65

Table 4. Real-World Results on Full-Unseen Tools. Success rate (SR) on Tiltaxes, with 10 rollouts per pattern.

MethodHittingSweepingHooking
TiltaxesTiltaxesTiltaxes
P01P02Avg.P01P02Avg.P01P02Avg.
FM0.000.400.200.100.300.200.000.200.10
FuncBridge0.201.000.600.700.500.600.600.600.60
AuthorsOpenReview ↗

Response to Reviewer HtrU (2/3)

Responses to Weaknesses and Questions


Question 1 (CW1, Q1):Unclear scope of unseen-tool generalization and insufficient real-world evaluation.

Answer: Thanks for highlighting the evaluation split. System-2 is pretrained on action-free keypoint trajectories from all original benchmark tools (10 simulation and 7 real-world, Fig. 3), while System-1 uses action-labeled data only from seen tools. Thus, the original evaluation is System-1-unseen, not full-unseen.

To evaluate full-unseen generalization, we add Canoe, Flask, and Tablet in simulation (Figs. 14–16, App. B.2) and Tiltaxes in the real world (Fig. 10, App. B.1). None are used in System-2 pretraining or System-1 fine-tuning (Tab. A). Additionally, training and evaluation use distinct initial configurations, and the human videos contain no full-unseen tools.

Table A. Training and Evaluation Data Splits. Tool and trajectory statistics for System-2 pretraining, System-1 fine-tuning, and evaluation. Reported per function.

DomainSystem-2 PretrainingSystem-1 Fine-tuningSystem-1-Unseen EvaluationFull-Unseen Evaluation
Simulation10 tools (900 traj.)Hammer, Mustard, Tobject, Pickaxe (360 traj.)Scoop, Shoe, Book (within the 10 tools)Canoe, Flask, Tablet (outside the 10 tools)
Real-World7 tools (280 traj.)Hammer, Mustard, Pickaxe (120 traj.)Tobject, Book, Shoe (within the 7 tools)Tiltaxes (outside the 7 tools)

As shown in Tabs. 2 and 4, FuncBridge achieves higher average SRs on full-unseen tools in both simulation and the real world, demonstrating that function-aware keypoint trajectories support functional generalization even to full-unseen tools. We also expand real-world sweeping and hooking to Tobject, Book, and Shoe, where FuncBridge maintains its advantage (Tab. 3).

Qualitative results are provided in Figs. 8–10 (App. B.1) for real-world experiments and Figs. 14–16 (App. B.2) for full-unseen simulation tools.


Question 2 (CW2, Q2, CW3): Lack of comparison with ATM pretrained on the same data and unclear distinctions from ATM and Im2Flow2Act

Answer: Thanks for the comments. To ensure a fair comparison, we pretrain ATM's tracker on the same action-free keypoint trajectories and fine-tune its downstream policy on the same action-labeled data as FuncBridge, using matched training budgets (App. A.3). We denote this baseline as ATM (Pretrained).

Table B (App. B.3, Tab. 9). Real-World Hitting Results on System-1-Unseen and Full-Unseen Tools. Average success rate (SR) across two functional patterns, with 10 rollouts per pattern.

System-1-UnseenFull-Unseen
MethodTobjectBookShoeTiltaxes
FM0.300.250.100.20
ATM (Pretrained)0.550.600.350.50
FuncBridge0.700.800.400.60

As shown in Tabs. 1–2 and Tab. B (App. B.3, Tab. 9), FuncBridge outperforms ATM (Pretrained) in average SR across all three functions in both simulation settings and on real-world hitting. Compared with ATM, FuncBridge uses flow matching to predict functional keypoint trajectories and introduces random keypoint perturbations during System-1 training to improve robustness to prediction errors (Sec. 3.3). These designs combine function-aware motion prediction with grounded execution across novel tools. Qualitative comparisons are provided in Figs. 14–16 (App. B.2).

To clarify our contribution, Tab. C compares the research focus, method, and evaluation of FuncBridge, ATM, and Im2Flow2Act.

Table C. Comparison with ATM and Im2Flow2Act.

MethodProblemMethodEvaluation
ATMLearns general motion information from action-free videos to improve robot generalization across manipulation tasks.Predicts future trajectories of specified points and uses them to guide downstream policy learning. Does not specifically study how functional representations affect generalization across tools.Evaluates performance across manipulation tasks, but does not explicitly target functional generalization across geometrically different tools using designated functional points.
Im2Flow2ActFocuses on learning manipulation skills from cross-domain data, including sim-to-real transfer.Uses object flow as an intermediate representation, with flow generation and a flow-conditioned policy.Evaluates cross-domain skill transfer rather than functional generalization across geometrically different tools.
FuncBridgeFocus on functional generalization: performing the same function with novel tools of different geometries.Systematically compares functional representations, selects 2D keypoint trajectories to represent functional motion, and proposes a two-stage policy combining functional keypoint reasoning and grounded execution.Evaluates functions across diverse tools, requiring the same function to be completed using a designated functional point or following a specified functional pattern.
AuthorsOpenReview ↗

Response to Reviewer HtrU (3/3)

Question 3 (CW3):Do keypoint trajectories offer advantages across different functions, and are they easier to predict?

Answer: To evaluate action groundability across functions, we extend the provided-reference comparison between object masks and keypoint trajectories from hitting to sweeping and hooking.

Table D (App. B.4, Tab. 10). Functional Representation Comparison Across Functions. Success rate (SR) on different functions when provided-reference object masks or keypoint trajectories condition the same execution policy.

FunctionObject MasksKeypoint Trajectories
Hitting0.320.49
Sweeping0.360.53
Hooking0.120.28

As shown in Tab. D (App. B.4, Tab. 10), keypoint trajectories outperform object masks across all three functions under the same provided-reference setting. These results demonstrate their stronger action groundability for different functional tasks.

Regarding prediction difficulty, functional generalization primarily depends on how the functional contact point moves relative to the target. Keypoint trajectories capture this essential motion, providing sufficient information to guide functional execution across different tools, as supported by Tab. D. Compared with videos and object masks, they remove unnecessary visual details and dense shape information, resulting in a compact, low-dimensional representation that is easier to predict.


Question 4 (Q3): Function conditioning and closed-loop execution

Answer: We train separate System-2 and System-1 models for hitting, sweeping, and hooking. Therefore, function is specified by the selected model, without requiring an additional task-conditioning token. Each model uses the designated functional and target points.

For keypoint updates and replanning, System-2 predicts future functional keypoint trajectories from the current observation and keypoint positions. System-1 then generates an action chunk conditioned on the predicted trajectories, current observation, and robot state (Sec. 3.3, Eqs. (3)–(4)). After executing each action chunk, we update the current keypoint positions from the new observation and predict new trajectories and actions, enabling closed-loop execution.


Question 5 (CW1, Q4): Robustness to viewpoint changes and background clutter

Answer: Robustness is important for policy models, and we thank the reviewer for this suggestion. We further evaluate FuncBridge under two separate visual perturbations across hitting, sweeping, and hooking: adding irrelevant tools to the background and rotating the camera by 10°. The evaluation settings are illustrated in Fig. 17(a) (App. B.5).

Table E (App. B.5, Fig. 17). Robustness Under Background Clutter and Viewpoint Changes. Success rate (SR) of FuncBridge across three functions under the standard setting, background clutter, and a 10° camera rotation.

FunctionStandardBackground ClutterCamera Rotation (10°)
Hitting0.360.260.31
Sweeping0.450.360.38
Hooking0.230.150.18

As shown in Tab. E, FuncBridge maintains satisfactory success rates across functions under different perturbations, demonstrating considerable robustness of keypoints-based functional representations.


Question 6 (CW1):Limited evaluation on VLM-based functional point selection.

Answer: We extend the original hitting-only VLM point-selection evaluation (Fig. 5) to sweeping and hooking, using manually specified points for comparison.

Table F (App. B.6, Tab. 11). VLM-Based Functional Point Selection Across Functions. Success rate (SR) of FuncBridge with manually specified and VLM-selected functional and target points across hitting, sweeping, and hooking.

FunctionManual PointsVLM-Selected Points
Hitting0.360.31
Sweeping0.450.42
Hooking0.230.23

Results in Tab. F (App. B.6, Tab. 11) show that FuncBridge can use a VLM to automatically select functional points across different functions, reducing reliance on manual specification.


Question 7 (Q4): Training variability and uncertainty in the main results and human-video gains

Answer: For human-video pretraining, we fix the System-2 checkpoint pretrained with 525 additional human videos and fine-tune System-1 using two new random seeds. The additional runs achieve success rates of 0.39 and 0.40, close to the original result of 0.39. These results indicate that the improvement from human-video pretraining is reasonably stable.

For the main experiments, FuncBridge consistently outperforms the baselines across all three functions (Tabs. 1–4), with substantial performance gains. We therefore believe these improvements mainly stem from the effectiveness of our method rather than variations caused by training randomness.

Comments on the paper

8 comments
AuthorsOpenReview ↗

Summary of the Rebuttal and the Major Changes of Revised Manuscript

Dear Reviewers,

We sincerely thank all of you for the careful reviews and constructive feedback. We particularly appreciate your recognition of FuncBridge’s contributions: Reviewer HtrU highlights a relevant problem with quantitative evidence across functions. Reviewer tWSQ values the meaningful and underexplored problem, systematic representation comparisons, and the scalability with heterogeneous data. Reviewer NPc1 recognizes the effectiveness of the geometry-aligned intermediate representation and extensive simulation and real-world validation.

We also recognize the reviewers' concerns regarding full-unseen generalization, fair and more comprehensive comparisons, and robustness. During the rebuttal period, we have conducted targeted experiments and analyses to address these concerns. We have incorporated the findings into the revised manuscript, with revised text highlighted in red and new figures and tables in blue. The main updates are summarized below:

1. Data Splits and Full-Unseen Generalization. We define System-1-unseen tools as included only in System-2 pretraining and full-unseen tools as excluded from both training stages. We further add full-unseen evaluations in simulation (Canoe, Flask, Tablet) and the real world (Tiltaxes). (§4.1, Tables 2 and 4; Appendix A.3)

2. Data-Matched ATM Comparisons. We evaluate ATM (Pretrained) with matched training data and budgets, showing that FuncBridge achieves higher performance across all three functions in simulation and on real-world hitting. (Tables 1–2; Appendices A.3 and B.3, Table 9)

3. Real-World Evaluation Expansion. We extend real-world sweeping and hooking from Book alone to Tobject, Book, and Shoe. (Table 3; Appendix B.1)

4. Representation and Prediction-Error Analysis Across Functions. We extend the provided-reference comparison to all three functions, further supporting the effectiveness of keypoint trajectories for action grounding. We also quantify System-2 prediction errors to help explain the performance gap between reference-guided execution and the complete pipeline. (Appendix B.4, Table 10; detailed reviewer responses)

5. Robustness of FuncBridge. We evaluate FuncBridge under background clutter and 10° camera rotation across all three functions, demonstrating its robustness to visual perturbations. (Appendix B.5, Figure 17)

6. VLM-based Autonomous Point Selection. We extend VLM-based functional point selection to all three functions, demonstrating the potential of FuncBridge for more autonomous functional tool use. (Appendix B.6, Table 11)

7. Methodological Scope and Related-Work Distinctions. We clarify how functional generalization differs from category-level and affordance generalization, and further distinguish FuncBridge from existing methods. (§1–2)

We believe these substantial revisions, together with the detailed rebuttal and enriched manuscript, have significantly improved the quality, clarity, and completeness of FuncBridge. We would greatly appreciate your feedback on whether our responses have addressed your concerns, and we would be happy to provide any further clarification during the discussion period.

Best regards,
Authors of Submission 31587

Dear Reviewers,

A quick reminder that the discussion period ends on November 18 (AoE). If you haven't already, please review the authors' rebuttal and provide a brief follow-up indicating whether your concerns have been adequately addressed or if further clarification is needed.

Your timely feedback would be greatly appreciated and will help facilitate the final decision.

Thank you for your time and contributions!

Best regards,

AC

AuthorsOpenReview ↗

Looking forward to your reply

Dear Reviewer NPc1,

We sincerely appreciate your time and effort in reviewing our submission and providing constructive feedback.

As the discussion phase will end in 1 day, we would be very grateful if you could take a moment to review our rebuttal responses. We look forward to hearing your feedback and would be happy to provide further clarifications or address any additional concerns.

Thank you again for your time and consideration.

Best regards,
Authors of Submission 31587

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.