- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents an affordance foundation model. From one RGB-D image and a language instruction, AFUN predicts a task-conditioned affordance mask (where to interact) and an object-centric 3D Bézier curve (how to interact).
Critical strengths
-
Predicting both where and how in one model is a useful formulation. Representing motion with the object’s trajectory, rather than the hand or gripper, makes the labels less dependent on the embodiment.
-
The gains are large and consistent (+23.9 mean gIoU; best on all motion test sets). The training-data control (Table 5) shows the gains do not come from data alone.
Critical weaknesses
-
The claim about the dataset scale in intro and first figure needs a quantitative comparison. The dataset is called “large-scale”, but the final set has 59,867 samples, fewer than HOVA-500K, which the paper already uses. Please add a quantitative comparison with prior affordance datasets (size, categories, modalities, release status).
-
Real-robot details are needed and should be enriched. Please report: the number of trials per task and how the setups vary between trials; how the curve is converted into robot commands; the main failure causes; the share of Franka data in training, which matters for interpreting the “no embodiment-specific fine-tuning” claim.
-
Curve fitting is also a concern. The motion labels are planar constant-curvature fits of the raw trajectories (App. C.2). Please report the fitting residual and how many samples are dropped at this step. Please also discuss which motions the representation cannot express, e.g., non-planar, multi-segment or discontinuous motions, and in-place rotations.
Questions for authors
See Weaknesses #3.
4 comments
3. Curve-fitting details
Fitting residual and drops. We define the fitting residual as ε = RMS distance between the tracked 3D points and the fitted arc, divided by the arc length (we take the larger of the raw-point and smoothed support-point RMS). Samples are dropped by thresholding ε. We keep ε < 0.08, which means an RMS deviation under 8% of the arc length. This removes 34% of fits. For human videos (VITRA) we set a separate threshold, ε < 0.125, which discards 89% of fits. Retained labels have a median residual of 11.7 mm (p90 31.8 mm), corresponding to a median ε of 0.048 (p90 0.082). SceneFun3D labels are converted directly from annotated articulation parameters and involve no fitting. We will add these to App. C.2.
Discussion of the representation. We agree the curve fitting process is planar, and it cannot capture the complex action, such as "stir the cereal". Regarding multiple motion, AFUN can predict a separate mask and curve for each moving part, such as the two drawer knobs in Figure 1(b). We acknowledge that long, multi-segment/discontinuous motions can not currently be represented by a single curve, Future work needed for predicting multiple curves with more control points. In-place rotations (e.g., turning a knob) are also not currently supported, but could be expressed as a partial, off-axis mask with a circular arc about the rotation axis. We will extend these into our limitations and future work section.
Reply to the responses by authors
Thank you for the detailed response, which addresses some of my concerns. The dataset scale table is especially welcome. Considering the remaining limitations, I will keep my original score. I consider this paper to be above the acceptance borderline.
Thank you for the follow-up and for your positive assessment of our paper. We are glad the dataset comparison table was helpful.
In the final version, we will add the dataset comparison, real-robot details and curve-fitting analysis, relax the “large-scale” claim, and expand the limitations section on the motions our curve cannot yet represent.
If there are specific remaining concerns that we could address during the discussion period, we would be happy to provide further clarification or results.
We thank the reviewer for the detailed and specific feedback, and for recognizing the where-and-how formulation and the consistency of our gains. To address your concerns, we provide a quantitative comparison with prior datasets, the requested real-robot details, and an analysis of the curve fitting and its limitations below.
1. Dataset scale and scene diversity
Our segmentation training relies on prior affordance segmentation datasets, and our AFUN dataset is mainly for motion training. We give a more detailed comparison in the table below with the datasets in our motion baselines. We have a larger number than our previous motion datasets, but it's still far from millions. We'll relax this claim in the final version.
2. Real-robot details
We ran 10 single-attempt trials per task (40 in total) with no retries or human intervention, fixed cameras and start pose, and randomized object position and in-plane orientation (table below; the microwave range is smaller due to reachability). A trial succeeded if the screwdriver was lifted clear of the table, the pot lid was moved clear of the pot, the drawer opened at least halfway, or the unlatched microwave door opened at least 80°, giving 36/40 successes (90%).
Deployment detail. AFUN takes one RGB-D view; AnyGrasp uses both views, and we pick its highest-scoring grasp whose trajectory has a valid IK solution. We sample the Bézier curve into waypoints in the robot frame, align its start with the grasp, rotate the gripper along the curve's tangent, time it with Ruckig, and execute joint-impedance commands through franky without replanning. Only workspace cropping is task-specific.
Failure analysis. All four failures happened on where a small error grows over time: a path that starts slightly off the drawer's rail or the microwave door's swing moves further from the correct path as the robot pulls, until the gripper slips off the handle. In most trials, the predicted motion can enable the robot to complete the task (16 of 20 drawer and microwave trials succeed).
The share of Franka data and cross-embodiment claim: Only 9.51% of the samples in our motion dataset involve a Franka robot. Our dataset is embodiment-agnostic because we track only how objects move and fit the curve to that motion. No robot state, action or kinematics is used in training. The majority of data is from SceneFun3D, which is human-annotated and embodiment-free. We also have egocentric data of humans manipulating objects.
Our method is cross-embodiment because, unlike VLAs, which directly predict actions, we predict how the object would move (the motion), and use IK to solve for the waypoints the robot should follow (see details above).