acceptodds
Under review as a conference paper at ICLR 2027

AFUN: Towards an Affordance Foundation Model for Functional Understanding

Abstract

Affordance understanding helps connect what a robot sees with what it should do. It provides an interpretable way for robots to manipulate objects in open, messy real-world environments. However, building a general affordance model is still difficult. Such a model needs to understand both where to interact with an object and how the object should move after interaction. It also needs to work across many different environments, objects, and tasks. Most existing methods solve only part of this problem. Some methods can find the relevant object region but do not predict a usable motion. Others can predict motion, but they do not scale well to diverse real-world settings. In this paper, we introduce AFUN, a step toward a general affordance foundation model for understanding object functionality. Given one RGB-D image and a language instruction, AFUN predicts a task-specific functional mask, showing where to interact, and a 3D motion curve, showing how the object should move after contact. To improve generalization, we build a large-scale data pipeline that converts data from robots, humans, simulations, and real-world scans into a unified affordance format. This format includes language instructions, functional masks, and object-centered 3D motion labels. We evaluate AFUN in three ways. For affordance segmentation, AFUN outperforms all baselines by a large margin across 8 test sets from 4 benchmarks, improving mean gIoU/cIoU by +23.9/+26.3. For contact-point prediction, it predicts much more accurate contact points, improving hit rate by 12.7-61.3% over the best baseline. For 3D motion prediction, it achieves the best results on all three test sets. Finally, AFUN can be used for real-world robot manipulation without robot-specific finetuning or task-specific heuristics, showing its ability to adapt to open-world affordance tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Affordance, Functionality Understanding, Robotics, Perception

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_A3SdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an affordance foundation model. From one RGB-D image and a language instruction, AFUN predicts a task-conditioned affordance mask (where to interact) and an object-centric 3D Bézier curve (how to interact).

Critical strengths
  1. Predicting both where and how in one model is a useful formulation. Representing motion with the object’s trajectory, rather than the hand or gripper, makes the labels less dependent on the embodiment.

  2. The gains are large and consistent (+23.9 mean gIoU; best on all motion test sets). The training-data control (Table 5) shows the gains do not come from data alone.

Critical weaknesses
  1. The claim about the dataset scale in intro and first figure needs a quantitative comparison. The dataset is called “large-scale”, but the final set has 59,867 samples, fewer than HOVA-500K, which the paper already uses. Please add a quantitative comparison with prior affordance datasets (size, categories, modalities, release status).

  2. Real-robot details are needed and should be enriched. Please report: the number of trials per task and how the setups vary between trials; how the curve is converted into robot commands; the main failure causes; the share of Franka data in training, which matters for interpreting the “no embodiment-specific fine-tuning” claim.

  3. Curve fitting is also a concern. The motion labels are planar constant-curvature fits of the raw trajectories (App. C.2). Please report the fitting residual and how many samples are dropped at this step. Please also discuss which motions the representation cannot express, e.g., non-planar, multi-segment or discontinuous motions, and in-place rotations.

Questions for authors

See Weaknesses #3.

4 comments

We thank the reviewer for the detailed and specific feedback, and for recognizing the where-and-how formulation and the consistency of our gains. To address your concerns, we provide a quantitative comparison with prior datasets, the requested real-robot details, and an analysis of the curve fitting and its limitations below.

1. Dataset scale and scene diversity

Our segmentation training relies on prior affordance segmentation datasets, and our AFUN dataset is mainly for motion training. We give a more detailed comparison in the table below with the datasets in our motion baselines. We have a larger number than our previous motion datasets, but it's still far from millions. We'll relax this claim in the final version.

Dataset (used by)ModalitiesEpisodesTrajectories / instancesRegionMotionLanguageScenesVerbs / objectsAffordance Data Released
HOVA-500K (GLOVER++)images–500K2D contact pointnoneclass labelsn.r.675 / 1,726✓
EPIC-KITCHENS (VRB)human700~54K2D contact points2D hand trajectory✗4597 / 300✗
EPIC-KITCHENS (VidBot)human495~50K3D contact points3D hand trajectory✓4097 / 289✗
HOI4D (General Flow)human2,05551,264object mask (input only)3D object point flow✓12010 / 14✗
A0 data (A0)human, robot, sim8,929~27Kcontact point2D object waypoints✓38672 / 16✓
AFUN (ours)robot, human, sim, scan22,466223,334part mask3D object curve✓1,720353 / 765upon acceptance
AFUN-HQ (ours)robot, human, sim, scan4,25059,867part mask3D object curve✓713139 / 178upon acceptance

2. Real-robot details

We ran 10 single-attempt trials per task (40 in total) with no retries or human intervention, fixed cameras and start pose, and randomized object position and in-plane orientation (table below; the microwave range is smaller due to reachability). A trial succeeded if the screwdriver was lifted clear of the table, the pot lid was moved clear of the pot, the drawer opened at least halfway, or the unlatched microwave door opened at least 80°, giving 36/40 successes (90%).

TaskPosition randomization rangeOrientation rangeSuccesses
Pick Up ScrewdriverApprox. 20 x 20 cmFull 360°10/10
Take Off Pot LidApprox. 30 x 20 cmFull 360°10/10
Open DrawerApprox. 20 x 10 cmApprox. ±30°8/10
Open MicrowaveApprox. 10 x 10 cmApprox. ±20°8/10

Deployment detail. AFUN takes one RGB-D view; AnyGrasp uses both views, and we pick its highest-scoring grasp whose trajectory has a valid IK solution. We sample the Bézier curve into waypoints in the robot frame, align its start with the grasp, rotate the gripper along the curve's tangent, time it with Ruckig, and execute joint-impedance commands through franky without replanning. Only workspace cropping is task-specific.

Failure analysis. All four failures happened on where a small error grows over time: a path that starts slightly off the drawer's rail or the microwave door's swing moves further from the correct path as the robot pulls, until the gripper slips off the handle. In most trials, the predicted motion can enable the robot to complete the task (16 of 20 drawer and microwave trials succeed).

The share of Franka data and cross-embodiment claim: Only 9.51% of the samples in our motion dataset involve a Franka robot. Our dataset is embodiment-agnostic because we track only how objects move and fit the curve to that motion. No robot state, action or kinematics is used in training. The majority of data is from SceneFun3D, which is human-annotated and embodiment-free. We also have egocentric data of humans manipulating objects.

Our method is cross-embodiment because, unlike VLAs, which directly predict actions, we predict how the object would move (the motion), and use IK to solve for the waypoints the robot should follow (see details above).

3. Curve-fitting details

Fitting residual and drops. We define the fitting residual as ε = RMS distance between the tracked 3D points and the fitted arc, divided by the arc length (we take the larger of the raw-point and smoothed support-point RMS). Samples are dropped by thresholding ε. We keep ε < 0.08, which means an RMS deviation under 8% of the arc length. This removes 34% of fits. For human videos (VITRA) we set a separate threshold, ε < 0.125, which discards 89% of fits. Retained labels have a median residual of 11.7 mm (p90 31.8 mm), corresponding to a median ε of 0.048 (p90 0.082). SceneFun3D labels are converted directly from annotated articulation parameters and involve no fitting. We will add these to App. C.2.

Discussion of the representation. We agree the curve fitting process is planar, and it cannot capture the complex action, such as "stir the cereal". Regarding multiple motion, AFUN can predict a separate mask and curve for each moving part, such as the two drawer knobs in Figure 1(b). We acknowledge that long, multi-segment/discontinuous motions can not currently be represented by a single curve, Future work needed for predicting multiple curves with more control points. In-place rotations (e.g., turning a knob) are also not currently supported, but could be expressed as a partial, off-axis mask with a circular arc about the rotation axis. We will extend these into our limitations and future work section.

Reviewer_A3SdOpenReview ↗

Reply to the responses by authors

Thank you for the detailed response, which addresses some of my concerns. The dataset scale table is especially welcome. Considering the remaining limitations, I will keep my original score. I consider this paper to be above the acceptance borderline.

Thank you for the follow-up and for your positive assessment of our paper. We are glad the dataset comparison table was helpful.

In the final version, we will add the dataset comparison, real-robot details and curve-fitting analysis, relax the “large-scale” claim, and expand the limitations section on the motions our curve cannot yet represent.

If there are specific remaining concerns that we could address during the discussion period, we would be happy to provide further clarification or results.

Reviewer_JhMkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses open-world affordance prediction. To build an affordance foundation model, the paper proposes a scalable data pipeline, a model architecture, and a training scheme for stable training. The experimental results show that the proposed model outperforms existing methods on affordance mask segmentation, contact point prediction, and 3D motion prediction. In addition, the authors deploy the model on a real robot as a demonstration.

Critical strengths
  • The paper is well written and easy to follow. In particular, the Introduction clearly explains the motivation for an open-world affordance foundation model.
  • The paper proposes a data pipeline that converts various types of datasets into a consistent format for training, which is highly practical. Using this pipeline, the authors build a large-scale dataset and use it to train the model.
  • The experiments are comprehensive. The results show strong performance of the proposed method on various affordance-related tasks: mask segmentation, contact point prediction, and 3D motion prediction.
Critical weaknesses
  • The paper does not evaluate the quality (accuracy) of the data that the proposed pipeline annotates automatically. Since the data pipeline is one of the main contributions of this paper, this evaluation is important. The data quality is evaluated only indirectly, through the final performance of the model trained on the data. A direct evaluation of the data itself (i.e., its accuracy) is also needed.
  • The source code is not included in the submission, which raises concerns about reproducibility. Since the data pipeline is one of the main contributions, the contribution of this paper will be limited if the code is not released, even though the paper itself is of high quality. Releasing the code would also benefit future research in this field.
  • In addition to the code, releasing the created dataset would make the contribution of this paper stronger.
  • The paper mentions a manual annotation process, but does not describe its details.
Questions for authors

Will the source code and the created dataset be released?

3 comments

We thank the reviewer for the encouraging feedback on the writing and the results, and for recognizing the data pipeline as a clear contribution. To address your concerns, we provide a direct evaluation of the data quality, details of the manual annotation process, and anonymous links to the code, dataset and checkpoints below.

1. Data pipeline accuracy

We would like to clarify that our data pipeline acts primarily as a filter: In our dataset, the end-to-end pipeline (including the human quality control) removes 95.2% of the original data (e.g., clips with poor task grounding, occlusion, unreliable segmentation, or insufficient motion), as shown in Table 9 and Section 3 of the paper.

We reviewed the human QC annotation record and report a 38.4% average acceptance rate from the result of the pipeline in all robotic data sources, serving as a human-verified accuracy..

To investigate this further, we re-ran the pipeline on 10,000 random DROID samples with GPT-6 in place of Qwen. With GPT-6, the pipeline filters more strictly, achieving a higher human acceptance rate of 70.00%. Details of the 10,000 comparison are in the table below.

Pipeline VLMIntervals keptHuman acceptance
Qwen3.5-35B (paper)812327 (40.27%)
GPT-6370259 (70.00%)

2. Manual annotation and data details

We built a review tool in which annotators filter out the noisy samples that survive the pipeline. The annotator sees the RGBD image, the point cloud, and the 3D trajectory along with the language, and determines if the trajectory and segmentation are aligned with the task. A screenshot of the tool is provided at https://github.com/AFUN-anonymous/AFUN-anonymous/blob/main/assets/annotator.png.

3. Anonymous code release and dataset release

We provide the full training, data pipeline, and evaluation code at https://github.com/AFUN-anonymous/AFUN-anonymous.git, and the dataset and checkpoints at https://huggingface.co/afun-anonymous. The code and dataset are ready and will be publicly released upon acceptance.

Thank you for the response. The anonymous links to the code, dataset, and checkpoints, together with the commitment to release them upon acceptance, resolve my concern about the release. However, I still have the following concerns about the data pipeline.

1. The accuracy of the labels is still not shown

The 38.4% acceptance rate in the rebuttal shows what proportion of pipeline outputs the annotators accepted, not how accurate the accepted labels are, so my original concern remains. A direct evaluation of the label quality matters for three reasons:

  • The pipeline is one of the main contributions. The downstream results, including the data controls in Table 5, show that the data are useful for training, but not that the labels are accurate, since a model can perform well even with noisy labels.
  • The AFUN test set in Table 3 is built from the pipeline output. Label errors therefore become ground-truth errors and can affect the results on this test set.
  • The dataset will be released. Without knowing how much error the labels contain, it is difficult for users to judge whether the dataset suits their purposes.

2. The reproducibility of the human verification remains unclear

The rebuttal describes the review tool but reports neither the number of annotators nor the inter-annotator agreement. Even with the released dataset, applying the pipeline to new data with comparable quality requires reproducing this step. Without this information, it is unclear how consistent the judgments are, and thus whether others could reproduce them.

3. The scalability claim is not sufficiently supported

The paper describes the pipeline as scalable. However, an acceptance rate of about 40% on the robotic sources means that about 60% of those outputs are rejected, so obtaining usable data currently relies on checking every output by hand. Even with GPT-6, 30% is still rejected. Accepting or rejecting a sample may take less effort than annotating it from scratch, but since the human cost is not reported, it remains unclear whether the pipeline is scalable in practice.

Rating

My concern about the release has been resolved. However, the data pipeline is one of the main contributions, and the evidence for its output quality, the reproducibility of its human verification, and its scalability is still insufficient. Therefore, I will keep my rating. If I have missed information that already addresses these concerns, I am happy to reconsider my rating.

Thank you for the follow-up, and we apologize that our first response was unclear and caused miscommunication. We address the three points below.

1. The accuracy of the labels

The 38.4% is measured on the raw pipeline outputs and describes the automatic pipeline before verification, not how accurate the accepted labels are. To measure the accuracy of the accepted labels directly, we ran an additional audit of the human-verified AFUN-HQ set, which is the one used to train our network. We took all 121 samples of the AFUN test set and a random sample of 360 training samples, shuffled them together, and re-reviewed them.

Training setTest set
Re-reviewed360121
Kept346 (96.11%)121 (100.00%)

The test set was curated with extra care: after sampling, it went through a second human quality-control pass (Section 5.2.3 of the paper), and in this audit all 121 samples were kept again, so ground-truth errors on the AFUN test set in Table 3 should be rare. For the training set, 14 of the 360 audited samples (3.89%) were rejected because of annotator oversights, giving an estimated label accuracy of 96.11% (95% confidence interval: 93.6–97.7%).

2. Reproducibility of the human verification

Annotators. The verification was done by two annotators and took 32.8 annotator-hours in total for the full AFUN-HQ set.

Criteria. A sample is kept only if all three hold; otherwise it is discarded:

  • (i) the mask lies on the object part that has to be contacted to carry out the instruction;
  • (ii) the motion curve starts at the masked part;
  • (iii) the direction and shape of the curve match the instructed action.

Consistency. The audit in point 1 is a second, blind pass over samples that were accepted into AFUN-HQ. It kept 96.11% of the 360 training samples and 100% of the 121 test samples, so the two passes agree on at least 96% of the accepted samples.

Release. The review tool is in the anonymous repository. With the dataset we will also release all keep/discard decisions, so the verification can be inspected and repeated on new data.

3. Scalability

We agree that the final dataset relies on human verification, and we will relax the "scalable" claim and state this clearly in the final version. What the pipeline does is make the human part small.

The pipeline starts from 1,242,740 intervals (line 178). Without it, an annotator would have to screen every clip for problems such as occlusion, a wrong camera or missing motion, and then draw a mask and mark a 3D trajectory for each remaining one. The pipeline does all of this automatically: screening, masks, tracking, 3D lifting and curve fitting. Regarding "checking every output by hand": the human never has to look at the raw inputs. Only the small fraction of samples that pass all automatic stages is shown to a human, and for those the task is a yes/no decision. Each decision takes a few seconds, and less for SceneFun3D, where only the validity of the viewpoint is checked. In total, the manual work behind the dataset is about 32.8 annotator-hours for a starting pool of about 1.24M intervals.

The pipeline also adapts to a wide variety of sources, such as robot teleoperation, simulation, human egocentric video and 3D scans (see Figures 10 and 11 in the appendix for visualization). Because it only needs a trackable object in an RGB-D video, adding a new source only requires a new preprocessing adapter (Appendix C).

We hope these results address your remaining concerns, and we are happy to provide any further information.

Reviewer_TQDWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents AFUN, a model that takes a single RGB-D observation and a language instruction and predicts both a task-relevant affordance mask and a 3D post-contact motion curve. The authors also build a unified data pipeline that converts human videos, robot demonstrations, simulation data, and scene scans into a common format with language instructions, affordance masks, and 3D motion labels, yielding 59,867 training samples. AFUN combines frozen Qwen3-VL, SAM3, and Sonata models, using a MetaQuery interface and a motion decoder to predict masks and Bézier curves. It performs well across several segmentation and motion test sets, and demonstrations on four real-world robot tasks show its potential for practical use.

Critical strengths
  1. The authors bring together data from different sources and standardize the language, masks, depth, and motion annotations. They also filter the data and manually check its quality. This is one of the paper’s clearest contributions.
  2. Predicting both where to interact and how to move after contact is more useful for robot manipulation than predicting affordance regions alone. It also makes the connection between perception and action easier to understand.
Critical weaknesses

The methodological novelty appears limited. AFUN follows the established VLM-based segmentation paradigm used in works such as 3D-AffordanceLLM and SegPoint, adopting the existing MetaQuery interface, stronger pretrained components, and an additional motion prediction branch. These choices have engineering value, but the new modeling contribution remains unclear. Same-data comparisons help, yet differences in backbones and training procedures still make it difficult to isolate the method’s contribution. Predicting masks and motion curves together also does not necessarily demonstrate effective joint learning. A comparison with independently trained motion prediction is needed to show whether AFUN improves over simply adding a standard motion head to an existing segmentation model.

Questions for authors
  1. With the same pretrained components, training data, and compute budget, how does AFUN compare with an existing VLM segmentation framework plus a standard 3D motion head? Could you provide a direct comparison to isolate the contributions of MetaQuery and the joint architecture?
  2. Does joint segmentation and motion training improve motion prediction? Could you compare separate training, motion training with the segmentation branch frozen, and full joint training?
2 comments

Thank you for the careful and constructive review, and for recognizing the value of jointly predicting where and how to interact and the practicality of our data pipeline. To address your concerns, we clarify how AFUN differs from prior VLM-based segmentation methods and provide new data-matched comparisons below that isolate the contributions of MetaQuery and of joint training.

1. Architecture and novelty

Novelty and difference from 3D-AffordanceLLM and SegPoint: We agree that the VLM-based segmentation paradigm itself is not new; 3D-AffordanceLLM and SegPoint apply it to object point clouds and room scans. AFUN differs in three ways.

  1. Both works assume clean, prepared point clouds: isolated PartNet shapes for 3D-AffordanceLLM (its partial views are rendered from four virtual cameras, with no sensor noise or clutter) and multi-view ScanNet reconstructions for SegPoint. Such scans are rarely available in deployment. AFUN instead works on a single real-world, potentially noisy RGB-D image, the robot's native observation.
  2. To our knowledge, AFUN is the first to predict a task-conditioned mask and 3D post-contact motion together, end to end, from one RGB-D frame and an instruction. Prior VLM methods stop at a mask or grasp, and motion models such as A0 and VRB rely on contact points or heatmaps, which often struggle to locate the object in the first place. A0 hits the ground-truth region in at most 22.7% of cases, versus at least 67.3% for AFUN (in manuscript Table 2).
  3. We also introduce a new affordance representation that combines a task-conditioned mask, lifted to 3D with depth, with an anchored 3D Bézier curve for motion. Unlike articulation parameters, it covers both articulated and free motion in one representation, with lower error than the OPD representation (Table 6).

MetaQuery design choice: MetaQuery passes more information (32 query tokens vs. a single token) to the downstream models with about 43% fewer trainable parameters (≈4.7M vs. ≈8.3M in Stages 1–2). The table below compares its trainable parameters and conditioning with the LoRA + <AFF> interface for Stages 1–2. Under the same data seen, LoRA + <AFF> only reaches 66.2 gIoU vs. MetaQuery's 69.3 (Table 4). We also extends this comparison to motion in this rebuttal. In both comparisons, LoRA with special tokens performs slightly worse on most metrics.

LoRA + <AFF>MetaQuery
LLM bodyLoRA rank 8, alpha 16, dropout 0.05, on q_proj and v_proj in all 36 layers: 3.83Mfrozen
New token rows<AFF>, <MOT>: 2 rows, 8K64 MetaQueries + 2 boundary tokens: 66 rows, 0.27M (34 rows in Stage 1)
LLM → SAM3 projector4096→1024→256: 4.46Msame: 4.46M
Total trainable, Stages 1–2≈ 8.3M≈ 4.7M
SAM3 conditioning (tokens × dim)1 × 256: projected <AFF> hidden state32 × 256: projected semantic MetaQuery states
Motion-decoder conditioning (tokens × dim)1 × 4096: <MOT> hidden state32 × 4096: motion MetaQuery states

Pretrained components. Stronger pretrained components alone do not explain all the gains. The same Qwen3-VL-8B and SAM3 used zero-shot reach 42.5 mean gIoU, versus 69.3 for AFUN (Table 1), and AffordanceNet retrained on our Stage 2 data reaches 56.8, versus 71.1 for AFUN-Seg (Table 5).

2. Controlled comparison with matched components; separate, frozen, and joint training

To address your concern, we trained three additional models to compare with AFUN. All three are trained on the same data as AFUN, with the same number of training samples seen, so the comparison is data-matched:

  1. AFUN Stage 3 with the segmentation branch (all Stage 2 parameters) frozen.
  2. AFUN with LoRA and a special <AFF> token (following prior work), trained through Stage 3 with a <MOT> token for motion.
  3. One of our baselines, AffordanceNet (RAGNet), with an additional geometry encoder and motion decoder. We add another special token <MOT> for motion cross-attention. We initialize it from the AffordanceNet trained on the AFUN corpus (as in Table 5).
VariantAFUN test Abs. ADE / FDE ↓AFUN test CIM ↑SceneFun3D Abs. ADE / FDE ↓SceneFun3D CIM ↑RoboMIND2 Abs. ADE / FDE ↓RoboMIND2 CIM ↑
AFUN (default)0.098 / 0.13981.00.351 / 0.44167.30.254 / 0.32362.2
Frozen segmentation0.147 / 0.19269.40.484 / 0.56349.50.256 / 0.34762.2
LoRA + <AFF> <MOT> token0.124 / 0.15779.30.363 / 0.42266.00.269 / 0.34559.6
AffordanceNet + motion decoder0.130 / 0.14971.90.495 / 0.56344.80.351 / 0.41133.3

The motion head predicts from SAM3's per-object features, which only learn to contain semantic information in stage 2. Keeping it frozen makes it harder to converge. Joint training lets the motion loss add motion information to them without changing much mask quality.

For the other two variants, predicting the right object is crucial, since a wrong object could lead to an entirely different motion. AffordanceNet lags behind in segmentation, which makes its motion decoder harder to train. The LoRA + token variant is slightly worse than AFUN on most metrics, but due to extra training parameters, it takes more compute to converge on the same dataset.

Comments on the paper

1 comment

We sincerely thank all reviewers for their careful reading of our paper and their thoughtful, specific feedback. We are glad they found predicting both where and how to interact a useful formulation (TQDW, A3Sd), the data pipeline practical and a clear contribution (TQDW, JhMk), the paper well written (JhMk), and the gains large and consistent (JhMk, A3Sd). In this rebuttal, we add matched comparisons that isolate the contributions of MetaQuery (vs. LoRA + special tokens) and of joint training (TQDW); a direct evaluation of data quality, annotation details and code/data release (JhMk); and a dataset comparison, real-robot details and curve-fitting analysis (A3Sd).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.