acceptodds
Under review as a conference paper at ICLR 2027

UniAfford: Token-Routed Multitask Learning for Generalizable 2D-3D Affordance Perception

Abstract

Affordance perception aims to localize actionable regions that support embodied interaction, yet 2D and 3D affordance grounding have evolved as separate problems, with different task definitions, supervision formats, datasets, and evaluation protocols. This fragmentation limits the learning of transferable object–affordance semantics across visual and geometric spaces. We propose **Token Router for Tasks**, a general multitask training paradigm for MLLM-based systems that routes contextual hidden states to task-specific branches without requiring the language head to generate predefined task markers. Routed states are supervised directly by branch-specific objectives, enabling dense prediction losses to shape shared MLLM representations. We instantiate this paradigm as **UniAfford**, a unified framework for generalizable 2D–3D affordance perception, together with **UniAfford-Data**, a unified dataset that integrates pixel-level 2D annotations, point-level 3D annotations, and language instructions under a shared object–affordance taxonomy, supporting heterogeneous supervision through semantic-level 2D–3D pairing. UniAfford adopts an MLLM as a shared semantic hub and a modality-aware token router to produce image- and point-cloud-affordance queries. These queries respectively condition a SAM-style pixel decoder and a SONATA-based point decoder, enabling flexible 2D, 3D, and joint affordance inference from image-only, point-cloud-only, or paired multimodal inputs. Extensive experiments demonstrate strong zero-shot generalization across 2D and 3D affordance benchmarks without target-specific fine-tuning, alongside state-of-the-art branch-wise performance under modality-isolated training and evaluation protocols. Ablations demonstrate the importance of token routing, joint 2D–3D supervision, and decoder coupling, while language-head diagnostics show that routed latent states carry meaningful object–affordance semantics. Code and data will be publicly released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Affordance Perception, MLLMs, Embodied Intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_VnAaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Token Router for Tasks, a general multitask training paradigm for MLLM-based systems. Unlike LISA-style methods that couple task dispatch with the generation of predefined task tokens, the paradigm predicts branch assignments directly from contextual MLLM hidden states without requiring the language head to generate task markers. Routing anchors provide only routing supervision and are excluded from the language modeling loss; the routed states are supervised by branch-specific objectives — SAM-style pixel-level and SONATA-style point-level dense prediction losses — which in turn shape the shared MLLM representations. The authors instantiate this paradigm as UniAfford, a unified framework for generalizable 2D–3D affordance perception, supported by UniAfford-Data, a unified dataset that organizes 25k image and 69k point-cloud samples under a shared object–affordance taxonomy with semantic-level 2D–3D pseudo-pairing.

Critical strengths
  1. The unification of 2D and 3D affordance grounding is a novel and well-motivated contribution at the level of task organization, as prior works have treated these as separate problems.
  2. One model plus one unified dataset serves both modalities. The architecture demonstrates strong OOD zero-shot transfer (27.52 gIoU on AGD20K and 83.55 AUC on GEAL*) as well as state-of-the-art branch-wise performance (71.19 gIoU on ReasonAff versus 67.41 for Affordance-R1; 14.25 mIoU on PIAD versus 9.73 for DAG).
  3. The ablations are thorough and controlled: learned routing improves 3D mIoU from 17.46 to 34.56, joint 2D–3D supervision improves 2D gIoU from 41.41 to 68.79, and the language-head diagnostics confirm that routed states carry object–affordance semantics.
  4. 2D inference is notably efficient compared with reasoning-based baselines.
Critical weaknesses
  1. The framework stacks three large backbones (Qwen3-VL, SAM, and SONATA). The MLLM dominates the 3D inference cost (1,930.87 GFLOPs versus 21.15 for GREAT and 13.68 for IAGNet), which raises doubts about the feasibility of real-world robotic deployment.
  2. The semantic-level pseudo-pairing connects image and point-cloud instances only through shared object–affordance labels, without instance-level spatial correspondence, which the authors themselves acknowledge limits geometric consistency supervision.
  3. At the method level, the components are mostly combinations of existing techniques (MLLM + LoRA + SAM decoder + SONATA decoder + a lightweight router), so the novelty lies mainly in task organization and the training interface rather than in methodological breakthroughs.
Questions for authors
  1. The paper lacks comparisons or discussions with newer related methods, such as A4-Agent (ECCV 2026). Could the authors add these comparisons or discuss their relation to this work?
2 comments
AuthorsOpenReview ↗

Response to Reviewer VnAa — Part 1/2

Thank you for your time and thoughtful feedback. We appreciate your constructive comments and the opportunity to clarify our work. We address your concerns point by point below and welcome any further questions or requests for clarification.

W1: Computational Cost and Deployment Feasibility

We introduce the MLLM to support the unified task capabilities studied in this work: connecting the functional semantics in language instructions to pixel-level 2D and point-level 3D grounding, while allowing dense supervision from both modalities to jointly shape shared representations. Specialized 3D models focus on efficient prediction within specific input and output spaces, whereas UniAfford also performs cross-modal instruction understanding, task dispatch, and joint learning. Its computational cost should therefore be assessed in relation to these capabilities.

In this design, the MLLM provides shared semantic representations, modality-specific encoders and decoders provide precise spatial localization, and token routing connects the two. The H2 and H3 results support the architecture’s standalone branch capabilities and the contributions of joint supervision, learned routing, and similarity-based coupling, respectively. Thus, introducing the MLLM serves the central research objective of this work, rather than merely improving metrics by increasing model size.

We acknowledge that these capabilities incur higher computational costs than specialized 3D models. To provide a reference for deployment feasibility, we deployed the 2B-parameter model on a single RTX 4090 GPU, using BF16 inference while retaining FP32 for the 3D branch. With the native Transformers implementation, inference takes approximately 1.3 seconds per single-modality sample on average. This indicates potential for deployment as a perception module when suitable computing resources are available, while leaving room for further deployment optimization.

InputBF16 Time/SampleBF16 TFLOPs
2D1.330 s12.342
3D1.304 s0.648
2D + 3D1.969 s12.442

W2: Semantic Pseudo-Pairing and Spatial Correspondence

The main advantage of semantic-level pseudo-pairing is that it enables joint training to exploit larger amounts of existing 2D and 3D annotations, without being constrained by the scale of instance-aligned data. Requiring images and point clouds to depict the same physical instance and possess spatial correspondence would require additional data collection and annotation, while also making it difficult to directly integrate existing independent datasets.

By unifying object–affordance semantics, we organize images and point clouds from different sources into training pairs while preserving their respective pixel-level and point-level annotations. This both expands the data available for joint training and increases instance, appearance, and geometric diversity under the same functional semantics, enabling the model to learn shared functional representations from different observations. Semantic-level pseudo-pairing therefore represents an explicit design choice that balances data scale against spatial correspondence.

W3: Methodological Contributions

The methodological contributions of this work primarily concern task dispatch and semantic–spatial coupling. Token routing predicts task branches directly from contextual hidden states, removing the dependence of task dispatch on predefined marker generation by the language head. With the same backbones, decoders, and training data, this design improves 2D gIoU from 62.28 to 68.79 and 3D mIoU from 17.46 to 34.56 compared with the fully supervised fixed-anchor baseline.

For the decoding interface, we initially explored a LISA-like prompt-style architecture that feeds task representations into downstream networks as prompts. However, this approach did not fully exploit the benefits of joint multitask learning. We therefore introduced similarity-based semantic–spatial coupling, which explicitly establishes matching relationships between task queries and dense spatial features. Compared with the corresponding prompt-style coupling variants, this design improves 2D gIoU from 36.97 to 68.79 and 3D mIoU from 14.31 to 34.56. These results indicate that the coupling design is an important source of the performance gains and constitutes one of the methodological contributions of this work.

Furthermore, the unified taxonomy and semantic-level pseudo-pairing enable these mechanisms to exploit large-scale 2D and 3D annotations without requiring instance alignment. Our contribution therefore lies in applying these dispatch and coupling designs to heterogeneous 2D–3D dense supervision through a shared MLLM, and validating their effects through ablations that hold the other components fixed.

(Due to the length limit, our response continues in Part 2/2.)

AuthorsOpenReview ↗

Response to Reviewer VnAa — Part 2/2

Q1: Comparison with A4-Agent

We thank the reviewer for pointing out this related work. A4-Agent coordinates multiple pretrained models at test time to perform reasoning and grounding without task-specific training. UniAfford, in contrast, learns a shared interface for instruction-conditioned pixel-level and point-level prediction through training. The two approaches emphasize different training paradigms and task coverage. We will include the results reported in the original A4-Agent paper on ReasonAff in the revised manuscript. The comparison with UniAfford’s H2 results is provided below:

MethodReasoninggIoU ↑cIoU ↑P50 ↑P50–95 ↑
A4-AgentYes70.5264.6275.2455.22
UniAfford (H2)No71.1973.6380.9455.89

Table R5: A4-Agent’s results on ReasonAff compared with the results in Table 3(a).

Please let us know if any point needs further clarification or if you have additional questions. We will respond promptly. Thank you again for your time and thoughtful feedback.

Reviewer_EMxaOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed UniAfford, by proposing a unified framework for 2D and 3D affordance grounding. The authors also propose TRT that routes MLLM hidden states to task-specific branches. The authors also proposed UniAfford-Data, a dataset integrating pixel-level and point-level 2D and 3D annotations under similar taxonomy

Critical strengths
  • Routing MLLM states to task-specific branches without using the language head to generate predefined task markers resolves some of the existing limitations of LISA-style interfaces.

  • The ablation studies showing 2D gIoU improvement from 41.41 to 68.79 and 3D mIoU from 30.07 to 34.56 with the same frozen branch backbones is a strong empirical achievement.

  • No large-scale dataset provides spatially aligned pixel-level and point-level annotations for the same physical instances.Hence the semantic pseudo pairing approach is interesting.

  • Projecting routed hidden states through the language head and inspecting vocabulary token frequencies (Appendix C.4) is an informative analysis

Critical weaknesses
  • The paper defines "zero-shot" as "cross-dataset transfer without target-specific adaptation, rather than requiring every target object or affordance category to be absent from the training taxonomy." This is probably inaccurate definition. AGD20K contains 36 affordance categories across 50 objects. UniAfford-Data contains 162 affordance categories. The overlap is nowhere quantified. GEAL is derived from LASO-C, which uses the same 3D-AffordanceNet source/categories as AGPIL and PIADv2 and both are training sources. The model has been trained on many of the exact <object, affordance> pairs it is evaluated on. So, "Cross-dataset transfer" or "instance-level generalization" would be more accurate than "OOD zero-shot." The comparison, therefore, against baseline zero-shot methods is not apple to apple comparison.

  • Every table reports single-run numbers. LoRA fine-tuning is initialization-sensitive. A 1–3% variance on these tasks is typical. The router itself is stochastic. Some claimed gains are very small. Without multi-seed validation, these can be within noise.

  • Missing some of the important baselines like Affogato, SeqAfford, LMAffordance3D, etc.

  • Can the authors provide some evidence to show that the ReasonAff images used as training data in UniAfford-Data are disjoint from the ReasonAff test set in H2 evaluation? Similarly for PIADv2? Please provide sample count numericals.

Questions for authors

Please address the concerns raised in the weakness section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer EMxa — Part 1/2

Thank you for your time and thoughtful feedback. We appreciate your constructive comments and the opportunity to clarify our work. We address your concerns point by point below and welcome any further questions or requests for clarification.

W1: Cross-Dataset Transfer and Semantic Coverage

Thank you for pointing out the ambiguity in terminology. We follow the zero-shot cross-dataset evaluation paradigm used by Affordance-R1 on AGD20K, assessing transfer without target-specific adaptation rather than assuming that all target categories are absent from the source training data. To avoid confusion, we will replace “OOD zero-shot” with “cross-dataset transfer without target-specific adaptation” and distinguish the novelty of semantic categories/combinations from instance independence.

The additional semantic coverage statistics are provided below. AGD20K includes both categories and combinations covered by the source training data and those not covered. GEAL*, which uses original assets from 3D-AffordanceNet, has substantial semantic overlap with AGPIL/PIADv2, the source pools used to construct UniAfford-Data.

Test setAudited sourceObject category coverageAffordance category coverageObject–affordance combination coverage
AGD20KUniAfford-Data18/50 (36.00%)19/36 (52.78%)22/123 (17.89%)
GEAL*AGPIL/PIADv2 source pools23/23 (100%)18/18 (100%)57/58 (98.28%)

Table R3: Semantic coverage of the target test sets. Denominators indicate the total number of categories or combinations in each target set. Coverage of source pools and coverage of the actual training set are distinguished by the scope specified in the table.

Regarding comparison fairness, the 3D methods do not use mutually independent upstream assets: OpenAD directly uses 3D-AffordanceNet, while PIAD used by IAGNet, PIADv2 used by GREAT, and AGPIL/PIADv2 used by UniAfford also reuse this source to varying degrees. Nevertheless, these datasets are not identical. For example, PIADv2 additionally includes sources such as 3DIR and Objaverse, and the datasets provide different image, language, and affordance supervision. Therefore, for cross-dataset transfer on GEAL*, UniAfford-Data does not provide an evident additional data advantage over these baselines, particularly in terms of object–affordance combinations.

W2: Result Stability and Training Cost

LoRA is primarily used for parameter-efficient fine-tuning of the MLLM. We conducted a full fine-tuning comparison and found that the performance difference between the two approaches was within 1%; we therefore adopted LoRA to reduce training costs. We recognize the value of evaluation across multiple random seeds for assessing result stability. However, given the substantial computational cost of MLLM training and our current resource constraints, conducting multiple independent training runs for all experiments poses considerable practical challenges.

Moreover, our conclusions regarding the key design choices primarily rely on controlled ablations under the same experimental settings, rather than solely on small differences in benchmark rankings. For example, compared with the fixed-anchor baseline, learned routing improves 2D gIoU and 3D mIoU by 6.51 and 17.10 percentage points, respectively. Compared with the corresponding single-modality training variants, joint training improves these metrics by 27.38 and 4.49 percentage points, respectively. These results show consistent improvements across both output spaces and provide empirical support for the effectiveness of the corresponding design choices.

W3: Additional Baseline Comparisons

LMAffordance3D is already included in Table 3(b), and we will make this comparison clearer. For Affogato (ECCV 2026), we have not yet obtained an official implementation sufficient to reliably reproduce its reported results, so we cannot currently provide verified reproduction results.

For SeqAfford, its public repository states that the provided code is not the final complete version and points to the subsequent OpenHOI implementation. To supplement the comparison with this line of work, we evaluated the publicly released OpenHOI HOIAffordanceMLLM weights on the same GEAL* test tasks for single-affordance grounding. The results are shown below.

MethodAUC ↑mIoU ↑SIM ↑MAE ↓
OpenHOI (subsequent implementation of SeqAfford)70.889.280.4930.137
UniAfford (H1)83.5514.670.5650.102

Table R4: Evaluation on GEAL* using the official OpenHOI weights trained on 3D-AffordanceNet.

(Due to the length limit, our response continues in Part 2/2.)

AuthorsOpenReview ↗

Response to Reviewer EMxa — Part 2/2

W4: Separate Training Protocols for H1 and H2

It is necessary to distinguish the joint-training transfer protocol in H1 from the within-benchmark training and evaluation protocol in H2. H2 uses models trained separately on the corresponding benchmark training sets: the ReasonAff experiment uses the ReasonAff training split, and the PIADv2 experiment uses the corresponding PIADv2 training split, with evaluation conducted on the official test splits. The results in Table 3 are not obtained by directly applying the checkpoint jointly trained on UniAfford-Data to these benchmark tests. Therefore, they do not involve additional training data carried over through that jointly trained checkpoint.

Please let us know if any point needs further clarification or if you have additional questions. We will respond promptly. Thank you again for your time and thoughtful feedback.

Reviewer_SNMYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents UniAfford, a multitask framework that unifies affordance grounding in 2D images and 3D point clouds. The authors also introduce UniAfford-Data, which integrates RAGNet, ReasonAff, PIADv2, and AGPIL under a shared object–affordance taxonomy. It forms semantic-level pseudo-pairs of 2D and 3D samples without requiring instance-level correspondence. However, the paper does not yet provide sufficient causal evidence to substantiate the benefits of learned routing.

Critical strengths
  1. Unifying the traditionally separate pixel-level 2D and point-level 3D affordance supervision within a single framework represents a meaningful research direction, particularly for data-scarce embodied perception scenarios.
  2. The paper provides a comprehensive empirical evaluation covering 2D and 3D results, making the experimental study more complete.
Critical weaknesses
  1. The full model is exposed to more samples, categories, and supervision signals than the 2D-only and 3D-only variants. Its gains may thus result from additional data, optimization steps, or broader language-label coverage rather than 2D–3D knowledge transfer. A controlled comparison with matched sample counts and optimization budgets is needed.
  2. The paper does not clarify whether shape-ID deduplication or instance-overlap checks were performed across AGPIL, PIADv2, and the LASO-derived GEAL* evaluation set. Thus, it remains unclear whether the model was trained on clean versions of the target point clouds and subsequently evaluated on their Scale, Jitter, or Rotate variants. If such overlap exists, the reported AUC of 83.55 would not constitute reliable evidence of OOD generalization. The authors should clarify this issue and, if necessary, re-evaluate the model using strictly instance-disjoint splits based on the original shape IDs.
  3. The system largely combines several strong pretrained components, including Qwen3-VL, SigLIP, SAM, and SONATA, with a lightweight router. Similarly, UniAfford-Data mainly reindexes and semantically pairs existing datasets. The paper lacks a strong marker-token baseline trained with the same MLLM, decoders, and data. It is thus unclear whether the gains arise from the proposed routing mechanism or from stronger backbones, pretrained knowledge, and increased data scale.
  4. Routing targets are defined by the positions of <img-aff> and <pc-aff>. Although these anchors are excluded from the language modeling loss, they still affect subsequent hidden states under teacher forcing. At inference time, however, the model is claimed not to generate these markers. The paper does not clearly specify the inference-time generation sequence, stopping criterion, or how the resulting train–test mismatch is avoided.
Questions for authors

pls see weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer SNMY — Part 1/2

Thank you for your time and thoughtful feedback. We appreciate your constructive comments and the opportunity to clarify our work. We address your concerns point by point below and welcome any further questions or requests for clarification.

W1: Sources of Joint-Training Gains

Thank you for raising the question of what accounts for the performance gains. We examine this issue through the modality-isolated experiments in H2 and controlled ablations at different levels in H3. It is important to distinguish between comparisons of single-modality and joint training, and comparisons of methodological designs under the same joint supervision. The former assess the overall benefits of joint training, whereas the latter examine whether performance can be explained solely by additional data and supervision.

First, H2 trains and evaluates the 2D and 3D branches separately on their respective benchmarks, without training supervision or auxiliary visual observations from the other modality, and still achieves strong benchmark performance. This shows that UniAfford’s standalone branch capabilities do not depend on additional cross-modal data. H3 further shows that, with the branch backbones unchanged, joint training improves 2D gIoU from 41.41 to 68.79 and 3D mIoU from 30.07 to 34.56, demonstrating additional benefits of the joint-training setting beyond the single-modality capabilities.

The component ablations under the same joint-training setting in H3 address the data-volume explanation more directly. These variants use the same training samples, category coverage, and supervision signals, with matched training budgets. Their performance differences therefore cannot be attributed to the full model seeing more data or undergoing more optimization:

  • Routing mechanism: The fixed-anchor baseline generates task-marker tokens through the language head and uses them to select the corresponding hidden states for the respective decoding branches. With the backbones, decoders, and training settings held constant, replacing this mechanism with learned routing improves 2D gIoU from 62.28 to 68.79 and 3D mIoU from 17.46 to 34.56. This shows that directly predicting task branches from contextual states provides additional benefits under the same data and supervision, while reducing interference from language markers during joint training.
  • Semantic–spatial coupling: The prompt-style variant inspired by LISA supplies task representations as prompts to the downstream network, whereas our similarity-based coupling explicitly establishes correspondence between task queries and dense spatial features. Under the same joint supervision, the latter improves 2D gIoU from 36.97 to 68.79 and 3D mIoU from 14.31 to 34.56. This indicates that providing the same training data and task representations alone is insufficient to achieve the full model’s performance; how shared semantic representations are used for dense spatial prediction is also critical.

Thus, the existing experiments demonstrate both the overall benefits of joint training and the independent methodological contributions of routing and semantic–spatial coupling through comparisons with matched data and training budgets. We do not attribute the entire difference between single-modality and joint training to cross-modal knowledge transfer. Our conclusion is that the full model’s performance cannot be explained solely by more samples, broader label coverage, or more optimization steps; the specific mechanisms through which it uses joint supervision are also important sources of improvement.

W2: Evaluation after Excluding Overlapping Assets

We have added an audit of overlap at the original-asset level between GEAL* and the training sources. Conservatively including PIADv2 validation assets in the audit, we found that 400 of the 1,035 assets in GEAL* overlap with the examined AGPIL/PIADv2 source pools. After excluding all affordance tasks and corruption variants associated with these assets, we retain 635 assets and 4,248 evaluation tasks as the GEAL** test set. UniAfford achieves AUC scores of 83.55 and 82.11 on the full GEAL* set and the asset-excluded GEAL** subset, respectively. These results support the model’s transfer capability on test instances after removing assets with confirmed source overlap. The detailed results are provided below:

MethodEvaluation setAUC ↑mIoU ↑SIM ↑MAE ↓
UniAfford (H1)GEAL*83.5514.670.5650.102
UniAffordGEAL**83.16 (-0.47%)14.06 (-4.16%)0.568 (+0.53%)0.106 (+3.92%)

Table R1: Re-evaluation of UniAfford on GEAL** after removing overlapping assets.

(Due to the length limit, our response continues in Part 2/2.)

AuthorsOpenReview ↗

Response to Reviewer SNMY — Part 2/2

W3: Routing under Matched Architecture and Data

Our methodological contribution is a shared task interface that allows pixel-level and point-level dense supervision to jointly optimize MLLM representations. To distinguish the benefits of routing from those of pretrained backbones, Table 4 compares token routing with the task-marker generation paradigm exemplified by LISA, while keeping the backbones, dense prediction decoders, and training data identical.

Specifically, during training, the fixed-anchor baseline applies language cross-entropy supervision to all valid response tokens, including task-marker tokens, without additional routing supervision. At inference, the language head autoregressively generates task-marker tokens, and the corresponding hidden states are then passed to the decoding branches. In contrast, UniAfford uses routing-supervision anchors to train a separate routing head that predicts branches directly from contextual states. Task dispatch does not depend on the language head generating specified tokens.

Under this comparison, UniAfford improves 2D gIoU from 62.28 to 68.79 and 3D mIoU from 17.46 to 34.56. This ablation therefore directly supports the advantage of learned routing over the LISA-style task-marker generation interface, and the performance difference cannot be attributed solely to stronger pretrained backbones. We will add the language supervision and autoregressive inference details of the fixed-anchor baseline to the revised manuscript to resolve the ambiguity in the original description.

W4: Training and Inference Procedures

Thank you for pointing out that the training and inference procedures were insufficiently described. During training, <img-aff> and <pc-aff> serve as routing-supervision anchors that determine branch labels. They are excluded from the language cross-entropy loss but remain in the teacher-forced input sequence and therefore affect subsequent contextual states.

At inference, the current hidden state is used for both route prediction and language-token prediction. When the router selects a non-text branch, that hidden state is passed to the corresponding dense prediction decoder. Subsequent autoregressive generation still uses the embedding of the token predicted by the language head as the next input; it neither feeds the hidden state back directly nor forces a branch anchor into the sequence. Task dispatch is therefore determined by the routing head’s prediction and does not depend on whether the language head generates a specified task marker.

For presentation, we display the corresponding branch marker at each position where a non-text branch is activated, identifying its associated dense prediction output. This substitution is used only for displaying results and does not change the feedback of the actual generated sequence or any subsequent computation. Generation terminates when EOS is produced or the configured maximum generation length is reached.

Please let us know if any point needs further clarification or if you have additional questions. We will respond promptly. Thank you again for your time and thoughtful feedback.

Reviewer_WbeZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Affordance perception identifies actionable regions in images and point clouds, but the two modalities are typically studied separately. This fragmentation limits the reuse of functional semantics across heterogeneous annotations. UniAfford combines a shared MLLM, learned token routing, modality-specific dense decoders, and a taxonomy-aligned dataset of semantically paired 2D–3D observations. Experiments report strong ReasonAff and PIAD results, cross-dataset transfer to AGD20K and GEAL*, and improvements from routing and joint supervision.

Critical strengths
  1. The unified taxonomy makes heterogeneous pixel-level and point-level annotations jointly usable without requiring expensive instance-aligned data collection.
  2. Predicting branch assignments directly from contextual states offers a useful alternative to dispatch through language-generated task markers.
  3. Reported results are strong in both output spaces, including 71.19 gIoU on ReasonAff and 14.25 mIoU on PIAD.
Critical weaknesses
  1. GEAL* test independence remains unverified because its LASO source and the PIADv2 and AGPIL training sources reuse 3D-AffordanceNet geometry, yet no underlying shape-ID overlap audit is reported.
  2. Table 4 lacks an unpaired multitask baseline with identical supervision and inference inputs, preventing attribution of joint-learning gains specifically to semantic pseudo-pairing.
  3. The fixed-anchor ablation does not specify whether anchor generation receives language-model supervision, leaving the fairness of the central routing comparison uncertain. For example, fixed-anchor routing vs. routing by special token or language token.
Questions for authors
  1. Are GEAL* test shapes disjoint from all training sources at the underlying asset level, and what results remain after enforcing this separation?
  2. What tokens are fed back after non-text routing decisions, and how are zero-query or excess-query predictions scored?
  3. Does semantic pairing outperform unpaired multitask training under matched data, computation, and inference inputs?
  4. Do the routing and 3D grounding advantages persist with fully supervised marker-generation baselines and matched task prompts, respectively?
2 comments
AuthorsOpenReview ↗

Response to Reviewer WbeZ — Part 1/2

Thank you for your time and thoughtful feedback. We appreciate your constructive comments and the opportunity to clarify our work. We address your concerns point by point below and welcome any further questions or requests for clarification.

W1 & Q1: Asset-Level Overlap Audit and Evaluation

Given the scarcity of 3D affordance data, existing datasets do share source assets. When selecting the zero-shot evaluation set, we deliberately sought to avoid substantial overlap with the training data in UniAfford-Data. We traced the original assets underlying the GEAL* samples and checked their overlap with the AGPIL and PIADv2 source splits, conservatively including PIADv2 validation assets in the exclusion set. Among the 1,035 original assets underlying the 7,248 test tasks in GEAL*, 400 overlap with the AGPIL/PIADv2 sources, corresponding to 400/1,035 assets (38.65%) and 3,000/7,248 test tasks (41.39%). We denote the dataset obtained by removing these overlapping assets and their associated tasks as GEAL**. However, because GEAL applies perturbations of varying degrees to its original assets, including deformation, missing points, and point dispersion, GEAL* contains no point clouds and annotations that are exactly identical to those in the UniAfford-Data training set.

After removing the overlapping original assets and their associated test tasks, our model’s results change as follows:

MethodEvaluation setAUC↑mIoU↑SIM↑MAE↓
UniAfford (H1)GEAL*83.5514.670.5650.102
UniAffordGEAL**83.16 (-0.47%)14.06 (-4.16%)0.568 (+0.53%)0.106 (+3.92%)

Table R1: Re-evaluation of UniAfford on GEAL** after removing overlapping assets.

W2 & Q3: Semantic Pseudo-Pairing versus Unpaired Multitask Training

Thank you for this suggestion. UniAfford-Data integrates pixel-level and point-level supervision under a shared object–affordance taxonomy, enabling a shared model to learn jointly from both modalities. Explicit semantic-level pseudo-pairing further provides training examples with joint 2D–3D inputs and outputs, helping the model fully handle tasks involving different modality combinations while preserving the original observations, annotations, and semantic associations for each modality.

To directly examine the additional effect of presenting both tasks together, we constructed a fixed supplementary ablation subset from the existing sources of UniAfford-Data and trained two variants: semantic-level pseudo-pairing (our method) and unpaired multitask training (the control). Both use the same underlying data, segmentation supervision, model configuration, and main optimization hyperparameters. We evaluate them under two input settings: unimodal validation, where each example contains either 2D or 3D input, and mixed-input validation, where examples may contain either a single modality or joint 2D–3D inputs. Within each setting, both models use the same underlying tasks, visual data, and ground-truth annotations.

Both validation settings contain the same 614 2D tasks and 902 3D tasks. All results use the model’s native routing outputs. If the model does not route to the requested modality, that branch retains an empty prediction and is included in the evaluation.

Evaluation inputTraining schemegIoU↑cIoU↑AUC↑mIoU↑SIM↑MAE↓
Unimodal validationSemantic-level pseudo-pairing79.5282.5788.2522.660.62850.0879
Unimodal validationUnpaired multitask training80.4078.9388.1823.410.62260.0901
Mixed-input validationSemantic-level pseudo-pairing79.4782.4688.1322.610.62910.0879
Mixed-input validationUnpaired multitask training69.6071.3787.7523.120.61560.0915

Table R2: Comparison of paired and unpaired multitask training on a fixed supplementary subset.

Under unimodal validation, the two variants exhibit trade-offs across metrics, and both produce all requested modality outputs. Under mixed-input validation, the model trained with semantic-level pseudo-pairing completes all tasks, whereas the model trained with unpaired multitask supervision omits one output branch for every dual-modality request, yielding a dual-modality output incompleteness rate of 100% (84/84). Specifically, in the reported evaluation, it produces only the 3D output and omits the 2D output, resulting in 2D gIoU and cIoU scores that are 9.87 and 11.09 percentage points lower, respectively, than those of semantic-level pseudo-pairing. This shows that, under the evaluated setting, unpaired multitask training enables the model to perform each unimodal task individually but does not enable it to combine these capabilities into complete joint 2D–3D outputs.

(Due to the length limit, our response continues in Part 2/2.)

AuthorsOpenReview ↗

Response to Reviewer WbeZ — Part 2/2

W3 & Q4: Marker-Generation Baseline and Matched Task Prompts

Thank you for pointing out that this implementation detail was insufficiently described. The fixed-anchor baseline in Table 4 dispatches tasks through task markers generated by the language head. During training, we apply standard autoregressive language cross-entropy supervision to all target tokens in valid responses, including <img-aff> and <pc-aff>, while retaining the corresponding 2D and 3D dense prediction losses. This baseline does not use separate routing classification supervision or routing structure regularization.

At inference, the model generates a response autoregressively. When the language head generates a task marker, its associated hidden state is passed to the corresponding decoder branch. Thus, this baseline already includes explicit marker-generation supervision. Its distinction from our method is that the baseline determines task dispatch through the identities of tokens generated by the language head, whereas our method directly predicts branches from contextual states using a separate routing head. Section 5.4 and Appendix C.5 already state that the backbones, decoders, and training data are kept identical, but the differences in losses and inference procedures were not made explicit. We will add these details, particularly clarifying that our method’s exclusion of <img-aff> and <pc-aff> from the language modeling loss does not apply to the fixed-anchor baseline.

To avoid ambiguity, we will rename “fixed-anchor” to “marker-generation” in the revised manuscript, distinguishing the supervision anchors used for routing from the task-marker tokens generated by the language head.

Regarding task prompts in the 3D comparisons, all methods receive semantically equivalent information about the object category and target affordance, adapted to their supported input interfaces. For methods that support natural-language input, we provide instructions constructed from the same object–affordance information. For methods that use an open-vocabulary query interface, such as OpenAD, we provide the category terms corresponding to the same task. Thus, although prompt formats may differ, the semantic information specifying the grounding task is consistent across methods. The 3D comparison results reported in the paper already use this setting, and we will clarify it in the main paper.

Q2: Autoregressive Feedback and Query Handling

Non-text routing does not replace the language model’s autoregressive feedback mechanism. At each step, the current hidden state is used to predict both the routing branch and the language token. When the router selects a non-text branch, that state is additionally passed to the corresponding dense prediction decoder. Subsequent generation still uses the embedding of the token predicted by the language head, rather than directly feeding back the hidden state or forcibly inserting a task marker. Task dispatch is therefore determined by the routing head, while the language sequence follows the standard token-prediction and embedding-feedback process.

During training: If no query is predicted for a valid task branch, the hidden state associated with that branch’s ground-truth marker is used as a fallback so that the dense prediction branch still receives supervision. Meanwhile, the routing classification loss corrects the missed branch prediction. For excess queries, routing classification supervision and the existence and sparsity objectives jointly suppress incorrect and redundant routing, encouraging the query count to match the task requirements.

During evaluation: The fallback based on ground-truth markers is disabled. Tasks for which no query is generated remain in the full test set and are scored using near-zero prediction maps under the same evaluation metrics. For single-target tasks, once a branch has triggered decoding, repeated activation of that branch is masked at subsequent generation steps. Text generation and other available branches continue normally until EOS or exhaustion of the generation budget. Thus, this protocol does not involve post-generation query truncation, selection of the best prediction, or exclusion of failed samples.

Please let us know if any point needs further clarification or if you have additional questions. We will respond promptly. Thank you again for your time and thoughtful feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revision

We thank all reviewers for their careful reading and constructive feedback. We have responded to the concerns raised and revised the manuscript to clarify the methodology, make the evaluation protocols more transparent, and strengthen the experimental support for our contributions. The revised manuscript has been uploaded, with additions and revisions highlighted in blue for ease of review.

The main changes are as follows:

  • Clarified evaluation protocols and generalization claims. We removed the broad OOD characterization and clarified the cross-dataset zero-shot setting as evaluation without target-specific training or fine-tuning. We also made the H2 branch-wise protocol explicit: each branch is independently initialized from pretrained backbones, trained on the corresponding benchmark’s training split, and evaluated on its designated test split, without inheriting UniAfford-Data-trained weights.

  • Added semantic-coverage statistics and an asset-overlap audit. The appendix now reports object-category, affordance-category, and object–affordance combination coverage. We also document the original-asset audit for GEAL* and provide evaluation results after excluding overlapping assets, giving more concrete support for instance-level generalization.

  • Clarified the marker-generation baseline and routing comparison. We renamed the baseline consistently and specified its language supervision, dense prediction losses, and inference procedure. The revised discussion distinguishes language-head-based marker generation from learned routing under matched backbones, decoders, and training data.

  • Expanded the training and inference description. We clarified the role of routing-anchor labels, autoregressive token feedback, training-time query fallback, evaluation-time handling of missing or repeated queries, and generation stopping conditions.

  • Added a direct pseudo-pairing comparison. We included supplementary experiments comparing semantic pseudo-pairing with independent multitask training using the same underlying data and supervision. The analysis distinguishes single-modal grounding performance from the ability to produce complete dual-modal outputs.

  • Improved comparison context and presentation. We clarified the matched task-specifying semantics used in the 3D comparisons, expanded the discussion of reasoning-based methods, and added the A4-Agent comparison in the appendix with its training-setting differences explicitly stated. References from the main text guide readers to the new supporting analyses.

Throughout the revision, we have aimed to make the paper easier to follow and the relationship between each experimental result and its supporting claim more explicit. We hope these revisions address the reviewers’ concerns and would welcome any further feedback on points that may still require clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.