- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Token Router for Tasks, a general multitask training paradigm for MLLM-based systems. Unlike LISA-style methods that couple task dispatch with the generation of predefined task tokens, the paradigm predicts branch assignments directly from contextual MLLM hidden states without requiring the language head to generate task markers. Routing anchors provide only routing supervision and are excluded from the language modeling loss; the routed states are supervised by branch-specific objectives — SAM-style pixel-level and SONATA-style point-level dense prediction losses — which in turn shape the shared MLLM representations. The authors instantiate this paradigm as UniAfford, a unified framework for generalizable 2D–3D affordance perception, supported by UniAfford-Data, a unified dataset that organizes 25k image and 69k point-cloud samples under a shared object–affordance taxonomy with semantic-level 2D–3D pseudo-pairing.
Critical strengths
- The unification of 2D and 3D affordance grounding is a novel and well-motivated contribution at the level of task organization, as prior works have treated these as separate problems.
- One model plus one unified dataset serves both modalities. The architecture demonstrates strong OOD zero-shot transfer (27.52 gIoU on AGD20K and 83.55 AUC on GEAL*) as well as state-of-the-art branch-wise performance (71.19 gIoU on ReasonAff versus 67.41 for Affordance-R1; 14.25 mIoU on PIAD versus 9.73 for DAG).
- The ablations are thorough and controlled: learned routing improves 3D mIoU from 17.46 to 34.56, joint 2D–3D supervision improves 2D gIoU from 41.41 to 68.79, and the language-head diagnostics confirm that routed states carry object–affordance semantics.
- 2D inference is notably efficient compared with reasoning-based baselines.
Critical weaknesses
- The framework stacks three large backbones (Qwen3-VL, SAM, and SONATA). The MLLM dominates the 3D inference cost (1,930.87 GFLOPs versus 21.15 for GREAT and 13.68 for IAGNet), which raises doubts about the feasibility of real-world robotic deployment.
- The semantic-level pseudo-pairing connects image and point-cloud instances only through shared object–affordance labels, without instance-level spatial correspondence, which the authors themselves acknowledge limits geometric consistency supervision.
- At the method level, the components are mostly combinations of existing techniques (MLLM + LoRA + SAM decoder + SONATA decoder + a lightweight router), so the novelty lies mainly in task organization and the training interface rather than in methodological breakthroughs.
Questions for authors
- The paper lacks comparisons or discussions with newer related methods, such as A4-Agent (ECCV 2026). Could the authors add these comparisons or discuss their relation to this work?
2 comments
Response to Reviewer VnAa — Part 2/2
Q1: Comparison with A4-Agent
We thank the reviewer for pointing out this related work. A4-Agent coordinates multiple pretrained models at test time to perform reasoning and grounding without task-specific training. UniAfford, in contrast, learns a shared interface for instruction-conditioned pixel-level and point-level prediction through training. The two approaches emphasize different training paradigms and task coverage. We will include the results reported in the original A4-Agent paper on ReasonAff in the revised manuscript. The comparison with UniAfford’s H2 results is provided below:
| Method | Reasoning | gIoU ↑ | cIoU ↑ | P50 ↑ | P50–95 ↑ |
|---|---|---|---|---|---|
| A4-Agent | Yes | 70.52 | 64.62 | 75.24 | 55.22 |
| UniAfford (H2) | No | 71.19 | 73.63 | 80.94 | 55.89 |
Table R5: A4-Agent’s results on ReasonAff compared with the results in Table 3(a).
Please let us know if any point needs further clarification or if you have additional questions. We will respond promptly. Thank you again for your time and thoughtful feedback.
Response to Reviewer VnAa — Part 1/2
Thank you for your time and thoughtful feedback. We appreciate your constructive comments and the opportunity to clarify our work. We address your concerns point by point below and welcome any further questions or requests for clarification.
W1: Computational Cost and Deployment Feasibility
We introduce the MLLM to support the unified task capabilities studied in this work: connecting the functional semantics in language instructions to pixel-level 2D and point-level 3D grounding, while allowing dense supervision from both modalities to jointly shape shared representations. Specialized 3D models focus on efficient prediction within specific input and output spaces, whereas UniAfford also performs cross-modal instruction understanding, task dispatch, and joint learning. Its computational cost should therefore be assessed in relation to these capabilities.
In this design, the MLLM provides shared semantic representations, modality-specific encoders and decoders provide precise spatial localization, and token routing connects the two. The H2 and H3 results support the architecture’s standalone branch capabilities and the contributions of joint supervision, learned routing, and similarity-based coupling, respectively. Thus, introducing the MLLM serves the central research objective of this work, rather than merely improving metrics by increasing model size.
We acknowledge that these capabilities incur higher computational costs than specialized 3D models. To provide a reference for deployment feasibility, we deployed the 2B-parameter model on a single RTX 4090 GPU, using BF16 inference while retaining FP32 for the 3D branch. With the native Transformers implementation, inference takes approximately 1.3 seconds per single-modality sample on average. This indicates potential for deployment as a perception module when suitable computing resources are available, while leaving room for further deployment optimization.
W2: Semantic Pseudo-Pairing and Spatial Correspondence
The main advantage of semantic-level pseudo-pairing is that it enables joint training to exploit larger amounts of existing 2D and 3D annotations, without being constrained by the scale of instance-aligned data. Requiring images and point clouds to depict the same physical instance and possess spatial correspondence would require additional data collection and annotation, while also making it difficult to directly integrate existing independent datasets.
By unifying object–affordance semantics, we organize images and point clouds from different sources into training pairs while preserving their respective pixel-level and point-level annotations. This both expands the data available for joint training and increases instance, appearance, and geometric diversity under the same functional semantics, enabling the model to learn shared functional representations from different observations. Semantic-level pseudo-pairing therefore represents an explicit design choice that balances data scale against spatial correspondence.
W3: Methodological Contributions
The methodological contributions of this work primarily concern task dispatch and semantic–spatial coupling. Token routing predicts task branches directly from contextual hidden states, removing the dependence of task dispatch on predefined marker generation by the language head. With the same backbones, decoders, and training data, this design improves 2D gIoU from 62.28 to 68.79 and 3D mIoU from 17.46 to 34.56 compared with the fully supervised fixed-anchor baseline.
For the decoding interface, we initially explored a LISA-like prompt-style architecture that feeds task representations into downstream networks as prompts. However, this approach did not fully exploit the benefits of joint multitask learning. We therefore introduced similarity-based semantic–spatial coupling, which explicitly establishes matching relationships between task queries and dense spatial features. Compared with the corresponding prompt-style coupling variants, this design improves 2D gIoU from 36.97 to 68.79 and 3D mIoU from 14.31 to 34.56. These results indicate that the coupling design is an important source of the performance gains and constitutes one of the methodological contributions of this work.
Furthermore, the unified taxonomy and semantic-level pseudo-pairing enable these mechanisms to exploit large-scale 2D and 3D annotations without requiring instance alignment. Our contribution therefore lies in applying these dispatch and coupling designs to heterogeneous 2D–3D dense supervision through a shared MLLM, and validating their effects through ablations that hold the other components fixed.
(Due to the length limit, our response continues in Part 2/2.)