acceptodds
Under review as a conference paper at ICLR 2027

AffiMoE: Fine-Grained Text-Visual Alignment via Token-Wise Expert Routing for Language-Grounded Affordance Reasoning

Abstract

Affordance grounding aims to localize object regions that enable a specified action, requiring models to connect high-level task intent with fine-grained functional structures. Recent LLM- and VLM-based approaches improve open-world affordance reasoning through language supervision, while RAGNet further extends this paradigm with human-like instructions and reasoning-aware segmentation. However, these methods still compress rich multimodal semantics into compact affordance representations, leaving the correspondence between language-level actions and fine-grained functional regions largely implicit. To address this limitation, we propose AffiMoE, a language-grounded affordance reasoning framework that explicitly strengthens fine-grained text-visual alignment through token-wise expert routing. Specifically, AffiMoE uses cross-attention to align action semantics with dense visual features, producing localized representations that better capture action-relevant object parts. Since precise affordance grounding requires both global object context and local functional evidence, we further treat globally aligned multimodal features and locally aligned visual features as complementary experts, and introduce a token-wise mixture-of-experts router to adaptively integrate them at each spatial location. In this way, AffiMoE preserves coherent high-level reasoning while improving the localization of action-relevant functional regions. To rigorously evaluate this capability, we introduce a new benchmark, and extensive experiments show that AffiMoE achieves state-of-the-art performance across multiple benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Affordance segmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fTUkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AffiMoE for language-grounded affordance segmentation. It uses GPT to identify the target object, infers the intended action, and generate a bounding box. It then introduces cross-attention between MLLM representations and SAM 3 features, followed by token-wise routing between global and locally aligned representations. It also constructs AffordBench for training and manually revise UMD annotations for evaluation.

Critical strengths
  1. The paper addresses a meaningful problem: identifying action-specific functional regions rather than merely localizing objects. The proposed MoE-style routing provides a well-motivated way to selectively combine global semantics with locally aligned visual evidence, supporting token-level fine-grained text–visual alignment.

  2. The fine-grained alignment module shows clear benefits. In Table 2, adding FGA without LoRA improves UMD-relabeled performance from 73.82/72.40 to 82.71/80.79 cIoU/gIoU.

  3. The human-verified training annotations are a useful contribution. The additional HANDAL results also provide positive evidence: AffiMoE improves over AffordanceVLM using the same training data.

Critical weaknesses
  1. The routing mechanism is underspecified. Section 3.3 does not precisely define the response aggregation function, explain how scores from the two branches are calibrated, or clarify which tokens are routed. Larger feature responses are not necessarily more reliable evidence.

  2. MoE provides limited and inconsistent gains. In Table 2, adding MoE with LoRA improves UMD-relabeled cIoU by 1.48 points but decreases gIoU by 0.85 points. In Table 4, its gIoU advantage over shared-weight fusion is only 0.10 points, without repeated-run statistics.

  3. Architectural gains are not sufficiently isolated. GPT already resolves the object, action, and bounding box before downstream segmentation. Whether baselines share this preprocessing is unclear, and the substantially different SAM 3 baseline results in Tables 2 and 3 require an explicit configuration summary.

  4. Fine-grained alignment needs more targeted evaluation. UMD and UMD-relabeled use the same images, while aggregate IoU alone does not establish action-specific alignment. Same-object/different-action tests, boundary metrics, and per-affordance results would strengthen the evidence.

Questions for authors

check the weakness, and I will raise my score if you can solve my questions.

Reviewer_SfTbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents AffiMoE for segmenting functional object regions from natural-language instructions. The system first uses GPT to identify the target object and action, predict a bounding box, and crop the image before performing fine-grained affordance segmentation. Built on AffordanceVLM, AffiMoE introduces SAM3 features and cross-attention within the language model to retrieve action-relevant local visual information. Token-wise routing then selects between the original multimodal features and locally aligned features, with the final [AFF] token conditioning the segmentation decoder. The authors also introduce AffordBench, containing 3,144 images and 5,510 object–action annotations, and provide revised UMD annotations. The method achieves strong results on both UMD annotation versions and HANDAL.

Critical strengths

AffordBench includes human-verified and manually refined annotations. Reporting results on both original and revised UMD annotations is helpful, while the same-training-data HANDAL experiments provide additional evaluation evidence. Giving language representations direct access to high-resolution segmentation features is a practically useful direction.

Critical weaknesses
  1. l baselines receive the same GPT-generated boxes, crops, and rewritten instructions. This makes it difficult to separate architectural gains from preprocessing benefits. Comparisons using full images, shared predicted crops, and ground-truth crops would help.
  2. The core MoE routing gains are small and inconsistent. Tables 2 and 4 suggest that routing changes the predicted regions, but do not establish consistent improvements in fine-grained localization. Multiple seeds, confidence intervals, and direct comparisons with learnable token-wise soft gating are missing.
  3. The methodological novelty appears limited. The paper should more clearly distinguish its design from established cross-attention and gated-fusion approaches.
Questions for authors
  1. Do all baselines use identical GPT localization, cropping, and instruction rewriting? Could you compare methods under shared preprocessing and report the gains from GPT preprocessing separately?
  2. Why does routing improve some cIoU results while reducing several gIoU results? Could you provide multi-seed results and comparisons with learnable token-wise soft gating and standard residual cross-attention?
Reviewer_2bSWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose AffiMoE, which builds on AffordanceVLM/RAGNet, uses dense visual features from the SAM3 family and layer-wise cross-attention with an MLLM for fine-grained alignment, and applies token-level hard routing to select between global semantic features and locally aligned features. The authors also construct AffordBench, containing 5,510 object–action records across 3,144 images and covering seven affordance categories. The authors also evaluate the method on UMD and UMD-relabeled, a relabeled version of the same images, and provide additional results on HANDAL.

Critical strengths
  • The existing experiments directly support the contribution of fine-grained alignment to task performance. For example, the results in Table 2 indicate that the model needs to incorporate information about “which action to perform” into local visual features.
  • The authors construct AffordBench by selecting 5,510 records from 47,040 candidates through two rounds of manual review, with 751 masks manually reannotated. The training sources are diverse, and functional regions occupy a median of only 0.81% of the full image area, posing a practical challenge for precise localization.
Critical weaknesses
  • The gains from routing are limited and inconsistent: Adding MoE to the full LoRA+FGA configuration improves cIoU on UMD but reduces gIoU. Compared with shared weights, the gains in Table 4 are very limited, and no uncertainty estimates are provided. The authors have already acknowledged that gIoU does not improve consistently. The current evidence more strongly supports local alignment than the general superiority of more complex token-level hard routing. Since the latter is an important contribution distinguishing the paper from standard cross-attention fusion, this directly affects the paper’s core contribution.
  • The full comparison and routing implementation remain difficult to verify: Section 4.1 states that AffordanceVLM in the main comparison was fine-tuned on AffordBench, yet its scores in Table 1 are identical to those of the RAGNet-only baseline in Table 3 and differ substantially from the baseline in Table 2. It is also unclear whether all baselines receive the same GPT-generated object boxes and action parsing. If only the proposed method converts implicit intent into explicit object–action pairs and crops the images, the final performance gap cannot be attributed entirely to FGA/MoE. The aggregation function ρ, the tokens being routed, and the gradient handling for hard gating are also insufficiently defined, further limiting reproducibility.
Questions for authors
  • Row-by-row comparison of experimental settings. Could the authors specify, for each row in Tables 1–3, the training data, initialization, trainable modules, use of GPT parsing and cropping, test instruction format, and source of bounding boxes? Please explain in particular the inconsistency in Tab 2-3. If these results follow different protocols, please label them explicitly in the tables.
  • Routing definition and benefits. Is ρ\rho a mean, a norm, or a learnable function? Are the two feature streams normalized to the same scale, and does routing cover visual tokens, text tokens, or the entire sequence? How is hard selection trained? With the same frontend and parameter budget, do results across multiple seeds and statistics broken down by action and region size show that routing consistently outperforms shared weights, and can they explain why cIoU improves while gIoU declines?
  • Test annotations and language difficulty. How many independent images, objects, and action instances are included in UMD and its relabeled version, respectively? Were annotators blinded to model outputs during relabeling? When the final model is evaluated on realistic intent instructions with object names removed, are errors from the GPT frontend included in the end-to-end results? Appendix B.2 explicitly states that training and component ablations use four explicit object–action templates, so localization ability under templated conditions needs to be distinguished from full intent reasoning ability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.