acceptodds
Under review as a conference paper at ICLR 2027

From Fine-Grained Recognition to Fine-Grained Manipulation: Semantic Visual Primitives for Grounded Multimodal Reasoning

Abstract

Multimodal large language models (MLLMs) can often describe fine-grained visual details, such as part attributes or functional object regions, but may fail to faithfully ground these cues or use them for recognition and action. This reveals a fine-grained semantic grounding gap, where intermediate evidence remains plausible in language or localized by geometry-only primitives, yet is not structured as grounded evidence for reasoning. In this paper, we propose Semantic Visual Primitive Reasoning (SVP-R), a unified framework that turns visual primitives into structured evidence that binds geometry, semantics, and action cues. Each Semantic Visual Primitive binds *where* a part is, *what* property it exhibits, and *how* it can support an action, forming an evidence graph for fine-grained recognition and manipulation. To prevent such primitives from becoming post-hoc rationales, we introduce graph binding, a shared objective that verifies the evidence graph at the node, edge, and graph levels, instantiated as Reinforced Evidence Binding (REB) for recognition and Affordance Evidence Binding (AEB) for manipulation. Extensive experiments on fine-grained recognition and manipulation show that SVP-R consistently outperforms existing grounded reasoning and VLA methods, and interventions confirm that its decisions follow the grounded evidence. These results suggest that fine-grained multimodal reasoning requires binding visual details into structured evidence for perception and action.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Fine-Grained Semantic Grounding, Semantic Visual Primitives, Multimodal Large Language Models, Faithful Visual Reasoning, Fine-Grained Manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_THZEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper points to a "fine-grained semantic grounding gap" in multimodal LLMs: a model can describe a part's attributes verbally, yet fails to tie them to the image evidence or to the final decision or action. To address this, the authors propose SVP-R, which casts evidence as Semantic Visual Primitives, binding together a part's location, its attribute, and the action affordance it supports. These primitives are organized into an evidence graph and validated through "graph binding" at the node, edge, and graph levels. The idea is instantiated twice: as Reinforced Evidence Binding (REB, a correctness-gated GRPO reward) for fine-grained recognition, and as Affordance Evidence Binding (AEB, a training loss) for fine-grained manipulation. The evaluation spans CUB/Dogs/Cars (both within-domain and held-out), a MetaFine localization split, five simulated manipulation tasks (MetaFine plus RoboTwin 2.0), and three real-robot tasks, together with intervention and ablation studies.

Critical strengths
  • A shared interface across recognition and manipulation. Binding "where/what/how" into a single primitive and applying graph binding to both tasks is a coherent conceptual contribution. Showing this interface working in perception and control alike supports the paper's central motivation.
  • Intervention-based analysis of grounding. The degradation observed under evidence removal, swapping, shuffling, and jitter, combined with the interface-layer analysis, provides evidence that predictions genuinely depend on the intermediate primitives rather than merely being accompanied by them.
Critical weaknesses
  1. The main baseline gains are not isolated from the training recipe. Table 6 supports a benefit from the binding terms within SVP-R, but it does not resolve the lack of supervised warm-up for GRIT and Visual-RFT in Table 1. A common supervised starting point is needed to establish how much of the larger comparative gain is specific to binding.
  2. Dependence on evidence and correctness of evidence are not fully separated. Corrupting a primitive can show that the model uses it, but does not by itself establish that its location and attribute are both correct. The automatically generated Dogs/Cars targets make independent semantic validation especially important; the CUB labels offer a direct setting in which to test this part of the claim.
Questions for authors
  1. How much of the recognition gain comes from binding rather than supervised warm-up? Table 6 already shows benefits from the REB terms within the 3B recipe. To interpret the much larger Cars gains in Table 1, I would like a comparison with GRIT and Visual-RFT starting from the same supervised checkpoint and using matched RL data and update budgets.

  2. Does graph binding improve the correctness of the evidence, not just dependence on it? Does the grounding hit rate in Table 3 require both a correct part location and a correct attribute, or only localization? Using CUB's official annotations, I would like this joint measure for full REB and the ablated rewards in Table 6 alongside classification accuracy. The result would directly test the verified-evidence claim without relying on agreement with automatically generated Dogs/Cars labels.

8 comments
AuthorsOpenReview ↗

Thank you for your constructive comments, and the revised version has been uploaded.

Thank you again for your constructive comments. We have uploaded the revised paper, with all changes marked in blue.

The joint correctness of the emitted evidence, together with the reward that contains only localization and semantic information, has been added to Tab. 8 and Sec. 4.3. As two tables have been added, Tab. 3 and Tab. 6 in our responses are now Tab. 5 and Tab. 8.

As the discussion period is ending soon, we would greatly appreciate it if you could kindly take a look at our responses, and we sincerely hope they have addressed your concerns. We are happy to provide any further clarification.

AuthorsOpenReview ↗

Thank you for your time and consideration. We would appreciate your further feedback on our rebuttal.

Dear Reviewer,

We would appreciate it if you could kindly review our rebuttal when you have a chance!

We would also like to note that Reviewer Mpon provided positive feedback following our rebuttal and maintained their positive score of 4. We welcome any further questions or concerns and would be happy to address them.

Thank you for your time and consideration!

AuthorsOpenReview ↗

A sincere request for your feedback before the discussion closes

Dear Reviewer THZE,

We sincerely apologize for writing to you again. As the discussion closes very soon, we would be truly grateful if you could spare a few minutes for our responses. We have completed both experiments exactly as you suggested.

  1. Supervised warm-up (Q1). We re-trained GRIT and Visual-RFT from the same supervised checkpoint as SVP-R, with matched RL data and update budgets. SVP-R still leads on both datasets, 91.29 against 89.70 on CUB and 91.42 against 89.03 on Cars.

  2. Correctness of the evidence (Q2). Using the official CUB annotations, the joint correctness of location and attribute rises from 47.3 under the node-only reward to 69.2 under the full reward, while accuracy rises from 80.71 to 84.87. The result is now in Tab. 8 and Sec. 4.3 of the revised paper.

If these results address your concerns, we would deeply appreciate it if you could kindly consider raising your score. If anything remains unclear, we will respond immediately. Thank you very much for your time.

Best regards, The Authors

Reviewer_S8HPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes SVP-R, a framework that connects part locations, semantic attributes, and action affordances through structured visual evidence. A graph-binding objective checks this evidence at the node, edge, and graph levels. The framework is implemented through RL for fine-grained recognition and robotic manipulation. Experiments report improvements in classification, localization, and manipulation, with ablation studies examining whether predictions depend on the generated evidence.

Critical strengths

1.The paper addresses a relevant problem: plausible visual explanations do not necessarily reflect the evidence used for a prediction. The focus on connecting intermediate evidence to decisions is well motivated.

2.The intervention experiments are the strongest part of the evaluation. On CUB, shuffling attributes across parts reduces accuracy from 84.87% to 41.04%, suggesting that the model uses the part–attribute associations. The manipulation interventions provide complementary evidence of this dependence.

3.The experiments cover both visual recognition and robotic manipulation, including real-robot tasks. The reward and interface-layer ablations help explain the design choices.

Critical weaknesses

1.The reward depends on a fixed annotation space. Attribute verification checks whether a predicted part–attribute pair belongs to the annotated set. While multiple traces are allowed, a valid trace using correct but unannotated attributes may receive no semantic credit.

2.The generalization claims need stronger support. The Held-out split tests new classes within the same datasets, with largely shared part structures and attribute vocabularies. This supports class-level transfer, but leaves transfer to new visual domains or attribute vocabularies untested.

3.The broader framing needs stronger support. Recognition and manipulation seem largely disconnected. The evaluation is also limited in scope. Results on V*, HR-Bench, or a general visual recognition benchmark would help clarify whether the gains extend beyond the specific classification and manipulation tasks studied.

4.The relationship to closely related work needs clarification. SaGe [1] also organizes structured visual evidence and combines supervised training with graph-related reinforcement rewards for grounding and question relevance.

[1] Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models

Questions for authors

Listed in the "Weaknesses" section.

5 comments
AuthorsOpenReview ↗

Thank you for your constructive comments, and the revised version has been uploaded.

Thank you again for your constructive comments. We have uploaded the revised paper, with all changes marked in blue.

The results on V* and CV-Bench-2D have been added to Sec. 4.1, Sec. 4.2 and App. A.1, and SaGe is now cited and discussed in the related work (Sec. 2). As two tables have been added, Tab. 2, Tab. 4 and Tab. 5 in our responses are now Tab. 4, Tab. 6 and Tab. 7.

As the discussion period is ending soon, we would greatly appreciate it if you could kindly take a look at our responses and let us know whether they address your concerns. We are happy to provide any further clarification.

AuthorsOpenReview ↗

Thank you for your time and consideration. We would appreciate your further feedback on our rebuttal.

Dear Reviewer,

We would appreciate it if you could kindly review our rebuttal when you have a chance! We would also like to note that Reviewer Mpon provided positive feedback following our rebuttal and maintained their positive score of 4. We welcome any further questions or concerns and would be happy to address them.

Thank you for your time and consideration.

AuthorsOpenReview ↗

A sincere request for your feedback on our responses

Dear Reviewer S8hp,

Thank you again for your constructive comments, which gave us a broader view from which to validate our method. With the discussion closing very soon, we would be sincerely thankful if you could find a moment to look at our responses.

Following your suggestion, we evaluated SVP-R on V*, where it reaches 87.96 and exceeds DeepEyes (85.6) trained on the same data domain, in a single inference pass. The same model, evaluated on CV-Bench-2D with no training data from it, reaches 79.83 and exceeds Qwen2.5-VL-32B (76.2), which supports its cross-domain transfer. Both results are now in Tab. 2 and Tab. 3 of the revised paper.

We also clarified the relation to SaGe in terms of motivation and method design, and this discussion has been added to the related work.

If our responses resolve your concerns, we would be deeply grateful if you could kindly consider raising your score. Your assessment means a great deal to us, and we are ready to answer any remaining question right away. Thank you again sincerely for your time and effort.

Best regards, The Authors

Reviewer_MponOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a novel framework for grounding recognition and manipulation with multimodal large language models in images. It builds on previous work that used chain-of-thought prompting with images and anchored reasoning traces with visual primitives (bounding boxes of local features/parts). Specifically, it addresses the gap that previous work using visual primitives only makes use of localization of parts, but not the semantic content in the regions of interest.

The framework is applied and evaluated on recognition and robotic manipulation. Training is supervised on tuples of semantic visual primitives that contain the localization of a part, a semantic descriptor, and, for manipulation, the possible poses the part allows for. The primitives are structured as a graph, and the loss/reward is computed on this graph.

The evaluation is performed on a wide range of datasets, and the framework is compared against a range of other methods that were re-trained on the same data, making the comparison of results fair.

Critical strengths

This work introduces a well-thought-out framework for improving fine-grained visual recognition and robotic manipulation using MLLMs. The results show that this framework can lead to substantial improvements across both tasks. A low-hanging fruit would have been to simply extend previous visual primitive methods with semantic descriptors in the reasoning traces; however, the authors went beyond that by introducing supervision on the graph structure, which prevents the model from hacking the reward and binds the outcome to the reasoning traces, as evidenced by the ablation experiments.

Beyond the results, the methodology of the work is very strong. Baselines were re-trained on the same data to allow for a fair comparison; a good range of datasets was evaluated, as well as different model sizes. I appreciate that the paper is not cramped with results, but instead uses very targeted ablations that support the contributions.

Critical weaknesses

While the general idea, setup, and evaluation are very straightforward to understand, the mathematical framework is a bit hard to follow at times. Firstly, the profile W\mathbf{W} in Eq. 5 seems to be a crucial component of the supervision; however, it is not clear to me how it is computed. Furthermore, the explanation of the rewards in Eq. 6 falls very short, as these variables are only introduced in this equation. It is also not clear to me what "one statistic per level" refers to in the paragraph of Eq. 6.

I believe that this issue can be easily addressed, for example, through an additional appendix that explains the mathematical framework more thoroughly.

Questions for authors
  1. It would be interesting to know how impactful the supervision using the graph structure is. Did you run any experiments where you only extended the reasoning traces with semantic information on top of the localization? If yes, how did this perform compared to the full supervision setup using the graph?
  2. Is it correct that you only supervised the centers of bounding boxes in this framework and not their extent (this is what I understood based on Eq. 4 at least)? If so, how do you explain that the model learnt the extent of bounding boxes so well as depicted in Figure 3?
  3. Could you offer more explanation on some of the mathematical framework? Specifically, I would like to know more about the profile W\mathbf{W} in Eq. 5 and the rewards in Eq. 6.
  4. What do you mean by "REB takes one statistic per level" in the context of recognition in the paragraph "Reward" around Eq. 6?
3 comments
AuthorsOpenReview ↗

Thank you for your positive and constructive comments, and the revised version has been uploaded.

Thank you again for your positive and constructive comments!

We have uploaded the revised paper, with all changes marked in blue. The mathematical framework is now clarified in Sec. 3.2, Sec. 3.3 and App. D, and the comparison with the reward that supervises only localization and semantics has been added to Tab. 8 and Sec. 4.3.

All revisions are consistent with our responses above. We sincerely thank you for your time and support.

I have read your revised version, and it answers all my open questions!

AuthorsOpenReview ↗

Thank you for your feedback and support

Dear Reviewer Mpon,

Thank you very much for taking the time to read our revised version, and we are glad that it answers all your questions. Your comments on the mathematical framework made the paper clearer, and the appendix you suggested is now part of it.

We sincerely appreciate your careful review and your support of our work.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.