- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RACE, which achieves spatial grounding through the co-evolution of referent semantics and spatial predictions. It uses affordance probability maps as an intermediate representation for grounding and introduces an explicit architecture for progressively refining this map.
Critical strengths
- The co-evolution of semantics and affordance enables progressive refinement of grounding predictions. The explicit affordance representation supports residual delta updates, allowing the model to localize the target object more accurately through coarse-to-fine refinement. The explicit affordance delta updates in a corarse to fine manner is novel for sptial grounding.
Critical weaknesses
- The paper lacks sufficient novelty. Though the explicit affordance used in spatial grounding is novel, its central idea of reciprocal interaction between semantics and spatial predictions has already been explored in FlowSeg. In my view, the two methods do not differ substantially at a conceptual level: RACE uses an explicit affordance representation for grounding, whereas FlowSeg uses visual queries.
- The experimental evaluation is insufficient. The paper does not compare against FlowSeg, and the existing experiments do not sufficiently isolate the benefits of using an explicit affordance probability map for grounding. More baseline comparisons and ablation studies are needed, particularly a variant that retains co-evolution without using an explicit affordance representation, to assess whether this representation provides additional benefits.
Questions for authors
- Would grounding performance differ substantially if spatial predictions were represented without an explicit affordance map, for example using visual queries as in FlowSeg or another representation? If so, why?If there is no substantial difference, what distinguishes the proposed method from FlowSeg?
5 comments
Official Comment by Reviewer GAJc
It remains unclear why RACE places greater emphasis than FlowSeg on preserving the backbone’s semantic and QA capabilities. I do not see how this distinction follows from the proposed mechanism or architectural design. FlowSeg also includes a language modeling loss and uses a dedicated decoder for spatial grounding. Explicitly maintaining a spatial distribution alone does not appear to explain better preservation of the backbone’s semantic and QA capabilities. Could the authors clarify which specific architectural or training choices support this claimed distinction?
Clarifying the Differences Between RACE and FlowSeg
We really appreciate your question. We clarify the differences between RACE and FlowSeg in terms of application, motivation, and mechanism.
1. Application. General and embodied grounding/QA in RACE vs. Segmentation in general scenes in FlowSeg.
RACE targets diverse grounding and spatial QA tasks across general and embodied settings within a single model. Our training data and evaluations cover multiple settings and tasks, with results showing strong overall performance. FlowSeg focuses on language-conditioned instance segmentation in general visual scenes, and its reported evaluations focus on segmentation benchmarks.
2. Motivation. Semantic–spatial correlation in RACE vs. Semantic disambiguation in FlowSeg.
RACE uses iterative interaction to jointly refine semantic understanding and spatial grounding: semantics guide spatial distribution updates, while spatial evidence feeds back to refine referent semantics. FlowSeg focuses on aligning language conditions with candidate masks for semantic disambiguation.
3. Mechanism. Explicit spatial distributions in RACE vs. Implicit spatial representations in FlowSeg.
RACE explicitly maintains and updates a spatial distribution across stages. Grounding supervision acts directly on this distribution, which weights the visual evidence fed back for semantic refinement and serves as the final grounding prediction. The same distribution connects supervision, feedback, and prediction, without a separate set of latent visual queries carrying the spatial state. FlowSeg instead introduces an additional set of visual queries and jointly refines these queries and language conditions through bidirectional interaction in latent space.
Response to Reviewer GAJc
A1: Thank you for this question. Our comparison with a FlowSeg-style latent co-evolution decoder shows accuracy gains of 3.04–7.80 percentage points across the reported subsets and benchmarks. RACE and FlowSeg share bidirectional semantic–visual interaction, but their differences go beyond performance and task settings:
1.Motivation. FlowSeg focuses on aligning language conditions with candidate masks. RACE aims to preserve the backbone’s semantic and QA capabilities, using a dedicated spatial decoder for task-relevant grounding. This design assigns distinct roles to general semantic understanding and task-relevant spatial inference, while coupling semantic and spatial states through iterative feedback for fine-grained grounding.
2.Mechanism. FlowSeg refines language conditions and visual queries; RACE unifies heterogeneous annotations into distributional targets and explicitly maintains the same spatial distribution for both grounding prediction and feedback evidence selection. Grounding supervision therefore directly constrains feedback weighting, enabling co-evolution between referent semantics and explicit spatial hypotheses.
3.Experimental Comparisons. To evaluate an alternative representation, we constructed a FlowSeg-style latent co-evolution decoder with 16 visual queries propagated across stages through bidirectional interaction with semantic representations. It uses the same backbone initialization, training data, and evaluation protocol as RACE.
These results support the benefit of an explicit affordance distribution in our grounding setting. The coupling between grounding supervision and feedback weighting may help explain these gains.