acceptodds
Withdrawn from ICLR 2027

Complementary Goal Grounding via Prototype Convolution for OmniGoal Navigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

OmniGoal Navigation requires a single policy to locate goals in different forms, such as categories, images, or instance descriptions. Existing OmniGoal approaches primarily match heterogeneous goals with current observations, while navigation experience associated with different goal forms remains largely unmodeled. However, experience across goal forms can provide complementary cues for locating the same or related concepts, such as appearance from image goals and detailed textual descriptions from instance goals. To incorporate such complementary experience into a unified navigation policy, this work introduces Prototype Convolution(ProtoConv). A prototype graph organizes prior navigation experience into multi-form concept prototypes and their spatial relations. Given a goal, ProtoConv retrieves goal-conditioned prototype kernels and convolves them with an online semantic graph constructed from current observations. The convolution combines cross-form cues that transfer appearance and descriptive attributes for the same concept with cross-concept spatial correlations that support target localization through related observations. By integrating these cues, ProtoConv guides navigation with complementary experience across goal forms and concepts. Evaluations on HM3D, MP3D, and GOAT-Bench demonstrate consistent effectiveness across ObjectGoal, ImageGoal, and Instance Navigation without form-specific policy design.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Navigation, Zero-Shot Goal Navigation, Computer Vision, Graph Matching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Az4eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ProtoConv for navigation to goals specified by categories, images, or instance descriptions. A prototype graph stores prior observations in multiple forms and their relations. Each goal retrieves three form-specific roots, which are expanded into kernels using random walks with restart. These kernels are matched against an online semantic graph, and the responses guide waypoint selection. Experiments cover ObjectNav, image- and text-specified instance navigation, and sequential goals on GOAT-Bench.

Critical strengths
  1. The method allows a goal expressed in one form to retrieve appearance features, descriptions, and spatial context from prior observations. Separating the prototype graph from the online semantic graph gives a clear account of how cross-scene experience can support current goal grounding within a shared navigation framework.

  2. ProtoConv retrieves goal-conditioned kernels to focus on relevant prior knowledge. Its proposed matching function considers both node content and relation sequences, providing contextual evidence beyond appearance similarity. The explicit kernels and spatial response maps also make it easier to inspect which retrieved cues support waypoint selection.

Critical weaknesses
  1. Equation (3) defines a probability distribution with at least α\alpha mass at the root; Appendix A.2 confirms that the probabilities sum to one. With α=0.5\alpha=0.5, every non-root node therefore has probability at most 0.5. Applying the cutoff p(v)≥ρp(v) ≥\rho with ρ=0.6\rho = 0.6 in Eq. (4) excludes all non-root nodes, leaving only the explicitly retained root. Under the stated rule, the kernels would therefore contain no non-root relational context, which is inconsistent with the multi-node examples in Figure 3 and makes the relation-matching gains reported in Table 2 difficult to reconcile with the formulation. The preceding text instead describes retaining a top fraction, but Appendix A.2 repeats the absolute cutoff. The authors need to clarify the implemented rule and report the resulting kernel sizes. This may be a specification error, but it leaves the central mechanism and its experimental interpretation unresolved.

  2. Equation (5) does not define its normalization factor Zℓ(u)Z_\ell (u). The relation indicator also refers to a path from each root to a node without specifying how to handle multiple paths with different relation sequences, or whether cycles and repeated nodes are permitted. These choices determine which evidence contributes and when propagation terminates. Appendix A.3 repeats the path indicator without resolving these ambiguities. Explicit definitions or pseudocode for normalization, path selection or aggregation, and termination are needed to reproduce the response scores on which waypoint selection depends.

Questions for authors
  1. Which kernel-retention rule was actually used in the reported experiments: an absolute probability cutoff, a top-ranked fraction, or another rule? At α=0.5\alpha=0.5 and ρ=0.6\rho = 0.6, how many nodes did the resulting kernels typically contain?

  2. What is the exact definition of Zℓ(u)Z_\ell (u) in Eq. (5), including when there are no valid matches at a given depth? Does normalization depend on all candidate node pairs or only relation-consistent pairs?

  3. When several paths connect a root to the same node, which paths contribute to Eq. (5), and how are their contributions combined? Are edge direction and repeated nodes taken into account, and what terminates propagation in graphs with cycles?

Reviewer_SK1uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes ProtoConv for navigation with category, image, and text goals. It builds a multimodal prototype graph from training-environment observations, retrieves goal-conditioned kernels through a restart random walk, and matches these kernels against an online semantic graph. Graph responses are projected onto a spatial map to select navigation waypoints. Experiments on HM3D, MP3D, and GOAT-Bench report improvements and examine knowledge forms, kernel construction, and relation matching.

Critical strengths

The paper identifies a useful opportunity to reuse multimodal observations across goal forms: a category goal can benefit from stored appearance evidence, while an image or text goal can benefit from contextual object relations. Organizing this evidence into cross-scene prototypes and matching it against online observations is a coherent approach to unified navigation.

Critical weaknesses
  1. The stated kernel construction cannot produce the claimed relational behavior under the reported defaults. Equation (3) and Appendix A.2 define a normalized probability distribution with pi(ri)≥αp_i(r_i)\geq\alpha. For α=0.5\alpha=0.5, every non-root node therefore satisfies pi(v)≤0.5p_i(v)\leq0.5. Equation (4) retains nodes with pi(v)≥ρp_i(v)\geq\rho, where the reported default is ρ=0.6\rho=0.6, and explicitly adds the root. Every kernel must consequently reduce to a single root node. No relational neighborhood survives, and Node and Node+Edge matching coincide because the depth-zero relation indicator is one. Table 2's differences between these variants are therefore unexplained by the stated algorithm. The description of retaining a "top-ρ\rho fraction" conflicts with the absolute cutoff repeated in the appendix. Until the actual implementation is established, the experiments cannot substantiate the proposed cross-form and cross-concept convolution mechanism.
  2. The paper does not establish its incremental contribution under comparable experimental conditions. UniGoal already unifies goal forms through graph matching, and TrajRAG already retrieves cross-scene navigation experience. The distinctive contribution therefore needs to be established at the prototype-kernel and convolution stages. However, ProtoConv accesses an offline multimodal prototype bank and uses Qwen3.7-Plus, whereas 3D-Mem constructs its scene memory online and reports results using GPT-4o. The comparisons do not isolate these resource differences. Sensor assumptions are also unclear: Section 3 describes RGB observations, while Section 4.1 and Figure 2 require RGB-D, and the source of camera poses for online mapping is unspecified. Table 5 reports 28.8/15.8 for 3D-Mem, compared with its original full-set scores of 62.9/44.7, without explaining the source and configuration of this result. The SOTA claim also lacks sufficiently comprehensive comparisons with recent methods, including HGR (Hypothesis Graph Refinement, ECCV 2026). These gaps leave both the contribution of the proposed matching mechanism and its claimed comparative advantage unestablished.
Questions for authors
  1. What kernel-retention rule was actually implemented, and what kernel sizes result at α=0.5\alpha=0.5 and ρ=0.6\rho=0.6?
  2. How are Zℓ(u)Z_\ell(u), multiple root-to-node paths, the Gaussian bandwidth, and the target-verification and STOP rules defined?
  3. What sources and common sensor and evaluation conditions underlie Table 5, particularly the 3D-Mem scores of 28.8/15.8?
  4. What controlled evidence establishes the contribution of prototype convolution beyond existing graph matching and experience retrieval when the stored evidence and VLM backend are held fixed?
Reviewer_GpJKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ProtoConv, a unified framework for navigation using object categories, images, or text goals. It builds a prototype graph from past navigation experience, capturing multimodal object information and spatial relationships. Relevant graph information is matched with the current scene to guide navigation. Experiments on HM3D, MP3D, and GOAT-Bench show promising results across different navigation tasks.

Critical strengths
  1. Evaluation covers several goal modalities plus sequential navigation in GOAT-Bench. Ablations show steady gains from incorporating category, image, and text knowledge, and from relation-consistent matching instead of node-only matching, which provides initial evidence for the proposed mechanism.

  2. The paper highlights a key weakness in unified navigation methods: supporting multiple goal formats doesn’t mean the model transfers knowledge across them. Pulling in cross-form and relational context from prior experience is a meaningful fix.

Critical weaknesses
  1. The hyperparameters reported for Section 4.2 do not seem mathematically compatible. Under a restart probability of alpha = 0.5, any non-root node has a stationary probability of at most 0.5, while the retention threshold is set at 0.6. Since the appendix describesit as a probability cutoff, each kernel would reduce to its root node alone. Relational expansion and multi-node, cross-form matching, which are the paper’s main contributions, would then never be activated, and it becomes unclear how the reported gains from graph structure and relation-consistent matching were obtained. The authors should state the thresholding rule actually used and reconcile the experimental results with it.
  2. Graph construction relies on Grounding DINO and MobileSAM detections, VLM-generated descriptions and relations, prototype merging, and frequency counts, yet none of these stages is quantitatively audited. The paper does not report detection or segmentation errors, relation precision and recall, merge quality, handling of duplicate evidence, or how many candidate edges are kept versus filtered. As a result, the gains cannot be confidently attributed to sound relational reasoning rather than VLM priors or a favorable but noisy pipeline.
  3. . The method uses a lot of components, including detection, segmentation, mapping, VLM calls, and graph matching, but the paper does not report runtime, memory, or compute, and it is unclear whether the VLM runs online. Baselines are also not rerun under the same setup. The gain over MSGNav on GOAT-Bench is small (53.1 vs. 52.0 SR; 30.7 vs. 29.6 SPL), so the added complexity is hard to justify.
Questions for authors
  1. Please provide the exact graph-construction protocol used for all reported numbers: the VLM model/version, frozen prompts, decoding parameters, confidence thresholds, prototype-merging criterion, edge-deduplication key, and the precise training-environment splits used to build each prototype graph.
  2. How robust is ProtoConv to graph errors? Please evaluate performance when controlled fractions of relation edges are removed, corrupted, or shuffled, and when cross-form links are perturbed. This would directly test whether the relation-aware mechanism, rather than the overall perception/VLM stack, causes the gains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.