acceptodds
Withdrawn from ICLR 2027

Text Assisted Event Stream Visual Place Recognition via Multi-Scale Spatial-Temporal Optimal Transport

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Benefiting from the high dynamic range, high temporal resolution and low energy consumption of event cameras, low‑altitude economy research leveraging event cameras has become a popular topic. Several existing attempts introduce natural‑language descriptions into visual place recognition (VPR) to mitigate the sparsity of structural spatial information inherent to event cameras. Nevertheless, these methods still fail to fully address two core challenges: effective encoding of event streams and semantic alignment between event streams and text. In this paper, we propose a multi‑scale spatial‑temporal optimal transport approach termed MSTOT‑EPR to tackle the above challenges. Specifically, given event streams and scene text information, we first adopt a visual encoder and a text encoder to extract respective feature representations. Optimal‑transport constraints are then applied to achieve cross‑modal alignment. Afterwards, a cross‑modal bottleneck fusion module is designed for further bimodal feature fusion. Furthermore, we develop multi‑scale spatial optimal transport and inter‑frame temporal optimal transport modules to mine high‑quality spatial‑temporal event features for scene recognition. Extensive experiments on the NYC‑Event‑VPR and EPRBench datasets fully validate the effectiveness of our proposed framework for event‑based VPR. The source code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Event-based Vision, Visual Place Recognition, Optimal Transport

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_42BdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses semantic sparsity and spatial-temporal correspondence in event-based visual place recognition. It proposes MSTOT-EPR, combining event-text optimal transport (OT) alignment and bottleneck fusion with multi-scale spatial OT and anchor-based temporal OT. DINOv2 and CLIP provide visual and textual features, while SALAD aggregates the fused features into retrieval descriptors. Experiments on NYC-Event-VPR and EPRBench report improved event-based retrieval accuracy. However, the methodological novelty and experimental controls are insufficiently established.

Critical strengths
  1. The paper is well-written and clearly organized, making the proposed framework easy to follow from motivation to methodology and experiments.
  2. The paper includes extensive experiments on two event-based VPR benchmarks, with comparisons against a broad range of baselines and ablation studies examining the contributions of the proposed components. The results provide empirical support for the effectiveness of the approach.
Critical weaknesses
  1. The experimental analysis provides limited insight into why the proposed components help. Targeted evaluations or visualizations in challenging scenarios, such as semantically ambiguous places, large scale changes, and substantial inter-frame displacement, would help assess whether each component addresses its intended challenge.
  2. Comparison with related text-assisted methods is missing. The paper does not discuss whether other methods on EPRBench also leverage text descriptions, nor does it clarify how the proposed approach differs from them. In particular, SG-VPR is a strong and directly relevant baseline; a detailed analysis of the differences between SG-VPR and MSTOT-EPR should be provided.
  3. Computational cost is not reported. The paper does not report runtime or efficiency metrics (e.g., inference time, training time, or parameter count), which is important for assessing the practicality of the proposed framework.
  4. Symbols in Eq. (8) are not explained. The operator g_o appears in Eq. (8) without any definition or explanation of its role. If it is intended to be a learnable projection layer, as is the case for g_v and g_w in Sec. 3.3, this should be stated explicitly.
Questions for authors

Given that SG-VPR already incorporates textual scene descriptions into event-based VPR, what is novel in the second claimed contribution beyond the OT-based alignment mechanism, and how does this mechanism differ from SG-VPR’s cross-modal fusion?

Reviewer_EBrBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates visual place recognition with event camera data and proposes a text-augmented multi-scale spatio-temporal optimal transport framework, MSTOT-EPR. The proposed method partitions the event stream into five consecutiveevent frames. Visual features are extracted via DINOv2, and scene description features are extracted using the CLIP textencoder. The pipeline proceeds sequentially as follows: optimal-transport-based alignment between visual patches and texttokens, followed by cross-modal fusion using bottleneck tokens; cross-scale spatial optimal transport to aggregate spatialfeatures at different resolutions inter-frame optimal transport anchored at the middle frame together with gated tempor aggregation; and global place descriptor generation with SALAD for retrieval. The model is jointly optimized with retrievalloss, local alignment loss, and global MMD alignment loss. Evaluations are conducted on the NYC-Event-VPR and EPRBench datasets. Ablation studies on components and hyperparameters validate the design choices.

Critical strengths

The spatial sparsity, temporal dynamics of event data, and the modal gap between event signals and natural language pose notable challenges for place recognition. This work addresses these issues via local cross-modal alignment, multi-scale spatial modeling, and inter-frame alignment, respectively, yielding a coherent overall design.

Critical weaknesses

W1.Existing ablations are insufficient to demonstrate that the performance gains stem from the proposed OT design. W2.The source of text prompts and the inference protocol remain unclear, which directly hinders credible interpretation of results and fair comparisons.

Questions for authors

Q1.With identical inputs, backbones, training configurations and approximate model capacity, what is the performance gain of OT over simple alternatives? Q2.Please clarify the exact modules included in the baseline and the BF toggle in Table 2. Does the baseline adopt five frames, text, local cross-modal OT, and alignment losses?

Reviewer_dE9SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MSTOT-EPR for text-aided event-camera visual place recognition. The method aligns event tokens with scene-description text, performs multimodal fusion via a bottleneck layer, and further leverages multi-scale spatial optimal transport and inter-frame temporal optimal transport before aggregation. Experiments demonstrate competitive performance on the NYC-Event-VPR and EPRBench datasets.

Critical strengths
  1. In the event domain, visual place recognition is a crucial problem. Due to the inherent asynchronous nature of the event stream, the authors do not directly treat the event stream as static frames; instead, they explicitly design an alignment method to tackle this issue.

  2. The method proposed in this paper achieves promising results on two classic Event VRR datasets.

Critical weaknesses
  1. The method proposed in this paper needs to clarify how the scene text is obtained. How can we acquire scene descriptions for an unseen scene? If such descriptions have to be obtained in advance or retrieved via an auxiliary model, the practicability and efficiency will be greatly reduced.
  2. The performance of the method in this paper is compared on NYC-Event-VPR and EPRBench, but performance comparisons on some other classic Event VRR benchmarks are missing. Is this a limitation caused by the unavailability of textual scene descriptions?
  3. The innovation of this paper is somewhat insufficient. The proposed multi-scale method and cross-modal fusion method are relatively common modules. The authors should further clarify the innovation of this paper.
Questions for authors

How can we obtain these text descriptions for novel scenes? Should the ground truth be acquired in advance, or should it be obtained using other models such as LLMs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.