acceptodds
Withdrawn from ICLR 2027

Beyond Correspondence: Understanding Response Verification in Image–Event Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Image–event retrieval matches static images with the brightness changes recorded by event cameras. Candidate verification can refine learned similarity rankings. Understanding its effects requires examining the response cues that distinguish candidates and their influence on initially correct matches. We present a controlled evaluation framework that holds trained models and their top-five candidates fixed while comparing image-derived responses with observed event responses. We compare acquisition-informed and static response representations and test whether combining each response score with the initial similarity score improves on response-only ranking. Experiments with three models on two controlled benchmarks show that acquisition-informed verification improves mean bidirectional top-1 accuracy over initial rankings. Preserving coarse temporal structure aids verification on N-Caltech101, while static gradients outperform acquisition-informed responses for the EventBind and CEIA retrieval models on mini N-ImageNet. With a validation-selected weight held fixed, score fusion improves three-run mean bidirectional top-1 accuracy by 1.60–6.03 percentage points over response-only ranking across eight test settings, while also outperforming initial rankings. A CEIA case study shows how fusion reduces displacement of initially correct matches. These findings provide an empirical basis for evaluating response representations and their integration with learned correspondence in image–event retrieval. Code is available in an https://anonymous.4open.science/r/response-verification-review anonymous repository.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
image-event retrieval, cross-modal retrieval, event-based vision, representation learning, response verification, reranking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZhfBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies paired image-event instance retrieval and asks whether a training-free verification step can fix the top-1 errors left by learned image-event aligners. Keeping each trained model and its top-five shortlist fixed, it reranks the shortlist by comparing hand-crafted responses, either a simulation of the benchmark's documented camera motion or static image gradients matched against event-count gradients, and it also fuses this response score with the original correspondence score. Experiments with three trained rankers on two display-recorded event benchmarks report gains from reranking, an advantage for static gradients on mini N-ImageNet, and further gains from fusion.

Critical strengths
  1. The controlled protocol is the right way to ask this question. Freezing each checkpoint and its shortlist means every change comes from the response representation alone, and the paper reports per-seed and per-direction results, including the case where verification hurts.

  2. A training-free comparison of image edges with event activity corrects many errors of CLIP-based aligners. It needs no training, and the rescue and damage accounting shows why fusion helps where response-only ranking does not.

Critical weaknesses
  1. The evidence does not reach beyond display-recorded benchmarks, and the verifier is built from those benchmarks' acquisition protocols. Both datasets were made by showing an image on a monitor and moving an event camera along a documented path. The N-Caltech101 response replays that saccade path with a training-calibrated raster scale (Eq. 51 to 54), and the mini N-ImageNet response uses a display transform and translation extent fitted on training pairs (Eq. 66). Here the event stream is close to a deterministic function of the displayed image, so rendering the image, or just taking its gradients, should match the events well. The title and abstract speak about image-event retrieval in general, and natural recordings are left to future work. I am not convinced the takeaway survives outside this regime. Supporting the general framing needs real paired data, for example frames and events from a DAVIS-type sensor with unknown motion. If the claims are instead restricted to display-recorded benchmarks, I am unsure what the community learns beyond them.

  2. The acquisition-informed response, presented as the main verifier, is not shown to beat simple static baselines. On mini N-ImageNet the static gradient response G beats S by 3.57 and 4.61 pp (Table 2), and under controlled motion the plain spatial activity score A is the best response-only ranker on both subsets (Table 20). On N-Caltech101, G and A are not reported on the test set, although A is defined for that dataset in Sec. B.5. All compared verifiers are the authors' own hand-crafted responses, with no learned reranker such as the pairwise classifier in [1], which the submission cites. Predicting brightness increments from image gradients and matching them to events is standard in event-based tracking, for example EKLT [2].

  3. The writing makes the method look heavier than it is. A few lines of method are spread over 94 numbered equations, some repeated (Eq. 13 to 15 reappear as Eq. 41, 47 and 48), and PP-UOT, which is not a contribution, fills much of the method section.

[1] Cross-modal Place Recognition in Image Databases using Event-based Sensors, arXiv 2023.
[2] EKLT, Asynchronous Photometric Feature Tracking Using Events and Frames, IJCV 2020.

Questions for authors
  1. How do G and A perform on the N-Caltech101 test set with the same shortlists, alone and fused with correspondence? If they match or exceed S there, the case for acquisition-informed verification rests on mini N-ImageNet controls that already favor static responses. This one matters most for my rating.

  2. PP-UOT drops from 78.9 on internal validation to 41.0 on Original Test (Table 10), while EventBind and CEIA do not drop. What differs between the training-pool recordings and the released validation recordings, and does this shift affect the fusion weight, which was selected on the internal validation split?

Reviewer_FzUgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies image–event retrieval, where the goal is to match an image with its corresponding event-camera recording, and explores whether a verification stage can improve retrieval performance. Given the top five candidates retrieved by existing models such as EventBind and CEIA, or the authors’ proposed optimal-transport-based ranker (PP-UOT), the method reranks candidates by comparing compact representations derived from images and event recordings. The main approach uses acquisition-informed responses that simulate the documented camera motion used to record each benchmark to predict brightness changes and compare them with observed events. The authors also investigate simpler gradient-based responses and combine verification scores with the original retrieval similarities. Experiments on N-Caltech101 and mini N-ImageNet show that verification generally improves top-1 retrieval accuracy, with different response types performing better across datasets. Combining verification and retrieval scores consistently outperforms either approach alone in the evaluated settings.

Critical strengths

1- The experimental evaluation is carefully controlled, keeping the trained models and top-five candidates fixed to isolate the effect of verification. Results across three training runs, paired confidence intervals, and internal controls such as random reranking and shuffled acquisition orders strengthen the reliability of the findings.

2-The score fusion analysis is convincing, showing consistent improvements over both the initial and response-only rankings across all evaluated settings. The correction and displacement analysis further explains these gains by showing how fusion reduces the loss of initially correct matches while preserving the benefits of verification.

Critical weaknesses

1- The benefit of the acquisition-informed response is unclear. On mini N-ImageNet, simpler static gradient responses outperform the proposed acquisition-informed response, which even reduces CEIA’s image-to-event accuracy when used alone. On controlled-motion subsets, static responses achieve the best response-only results, while adding acquisition information provides little additional improvement. On N-Caltech101, preserving acquisition order offers only a small gain, and no static baseline is provided. These findings suggest that most improvements come from spatial structure rather than modeling the acquisition process. The paper should better explain when and why acquisition information is beneficial, especially given its additional requirements.

2- The evaluation is limited in scope and comparisons. All experiments use screen-recorded static images with known camera motion, leaving generalization to natural event recordings unclear. The method is also not compared against existing reranking and verification approaches, such as learned rerankers used for event-to-image matching, classical spatial verification from image retrieval, or comparing candidates with images reconstructed from events (e.g., E2VID), several of which are discussed in the related work. Furthermore, improvements are uneven across retrieval models, with larger gains for weaker rankers and no statistically significant improvement from the acquisition-informed response for CEIA on mini N-ImageNet. Several important results are also presented only in the appendix, making it harder to assess the main claims from the paper itself.

Questions for authors

1- Can you compare the proposed verification with at least one existing reranking approach under the same fixed top-five candidates, for example a lightweight learned reranker trained on the same training pairs, or matching candidates against images reconstructed from the event recordings?

2- Can you evaluate the static response and fusion on a dataset with natural event recordings and unknown camera motion? Even a small-scale experiment would help show whether the gains from verification and fusion transfer beyond screen-recorded benchmarks.

Reviewer_VCFTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores a second stage reranking for image-event retrieval, where each event recording was captured by moving an event camera in front of a displayed static image. A CLIP-like trained model provides the initial ranking and a fixed top-5 shortlist, which is reranked by a hand-crafted score that compares the image with the recorded events. The motion-based variant simulates which brightness changes the image would produce under the documented camera motion and compares them, pooled to an 8x8 grid, with the equally pooled observed events. The static variants simply compare edge maps of the image and of the accumulated event counts.
The authors show that a mixed score of the trained model and handcrafte reranking wins in all combinations, beating both the original ranking and the hand-crafted score alone. On mini N-ImageNet the static edge maps work better than the motion simulation.

Critical strengths

S1) Reranking the fixed top-5 candidates with the hand-crafted score, and even more so mixing it with the model score, raises the top-1 accuracy of EventBind and CEIA on both datasets, and in every seed. The score is cheap to precompute and the mixing weight was tuned on the validation data of another model.

S2) The authors are honest in their analysis and report that a plain edge map beats their own motivated score, support their claims with reordering analysis, and count how many correct matches are rescued and how many are displaced, which explains why fusion helps.

Critical weaknesses

W1) The motion-based score does not seem to be necessary. On mini N-ImageNet, the static edge-map score and even the simplest activity score work better, but this is only shown in the appendix (Tables 17 and 20). In Table 20, the score built for vertical motion even beats the one built for horizontal motion on horizontal recordings, so it is not clear whether the simulation captures the motion at all. On N-Caltech101 no static score is reported. Please report the static scores on N-Caltech101 for all three models.

W2) The method follows how the benchmarks were obtained. Both datasets are static images shown on a monitor and filmed under a known camera motion, so the events are simply the edges of the image blurred by that known motion. It is unclear which practical retrieval problem this solves, and nothing indicates that the findings transfer to real recordings with unknown motion, moving scenes or changing illumination.

W3) The hand-crafted score is never evaluated as a retrieval method on its own. It is a precomputable low-dimensional vector, so ranking the whole gallery with it, alone or fused, costs nothing extra, and without these numbers it is unclear whether the learned model contributes anything beyond supplying a shortlist. Furthermore, a shortlist of three is best on validation and accuracy drops quickly for longer ones (Table 35), yet fusion is never evaluated beyond five candidates, where keeping the model score should matter most.

W4) The role of PP-UOT is unclear. It is an elaborate ranker trained from scratch that only serves as a third starting point and is not claimed as a contribution, yet its accuracy on mini N-ImageNet roughly halves between internal validation and the official test split. This is left unexplained, although the fusion weight and the shortlist size are selected on PP-UOT validation data.

Minor weaknesses:

MW1) Fig. 1 and Sec. 4.3 are hard to follow, and Fig. 3 is hard to understand without the nested score combinations of Appendix C.5. The outcome of this whole analysis is just that the motion-based score adds almost nothing once the static one is present.

MW2) The abstract and Sec. 4.2 present the motion-based verification as a general improvement, although for CEIA on mini N-ImageNet the overall change is not significant and one direction gets worse, and one PP-UOT even loses on the N-Caltech101 test.

MW3) The initial models see only a capped number of events whereas the hand-crafted scores use all of them (Appendix C.2), so part of the gain may come from events that the initial ranking never saw.

Questions for authors

Q1) Please report the static scores on N-Caltech101 for all three models.

Q2) Do any of the findings hold on recordings that are not displayed images?

Q3) How well does the hand-crafted score rank the whole gallery on its own, and fused with the model score, also with longer shortlists?

Q4) Why does PP-UOT lose most of its accuracy between internal validation and the official test split on mini N-ImageNet while CEIA and EventBind gain?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.