acceptodds
Withdrawn from ICLR 2027

MomentRoute: Asymmetric Resolution Routing for Budgeted Visual Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Budgeted visual memory must search a broad video history while preserving the visual detail needed to answer a query. We show that these two tasks have different resolution requirements: across two cross-modal rerankers, reducing candidate resolution from to lowers selection recall by only 0.23–0.26 percentage points, whereas reducing three selected frames to lowers the 72B reader’s conditional answer accuracy by 10.4 points. MomentRoute exploits this asymmetry by separating candidate localization from evidence reading: it reranks 32 thumbnails in the cloud, then requests three frames for answering. On 1,200 constructed Ego4D-NLQ QA episodes, this allocation reaches 65.1% accuracy, compared with 60.1% for direct thumbnail upload and 58.4% for compressed full-buffer video. Relative to adaptive edge selection with three high-resolution frames, cross-modal reranking improves QA by +5.2 pp (). Accuracy–resource curves expose the accompanying trade-offs: four-frame edge selection reaches 64.1% at 1,009 KiB, compared with 65.1% at 1,136 KiB for two-stage routing (exact McNemar ). Edge selection provides lower TTFT; two-stage routing uses 25% fewer reader visual tokens (768 vs. 1,024). Together, these measurements show how allocating resolution by processing stage preserves broad candidate search and detailed evidence reading, while making the communication and inference trade-offs explicit. Code and benchmark data will be made publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual memory, multimodal assistants, edge-cloud inference, cross-modal reranking, resolution allocation, video question answering

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_txkzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to select a small number of key frames from video history stored on a device to answer user queries under limited communication bandwidth and inference resources. The authors argue that candidate frame retrieval and visual question answering have different resolution requirements. Low-resolution images generally retain enough information to determine whether a candidate frame is relevant to a query, while higher-resolution images are more useful for answering questions that require visual details.

Based on this observation, the authors propose MomentRoute. The method first ranks frames from the video history by computing the similarity between cached visual features and the user query. The top 32 low-resolution thumbnails are then uploaded to the cloud, where a cross-modal model reranks the candidates. Redundant frames are removed based on temporal distance and visual similarity, and the final three high-resolution frames are sent to a vision-language model for question answering.

The authors evaluate frame retrieval on datasets including Ego4D-NLQ, EPIC-Kitchens-100, and QVHighlights, and conduct visual question answering experiments on a constructed Ego4D-NLQ multiple-choice dataset and EgoSchema. On 1,200 Ego4D-NLQ QA samples, the two-stage method achieves 65.1% accuracy, improving by 5.2 percentage points over edge-based selection using three high-resolution frames. The paper also analyzes the trade-offs between accuracy, upload cost, and inference latency under different frame budgets and transmission constraints.

Critical strengths

The paper compares the performance of candidate frame reranking and visual question answering at different image resolutions. The results show that, under the authors' experimental settings, reducing the resolution of candidate frames has only a small impact on retrieval recall, while reducing the resolution of images provided to the vision-language model affects question answering accuracy.

Critical weaknesses
  1. The paper lacks novelty, the method description is incomplete, and the overall logical organization and writing quality are poor.

The proposed method mainly combines existing techniques, including visual feature retrieval, cross-modal reranking, temporal and visual redundancy suppression, and high-resolution image selection. The authors do not clearly explain what new technical mechanism this paper introduces compared with existing coarse-to-fine video retrieval methods. The paper emphasizes that different processing stages have different resolution requirements, but this experimental observation alone does not sufficiently demonstrate the novelty of the proposed method.

In addition, the paper lacks a clear and complete description of the method. Figure 1 only shows the data transmission process between the edge device and the cloud, without showing how the core modules actually work. Algorithm 1 mainly describes similarity-based frame ranking and redundancy removal, but does not fully explain how the cross-modal reranking scores are calculated in the two-stage method or how the reranking results are used for subsequent frame selection. I could not even find a proper method diagram, and it is really difficult for me to clearly understand the core method of this paper. Important technical details are scattered across different sections, without a unified and complete description of the method. This makes it difficult for me to understand the information flow and the specific implementation of the entire method.

  1. First, the abstract does not follow a reasonable order of presenting the research problem, limitations of existing methods, main challenges, proposed method, and experimental results. Without sufficiently explaining the research motivation and technical challenges, the authors directly introduce experimental results such as recall and question answering accuracy at different resolutions, followed by upload size, inference latency, and the number of visual tokens. These different metrics lack necessary explanations and logical connections, making the abstract difficult to understand. It is also difficult to identify the main contribution of the paper from the abstract.

Similar problems also appear in the main text. The paper switches between candidate frame coverage, retrieval recall, redundancy removal strategies, cross-modal reranking, resolution allocation, and communication and inference costs, without clearly establishing the logical relationships between these topics. Some equations and experimental analyses are introduced before the core method has been sufficiently explained, making the paper more difficult to read and understand.

Overall, the paper does not clearly establish the logical connection between the research problem, method design, and experimental conclusions. This not only affects the readability of the paper but also makes it difficult to properly assess the actual contribution and effectiveness of the method.

  1. The paper emphasizes that, under the same three-frame reading budget, the two-stage method improves question answering accuracy from 59.9% to 65.1%. However, this result only shows that cross-modal reranking can improve evidence selection under a fixed three-frame budget. It does not demonstrate that the overall system has higher resource efficiency.

Table 5 shows that the four-frame edge selection method achieves 64.1% question answering accuracy, with an upload size of 1,009 KiB and a TTFT of 628.0 ms. In comparison, the proposed two-stage method achieves 65.1% accuracy, but the upload size increases to 1,136 KiB and the TTFT increases to 814.9 ms. The accuracy difference between the two methods is not obvious. Although the two-stage method uses fewer reader visual tokens, it also introduces additional cloud reranking computation and communication overhead. Therefore, the current experiments do not sufficiently explain under what conditions these additional costs are necessary, nor do they convincingly demonstrate the advantages of the method under overall resource constraints.

In addition, some resource statistics in the paper lack clear explanations. Appendix Table 6 reports an upload size of 1,472 KiB for the Full-32 high-resolution configuration, while Table 18 reports that each high-resolution image is approximately 250.7 KiB. The paper does not sufficiently explain how these numbers are calculated or clearly state whether different configurations use the same compression quality and transmission settings. This makes it difficult to verify the fairness of the communication cost comparisons between different methods.

Furthermore, the paper mainly uses multiple-choice questions constructed from Ego4D-NLQ narrations for downstream visual question answering evaluation. Although the authors perform text filtering and some manual checks, there is still insufficient evidence that this construction can represent real visual memory question answering tasks, especially those involving event order, repeatedly appearing objects, and information integration across time. Therefore, the current experiments provide limited support for the generalization ability of the method.

Questions for authors
  1. Regarding the novelty and completeness of the method

The paper mainly combines existing techniques such as visual feature retrieval, cross-modal reranking, and temporal redundancy removal. Can the authors clearly explain what new technical mechanism this paper introduces compared with existing coarse-to-fine video retrieval methods? In addition, Figure 1 only shows the data transmission process, and Algorithm 1 does not fully explain the implementation of the two-stage method. Can the authors provide a complete method framework diagram and explain how the cross-modal reranking scores are calculated, whether they are combined with the original retrieval scores, and how they are used for subsequent frame selection?

  1. Regarding the logical organization and method description

The overall logic of the paper is difficult to follow, especially the abstract and method sections. The authors introduce a large number of experimental results and technical metrics before sufficiently explaining the research motivation and main technical challenges. The equations, algorithms, and experimental analyses in the main text also lack clear connections. Can the authors reorganize the main research problem, the specific limitations of existing methods, and explain what problem each proposed module is intended to solve? In particular, can the authors explain more clearly the relationship between the method design and the experimental conclusions?

  1. Regarding experimental comparisons and resource statistics

Table 5 shows that the four-frame edge selection method achieves similar question answering accuracy to the two-stage method with lower upload size and TTFT, and the accuracy difference between the two methods is not clearly distinguishable. Can the authors explain under what specific resource constraints the two-stage method has clear advantages over directly selecting more high-resolution frames?

In addition, Appendix Table 6 reports an upload size of 1,472 KiB for the Full-32 high-resolution configuration, while Table 18 reports approximately 250.7 KiB per high-resolution image. Can the authors explain how these numbers are calculated and whether different methods use the same image compression quality and transmission conditions?

  1. Regarding the question answering dataset and the generalization ability of the method

The paper mainly uses a four-choice question answering dataset constructed from Ego4D-NLQ narrations for evaluation. How do the authors ensure that these questions can represent real visual memory question answering tasks? In particular, is the proposed method still effective for questions that require understanding event order, repeated appearances of objects, and integration of information across time? Can the authors provide experimental results for different question types to demonstrate that the improvements are not limited to relatively simple visual recognition questions?

Reviewer_4tmWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents an approach for budgeted visual memory for Question answering.

The key observation of the paper is that searching and reranking video frames to gather evidence for answering a question can be done in a lower resolution without significant change in recall performance. On the other hand, providing the VLM with this evidence in low resolution to answer the question does have a significant impact in answer accuracy.

Based on this, the paper provides an asymmetric routing approach in which the top-k (32) candidate (frames) are first reranked using cloud re-ranker using low resolution images(224x224), from which M frames are selected to be processed by the VLM at a higher resolution (448x448).

The authors provide experiments on few video datasets ( Ego4D-NLQ, EgoSchema, Ego-4D etc).

Critical strengths

Evaluating on four datasets is reasonable.

Critical weaknesses
  • The paper adopts a coarse-to-fine approach, where retrieval is done at low resolution and question answering is done at high-resolution. Coarse-to-fine processing is a extremely common strategy in computer vision, while it may not have been directly applied in this way, there's nothing impactful to learn from the paper as it stands.
  • The whole method felt like a very convoluted way of stating a simple enough heuristic algorithm that can be stated as follows:. first find a set of candidates using cosine similarity, re-rank the top k (32) frames at lower resolution(224x224) using a cloud re-ranker, then select M frames after exclusion, use these M frames at higher resolution to answer the question using a VLM.
  • The paper seems to have very limited novelty or contribution apart from this heuristic, and all the comparison mostly has been design choices in this framework (like various LLM, different matchers for Re-ranking etc).
  • The paper looks AI written, without enough proof reading or polishing.
  • The paper is extremely difficult to comprehend especially the problem formulation, bottleneck analysis and method.
  • The method introduces lot of terms like MMR, DPP etc, without any references, making it difficult for a reader, who is not familiar with this area.
Questions for authors

No questions.

Reviewer_qTKBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents MomentRoute, an edge-cloud visual memory system that uses different image resolutions for candidate retrieval and downstream question answering. The proposed two-stage pipeline first uploads 32 low-resolution thumbnails for cross-modal reranking, then retrieves three high-resolution frames for a VLM reader. The authors also introduce adaptive temporal and visual exclusion for frame selection. Experiments on Ego4D-NLQ, EPIC-Kitchens, QVHighlights, and EgoSchema examine retrieval recall, QA accuracy, bandwidth, and latency. On the constructed Ego4D-NLQ multiple-choice QA benchmark, the two-stage approach achieves 65.1% accuracy with a 72B reader, compared with 59.9% for three-frame edge selection and 60.1% for direct thumbnail upload.

Critical strengths
  1. Useful empirical investigation of resolution allocation. The paper provides evidence that candidate reranking is relatively insensitive to reducing frame resolution from 448 × 448 to 224 × 224, while downstream answering is more sensitive to resolution. The resolution sweeps and comparisons across rerankers support this observation. This is a useful finding for designing resource-constrained visual memory systems.
Critical weaknesses
  1. Limited novelty and insufficient justification of the proposed problem and solution.

This paper does not establish a sufficiently clear contribution beyond combining existing techniques. The proposed pipeline uses embedding-based candidate retrieval, cross-modal reranking, temporal and visual exclusion, and high-resolution frame delivery. These are reasonable design choices, but the paper does not clearly explain what is fundamentally new compared with existing coarse-to-fine video retrieval methods such as VideoTree and T*, or established relevance-diversity selection methods.

The necessity of the proposed system is unclear. Table 5 shows that a simpler four-frame edge-selection method achieves statistically indistinguishable QA accuracy while requiring less upload bandwidth and lower latency. The proposed method saves reader tokens but adds cloud reranking computation, and the paper does not demonstrate a clear overall system-level advantage.

  1. Several reported comparisons and numerical conclusions are inconsistent or potentially misleading.

A particularly concerning example appears in Appendix Table 15. The last column is labeled "Δ Best," but the reported improvements do not consistently correspond to the best-performing baseline in each row.

For the Q1 Fast-Pace subset, MR-Auto achieves 42.52%, while the best baseline reaches 41.90%. The actual improvement over the best baseline is therefore +0.62 pp, rather than the reported +3.50 pp. Similarly, for Q4 Slow-Fixation, MR-Auto achieves 40.76% against a best baseline of 39.52%, giving +1.24 pp rather than the reported +8.35 pp.

This is not simply a formatting issue. The claimed benefit of adaptive calibration is one of the paper's main contributions, and the presentation exaggerates its advantage over the strongest available alternatives.

  1. The method formulation and implementation are not sufficiently clear.

Section 3 describes a two-stage pipeline in which 32 candidates are first reranked by a cross-modal model, followed by temporal and visual exclusion. However, Algorithm 1 primarily describes selection using the original embedding similarities and only mentions the reranking stage in its final line.

It is unclear how the reranker scores replace the original scores, whether the exclusion thresholds are recomputed, and how the returned candidate indices are handled between the edge and cloud stages.

The oracle analysis in Section 2.1 also needs clarification. Equation (1) is presented as an exact decomposition of the miss rate, but its intermediate terms are differences between particular oracle definitions rather than independently measured error categories. The interpretation of these terms as distinct causes of failure is stronger than what the algebra alone establishes.

Given that the paper relies heavily on this decomposition to motivate the proposed method, the definitions and their interpretation need to be more precise.

  1. The paper lacks qualitative evidence and meaningful failure analysis.

Despite focusing on visual memory retrieval, the paper does not provide representative examples from the evaluated videos showing the original candidate frames, reranking results, selected high-resolution evidence, and final QA outputs. I cannot inspect whether the method successfully retrieves visually informative frames, whether it selects redundant or irrelevant content, or which types of questions benefit from high-resolution evidence.

  1. The presentation and organization need substantial improvement.

The manuscript is difficult to follow because of the large number of method variants, oracle definitions, abbreviations, and overlapping experimental comparisons. Important distinctions between measured QA accuracy, projected accuracy, and oracle results are distributed across the main text and appendix.

Questions for authors
  1. What is the main technical novelty of MomentRoute beyond combining existing coarse-to-fine retrieval, cross-modal reranking, and diversity-based frame selection? In particular, how does the proposed approach differ from prior coarse-to-fine video retrieval frameworks such as VideoTree and T*?

  2. Could the authors clarify the exact execution order in Algorithm 1? Section 3 states that exclusion is applied after cloud reranking, whereas the pseudocode appears to select frames using the original cosine scores before describing the reranking stage. It is not clear how the reranker scores enter the final selection procedure.

  3. How was the constructed Ego4D-NLQ QA benchmark validated for visual answerability? In particular, were questions checked against the actual selected video frames rather than their source narrations? A small set of question–frame–answer examples, including failure cases, would help establish what the benchmark is measuring.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.