- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how to select a small number of key frames from video history stored on a device to answer user queries under limited communication bandwidth and inference resources. The authors argue that candidate frame retrieval and visual question answering have different resolution requirements. Low-resolution images generally retain enough information to determine whether a candidate frame is relevant to a query, while higher-resolution images are more useful for answering questions that require visual details.
Based on this observation, the authors propose MomentRoute. The method first ranks frames from the video history by computing the similarity between cached visual features and the user query. The top 32 low-resolution thumbnails are then uploaded to the cloud, where a cross-modal model reranks the candidates. Redundant frames are removed based on temporal distance and visual similarity, and the final three high-resolution frames are sent to a vision-language model for question answering.
The authors evaluate frame retrieval on datasets including Ego4D-NLQ, EPIC-Kitchens-100, and QVHighlights, and conduct visual question answering experiments on a constructed Ego4D-NLQ multiple-choice dataset and EgoSchema. On 1,200 Ego4D-NLQ QA samples, the two-stage method achieves 65.1% accuracy, improving by 5.2 percentage points over edge-based selection using three high-resolution frames. The paper also analyzes the trade-offs between accuracy, upload cost, and inference latency under different frame budgets and transmission constraints.
Critical strengths
The paper compares the performance of candidate frame reranking and visual question answering at different image resolutions. The results show that, under the authors' experimental settings, reducing the resolution of candidate frames has only a small impact on retrieval recall, while reducing the resolution of images provided to the vision-language model affects question answering accuracy.
Critical weaknesses
- The paper lacks novelty, the method description is incomplete, and the overall logical organization and writing quality are poor.
The proposed method mainly combines existing techniques, including visual feature retrieval, cross-modal reranking, temporal and visual redundancy suppression, and high-resolution image selection. The authors do not clearly explain what new technical mechanism this paper introduces compared with existing coarse-to-fine video retrieval methods. The paper emphasizes that different processing stages have different resolution requirements, but this experimental observation alone does not sufficiently demonstrate the novelty of the proposed method.
In addition, the paper lacks a clear and complete description of the method. Figure 1 only shows the data transmission process between the edge device and the cloud, without showing how the core modules actually work. Algorithm 1 mainly describes similarity-based frame ranking and redundancy removal, but does not fully explain how the cross-modal reranking scores are calculated in the two-stage method or how the reranking results are used for subsequent frame selection. I could not even find a proper method diagram, and it is really difficult for me to clearly understand the core method of this paper. Important technical details are scattered across different sections, without a unified and complete description of the method. This makes it difficult for me to understand the information flow and the specific implementation of the entire method.
- First, the abstract does not follow a reasonable order of presenting the research problem, limitations of existing methods, main challenges, proposed method, and experimental results. Without sufficiently explaining the research motivation and technical challenges, the authors directly introduce experimental results such as recall and question answering accuracy at different resolutions, followed by upload size, inference latency, and the number of visual tokens. These different metrics lack necessary explanations and logical connections, making the abstract difficult to understand. It is also difficult to identify the main contribution of the paper from the abstract.
Similar problems also appear in the main text. The paper switches between candidate frame coverage, retrieval recall, redundancy removal strategies, cross-modal reranking, resolution allocation, and communication and inference costs, without clearly establishing the logical relationships between these topics. Some equations and experimental analyses are introduced before the core method has been sufficiently explained, making the paper more difficult to read and understand.
Overall, the paper does not clearly establish the logical connection between the research problem, method design, and experimental conclusions. This not only affects the readability of the paper but also makes it difficult to properly assess the actual contribution and effectiveness of the method.
- The paper emphasizes that, under the same three-frame reading budget, the two-stage method improves question answering accuracy from 59.9% to 65.1%. However, this result only shows that cross-modal reranking can improve evidence selection under a fixed three-frame budget. It does not demonstrate that the overall system has higher resource efficiency.
Table 5 shows that the four-frame edge selection method achieves 64.1% question answering accuracy, with an upload size of 1,009 KiB and a TTFT of 628.0 ms. In comparison, the proposed two-stage method achieves 65.1% accuracy, but the upload size increases to 1,136 KiB and the TTFT increases to 814.9 ms. The accuracy difference between the two methods is not obvious. Although the two-stage method uses fewer reader visual tokens, it also introduces additional cloud reranking computation and communication overhead. Therefore, the current experiments do not sufficiently explain under what conditions these additional costs are necessary, nor do they convincingly demonstrate the advantages of the method under overall resource constraints.
In addition, some resource statistics in the paper lack clear explanations. Appendix Table 6 reports an upload size of 1,472 KiB for the Full-32 high-resolution configuration, while Table 18 reports that each high-resolution image is approximately 250.7 KiB. The paper does not sufficiently explain how these numbers are calculated or clearly state whether different configurations use the same compression quality and transmission settings. This makes it difficult to verify the fairness of the communication cost comparisons between different methods.
Furthermore, the paper mainly uses multiple-choice questions constructed from Ego4D-NLQ narrations for downstream visual question answering evaluation. Although the authors perform text filtering and some manual checks, there is still insufficient evidence that this construction can represent real visual memory question answering tasks, especially those involving event order, repeatedly appearing objects, and information integration across time. Therefore, the current experiments provide limited support for the generalization ability of the method.
Questions for authors
- Regarding the novelty and completeness of the method
The paper mainly combines existing techniques such as visual feature retrieval, cross-modal reranking, and temporal redundancy removal. Can the authors clearly explain what new technical mechanism this paper introduces compared with existing coarse-to-fine video retrieval methods? In addition, Figure 1 only shows the data transmission process, and Algorithm 1 does not fully explain the implementation of the two-stage method. Can the authors provide a complete method framework diagram and explain how the cross-modal reranking scores are calculated, whether they are combined with the original retrieval scores, and how they are used for subsequent frame selection?
- Regarding the logical organization and method description
The overall logic of the paper is difficult to follow, especially the abstract and method sections. The authors introduce a large number of experimental results and technical metrics before sufficiently explaining the research motivation and main technical challenges. The equations, algorithms, and experimental analyses in the main text also lack clear connections. Can the authors reorganize the main research problem, the specific limitations of existing methods, and explain what problem each proposed module is intended to solve? In particular, can the authors explain more clearly the relationship between the method design and the experimental conclusions?
- Regarding experimental comparisons and resource statistics
Table 5 shows that the four-frame edge selection method achieves similar question answering accuracy to the two-stage method with lower upload size and TTFT, and the accuracy difference between the two methods is not clearly distinguishable. Can the authors explain under what specific resource constraints the two-stage method has clear advantages over directly selecting more high-resolution frames?
In addition, Appendix Table 6 reports an upload size of 1,472 KiB for the Full-32 high-resolution configuration, while Table 18 reports approximately 250.7 KiB per high-resolution image. Can the authors explain how these numbers are calculated and whether different methods use the same image compression quality and transmission conditions?
- Regarding the question answering dataset and the generalization ability of the method
The paper mainly uses a four-choice question answering dataset constructed from Ego4D-NLQ narrations for evaluation. How do the authors ensure that these questions can represent real visual memory question answering tasks? In particular, is the proposed method still effective for questions that require understanding event order, repeated appearances of objects, and integration of information across time? Can the authors provide experimental results for different question types to demonstrate that the improvements are not limited to relatively simple visual recognition questions?