- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a well-curated benchmark built primarily from real-world industrial data and enriched with diverse logistics safety scenarios. The authors evaluate mainstream multimodal large language models on the benchmark and commit to publicly releasing the dataset upon acceptance.
Critical strengths
-This work collects images and videos from operational warehouse environments and involves domain experts in multiple rounds of annotation and validation. Dense objects, occlusions, and low-resolution imagery also make the benchmark representative of the visual challenges encountered in practical warehouse safety monitoring.
-The benchmark covers multiple levels of capability, from object and attribute perception to operational rule understanding, hazard identification, and response planning. It also combines single-choice, multiple-choice, and open-ended questions. This design helps distinguish weaknesses in perception, domain knowledge, and risk reasoning.
-Beyond comparing open-source and proprietary models against human performance, the paper examines reasoning budgets, error categories, and the relationship between risk recall and false alarm rates. These analyses suggest concrete directions for improvement.
Critical weaknesses
The paper does not include controlled comparisons using single-frame inputs or shuffled frame orders, making it unclear how much model performance depends on temporal reasoning. Adding these ablations and separately reporting results on tasks that inherently require temporal information would strengthen the benchmark’s value for video understanding.
Questions for authors
The main text claims that leading models surpass novice humans, but Table 1 reports an overall accuracy of 0.71 for the best model, below the students’ 0.73. These discrepancies should be clarified, and the evaluation protocol and corresponding claims should be made consistent to strengthen confidence in the human–model comparisons.
4 comments
We sincerely thank you again for your valuable time and thoughtful feedback. During the rebuttal period, we have comprehensively addressed your comments, and hope that our responses have clarified the key aspects. We would be pleased to provide additional clarifications if needed.
Thank you for your response. My concern has been solved and I will maintain my original score. Good Luck.
Thank You for Your Positive Feedback
We are pleased that the rebuttal response has effectively addressed your concerns. The relevant discussion has been included in the revised manuscript to enhance clarity and facilitate readers’ understanding. We sincerely appreciate your valuable time and positive feedback.
Response to Reviewer 66Pt
Dear Reviewer 66Pt,
We sincerely thank you for the positive feedback and recognition of our work. Below, we provide comprehensive clarifications regarding your comments.
[W1]: Video Input Ablations and Analysis
Response:
a. Following your constructive comment, we have compared the impact of six different video input strategies on model performance and are excited to share our results here. Specifically, we randomly sampled a subset of 500 VQA pairs across 10 evaluation tasks, and obtained the accuracies of Qwen3.7-Plus and GPT-5.5 under these six input settings, as shown in the tables below. Note that the key frame was manually selected as the frame containing direct evidence for answering the question.
b. These results lead to two valuable conclusions.
(i) LogiScope-VQA assesses abilities ranging from single-frame understanding to multi-frame information integration. Key-frame-only inputs largely preserve performance on FM, CP, and WC but reduce EOC accuracy sharply in both models. Removing the key frame also affects FM much more than CP, revealing differences in evidence redundancy alongside contextual requirements.
(ii) Evaluation tasks differ in their sensitivity to temporal frame order. Reversing or shuffling frames substantially degrades EOC performance, moderately affects FP-S, FP-C, SR and PAC, and has little impact on other tasks like PSD. These contrasting responses reveal task-dependent sensitivity to sequence organization beyond evidence availability.
Qwen3.7-Plus:
GPT-5.5:
[Q1]: Difference Between the Textual Phrasing and Table 1
Response:
Thank you for spotting this oversight. To clarify, the leading models indeed fall slightly short of the novice students in overall accuracy (0.71 vs. 0.73). The values in Table 1 are correct, and we have revised the statement accordingly (Page 6, Line 316-317). Additionally, we have carefully reviewed and updated the description of the evaluation protocol in Appendix C.4 (Page 21, Line 1088).
Summary: Thank you again for your insightful suggestions, which have helped to strengthen our manuscript. The additional video input ablations reveal LogiScope-VQA’s rationality in spatio-temporal perception and reasoning, and enrich the exploration of video understanding in industrial logistics scenarios. We hope these additional analyses clarify the practical value of our benchmark, and we are pleased to further discuss and resolve any further concerns.