acceptodds
Under review as a conference paper at ICLR 2027

LogiScope-VQA: Benchmarking Vision-Language Models for Logistics Hazard Identification in Industrial Scenarios

Abstract

Large Multimodal Models (LMMs) large-scale deployment in industrial warehouse settings specifically necessitates that models exhibit human-expert-level hazard-oriented perception, understanding, and reasoning capabilities. However, the scarcity of real industrial data, tightly coupled to commercial terms, significantly hampers further advancement. To bridge this gap, we curate LogiScope-VQA to investigate the practical applicability of mainstream LMMs in real-world logistics operations. LogiScope-VQA comprises 2,476 images and 2,918 videos primarily sourced from real-world logistics parks, along with 10,274 VQAs meticulously curated and validated by human annotators. Grounded in 18 core objects and 20 risk types, we devise 39 subtasks aligned with three principal themes: industrial element perception, warehouse knowledge understanding, and potential risk reasoning. Furthermore, we incorporate dynamic thinking-budget configurations and dual-dimensional risk bias analyses to elucidate the properties of LMMs. Extensive experiments unveil that even powerful proprietary models, including GPT-5.5, Gemini-3-Pro, and Claude-Opus-4.7, exhibit a significant gap relative to human performance. The unique challenge of jointly integrating perception, understanding, and reasoning for hazard identification poses substantial headroom for further improvement on LogiScope-VQA. We additionally reveal the pervasive security bias issue that impedes LLMs' practical deployment in real-world settings. We provide evaluation code at the anonymous repository and upload representative data samples as supplementary material. The full project will be fully open-sourced upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
Industrial Benchmark, Logistics Hazard Identification, Visual Question Answering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_66PtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a well-curated benchmark built primarily from real-world industrial data and enriched with diverse logistics safety scenarios. The authors evaluate mainstream multimodal large language models on the benchmark and commit to publicly releasing the dataset upon acceptance.

Critical strengths

-This work collects images and videos from operational warehouse environments and involves domain experts in multiple rounds of annotation and validation. Dense objects, occlusions, and low-resolution imagery also make the benchmark representative of the visual challenges encountered in practical warehouse safety monitoring.

-The benchmark covers multiple levels of capability, from object and attribute perception to operational rule understanding, hazard identification, and response planning. It also combines single-choice, multiple-choice, and open-ended questions. This design helps distinguish weaknesses in perception, domain knowledge, and risk reasoning.

-Beyond comparing open-source and proprietary models against human performance, the paper examines reasoning budgets, error categories, and the relationship between risk recall and false alarm rates. These analyses suggest concrete directions for improvement.

Critical weaknesses

The paper does not include controlled comparisons using single-frame inputs or shuffled frame orders, making it unclear how much model performance depends on temporal reasoning. Adding these ablations and separately reporting results on tasks that inherently require temporal information would strengthen the benchmark’s value for video understanding.

Questions for authors

The main text claims that leading models surpass novice humans, but Table 1 reports an overall accuracy of 0.71 for the best model, below the students’ 0.73. These discrepancies should be clarified, and the evaluation protocol and corresponding claims should be made consistent to strengthen confidence in the human–model comparisons.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 66Pt

Dear Reviewer 66Pt,

We sincerely thank you for the positive feedback and recognition of our work. Below, we provide comprehensive clarifications regarding your comments.

[W1]: Video Input Ablations and Analysis

Response:

a. Following your constructive comment, we have compared the impact of six different video input strategies on model performance and are excited to share our results here. Specifically, we randomly sampled a subset of 500 VQA pairs across 10 evaluation tasks, and obtained the accuracies of Qwen3.7-Plus and GPT-5.5 under these six input settings, as shown in the tables below. Note that the key frame was manually selected as the frame containing direct evidence for answering the question.

b. These results lead to two valuable conclusions.

(i) LogiScope-VQA assesses abilities ranging from single-frame understanding to multi-frame information integration. Key-frame-only inputs largely preserve performance on FM, CP, and WC but reduce EOC accuracy sharply in both models. Removing the key frame also affects FM much more than CP, revealing differences in evidence redundancy alongside contextual requirements.

(ii) Evaluation tasks differ in their sensitivity to temporal frame order. Reversing or shuffling frames substantially degrades EOC performance, moderately affects FP-S, FP-C, SR and PAC, and has little impact on other tasks like PSD. These contrasting responses reveal task-dependent sensitivity to sequence organization beyond evidence availability.

Qwen3.7-Plus:

SettingFP-SFP-CCPWCSRORPACFMPSDEOCOverall
Ordered Frames (baseline)0.690.650.890.910.590.400.470.830.530.680.68
Reversed Frames0.560.540.870.910.500.400.390.830.520.260.59
Shuffled Frames0.520.540.870.910.470.350.410.810.530.260.58
Key Frame Removed0.450.410.850.890.410.300.350.400.480.190.48
Key Frame Only0.600.600.890.910.550.400.360.830.530.490.63
Random Single Frame0.420.410.750.890.380.250.310.400.410.210.46

GPT-5.5:

SettingFP-SFP-CCPWCSRORPACFMPSDEOCOverall
Ordered Frames (baseline)0.630.680.780.920.610.200.770.810.640.850.72
Reversed Frames0.520.600.770.920.500.200.640.810.630.230.61
Shuffled Frames0.500.570.780.910.510.200.680.810.620.250.61
Key Frame Removed0.400.460.720.910.390.150.650.500.540.210.52
Key Frame Only0.590.610.780.920.550.150.650.830.620.660.67
Random Single Frame0.400.440.700.890.300.150.620.560.520.150.49

[Q1]: Difference Between the Textual Phrasing and Table 1

Response:

Thank you for spotting this oversight. To clarify, the leading models indeed fall slightly short of the novice students in overall accuracy (0.71 vs. 0.73). The values in Table 1 are correct, and we have revised the statement accordingly (Page 6, Line 316-317). Additionally, we have carefully reviewed and updated the description of the evaluation protocol in Appendix C.4 (Page 21, Line 1088).

Summary: Thank you again for your insightful suggestions, which have helped to strengthen our manuscript. The additional video input ablations reveal LogiScope-VQA’s rationality in spatio-temporal perception and reasoning, and enrich the exploration of video understanding in industrial logistics scenarios. We hope these additional analyses clarify the practical value of our benchmark, and we are pleased to further discuss and resolve any further concerns.

We sincerely thank you again for your valuable time and thoughtful feedback. During the rebuttal period, we have comprehensively addressed your comments, and hope that our responses have clarified the key aspects. We would be pleased to provide additional clarifications if needed.

Thank you for your response. My concern has been solved and I will maintain my original score. Good Luck.

AuthorsOpenReview ↗

Thank You for Your Positive Feedback

We are pleased that the rebuttal response has effectively addressed your concerns. The relevant discussion has been included in the revised manuscript to enhance clarity and facilitate readers’ understanding. We sincerely appreciate your valuable time and positive feedback.

Reviewer_e5b6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LogiScope-VQA, a benchmark for evaluating vision-language models on logistics safety tasks. It contains 10,274 question-answer pairs based on 2,476 images and 2,918 videos, mainly collected from real warehouse environments. The tasks cover industrial element perception, warehouse knowledge understanding, and potential risk reasoning. The authors evaluate 20 models and compare their performance with human participants. They also study the effects of thinking budgets, risk judgment bias, and the reliability of an LLM-based evaluator. The reported results suggest that fine-grained perception and reliable hazard identification remain challenging for current models.

Critical strengths
  1. The benchmark addresses a useful problem with data that is difficult to obtain. Real warehouse surveillance contains small objects, occlusion, and complex interactions between workers and equipment. These conditions are relevant to practical safety monitoring and provide a useful evaluation setting beyond common natural-image benchmarks.

  2. The benchmark covers several related capabilities, including perception, domain knowledge, and risk reasoning. The use of domain experts for annotation is also a positive aspect. With a reliable evaluation protocol and a complete data release, this resource could help researchers identify specific weaknesses of models in industrial scenarios.

Critical weaknesses
  1. The visual input protocol is unclear for some evaluated models.

Appendix C.2 states that all models directly receive an ordered sequence of 2–10 images. However, Table 1 includes GLM-4.7, whose official documentation lists text as its input modality (https://docs.z.ai/guides/llm/glm-4.7). It is therefore unclear how this model processes the visual input in the reported experiments. If an additional vision model, image captioning step, or tool is used, this component should be described and considered in the comparison. If the model name is incorrect, the exact endpoint or checkpoint should be provided. This issue matters because the paper uses these results to draw conclusions about visual perception and differences between open-source and proprietary models.

  1. Several reported results and evaluation details are inconsistent.

First, Table 3 reports an accuracy of 74.4% with Cohen's kappa of 0.81 for Phi-4-Mini, and 87.2% with kappa of 0.89 for Claude-Opus-4.7. If accuracy is the agreement with human judgments on the same binary decisions, standard Cohen's kappa cannot exceed this agreement rate. The calculation or the meaning of these metrics needs clarification.

Second, the thinking results in Table 2 are difficult to reconcile with Table 1, which is also reported to use thinking mode. For example, MiniMax-M2.1 obtains 0.47, 0.49, and 0.54 on the three perception subtasks in Table 1, but only 0.10 for the perception category in Table 2. These values cannot be explained by a weighted average of the same subtask results.

Finally, Section 3.1 describes three expert participants, while Appendix C.3 describes only one expert. The descriptions of how questions are assigned to students also differ. Together, these issues reduce my confidence in the reported model comparisons, judge validation, and human baseline.

Questions for authors

The following questions are clarification requests that could affect my recommendation.

  1. How are visual inputs processed by GLM-4.7 and the other text-generation models included in Table 1? Please provide the exact model identifiers, any additional visual processing components, and an example input request.

  2. How are accuracy and Cohen's kappa calculated in Table 3? Do they use the same samples and human reference labels? Please provide the number of evaluated judgments and the corresponding confusion matrices.

  3. Do Tables 1 and 2 use the same questions, inference settings, and aggregation method? If not, please explain the differences. Please also clarify the actual number of human experts, how questions were assigned to participants, and whether the evaluation experts were involved in creating the ground-truth answers.

7 comments
AuthorsOpenReview ↗

Response to Reviewer e5b6 (1/3)

Dear Reviewer e5b6,

Thanks for your holistic comments, and we have provided detailed responses as follows.

[W1 & Q1]: Additional evaluation details for the incorporated LLMs.

Response:

a. To clarify, we employ visual assistant models from the same model family to support the evaluation of LLMs in industrial logistics scenarios, as reported in Table 1. The exact model identifiers are listed below.

Final-answer LLMVisual assistant
GLM-4.7, GLM-5.2GLM-4.6V
MiniMax-M2.1, MiniMax-M2.5MiniMax-VL-01
Kimi-K2-ThinkingKimi-VL-A3B-Thinking-2506

b. The LLM-plus-visual-assistant evaluation pipeline is implemented in strict accordance with the official documentation [1-4] on deploying LLMs in multimodal settings. Specifically, for each sample, we first assign identifiers I₁, I₂, ..., Iₙ to the 2-10 input images to preserve their original order. Next, we submit the ordered images together with the question, all answer options, and a shared visual-evidence prompt in one visual-analysis request. Then the visual assistant performs the joint multi-image analysis and returns textual visual evidence, including per-image evidence, cross-image relationships, and explicit uncertainties. Finally, the final-answer LLM performs reasoning and answers the question based on this evidence and the original task context. Its final answer is processed and evaluated using the same metrics as the answers of the other MLLMs. We have added the pipeline description and prompt templates to the Appendix C.3 (Page 20, Line 1030-1079).

c. Such design broadens benchmark coverage and reflects practical deployments that combine specialized perception models with general-purpose LLMs. Using a fixed visual assistant within each model family also supports more controlled comparisons across LLM versions. Accordingly, this paper measures the industrial hazard identification performance of the combined systems, encompassing both the visual assistants’ perceptual capabilities and the LLMs’ reasoning capabilities.

References:

[1] GLM multi-image input example

[2] GLM-4.6V visual-input documentation

[3] MiniMax text-generation API documentation

[4] Kimi-VL vLLM serving guide

[W2.1 & Q2]: Clarification about the computation details of accuracy and Cohen's kappa.

Response:

a. Importantly, the unexpected trend between accuracy and Cohen's kappa observed in Table 3 is due to their different computation levels, rather than calculation errors. As stated in Appendix C.1 (Page 17, Line 783-785), the ground truth for an open-ended question contains multiple key points, and a response is considered correct only if all key points are identified correctly and none is omitted. Accuracy is therefore computed at the question level, consistent with Table 1, whereas Cohen's kappa is computed at the key-point level.

AuthorsOpenReview ↗

Response to Reviewer e5b6 (2/3)

b. Additionally,we provide the complete experimental workflow and calculation formulas for reference. First, we randomly select 100 open-ended questions and use the VQA responses of four models: Qwen3.7-Plus, Gemini-3.1-Pro, InternVL3.5, and Kimi-K2.6, yielding 400 evaluated responses. Three judge models, Phi-4-Mini, DeepSeek-V4-Pro, and Claude-Opus-4.7, and two human judges independently judge the correctness of every key point in these responses. When a judge-model call fails, we exclude the affected response and report the number of valid samples as NvalidN_{\mathrm{valid}}. Human-human comparisons contain no failed calls. Second, we compare each judge model's judgments with those of the human judges. Table 3 reports the average of the comparisons across the four model-response sets. Third and most importantly, the three metrics are computed at different levels. Accuracy and F1 are computed at the question level from the confusion-matrix counts TPqTP_{q}, TNqTN_{q}, FPqFP_{q}, and FNqFN_{q}:

Acc=(TPq+TNq)/NvalidAcc = (TP_{q} + TN_{q}) / N_{valid}

F1=2∗TPq/(2∗TPq+FPq+FNq)F1 = 2 * TP_{q} / (2 * TP_{q} + FP_{q} + FN_{q})

To provide a finer-grained assessment, Cohen's kappa is computed at the key-point level to assess whether a judge model identifies the same specific omissions or errors as the human judges. Using the key-point-level confusion-matrix counts TPkTP_{k}, TNkTN_{k}, FPkFP_{k}, and FNkFN_{k}, where Mvalid=TPk+TNk+FPk+FNkM_{\mathrm{valid}} = TP_{k} + TN_{k} + FP_{k} + FN_{k}, we compute

po=TPk+TNkMvalid\displaystyle p_o = \frac{TP_k + TN_k}{M_{valid}}

pe=(TPk+FPk)×(TPk+FNk)+(TNk+FPk)×(TNk+FNk)Mvalid2\displaystyle p_e = \frac{(TP_k + FP_k) \times (TP_k + FN_k) + (TN_k + FP_k) \times (TN_k + FN_k)}{M_{valid}^{2}}

Cohen’s kappa=po−pe1−pe\displaystyle \text{Cohen's kappa} = \frac{p_o - p_e}{1 - p_e}

c. Following your helpful suggestions, we provide detailed confusion matrices used to calculate the accuracy and Cohen’s kappa scores, ensuring that these metric calculations are fully reproducible. This further strengthens the rigor and reliability of LLM-as-a-Judge evaluation in Table 3.

Question-level confusion matrix for accuracy calculation:

Judge SetupModelN_validTN_qTP_qFN_qFP_qAcc. (%)F1 (%)
Phi-4-MiniQwen3.7-Plus99334314976.7778.90
Gemini-3.1-Pro98294317973.4776.79
InternVL3.51003141151372.0074.55
Kimi-K2.698304415975.5178.57
Average—————74.4477.20
DeepSeek-V4-ProQwen3.7-Plus100414216183.0083.17
Gemini-3.1-Pro99364614382.8384.40
InternVL3.599394016479.8080.00
Kimi-K2.6100364416480.0081.48
Average—————81.4182.26
Claude-Opus-4.7Qwen3.7-Plus99315441085.8688.52
Gemini-3.1-Pro10031583889.0091.34
InternVL3.510036515887.0088.70
Kimi-K2.699295721186.8789.76
Average—————87.1889.58
Human-HumanQwen3.7-Plus10040553295.0095.65
Gemini-3.1-Pro10038529190.0091.23
InternVL3.510043515194.0094.44
Kimi-K2.610039528191.0092.04
Average—————92.5093.34

Key-point-level confusion matrix for Cohen's kappa calculation:

Judge SetupModelM_validTN_kTP_kFN_kFP_kCohen's kappa
Phi-4-MiniQwen3.7-Plus4849536115130.8342
Gemini-3.1-Pro4788835820120.8035
InternVL3.54909735520180.7854
Kimi-K2.64799035917130.8171
Average—————0.8100
DeepSeek-V4-ProQwen3.7-Plus4901013602270.8360
Gemini-3.1-Pro483943641870.8495
InternVL3.548310234722120.8105
Kimi-K2.6490953652190.8243
Average—————0.8301
Claude-Opus-4.7Qwen3.7-Plus484903747130.8740
Gemini-3.1-Pro49093383680.9121
InternVL3.549010436510110.8803
Kimi-K2.6485923765120.8934
Average—————0.8899
Human-HumanQwen3.7-Plus490102375760.9231
Gemini-3.1-Pro490973761340.8974
InternVL3.5490111369640.9435
Kimi-K2.64901003761040.9163
Average—————0.9201

Thank you for your perceptive observation, we have consolidated the detailed description of the LLM-as-a-Judge evaluation procedure and associated calculation details in the newly added Appendix C.5 (Page 21, Lines 1094–1129) to facilitate readers’ precise understanding.

AuthorsOpenReview ↗

Response to Reviewer e5b6 (3/3)

[W2.2 & Q3]: Updated thinking-mode metrics in Table 2.

Response:

(1) Thank you for your kind reminder. For MiniMax-M2.1 on the perception task, the correct values under non-thinking and thinking modes are 0.54 and 0.50, respectively. We sincerely apologize for this oversight in reporting the metrics. During the rebuttal period, we re-verified all experimental results to confirm that the entries in Table 2 (page 7, line 331) are accurate.

(2) To further clarify, Table 1 and the thinking setting in Table 2 utilize identical evaluation configurations, including the prompt template, temperature, thinking mode and maximum number of new tokens. Table 2 differs in that the evaluations are conducted on a randomly sampled subset of the data, and results under the non-thinking setting are also assessed for comparison.

[W2.3 & Q3]: Inconsistency of recruited human expert numbers.

Response: Thank you for bringing this to our attention. Given the substantial effort and resources required for data collection and benchmark construction, the manuscript underwent multiple revisions, during which this minor error was inadvertently introduced. To clarify, three experts each completed all questions, and their mean score was utilized as the logistics expert performance reference. The main manuscript is correct, and we have corrected Appendix C.4 (Page 21, Line 1088) accordingly.

(i) For the novice baseline, we recruited 10 paid college students with no background in industrial logistics. The 10,274 questions were randomly assigned among them, so each student answered approximately 1,000 questions.

(ii) For the expert baseline, we recruited three paid logistics-safety experts, each with more than three years of experience in logistics security. All participants independently answered their assigned questions without any external assistance.

(iii) Additionally, the experts responsible for ground-truth annotation and those participating in the human-baseline evaluation constituted two entirely non-overlapping groups. This aims to mitigate bias from prior exposure to reference answers and enhance the independence and validity of human performance assessment.

Summary: We sincerely thank you again for your thorough review comments. We have provided holistic clarifications regarding the visual input protocol, judge validation metrics, and logistics expert involvement. Furthermore, we publicly commit on OpenReview to open-sourcing all hazard identification data and maintaining a continuously updated leaderboard to overcome the substantial data barriers to LLM evaluation in industrial scenarios. We would also be very pleased to engage in further discussion and address any additional questions.

Reviewer_boJLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper provides a solid and (mostly) real-world VQA benchmark on logistics hazard identification in industrial scenarios. Data comes from internal corporate monitoring and is licensed under CC BY-NC-SA.

Critical strengths

S1: Solid resouces

This paper obtains 5,394 visual samples and 10,274 VQAs from 3.5 million surveillance clips with expert annotation and cross-validation. The overall project volume was large. This benchmark constructino pipeline is clear and reasonable. Compared to simulated resources like IndustryEQA, LogiScope-VQA uses real-world data, thus potentially being more valuable.

S2: Informative analysis

The analysis of thinking budget and risk bias provides insights of the capability of current LMMs in logistics hazard identification.

Critical weaknesses

W1: Generalizability concerns

  • While being a solid benchmark, the majority of data is collected from one single logistics company, potentially compromising the generalizability.
  • More than 20% data is generated or editted, which might introduce style biases. Marginal performance disparity alone can't validate the authenticity and quality of synthetic data.
Questions for authors

W1: Can you provide more analysis of synthetic data. What tasks use synthetic data? How to assess the quality and authenticity of the synthetic data?

4 comments
AuthorsOpenReview ↗

Response to Reviewer boJL (1/2)

Dear Reviewer boJL:

We are sincerely grateful for the positive assessment of our work and the constructive feedback. Herein, we provide thorough clarifications to enhance comprehension of this research work.

[W1.1]: Holistic discussion of the generalizability of the collected database

Response:

a. Due to restrictive corporate security policies and commercial confidentiality considerations, collecting and releasing cross-company logistics safety dataset is largely infeasible. Indeed, the vast majority of logistics companies currently keep their internal safety risk data private to safeguard future business prospects. One of the primary motivations for this work is to break down severe data barriers between logistics companies. To this end, we will make the industrial benchmark freely open-source to facilitate cross-company dataset construction and LLM advancement in industrial logistics.

b. The constructed LogiScope-VQA exhibits substantial data diversity, spanning broad spatiotemporal contexts as well as heterogeneous modalities, core objects, risk types, and task formats. As a global leader in smart logistics, Cainiao company offers highly diverse, real-world warehousing environments for data collection through its worldwide operations and logistics network. The database is collected across more than 20 countries and regions from over 40,000 surveillance camera feeds, yielding over 3.5 million raw clips within one year. After rigorous filtering, 5,394 high-quality visual samples are retained, covering over 20 operational scenarios.

c. To further enhance generalizability, we deliberately incorporate synthetic data during benchmark construction to systematically diversify logistics risk factors. Concretely, the image-editing model is introduced to expand a broad range of common logistics hazards while preserving the authentic industrial context of the original images. These controlled variations broaden the benchmark’s coverage of risk scenarios, supporting generalizable evaluation of LLM-based hazard identification. (See the subsequent response for in-depth analyses of synthetic data quality.)

[W1.2 & Q1]: In-depth exploration into the quality of synthetic data

Response:

Thank you for this valuable comment. We address your concerns regarding the authenticity, reliability, and potential style bias of our synthetic data from five aspects: (a) blind visual discrimination, (b) controlled editing evaluation, (c) performance consistency, (d) limited data proportion, and (e) strict quality control.

(a) To evaluate visual authenticity, we have conducted a blind discrimination experiment, which reveals that our synthetic data exhibits convincing realism and poses challenges to humans, general-purpose LMMs, and specialized detectors. Humans and LMMs show limited discrimination capability, while the specialized detector (AIDE) still misclassifies most synthetic images as real.

Specifically, we sampled 100 synthetic and 100 real images (excluding original–synthetic counterpart pairs), and evaluated both full images and random crops. Three human volunteers, two general-purpose LMMs (Qwen3.5-Plus and GPT-5.5), and one specialized synthetic-image discriminator (AIDE) independently classified each sample as real or synthetic based solely on visual content. We computed RRR_R, the proportion of real images classified as real; RSR_S, the proportion of synthetic images classified as real; and discriminability sensitivity d′d'. In the equations, X∈{R,S}X\in\{R,S\}, NXN_X is the number of images from source XX, NX→RN_{X\rightarrow R} is the number labeled real, and Φ−1\Phi^{-1} is the inverse standard normal CDF. d′d' measures perceptual discriminability: larger positive values indicate easier separation, whereas values near zero indicate visual indistinguishability.

$\displaystyle R_X = \frac{N_{X\rightarrow R}}{N_X}, \quad \widetilde R_X = \frac{N_{X\rightarrow R}+0.5}{N_X+1}, \quad X \in \lbrace R,S \rbrace.$

d′=Φ−1(R~R)−Φ−1(R~S).d' = \Phi^{-1}(\widetilde R_R) - \Phi^{-1}(\widetilde R_S).

The detailed results are presented below:

JudgeFull Image RRR_R (%) ↑Full Image RSR_S (%) ↑Full Image d′d' (→ 0)Random Crop RRR_R (%) ↑Random Crop RSR_S (%) ↑Random Crop d′d' (→ 0)
Volunteer 175.071.0+0.1278.076.0+0.06
Volunteer 271.068.0+0.0874.075.0-0.03
Volunteer 380.071.0+0.2882.077.0+0.17
Human Mean75.370.0+0.1678.076.0+0.07
Qwen3.5-Plus76.071.0+0.1579.075.0+0.13
GPT-5.579.072.0+0.2281.077.0+0.14
AIDE95.064.0+1.2594.066.0+1.11
AuthorsOpenReview ↗

Response to Reviewer boJL (2/2)

(b) Our controlled editing experiment shows that models generally respond correctly to relevant edits while preserving unaffected answers, mitigating concerns about disruptive style bias.

Specifically, we evaluated 200 original–edited image pairs using GPT-5.5, Qwen3.7-Plus, and Kimi-K2.6. Each pair was bound with two verified questions: AA, whose answer changes after editing, and BB, whose answer remains unchanged. Each question was evaluated independently on both images, yielding 800 VQA instances per model. We report three metrics: original-image accuracy computed over 400 questions, update rate, and retention rate. For each model, update and retention rates are computed on the subset S\mathcal S where both original-image answers are correct, with M=∣S∣M=|\mathcal S|:

Update Rate=1M∑i∈S1[y^i,AE=yi,AE].\text{Update Rate} = \frac{1}{M}\sum_{i\in\mathcal S} \mathbf{1}[\hat y_{i,A}^{E}=y_{i,A}^{E}].

Retention Rate=1M∑i∈S1[y^i,BE=yi,BO].\text{Retention Rate} = \frac{1}{M}\sum_{i\in\mathcal S} \mathbf{1}[\hat y_{i,B}^{E}=y_{i,B}^{O}].

Here, y^\hat y and yy denote predicted and ground-truth answers, and O/EO/E denote original/edited images. The results are shown below:

ModelAccAcc (%)MMUpdate RateUpdate~Rate (%) ↑\uparrowRetention RateRetention~Rate (%) ↑\uparrow
GPT-5.570.510579.094.3
Qwen3.7-Plus65.39176.993.4
Kimi-K2.660.07774.090.9

(c) As also shown in Appendix C.5, model performance remains consistent between synthetic and real data, further supporting the fidelity of our synthetic data from a performance-consistency perspective.

(d) Questions involving synthetic data account for only 7.65% of our benchmark, enriching rare scenarios with minimal impact on the overall evaluation. The distribution of these 786 questions across tasks is as follows:

Industrial Element PerceptionWarehouse Knowledge UnderstandingPotential Risk ReasoningTotal
432242112786

(e) Synthetic data is constructed under strict quality control. All synthetic images and video frames are edited from real samples, and subsequent filtering rigorously removes any instances exhibiting obvious artifacts, physical implausibility, or noticeable style inconsistencies.

Summary: Thank you again for your insightful comments. The additional analytical experiments provide further evidence supporting the generalizability of our benchmark and the quality of the synthetic data. We hope that our responses could satisfactorily address your previous concerns, and we would be delighted to provide any further clarification you may need.

Thank you for your response. The revision addresses most of my concerns, and I maintain my weak accept rating.

AuthorsOpenReview ↗

Thank You for Your Positive Assessment

We are very pleased to hear that your initial concerns have been addressed. We have meticulously incorporated the relevant clarifications from our discussions into the revised manuscript to improve its clarity and precision. Thank you again for your positive assessment and valuable suggestions.

Comments on the paper

2 comments
Area_Chair_jfLsOpenReview ↗

Please engage with the authors rebuttal by Nov 18

Dear Reviewers,

The authors have posted their rebuttal. Please engage with it by November 18, 2026, noting which of your concerns are resolved or which remain. Even brief engagement helps greatly.

Best regards,
Area Chair

Dear Area Chair,

We would like to express our sincere gratitude to you and all reviewers for the valuable comments and thorough feedback. Herein, we briefly summarize the core contributions, rebuttal summary, and practical value for reference.

Key Contributions:

a. Scarce and authentic industrial data. All three reviewers appreciate the scarcity and practical value of our real-world industrial logistics safety data, mirroring the visual challenges of practical warehouse monitoring.

Reviewer boJL: "Solid resources. LogiScope-VQA uses real-world data and is more valuable." Reviewer e5b6: "The benchmark addresses a useful problem with data that is difficult to obtain." Reviewer 66Pt: "Representative of the visual challenges in warehouse safety monitoring."

b. Substantial effort and rigorous construction. All three reviewers acknowledge the substantial project effort and our rigorous construction pipeline, from massive data curation to multi-round expert annotation and validation.

Reviewer boJL: "The overall project volume was large and construction pipeline is clear and reasonable." Reviewer e5b6: "Use of domain experts for annotation is a positive aspect." Reviewer 66Pt: "Domain experts in multiple rounds of annotation and validation."

c. Multi-level and diagnostic evaluation. Reviewers e5b6 and 66Pt commend that the multi-level capability coverage and diverse question formats effectively diagnose specific model weaknesses.

Reviewer e5b6: "With a reliable evaluation protocol, this resource could identify specific weaknesses of models in industrial scenarios." Reviewer 66Pt: "The benchmark covers multiple levels of capability and combines single-choice, multiple-choice, and open-ended questions."

Rebuttal Summary:

(i) Reviewer boJL requests clarification on the dataset generalizability and synthetic data quality. Accordingly, we present the dataset's broad spatiotemporal coverage, collecting from 3.5 million raw clips, 20+ countries, 40,000+ cameras. The blind visual discrimination and controlled editing evaluations effectively confirm the high realism and negligible style bias of our synthetic data.

(ii) Reviewer e5b6 mainly seeks clarification on the visual input protocol for LLMs and reported metrics in LLM-as-a-Judge reliability. Therefore, we detail the LLM-plus-visual-assistant evaluation pipeline and provide complete confusion matrices clarifying the question-level and key-point-level metric computation. The follow-up discussion further incorporates ranking stability and self-preference bias analyses to consolidate judge reliability. Moreover, we thoroughly proofread the manuscript and rectify all writing issues.

(iii) Reviewer 66Pt's main concern is temporal-reasoning ablations. To address this, our six-setting video input ablation study highlights LogiScope-VQA's strength in evaluating capabilities ranging from single-frame understanding to multi-frame information integration and temporal reasoning.

Overall, the initial questions regarding evaluation pipeline, synthetic data quality and temporal ablations have been resolved through our extensive experiments and clarifications. Upon conclusion of the rebuttal phase, all three reviewers indicate that their primary concerns have been addressed. We are pleased that all three reviewers consistently provide positive recommendations regarding the acceptance of our paper.

Practical Value:

Currently, the vast majority of logistics companies keep their internal safety risk data confidential to safeguard future commercial prospects. The scarcity of logistics safety benchmarks hinders the holistic evaluation and practical deployment of MLLMs in industrial settings. A primary motivation for this work is to overcome the severe data barriers, accomplishing rigorous evaluation of MLLMs' ability to identify hazards in real-world industrial environments. This work introduces a comprehensive MLLM benchmark specifically designed for hazard identification in industrial logistics, covering 18 core objects, 20 risk types, 39 subtasks. Upon acceptance of the paper, we will release the benchmark as a publicly and freely available dataset to support further advances in MLLMs for industrial logistics.

Thank you again for your valuable time, dedicated effort and thoughtful consideration of our work.

Sincerely,

Authors of ICLR Submission 1337

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.