- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
No generative AI was used for the review of that paper. The paper has not been analyzed by an LLM, the review has not been passed through an LLM. This is in order to keep the submission truly confidential and reflect the opinion of the reviewer.
This paper introduces the Region Detection task and a dataset targeting this task called RegionDet.
The region detection task targets visual scenes concepts (visual states from the authors' words) that cannot be as clearly defined as well-bounded objects of traditional object detection. Specifically, RegionDet contains 8 categories: "construction", "crossing", "damage", "queuing", "talking", "vendor", "waiting", and "walking".
The paper further evaluates closed-set and open-vocabulary detectors on RegionDet, revealing a gap in generalization for both approaches. Visual states detection is hard and that closed-set model do not adapt that well, even after training.
Critical strengths
- The idea of concept localization (visual states, from the paper), is interesting. Although it shows some resemblance with "From Captions to Visual Concepts and Back" Fang et al., CVPR 2015 and other visual scene graph understanding works, the categories defined in this submission are different and also involve multiple entities at once.
- Multiple annotations categories can co-exist in a single image, increasing the diversity in the samples.
- Based on the information in the paper, the annotations of the dataset seem to have been carefully crafted, without AI intervention and with multiple labeling rounds, suggesting high quality.
- The paper mentioned the use of the official repository of all approaches, which is good for reproducibility and enhances the trust in the reported results. Overall, the amount of approaches evaluated is greatly appreciated and provides a comprehensive landscape of models' capabilities.
- Reporting results at different AP levels is important and instructive as it shows that closed-set model do have some understanding of these categories, but struggle to accurately locate them on the image.
- The paper includes a limitation section.
Critical weaknesses
I split this section in critical weaknesses and minor weaknesses.
I have substantial concerns in the paper and am unsure that they can be fixed during review since, in my opinion, the major weaknesses greatly overweight the strengths of the paper. Nonetheless, if authors would like to engage in reviewing and discussions, I will be happy to help them improve the paper as much as possible, based on my review.
Critical weaknesses
Philosophical concerns and Region Detection task format
- Philosophical justification of the task. The most important philosophical concern that I have relates to the choice of using bounding boxes as representation of visual states. The name "state" itself, greatly hints towards a finegrained representation, such as edges of scene graphs. Authors also mention rightfully that visual states that visual states are "scene context, object relations, and human activities" (L. 14-15). All which are appropriate graph representations. In that aspect, the related works section would benefit from a dedicated paragraph discussing visual scene graph understanding approaches and datasets, as they already study visual states. Useful references involve, but are not limited to: "Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations", Krishna et al., IJCV 2017 "What are they doing? : Collective Activity Classification Using Spatio-Temporal Relationship Among People." W. Choi et al., ICCV 2009. Could authors comment on what are the advantages of that bounding box formulation of scene graph? And incorporate these changes in the related work section?
- Bounding box vs. Scene graph. Following 1., I am not sure how the bounding box formulation is a better task format than scene graph itself. The bounding box format can be recovered from the scene graph format since scene graph models usually already detect objects. In that case, the union of the boxes of objects (e.g. humans) sharing a common "waiting" edge with another stuff segment (e.g. road) is sufficient to recover the full bounding box of the visual state. It feels to me like the granularity chosen for the task is potentially improper. Could authors comment and give examples of where their format is better than scene graph understanding? Could authors benchmark state-of-the-art scene graph understanding networks too?
- Downstream tasks. Naturally stemming from 1. and 2., it is unclear what the actual applications are for this task. Clearly stating downstream use cases would strengthen the paper's contributions. Could authors clearly explain downstream application of this task and dataset?
Dataset and experimental concerns
- Important missing information. There are in my opinion important missing information related to the proposed RegionDet dataset. Since images are collected from public web sources, clearly stating the licenses and sources is extremely important. Then, it is not clear what rationale has been used to select the classes in the first place. In the COCO dataset paper, authors clearly mention their strategy for class selection, it would strengthen the paper to clearly discuss it. It is also important to state the how the collectors were selected (professional? Not professional? Were they hired? Were they volunteers? Did they have incentives to properly annotate?, etc.). The dataset would also benefit from person de-identification, if possible. Otherwise, that would be appreciated at least when showing qualitative results, in the manuscript. It especially feels strange that "vendor" would be a visual state, since it is more of a well defined region. Could authors please add all these information to the manuscript? Could authors clarify how and why they chose these classes initially?
- Open-sourcing the code. While I do understand that the code and dataset remain private during the review phase, there is no mention of whether the dataset and the scripts for the experiments will be made open-sourced after potential acceptance. Can authors clarify what the status of the code and dataset will be?
- Questionable open-set evaluation protocol. Open-set detectors are evaluated in zero-shot. This setting is interesting, but to clearly compare with the closed-set models, it would be important to also finetune open-set detectors on the training split of RegionDet to see if they benefit from this training and can adapt to the data. Could authors please provide experimental results under this setting too?
- Lack of implementation details. Please specify the link of the official implementation used for each approach, and the default settings used (LR, batch size, optimizer, etc.). This can be done in the supplementary materials and referred in "implementation details" (L. 318).
- Improving performances. Generally speaking, authors do not investigate performance improvement using this dataset or by developing a method related to it. This is usually expected from papers introducing datasets. This would also better justify data collection for Region Detection.
Miscellaneous
- References. There are many missing references which impede reading. I initially thought that DINO referred to "Emerging Properties in Self-Supervised Transformers" (M. Caron et al. 2021, ICCV), but it actually comes from "DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection" (H. Zhang et al., ICLR 2023). I found baselines not be properly referenced. Could authors clearly indicate the reference of the models next to their table entries and cite them accordingly?
- Classification vs. Localization. I would be curious to know whether the VLMs can correctly identify the category independently of the location in the image. Perhaps they are good at understanding that there is "damage" but simply not good at localizing it. For each sample in the dataset, authors could construct a "yes"/"no" binary classification experiment asking whether there is the category on the sample. That would give a rough idea of whether VLMs are simply missing the localization of the concept or simply not realizing that the concept is present in the image in the first place. Could authors please share the results of this experiment?
Minor weaknesses
- Clarifying the difference between Region Detection and RegionDet. Although clarified after the method section, the sentence "Region Detection is a benchmark setting" (L. 18) in the abstract can confuse readers.
- ICLR Format. References should be in parentheses. Please refer to the official template.
- Highlighting results. Result tables are quite blunt, it would be appreciated if authors could bold important/best performing numbers, or highlight particular outcomes in the tables directly.
- Title and description of table 6. I do not understand what experiment has been conducted on table 6. The caption is not enough to clearly understand the setting and there is only one occurrence of that table in the manuscript, line 391, which also does not explain the experiment that much. Could author please better describe it?
- Prompt templates. Please make a small table containing the exact prompts you use for each prompting setting. You can use "{category}" to indicate the category variable inside the prompt.
- Class imbalance. Authors mention that "construction", "damage" and "vendor" classes have greater appearance in scene variation (L. 241), but the justification is not clear. When running the numbers, these 3 classes make up 80% of the training data of RegionDet. Could authors kindly re-explain that choice?
- Review 2nd key finding claims. The second key finding is not sharp enough. While closed-set detectors are trained on the dataset (L.252), zero-shot ones are not (L. 255). At L. 453, authors claim "Second, performance cannot be explained by detection paradigm alone". This is true in what is shown, but without retraining the open-set detectors on the RegionDet dataset, this claim feels very weak.
- Quality control. Since there has been annotator cross-checking, and since the annotated regions are ambiguous it would be interesting if the paper used the weighted disagreement rate metric for evaluation, which also take into account the difficulty of the annotation for human reviewers. It has been introduced in that paper: "Intrinsic Images in the Wild", Bell et al., SIGGRAPH 2014. and an example usage, also in scene graph understanding, has been used in: "Holistic Order Prediction in Natural Scenes", Musacchio et al., NeurIPS 2025. I understand that this might be complicated to achieve since the metadata of the annotations might not have been tracked. So this criticism should be just taken as a suggestion for improvement.
- Visualizations. Visualizations of attention map or gradcams from the model might be interesting to understand the importance of regions in their representations.
- Circular definitions. Technically speaking all the definitions in table 2 are circular. I would invite authors to adjust them.
Questions for authors
Please refer to the weaknesses section.
In the current state of the paper, the weaknesses greatly outweigh the strengths of the paper.
I also believe that the current format and writing of the manuscript is not satisfactory for acceptance at a top tier conference such as ICLR.
Improving my score would require clearly assessing all my concerns.
Nonetheless, I am greatly worried about the task format itself and unless authors can simply and clearly convince that there is an advantage over scene graph, I do not think I will improve my score.
2 comments
Dear Reviewer,
Thank you for your response and for continuing to follow up on our manuscript as the discussion period draws to a close.
Given the low initial ratings, our concurrent rebuttal commitments for other submissions, and our limited time and capacity, we decided not to pursue a rebuttal for this manuscript in this round. We will carefully consider all reviewers’ comments, further improve this work, and submit it to another conference after thorough revision. Thank you for your advice and encouragement.
Even though we did not submit a rebuttal, you still took the initiative to follow up and explain your final assessment. We sincerely appreciate your conscientious and responsible approach to the review process. Thank you for your time and effort.
Final remarks before end of the rebuttal period.
Dear authors, dear AC,
Since the authors have not produced any rebuttal and since the rebuttal period will end soon, I will maintain my initial score. I believe that in the current state, this manuscript should be rejected. Nonetheless, I encourage the authors to improve their manuscript and submit to another conference.
Thank you.