acceptodds
Under review as a conference paper at ICLR 2027

RegionDet: A Benchmark for Region Detection Beyond Object Instances

Abstract

Object detection is a fundamental task in computer vision and has achieved remarkable progress on standard benchmarks by localizing discrete and well-bounded object instances. However, many visual targets in real-world scenarios are not individual objects, but regions defined by visual states, scene context, object relations, and human activities, such as construction areas, damaged road regions, queues, group conversations, and vendor regions. Existing detection benchmarks are mainly built around object instances, providing limited support for systematically evaluating such region targets. To address this gap, we introduce Region Detection, a benchmark setting that extends conventional object detection beyond object instances, and construct RegionDet, a benchmark for region target localization. RegionDet contains eight region categories, including Construction, Crossing, Damage, Queuing, Talking, Vendor, Waiting, and Walking, with COCO-style bounding-box annotations and evaluation protocols. We evaluate representative closed-set and zero-shot/open-vocabulary detectors on RegionDet. Results show that closed-set detectors can partially learn region-level patterns under supervision, while zero-shot/open-vocabulary detectors remain limited even with region-oriented textual prompts, suggesting that current vision-language detectors still struggle to generalize to region targets. Further analyses highlight key challenges in Region Detection, including weak boundary cues, strong context dependency, and insufficient relation-level region understanding.

Primary area:
datasets and benchmarks
Keywords:
Region Detection, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zQASOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

No generative AI was used for the review of that paper. The paper has not been analyzed by an LLM, the review has not been passed through an LLM. This is in order to keep the submission truly confidential and reflect the opinion of the reviewer.

This paper introduces the Region Detection task and a dataset targeting this task called RegionDet.

The region detection task targets visual scenes concepts (visual states from the authors' words) that cannot be as clearly defined as well-bounded objects of traditional object detection. Specifically, RegionDet contains 8 categories: "construction", "crossing", "damage", "queuing", "talking", "vendor", "waiting", and "walking".

The paper further evaluates closed-set and open-vocabulary detectors on RegionDet, revealing a gap in generalization for both approaches. Visual states detection is hard and that closed-set model do not adapt that well, even after training.

Critical strengths
  • The idea of concept localization (visual states, from the paper), is interesting. Although it shows some resemblance with "From Captions to Visual Concepts and Back" Fang et al., CVPR 2015 and other visual scene graph understanding works, the categories defined in this submission are different and also involve multiple entities at once.
  • Multiple annotations categories can co-exist in a single image, increasing the diversity in the samples.
  • Based on the information in the paper, the annotations of the dataset seem to have been carefully crafted, without AI intervention and with multiple labeling rounds, suggesting high quality.
  • The paper mentioned the use of the official repository of all approaches, which is good for reproducibility and enhances the trust in the reported results. Overall, the amount of approaches evaluated is greatly appreciated and provides a comprehensive landscape of models' capabilities.
  • Reporting results at different AP levels is important and instructive as it shows that closed-set model do have some understanding of these categories, but struggle to accurately locate them on the image.
  • The paper includes a limitation section.
Critical weaknesses

I split this section in critical weaknesses and minor weaknesses.

I have substantial concerns in the paper and am unsure that they can be fixed during review since, in my opinion, the major weaknesses greatly overweight the strengths of the paper. Nonetheless, if authors would like to engage in reviewing and discussions, I will be happy to help them improve the paper as much as possible, based on my review.

Critical weaknesses

Philosophical concerns and Region Detection task format

  1. Philosophical justification of the task. The most important philosophical concern that I have relates to the choice of using bounding boxes as representation of visual states. The name "state" itself, greatly hints towards a finegrained representation, such as edges of scene graphs. Authors also mention rightfully that visual states that visual states are "scene context, object relations, and human activities" (L. 14-15). All which are appropriate graph representations. In that aspect, the related works section would benefit from a dedicated paragraph discussing visual scene graph understanding approaches and datasets, as they already study visual states. Useful references involve, but are not limited to: "Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations", Krishna et al., IJCV 2017 "What are they doing? : Collective Activity Classification Using Spatio-Temporal Relationship Among People." W. Choi et al., ICCV 2009. Could authors comment on what are the advantages of that bounding box formulation of scene graph? And incorporate these changes in the related work section?
  2. Bounding box vs. Scene graph. Following 1., I am not sure how the bounding box formulation is a better task format than scene graph itself. The bounding box format can be recovered from the scene graph format since scene graph models usually already detect objects. In that case, the union of the boxes of objects (e.g. humans) sharing a common "waiting" edge with another stuff segment (e.g. road) is sufficient to recover the full bounding box of the visual state. It feels to me like the granularity chosen for the task is potentially improper. Could authors comment and give examples of where their format is better than scene graph understanding? Could authors benchmark state-of-the-art scene graph understanding networks too?
  3. Downstream tasks. Naturally stemming from 1. and 2., it is unclear what the actual applications are for this task. Clearly stating downstream use cases would strengthen the paper's contributions. Could authors clearly explain downstream application of this task and dataset?

Dataset and experimental concerns

  1. Important missing information. There are in my opinion important missing information related to the proposed RegionDet dataset. Since images are collected from public web sources, clearly stating the licenses and sources is extremely important. Then, it is not clear what rationale has been used to select the classes in the first place. In the COCO dataset paper, authors clearly mention their strategy for class selection, it would strengthen the paper to clearly discuss it. It is also important to state the how the collectors were selected (professional? Not professional? Were they hired? Were they volunteers? Did they have incentives to properly annotate?, etc.). The dataset would also benefit from person de-identification, if possible. Otherwise, that would be appreciated at least when showing qualitative results, in the manuscript. It especially feels strange that "vendor" would be a visual state, since it is more of a well defined region. Could authors please add all these information to the manuscript? Could authors clarify how and why they chose these classes initially?
  2. Open-sourcing the code. While I do understand that the code and dataset remain private during the review phase, there is no mention of whether the dataset and the scripts for the experiments will be made open-sourced after potential acceptance. Can authors clarify what the status of the code and dataset will be?
  3. Questionable open-set evaluation protocol. Open-set detectors are evaluated in zero-shot. This setting is interesting, but to clearly compare with the closed-set models, it would be important to also finetune open-set detectors on the training split of RegionDet to see if they benefit from this training and can adapt to the data. Could authors please provide experimental results under this setting too?
  4. Lack of implementation details. Please specify the link of the official implementation used for each approach, and the default settings used (LR, batch size, optimizer, etc.). This can be done in the supplementary materials and referred in "implementation details" (L. 318).
  5. Improving performances. Generally speaking, authors do not investigate performance improvement using this dataset or by developing a method related to it. This is usually expected from papers introducing datasets. This would also better justify data collection for Region Detection.

Miscellaneous

  1. References. There are many missing references which impede reading. I initially thought that DINO referred to "Emerging Properties in Self-Supervised Transformers" (M. Caron et al. 2021, ICCV), but it actually comes from "DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection" (H. Zhang et al., ICLR 2023). I found baselines not be properly referenced. Could authors clearly indicate the reference of the models next to their table entries and cite them accordingly?
  2. Classification vs. Localization. I would be curious to know whether the VLMs can correctly identify the category independently of the location in the image. Perhaps they are good at understanding that there is "damage" but simply not good at localizing it. For each sample in the dataset, authors could construct a "yes"/"no" binary classification experiment asking whether there is the category on the sample. That would give a rough idea of whether VLMs are simply missing the localization of the concept or simply not realizing that the concept is present in the image in the first place. Could authors please share the results of this experiment?

Minor weaknesses

  1. Clarifying the difference between Region Detection and RegionDet. Although clarified after the method section, the sentence "Region Detection is a benchmark setting" (L. 18) in the abstract can confuse readers.
  2. ICLR Format. References should be in parentheses. Please refer to the official template.
  3. Highlighting results. Result tables are quite blunt, it would be appreciated if authors could bold important/best performing numbers, or highlight particular outcomes in the tables directly.
  4. Title and description of table 6. I do not understand what experiment has been conducted on table 6. The caption is not enough to clearly understand the setting and there is only one occurrence of that table in the manuscript, line 391, which also does not explain the experiment that much. Could author please better describe it?
  5. Prompt templates. Please make a small table containing the exact prompts you use for each prompting setting. You can use "{category}" to indicate the category variable inside the prompt.
  6. Class imbalance. Authors mention that "construction", "damage" and "vendor" classes have greater appearance in scene variation (L. 241), but the justification is not clear. When running the numbers, these 3 classes make up 80% of the training data of RegionDet. Could authors kindly re-explain that choice?
  7. Review 2nd key finding claims. The second key finding is not sharp enough. While closed-set detectors are trained on the dataset (L.252), zero-shot ones are not (L. 255). At L. 453, authors claim "Second, performance cannot be explained by detection paradigm alone". This is true in what is shown, but without retraining the open-set detectors on the RegionDet dataset, this claim feels very weak.
  8. Quality control. Since there has been annotator cross-checking, and since the annotated regions are ambiguous it would be interesting if the paper used the weighted disagreement rate metric for evaluation, which also take into account the difficulty of the annotation for human reviewers. It has been introduced in that paper: "Intrinsic Images in the Wild", Bell et al., SIGGRAPH 2014. and an example usage, also in scene graph understanding, has been used in: "Holistic Order Prediction in Natural Scenes", Musacchio et al., NeurIPS 2025. I understand that this might be complicated to achieve since the metadata of the annotations might not have been tracked. So this criticism should be just taken as a suggestion for improvement.
  9. Visualizations. Visualizations of attention map or gradcams from the model might be interesting to understand the importance of regions in their representations.
  10. Circular definitions. Technically speaking all the definitions in table 2 are circular. I would invite authors to adjust them.
Questions for authors

Please refer to the weaknesses section.

In the current state of the paper, the weaknesses greatly outweigh the strengths of the paper.

I also believe that the current format and writing of the manuscript is not satisfactory for acceptance at a top tier conference such as ICLR.

Improving my score would require clearly assessing all my concerns.

Nonetheless, I am greatly worried about the task format itself and unless authors can simply and clearly convince that there is an advantage over scene graph, I do not think I will improve my score.

2 comments
Reviewer_zQASOpenReview ↗

Final remarks before end of the rebuttal period.

Dear authors, dear AC,

Since the authors have not produced any rebuttal and since the rebuttal period will end soon, I will maintain my initial score. I believe that in the current state, this manuscript should be rejected. Nonetheless, I encourage the authors to improve their manuscript and submit to another conference.

Thank you.

Dear Reviewer,

Thank you for your response and for continuing to follow up on our manuscript as the discussion period draws to a close.

Given the low initial ratings, our concurrent rebuttal commitments for other submissions, and our limited time and capacity, we decided not to pursue a rebuttal for this manuscript in this round. We will carefully consider all reviewers’ comments, further improve this work, and submit it to another conference after thorough revision. Thank you for your advice and encouragement.

Even though we did not submit a rebuttal, you still took the initiative to follow up and explain your final assessment. We sincerely appreciate your conscientious and responsible approach to the review process. Thank you for your time and effort.

Reviewer_mkoeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose RegionDet, a benchmark dataset for region-level detection. Unlike existing object detection benchmarks that primarily focus on instance-level or object-level detection, RegionDet focuses on detecting regions and requires holistic consideration of the relationships among entities within a region, as well as between entities and their surrounding background. The authors evaluate and analyze several existing closed-set object detectors, open-set object detectors, and vision-language models (VLMs) on RegionDet.

Critical strengths

S1: Promising and practically relevant task formulation. The proposed region-level detection task is motivated by realistic application scenarios where understanding a region as a whole, rather than detecting individual entities independently, is important. The task formulation has potential practical relevance and may provide a useful new direction for visual understanding.

S2: Broad evaluation of existing methods. The authors conduct a relatively comprehensive evaluation of existing approaches, including closed-set object detectors, open-set object detectors, and vision-language models. They also design task-specific evaluation protocols for different types of models, which provides a useful comparison of their capabilities on the proposed benchmark.

S3: Clear presentation. The paper is clearly written and well organized. The motivation, task formulation, benchmark construction, and experimental analysis are presented in a logical and accessible manner, making the paper relatively easy to follow.

Critical weaknesses

W1. Potential ambiguity and overlap among category semantics. Unlike instance-level object detection, where category definitions are typically more clearly separated, the eight categories in RegionDet may exhibit substantial semantic overlap. For example, Walking, Talking, Waiting, Queuing, and Crossing are not necessarily mutually exclusive: a person may be walking while talking. Similarly, Construction and Damage may naturally co-occur within the same region. Such semantic overlap introduces inherent ambiguity into the annotation and evaluation, and may partially explain the relatively limited performance of existing models. In this case, the difficulty may arise not only from the limitations of the model architectures, but also from ambiguity inherent in the task definition and dataset.

The authors state that “Cases without sufficient evidence for determining category identity or region membership were excluded” (L214–215). However, this filtering strategy may itself limit the applicability of RegionDet to real-world scenarios, where multiple activities or semantic conditions can naturally coexist within the same region. Rather than excluding regions that potentially contain multiple categories, a multi-label formulation could arguably provide a more faithful representation of such real-world situations. At minimum, the authors should provide a more detailed analysis of the extent of category overlap and discuss how such ambiguity affects both annotation consistency and benchmark performance.

W2: Limited analysis of the characteristics and challenges of the proposed dataset. Although the authors evaluate a broad range of existing methods and identify several interesting findings, the analysis is primarily focused on comparing model performance rather than investigating the intrinsic characteristics of RegionDet. In particular, the paper could provide deeper analysis of why existing object detectors struggle with this task and whether they indeed lack the ability to model the relationships among entities and between entities and their surrounding context, which the authors identify as a key distinction between region-level and conventional object detection. For example, analyses based on learned representations, attention maps, feature similarity, or controlled experiments could help determine whether current models fail to capture such relational information. Such analysis would not only provide stronger evidence for the motivation of RegionDet, but could also reveal concrete research directions for developing models specifically suited to this benchmark. This would substantially strengthen the contribution beyond dataset construction and empirical benchmarking.

W3: Ambiguity in region boundary annotations. Unlike conventional object detection, where the spatial extent of an isolated object can often be defined relatively clearly when significant occlusion is absent, RegionDet requires annotating the spatial extent of an entire region based on factors such as the entities within the region, their activities, and the surrounding context. Consequently, the boundaries of a region may be inherently less well-defined than those of conventional object instances. Such annotation ambiguity could introduce inconsistencies in the supervision and make precise localization intrinsically difficult. This may also be related to the first finding in Sec. 4.6: “First, region-level patterns are learnable under supervision, but precise localization remains difficult.” The authors should investigate whether the localization difficulty is partly caused by inherent uncertainty or inconsistency in the region boundary annotations, rather than solely reflecting a limitation of existing detection models.

W4: Lack of evaluation metrics specifically designed for RegionDet. The authors emphasize that region-level detection has several characteristics that distinguish it from conventional object detection. However, the current evaluation appears to rely primarily on standard object detection metrics such as AP, AP50, and AP75. It is unclear whether these metrics can fully capture the capabilities required by RegionDet, particularly given the semantic overlap among categories and the potentially ambiguous spatial extent of regions. A dedicated evaluation protocol or a set of complementary metrics tailored to the unique properties of RegionDet would therefore be valuable. For example, evaluation could explicitly assess the model's ability to recognize region-level semantics, handle overlapping or co-occurring categories, and localize semantically meaningful regions despite inherently ambiguous boundaries. Developing such an evaluation framework would also provide a more comprehensive assessment of the challenges identified in Weaknesses 1 and 3.

Questions for authors

Q1: The dataset is divided evenly between the training and test sets, with 4,005 images in each split. This is somewhat unusual compared with many established vision benchmarks, which typically allocate a substantially larger portion of the data to training (e.g., COCO uses approximately 80% of its images for training, with the remaining images used for validation and evaluation). Could the authors provide a clearer rationale for adopting a 1:1 train/test split, and discuss whether this choice has any implications for the amount of training data available for learning the proposed region-level concepts?

More importantly, could the authors provide more details about the principles used to construct the train/test split? Is the split performed randomly at the image level, or are there additional constraints based on the image source, scene, event, video sequence, or other metadata? Since the images are collected from public web sources and may contain visually similar or temporally related samples, I am concerned that a purely image-level random split could result in highly similar images appearing in both training and test sets. This could lead to train-test leakage and potentially overestimate the benchmark performance. Did the authors explicitly perform any deduplication or similarity-based filtering, or otherwise design the splitting procedure to ensure sufficient independence between the training and test sets? Please clarify the splitting protocol and how the authors mitigate this potential issue.

Reviewer_JBSMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Based on the paper's claim, it proposes Region Detection. It is a box-level detection task targeting finding specific regions rather than single instances. It introduces RegionDet, a dataset containing 8,010 web images. The authors benchmark three groups of models:14 closed-set detectors, 15 open-vocabulary detectors, and two VLMs. They conclude that region-level patterns are hard for these models but learnable under supervision, and suggest that weak boundary cues, context dependency, and insufficient relation-level understanding are the key challenges.

Critical strengths

Broad benchmarking results for detectors. The sweep covers a wide range of detectors, including classic one-stage, two-stage, diffusion-based, and DETR-style baselines. The zero-shot results (≤2.1 AP under all three prompt types) show a clear limitation for current open-vocabulary detectors.

Critical weaknesses

The validity of the benchmark.

  • Many images from the figures of the paper appear to be screenshots of web image-search previews, with commercial stock-photo watermarks and website badges that may affect the evaluation.
  • Too many details are missing. See Questions for Authors.
  • Single-category images: None of the annotated examples in the paper shows different categories co-occurring in one image. Localization only separates instances of the same category. If images contain a single category, recognition can be solved at the image level. It is then unclear what box AP measures beyond image-level classification plus splitting same-category instances.

Unsupported claims.

  • The closest related work is missing. The abstract (L16–18) states that existing benchmarks are limited for region targets. The paper compares only against object-instance, segmentation, action/HOI, and referring benchmarks. It omits the settings closest to Region Detection: Visual Phrases (CVPR 2011) and phrase detection in VRD (ECCV 2016); they both detect a relation-defined composite as a single box.
  • The stated challenges are asserted, not analyzed. The abstract claims that further analyses reveal weak boundary cues, strong context dependency, and insufficient relation-level understanding. The evidence, however, only consists of per-category AP, AP50/AP75, and qualitative figures. The submission has no supplementary or auxiliary experiments to help with the analysis. It remains unclear how much each factor contributes to the observed failures.
  • The VLM conclusions rely on too little evidence. MLLMs are most relevant to relation- and context-defined targets and regions, yet the paper only covers two relatively old models. The VLM study omits stronger grounding-capable models available at submission time, such as Qwen3-VL or Gemma variants.
Questions for authors
  • Data source and licensing. Which websites were the images collected from, and under what licenses? What fraction of images contains watermark overlays? Given the sources' terms of use, can the images be redistributed?
  • Dataset split. How was the train/test split made? Without a validation split, how were checkpoints selected for the closed-set models?
  • Annotation agreement. The definition of the region and its size can be highly biased; please report IoU between independent double annotations, and human evaluation results, at least on a subset.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.