acceptodds
Under review as a conference paper at ICLR 2027

ELSA: Evaluating localization of social activities in urban streets using open-vocabulary detection

Abstract

We introduce ELSA: Evaluating Localization of Social Activities, to our knowledge the first multi-label benchmark for detecting individual and group activities in still street-level images. ELSA draws on theoretical frameworks in urban sociology and design, and covers in-the-wild scenes in which group sizes and activity types vary significantly. The dataset includes 1,500 manually annotated images with more than 6,000 bounding boxes. Existing Open-Vocabulary Detection (OVD) models face a number of challenges. They often struggle with semantic consistency across diverse inputs and are sensitive to slight variations in input phrasing, leading to inconsistent performance. The calibration of their predictive confidence, especially in complex multi-label scenarios, remains suboptimal, frequently resulting in overconfident predictions that do not accurately reflect their context understanding. We expose these limitations and show that modern OVD models perform poorly on ELSA. We further propose Normalized Log-Sum-Exp (N-LSE), a context-aware confidence score for these models. Unlike the commonly used Max-Logit, which relies on a single token, N-LSE accounts for every label in the query, yielding a less inflated ranking and better detection results. Finally, we present Dynamic Box Aggregation (DBA), an evaluation algorithm that penalizes contradictory predictions for the same person or group and recovers correct boxes that NMS discards, providing a more reliable AP. We release our dataset and code.

Primary area:
datasets and benchmarks
Keywords:
computer vision, grounded vision-language models, open-vocabulary object detection, social interaction recognition, benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_daQsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ELSA is a test-only benchmark for open-vocabulary detection of people, pairs and groups in still street-level images of New York City. The main contribution is a task definition and a dataset together with evaluation protocols and the evaluation results on major detectors.

  • Data: 1,500 images and 6,384 boxes. Each box carries multiple labels from 26 categories (Condition, State, Activity, Other), forming 265 combinations. These are turned into 1,361 templated prompts (L090–091).
  • N-LSE: a confidence score that averages token confidences within each label and takes a log-mean-exp over labels, replacing Max-Logit.
  • DBA: an evaluation procedure that keeps near-top boxes per person and counts contradictory labels as false positives, presented as "more reliable" than NMS-AP.
  • Results: eight detector variants and a two-stage Grounding DINO + SigLIP 2 baseline are evaluated zero-shot. All score about 2× chance. N-LSE beats Max-Logit in 14 of 16 cells, and DBA yields more true positives than NMS.
Critical strengths

A clearly defined, theory-grounded compositional task with careful evaluation hygiene.

  • Explicit label-consistency rules (App. A.4).
  • A fixed dataset-level box budget, so detectors with different score scales are comparable (L275–281).
  • Chance and ceiling references for every model (Tables 1, 5).
  • This makes the very low numbers interpretable, and the benchmark usable if its data quality is established.

An informative diagnostic result: the two-stage baseline is about 10× better than end-to-end detectors (Table 2: 0.448 vs 0.0435). This points to description grounding, not person detection, as the bottleneck, a useful finding for OVD research.

Critical weaknesses

Evaluating only detection models is not justified.

  • All evaluated systems are open-vocabulary detectors (MDETR, OWL-ViT, Detic, Grounding DINO, LLMDet). Even stronger detectors the paper cites (OWLv2, GLIP, YOLO-World, Florence-2; L123–136) are not run, and vision-language models are not evaluated or discussed.
  • The task is to locate people from compositional descriptions such as "a group sitting and dining", which is where vision-language models are strongest. The paper's own results point the same way: adding an image–text model in the two-stage baseline improves AP about 10× (Table 2), so understanding the description, not finding people, seems to be the bottleneck.
  • On a closely related multi-person referring benchmark with interaction descriptions, a general VLM far outperforms a strong detector (HumanRef, arXiv 2503.08507, Table 4: average DF1 56.2 for Qwen2.5-VL-7B vs 23.3 for DINO-X).
  • As it stands, the near-chance results describe older detection pipelines rather than how hard the task is for current systems, which limits what the benchmark tells the community.
  1. The dataset's central novelty, "pair" and "group", is not defined or validated.
  • Boxing pairs and groups as units is what distinguishes ELSA from existing person datasets; e.g., driving datasets such as JAAD and TITAN label only individuals. The paper should say so explicitly.

  • Yet the only definition is that Condition is "the social configuration of the subjects as alone, pair, or group" (L191–192). The paper does not say when two nearby people form a pair rather than two individuals, whether a group means three or more, whether proximity or interaction is required, or how far a group box extends.

  • No inter-annotator agreement is reported for labels or boxes (L200–205). Every prompt includes a Condition label, and a match requires IoU 0.75–0.90, so part of the near-chance performance may reflect annotation ambiguity rather than model failure. The dataset is also small and single-city.

  • The two proposed methods do not offset this. Both are tested only on ELSA, and DBA's "more reliable" claim rests mainly on keeping more boxes per person than the NMS baseline, not on any validation of reliability. It is preferred to give operational pair/group criteria, double-annotate a subset, and report agreement for Condition labels and pair/group box extents.

Questions for authors
  • How does ELSA differ from JAAD, PIE, TITAN, nuScenes attributes and Refer-KITTI? Please add a dataset comparison table.

  • How do the current method compare against the the grounding-capable VLMs both from the quality and the cost perspective?

5 comments

We thank the reviewer for the careful analysis and constructive suggestions, which motivated additional experiments beyond open-vocabulary detectors and a richer analysis of related work for the revised manuscript.

How do the current method compare against the grounding-capable VLMs both from the quality and the cost perspective?

ELSA primarily targets open-vocabulary detection (OVD) models, which produce bounding boxes with associated confidence scores for textual queries. Generative grounding models such as Qwen and Gemma operate differently: they generate coordinates as sequences of tokens, with each token conditioned on preceding outputs. Although these models can be evaluated for localization accuracy using measures such as IoU-based recall, their generated outputs do not necessarily provide confidence scores equivalent to those of OVD models. This distinction matters for DBA, which uses numerical differences between detection scores to retain competing predictions. Applying DBA to generative grounding models therefore requires defining and validating a suitable per-box confidence measure. Running the same evaluation procedure does not, by itself, establish comparability across these model families.

However, we address the reviewer concern by first enriching our evaluation with additional detectors (OWLv2 and YOLO-World) and secondly providing complementary grounding results for four VLMs (Florence-2, Qwen3-VL-8B, Qwen3.5-9B, and Gemma 4 E4B) under one possible evaluation protocol. Following the official Qwen grounding examples, we requested all matching boxes for each prompt. For Qwen and Gemma, we then defined a proxy confidence score for each box as the geometric mean of the probabilities of the generated tokens encoding its coordinates. We run Florence-2 with its built-in open-vocabulary detection task and score its boxes in the same way from their location tokens, as its released code does for its object detection task.

These proxy scores reflect generated coordinate-token probabilities, not validated confidence in detection correctness. Thus, applying DBA's fixed margin does not establish comparability with native OVD scores. Localization can instead be compared using score-independent measures such as IoU-based recall, with the same set of prompts and an explicitly defined output budget.

We present these additional experiments as supplementary grounding results rather than a directly comparable extension of the OVD benchmark. Under the tested scoring procedures, the evaluated generative VLMs also achieve low absolute AP, although their relative performance varies across DBA-AP and NMS-AP. Because these comparisons depend on proxy confidence scores whose interpretation differs from native detector scores, they do not establish a definitive ranking across model families. Qwen inference was approximately 12 times slower than our longest detector runs under the tested configuration. We will report the inference settings and computational costs alongside these results and clarify their methodological limitations in the supplementary material.

We plan to include the additional detectors in the main table of the paper, and clarify limitations for the use of VLMs in ELSA together with initial findings in our final supplementary material.

ModelVariationScoringGlobalCSCSACSOAlonePairGroup
Reference
OWLViT-B/32DBA-AP0.00030.00210.000100.00040.00050
OWLViT-B/32NMS-AP0.00170.0155000.005100
GdinoSwin-BDBA-AP0.00230.00680.00150.00540.00320.00150.0024
GdinoSwin-BNMS-AP0.04350.1770.02910.02860.04490.07210.0097
New Detectors
OWLv2 ViT-B/16DBA-AP0.00300.01600.00150.00220.00370.00010.0056
OWLv2 ViT-B/16NMS-AP0.02620.2220.00080.01500.03670.02810.0129
YOLOv2-XDBA-AP0.00460.03320.00110.00120.004400.0100
YOLOv2-XNMS-AP0.01040.08470.00120.00330.02750.00170.0022
VLMs models
Florence-2LargeDBA-AP0.00080.00300.00060.00050.00150.00080.0002
Florence-2LargeNMS-AP0.00100.00670.000300.00100.00160.0002
Qwen3.5-9BDBA-AP0.00020.00070.00010.00010.000300.0002
Qwen3.5-9BNMS-AP0.00010.0008000.000300
Qwen3VL-8BDBA-AP0.00400.00780.00360.00040.00610.00390.0019
Qwen3VL-8BNMS-AP0.00260.00350.00240.00460.00310.00310.0015
Gemma4 E4BDBA-AP0.00020.00170.00010.000010.00070.000020.000006
Gemma4 E4BNMS-AP0.00010.00050.0000100.000200.000005

How does ELSA differ from JAAD, PIE, TITAN, nuScenes attributes and Refer-KITTI? Please add a dataset comparison table

Most of the suggested datasets use forward-facing videos captured from vehicle-mounted cameras, limiting the sidewalk areas which can be extracted.

The main goal of those works is also different, as they are centered on autonomous driving, targeting tasks such as crossing behavior (JAAD, PIE), trajectory forecasting (TITAN, PIE) and tracking (nuScenes). Their annotations do not cover groups, pairs, nor the range of complex social interactions in ELSA. More broadly, the suggested datasets primarily study how pedestrians interact with the car, whereas ELSA models how people interact with each other in public space. Finally, ELSA combines structured Condition–State–Activity annotations with an evaluation framework that enables systematic comparisons across social configurations and activity types. DBA further exposes contradictory predictions with similar confidence that NMS-based evaluation can conceal. Together, these contributions help diagnose model vulnerabilities and guide the development of more reliable methods for analyzing social interactions in public spaces.

Following reviewer GsFJ suggestion, we have also included an quantitative evaluation and comparison with other benchmarks that cover single individual characteristics of ELSA (CityPersons for person detection in urban imagery, OVDEval for open-vocabulary detection, and OmniLabel for detection using compositional prompts). We will incorporate this discussion and the following comparison table below into the main manuscript.

DatasetImageryStill imagesLocationTaskAnnotated unitPair/group boxesSocial activities¹Language queries
JAADDashcam videoNoN. America, EuropeCrossing behaviorPedestrianCrowd boxes onlyNoNo
PIEDashcam videoNoTorontoCrossing intention, trajectory predictionPedestrianNoNoNo
TITANOn-board GoPro videoNoTokyoTrajectory forecastingPerson, vehicleNoLimited²No
Refer-KITTIforward-facing driving videoNoKarlsruheReferring multi-object trackingCar, pedestrianNoNoYes
nuScenes360° camera rigNoBoston, Singapore3D detection, trackingIndividual objectNoNoNo
ELSAPanoramic street-level imageryYesNew York CitySocial-activity localizationIndividual, pair, groupYesYesYes, with paraphrases

¹ Includes activities such as dining, talking, shopping, and taking photos.

² Talking in groups and phone-related activities.

We hope that the evaluation of six additional models, the clarified distinction between VLMs and OVDs, and the expanded comparison of ELSA with existing benchmarks have addressed the reviewer’s main concerns.

The dataset's central novelty, "pair" and "group", is not defined or validated. [...] The paper does not say when two nearby people form a pair rather than two individuals, whether a group means three or more, whether proximity or interaction is required, or how far a group box extends. No inter-annotator agreement is reported for labels or boxes (L200–205).

We thank the reviewer for highlighting the need to clarify our annotation criteria and quality-control procedures.

Annotation criteria. ELSA aims to capture visually identifiable social configurations rather than establish actual social relationships, which cannot always be determined from still images. Our annotation guidelines distinguish three configurations: alone, when an individual is not visibly participating in a shared activity or interaction; pair, when two individuals form a recognizable social unit through shared activity or interaction; and group, when three or more individuals form such a unit. Social configurations were identified using a combination of spatial proximity and observable visual cues, including gestures, head and body orientation, posture, and apparent shared activities. Pair and group bounding boxes enclose the visible extent of the individuals belonging to the identified social unit, excluding nearby individuals who are not part of it. In most cases, these configurations were visually apparent, although some images presented ambiguity, particularly when individuals were close together without clear evidence of interaction. Such cases were flagged for further review during the annotation process. We will clarify these operational criteria in the revised manuscript.

Annotation quality control. Written guidelines were shared with all annotators.Three PhD students and one postdoctoral researcher checked the existing boxes and labels, corrected errors, and reviewed ambiguous cases together. Uncertain cases and disagreements were explicitly flagged, corrected, and reviewed again, followed by a final review by the postdoctoral researcher, whose expertise spans urban planning and computer science.

We recognize that annotation consistency does not establish the validity of the underlying social interpretation. Our iterative review process was designed to reduce inconsistencies and address ambiguous cases, although we did not conduct a separate quantitative inter-annotator agreement study. We will document these procedures and acknowledge the limitations of inferring social configurations from still images.

Reviewer_iPK7OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ELSA is a benchmark for localizing individual and group activities in still street-view images. It contains 1,500 images, 6,384 multi-label boxes, and 1,361 prompts derived from label combinations and synonyms. The paper evaluates open-vocabulary detectors, proposes a label-wise N-LSE score, and introduces DBA to penalize contradictory predictions for overlapping targets.

Critical strengths
  1. The dataset addresses a useful and challenging application beyond object nouns: identifying social configurations, states, and activities jointly. Its structured labels, manual annotation, expert review, and prompt variants provide a concrete resource for this problem.

  2. The evaluation includes several detector families, LLMDet, and a two-stage SigLIP 2 baseline. The budget, temperature, and aggregation analyses help characterize the observed limitations rather than relying on a single comparison.

Critical weaknesses
  1. The claim that DBA is a more reliable AP needs stronger validation. DBA uses GT-assisted grouping and an absolute score margin before converting contradictory retained predictions into false positives. Rescaling scores from [0.9,0.6] to [0.45,0.30] preserves ranking but changes retention at margin 0.2. The reported margin sweep does not establish comparability across detector score scales or agreement with human semantic judgments. The public implementation also ignores subsequent same-prompt duplicates inside a DBA group. These choices need justification and controlled metric checks before DBA can support cross-model conclusions.

  2. The N-LSE comparison conflates relevant-token selection, within-label averaging, and cross-label aggregation. A mean using exactly the same label/token mapping is a necessary simple control. Improved AP establishes ranking benefits, not probability calibration. The strongest Grounding DINO variant improves global mAP only from 0.230% to 0.240% in Table 1, and the low absolute scores make it important to separate localization, semantics, and detection-budget effects when interpreting the benchmark.

Questions for authors
  1. How stable are DBA model rankings under monotone score rescaling, and what evidence links DBA more closely than standard AP to human judgments of semantic correctness?

  2. How does N-LSE compare with a simple mean over the same selected label confidences? Are calibration claims supported by a calibration-specific evaluation?

  3. Can the localization, correct-prompt, and retained-budget recall decomposition in the released analysis be reported to explain the very low AP values?

2 comments

We thank the reviewer for taking the time to review our manuscript and for providing valuable and constructive feedback.

The public implementation ignores subsequent same-prompt duplicates inside a DBA group.

We thank the reviewer for identifying a rule that should have been stated explicitly. We will clarify the duplicate-handling in the manuscript and make it explicit in the pseudocode. We also checked how much it matters for the results. In all reported experiments, the rule applies to no box at any of the IoU thresholds for any model. The reported numbers would therefore be the same whether such duplicates are ignored or counted as false positives.

The rule itself is intentional. DBA treats repeated correct detections, incorrect detections, and contradictory descriptions differently. Repeating a correct detection does not indicate the same failure as predicting an incorrect label or assigning similar scores to incompatible descriptions. This was the argument of NMS to begin with. DBA counts the correct match once while ensuring that the latter failures affect the evaluation. NMS-AP also excludes suppressed predictions from AP rather than counting them as false positives, although its selection rule differs from ours (Yao et al., 2024b). TIDE likewise distinguishes duplicate detections from classification and localization errors and analyzes their effects separately (Bolya et al., 2020).

How stable are DBA model rankings under monotone score rescaling, and what evidence links DBA more closely than standard AP to human judgments of semantic correctness?

The model ranking under DBA-AP is stable under rescaling and also when the scores of all detectors are placed on one common scale. In all settings we tested, the rank correlation with the order in the paper is at least 0.88 and the two best variants never change. We ran two checks:

Linear rescaling. The margin sweep (0.05 to 0.5) had already been run separately for each model. The paper did not explicitly state this; we will add a comment in the revised manuscript. For DBA’s retention rule, applying margins of 0.05, 0.10, 0.20, 0.30, and 0.50 to the original scores is equivalent to multiplying those scores by 4, 2, 1, 2/3, and 0.4, respectively, and using a fixed margin of 0.20. In the reviewer’s example, halving the scores while keeping the margin at 0.20 gives the same retention decisions as applying a margin of 0.40 to the original scores.

Non-linear rescaling. We rescale the scores with three monotone functions: the square root, square and logarithm. As a fourth rescaling, we replace every score by its percentile rank among the model’s boxes, for instance, a score at the 95th percentile becomes 0.95. The result depends only on the order of a model’s scores, not on how widely they are spread, so the scores of all models end up on the same scale. We repeated the sweep for each variant. The following table shows the Spearman rank correlation between the model order under global DBA-AP in each setting and the order in the paper (the eight model variants of Table 2).

MarginNo rescalingSquare rootSquareLogRank
0.050.950.930.90.880.95
0.100.9510.950.950.95
0.20 (paper)11111
0.3011111
0.5011111

At margins of 0.20, 0.30, and 0.50, all eight variants retained their original ranking under every tested transformation, including conversion to within-detector percentile ranks. At margins of 0.05 and 0.10, no variant moved more than two positions. Grounding DINO Swin-B and LLMDet ranked highest across all settings.

Absolute DBA-AP values depend on the score margin. The rescaling experiments assess the stability of model rankings. We have not independently tested whether DBA-AP agrees more closely than NMS-AP with human judgments of semantic correctness. We will therefore replace the general claim of greater reliability with a more precise statement. When a detector assigns competing descriptions to the same target, DBA retains predictions within the specified margin of the highest score and penalizes those that contradict one another under ELSA’s Condition and State rules. NMS can conceal such a contradiction by suppressing one of the predictions.

How does N-LSE compare with a simple mean over the same selected label confidences? Are calibration claims supported by a calibration-specific evaluation?

As stated in Sec. 3, the arithmetic mean is already part of the proposed N-LSE aggregation. As β → 0, N-LSE converges to the mean of the selected label confidences. The suggested comparison therefore evaluates two configurations within the same aggregation family. We compared the mean with our fixed setting of β = 1, keeping token selection and within-label averaging unchanged. The following table reports global mAP on ELSA in percent.

ModelMax-LogitN-LSE (=1)Mean
GD Swin-B0.22950.24050.2380
GD Swin-T (1)0.09620.10330.1026
GD Swin-T (2)0.05240.10430.1032
LLMDet0.16150.20850.2029

The mean performs comparably to N-LSE at β = 1, consistent with their relationship within the same family. These results support aggregating evidence across the selected labels, with both configurations improving over Max-Logit. We fixed β = 1 without tuning it, and will add the mean as the β → 0 configuration in the appendix’s temperature table.

We also agree that AP measures ranking and not calibration. We computed the expected calibration error of the retained boxes using ten score bins and an IoU threshold of 0.75.

ModelMax-LogitN-LSE (=1)
GD Swin-B0.4730.213
GD Swin-T (1)0.6140.260
GD Swin-T (2)0.6620.255
LLMDet base0.5300.198

Neither of the scores is calibrated, although the numbers above should be read carefully and with the difficulty of the benchmark in mind. Since fewer than one in a hundred retained boxes is correct, a calibrated score would have to assign almost every box a value below 0.01. We will stop claiming calibration gains. We remove the corresponding wording from the paper and claim a better ranking only.

Can the localization, correct-prompt, and retained-budget recall decomposition in the released analysis be reported to explain the very low AP values?

Below, we provide the requested decomposition for the eight model variants in Table 2. We will report this in the revised manuscript.

Our table shows three recalls. Each is a share of ground truth boxes that are matched by a corresponding predicted box. The localization (class-agnostic) recall in the first column allows for a box of any prompt, ignoring the label. The correct-prompt recall allows only boxes of the correct prompt (where synonymous prompts are counted separately, as in our AP). Both are measured on the ten highest scoring boxes per image and prompt. The retained budget recall in the third column allows only boxes of the correct prompt after the detection budget is applied.

All of the values are averaged over the paper’s IoU thresholds and label combinations; they are in percent:

VariantLocalization recallCorrect-prompt recallRetained-budget recallmAP
GD Swin-B72.041.213.60.230
GD Swin-T (1)65.526.93.70.096
GD Swin-T (2)57.619.82.10.052
LLMDet70.233.314.20.161
MDETR ResNet-10156.311.12.40.051
MDETR EfficientNet-B554.411.13.00.049
Detic13.57.03.50.074
OWL-ViT8.11.71.70.038

The table shows that the low AP cannot be attributed to a single cause and that the main cause differs between the models. Detic and OWL-ViT match few ground-truth boxes with any box at these IoU thresholds. Note however that the localization recall favors models that return many boxes, so the low value of OWL-ViT does not necessarily imply poor localization. Both MDETR variants match more than half of the ground-truth boxes, but only 11 % with the correct prompt.

Reviewer_GsFJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce ELSA, a benchmark for detecting individual and group activities in still street-level images. They also test the performance of Open-Vocabulary Detection (OVD) models on this benchmarks, showing that SOTA approaches perform poorly on it. Another valuable contribution is the introduction of Normalized Log-Sum-Exp (N-LSE), a context-aware confidence score for the OVD models. Finally, the authors propose an innovative evaluation algorithm, Dynamic Box Aggregation (DBA), that penalizes contradictory predictions for the same person or the same group. They also release the dataset and the code for reproducibility and adoption from the research communities.

Critical strengths
  • The multi-label benchmark is useful for researcher and practitioners working in computer vision with a focus on recognition of behaviors and social interactions;

  • The number of OVD models tested is adequate;

  • The context-aware confidence score and the evaluation algorithm are sounded and provide a mode valuable evaluation strategy;

  • The paper is clear and well-written.

Critical weaknesses
  • The dataset per se seems a better fit for computer vision conferences;

  • The authors should be better clarify and motivate the relevance of performing the recognition of human activities and social interactions in still images vs. videos;

  • It would be very relevant not just having different OVD models tested on the proposed benchmark but the same OVD models tested on the ELSA and other alternative benchmarks. This would be provide a better and experimental justification of the need of introducing this new one;

  • More details should be provided on the human annotation. The authors state at page 4 that "a team of seven trained annotators manually corrected the initial boxes ...". How is this done? Do multiple annotators correct or label the same image/box? Is there any kind of agreement among the annotators?

Questions for authors
  • Could the authors provide an analysis of the activities/interactions more difficult to be detected by the different tested models or viceversa the ones where they perform better?

  • Is Google Street View a good data source? It seems a dataset where situations/images with people on the streets tend to be not captured. Could you give some details/statistics on the impact of the different data sources in building the benchmark?

4 comments

It would be very relevant not just having different OVD models tested on the proposed benchmark but the same OVD models tested on other alternative benchmarks.

We thank the reviewer for this valuable suggestion, which prompted a more informative analysis of how ELSA’s combined requirements challenge existing models. We agree that evaluating the same models on established benchmarks provides stronger experimental support for ELSA’s contribution.

We therefore selected three datasets that capture complementary aspects of ELSA: CityPersons for person detection in urban imagery, OVDEval for open-vocabulary detection, and OmniLabel¹ for detection with compositional prompts. We also appreciate the reviewer’s positive assessment of the breadth of our original model evaluation. Following reviewer daQs’s suggestion, we further expanded the set of models in both this comparison and our main analysis.

Table 1 compares performance on CityPersons with three increasingly demanding settings on ELSA, using person detection as a first common reference point. When evaluation is restricted to person detection, models achieve comparable performance on the two datasets. However, introducing ELSA’s social configurations—whether people are alone, in pairs, or in groups—already produces a substantial drop: average mAP falls to approximately one-quarter of that on CityPersons (15.7 versus 63.3). Evaluating ELSA’s full label combinations further reduces performance. This progression suggests that person detection in urban imagery alone does not capture the difficulty of recognizing and localizing the social activities represented in ELSA.

ModelCityPersons "a person"ELSA "a person"ELSA conditionsELSA (all prompts)
GD Swin-B82.287.122.40.230
GD Swin-T (1)70.684.519.40.096
GD Swin-T (2)91.182.817.40.052
LLMDet base72.585.820.20.161
OWLv272.971.616.80.193
OWL-ViT B/3235.456.55.40.038
Detic R5070.178.09.50.074
MDETR B529.468.614.20.049
MDETR RN10131.464.513.80.051
YOLO-World X77.381.517.70.182

Table 2 provides complementary comparisons with OVDEval and OmniLabel. Across the 10 evaluated models, average mAP is 25.8 on OVDEval Relationship, 28.1 on the other four OVDEval subsets, and 17.9 on OmniLabel, compared with 0.11 on the full ELSA benchmark—approximately two orders of magnitude lower. When each model is evaluated using its native score, only Grounding DINO Swin-B exceeds 0.2% baseline mAP on ELSA.

Together, these experiments show that models perform substantially better on benchmarks covering individual aspects of the task, supporting ELSA’s contribution as a benchmark that brings these challenges together.

We will include both tables and the accompanying analysis in the revised manuscript.

ModelOVDEval RelationshipOVDEval other fourOmniLabel descriptionsELSA
GD Swin-B41.737.923.70.230
GD Swin-T (1)36.630.720.90.096
GD Swin-T (2)26.533.224.50.052
LLMDet base31.531.518.20.161
OWLv214.626.919.50.193
OWL-ViT B/321.712.512.70.038
Detic R5022.927.310.00.074
MDETR B528.724.116.40.049
MDETR RN10128.125.816.30.051
YOLO-World X25.231.217.00.182

In Table 1, the first three columns report AP at an IoU threshold of 0.5. In Table 2, OVDEval and OmniLabel follow their official protocol (AP averaged over IoU thresholds from 0.50 to 0.95). The ELSA column in both tables reports mAP averaged over IoU thresholds from 0.75 to 0.90, following ELSA’s evaluation protocol.

¹ Excluding Objects365 as its images were unavailable from the official source.

Regarding the annotation process and composition of ELSA.

Human annotation. Three PhD students and one postdoctoral researcher checked the existing boxes and labels, corrected errors, and reviewed ambiguous cases together. They then took turns auditing one another’s annotations, and identified errors and disagreements were corrected. The postdoctoral researcher, whose background spans urban planning and computer science, conducted a final review. This process provided cross-annotator quality control. We will better clarify this procedure in the manuscript.

Activity-level performance. We rank activities by mAP and recall. First, we notice that activities were generally difficult for the evaluated detectors to localize. Even the highest activity-level mAP on ELSA was only 0.16%, for load/unload packages from car/truck. However, activities with higher mAP and recall usually exhibit the presence of large peculiar items or objects (waiting at the bus stop, loading/unloading item) or unique patterns (sport activities). On the other hand, harder activities involve smaller elements (pet interaction) or interaction with scene elements that are commonly present such as restaurants. Our analysis also reverses the intuition that most common activities are the easiest to find. The three most common values (talking, crossing sidewalk, and phone interaction) sit exactly in the middle of this ranking.

Imagery sources. We thank the reviewer for highlighting the lack of details on the distribution of data providers in ELSA. We plan to extend the following line in Sec. 3.1 “Dataset Statistics”: “ELSA consists of 1,500 images selected from Google (122), Bing (932), and Apple (446), with a total of 6,384 annotated bounding boxes for social and individual activities, encompassing 26 distinct labels.”

The resulting sidewalk-views produced by our pre-processing pipeline have different resolutions according to the different providers. We obtain 1024x561 pixels images from Apple and Bing and 768x433 from Google. This motivates our choice to favour the first two in the dataset construction. This however does not imply that Google is a worse source. The final average number of boxes per image is actually higher for Google (4.68) than Apple (4.14) and Bing (4.25). We also examine the five rarest activities after normalizing their frequencies by the total number of boxes, finding that Google shows comparable representation (0.94 of the dataset average), compared with Apple (1.3) and for Bing (0.87).

Relevance of performing the recognition of human activities and social interactions in still images vs. videos.

We motivate the use of still images in the introduction (L062–L070), mainly through the cost of city-wide video. We will extend this motivation with three arguments.

(1) Coverage. Street-level imagery covers the street networks of entire cities and is a standard data source in urban analytics (Biljecki & Ito, 2021) (2) Time. The same places are captured repeatedly, which already supports city-wide studies of change over time. (3) Extensibility. Each ELSA image is stored as a panorama ID with its coordinates and heading. The library we use to retrieve the panoramas supports nine street-level services, enabling easy future extension of our work to new cities.

I'm happy with the new experiments and clarifications produced during the rebuttal. I'm still convinced that this paper would be a better fit for a computer vision or a multimedia conference. Overall, I've decided to move the score to "weak accept".

We thank the reviewer for raising their score following our additional experiments and analysis.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall response

We sincerely thank all reviewers for their insightful feedback and constructive suggestions. Their feedback has been incorporated to further improve the paper.

We are pleased that the reviewers recognized the value of our benchmark for evaluating social configurations, states, and activities beyond conventional object recognition [GsFJ, iPK7]; its theory-grounded formulation, structured annotations, and careful evaluation design [iPK7, daQs]; the soundness of the proposed confidence scoring and evaluation methodology [GsFJ]; and the breadth of model comparisons and diagnostic experiments [GsFJ, iPK7, daQs.] We also appreciate the recognition of our two-stage baseline as an informative result highlighting description grounding as a potential bottleneck in OVD [daQs], and the clarity of the paper's presentation[ GsFJ.]

In response to the reviewers’ suggestions, we conducted additional experiments and clarified several aspects of the benchmark.

  • Reviewer GsFJ. We compared the same detectors across ELSA, CityPersons, OVDEval, and OmniLabel and showed that, on the same ELSA images, distinguishing individuals, pairs, and groups was substantially harder than detecting people. We also analyzed performance by activity and clarified how the annotations were checked, how each imagery provider contributed to the dataset, and why we used still images.

  • Reviewer iPK7. We tested DBA under score rescaling and found that the detector ranking remained unchanged at the margin used in the paper. We also showed that an arithmetic mean over the same label confidences performed comparably to N-LSE and reported how recall changed when a correct prompt was required and the detection budget was applied.

  • Reviewer daQs. We added OWLv2 and YOLO-World and provided supplementary results for grounding-capable VLMs, explaining how their use of coordinate-token probabilities as confidence scores limits direct comparison with detectors. We also compared ELSA with related pedestrian datasets and clarified how the team identified pairs and groups, determined whom to include in each box, and reviewed ambiguous cases.

The individual responses below provide the full results and address the remaining questions. We thank the reviewers again for their feedback and welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.