- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ELSA is a test-only benchmark for open-vocabulary detection of people, pairs and groups in still street-level images of New York City. The main contribution is a task definition and a dataset together with evaluation protocols and the evaluation results on major detectors.
- Data: 1,500 images and 6,384 boxes. Each box carries multiple labels from 26 categories (Condition, State, Activity, Other), forming 265 combinations. These are turned into 1,361 templated prompts (L090–091).
- N-LSE: a confidence score that averages token confidences within each label and takes a log-mean-exp over labels, replacing Max-Logit.
- DBA: an evaluation procedure that keeps near-top boxes per person and counts contradictory labels as false positives, presented as "more reliable" than NMS-AP.
- Results: eight detector variants and a two-stage Grounding DINO + SigLIP 2 baseline are evaluated zero-shot. All score about 2× chance. N-LSE beats Max-Logit in 14 of 16 cells, and DBA yields more true positives than NMS.
Critical strengths
A clearly defined, theory-grounded compositional task with careful evaluation hygiene.
- Explicit label-consistency rules (App. A.4).
- A fixed dataset-level box budget, so detectors with different score scales are comparable (L275–281).
- Chance and ceiling references for every model (Tables 1, 5).
- This makes the very low numbers interpretable, and the benchmark usable if its data quality is established.
An informative diagnostic result: the two-stage baseline is about 10× better than end-to-end detectors (Table 2: 0.448 vs 0.0435). This points to description grounding, not person detection, as the bottleneck, a useful finding for OVD research.
Critical weaknesses
Evaluating only detection models is not justified.
- All evaluated systems are open-vocabulary detectors (MDETR, OWL-ViT, Detic, Grounding DINO, LLMDet). Even stronger detectors the paper cites (OWLv2, GLIP, YOLO-World, Florence-2; L123–136) are not run, and vision-language models are not evaluated or discussed.
- The task is to locate people from compositional descriptions such as "a group sitting and dining", which is where vision-language models are strongest. The paper's own results point the same way: adding an image–text model in the two-stage baseline improves AP about 10× (Table 2), so understanding the description, not finding people, seems to be the bottleneck.
- On a closely related multi-person referring benchmark with interaction descriptions, a general VLM far outperforms a strong detector (HumanRef, arXiv 2503.08507, Table 4: average DF1 56.2 for Qwen2.5-VL-7B vs 23.3 for DINO-X).
- As it stands, the near-chance results describe older detection pipelines rather than how hard the task is for current systems, which limits what the benchmark tells the community.
- The dataset's central novelty, "pair" and "group", is not defined or validated.
-
Boxing pairs and groups as units is what distinguishes ELSA from existing person datasets; e.g., driving datasets such as JAAD and TITAN label only individuals. The paper should say so explicitly.
-
Yet the only definition is that Condition is "the social configuration of the subjects as alone, pair, or group" (L191–192). The paper does not say when two nearby people form a pair rather than two individuals, whether a group means three or more, whether proximity or interaction is required, or how far a group box extends.
-
No inter-annotator agreement is reported for labels or boxes (L200–205). Every prompt includes a Condition label, and a match requires IoU 0.75–0.90, so part of the near-chance performance may reflect annotation ambiguity rather than model failure. The dataset is also small and single-city.
-
The two proposed methods do not offset this. Both are tested only on ELSA, and DBA's "more reliable" claim rests mainly on keeping more boxes per person than the NMS baseline, not on any validation of reliability. It is preferred to give operational pair/group criteria, double-annotate a subset, and report agreement for Condition labels and pair/group box extents.
Questions for authors
-
How does ELSA differ from JAAD, PIE, TITAN, nuScenes attributes and Refer-KITTI? Please add a dataset comparison table.
-
How do the current method compare against the the grounding-capable VLMs both from the quality and the cost perspective?
5 comments
How does ELSA differ from JAAD, PIE, TITAN, nuScenes attributes and Refer-KITTI? Please add a dataset comparison table
Most of the suggested datasets use forward-facing videos captured from vehicle-mounted cameras, limiting the sidewalk areas which can be extracted.
The main goal of those works is also different, as they are centered on autonomous driving, targeting tasks such as crossing behavior (JAAD, PIE), trajectory forecasting (TITAN, PIE) and tracking (nuScenes). Their annotations do not cover groups, pairs, nor the range of complex social interactions in ELSA. More broadly, the suggested datasets primarily study how pedestrians interact with the car, whereas ELSA models how people interact with each other in public space. Finally, ELSA combines structured Condition–State–Activity annotations with an evaluation framework that enables systematic comparisons across social configurations and activity types. DBA further exposes contradictory predictions with similar confidence that NMS-based evaluation can conceal. Together, these contributions help diagnose model vulnerabilities and guide the development of more reliable methods for analyzing social interactions in public spaces.
Following reviewer GsFJ suggestion, we have also included an quantitative evaluation and comparison with other benchmarks that cover single individual characteristics of ELSA (CityPersons for person detection in urban imagery, OVDEval for open-vocabulary detection, and OmniLabel for detection using compositional prompts). We will incorporate this discussion and the following comparison table below into the main manuscript.
| Dataset | Imagery | Still images | Location | Task | Annotated unit | Pair/group boxes | Social activities¹ | Language queries |
|---|---|---|---|---|---|---|---|---|
| JAAD | Dashcam video | No | N. America, Europe | Crossing behavior | Pedestrian | Crowd boxes only | No | No |
| PIE | Dashcam video | No | Toronto | Crossing intention, trajectory prediction | Pedestrian | No | No | No |
| TITAN | On-board GoPro video | No | Tokyo | Trajectory forecasting | Person, vehicle | No | Limited² | No |
| Refer-KITTI | forward-facing driving video | No | Karlsruhe | Referring multi-object tracking | Car, pedestrian | No | No | Yes |
| nuScenes | 360° camera rig | No | Boston, Singapore | 3D detection, tracking | Individual object | No | No | No |
| ELSA | Panoramic street-level imagery | Yes | New York City | Social-activity localization | Individual, pair, group | Yes | Yes | Yes, with paraphrases |
¹ Includes activities such as dining, talking, shopping, and taking photos.
² Talking in groups and phone-related activities.
We hope that the evaluation of six additional models, the clarified distinction between VLMs and OVDs, and the expanded comparison of ELSA with existing benchmarks have addressed the reviewer’s main concerns.
The dataset's central novelty, "pair" and "group", is not defined or validated. [...] The paper does not say when two nearby people form a pair rather than two individuals, whether a group means three or more, whether proximity or interaction is required, or how far a group box extends. No inter-annotator agreement is reported for labels or boxes (L200–205).
We thank the reviewer for highlighting the need to clarify our annotation criteria and quality-control procedures.
Annotation criteria. ELSA aims to capture visually identifiable social configurations rather than establish actual social relationships, which cannot always be determined from still images. Our annotation guidelines distinguish three configurations: alone, when an individual is not visibly participating in a shared activity or interaction; pair, when two individuals form a recognizable social unit through shared activity or interaction; and group, when three or more individuals form such a unit. Social configurations were identified using a combination of spatial proximity and observable visual cues, including gestures, head and body orientation, posture, and apparent shared activities. Pair and group bounding boxes enclose the visible extent of the individuals belonging to the identified social unit, excluding nearby individuals who are not part of it. In most cases, these configurations were visually apparent, although some images presented ambiguity, particularly when individuals were close together without clear evidence of interaction. Such cases were flagged for further review during the annotation process. We will clarify these operational criteria in the revised manuscript.
Annotation quality control. Written guidelines were shared with all annotators.Three PhD students and one postdoctoral researcher checked the existing boxes and labels, corrected errors, and reviewed ambiguous cases together. Uncertain cases and disagreements were explicitly flagged, corrected, and reviewed again, followed by a final review by the postdoctoral researcher, whose expertise spans urban planning and computer science.
We recognize that annotation consistency does not establish the validity of the underlying social interpretation. Our iterative review process was designed to reduce inconsistencies and address ambiguous cases, although we did not conduct a separate quantitative inter-annotator agreement study. We will document these procedures and acknowledge the limitations of inferring social configurations from still images.
We thank the reviewer for the careful analysis and constructive suggestions, which motivated additional experiments beyond open-vocabulary detectors and a richer analysis of related work for the revised manuscript.
ELSA primarily targets open-vocabulary detection (OVD) models, which produce bounding boxes with associated confidence scores for textual queries. Generative grounding models such as Qwen and Gemma operate differently: they generate coordinates as sequences of tokens, with each token conditioned on preceding outputs. Although these models can be evaluated for localization accuracy using measures such as IoU-based recall, their generated outputs do not necessarily provide confidence scores equivalent to those of OVD models. This distinction matters for DBA, which uses numerical differences between detection scores to retain competing predictions. Applying DBA to generative grounding models therefore requires defining and validating a suitable per-box confidence measure. Running the same evaluation procedure does not, by itself, establish comparability across these model families.
However, we address the reviewer concern by first enriching our evaluation with additional detectors (OWLv2 and YOLO-World) and secondly providing complementary grounding results for four VLMs (Florence-2, Qwen3-VL-8B, Qwen3.5-9B, and Gemma 4 E4B) under one possible evaluation protocol. Following the official Qwen grounding examples, we requested all matching boxes for each prompt. For Qwen and Gemma, we then defined a proxy confidence score for each box as the geometric mean of the probabilities of the generated tokens encoding its coordinates. We run Florence-2 with its built-in open-vocabulary detection task and score its boxes in the same way from their location tokens, as its released code does for its object detection task.
These proxy scores reflect generated coordinate-token probabilities, not validated confidence in detection correctness. Thus, applying DBA's fixed margin does not establish comparability with native OVD scores. Localization can instead be compared using score-independent measures such as IoU-based recall, with the same set of prompts and an explicitly defined output budget.
We present these additional experiments as supplementary grounding results rather than a directly comparable extension of the OVD benchmark. Under the tested scoring procedures, the evaluated generative VLMs also achieve low absolute AP, although their relative performance varies across DBA-AP and NMS-AP. Because these comparisons depend on proxy confidence scores whose interpretation differs from native detector scores, they do not establish a definitive ranking across model families. Qwen inference was approximately 12 times slower than our longest detector runs under the tested configuration. We will report the inference settings and computational costs alongside these results and clarify their methodological limitations in the supplementary material.
We plan to include the additional detectors in the main table of the paper, and clarify limitations for the use of VLMs in ELSA together with initial findings in our final supplementary material.