acceptodds
Withdrawn from ICLR 2027

SURE-Bench: Sensor Understanding and Reasoning with Evidence Benchmark for Human Activity

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

19 views

Abstract

Recent human activity benchmarks increasingly incorporate heterogeneous sensors for multimodal understanding and reasoning. However, existing evaluations primarily verify the final task output or answer. Recent works indicate that correct answers do not necessarily mean that a model relies on the intended evidence, as VQA models can exploit shortcuts while still producing the expected answer. To address this issue, we introduce the Sensor Understanding and Reasoning with Evidence Benchmark (SURE-Bench), a multimodal and multi-view benchmark that evaluates both answer correctness and sensor-evidence correctness. SURE-Bench contains synchronized thermal, event camera, LiDAR, and radar observations across 61 indoor activity scenarios, with calibrated motion capture providing physical ground truth. We construct 687 sensor-exclusive atomic questions, each answerable from exactly one target modality, and compose them into 220 compound questions requiring specific sensor pairs. The required sensor evidence is therefore known by construction, allowing us to evaluate whether a model identifies the modalities required to support its answer. Experiments with recent VLMs show that answer correctness and sensor-evidence correctness do not consistently agree. Specifically, models may answer correctly without identifying the required sensor pair, or conversely, identify the required sensor pair without answering the question correctly. To the best of our knowledge, SURE-Bench is the first benchmark for indoor human activity to quantitatively evaluate answer correctness and sensor-evidence correctness separately, supporting research toward multimodal models whose answers are grounded in the required sensor evidence.

Primary area:
datasets and benchmarks
Keywords:
Multimodal sensing, Benchmark, Humanpose, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2doAOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SURE-Bench tests whether models answer human-activity questions using the intended sensor evidence. It records 61 indoor activities from 22 people with thermal, event, LiDAR, radar, and Vicon motion capture. Single-sensor atomic questions are composed into compound questions requiring known sensor pairs. Models must provide both the answer and the supporting modalities.

Critical strengths
  1. The motivation is clear, and the atomic-then-compound construction is a neat way to know the required evidence by design.

  2. The sensor setup (four rigs, calibrated to Vicon coordinates) is solid, and the tolerance rules for quantitative questions are well motivated (Appendix A.3).

  3. The removal experiments (Tables 5 and 6) are the right kind of check, even if the results are weak.

Critical weaknesses
  1. The sensor-evidence metric can be solved without the sensors. A question like "Is anyone's face unusually warm?" says "thermal" in plain words, and "who reacts first, by how many ms?" says "event camera." A text-only model with basic knowledge of sensors can name the right pair without seeing any observation. So the evidence accuracy mostly measures whether the model knows what each sensor is for, not whether its answer is based on that sensor. The paper itself notes that the evidence field asks which sensors are needed, not which were used. Please add a text-only baseline where the model gets only the question and the list of sensor names. If text-only evidence accuracy is close to the numbers in Table 3, the main finding does not hold.

2.220 compound questions, with only 17 to 20 for three of the six sensor pairs. One percentage point in those pairs is a fraction of a question. 47 of 61 recordings are "clear," and results by visibility condition (a stated motivation) are not reported. All participants are recorded in one room.

  1. The models barely use the sensors even on atomic questions. In Table 5, the gap between giving the target sensor and giving only the other three is +0.9 to +13.4 points, and negative for Mistral. Non-target accuracy (37% to 43%) is close to target accuracy for most models. Either the questions can be answered without the sensor (which breaks the "sensor-exclusive" claim), or the models are guessing. A chance baseline and a text-only baseline per answer type are needed here too.
Questions for authors
  1. What is the label distribution for compound questions, and what do always-yes and always-no score?

  2. What is the evidence accuracy of a text-only model?

  3. Which VLM did the exclusivity audit, and how many questions did it remove?

  4. How many candidate questions did the LLM produce, and how many were removed or edited by experts? What was the agreement between experts?

  5. How was thermal ground truth defined?

Reviewer_BjkzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces SURE-Bench for evaluating human activity question answering with thermal, event-camera, LiDAR, and radar observations. It collects synchronized recordings with motion-capture ground truth and constructs sensor-exclusive atomic questions that are combined into questions requiring two sensor modalities. The benchmark evaluates whether VLMs give the correct answer and identify the expected sensor pair. Experiments with five VLMs show that these two measures often disagree.

Critical strengths
  1. The dataset combines synchronized thermal, event-camera, LiDAR, and radar observations with motion-capture ground truth. This supports questions involving different types of physical measurements.

  2. The authors construct compound questions from expert-reviewed atomic questions. They score answers and identified sensor pairs separately, so errors in each can be compared.

Critical weaknesses
  1. The sensor-evidence metric does not establish that the model uses the required sensor observations. It only checks whether the model names the predefined sensor pair. In many questions, words such as temperature, distance, or motion may reveal the expected sensor without examining the measurements. The modality-removal tests also show little change in atomic answer accuracy for several models when the designated sensor is absent. A question-only baseline and stronger tests of dependence on the observations are needed before interpreting sensor-pair identification as evidence-grounded reasoning.

  2. The compound-question evaluation raises concerns. All 220 compound questions have binary answers, yet the reported answer accuracy is only 7.3% to 16.8% across models. The paper does not explain these unusually low results through the yes/no label distribution, invalid-response rates, or simple guessing and majority-class baselines. Without these details, it is difficult to tell whether the low accuracy reflects poor multimodal reasoning, the question design, or the scoring procedure.

Questions for authors
  1. How accurately can the evaluated models identify the required sensor pair from the question text alone, without any sensor observations?

  2. How should the small accuracy gaps between target and non-target modalities in the atomic-question ablation be reconciled with the claim that each atomic question requires exactly one sensor?

  3. What is the yes/no label distribution for the compound questions? How are invalid answers scored, and what accuracies do random guessing and majority-class prediction achieve?

Reviewer_e17UOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SURE-Bench, a multimodal and multi-view benchmark that evaluates both answer correctness and sensor-evidence correctness for indoor human activity understanding. The benchmark incorporates synchronized observations from four sensor modalities across 61 indoor activity scenarios, with 687 sensor-exclusive atomic questions and 220 compound questions requiring specific sensor pairs. By explicitly defining the required sensor modalities for each question, SURE-Bench enables the evaluation of whether models correctly identify the evidence needed to support their answers. Experiments on recent VLMs reveal inconsistencies between answer correctness and sensor-evidence correctness, highlighting the importance of evaluating evidence grounding beyond final answer accuracy.

Critical strengths
  1. The benchmark incorporates synchronized observations from four heterogeneous sensor modalities, including thermal cameras, event cameras, LiDAR, and radar, providing a diverse and complementary foundation for evaluating multimodal understanding and reasoning.
  2. The construction of sensor-exclusive atomic questions and compound questions requiring specific sensor pairs provides an interesting and systematic way to evaluate evidence correctness, distinguishing SURE-Bench from conventional benchmarks that primarily focus on final answer accuracy.
  3. The benchmark addresses an important yet underexplored aspect of multimodal understanding by evaluating not only answer correctness but also whether models identify the sensor evidence required to support their answers.
Critical weaknesses
  1. The benchmark is relatively small, covering only 61 activity scenarios and 220 compound questions, which may limit its diversity and the generalizability of the evaluation results.
  2. The benchmark primarily focuses on indoor human activity scenarios, which may introduce inherent dataset biases and exploitable textual shortcuts, potentially allowing models to answer some questions without relying on the intended sensor evidence.
Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.