- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SURE-Bench tests whether models answer human-activity questions using the intended sensor evidence. It records 61 indoor activities from 22 people with thermal, event, LiDAR, radar, and Vicon motion capture. Single-sensor atomic questions are composed into compound questions requiring known sensor pairs. Models must provide both the answer and the supporting modalities.
Critical strengths
-
The motivation is clear, and the atomic-then-compound construction is a neat way to know the required evidence by design.
-
The sensor setup (four rigs, calibrated to Vicon coordinates) is solid, and the tolerance rules for quantitative questions are well motivated (Appendix A.3).
-
The removal experiments (Tables 5 and 6) are the right kind of check, even if the results are weak.
Critical weaknesses
- The sensor-evidence metric can be solved without the sensors. A question like "Is anyone's face unusually warm?" says "thermal" in plain words, and "who reacts first, by how many ms?" says "event camera." A text-only model with basic knowledge of sensors can name the right pair without seeing any observation. So the evidence accuracy mostly measures whether the model knows what each sensor is for, not whether its answer is based on that sensor. The paper itself notes that the evidence field asks which sensors are needed, not which were used. Please add a text-only baseline where the model gets only the question and the list of sensor names. If text-only evidence accuracy is close to the numbers in Table 3, the main finding does not hold.
2.220 compound questions, with only 17 to 20 for three of the six sensor pairs. One percentage point in those pairs is a fraction of a question. 47 of 61 recordings are "clear," and results by visibility condition (a stated motivation) are not reported. All participants are recorded in one room.
- The models barely use the sensors even on atomic questions. In Table 5, the gap between giving the target sensor and giving only the other three is +0.9 to +13.4 points, and negative for Mistral. Non-target accuracy (37% to 43%) is close to target accuracy for most models. Either the questions can be answered without the sensor (which breaks the "sensor-exclusive" claim), or the models are guessing. A chance baseline and a text-only baseline per answer type are needed here too.
Questions for authors
-
What is the label distribution for compound questions, and what do always-yes and always-no score?
-
What is the evidence accuracy of a text-only model?
-
Which VLM did the exclusivity audit, and how many questions did it remove?
-
How many candidate questions did the LLM produce, and how many were removed or edited by experts? What was the agreement between experts?
-
How was thermal ground truth defined?