- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper present a new benchmark and a new method for non-contact human-object interaction. the benchmark is designed to evaluate how MLLMs can find the target of non-contact human–object interactions (gaze, pointing, contextual intention), with human-verified boxes and wrong-target alternatives labelled by error type (drift, distractor, ambiguity). Regarding the method (CLASP) it removes error-related directions found at layer 20 from three layers selected by a contrast score, folds them into the weights without gradient training, and improves grounding on seven open-source MLLMs.
Critical strengths
- The benchmark is very valuable: it puts gaze, pointing and contextual intention under one task with shared question types. Its target boxes are human-verified with reported agreement, and its wrong-target alternatives are labelled by error type, so it can show why a model grounds the wrong object, not just whether it does.
- CLASP is cheap at inference: it is a single-pass edit folded into the weights, with no gradient training and only two fitted scalars. It improves all seven open-source backbones while keeping general capability.audit, and detailed appendices that support reproducibility.
Critical weaknesses
- Evaluation protocol
- The localization prompt is not adapted to each model. All MLLMs and CLASP get the same generic localization prompt, which asks for JSON boxes in [0,1] coordinates (Appendix A.3). Many models are trained with their own box format: Qwen2.5-VL outputs absolute pixel coordinates, and InternVL typically uses a 0–1000 grid. Forcing one format on all models may lower scores for reasons unrelated to grounding. Several strong grounding models (Qwen2.5-VL-7B, InternVL3.5-8B) score at or below chance AUC in Table 2, which supports this concern. It also suggests that part of CLASP's gain (and PEFT baselines) may come from fixing this format mismatch rather than improving grounding.
- Table 3b compares unequal training budgets. Table 3b is labelled "1%-Data PEFT Baselines", but CLASP uses 1,491 annotated training images (estimation, diagnosis and calibration pools), while the PEFT baselines use only 42. The fair comparison is in Appendix Table A16, where LoRA and DoRA use the same 1,491 images. There, the gap shrinks to +0.014 AUC and +0.018 mAP over DoRA. A difference this small could come from uneven hyperparameter tuning. Also, the Budget-matched DoRA (AUC 0.740, mAP 0.528) performs about as well as the single-layer CLASP variant in Table 4 (AUC 0.739, mAP 0.531). The diagnostic subspace alone therefore brings no clear advantage over standard PEFT, and the remaining gain may come mainly from editing several layers.
- Table 3c should include additional experiments for the open-source backbones: standard supervised fine-tuning and linear probing from the top-3 layers for grounding. Without these very straightforward baselines, it's not clear if CLASP is really improving over the general-purpose architecture and fine-tuning methods.
- A simple grounding baseline is missing. A two-step pipeline would be easy to add: build a referring phrase (e.g., "the object the person is looking at / pointing to"), or have an MLLM produce it, then give it to a dedicated grounding model such as Florence-2 or Grounding DINO.
- Minor: gaze annotations are box-level only. Unlike GazeFollow, the benchmark has no 2D gaze point or gaze heatmap. Gaze is evaluated only through the target object's box, which limits how precisely gaze-following ability can be measured.
Questions for authors
questions:
- The role of human annotator is not really clear to me. In 3.2 and 3.3, describe the work of the annotators. Then, in the evalution setup 4.1, it's said "Five independent human evaluators assess 1,080 test items". So these samples are checked twice? so the human performance score can be seen as a measure of annotator agreement?
- The directions are computed on layer 20 and edits are applied on other layers. How does this compare with computing the directions separately at each edited layer? Why should the layer chosen for estimating directions be different from the layers chosen for editing?
suggestions:
- Florence-2-large is used for annotations for Objects365 and SWiG. It would it be interesting to include in table 2. I guess that it is probably the best model for the task if it has been used for annotations. I understand that using the same model for annotations and evaluation has some risks that should be discussed but the performance of Florence-2 could provide interesting light on the problem.
- Table A.12. I suggest adding a column to indicate zero-shot or fine-tuned
- ALS is never defined
- Figure 3 is really hard to read. Simplify it by removing most of the text and equations
- I find the dataset valuable for the community but with limited novelty from a technical standpoint (which is not a problem). thereforen, I would recommend allocating more space to the method and its evaluation.
7 comments
Responses to Reviewer Ze5B (Part II)
The native conventions raise the AUC of the frozen models by 0.041 and 0.037, which confirms a small format effect. Under native conventions, CLASP still improves AUC by 0.185 on Qwen2.5-VL-7B and by 0.215 on InternVL3.5-8B. This retains about 90% of its gains under the common format (0.209 and 0.237). The gain of CLASP therefore holds across coordinate formats.
(3) Third, the AUC near chance level comes from where the frozen models place their boxes. For each query, we take the highest-confidence box of the two frozen models. A box is counted as on the target, on a verified negative, or on the designated person when its IoU with that region is at least 0.5. The results are shown below and in Table A22 of the revised paper.
| Model | Highest-confidence box | Share (%) | Min.D | AUC |
|---|---|---|---|---|
| Qwen2.5-VL-7B | On target | 9.4 | 0.032 | 0.945 |
| On verified negative | 8.5 | 0.128 | 0.701 | |
| On designated person | 19.6 | 0.344 | 0.566 | |
| Elsewhere | 59.8 | 0.611 | 0.372 | |
| Invalid output | 2.7 | 1.000 | 0.500 | |
| All | 100.0 | 0.474 | 0.495 | |
| InternVL3.5-8B | On target | 10.3 | 0.032 | 0.942 |
| On verified negative | 15.1 | 0.127 | 0.722 | |
| On designated person | 19.7 | 0.318 | 0.559 | |
| Elsewhere | 52.6 | 0.440 | 0.360 | |
| Invalid output | 2.3 | 1.000 | 0.500 | |
| All | 100.0 | 0.340 | 0.517 |
The frozen models place their first box on the target in only about 10% of queries. About 20% of the boxes stay on the designated person, and more than half fall on regions that match neither the target nor any verified alternative. The AUC near chance level therefore reflects a failure to follow the cue from the person to the target. This is precisely the capability gap that LongHOI-Bench is designed to reveal. Section 4.1 and Table 7b of the revised paper report the native-coordinate results, and Section 4.1 also reports the box locations.
W2: The budgets are stated in the caption of Table 3, and Table A16 provides the matched comparison on the same checkpoint. In the revised Table 3b, we add a training-image column and the 1,491-image LoRA and DoRA rows from Table A16. The image-budget-matched comparison is shown below, in Table A16, and in the revised Table 3b. Task scores are pooled, capability scores are percentages, and edited rows report the three-seed mean ± SD. For CLASP, the 1,491 images are used for estimation, diagnosis, and calibration without gradient training.
| Method | Train images | AUC | mAP | ACC | POPE F1 | MMBench ACC |
|---|---|---|---|---|---|---|
| Frozen backbone | 0 | 0.5167 | 0.2896 | 0.5692 | 87.43 | 81.72 |
| LoRA | 1,491 | 0.7278±0.0084 | 0.5159±0.0072 | 0.6757±0.0051 | 86.68±0.11 | 80.94±0.13 |
| DoRA | 1,491 | 0.7397±0.0061 | 0.5279±0.0049 | 0.6848±0.0046 | 86.77±0.09 | 81.03±0.11 |
| CLASP | 1,491 | 0.7538±0.0031 | 0.5461±0.0032 | 0.7261±0.0028 | 86.91±0.06 | 81.19±0.08 |
Responses to Reviewer Ze5B (Part III)
(1) First, the matched gap exceeds the variation across seeds. CLASP outperforms DoRA on AUC, mAP, and ACC, and each gap is larger than the three-seed standard deviations of both methods. The ACC gap reaches about nine times the standard deviation of DoRA. CLASP also retains higher POPE F1 and MMBench accuracy.
(2) Second, the gain comes from which layers are edited, beyond how many. Table 4 already isolates the effect of the layer count. With the same subspace, 30 random three-layer sets reach 0.709 ± 0.019 AUC, and the three layers with the lowest scores reach 0.687. Both remain below single-layer CLASP at 0.739, and three edited layers improve over one layer only when they are selected by ALS. As you note, single-layer CLASP already performs on par with DoRA trained on the same 1,491 images, and it does so without any gradient update. The diagnostic subspace and the layer selection therefore contribute together.
W3: Following your suggestion, we have added both baselines on InternVL3.5-8B with all 4,160 training images. SFT updates all parameters for three epochs, and the checkpoint is selected by validation mAP. Linear probing fits a linear box regressor on the concatenated residual states of layers 19, 21, and 24 (12,288 dimensions). It outputs a single box without category or confidence, so mAP and ACC do not apply. The pooled results are shown below and in Table A23 of the revised paper.
| Method | Train images | Trainable | AUC | mAP | ACC | POPE F1 | MMBench ACC |
|---|---|---|---|---|---|---|---|
| Frozen backbone | 0 | 0% | 0.517 | 0.290 | 0.569 | 87.43 | 81.72 |
| Linear probing (layers 19, 21, 24) | 4,160 | 0.0006% | 0.708 | N/A | N/A | N/A | N/A |
| SFT | 4,160 | 100% | 0.764 | 0.584 | 0.743 | 85.11 | 78.17 |
| CLASP | 1,491 | 0% | 0.754 | 0.546 | 0.726 | 86.91 | 81.19 |
SFT uses 2.8 times more training images and updates every parameter. It exceeds CLASP by only 0.010 AUC, 0.038 mAP, and 0.017 ACC. It loses 2.32 points of POPE F1 and 3.55 points of MMBench accuracy, beyond the one-point preservation tolerance, while CLASP loses 0.52 and 0.53. Linear probing reaches 0.708 AUC, below CLASP. The three edited layers already encode much of the target location, and CLASP makes this information usable in generation without training. Section 5.2 of the revised paper reports the key numbers of these results.
Responses to Reviewer Ze5B (Part I)
To Reviewer Ze5B: Thanks for the constructive comments. Responses to specific comments are given below, which will also be added to the revised paper.
W1: Thanks for raising this important point. We address the concern on the localization prompt with three analyses.
(1) First, the models follow the requested format. The table below reports how often their localization outputs fail the parsing checks of Appendix A.3 (Table A20 of the revised paper). Every model produces valid outputs for at least 95.6% of the queries in each interaction.
(2) Second, following your suggestion, we have evaluated the two models you named under their native conventions. Qwen2.5-VL-7B uses pixel coordinates of the processed image, and InternVL3.5-8B uses integers from 0 to 1000. Both the designated person box and the output follow the native convention, and CLASP keeps the same subspace, layers, and weights without recalibration. The pooled results are shown below, in Table 7b, and in Table A21 of the revised paper.