acceptodds
Under review as a conference paper at ICLR 2027

LongHOI-Bench: Benchmarking and Improving Non-Contact Human-Object Interaction Understanding in MLLMs

Abstract

Understanding non-contact human-object interactions requires identifying a person's target from gaze, pointing, or contextual evidence. While multimodal large language models (MLLMs) support diverse visual tasks, existing benchmarks largely evaluate these cues separately, limiting systematic assessment of person-target understanding. To address this gap, we introduce **LongHOI-Bench**, containing 6,240 images and 56,880 questions spanning localization, attribute discrimination, spatial relations, and contextual intention. Human-verified targets and typed negative alternatives connect task performance with three grounding errors: *drift*, *distractor*, and *ambiguity*. We benchmark 13 proprietary and open-source MLLMs on LongHOI-Bench. Results reveal substantial limitations, with the strongest model reaching 68.5% mean accuracy across interaction types while humans exceed 92% in each interaction category. A balanced error audit identifies distractors as the most frequent failure. Motivated by these findings, we propose **CLASP**, which combines *error-type diagnosis* with *contrast-based layer selection* and weighted suppression through analytic weight editing. On paired localization queries, CLASP corrects 69.8% of distractor errors while introducing errors in 4.2% of initially correct predictions. Direction controls show that diagnostic relevance matters beyond captured contrast energy; the intervention also improves external gaze and pointing performance without recalibration. The benchmark and code will be publicly released upon publication.

Primary area:
datasets and benchmarks
Keywords:
multimodal large language models, non-contact human-object interaction, visual grounding, benchmark, diagnostic rank selection, subspace editing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Ze5BOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper present a new benchmark and a new method for non-contact human-object interaction. the benchmark is designed to evaluate how MLLMs can find the target of non-contact human–object interactions (gaze, pointing, contextual intention), with human-verified boxes and wrong-target alternatives labelled by error type (drift, distractor, ambiguity). Regarding the method (CLASP) it removes error-related directions found at layer 20 from three layers selected by a contrast score, folds them into the weights without gradient training, and improves grounding on seven open-source MLLMs.

Critical strengths
  • The benchmark is very valuable: it puts gaze, pointing and contextual intention under one task with shared question types. Its target boxes are human-verified with reported agreement, and its wrong-target alternatives are labelled by error type, so it can show why a model grounds the wrong object, not just whether it does.
  • CLASP is cheap at inference: it is a single-pass edit folded into the weights, with no gradient training and only two fitted scalars. It improves all seven open-source backbones while keeping general capability.audit, and detailed appendices that support reproducibility.
Critical weaknesses
  1. Evaluation protocol
  • The localization prompt is not adapted to each model. All MLLMs and CLASP get the same generic localization prompt, which asks for JSON boxes in [0,1] coordinates (Appendix A.3). Many models are trained with their own box format: Qwen2.5-VL outputs absolute pixel coordinates, and InternVL typically uses a 0–1000 grid. Forcing one format on all models may lower scores for reasons unrelated to grounding. Several strong grounding models (Qwen2.5-VL-7B, InternVL3.5-8B) score at or below chance AUC in Table 2, which supports this concern. It also suggests that part of CLASP's gain (and PEFT baselines) may come from fixing this format mismatch rather than improving grounding.
  • Table 3b compares unequal training budgets. Table 3b is labelled "1%-Data PEFT Baselines", but CLASP uses 1,491 annotated training images (estimation, diagnosis and calibration pools), while the PEFT baselines use only 42. The fair comparison is in Appendix Table A16, where LoRA and DoRA use the same 1,491 images. There, the gap shrinks to +0.014 AUC and +0.018 mAP over DoRA. A difference this small could come from uneven hyperparameter tuning. Also, the Budget-matched DoRA (AUC 0.740, mAP 0.528) performs about as well as the single-layer CLASP variant in Table 4 (AUC 0.739, mAP 0.531). The diagnostic subspace alone therefore brings no clear advantage over standard PEFT, and the remaining gain may come mainly from editing several layers.
  • Table 3c should include additional experiments for the open-source backbones: standard supervised fine-tuning and linear probing from the top-3 layers for grounding. Without these very straightforward baselines, it's not clear if CLASP is really improving over the general-purpose architecture and fine-tuning methods.
  • A simple grounding baseline is missing. A two-step pipeline would be easy to add: build a referring phrase (e.g., "the object the person is looking at / pointing to"), or have an MLLM produce it, then give it to a dedicated grounding model such as Florence-2 or Grounding DINO.
  1. Minor: gaze annotations are box-level only. Unlike GazeFollow, the benchmark has no 2D gaze point or gaze heatmap. Gaze is evaluated only through the target object's box, which limits how precisely gaze-following ability can be measured.
Questions for authors

questions:

  • The role of human annotator is not really clear to me. In 3.2 and 3.3, describe the work of the annotators. Then, in the evalution setup 4.1, it's said "Five independent human evaluators assess 1,080 test items". So these samples are checked twice? so the human performance score can be seen as a measure of annotator agreement?
    • The directions are computed on layer 20 and edits are applied on other layers. How does this compare with computing the directions separately at each edited layer? Why should the layer chosen for estimating directions be different from the layers chosen for editing?

suggestions:

  • Florence-2-large is used for annotations for Objects365 and SWiG. It would it be interesting to include in table 2. I guess that it is probably the best model for the task if it has been used for annotations. I understand that using the same model for annotations and evaluation has some risks that should be discussed but the performance of Florence-2 could provide interesting light on the problem.
  • Table A.12. I suggest adding a column to indicate zero-shot or fine-tuned
  • ALS is never defined
  • Figure 3 is really hard to read. Simplify it by removing most of the text and equations
  • I find the dataset valuable for the community but with limited novelty from a technical standpoint (which is not a problem). thereforen, I would recommend allocating more space to the method and its evaluation.
7 comments
AuthorsOpenReview ↗

Responses to Reviewer Ze5B (Part I)

To Reviewer Ze5B: Thanks for the constructive comments. Responses to specific comments are given below, which will also be added to the revised paper.

W1: Thanks for raising this important point. We address the concern on the localization prompt with three analyses.

(1) First, the models follow the requested format. The table below reports how often their localization outputs fail the parsing checks of Appendix A.3 (Table A20 of the revised paper). Every model produces valid outputs for at least 95.6% of the queries in each interaction.

MethodGazePointingContextualAll
# Localization queries6837191,2192,621
CogVLM2.83.93.73.5
LLaVA-v1.5-7B3.12.60.71.8
VILA-8B2.52.22.52.4
Llama3.2-Vision-11B1.93.22.52.6
InternLM-XComposer2-7B4.03.23.03.3
mPLUG-Owl-7B4.02.43.43.3
Qwen2.5-VL-3B3.72.52.12.6
Qwen2.5-VL-7B3.81.72.62.7
InternVL3.5-2B4.42.82.83.2
InternVL3.5-8B1.92.92.12.3
Gemini-2.0-Flash0.71.31.11.1
GPT-4o0.90.40.30.5
GPT-50.11.31.41.0
CLASP (InternVL3.5-8B)1.21.41.11.2

(2) Second, following your suggestion, we have evaluated the two models you named under their native conventions. Qwen2.5-VL-7B uses pixel coordinates of the processed image, and InternVL3.5-8B uses integers from 0 to 1000. Both the designated person box and the output follow the native convention, and CLASP keeps the same subspace, layers, and weights without recalibration. The pooled results are shown below, in Table 7b, and in Table A21 of the revised paper.

MethodFormatInvalid (%)Min.DAUCmAP
Qwen2.5-VL-7BCommon2.70.4740.4950.299
Qwen2.5-VL-7BNative1.60.4370.5360.333
Qwen2.5-VL-7B + CLASPCommon1.20.2160.7040.475
Qwen2.5-VL-7B + CLASPNative0.70.2030.7210.489
InternVL3.5-8BCommon2.30.3400.5170.290
InternVL3.5-8BNative1.40.3110.5540.329
InternVL3.5-8B + CLASPCommon1.20.1610.7540.546
InternVL3.5-8B + CLASPNative0.90.1520.7690.561
AuthorsOpenReview ↗

Responses to Reviewer Ze5B (Part II)

The native conventions raise the AUC of the frozen models by 0.041 and 0.037, which confirms a small format effect. Under native conventions, CLASP still improves AUC by 0.185 on Qwen2.5-VL-7B and by 0.215 on InternVL3.5-8B. This retains about 90% of its gains under the common format (0.209 and 0.237). The gain of CLASP therefore holds across coordinate formats.

(3) Third, the AUC near chance level comes from where the frozen models place their boxes. For each query, we take the highest-confidence box of the two frozen models. A box is counted as on the target, on a verified negative, or on the designated person when its IoU with that region is at least 0.5. The results are shown below and in Table A22 of the revised paper.

ModelHighest-confidence boxShare (%)Min.D AUC
Qwen2.5-VL-7BOn target9.40.0320.945
On verified negative8.50.1280.701
On designated person19.60.3440.566
Elsewhere59.80.6110.372
Invalid output2.71.0000.500
All100.00.4740.495
InternVL3.5-8BOn target10.30.0320.942
On verified negative15.10.1270.722
On designated person19.70.3180.559
Elsewhere52.60.4400.360
Invalid output2.31.0000.500
All100.00.3400.517

The frozen models place their first box on the target in only about 10% of queries. About 20% of the boxes stay on the designated person, and more than half fall on regions that match neither the target nor any verified alternative. The AUC near chance level therefore reflects a failure to follow the cue from the person to the target. This is precisely the capability gap that LongHOI-Bench is designed to reveal. Section 4.1 and Table 7b of the revised paper report the native-coordinate results, and Section 4.1 also reports the box locations.

W2: The budgets are stated in the caption of Table 3, and Table A16 provides the matched comparison on the same checkpoint. In the revised Table 3b, we add a training-image column and the 1,491-image LoRA and DoRA rows from Table A16. The image-budget-matched comparison is shown below, in Table A16, and in the revised Table 3b. Task scores are pooled, capability scores are percentages, and edited rows report the three-seed mean ± SD. For CLASP, the 1,491 images are used for estimation, diagnosis, and calibration without gradient training.

MethodTrain images         AUC         mAP         ACC  POPE F1MMBench ACC
Frozen backbone00.51670.28960.569287.4381.72
LoRA1,4910.7278±0.00840.5159±0.00720.6757±0.005186.68±0.1180.94±0.13
DoRA1,4910.7397±0.00610.5279±0.00490.6848±0.004686.77±0.0981.03±0.11
CLASP1,4910.7538±0.00310.5461±0.00320.7261±0.002886.91±0.0681.19±0.08
AuthorsOpenReview ↗

Responses to Reviewer Ze5B (Part III)

(1) First, the matched gap exceeds the variation across seeds. CLASP outperforms DoRA on AUC, mAP, and ACC, and each gap is larger than the three-seed standard deviations of both methods. The ACC gap reaches about nine times the standard deviation of DoRA. CLASP also retains higher POPE F1 and MMBench accuracy.

(2) Second, the gain comes from which layers are edited, beyond how many. Table 4 already isolates the effect of the layer count. With the same subspace, 30 random three-layer sets reach 0.709 ± 0.019 AUC, and the three layers with the lowest scores reach 0.687. Both remain below single-layer CLASP at 0.739, and three edited layers improve over one layer only when they are selected by ALS. As you note, single-layer CLASP already performs on par with DoRA trained on the same 1,491 images, and it does so without any gradient update. The diagnostic subspace and the layer selection therefore contribute together.

W3: Following your suggestion, we have added both baselines on InternVL3.5-8B with all 4,160 training images. SFT updates all parameters for three epochs, and the checkpoint is selected by validation mAP. Linear probing fits a linear box regressor on the concatenated residual states of layers 19, 21, and 24 (12,288 dimensions). It outputs a single box without category or confidence, so mAP and ACC do not apply. The pooled results are shown below and in Table A23 of the revised paper.

MethodTrain imagesTrainableAUCmAPACCPOPE F1MMBench ACC
Frozen backbone00%0.5170.2900.56987.4381.72
Linear probing (layers 19, 21, 24)4,1600.0006%0.708N/AN/AN/AN/A
SFT4,160100%0.7640.5840.74385.1178.17
CLASP1,4910%0.7540.5460.72686.9181.19

SFT uses 2.8 times more training images and updates every parameter. It exceeds CLASP by only 0.010 AUC, 0.038 mAP, and 0.017 ACC. It loses 2.32 points of POPE F1 and 3.55 points of MMBench accuracy, beyond the one-point preservation tolerance, while CLASP loses 0.52 and 0.53. Linear probing reaches 0.708 AUC, below CLASP. The three edited layers already encode much of the target location, and CLASP makes this information usable in generation without training. Section 5.2 of the revised paper reports the key numbers of these results.

Reviewer_KzjYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces LongHOI-Bench for non-contact human–object interaction understanding across gaze, pointing, and contextual intention. The benchmark contains 6,240 images and 56,880 questions, with human-verified targets and typed negative alternatives. The paper also proposes CLASP, which selects contrastive feature directions using error-type diagnosis and distributes their suppression across attention-output projections. Experiments report substantial improvements over the frozen backbone and transfer to external gaze and pointing datasets.

Critical strengths
  • The paper addresses a meaningful problem: recognizing scene objects does not necessarily imply correctly associating them with a designated person. Linking localization and subsequent questions through verified targets is potentially useful.
  • The construction and evaluation include several thoughtful controls: human target establishment before model-assisted mining, image-disjoint splits, annotation agreement, construction-sensitivity checks, and external transfer without recalibration.
  • The intervention is practical because it can be folded into existing weights. The rank, direction-subset, layer-allocation, and strength experiments provide useful empirical evidence.
Critical weaknesses
  • The benchmark’s construct validity needs stronger support. Several representative questions in Figure 2 concern directly named objects or person attributes, such as clothing and cap colors, and can potentially be answered without resolving the person–target relation. The current aggregate scores therefore mix relational grounding with ordinary visual recognition. Language-only and cue-masking controls are helpful, but do not establish the proportion of questions that actually require the intended relation.

  • The PEFT comparisons remain confounded by backbone configuration. Appendix H.1 states that the methods use different language hidden widths, and Appendix M.3 explicitly notes that image-budget matching does not establish identical backbone configurations. Consequently, the reported advantage over image-budget-matched DoRA cannot yet be attributed cleanly to the adaptation method.

Questions for authors
  • What fraction of questions requires resolving the person–target relation? Can results be reported separately for this subset, ideally including designated-person swaps on suitable multi-person images?
  • Can the key LoRA/DoRA comparisons be repeated using the exact same pretrained checkpoint, preprocessing, image budget, and evaluation interface as CLASP? Please clarify the reported 4096-versus-5120 hidden-width difference.
4 comments
AuthorsOpenReview ↗

Responses to Reviewer KzjY (Part I)

To Reviewer KzjY: Thanks for the constructive comments. Responses to specific comments are given below, which will also be added to the revised paper.

W1 & Q1: Thank you for the insightful question. All localization queries require the person-target relation by construction. Each query asks for the object referred to by the gaze, pointing, or contextual intention of the designated person. AUC, mAP, and Min.D in Table 2 are therefore computed only on such queries.

For the multiple-choice questions, we have further annotated relation dependence for every test question. A question is labeled as dependent when the target can be identified only through the cue of the designated person, and as independent otherwise. Two annotators labeled all 7,524 multiple-choice test questions independently, and a third annotator adjudicated the 319 disagreements (initial agreement 95.8%, Cohen's κ = 0.855). In total, 6,199 questions (82.4%) are dependent and 1,325 (17.6%) are independent. The accuracy on each question family and each subset is shown below and in Table A17 of the revised paper.

MethodAttributeSpatialIntentionDependentIndependent
# Questions2,5192,4472,5586,1991,325
CogVLM0.3980.4210.3340.3500.542
LLaVA-v1.5-7B0.5430.4620.4890.4670.646
VILA-8B0.4600.4060.4310.4010.579
Llama3.2-Vision-11B0.5770.5250.4620.4970.634
InternLM-XComposer2-7B0.4860.4470.4760.4380.618
mPLUG-Owl-7B0.3090.2110.1830.2080.358
Qwen2.5-VL-3B0.5140.4460.4730.4520.599
Qwen2.5-VL-7B0.5550.4860.4680.4660.673
InternVL3.5-2B0.5230.5070.4570.4670.626
InternVL3.5-8B0.6130.5780.5180.5130.834
Gemini-2.0-Flash0.6870.5750.6410.6090.758
GPT-4o0.7150.6670.6760.6610.803
GPT-50.6180.6130.6320.5930.751
CLASP (InternVL3.5-8B)0.7490.7330.6970.6960.867
Human0.9540.9480.9510.9480.963

(1) First, all 13 models are less accurate on dependent questions than on independent ones, with gaps from 13.7 to 32.1 points (17.7 on average). Humans reach 94.8% and 96.3% on the two subsets, a gap of only 1.5 points. The deficit is therefore specific to resolving the person-target relation.

(2) Second, the model ranking on the dependent subset closely follows the overall ranking (Spearman ρ = 0.982), so the conclusions of Table 2 hold on the dependent subset.

(3) Third, CLASP raises the accuracy of InternVL3.5-8B by 18.3 points on dependent questions (51.3% to 69.6%) and by 3.3 points on independent ones (83.4% to 86.7%). The gain concentrates on the capability that LongHOI-Bench targets.

AuthorsOpenReview ↗

Responses to Reviewer KzjY (Part II)

In addition, following your suggestion, we have conducted a designated-person swap on 237 multi-person test images. Each image contains two persons with the same interaction type, different target instances, and relations that can be confirmed from non-contact cues. Images in which the target can be identified only through holding or direct contact are excluded. The target of the second person was annotated by two annotators independently and verified by a third. The image and the question are kept fixed, and only the designated person box is exchanged, which gives 474 queries. Pair ACC requires correct answers under both designations, and Invariant reports the share of pairs that receive the same answer under both designations. The results are shown below and in Table A18 of the revised paper.

MethodOriginal ACCSwapped ACCPair ACCInvariant (%)
InternVL3.5-8B0.5440.4890.30848.1
GPT-4o0.6960.6460.49833.8
CLASP (InternVL3.5-8B)0.7760.7380.65823.2
Human0.9700.9660.9621.7

Humans change their answer with the designated person in almost every pair (1.7% invariant) and answer both designations correctly in 96.2% of pairs. InternVL3.5-8B returns the same answer for both persons in 48.1% of pairs, and GPT-4o does so in 33.8%. A large share of model answers is thus produced without using the designated person, which is precisely the capability gap that LongHOI-Bench is designed to reveal. CLASP reduces the invariant rate to 23.2% and raises pair accuracy from 30.8% to 65.8%.

Both analyses are added to Appendix N.1 of the revised paper, and Section 4.2 and Table 7a report the key numbers. Figure 2 originally illustrated all four question families. Several of its attribute and spatial examples, such as the questions on clothing and cap colors, belong to the independent subset. In the revised Figure 2, these examples are replaced with dependent test questions from LongHOI-Bench on the same images. The figure now reflects the dependent questions, which make up 82.4% of the multiple-choice questions. The dependence labels will be released with the benchmark.

W2 & Q2: You are right on this point. The hidden width of 5,120 in Appendix H.1 is a reporting error. All PEFT baselines were trained on the same pretrained InternVL3.5-8B checkpoint as CLASP, whose language model has a hidden width of 4,096. The logged trainable parameter count of LoRA confirms this. Its value of 76,677,120 matches rank 80 on the q, k, v, and o projections of this checkpoint (36 layers, 8 key-value heads, head dimension 128). The trainable parameter counts in Appendix H.1 were also computed with square key and value projections, and they are corrected to 76,677,120 for LoRA and 23,371,776 for DoRA. The pretrained checkpoint, preprocessing, image budget, and evaluation interface are identical across CLASP, LoRA, and DoRA, so the results in Table 3b and Table A16 remain unchanged.

The image-budget-matched comparison on this checkpoint is shown below, in Table A16, and in the revised Table 3b. Task scores are pooled, capability scores are percentages, and edited rows report the three-seed mean ± SD. For CLASP, the 1,491 images are used for estimation, diagnosis, and calibration without gradient training.

MethodTrain images         AUC         mAP         ACC  POPE F1MMBench ACC
Frozen backbone00.51670.28960.569287.4381.72
LoRA1,4910.7278±0.00840.5159±0.00720.6757±0.005186.68±0.1180.94±0.13
DoRA1,4910.7397±0.00610.5279±0.00490.6848±0.004686.77±0.0981.03±0.11
CLASP1,4910.7538±0.00310.5461±0.00320.7261±0.002886.91±0.0681.19±0.08

On the same checkpoint and under the same image budget, CLASP outperforms DoRA on all three task metrics. It also retains higher POPE F1 and MMBench accuracy. The advantage over budget-matched DoRA can therefore be attributed to the adaptation method. The width and the parameter counts are corrected in Appendix H.1, and the statement on backbone configurations in Appendix M.3 is removed.

We appreciate the suggestions on construct validity and the PEFT comparison. If there are further settings or variants you would like to discuss, we would be happy to conduct more experiments and perform further analysis.

Thanks for the additional experiments and clarifications. The relation-dependent breakdown and person-swap experiment address my concern about whether the questions require person–target grounding. The person-swap results are particularly helpful, as are the stronger gains on the relation-dependent subset. The clarification that the PEFT baselines use the same checkpoint, together with the corrected hidden width and parameter counts, also resolves my concern about the comparison. My main concerns have been addressed, and I maintain my support for acceptance.

AuthorsOpenReview ↗

Thank You to Reviewer KzjY

To Reviewer KzjY: Thank you for the follow-up and for maintaining your support for acceptance. We are glad that the breakdown by relation dependence, the designated-person swap, and the clarified PEFT configuration address your concerns. The person swap you suggested has made the evidence for person-target grounding much stronger. These analyses are included in Appendix N and Table 7 of the revised paper, and the dependence labels will be released with the benchmark.

Reviewer_9PLYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LongHOI-Bench, a benchmark for evaluating non-contact human-object interaction understanding in MLLMs. It contains 6,240 images and 56,880 questions covering gaze, pointing, and contextual intention. The authors identify three types of grounding errors: drift, distractor, and ambiguity, with distractors being the most common. They further propose CLASP, which identifies error-related feature directions and suppresses them across selected model layers to improve target grounding. Experiments show substantial improvements across several MLLMs and external datasets.

Critical strengths
  1. Meaningful benchmark with careful annotations. The paper brings together different types of non-contact interactions under a unified evaluation framework. The typed negative samples provide useful information about grounding errors, and the dataset construction includes detailed human verification and agreement statistics.
  2. Strong results and extensive experiments. CLASP significantly improves grounding performance across multiple models. The paper includes comprehensive ablations on feature directions, layer selection, and suppression strength. The improvements on external datasets also support the effectiveness of the approach.
Critical weaknesses
  1. The role of error type diagnosis is not fully established. Although CLASP uses grounding error labels to select feature directions, the selected directions are also the first four singular vectors. It is unclear whether the error labels provide a substantial benefit over standard rank selection. In addition, the feature differences are obtained by changing candidate box coordinates, which may not directly represent grounding errors.
  2. Some aspects of the evaluation need further support. Several questions in Figure 2 appear to test general visual recognition rather than person-target grounding, making the overall evaluation less focused on the proposed capability. The benchmark also evaluates relatively old MLLMs, and some PEFT baselines use different backbone configurations, which limits the strength of the comparisons.
Questions for authors
  1. Could the authors compare CLASP with a baseline that selects the number of SVD directions using validation performance without error type labels? This would help clarify the contribution of the error diagnosis.
  2. Since the feature differences are obtained by changing candidate box coordinates, how can the authors ensure that the selected directions reflect grounding errors rather than differences in coordinate representations?
  3. Some questions in Figure 2, such as those about clothing colors, do not seem to require person-target grounding. Could the authors clarify their role and report performance specifically on questions that depend on correct target identification?
  4. The evaluation includes relatively old models such as GPT-4o, GPT-5, and Gemini 2.0 Flash. Could the authors explain the model selection criteria and consider evaluating more recent MLLMs to verify whether the observed limitations still hold?
6 comments
AuthorsOpenReview ↗

Responses to Reviewer 9PLY (Part I)

To Reviewer 9PLY: Thanks for the constructive comments. Responses to specific comments are given below, which will also be added to the revised paper.

W1 & Q1: Thank you for the insightful suggestion. Error-type diagnosis determines how many leading directions are suppressed, and rules without error-type labels select other ranks. In the initial version of the paper, Table A2 already applied five spectral criteria without error-type labels to the same spectrum. They select k = 3, 7, 7, 7, and 8, and none of them selects the rank identified by diagnosis. The common 95% cumulative-energy rule selects k = 7, and the paired image-level bootstrap in Appendix D.4 shows that k = 4 exceeds k = 7 on AUC, mAP, and ACC with all 95% intervals above zero.

In the revised version, following your suggestion, we have further selected the rank by validation performance without error-type labels. Each rank from 1 to 8 is applied with the same layers, strength, and layer weights, and the validation split is not used by any stage of CLASP. The results are shown below and in Table A25 of the revised paper. Test scores repeat Table A3(a), and all scores are interaction-macro.

Rank kVal AUCVal mAPVal ACCTest AUCTest mAPTest ACC
10.68100.41860.64210.68370.42180.6489
20.73170.48880.69280.72260.48370.6834
30.75270.53750.70930.75190.54420.7203
40.75520.55500.72240.75840.55130.7256
50.75810.55790.72330.75680.55010.7242
60.75530.54780.72610.75430.54580.7214
70.75610.55090.71750.75220.54350.7191
80.74540.53040.71360.74760.53710.7138
Validation selection (k = 5)0.75810.55790.72330.75680.55010.7242
Error-type diagnosis (k = 4, CLASP)0.75520.55500.72240.75840.55130.7256

Validation selection chooses k = 5, which falls below k = 4 on all three test metrics. Diagnosis selects k = 4, the best rank on the test set, from the 617-image diagnosis pool alone. No edited model is evaluated on labeled validation data in this process. The error-type labels therefore identify the effective rank directly, while spectral rules and validation search settle on other ranks with lower test scores. Section 5.2 of the revised paper reports this result.

AuthorsOpenReview ↗

Responses to Reviewer 9PLY (Part II)

W1 & Q2: The selected directions reflect grounding errors for the following reasons.

(1) First, the paired design isolates the referred object. Each pair shares the image, the question, and the coordinate format. The two inputs differ only in whether the candidate box refers to the verified target or to a typed negative. Diagnosis retains a direction only when its projection energy separates the three error types, measured by η² against a permutation reference. All four selected directions, u1 to u4, exceed the permutation P99 of error-type separation, while u5 to u7 do not (Appendix B).

(2) Second, the gain holds without candidate coordinates. No candidate box is provided at test time, and the edited model receives only the image, the designated person box, and the question. The edit also improves outputs that contain no coordinates. Multiple-choice accuracy rises from 56.9% to 72.6%, and the gain concentrates on questions that require the person-target relation (see our response to W2 & Q3). In addition, the CLASP subspace overlaps with the Nullu HalluSpace, which is estimated from caption contrasts without any box coordinates. Their affinity at rank four is nearly four times the permutation-null P99 (Appendix H).

(3) Third, the gain persists under a different coordinate convention. If the subspace encoded the normalized coordinates used during estimation, a different convention at test time would remove much of the gain. We therefore apply the same subspace under the native convention of each backbone. Qwen2.5-VL-7B uses pixel coordinates of the processed image, and InternVL3.5-8B uses integers from 0 to 1000. Both the designated person box and the output follow the native convention, and the subspace, layers, and weights are unchanged. The pooled results are shown below, in Table 7b, and in Table A21 of the revised paper.

MethodFormatInvalid (%)Min.DAUCmAP
Qwen2.5-VL-7BCommon2.70.4740.4950.299
Qwen2.5-VL-7BNative1.60.4370.5360.333
Qwen2.5-VL-7B + CLASPCommon1.20.2160.7040.475
Qwen2.5-VL-7B + CLASPNative0.70.2030.7210.489
InternVL3.5-8BCommon2.30.3400.5170.290
InternVL3.5-8BNative1.40.3110.5540.329
InternVL3.5-8B + CLASPCommon1.20.1610.7540.546
InternVL3.5-8B + CLASPNative0.90.1520.7690.561

Under native conventions and without recalibration, CLASP improves AUC by 0.185 on Qwen2.5-VL-7B and by 0.215 on InternVL3.5-8B. This retains about 90% of its gains under the common format (0.209 and 0.237). The suppressed directions therefore capture grounding behavior that transfers across coordinate conventions. Section 5.2 and Table 7b of the revised paper report these numbers.

AuthorsOpenReview ↗

Responses to Reviewer 9PLY (Part III)

W2 & Q3: All localization queries require the person-target relation by construction. Each query asks for the object referred to by the gaze, pointing, or contextual intention of the designated person. AUC, mAP, and Min.D in Table 2 are therefore computed only on such queries.

For the multiple-choice questions, we have further annotated relation dependence for every test question. A question is labeled as dependent when the target can be identified only through the cue of the designated person, and as independent otherwise. Two annotators labeled all 7,524 multiple-choice test questions independently, and a third annotator adjudicated the 319 disagreements (initial agreement 95.8%, Cohen's κ = 0.855). In total, 6,199 questions (82.4%) are dependent and 1,325 (17.6%) are independent. The examples you cited belong to this 17.6%. The accuracy on each question family and each subset is shown below and in Table A17 of the revised paper.

MethodAttributeSpatialIntentionDependentIndependent
# Questions2,5192,4472,5586,1991,325
CogVLM0.3980.4210.3340.3500.542
LLaVA-v1.5-7B0.5430.4620.4890.4670.646
VILA-8B0.4600.4060.4310.4010.579
Llama3.2-Vision-11B0.5770.5250.4620.4970.634
InternLM-XComposer2-7B0.4860.4470.4760.4380.618
mPLUG-Owl-7B0.3090.2110.1830.2080.358
Qwen2.5-VL-3B0.5140.4460.4730.4520.599
Qwen2.5-VL-7B0.5550.4860.4680.4660.673
InternVL3.5-2B0.5230.5070.4570.4670.626
InternVL3.5-8B0.6130.5780.5180.5130.834
Gemini-2.0-Flash0.6870.5750.6410.6090.758
GPT-4o0.7150.6670.6760.6610.803
GPT-50.6180.6130.6320.5930.751
CLASP (InternVL3.5-8B)0.7490.7330.6970.6960.867
Human0.9540.9480.9510.9480.963

(1) First, all 13 models are less accurate on dependent questions than on independent ones, with gaps from 13.7 to 32.1 points (17.7 on average). Humans reach 94.8% and 96.3% on the two subsets, a gap of only 1.5 points. The deficit is therefore specific to resolving the person-target relation.

(2) Second, the model ranking on the dependent subset closely follows the overall ranking (Spearman ρ = 0.982), so the conclusions of Table 2 hold on the dependent subset.

(3) Third, CLASP raises the accuracy of InternVL3.5-8B by 18.3 points on dependent questions (51.3% to 69.6%) and by 3.3 points on independent ones (83.4% to 86.7%). The gain concentrates on the capability that LongHOI-Bench targets.

The independent questions thus provide a reference for ordinary recognition within the benchmark. The gap between the two subsets isolates the deficit specific to the person-target relation. We further swap the designated person on 237 multi-person test images. InternVL3.5-8B and GPT-4o return the same answer for both persons in 48.1% and 33.8% of pairs, against 1.7% for humans (Table A18).

The analysis is added to Appendix N.1 of the revised paper, and Section 4.2 and Table 7a report the key numbers. Figure 2 originally illustrated all four question families, and several of its attribute and spatial examples belong to this subset. In the revised Figure 2, these examples are replaced with dependent test questions from LongHOI-Bench on the same images. The figure now reflects the dependent questions, which make up 82.4% of the multiple-choice questions. The dependence labels will be released with the benchmark.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Our Responses & Paper Revisions

We sincerely thank all the reviewers for their valuable and constructive reviews. We appreciate that the reviewers acknowledged that LongHOI-Bench addresses a meaningful problem (Reviewers 9PLY, KzjY), brings gaze, pointing, and contextual intention under one evaluation framework (Reviewers 9PLY, Ze5B), provides human-verified targets with agreement statistics (Reviewers 9PLY, KzjY, Ze5B), and uses typed negative alternatives that reveal why a model grounds the wrong object (Reviewers 9PLY, Ze5B). The reviewers also recognized the thoughtful construction and evaluation controls (Reviewer KzjY), the practical training-free design of CLASP (Reviewers KzjY, Ze5B), the extensive ablations (Reviewers 9PLY, KzjY), the consistent gains across backbones and external datasets (Reviewers 9PLY, Ze5B), and the detailed appendices (Reviewer Ze5B).

We have made our best effort to address the concerns and revise the paper accordingly. The major modifications are summarized as follows.

  1. Relation dependence of the questions (Reviewers 9PLY, KzjY): We have annotated relation dependence for all 7,524 multiple-choice test questions (Cohen's κ = 0.855). Of these questions, 82.4% are relation-dependent. All 13 models score 13.7 to 32.1 points lower on them than on relation-independent questions, while humans score only 1.5 points lower (Table 7a and Table A17). When the designated person is swapped on 237 multi-person images, InternVL3.5-8B and GPT-4o return the same answer for both persons in 48.1% and 33.8% of pairs, against 1.7% for humans (Table 7a and Table A18). Figure 2 is revised accordingly.

  2. Localization protocol (Reviewer Ze5B): We report the invalid-output rate of every model (Table A20), evaluate the two named models under their native coordinate conventions (Table 7b and Table A21), and locate the highest-confidence boxes of the frozen models (Table A22). CLASP retains about 90% of its AUC gain under native conventions, and the near-chance AUC of the frozen models comes from boxes that miss the target.

  3. Additional baselines and models (Reviewers 9PLY, Ze5B): We have added supervised fine-tuning and linear probing (Table A23), two-stage grounding with Florence-2 and Grounding DINO (Table A24), and four recent MLLMs including GPT-6.1 Sol and Claude Fable 5.2 (Table A19).

  4. Analysis of CLASP (Reviewers 9PLY, Ze5B): We have added validation-based rank selection without error-type labels (Table A25) and layer-matched subspace estimation (Table A26), and we have shown that the CLASP gain persists under native coordinate conventions (Table A21).

  5. PEFT configuration (Reviewers 9PLY, KzjY): We have corrected the hidden width and the trainable parameter counts in Appendix H.1 and removed the corresponding statement in Appendix M.3. All PEFT baselines use the same InternVL3.5-8B checkpoint as CLASP, and all reported results remain unchanged.

  6. Presentation (Reviewer Ze5B): We have defined ALS and LDR, simplified Figure 3, added a training-setting column to Table A12, and reported the key ablation numbers in Section 5.2.

The major revised contents in the manuscript are highlighted in blue, and the new results are collected in Appendix N, with the key results summarized in Table 7 of the main text. Point-for-point responses to specific comments are given in the following reviewer-specific responses. We welcome any further discussions and will address any remaining concerns.

AuthorsOpenReview ↗

Summary of the Author-Reviewer Discussion

We thank the reviewers and the AC for their time and the constructive discussion. After the discussion, all three reviewers recommend acceptance, and Reviewer Ze5B has raised the rating from 2 to 3. All three reviewers state that the rebuttal largely addresses their concerns. We summarize below how the main concerns were addressed, including the experiments added during the discussion.

  1. Construct validity (Reviewers KzjY, 9PLY): LongHOI-Bench measures person-target grounding. Of the multiple-choice questions, 82.4% depend on this relation, and the model deficit concentrates on them, with an average drop of 17.7 points against 1.5 points for humans. Swapping the designated person confirms that models often ignore the person, returning the same answer for both persons in up to 48.1% of pairs against 1.7% for humans.

  2. Comparison with trained baselines (Reviewers KzjY, 9PLY, Ze5B): On the same backbone and at equal training data, CLASP leads both in grounding and in preserving general capability. All PEFT baselines share the InternVL3.5-8B checkpoint of CLASP, and the reported results remain unchanged after the hidden width in Appendix H.1 is corrected. Following the suggestions of Reviewer Ze5B, we further compared CLASP with LoRA, DoRA, full SFT, and linear probing on the same 1,491 images. CLASP reaches the highest AUC, mAP, and ACC and retains the most general capability, while full SFT loses more than one point on both POPE F1 and MMBench. These results will be added to Table A23 in the final version.

  3. Coordinate format (Reviewer Ze5B): The gain of CLASP holds across coordinate formats. Under native conventions, CLASP retains about 90% of its AUC gain without recalibration, which Reviewer Ze5B considers convincing. It also stays ahead of LoRA and DoRA retrained under the native format, and these results will be added to Table A21 in the final version.

  4. Diagnosis and baselines (Reviewers 9PLY, Ze5B): Error-type diagnosis selects the best test rank, which validation search without these labels misses. CLASP also outperforms two-step grounding with Florence-2 and Grounding DINO, and the strongest recent MLLM, Claude Fable 5.2, remains at least 18.2 points below humans.

Core contributions. LongHOI-Bench brings gaze, pointing, and contextual intention under one evaluation framework, with human-verified targets and typed negative alternatives that reveal why a model grounds the wrong object. All three reviewers highlight the benchmark as a strength. CLASP turns this error-type diagnosis into a training-free edit that leads trained baselines at equal data, preserves general capability, and transfers to external gaze and pointing datasets without recalibration.

We thank the reviewers again for the discussion, which has substantially strengthened the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.