acceptodds
Under review as a conference paper at ICLR 2027

LR-Bench: Diagnosing and Mitigating Left-Right Confusion in MLLMs for Human-Centric Spatial Understanding

Abstract

Human subject-centered left-right (LR) discrimination, i.e., answering “which is his left hand” in the depicted person's own frame of reference (FoR) rather than in screen coordinates, is a fundamental capability for human-centric vision-language applications. We find that current multimodal large language models (MLLMs) fail at this task in a systematic way. Most open-source models score near chance, and for most mirror pairs their prediction stays the same after the input is horizontally flipped, which means the errors are systematic rather than random. To diagnose this failure, we introduce LR-Bench, a benchmark of 8,048 human-annotated binary QA pairs over 2,942 in-the-wild images and 1,809 video clips, spanning appearance, action & pose, spatial relation, and temporal dynamics, and evaluated with binary accuracy, a mirror-paired consistency score, and an LLM-judged caption protocol. Beyond evaluation, we propose Spatial Latent Prompt (SLP), a plug-in that injects an input-derived human structural prior, a Mesh-Derived Part Map from an off-the-shelf mesh recovery model, into frozen visual tokens through learnable part embeddings and a person-relative positional channel. With fewer than 0.1M trainable parameters and a frozen backbone, SLP improves three MLLM backbones by 36.9–48.7 points on images and 27.9–35.8 points on videos, and clearly outperforms full-parameter SFT and LoRA trained on the same data. The gain therefore comes from how the structural prior is represented and injected, not simply from in-domain supervision. SLP thus establishes a lightweight, parameter-efficient paradigm for human-centric spatial perception in MLLMs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human-Centric Spatial Understanding, Subject-Centered Left-Right Discrimination

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RjysOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies left/right understanding in the depicted person’s own reference frame rather than screen coordinates. The authors introduce LR-Bench, with 8,048 image and video QA pairs, and use mirror-paired evaluation to examine model errors. They propose Spatial Latent Prompt (SLP), which encodes part maps from an external human mesh model and injects them into frozen visual tokens. Trained on LR QA data with fewer than 0.1M new parameters, SLP substantially improves accuracy and mirror-paired scores across three backbones, and improves left/right descriptions in captions.

Critical strengths
  1. LR-Bench covers both images and videos, with questions about appearance, action and pose, spatial relations, and temporal dynamics. All test answers are human-annotated and independently checked, and the questions explicitly use the person’s own reference frame (Section 3).

  2. The evaluation checks whether both original and mirrored inputs are answered correctly, as well as how often the answer stays unchanged after mirroring (Section 3.4 and Table 3). These measures reveal error patterns that a single accuracy score would miss.

  3. SLP produces large gains on Qwen3-VL, InternVL, and GLM backbones (Table 2), while keeping the backbone frozen and training fewer than 0.1M new parameters. Table 4 also checks the contributions of the semantic and positional channels separately.

Critical weaknesses
  1. SLP injects anatomical left/right labels predicted by an external mesh model into visual tokens, so an incorrect MPM can also mislead the model. Appendix C.4 and Figure 8 show such failures, but the paper does not quantify how MPM errors affect training or inference. There is also no mechanism to reduce or disable the injection when mesh confidence is low. I cannot tell how sensitive SLP is to incorrect priors, or whether it can perform worse than the original model under heavy occlusion or unusual poses.

  2. The SFT and LoRA baselines in Table 5 use the same training data but receive no MPM, while the visual-prompt baselines in Table 6 receive structural information without corresponding adaptation training. SLP receives both. These comparisons do not fully separate the effects of the prior, training, and injection interface. I would like to see alternative interfaces trained with the same MPM and QA data. As it stands, the results show that the full pipeline works, but do not establish that token-level injection itself is better.

  3. The positional encoding in Section 4.2 and Appendix B.1 measures offsets from the joint centroid of all human regions, without rotating the coordinates to match the queried person’s orientation. With multiple people, this centroid is not specific to the person being asked about. Table 4 reports the positional channel’s overall gain, but does not break down its contribution by viewpoint or person count; Table 9 compares the full SLP with the backbone. It remains unclear how much this encoding helps when the model needs to distinguish different people’s reference frames.

Questions for authors
  1. Have you checked SLP’s performance separately when the MPM is correct and when it is wrong? In particular, can incorrect left/right labels make SLP perform worse than the original backbone? What does the available analysis tell us about its sensitivity to these errors?

  2. Do you have results for other injection interfaces trained with the same MPM and 120k QA data? The baselines in Table 5 receive no MPM, and those in Table 6 receive no corresponding adaptation training. What evidence helps separate the benefit of the structural prior from that of token-level injection?

  3. In multi-person scenes, why is the joint centroid of all people a suitable positional reference for the person being asked about? Do you have viewpoint- and person-count breakdowns comparing semantic-only SLP with the full method, to show where the positional channel actually helps?

I’m currently on the fence between weak reject and weak accept. If the authors can address my main concerns convincingly, I would be open to raising my score.

4 comments
AuthorsOpenReview ↗

Rebuttal from authors #1

W1 & Q1: Sensitivity to an incorrect mesh prior

Summary. We answer this with three new measurements. The MPM is almost never left/right-inverted, with no such case among 200 inspected images. SLP loses at most 1.3 points when the prior is locally imprecise, which is the error mode that actually occurs. A fully inverted prior drops accuracy to 29.71 / 29.16, which bounds the worst case, but this failure mode is very rare in practice (0 observed in 200 inspected samples).

(1) Relying on the prior is the design, not an overlooked vulnerability. Our diagnosis shows the backbone does not itself possess reliable subject-centered laterality, and plain fine-tuning on the same data does not install it (Table 5). Therefore, we choose to rely on an external structural prior, as any prior-injection method must. The informative questions then become: how often is the prior actually wrong, and how gracefully does SLP degrade when it is?

(2) How often is the prior left/right-inverted in practice? We manually inspected 200 randomly sampled images and found no case in which the MPM's anatomical left/right was inverted. Two factors contribute to this. First, the person-detection stage applies its own confidence threshold. Second, the mesh-recovery model (SAM3D-Body) generalizes well in the wild and is particularly reliable on exactly the laterality distinction that MLLMs lack.

Beyond this targeted inspection, we also re-examined SLP's remaining failure cases, including those documented in Appendix C.8 (C.4 in the original paper): none of them is attributable to a left/right-inverted MPM either. Both inspections point to the same conclusion: left/right inversions are very rare in practice, and the MPM provides a reliable prior for SLP. We state this in Sec. 4.1 of the revised PDF, with the full analysis in Appendix C.4.

(3) SLP is robust to the error mode that does occur: correctly-sided but imprecise part maps. What mesh recovery exhibits in practice is not side flipping but spatial imprecision: under heavy occlusion and unusual poses (the regime of Appendix C.8), the part map remains correctly sided but can be locally misplaced. We measure robustness to exactly this mode by swapping each visual patch's prior with a random patch within a radius of 1 / 2 / 3 / 5 patches (perturbation levels 1–4; patch size 16, input long side 512), averaged over repeated runs:

Images

LevelOverallAppearanceAction & PoseSpatial
0 (clean)79.4881.4784.3672.60
178.81±0.1881.02±0.3281.73±0.3373.17±0.96
278.78±0.6180.93±0.8181.98±0.8873.00±0.70
378.39±0.3781.13±0.2980.89±0.1472.27±0.84
478.25±0.2380.60±0.2481.23±0.3872.33±0.45

Videos

LevelOverallAppearanceAction & PoseSpatialTemporal
0 (clean)78.6984.0986.7471.2065.69
178.24±0.1283.64±0.5786.21±0.7571.03±0.2565.10±0.24
278.04±0.2983.51±0.7886.01±0.6871.10±0.8764.57±0.70
378.10±0.3483.88±0.7286.06±0.3871.20±0.8764.26±1.13
477.76±0.6483.23±0.5785.48±0.7271.33±0.6464.19±0.97

Degradation is at most 1.3 points on images and 1.0 point on videos, leaving SLP more than 27 points above the frozen backbone throughout. Finally, as a deliberate stress test, we swap left and right everywhere in the MPM, and accuracy collapses to 29.71 / 29.16, below the frozen backbone (41.68 / 50.16). So the direct answer to the reviewer's question is yes: with a fully inverted prior, SLP is worse than the original model. This is the expected behavior of a method that actually uses its prior rather than ignoring it, and it bounds the worst case. Taken together, SLP is robust in the error regime that actually occurs (imprecise but correctly-sided priors), and the only failure regime is adversarially constructed and, as shown in (2), does not arise in practice. These analyses are added to the revised appendix (C.4, Sensitivity to the Mesh Prior), with the remaining qualitative failure cases in Appendix C.8 (Failure Cases).

AuthorsOpenReview ↗

Rebuttal from authors #2

W2 & Q2: Separating prior, training, and interface

Summary. With the prior, the data, and the training schedule all held fixed, SLP still outperforms every trained pixel-level interface, while training 222× fewer parameters at 3.0–3.6× lower cost. The difference is larger on open-ended captioning, where all three pixel + LoRA variants fall below the frozen backbone they started from.

We agree that this three-way attribution is the key question, and all three factors can now be separated explicitly. The paper already isolates two of them: training without the prior (Table 5: full-parameter SFT and LoRA on the same 120K QA data reach 59–62%, far below SLP) and the prior without training (Table 6: the same L/R labels rendered as visual prompts for the fully frozen backbone gain at most ~7 points). The missing cell is the same prior with training, injected through an alternative interface. We have now run exactly this control (New Experiment A; Table 7 of the revised PDF): using the same mesh-recovery output, the same 120K QA pairs, and the same optimization schedule, we trained three pixel-level interfaces with LoRA:

Injection interfaceLR-ImgLR-VidTrainable paramsTrain cost
MPM rendering as visual prompt + LoRA78.9076.0121.82M187.2 GPU·h
Pose keypoint as visual prompt + LoRA70.1568.4221.82M156.7 GPU·h
Part boxes as visual prompt + LoRA74.9373.6521.82M158.2 GPU·h
SLP (ours)79.4878.6998,304 (<0.1M)51.4 GPU·h

With the prior, the data, and the schedule all held fixed, token-level latent injection stays ahead of every trained alternative while training 222× fewer parameters at 3.0–3.6× lower cost. The strongest pixel variant (MPM rendering) narrows the QA gap on images, but the difference is clearer outside the QA format, on the caption-based evaluation (New Experiment B):

ModelImage Cov / PrecVideo Cov / Prec
Frozen backbone (Qwen3-VL-8B)68.20 / 64.8167.42 / 70.05
MPM visual prompt + LoRA59.11 / 69.9070.51 / 64.08
Pose keypoints + LoRA61.69 / 65.5072.20 / 66.51
Part boxes visual prompt + LoRA65.57 / 62.6168.37 / 65.14
SLP (ours)67.14 / 79.1967.76 / 82.88

We interpret this degradation as a disruption of the model's generalization ability caused by the change of the original model parameters. Trained on identical QA data, all three pixel-prompt + LoRA variants overfit the QA format data and end up with even lower caption performance than the frozen backbone (original Qwen3-VL-8B) (64.08–66.51 vs. 70.05), while SLP reaches 82.88 without changing backbone weights.

Putting the three factors together on LR-Vid:

  • the data without the prior reaches 61.48 (Table 5), and the prior without training reaches 57.32 (Table 6);
  • the prior with training through the best pixel interface reaches 76.01;
  • the same prior through token-level latent injection reaches 78.69, with the fewest trainable parameters and the backbone's general ability intact.

The interface is therefore not an implementation detail: with prior and data held fixed, how the prior is injected determines both the remaining gain and whether the model's existing capabilities are preserved.

AuthorsOpenReview ↗

Rebuttal from authors #3

W3 & Q3: The positional reference in multi-person scenes

Summary. The positional channel improves every person-count group and every viewpoint group, adding +9.62, +7.24 and +4.68 on Spatial Relation for one, two, and three or more people. A foreground-level anchor without any per-person conditioning is therefore already effective in multi-person scenes. Moreover, this design is also for the sake of engineering simplicity.

Design motivation. The positional channel is a deliberate design choice, not an approximation we settled for. A fully queried-person-specific centroid would require, at inference time, (i) parsing the question to identify the target person, (ii) matching that textual referent to a specific mesh instance, and (iii) recovering and centroiding that instance alone. This is a text-conditioned, per-person pipeline whose stages depend on one another. We instead run mesh recovery once per image, with no dependence on the question text and no per-person isolation, and anchor the positional frame on the foreground as a whole. This keeps the system simple and its stages decoupled. As the breakdown below shows, the foreground-level relative-position signal is already effective. Meanwhile, the fine-grained person identity is carried by the semantic channel.

Requested breakdown. Comparing semantic-only (SE) against full SLP on LR-Vid:

By person count

Person count# QAOverall SE → FullΔSpatial # QASpatial SE → FullΔ
12,47476.56 → 81.04+4.4852065.19 → 74.81+9.62
21,77173.46 → 77.13+3.6737361.39 → 68.63+7.24
≥350469.64 → 72.62+2.9810757.94 → 62.62+4.68

By viewpoint of the queried person

Viewpoint# QAOverall SE → FullΔSpatial # QASpatial SE → FullΔ
Front (48.6%)2,30877.77 → 82.11+4.3448666.05 → 75.10+9.05
Side (41.1%)1,95272.39 → 76.28+3.8941160.34 → 68.13+7.79
Other incl. back (10.3%)48969.12 → 72.19+3.0710359.22 → 65.05+5.83
  • The positional channel delivers clear gains in every group: on Spatial Relation it adds +9.62 with a single subject, +7.24 with two people, and +4.68 with three or more, on top of consistent overall gains (+4.48 / +3.67 / +2.98). Even a foreground-level relative-position signal without any per-person conditioning is therefore effective in multi-person scenes, not merely harmless. As the reviewer anticipated, the gain is largest when the centroid coincides with the queried person and decreases as the scene gets more crowded, but it never vanishes or reverses, which supports the coarse anchor as a good trade-off between simplicity and effectiveness.
  • The viewpoint breakdown follows the pattern reported in the paper's stratified analysis (Table 11; Table 9 in the original paper): the gain is largest on frontal views, where the viewer's frame is the mirror image of the subject's, and smallest on back and other views, where the two frames partly coincide.

This trade-off is now discussed in Appendix B.1 of the revised PDF.


We thank the reviewer for the helpful suggestions. We would be grateful if the reviewer could revisit the rating in light of this evidence, and we are glad to run further analyses during the discussion period.

We sincerely thank the reviewer for the constructive feedback. We hope our response has addressed your concerns. As the discussion deadline is approaching its end, we wanted to send a gentle final reminder. We fully respect your judgment and simply want to ensure no unresolved concern remains. Please let us know if anything further would be helpful.

Reviewer_5DC4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies human subject-centered left-right understanding in multimodal large language models. It introduces LR-Bench, comprising 8,048 human-annotated QA pairs over 2,942 images and 1,809 video clips, covering appearance, action and pose, spatial relations, and temporal dynamics. The proposed Spatial Latent Prompt (SLP) constructs a Mesh-Derived Part Map using an off-the-shelf human mesh estimator. It adds learnable part embeddings to existing visual tokens and encodes background positions relative to the human foreground centroid, while keeping the backbone frozen. A separate adapter is trained for each backbone on the same 120K QA examples, with fewer than 0.1M trainable parameters per adapter. The authors report substantial improvements across three backbones and evaluate binary QA, mirror-paired behavior, and open-ended captioning.

Critical strengths
  1. LR-Bench provides a focused evaluation of human subject-centered left-right understanding across images and videos. Human annotation with independent verification and complementary QA, mirror-paired, and caption-based evaluations make it a useful benchmark for this capability.

  2. SLP achieves substantial gains across three backbones with fewer than 0.1M trainable parameters per adapter. For example, Qwen3-VL-8B's video accuracy rises from 50.16% to 78.69%. Component ablations and inference-cost measurements further support its practical value.

Critical weaknesses
  1. The adaptation settings of the visual-prompt baselines in Table 6 could be clarified further.

It is not entirely clear to me from Section 5.4 whether the visual-prompt baselines, particularly the MPM visual-prompt baseline, were adapted on the same 120K examples as SLP. Appendix B.3 only specifies the Table 5 baselines, and a frozen backbone alone does not rule out adaptation of other components. It would be helpful if the authors could report any trainable components and their training settings. If no adaptation was performed, the comparison changes both the interface and the adaptation procedure, so it may be worth qualifying the conclusion attributing the advantage specifically to the injection interface.

  1. Clarification of Coverage in Tables 7 and 9.

For Qwen3-VL-8B + SLP video captioning, Table 7 reports 67.76% overall Coverage, while Table 9 reports 67.33% for full visibility and 56.14% for partial visibility. If these groups form a complete, non-overlapping partition of the same evaluation set under Section 3.4, the overall value is a QA-count-weighted average and cannot exceed 67.33%. Could the authors clarify any subset or aggregation differences and provide each group's QA count and correct/incorrect/not-mentioned counts?

  1. External validation.

Evaluating the method on other relevant benchmarks would further strengthen the evidence for its effectiveness and generalization.

  1. Choice of GPT-4.1.

The inclusion of GPT-4.1 alongside newer models is somewhat unexpected. A brief explanation of the baseline selection would be helpful.

  1. Typographical correction.

On page 2, line 107, "¡0.1M" appears to be a typo for "<0.1M".

Questions for authors

Training data availability.

The reproducibility statement already promises LR-Bench annotations and source identifiers, evaluation scripts, SLP code, and adapter weights. Will the 120K training QA pairs, their source identifiers, and data-construction scripts also be released? A release timeline and instructions for obtaining or reconstructing the visual data would be helpful.

8 comments
AuthorsOpenReview ↗

Rebuttal from authors #1

W1: Adaptation settings of the Table 6 baselines

Summary. The Table 6 baselines contain no trainable component at all. Following the suggestion, we have also added the trained counterpart, in which the same prior and the same 120K QA pairs are supplied to pixel-level interfaces with LoRA. SLP remains ahead on both subsets with 222× fewer trainable parameters, and the difference is larger on caption-based evaluation.

Clarification. The Table 6 visual-prompt baselines are entirely training-free: the structural information is supplied purely through pixels, the backbone is fully frozen, and no other module is adapted. We have stated this explicitly in Section 5.4 of the revised PDF, and Appendix B.3 now reports the complete settings of every baseline.

The trained counterpart. Following the suggestion, we present the trained control (New Experiment A; Table 7 of the revised PDF). Using the same mesh-recovery output, the same 120K QA pairs, and the same optimization schedule, we trained three pixel-level interfaces with LoRA:

  • MPM rendering as visual prompt: the same part map as ours, rendered into pixels;
  • Pose keypoints as visual prompt: a 24-joint skeleton from the same mesh output, left/right color-coded;
  • Part boxes as visual prompt: part-level boxes, left/right color-coded.
Injection interfaceLR-ImgLR-VidTrainable paramsTrain cost
MPM rendering as visual prompt + LoRA78.9076.0121.82M187.2 GPU·h
Pose keypoint as visual prompt + LoRA70.1568.4221.82M156.7 GPU·h
Part boxes as visual prompt + LoRA74.9373.6521.82M158.2 GPU·h
SLP (ours)79.4878.6998,304 (<0.1M)51.4 GPU·h

With the prior and the data held fixed, token-level latent injection stays ahead of every trained alternative while training 222× fewer parameters at 3.0–3.6× lower cost. The difference is clearer outside the QA format, on the caption-based evaluation (New Experiment B):

ModelImage Cov / PrecVideo Cov / Prec
Frozen backbone (Qwen3-VL-8B)68.20 / 64.8167.42 / 70.05
MPM visual prompt + LoRA59.11 / 69.9070.51 / 64.08
Pose keypoints + LoRA61.69 / 65.5072.20 / 66.51
Part boxes visual prompt + LoRA65.57 / 62.6168.37 / 65.14
SLP (ours)67.14 / 79.1967.76 / 82.88

We interpret this degradation as a disruption of the model's generalization ability caused by the change of the original model parameters. Trained on identical QA data, all three pixel-prompt + LoRA variants overfit the QA format data and end up with even lower caption performance than the frozen backbone (original Qwen3-VL-8B) (64.08–66.51 vs. 70.05), while SLP reaches 82.88 without changing backbone weights.

AuthorsOpenReview ↗

Rebuttal from authors #2

W2: Coverage in Tables 7 and 9 (Tables 8 and 11 in the revised PDF)

Summary. This was a bookkeeping error confined to the stratified appendix table. The overall numbers in Table 8, which are computed directly on the full set, are unaffected, and no main result or training run is involved. The corrected table and the full per-group counts are given below, and all conclusions in the paper still hold.

We thank the reviewer for this careful cross-check. The inconsistency came from corrupted attribute metadata in the stratified bookkeeping. The overall numbers in Table 8 (Table 7 in the original paper), computed directly on the full set, are unaffected. Table 11 (Table 9 in the original paper) has been recomputed from the same LLM-judge outputs (Coverage / Precision, %; n = QA pairs per group, 4,749 in total):

Front (n=2,308)Side (n=1,952)Other (n=489)1 person (n=2,474)2 people (n=1,771)≥3 (n=504)Full vis. (n=1,008)Partial vis. (n=3,741)
Qwen3-VL-8B67.29 / 68.6467.93 / 69.4666.05 / 79.2666.90 / 69.7969.73 / 70.6961.90 / 68.9162.00 / 73.2868.89 / 69.27
+ SLP67.85 / 84.2367.78 / 81.6367.28 / 81.4668.71 / 85.6568.49 / 79.9760.52 / 79.0265.97 / 77.7468.24 / 84.21

As requested, each group's QA count and correct / wrong / not-mentioned counts are reported in full (Mentioned = Correct + Wrong; Coverage = Mentioned / #QA; Precision = Correct / Mentioned):

Group# QAQwen3-VL-8B — Ment. / Cor. / Wrong / Not ment.+ SLP — Ment. / Cor. / Wrong / Not ment.
Front2,3081,553 / 1,066 / 487 / 7551,566 / 1,319 / 247 / 742
Side1,9521,326 / 921 / 405 / 6261,323 / 1,080 / 243 / 629
Other489323 / 256 / 67 / 166329 / 268 / 61 / 160
1 person2,4741,655 / 1,155 / 500 / 8191,700 / 1,456 / 244 / 774
2 people1,7711,235 / 873 / 362 / 5361,213 / 970 / 243 / 558
≥3 people504312 / 215 / 97 / 192305 / 241 / 64 / 199
Full visibility1,008625 / 458 / 167 / 383665 / 517 / 148 / 343
Partial visibility3,7412,577 / 1,785 / 792 / 1,1642,553 / 2,150 / 403 / 1,188

All qualitative conclusions of the paper hold under the corrected table. Notably, the Precision gain is far larger on partially visible subjects (+14.9) than on fully visible ones (+4.5), in line with the paper's claim that the 3D body prior helps most where visual evidence is incomplete. Table 11 and the corresponding text (Appendix C.1) are updated in the revision.

W3: External validation

Summary. On the human subset of MirrorQA, an independently constructed benchmark, SLP improves its frozen backbone from 44.79 to 82.48 without using any MirrorQA training data, and is balanced across the two sides.

We evaluated SLP on the human subset of MirrorQA (Liu et al., ACL 2026), the subject-centered laterality benchmark (image-only), without any tuning using its training data:

ModelH. leftH. rightHuman overall
LLaVA-v1.6-7B (LoRA; MirrorQA paper)60.3375.1467.72
Qwen2.5-VL-7B (LoRA; MirrorQA paper)74.1036.1955.17
InternVL3-8B (LoRA; MirrorQA paper)66.3940.6153.52
mPLUG-Owl3-7B (LoRA; MirrorQA paper)66.6756.9161.79
Qwen3-VL-8B (frozen; our backbone)36.8852.7244.79
Qwen3-VL-8B + SLP (ours)78.51 (285/363)86.46 (313/362)82.48 (598/725)

MirrorQA reports per-side accuracy only. The Human-overall column for its baselines is computed by us from the reported per-side accuracies and the official test counts (363 left / 362 right), the same 725-question split we evaluate on. Notably, these baselines are LoRA-finetuned on MirrorQA's own training set, yet remain strongly side-imbalanced (e.g., 74.10 left vs. 36.19 right), which independently confirms our Table 5 finding that plain fine-tuning cannot learn subject-centered laterality. SLP, trained on our 120K LR QA pairs without any MirrorQA data (the source datasets of our training set share no images with MirrorQA), lifts its frozen backbone from 44.79 to 82.48 with both sides more balanced.

We would also note that SLP leaves the backbone frozen, so the model's general capabilities are not corrupted by construction. The caption-based results in New Experiment B show that SLP preserves open-ended captioning quality while the LoRA-based alternatives degrade it.

AuthorsOpenReview ↗

Rebuttal from authors #3

W4: Choice of GPT-4.1

GPT-4.1 was included simply as a widely reported closed-source reference that our evaluation pipeline already supported. We agree that a newer model makes the comparison more informative, so following the suggestion we additionally evaluated GPT-5.6 Luna [1]:

CategoryImageVideo
Appearance67.8063.02
Action & Pose62.8361.53
Spatial Relation70.8047.70
Temporal Dynamics—63.70
Overall67.5159.36

Even this recent proprietary model remains far below an 8B open-source backbone equipped with SLP (79.48 / 78.69), and scores below chance on video Spatial Relation (47.70). This confirms that LR-Bench measures an ability that model scale alone has not delivered. In the revised PDF, Table 2 now reports GPT-5.6 Luna in place of GPT-4.1.

[1] OpenAI. GPT-5.6 System Card. July 2026.

W5: Typographical correction

Thank you. This is a LaTeX rendering artifact from a literal < in text mode. It is corrected to $<$0.1M in the revised PDF, and we have checked the paper for the same pattern.

Q: Training data availability

Yes. In addition to LR-Bench, we plan to release the 120K training QA pairs, and also the complete data-construction scripts.


We thank the reviewer for the supportive assessment and the careful reading. We would be grateful if the reviewer could revisit the rating in light of this evidence, and we are glad to provide further analyses during the discussion period.

Reviewer_ztvsOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies subject-centered anatomical left/right discrimination in multimodal large language models. It introduces LR-Bench, with 8,048 human-annotated binary QA pairs over 2,942 images and 1,809 video clips, covering appearance, action/pose, spatial relations, and temporal dynamics. Evaluation includes binary accuracy, mirror-paired correctness, and caption coverage/precision. The proposed Spatial Latent Prompt (SLP) projects predictions from an off-the-shelf human mesh recovery model into a labeled body-part map, then adds learned part embeddings and human-relative background position encodings to frozen visual tokens. Fewer than 0.1M parameters are trained on approximately 120K QA examples. Across three backbones, the paper reports large improvements in QA accuracy and mirror-paired performance, with additional gains in caption precision.

Critical strengths
  1. A useful, focused benchmark with substantial human annotation. Combining images and videos, especially temporal laterality and person-relative relations to external objects, extends existing image-based laterality evaluations. The two-pass annotation process, reported 94% initial agreement, and balanced labels are meaningful strengths. Mirror-paired evaluation is particularly useful for checking whether good original-input accuracy survives a controlled change in laterality.

  2. Large, consistent improvements across multiple backbones. Table 2 reports gains of 27.9-35.8 percentage points on videos and 36.9-48.7 points on images. For Qwen3-VL-8B, video accuracy rises from 50.16% to 78.69%, and MCS from 9.18% to 59.21%. The same-data SFT/LoRA comparison and semantic/positional ablations are valuable, while caption results suggest benefits beyond the binary output format. These findings make the overall structural-prior adaptation approach worth further investigation.

Critical weaknesses
  1. The experiments do not isolate the claimed advantage of the injection interface. Table 5 trains SFT/LoRA on the same 120K examples but without the structural prior. Table 6 supplies structural visual prompts to the frozen backbone, whereas SLP has a trained adapter. Thus, the comparison changes both the interface and whether the model has been trained to use that interface. The missing condition is the same prior and training data with an appropriately trained alternative interface. The results support the effectiveness of the combined prior-plus-adaptation system, but not the stronger claim in the abstract and Section 5.4 that the gains are specifically due to the proposed representation/injection rather than this combination. This distinction matters because learned additive prompts in frozen VLMs already have precedents such as PIN.

  2. The specific value of the mesh-derived representation is not established against relevant alternatives. Section 4.1 states that human parsing maps, citing Sapiens, discard limb identity and anatomical left/right. However, Sapiens' official labels explicitly include left/right hands, upper/lower arms, and upper/lower legs [1]. Clothing and occlusion may still make mesh-based coverage advantageous, but this needs a controlled comparison using the same latent interface and training with left/right-aware parsing or rasterized pose priors. Table 6's pixel-prompt comparisons do not provide that control. Given prior subject-centered laterality benchmarks such as MirrorQA and prior positional-insert methods, establishing the contribution of the chosen anatomical representation is central to the paper's methodological novelty.

[1] Sapiens official label definitions: https://github.com/facebookresearch/sapiens/blob/main/lite/demo/classes_and_palettes.py PIN: Dorkenwald et al., CVPR 2024, https://arxiv.org/abs/2402.08657 MirrorQA: Liu et al., ACL 2026, https://aclanthology.org/2026.acl-long.1879/

Overall, the benchmark and reported gains are promising, but these unresolved controls leave me leaning toward weak rejection. Matched-interface and matched-representation evidence could change this assessment.

Questions for authors
  1. Clarification and decisive comparison: Were any Table 6 visual-prompt baselines trained on the same 120K data? If not, can you compare against a trained alternative using the same MPM input? Please also specify the SFT/LoRA learning rates, LoRA rank, and validation-based hyperparameter selection, which are not given in Appendix B.3.

  2. Decisive comparison: With the SLP architecture and training fixed, how does MPM compare with left/right-aware human parsing and rasterized pose/keypoint priors? Please clarify how the Section 4.1 characterization of parsing applies to Sapiens, and whether any advantage comes specifically from coverage of clothing or occluded anatomy.

  3. Clarification of the diagnostic claims: What evidence distinguishes image-independent priors from flip-insensitive visual features or target/orientation recognition failures? Likewise, paired errors do not uniquely identify viewer-centered reasoning. A question-only or explicit-frame-of-reference control could help; otherwise, would you qualify these causal interpretations? Please provide the data split, feature extraction, and training details for the Figure 1 linear probe.

7 comments
AuthorsOpenReview ↗

Rebuttal from authors #1

W1 & Q1: Isolating the injection interface

Summary. With the prior, the data, and the training schedule all held fixed, SLP still outperforms every trained pixel-level interface, while training 222× fewer parameters at 3.0–3.6× lower cost. The difference is larger on open-ended captioning, where all three pixel + LoRA variants fall below the frozen backbone they started from.

We thank the reviewer for the helpful suggestion. Tables 5 and 6 are designed as two complementary ablations, i.e., the same data without the prior, and the same prior without training. Following the suggestion, we additionally provide the results of the same prior + data + training below, and they further confirm the advantage of our method.

We trained the alternative interfaces the reviewer asks for (added as Table 7 of the revised PDF), using the same 120K QA pairs and the same number of optimization steps, with the same mesh-recovery output supplied in three pixel renderings:

  • MPM rendering as visual prompt: the same part map as ours, rendered into pixels;
  • Pose keypoint as visual prompt: 24-joint keypoint heatmap with different colors for left/right keypoint;
  • Part boxes as visual prompt: part-level box prompt with different colors for left/right parts.
Injection interfaceLR-ImgLR-VidTrainable paramsTrain cost
MPM rendering as visual prompt + LoRA78.9076.0121.82M187.2 GPU·h
Pose keypoint as visual prompt + LoRA70.1568.4221.82M156.7 GPU·h
Part boxes as visual prompt + LoRA74.9373.6521.82M158.2 GPU·h
SLP (ours)79.4878.6998,304 (<0.1M)51.4 GPU·h

With prior and data fixed, token-level latent injection remains ahead of the trained alternatives, with 222× fewer trainable parameters and 3.0–3.6× lower training cost. The strongest pixel-level variant (MPM rendering) narrows the QA gap on images, but the difference is more significant outside the in-domain QA format, i.e., on the caption-based evaluation below:

ModelImage Cov / PrecVideo Cov / Prec
Frozen backbone (Qwen3-VL-8B)68.20 / 64.8167.42 / 70.05
MPM visual prompt + LoRA59.11 / 69.9070.51 / 64.08
Pose keypoints + LoRA61.69 / 65.5072.20 / 66.51
Part boxes visual prompt + LoRA65.57 / 62.6168.37 / 65.14
SLP (ours)67.14 / 79.1967.76 / 82.88

We interpret this degradation as a disruption of the model's generalization ability caused by the change of the original model parameters. Trained on identical QA data, all three pixel-prompt + LoRA variants overfit the QA format data and end up with even lower caption performance than the frozen backbone (original Qwen3-VL-8B) (64.08–66.51 vs. 70.05), while SLP reaches 82.88 without changing backbone weights.

Overall, traditional visual prompt methods update backbone weights, which can degrade the model's generalization ability. Instead, our latent interface keeps the knowledge of the original model intact, which is exactly one of our motivations as shown in Figure 1.

We thank the reviewer for the PIN reference. PIN is discussed in Related Work (Sec. 2.1): it adds a learnable but input-agnostic spatial vector to the frozen visual features. The key difference is that the prompt in SLP is input-derived and human-structure-specific, supplying anatomical left/right semantics.

Finally, we would note that the contribution of this work extends beyond the method itself: it spans the diagnosis of the failure mode, the construction and human annotation of LR-Bench, and its evaluation protocols, with SLP as the accompanying mitigation.

AuthorsOpenReview ↗

Rebuttal from authors #2

Q1: Hyperparameter selection

Summary. The complete settings of every run are now reported in Appendix B.3, and the comparison is robust to the choice: across learning rates and ranks, the strongest trained baseline stays between 77.48 and 78.98, all below SLP's 79.48.

Appendix B.3 now reports the complete settings. All Qwen3-VL-8B runs use 2 epochs with an identical batch size. Every LoRA run uses rank 8 with learning rate 1e-4, applied to the q/k/v/o and gate/up/down projections of all LLM layers (21,823,488 trainable parameters):

MethodTrainable paramsInitial LRLoRA rankGPUs × wall-clockGPU-hours
SLP (part embeddings + positional projection)98,3041e-2—24 × 2.14 h51.4
MPM visual prompt + LoRA21,823,4881e-4824 × 7.80 h187.2
Pose keypoints + LoRA21,823,4881e-4824 × 6.53 h156.7
Dense boxes + LoRA21,823,4881e-4824 × 6.59 h158.2

For the Table 5 baselines (same 120K QA pairs, no MPM), full-parameter SFT uses initial learning rate 2e-5 and LoRA uses the same 1e-4 / rank 8 as above. All runs share batch size 192, AdamW (β = (0.9, 0.999), ε = 1e-8), weight decay 0, gradient clipping 1.0, cosine decay with a 0.05 warmup ratio, and inputs resized to a long side of 512. All settings are reported in Appendix B.3 of the revised PDF.

Hyperparameter sensitivity. Regarding validation-based selection, we adopted the standard fine-tuning configuration for this backbone, and verified that the comparison is robust to this choice across learning rates and ranks (with MPM visual prompt + LoRA):

SettingLR-Img
LR 5e-5, rank 877.48
LR 1e-4, rank 8 (default)78.90
LR 2e-4, rank 878.56
LR 1e-4, rank 1678.98
LR 2e-4, rank 1677.91
AuthorsOpenReview ↗

Rebuttal from authors #3

W2 & Q2: The value of the mesh-derived representation, and the characterization of Sapiens

Summary. The reviewer is right about the label set, and we have corrected the wording in the revised PDF. The substance of the claim is unchanged: clothing categories take precedence over limb categories on covered regions, which is the case for 52.14% of LR-Bench image QA pairs. Under the same SLP interface and the same training, the parsing-based prior reaches 73.72 against 79.48 for the MPM, and the gap concentrates on exactly the covered-limb group.

We thank the reviewer for careful checking. The taxonomy is right: Sapiens' label set does include left/right hands, upper/lower arms, and upper/lower legs. The word "discards" in Section 4.1 was imprecise, and we have corrected it in the revised PDF. The exact meaning we want to convey still holds: in Sapiens' output, clothing categories take precedence over limb categories on covered regions. Whenever the subject is not in short sleeves, arm pixels are labeled upper clothes rather than left/right arm, so anatomical left/right labels are unavailable exactly where they are needed. We include two qualitative examples of Sapiens' segmentation in the revised appendix (Fig. 5), which make this precedence effect directly visible: limb regions covered by clothing carry clothing labels, not left/right limb labels.

How often does this matter on LR-Bench? We annotated every image QA pair for whether the subject's lateralized limbs are mostly visible, i.e., not covered by clothing and not outside the frame (automatic annotation with DeepSeek-Flash v4.1 at the same 512 input resolution; 3,299 QA pairs in total), together with the QA accuracy (%) of the two priors under the same SLP interface:

Limb visibility# QAShareSapiens parsingMPM (ours)
Both forearms and lower legs mostly visible1,57947.86%77.0780.05
Limbs partly or fully covered (clothing / out of frame)1,72052.14%70.6478.95

In 52.14% of image QA pairs the lateralized limbs are at least partly covered, and wherever a limb is covered the parsing output carries no left/right label on it. This is the common case rather than a corner case. Consistently, the accuracy gap between the two priors concentrates on exactly this group: −8.3 points on covered limbs (70.64 vs. 78.95), against −3.0 where both limb pairs are visible (77.07 vs. 80.05).

Two factors therefore explain the matched-representation result (73.72 vs. 79.48 on LR-Img, New Experiment C; Appendix C.3 of the revised PDF). On covered limbs the parsing prior carries no laterality signal, by construction of its taxonomy. And even where limbs are visible it still trails, because the segmentation quality of Sapiens degrades in the wild, especially on loose clothing, a known limitation that its authors attribute to training on fitted-clothing data (repository issues #8 and #13). The MPM has neither problem, since it assigns left/right labels from 3D body geometry regardless of clothing. This difference is directly visible by comparing the new appendix examples (Fig. 5) with the MPM visualizations in Figure 4.

Regarding rasterized pose/keypoint priors: under the latent interface, sparse keypoints can label only a small fraction of the visual tokens, so they cannot supply the dense per-token semantics that the interface requires. This is the part of Section 4.1's description (sparsity and misalignment with the dense token grid) that we believe is not in dispute. We therefore compare the pose prior in its trained pixel form, i.e., a 24-joint skeleton rendered from the same mesh-recovery output, where all three representations share the same interface and the same training (New Experiment A): keypoint prompts reach 70.15 / 68.42, below part boxes (74.93 / 73.65) and MPM rendering (78.90 / 76.01). The mesh-derived representation is therefore the strongest choice under a matched trained interface as well.

MirrorQA is discussed in Related Work (Sec. 2.2), and we additionally evaluate SLP on it (New Experiment E and our response to Reviewer 5DC4 W3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.