- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies left/right understanding in the depicted person’s own reference frame rather than screen coordinates. The authors introduce LR-Bench, with 8,048 image and video QA pairs, and use mirror-paired evaluation to examine model errors. They propose Spatial Latent Prompt (SLP), which encodes part maps from an external human mesh model and injects them into frozen visual tokens. Trained on LR QA data with fewer than 0.1M new parameters, SLP substantially improves accuracy and mirror-paired scores across three backbones, and improves left/right descriptions in captions.
Critical strengths
-
LR-Bench covers both images and videos, with questions about appearance, action and pose, spatial relations, and temporal dynamics. All test answers are human-annotated and independently checked, and the questions explicitly use the person’s own reference frame (Section 3).
-
The evaluation checks whether both original and mirrored inputs are answered correctly, as well as how often the answer stays unchanged after mirroring (Section 3.4 and Table 3). These measures reveal error patterns that a single accuracy score would miss.
-
SLP produces large gains on Qwen3-VL, InternVL, and GLM backbones (Table 2), while keeping the backbone frozen and training fewer than 0.1M new parameters. Table 4 also checks the contributions of the semantic and positional channels separately.
Critical weaknesses
-
SLP injects anatomical left/right labels predicted by an external mesh model into visual tokens, so an incorrect MPM can also mislead the model. Appendix C.4 and Figure 8 show such failures, but the paper does not quantify how MPM errors affect training or inference. There is also no mechanism to reduce or disable the injection when mesh confidence is low. I cannot tell how sensitive SLP is to incorrect priors, or whether it can perform worse than the original model under heavy occlusion or unusual poses.
-
The SFT and LoRA baselines in Table 5 use the same training data but receive no MPM, while the visual-prompt baselines in Table 6 receive structural information without corresponding adaptation training. SLP receives both. These comparisons do not fully separate the effects of the prior, training, and injection interface. I would like to see alternative interfaces trained with the same MPM and QA data. As it stands, the results show that the full pipeline works, but do not establish that token-level injection itself is better.
-
The positional encoding in Section 4.2 and Appendix B.1 measures offsets from the joint centroid of all human regions, without rotating the coordinates to match the queried person’s orientation. With multiple people, this centroid is not specific to the person being asked about. Table 4 reports the positional channel’s overall gain, but does not break down its contribution by viewpoint or person count; Table 9 compares the full SLP with the backbone. It remains unclear how much this encoding helps when the model needs to distinguish different people’s reference frames.
Questions for authors
-
Have you checked SLP’s performance separately when the MPM is correct and when it is wrong? In particular, can incorrect left/right labels make SLP perform worse than the original backbone? What does the available analysis tell us about its sensitivity to these errors?
-
Do you have results for other injection interfaces trained with the same MPM and 120k QA data? The baselines in Table 5 receive no MPM, and those in Table 6 receive no corresponding adaptation training. What evidence helps separate the benefit of the structural prior from that of token-level injection?
-
In multi-person scenes, why is the joint centroid of all people a suitable positional reference for the person being asked about? Do you have viewpoint- and person-count breakdowns comparing semantic-only SLP with the full method, to show where the positional channel actually helps?
I’m currently on the fence between weak reject and weak accept. If the authors can address my main concerns convincingly, I would be open to raising my score.
4 comments
Rebuttal from authors #2
W2 & Q2: Separating prior, training, and interface
Summary. With the prior, the data, and the training schedule all held fixed, SLP still outperforms every trained pixel-level interface, while training 222× fewer parameters at 3.0–3.6× lower cost. The difference is larger on open-ended captioning, where all three pixel + LoRA variants fall below the frozen backbone they started from.
We agree that this three-way attribution is the key question, and all three factors can now be separated explicitly. The paper already isolates two of them: training without the prior (Table 5: full-parameter SFT and LoRA on the same 120K QA data reach 59–62%, far below SLP) and the prior without training (Table 6: the same L/R labels rendered as visual prompts for the fully frozen backbone gain at most ~7 points). The missing cell is the same prior with training, injected through an alternative interface. We have now run exactly this control (New Experiment A; Table 7 of the revised PDF): using the same mesh-recovery output, the same 120K QA pairs, and the same optimization schedule, we trained three pixel-level interfaces with LoRA:
| Injection interface | LR-Img | LR-Vid | Trainable params | Train cost |
|---|---|---|---|---|
| MPM rendering as visual prompt + LoRA | 78.90 | 76.01 | 21.82M | 187.2 GPU·h |
| Pose keypoint as visual prompt + LoRA | 70.15 | 68.42 | 21.82M | 156.7 GPU·h |
| Part boxes as visual prompt + LoRA | 74.93 | 73.65 | 21.82M | 158.2 GPU·h |
| SLP (ours) | 79.48 | 78.69 | 98,304 (<0.1M) | 51.4 GPU·h |
With the prior, the data, and the schedule all held fixed, token-level latent injection stays ahead of every trained alternative while training 222× fewer parameters at 3.0–3.6× lower cost. The strongest pixel variant (MPM rendering) narrows the QA gap on images, but the difference is clearer outside the QA format, on the caption-based evaluation (New Experiment B):
| Model | Image Cov / Prec | Video Cov / Prec |
|---|---|---|
| Frozen backbone (Qwen3-VL-8B) | 68.20 / 64.81 | 67.42 / 70.05 |
| MPM visual prompt + LoRA | 59.11 / 69.90 | 70.51 / 64.08 |
| Pose keypoints + LoRA | 61.69 / 65.50 | 72.20 / 66.51 |
| Part boxes visual prompt + LoRA | 65.57 / 62.61 | 68.37 / 65.14 |
| SLP (ours) | 67.14 / 79.19 | 67.76 / 82.88 |
We interpret this degradation as a disruption of the model's generalization ability caused by the change of the original model parameters. Trained on identical QA data, all three pixel-prompt + LoRA variants overfit the QA format data and end up with even lower caption performance than the frozen backbone (original Qwen3-VL-8B) (64.08–66.51 vs. 70.05), while SLP reaches 82.88 without changing backbone weights.
Putting the three factors together on LR-Vid:
- the data without the prior reaches 61.48 (Table 5), and the prior without training reaches 57.32 (Table 6);
- the prior with training through the best pixel interface reaches 76.01;
- the same prior through token-level latent injection reaches 78.69, with the fewest trainable parameters and the backbone's general ability intact.
The interface is therefore not an implementation detail: with prior and data held fixed, how the prior is injected determines both the remaining gain and whether the model's existing capabilities are preserved.
Rebuttal from authors #3
W3 & Q3: The positional reference in multi-person scenes
Summary. The positional channel improves every person-count group and every viewpoint group, adding +9.62, +7.24 and +4.68 on Spatial Relation for one, two, and three or more people. A foreground-level anchor without any per-person conditioning is therefore already effective in multi-person scenes. Moreover, this design is also for the sake of engineering simplicity.
Design motivation. The positional channel is a deliberate design choice, not an approximation we settled for. A fully queried-person-specific centroid would require, at inference time, (i) parsing the question to identify the target person, (ii) matching that textual referent to a specific mesh instance, and (iii) recovering and centroiding that instance alone. This is a text-conditioned, per-person pipeline whose stages depend on one another. We instead run mesh recovery once per image, with no dependence on the question text and no per-person isolation, and anchor the positional frame on the foreground as a whole. This keeps the system simple and its stages decoupled. As the breakdown below shows, the foreground-level relative-position signal is already effective. Meanwhile, the fine-grained person identity is carried by the semantic channel.
Requested breakdown. Comparing semantic-only (SE) against full SLP on LR-Vid:
By person count
| Person count | # QA | Overall SE → Full | Δ | Spatial # QA | Spatial SE → Full | Δ |
|---|---|---|---|---|---|---|
| 1 | 2,474 | 76.56 → 81.04 | +4.48 | 520 | 65.19 → 74.81 | +9.62 |
| 2 | 1,771 | 73.46 → 77.13 | +3.67 | 373 | 61.39 → 68.63 | +7.24 |
| ≥3 | 504 | 69.64 → 72.62 | +2.98 | 107 | 57.94 → 62.62 | +4.68 |
By viewpoint of the queried person
| Viewpoint | # QA | Overall SE → Full | Δ | Spatial # QA | Spatial SE → Full | Δ |
|---|---|---|---|---|---|---|
| Front (48.6%) | 2,308 | 77.77 → 82.11 | +4.34 | 486 | 66.05 → 75.10 | +9.05 |
| Side (41.1%) | 1,952 | 72.39 → 76.28 | +3.89 | 411 | 60.34 → 68.13 | +7.79 |
| Other incl. back (10.3%) | 489 | 69.12 → 72.19 | +3.07 | 103 | 59.22 → 65.05 | +5.83 |
- The positional channel delivers clear gains in every group: on Spatial Relation it adds +9.62 with a single subject, +7.24 with two people, and +4.68 with three or more, on top of consistent overall gains (+4.48 / +3.67 / +2.98). Even a foreground-level relative-position signal without any per-person conditioning is therefore effective in multi-person scenes, not merely harmless. As the reviewer anticipated, the gain is largest when the centroid coincides with the queried person and decreases as the scene gets more crowded, but it never vanishes or reverses, which supports the coarse anchor as a good trade-off between simplicity and effectiveness.
- The viewpoint breakdown follows the pattern reported in the paper's stratified analysis (Table 11; Table 9 in the original paper): the gain is largest on frontal views, where the viewer's frame is the mirror image of the subject's, and smallest on back and other views, where the two frames partly coincide.
This trade-off is now discussed in Appendix B.1 of the revised PDF.
We thank the reviewer for the helpful suggestions. We would be grateful if the reviewer could revisit the rating in light of this evidence, and we are glad to run further analyses during the discussion period.
We sincerely thank the reviewer for the constructive feedback. We hope our response has addressed your concerns. As the discussion deadline is approaching its end, we wanted to send a gentle final reminder. We fully respect your judgment and simply want to ensure no unresolved concern remains. Please let us know if anything further would be helpful.
Rebuttal from authors #1
W1 & Q1: Sensitivity to an incorrect mesh prior
Summary. We answer this with three new measurements. The MPM is almost never left/right-inverted, with no such case among 200 inspected images. SLP loses at most 1.3 points when the prior is locally imprecise, which is the error mode that actually occurs. A fully inverted prior drops accuracy to 29.71 / 29.16, which bounds the worst case, but this failure mode is very rare in practice (0 observed in 200 inspected samples).
(1) Relying on the prior is the design, not an overlooked vulnerability. Our diagnosis shows the backbone does not itself possess reliable subject-centered laterality, and plain fine-tuning on the same data does not install it (Table 5). Therefore, we choose to rely on an external structural prior, as any prior-injection method must. The informative questions then become: how often is the prior actually wrong, and how gracefully does SLP degrade when it is?
(2) How often is the prior left/right-inverted in practice? We manually inspected 200 randomly sampled images and found no case in which the MPM's anatomical left/right was inverted. Two factors contribute to this. First, the person-detection stage applies its own confidence threshold. Second, the mesh-recovery model (SAM3D-Body) generalizes well in the wild and is particularly reliable on exactly the laterality distinction that MLLMs lack.
Beyond this targeted inspection, we also re-examined SLP's remaining failure cases, including those documented in Appendix C.8 (C.4 in the original paper): none of them is attributable to a left/right-inverted MPM either. Both inspections point to the same conclusion: left/right inversions are very rare in practice, and the MPM provides a reliable prior for SLP. We state this in Sec. 4.1 of the revised PDF, with the full analysis in Appendix C.4.
(3) SLP is robust to the error mode that does occur: correctly-sided but imprecise part maps. What mesh recovery exhibits in practice is not side flipping but spatial imprecision: under heavy occlusion and unusual poses (the regime of Appendix C.8), the part map remains correctly sided but can be locally misplaced. We measure robustness to exactly this mode by swapping each visual patch's prior with a random patch within a radius of 1 / 2 / 3 / 5 patches (perturbation levels 1–4; patch size 16, input long side 512), averaged over repeated runs:
Images
Videos
Degradation is at most 1.3 points on images and 1.0 point on videos, leaving SLP more than 27 points above the frozen backbone throughout. Finally, as a deliberate stress test, we swap left and right everywhere in the MPM, and accuracy collapses to 29.71 / 29.16, below the frozen backbone (41.68 / 50.16). So the direct answer to the reviewer's question is yes: with a fully inverted prior, SLP is worse than the original model. This is the expected behavior of a method that actually uses its prior rather than ignoring it, and it bounds the worst case. Taken together, SLP is robust in the error regime that actually occurs (imprecise but correctly-sided priors), and the only failure regime is adversarially constructed and, as shown in (2), does not arise in practice. These analyses are added to the revised appendix (C.4, Sensitivity to the Mesh Prior), with the remaining qualitative failure cases in Appendix C.8 (Failure Cases).