- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces HandHeldBench, a diagnostic benchmark for reference-guided handheld product insertion—compositing a product reference into a photograph of a person's empty hand so that identity, hand–object contact, scale, and occlusion are jointly plausible. The authors call the failure mode the Joint Identity–Interaction Reliability Gap. HandHeldBench has two 1K tracks: controlled-synthetic (Qwen3-VL renders product/person/grasp/scene quartets; 50 samples per category across 20 categories) and real-captured (918 unique references from 5,000+ captures, with empty-hand sources reconstructed by a Qwen-Image-Edit-2511 LoRA). HandHeldEval uses six metrics—NEPS (non-edit preservation), RSE (target-area ratio), TS-KR (target-scale keypoint retention), A-HFSSIM (aligned high-frequency SSIM), HOIS (a seven-question Qwen3-VL-32B judge), and DINO (full-image semantic similarity)—weighted into an Overall rank. Beyond evaluation, the authors release HandHeld-55k (54,923 quartets) and adapt FLUX.1-Kontext-dev and Qwen-Image-Edit-2511 with rank-64 LoRAs. Twelve editors (ten native plus two adapted) are scored on both tracks.
Critical strengths
-
Two complementary tracks with frozen provenance. The synthetic track isolates category and generation-condition confounds (p.4, lines 181–184); the real-captured track preserves natural framing and fine text (p.5, lines 219–224). Table 6 (p.16) freezes per-track source/reference/target/mask provenance with immutable identifiers.
-
Reproducibility documentation is thorough. Appendix D (p.20–21) lists per-model prompts, step counts, guidance, mask handling, and render resolution; Table 10 (p.21) records seeds and API versions; the Reproducibility Statement (p.10, lines 500–505) commits to frozen manifests and audited run records.
Critical weaknesses
-
Only two annotators per image; no inter-annotator agreement reported. Section 4.3 (p.8, lines 413–414) and Appendix E.1 (p.22, line 1167) state only that two ratings are averaged into the image-level score. With two raters, disagreements are averaged away rather than measured, and the human anchor in Table 4 inherits this uncertainty.
-
HOIS uses a Qwen3-VL-32B judge while Qwen-family editors are evaluated. Section 3.3 (p.5, lines 263–266) derives HOIS from Qwen3-VL-32B logits on seven atomic questions (Table 7, p.17). Qwen-Image-Edit-2511 is itself one of the twelve evaluated editors and the adapted backbone. The paper does not discuss whether the judge is calibrated toward Qwen outputs, nor does it cross-check against an independent VL judge.
-
LoRA adaptation raises Overall slightly while degrading structure, detail, and interaction. On synthetic (Table 2), FLUX.1-Kontext-dev + LoRA lifts Overall by +2.58 and NEPS by +24.86, but TS-KR drops by −15.21, A-HFSSIM by −10.24, and HOIS by −8.41. The real-captured track shows the same pattern (Table 3): Overall +1.95, NEPS +21.82, but TS-KR −14.26 and A-HFSSIM −13.50. The LoRA fixes scale (RSE 1.974 → 0.959) and preserves the non-edit region, but at the cost of keypoint alignment, high-frequency detail, and hand–object plausibility
Questions for authors
None