acceptodds
Withdrawn from ICLR 2027

Identity Meets Interaction: Diagnosing the Joint Identity–Interaction Reliability Gap in Reference-Guided Handheld Product Insertion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advances in instruction and reference-guided image editing enable models to follow complex instructions, transfer the appearance of reference content, and generate compelling visual results. However, editing guided by a reference object image requires more than a plausible composite: the object's identity-defining details also need to be preserved. This requirement is especially important in e-commerce, where product text, logos, and other identity cues carry commercial value. Existing models and benchmarks primarily focus on instruction following and overall visual quality, leaving the joint satisfaction of product identity and plausible hand-object interaction largely outside quantitative evaluation. To fill this gap, we introduce HandHeldBench, a diagnostic benchmark comprising 1,000 controlled-synthetic samples and 1,000 real-captured samples. We further propose HandHeldEval, an evaluation protocol for product scale, structural and high-frequency detail, identity similarity, and hand-object interaction plausibility. Results across mainstream general-purpose and task-specific editors on HandHeldBench show that current models cannot consistently achieve both plausible hand-object interaction and faithful preservation of product identity details. We define this recurring limitation as the Joint Identity-Interaction Reliability Gap. Beyond benchmarking, we construct HandHeld-55k, a training dataset of 54,923 validated editing quartets, and study reference-guided product-detail restoration as a strong post-processing baseline. Together, these components provide a systematic foundation for analyzing identity preservation and physical interaction in handheld product insertion.

Primary area:
datasets and benchmarks
Keywords:
Image Editing, Reference-Guided Image Editing, Hand-Object Interaction, Image Editing Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bYiWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HandHeldBench, a diagnostic benchmark for reference-guided handheld product insertion—compositing a product reference into a photograph of a person's empty hand so that identity, hand–object contact, scale, and occlusion are jointly plausible. The authors call the failure mode the Joint Identity–Interaction Reliability Gap. HandHeldBench has two 1K tracks: controlled-synthetic (Qwen3-VL renders product/person/grasp/scene quartets; 50 samples per category across 20 categories) and real-captured (918 unique references from 5,000+ captures, with empty-hand sources reconstructed by a Qwen-Image-Edit-2511 LoRA). HandHeldEval uses six metrics—NEPS (non-edit preservation), RSE (target-area ratio), TS-KR (target-scale keypoint retention), A-HFSSIM (aligned high-frequency SSIM), HOIS (a seven-question Qwen3-VL-32B judge), and DINO (full-image semantic similarity)—weighted into an Overall rank. Beyond evaluation, the authors release HandHeld-55k (54,923 quartets) and adapt FLUX.1-Kontext-dev and Qwen-Image-Edit-2511 with rank-64 LoRAs. Twelve editors (ten native plus two adapted) are scored on both tracks.

Critical strengths
  • Two complementary tracks with frozen provenance. The synthetic track isolates category and generation-condition confounds (p.4, lines 181–184); the real-captured track preserves natural framing and fine text (p.5, lines 219–224). Table 6 (p.16) freezes per-track source/reference/target/mask provenance with immutable identifiers.

  • Reproducibility documentation is thorough. Appendix D (p.20–21) lists per-model prompts, step counts, guidance, mask handling, and render resolution; Table 10 (p.21) records seeds and API versions; the Reproducibility Statement (p.10, lines 500–505) commits to frozen manifests and audited run records.

Critical weaknesses
  • Only two annotators per image; no inter-annotator agreement reported. Section 4.3 (p.8, lines 413–414) and Appendix E.1 (p.22, line 1167) state only that two ratings are averaged into the image-level score. With two raters, disagreements are averaged away rather than measured, and the human anchor in Table 4 inherits this uncertainty.

  • HOIS uses a Qwen3-VL-32B judge while Qwen-family editors are evaluated. Section 3.3 (p.5, lines 263–266) derives HOIS from Qwen3-VL-32B logits on seven atomic questions (Table 7, p.17). Qwen-Image-Edit-2511 is itself one of the twelve evaluated editors and the adapted backbone. The paper does not discuss whether the judge is calibrated toward Qwen outputs, nor does it cross-check against an independent VL judge.

  • LoRA adaptation raises Overall slightly while degrading structure, detail, and interaction. On synthetic (Table 2), FLUX.1-Kontext-dev + LoRA lifts Overall by +2.58 and NEPS by +24.86, but TS-KR drops by −15.21, A-HFSSIM by −10.24, and HOIS by −8.41. The real-captured track shows the same pattern (Table 3): Overall +1.95, NEPS +21.82, but TS-KR −14.26 and A-HFSSIM −13.50. The LoRA fixes scale (RSE 1.974 → 0.959) and preserves the non-edit region, but at the cost of keypoint alignment, high-frequency detail, and hand–object plausibility

Questions for authors

None

Reviewer_JZVgOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a “Joint Identity–Interaction Reliability Gap” in reference-guided handheld product insertion: editors cannot reliably preserve a product’s identity details and also produce a plausible hand–object interaction in the same image. To diagnose it, the authors introduce HandHeldBench, a 2K benchmark split into a synthetic track and a real-captured track. They pair it with HandHeldEval, a six-metric protocol that scores scale, structure, fine detail, interaction plausibility and background preservation separately. They also build HandHeld-55k, a training set of about 55K quartets, and propose a localized product-detail restoration baseline. Experiments on 12 editors show that no model succeeds on both axes, fine-tuning narrows the gap without closing it.

Critical strengths
  1. The paper formalizes a real e-commerce gap, joint identity preservation and hand–object interaction, that existing benchmarks never evaluate on the same output.

  2. Reporting scale, identity, interaction and preservation separately exposes hidden failure modes, for example oversized products inflating identity scores, which makes the benchmark more useful than a single leaderboard.

Critical weaknesses
  1. HOIS, the main measure on the interaction axis, does not correlate significantly with human judgments on either track (ρ\rho = 0.44 and 0.52, with confidence intervals crossing zero), and all the correlations rest on only 12 models. That undermines the central claim that HandHeldEval can reliably diagnose the joint gap.

  2. Empty-hand sources, training data and the HOIS judge all come from Qwen models (Qwen-Image-Edit, Qwen-Image, Qwen3-VL). This may bias the benchmark toward Qwen, which makes the reported gains from Qwen + LoRA and the model rankings harder to trust.

Questions for authors
  1. How much does HOIS agree with human contact ratings at the instance level, not just across 12 models, and is it robust to swapping Qwen3-VL for a non-Qwen judge?
  2. When the empty-hand sources are reconstructed with a non-Qwen editor, do the model rankings and the gains from Qwen + LoRA stay the same?
  3. How large is the effect of the detail-restoration pipeline on its own, measured by training with versus without restored targets?
  4. Why are the Overall weights set as they are (for example, HOIS at 0.30), and do the rankings change under other reasonable weightings?
Reviewer_5BsNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the challenge of preserving product identity while generating realistic hand–object interactions in image editing. The authors introduce HandHeldBench, a 2,000-sample benchmark, HandHeldEval, a multi-dimensional evaluation framework, and HandHeld-55k, a training dataset. Experiments across 10 image editing models reveal persistent limitations in both identity preservation and interaction quality. Additional fine-tuning and human evaluations demonstrate the usefulness of the dataset and evaluation framework.

Critical strengths

The paper introduces a comprehensive training dataset, benchmark, and evaluation framework. In particular, the fine-grained evaluation protocol provides a more systematic and interpretable assessment of product identity preservation and hand–object interaction quality.

Critical weaknesses
  • Limited contributions in scale and methodology. The proposed training dataset contains 55K samples across 16 product categories, offering relatively limited diversity. Moreover, the data construction and restoration pipelines largely rely on existing models and techniques, with limited methodological novelty.

  • Potential bias in benchmark construction. Both benchmark tracks reconstruct empty-hand source images by removing products from existing handheld images. This may preserve grasp-specific hand poses and spatial layouts, potentially simplifying the task compared to real-world scenarios. The paper lacks evaluation on independently captured empty-hand images.

  • Insufficient validation of the restoration pipeline. Approximately 69% of the training targets are enhanced using HandHeldRefiner, but the paper provides limited quantitative evidence that the restored details faithfully preserve product identity. More importantly, the contribution of this restoration process to downstream training performance is not isolated through ablation studies.

  • Limited alignment with human evaluation. Although the authors conduct a human study, the proposed HOIS metric shows relatively weak correlations with human judgments (Spearman ρ = 0.44–0.52), with confidence intervals including zero. Furthermore, the validation only reports model-level correlations, without examining instance-level agreement or inter-rater reliability, which limits confidence in the proposed evaluation protocol.

Questions for authors
  • Can the authors provide a data scaling experiment (e.g., 10K, 25K, and 55K samples) to demonstrate how training performance benefits from the proposed dataset? How diverse are the products and hand–object interactions beyond the reported category counts?

  • Can the authors evaluate a subset of independently captured empty-hand images to verify that the reverse construction pipeline does not simplify the interaction synthesis task? A comparison with the current benchmark would help establish its real-world applicability.

  • Can the authors compare fine-tuning with original targets versus HandHeldRefiner-enhanced targets under identical training settings? Additionally, can they quantitatively verify whether the restored text and logos remain faithful to the reference products?

  • Can the authors report instance-level correlations between HandHeldEval metrics and human judgments, particularly for HOIS? Reporting inter-rater agreement would also help establish the reliability of the human annotations and the proposed evaluation protocol.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.