acceptodds
Under review as a conference paper at ICLR 2027

Reference-Image-Guided Relation Editing via Token Bottlenecking and Semantic Shift Alignment

Abstract

Existing relation-centric image editing methods mainly rely on textual or spatial priors, which often fail to capture fine-grained relational configurations such as contact locations and relative poses. Compared with textual or spatial guidance, a reference image provides more fine-grained visual relation cues. Therefore, we focus on reference-image-guided relation editing, which transfers the relation depicted in a single reference image to a source image. However, this task presents two challenges. (i) Appearance information is entangled with relation semantics, making it difficult to isolate transferable relation semantics from the reference image. This issue is particularly pronounced in unified multimodal models, where the target generation process can directly attend to reference visual representations. As a result, the generated image relies more on reference appearance than on the intended relation cues. (ii) Relation editing is prone to unintended relation-irrelevant semantic shifts that progressively accumulate during denoising because such edits require coordinated pose and spatial adjustments, whose representations in generative models are coupled with identity, background, and style. To address these challenges, we propose Relation-Aware Consistency-Preserving Editing (RACE), which (i) introduces Relation-Aware Token Bottleneck Attention (RTBA) with a limited-capacity relation-token bottleneck that induces competition between relation and appearance information to promote relation-appearance disentanglement; and (ii) proposes Cross-Modal Semantic Shift Alignment (CSSA) with a Semantic Shift Predictor (SSP). CSSA guides editing toward the intended relation shift by aligning the editing-induced image semantic shift with the intended relation-text semantic shift. Additionally, we construct a 35K-scale Relation Transfer Dataset (RTD-35K) for training. Extensive experiments demonstrate that RACE achieves superior relation transfer accuracy and source-image consistency, while showing applicability to more challenging settings, including multi-pair relation editing and cross-category relation transfer. Our code will be publicly available.

Primary area:
generative models
Keywords:
AIGC, relation transfer, image editing, diffusion models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sSjeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RACE (Relation-Aware Consistency-Preserving Editing) for reference-image-guided relation editing, aiming to transfer fine-grained object relations from a reference image to a source image while preserving its original appearance and identity. RACE introduces Relation-Aware Token Bottleneck Attention (RTBA) to disentangle relation semantics from reference appearance, and Cross-Modal Semantic Shift Alignment (CSSA) with a Semantic Shift Predictor (SSP) to suppress unintended semantic changes during denoising. The paper also constructs RTD-35K, a dataset containing 35K image triplets across 235 relation categories. Experiments on HICORTBench and RTBench demonstrate competitive performance in relation transfer, source-image preservation, and generation quality.

Critical strengths
  1. Well-motivated and practically relevant task. The paper addresses reference-image-guided relation editing, a challenging and practical task. It clearly identifies the limitations of existing guidance methods and highlights the challenges of relation–appearance entanglement and unintended semantic shifts.
  2. Intuitive and well-structured method design. The proposed RACE framework combines RTBA to reduce reference-appearance leakage and CSSA to suppress unintended semantic shifts. The two components are complementary and well-motivated.
  3. Extensive experiments and promising results. The paper constructs RTD-35K, covering 235 relation categories, and demonstrates competitive performance through quantitative comparisons, ablation studies, and qualitative results on challenging scenarios.
Critical weaknesses
  1. Insufficient clarity and fairness in baseline comparisons. The paper uses different types of relation guidance across baselines. While FLUX.1-Kontext and OneHOI are adapted to accept reference images and retrained on RTD-35K, other models, such as OmniGen2, DyRef, and Nano Banana, also support reference-image inputs. However, the paper does not clearly specify their actual input configurations. Moreover, relevant methods such as OrionEdit, FreeEvent, and ReVersion are only compared qualitatively. The paper should clarify the input conditions for each baseline and provide more comprehensive quantitative comparisons under comparable settings.

  2. Inconsistent relation transfer performance and potential trade-off introduced by CSSA. Although RACE achieves superior relation consistency on HICORTBench, it underperforms DyRef on RTBench (0.2985 vs. 0.3443), without explaining this discrepancy. Furthermore, the ablation study shows that introducing CSSA decreases relation consistency from 0.3141 to 0.2985, despite improving EI. This suggests a potential trade-off between relation transfer accuracy and source-image preservation. The paper could further analyze this trade-off across different relation categories and CSSA loss weights.

  3. Lack of a dedicated quantitative ablation for RTBA. Although the paper provides qualitative ablation results for RTBA, it lacks a dedicated quantitative ablation that isolates its contribution.

Questions for authors

Could the paper include comparisons with more recent and stronger commercial image editing models, such as GPT Image 2? Nano Banana is a relatively outdated baseline, and comparisons with more advanced models would provide a more comprehensive evaluation of RACE's competitiveness.

Related work on balancing semantic editing and source preservation. SynPS (The Devil is in Attention Sharing: Improving Complex Non-rigid Image Editing Faithfulness via Attention Synergy, CVPR 2026) shares a similar motivation with CSSA, as both aim to balance intended semantic modifications and source-image preservation. More specifically, both methods leverage source–target differences during denoising to characterize editing-induced changes, rather than relying solely on final generated images. Although their technical implementations differ, the paper would benefit from citing SynPS and discussing the similarities and differences between the two approaches.

6 comments
AuthorsOpenReview ↗

Response to Reviewer sSje (Part 1 / 4)

W1 & Q1: Clarifying Comparison Configurations and Expanding Comparisons

We thank the reviewer for raising concerns about the fairness and completeness of our baseline comparisons. We address these concerns through two aspects: (1) clarifying the input configurations of different comparison methods; and (2) expanding quantitative and qualitative comparisons with additional and more recent models.

1. Clarifying Input Configurations of Comparison Methods

Following your suggestion, we have revised Sec. 4.2 to explicitly clarify the input configurations of all comparison methods. Specifically, OmniGen2, DyRef, Nano Banana, FluxKontext (w/ ref.), Qwen-Image-Edit, HOIEdit, Baseline (w/o train), and Baseline are evaluated with reference-image inputs.

2. Expanding Quantitative and Qualitative Comparisons

Expanded Baselines and Evaluation. We expand the quantitative comparisons by including OrionEdit, FreeEvent, ReVersion, and GPT Image 2. In response to your suggestion, we have also incorporated more recent models, including Nano Banana 2 and the latest Qwen-Image-2.1. We additionally introduce VLM-R, a complementary VLM-based evaluation metric using Qwen3.8-27B to assess fine-grained visual relation consistency on a scale of 0–10.

Quantitative Comparison. We further compare RACE with the newly added methods on HICORTBench. All methods are evaluated with reference-image inputs, and the results are summarized in the table below (Table 10). RACE achieves the best performance across all six metrics.

MethodReference ImageRelation ConsistencyEIHPSv2ImageRewardPickScoreVLM-R
ControlNet + IP-Adapter✓0.47020.45880.2612-0.195320.83005.4932
ReVersion✓0.17420.27920.2544-0.610820.67513.2938
Inverted Interaction✓0.29870.28210.27500.186421.32175.1200
FreeEvent✓0.23900.18650.2086-1.016720.66973.8080
OrionEdit✓0.47030.43030.25920.198621.11826.1837
Qwen-Image-2.1✓0.25650.24340.2476-0.697920.58683.7885
Nano Banana 2✓0.42430.45830.26890.455821.44918.1534
GPT-Image-2✓0.45450.44500.26490.096221.21037.5385
Ours✓0.47130.45980.27510.456421.46028.4268

Qualitative Comparison. We additionally include qualitative comparisons with the latest open-source model, Qwen-Image-2.1, in Fig. 10. While Qwen-Image-2.1 captures the general interaction, it struggles to reproduce the reference-specific fine-grained configurations. In contrast, RACE better transfers the reference interactions while preserving source content. Fig. 11 further compares RACE with commercial models, including Nano Banana 2 and GPT-Image-2. Although these models generate visually plausible results, they often exhibit reference-appearance leakage. For example, Nano Banana 2 reproduces the star-shaped detail from the reference image in the third row, while GPT-Image-2 introduces the reference box into its outputs in the first and second rows. In contrast, RACE transfers the intended interactions without such appearance leakage.

Additional Manuscript Details. We have included the additional comparison results and corresponding analyses in Supplementary Sec. D.6.

AuthorsOpenReview ↗

Response to Reviewer sSje (Part 2/ 4)

W2: Explaining the CSSA Trade-off and Hyperparameter Sensitivity

We thank the reviewer for this insightful observation regarding the decrease in relation consistency after introducing CSSA. We address this observation by (1) explaining the relation–preservation trade-off and (2) analyzing the sensitivity of CSSA to training steps and loss weight.


1. Explaining the Relation–Preservation Trade-off

Trade-off Mechanism. As discussed in the fourth paragraph of Sec. 1, CSSA aims to preserve the original appearance of the source image by aligning editing-induced image semantic shifts with the intended textual relation shift. However, since textual relation descriptions cannot fully capture fine-grained visual configurations, this semantic constraint may also limit fine-grained relation transfer by suppressing some necessary structural adjustments.

Empirical Evidence. The ablation study in Table 2 further confirms the trade-off introduced by CSSA between relation transfer and source-appearance preservation. Introducing CSSA slightly reduces relation consistency (0.3141→0.2985) while improving EI (0.3708→0.3851). This reflects its intended effect of better preserving source appearance.


2. Analyzing the sensitivity of CSSA to training steps and loss weight

(i) CSSA Loss Weight: As the reviewer pointed out, the CSSA loss weight plays an important role in balancing relation transfer and appearance preservation. We report the effects of the CSSA loss weight in the table below (Table 8, Supplementary Sec. D.2). Specifically, a small weight (λCSSA=0.5\lambda_{\mathrm{CSSA}}=0.5) provides insufficient semantic alignment to suppress relation-irrelevant shifts. Conversely, an excessively large weight (λCSSA=3.5\lambda_{\mathrm{CSSA}}=3.5) overemphasizes semantic alignment at the expense of the flow-matching objective, thereby degrading source-image preservation and image quality. In comparison, λCSSA=2.0\lambda_{\mathrm{CSSA}}=2.0 achieves the best overall balance across the six metrics.

ParameterValuerelation consistency ↑EI ↑HPSv2 ↑ImageReward ↑PickScore ↑VLM-R↑
λCSSA\lambda_{CSSA}0.50.26720.33770.27000.722321.75937.1413
λCSSA\lambda_{CSSA}1.250.27390.37470.29981.045722.02148.0320
λCSSA\lambda_{CSSA}2.0 (Ours)0.29850.38510.29131.269922.41898.5579
λCSSA\lambda_{CSSA}2.750.28160.34530.29401.337821.84188.4322
λCSSA\lambda_{CSSA}3.50.29200.35740.26980.704521.85348.1857

(ii) CSSA Joint-Training Steps: We also investigate the effect of CSSA joint-training steps, as shown in the table below (Table 8). Increasing the number of joint-training steps from 600 to 1,200 improves EI (0.3821→0.3851). In contrast, extending training to 2,100 steps degrades both relation consistency and EI. We hypothesize that prolonged CSSA training may cause SSP to learn a shortcut. Specifically, SSP may directly predict relation changes rather than model the complete image semantic shift.

ParameterValuerelation consistency ↑EI ↑HPSv2 ↑ImageReward ↑PickScore ↑VLM-R↑
N2N_26000.31820.38210.27840.941321.93928.4224
N2N_29000.28710.38330.27870.904422.05438.6857
N2N_21,200 (Ours)0.29850.38510.29131.269922.41898.5579
N2N_215000.27430.37180.30780.819822.12348.3689
N2N_22,1000.23800.30450.27030.693221.71037.7293
AuthorsOpenReview ↗

Response to Reviewer sSje (Part 3 / 4)

W3: Quantitatively and Qualitatively Evaluating RTBA

We thank the reviewer for suggesting a dedicated evaluation of RTBA. We provide both quantitative and qualitative ablations to further demonstrate its contribution to relation transfer and source-content preservation.

1. Quantitative Ablation

Quantitative Results. Following your suggestion, we have added a w/o RTBA variant to Table 2 to isolate the contribution of RTBA to the overall performance of RACE, as shown in the table below.

MethodRelation Consistency ↑EI ↑HPSv2 ↑ImageReward ↑PickScore ↑VLM-R ↑
Baseline0.27700.33080.25350.491821.23885.7437
w/o RTBA0.27150.35370.27790.924621.89615.5786
w/o CSSA0.31410.37080.27480.901221.93318.4369
w/o SSP0.27710.34170.27100.769921.78964.6917
w/o R-CFG0.17820.26300.27200.677321.79993.6385
Ours0.29850.38510.29131.269922.41898.5579

Quantitative Results Analysis. As shown in the table, removing RTBA leads to consistent performance degradation across all six metrics. These results demonstrate the contribution of RTBA to relation transfer, source-content preservation, and overall image quality.

Reviewer_BPs5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RACE, a framework designed for Reference-Image-Guided Relation Editing. Addressing the limitation of existing methods in transferring fine-grained relations from a reference image while preserving source content, owing to relation-appearance entanglement and unintended relation-irrelevant semantic shifts during editing. It consists of two complementary components: Relation-Aware Token Bottleneck Attention (RTBA), which routes reference information through limited-capacity relation tokens to reduce reference-appearance leakage, and Cross-Modal Semantic Shift Alignment (CSSA), which uses a Semantic Shift Predictor to align editing-induced image semantic changes with the intended textual relation changes. The authors also construct RTD-35K, a training dataset containing 35K image triplets spanning 235 relation categories. Evaluated on HICORTBench and RTBench, RACE demonstrates improved relation transfer, source-content preservation, and visual quality compared with existing methods.

Critical strengths
  1. The framework’s strength lies in addressing both reference appearance leakage and unintended changes to source content, enabling relation transfer while better preserving source identity, appearance, and background.

  2. RTBA uses a relation-token bottleneck to extract transferable relation cues while restricting direct access to reference appearance. CSSA aligns the semantic shift predicted from the denoising process with the intended textual relation shift. Ablations on RTBench support the contribution of these designs: removing SSP reduces EI from 0.3851 to 0.3417 and ImageReward from 1.2699 to 0.7699, while removing CSSA reduces ImageReward to 0.9012.

Critical weaknesses
  1. The novelty of this paper appears somewhat limited, as extracting and transferring visual relations from reference images has been explored in previous studies. For example, ReVersion learns transferable relation representations from exemplar images and applies them to novel objects, backgrounds, and styles. OrionEdit introduces information-flow masking and disentanglement mechanisms to regulate cross-image dependencies during editing, which are conceptually related to the proposed RTBA. However, the paper does not sufficiently clarify the methodological differences between RTBA and these existing approaches, particularly OrionEdit. A more thorough discussion and, where feasible, a direct experimental comparison would help establish the novelty of the proposed method.

  2. The theoretical motivation behind RTBA is not sufficiently justified. In Section 3.2, the authors state that "The limited capacity encourages relation and appearance information to compete within the relation tokens, thereby promoting the disentanglement of transferable relation cues from reference appearance." However, restricting the information capacity of relation tokens does not inherently guarantee that relational information will be prioritized over appearance information. The model may instead retain appearance features that are more strongly correlated with the generation objective. Therefore, additional quantitative or qualitative analyses would be necessary to substantiate this claim.

  3. The proposed method is built upon Bagel, yet Table 1 does not include the original Bagel model as a standalone baseline. Specifically, the authors should consider reporting results for Bagel with reference-image conditioning, as well as Bagel fine-tuned on RTD-35K without the proposed components. The fine-tuned baseline should use matched training data, training steps, and, where possible, comparable trainable parameter budgets to ensure a fair comparison.

  4. The ablation results in Table 2 reveal an interesting inconsistency with the intended role of CSSA. Specifically, removing CSSA increases relation consistency from 0.2985 to 0.3141, although the complete model achieves better EI and image quality scores. The authors should explain why introducing CSSA reduces relation consistency and investigate whether this behavior is sensitive to the loss weight or other training configurations.

[1] ReVersion: Diffusion-Based Relation Inversion from Images [2] OrionEdit: Bridging Reference and Source Images for Generalized Cross-Image Editing

Questions for authors

The authors construct HICORTBench from HICO-DET as a test benchmark and evaluate their method against a wide range of models. However, the paper does not explain how HICORTBench is constructed from HICO-DET, making the experiments difficult to reproduce.

In Section 3.2, the authors introduce their model pipeline. However, the authors do not describe the specific architecture of SSP, but only specify its inputs and output in Eq. (4). This makes it difficult to understand how SSP estimates the editing-induced semantic shift.

Although the main notations are summarized in Table 3, some symbols are not defined when first introduced, which makes the formulation difficult to follow. For example, r_ref, r_src. r_tgt in Section 3.1 is not identified as the relation label or description in the subject–relation–object triplet, and L in RTBA is not explicitly defined as the token-sequence length, leaving the sequence and attention-mask dimensions unclear.

7 comments
AuthorsOpenReview ↗

Response to Reviewer BPs5 (Part 1 / 5)

W1: Clarifying the Novelty over ReVersion and OrionEdit

We thank the reviewer for raising this point. We have further strengthened our previous discussions of the differences between RACE and ReVersion [1] (Sec. 1, second paragraph, Fig. 1(c)), as well as the masking mechanisms of RACE and OrionEdit [2] (Supplementary Sec. D.3, Table 9).


1. Theoretical Comparison

Comparison with ReVersion and OrionEdit. ReVersion learns relation concepts from multiple exemplars and associates them with textual identifiers, as illustrated in Fig. 1(c). However, this inversion-based approach requires relation-specific optimization and multiple same-relation exemplars, which makes it difficult to scale across diverse relation configurations in practical scenarios. In contrast, OrionEdit uses attention masks that let target denoising tokens directly attend to reference visual tokens, thereby helping preserve reference appearance. However, as discussed in the third paragraph of Sec. 1, directly accessing the full reference visual representations may introduce reference-appearance leakage, which conflicts with our task objective.

Our Method. RTBA uses a capacity-limited relation-token bottleneck to extract transferable relation cues while restricting direct access to reference appearance. Meanwhile, CSSA aligns editing-induced image semantic shifts with intended textual relation shifts, thereby suppressing relation-irrelevant changes and preserving source content. Together, these complementary mechanisms enable fine-grained relation transfer while effectively preserving source-image content. The table below further compares RACE, ReVersion, and OrionEdit in terms of their task paradigm, reference information, and core mechanism.

ComparisonReVersionOrionEditRACE (Ours)
Task ParadigmMulti-exemplar relation concept customizationGeneral cross-image editing with reference featuresSingle-reference fine-grained relation transfer
Transferable InformationLearned relation conceptsDisentangled appearance featuresFine-grained relations
Core MechanismRelation concept learning + textual identifier bindingAppearance-feature disentanglementRelation–appearance disentanglement

2. Experimental Comparison with ReVersion and OrionEdit

Qualitative Comparisons. Figure 10 presents qualitative comparisons with ReVersion, OrionEdit, and other methods. ReVersion exhibits limited relation-transfer capability when provided with only a single reference image, whereas OrionEdit suffers from severe reference-appearance leakage. In contrast, RACE better transfers reference relations while preserving source content.

Quantitative Comparisons and Mask Ablation. As shown in Tables 10 and 11, RACE achieves the best results across all six metrics on HICORTBench and leads in EI, ImageReward, PickScore, and VLM-R on RTBench, demonstrating a strong overall balance between relation transfer, source preservation, and image quality. Furthermore, we compare RTBA with SIGMA-style and OrionEdit-style attention masks under identical settings in Supplementary Sec. D.3. As shown in the table below (Table 9), RTBA achieves higher relation consistency (0.2985 vs. 0.1656 or 0.1353) and EI (0.3851 vs. 0.2398 or 0.2097). It also achieves better results across the other three metrics, demonstrating its effectiveness in balancing relation transfer and source-content preservation.

MethodRelation Consistency ↑EI ↑HPSv2 ↑ImageReward ↑PickScore ↑
SIGMA-style Group-Scoped Mask0.16560.23980.25580.251221.2910
OrionEdit-style Reverse-Causal Mask0.13530.20970.25290.223421.2264
RTBA (Ours)0.29850.38510.29131.269922.4189
AuthorsOpenReview ↗

Response to Reviewer BPs5 (Part 2 / 5)

W2: Explaining and Validating the RTBA Bottleneck Design

Thank you for raising this point. We address this concern from two aspects: (1) explaining how RTBA works jointly with training supervision to promote relation–appearance disentanglement; and (2) evaluating the robustness of RTBA under reference appearance variations.

Explanation of the Motivation Behind the RTBA Architecture. We argue that RTBA and training supervision jointly promote relation–appearance disentanglement. Specifically, RTBA imposes an architectural constraint through its relation-token bottleneck. This encourages relational and appearance information to compete for the bottleneck's limited capacity. Meanwhile, RTD-35K provides paired supervision through samples with fine-grained and visually consistent relations. The flow-matching objective further guides the model to transfer reference relations. Together, these designs encourage the model to selectively extract transferable relation cues.

Reference Appearance Sensitivity Experiment. Following your suggestion, we further evaluate RACE under reference appearance variations. Specifically, we create three variants by changing the reference identity, style, or both, while keeping the source image and reference relation configuration fixed.

Reference Appearance Robustness Results. The newly appended Fig. 19 illustrates the qualitative results under reference appearance variations. We keep the source image fixed and vary the reference identity (first and second rows) or style (third row), while maintaining the reference relation configurations. The four columns correspond to different fine-grained reference interactions. As shown, RACE consistently transfers these reference-specific interactions to the source image while preserving source identity and appearance. The quantitative results in the table below (Table 12) further confirm this robustness, with only minor changes in relation consistency, VLM-R scores, and EI.

Reference AppearanceRelation Consistency ↑EI ↑HPSv2 ↑ImageReward ↑PickScore ↑VLM-R ↑
Original0.47130.45980.27510.456421.46028.4268
Identity Changed0.46780.44520.27790.433221.45838.4132
Style Changed0.46470.44390.26050.351621.36728.4219
Identity + Style Changed0.46390.44050.26870.339821.30158.4105

Additional Manuscript Details. We have added this analysis and the corresponding results to Supplementary Sec. D.7.

AuthorsOpenReview ↗

Response to Reviewer BPs5 (Part 3 / 5)

W3: Completing and Clarifying the Bagel Baselines

We thank the reviewer for raising this point.

Baseline Definition. We clarify that the Baseline used in our ablation studies (e.g., Table 2) refers to a fine-tuned Bagel model trained under the same experimental settings as RACE and receiving identical inputs. We have explicitly clarified this definition in Sec. 4.3 of the revised manuscript.

Baseline Comparison. Following your suggestion, we have added both the fine-tuned Bagel baseline and the original Bagel model without task-specific fine-tuning (Baseline (w/o train)) to Table 1, enabling a straightforward comparison with other methods, including RACE.

Baseline Comparison Results. As shown in the table below (Table 1), RACE consistently outperforms the fine-tuned Bagel baseline across all six objective metrics and the reported user-study relation-consistency score, demonstrating the effectiveness of the proposed components beyond standard Bagel fine-tuning.

MethodRelation ConsistencyEIHPSv2ImageRewardPickScoreVLM-RUser Study: Relation ConsistencyUser Study: Editing ConsistencyUser Study: Image Quality
Baseline (w/o train)0.31960.30910.1923-1.373019.99364.593246.7248.9747.10
Baseline0.34160.33120.2159-0.652820.78995.103350.7651.7551.93
Ours0.47130.45980.27510.456421.46028.426882.9875.0381.16

Training and Inference Costs. In addition, the training and inference costs are reported in Supplementary Sec. C and Table 7 of the original manuscript. The corresponding results are reproduced in the table below. Compared with the Baseline, RACE incurs 13.66 additional training hours (21.0% overhead), while maintaining nearly identical inference time (35.08 vs. 35.33 s/image) and memory usage (29.77 vs. 29.71 GB).

MethodTraining Time (hours)Training StepsInference Time (seconds/image)Inference Memory (GB)
FLUX.1-Kontext99.637,85062.1537.36
OneHOI92.617,85054.0342.33
Baseline (Fine-tuned Bagel)64.897,85035.3329.71
Ours (RACE)78.556,650 + 1,20035.0829.77
Reviewer_yPEmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RACE transfers a designated object pair's relation from a reference image to a source image while preserving source appearance. It modifies Bagel with relation-token attention bottlenecks (RTBA), and trains a semantic shift predictor (SSP) aligned with textual relation changes through CSSA. Training uses source reconstruction and relation-aware classifier-free guidance. The authors construct 35K synthetic training triplets spanning 235 relation categories and evaluate on 1,000 HICO-derived and 1,500 synthetic test pairs. Results show improved relation/preservation tradeoffs, with qualitative demonstrations of multiple relations and cross-category transfer.

Critical strengths
  1. RTBA has useful controlled evidence. Routing reference information through relation tokens addresses appearance leakage while retaining source conditioning (Sec. 3.2; Fig. 8). In Table 2, the RTBA-containing configuration without CSSA improves relation consistency from 0.2770 to 0.3141 and EI from 0.3308 to 0.3708 over the baseline. Table 6 compares alternative masks with the same backbone and training settings. These controls support a contribution beyond simply providing another image.

  2. The system shows a strong reported empirical tradeoff. Table 1 improves HICORTBench relation consistency from the strongest baseline's 0.3984 to 0.4713 and EI from 0.3972 to 0.4598 (18.30% and 15.76% relative gains). Appendix D.4 trains reference-conditioned OneHOI and FluxKontext on the same data for the same number of steps. Table 7 supports the preservation/quality tradeoff on RTBench, although DyRef has higher relation consistency there. Table 4 reports essentially unchanged inference cost relative to Bagel. These results make the system practically promising.

Critical weaknesses
  1. The evidence does not yet establish that CSSA constrains relation-irrelevant image changes in the way claimed. Equations (9)-(18) motivate SSP as an approximation to image-semantic change, but the implemented objective jointly trains SSP, the text encoder, and its projection using relation-shift contrastive labels (Eq. 5; Appendix B.4). No independent image-semantic supervision or calibration for that approximation is described. Consequently, distinguishing relation transitions while ignoring appearance changes remains an alternative explanation; Appendix D.2 itself discusses such a shortcut. Figure 7's clustering in the learned representation does not independently resolve this. There is also an unexplained gap between the model-generated trajectories in the derivation and the noised ground-truth latents in Algorithm 1. Table 2 supports an empirical tradeoff: adding CSSA increases EI from 0.3708 to 0.3851 while reducing relation consistency from 0.3141 to 0.2985. It does not by itself identify the claimed mechanism, which is a central contribution.

  2. Validation of reference-specific interaction transfer, beyond recognizing a relation category, is limited. The motivation emphasizes contact locations and relative poses beyond a coarse relation label (Fig. 1), but the principal automatic relation metric is PViC embedding similarity, without a reported sensitivity check for those distinctions (Sec. 4.1). RACE receives object descriptions rather than a target-relation instruction at inference (Algorithm 2); the unresolved alternative is that it recognizes a category such as “carry” visually and produces a typical configuration, without matching the reference's particular contact/pose. RTBA's attention analysis and qualitative examples are relevant evidence, but do not isolate this distinction. RTD-35K's reference-skeleton conditioning supplies useful structural supervision; however, the published filtering criteria require the same relation without separately specifying a contact/pose fidelity test (Sec. 3.1; Appendix F, Fig. 18). The human study offers complementary evidence, yet uses only eight reference-source pairs; 24 raters cannot increase this input diversity. Moreover, 13 methods are split into six-method groups and normalized ranks are averaged without specifying group composition or balance (Appendix B.5). These gaps leave uncertainty about the claimed advantage in transferring the actual reference configuration while preserving source content.

Questions for authors
  1. Clarification/evidence request: What independently validates SSP's sensitivity to unwanted identity/background changes, beyond recognizing the relation transition? Please explain how Figure 7's examples and labels were constructed and whether its effect holds on independently assessed held-out image pairs.

  2. Clarification request: Is the velocity difference used by CSSA evaluated on ground-truth interpolation states or model-generated reverse trajectories? How does the implemented choice support the endpoint-shift interpretation in Eqs. (14)-(18)?

  3. Clarification/evidence request: What evidence distinguishes reference-specific configuration transfer from visual recognition of a coarse relation category? In particular, with the source image and object descriptions fixed, do references depicting the same relation category but different contacts/relative poses produce correspondingly different edits? Please clarify whether the current metrics or human judgments assess this distinction and source preservation separately. This would directly affect my assessment of the main task-level claim.

  4. Clarification request: Please provide the human-study group assignments, method exposure counts, input-pair selection procedure, and uncertainty across input pairs. How are normalized ranks made comparable across the six-method groups?

  5. Related-work and supervision clarification: Please cite and discuss RCEdit-500K: Reference Completion for Image-Conditioned Image Editing in relation to reference-conditioned editing and data construction, and Holistic Order Prediction in Natural Scenes in relation to explicit visual-relation representations. Please clarify RACE's specific contribution relative to these settings in its relation representation, transferable information, and supervision. In particular, beyond the reference-skeleton construction, what checks establish that retained RTD-35K reference-target pairs match in contact/relative pose rather than only in relation category?

7 comments
AuthorsOpenReview ↗

Response to Reviewer yPEm (Part 1 / 5)

W1 & Q1: Validating SSP's Appearance Sensitivity and Generalization

We thank the reviewer for the careful reading and thoughtful question, which helps us further clarify and validate SSP's sensitivity to appearance changes. We address the reviewer's concerns below by (1) explaining how our training design prevents SSP from relying solely on relation-transition prediction, and further validating its sensitivity to appearance changes; and (2) clarifying Fig. 7 and evaluating SSP's generalization to unseen relation categories.


1. SSP Sensitivity to Appearance and Background Changes

Preventing Relation-Only Shortcut Learning in SSP. With appropriate training, SSP remains sensitive to identity and background changes through contrastive learning. Specifically, during the warm-up stage of SSP, we use appearance-preserving source–target pairs as positive samples to align image and text semantic shifts, while treating appearance-changing source–reference pairs as negative samples to separate appearance-contaminated image shifts from the intended textual shifts. This design provides a favorable initialization for the subsequent joint training of SSP and discourages SSP from taking shortcuts based solely on relation transitions.

Experiment on SSP Sensitivity to Appearance Changes. Furthermore, following your suggestion, we evaluate the SSP checkpoint obtained after 1,200 joint-training steps on two groups of held-out image pairs: an appearance-preserving group and an appearance-changing group. The evaluation covers 20 held-out relation categories and 15 relatively rare relation categories, comprising 800 image-change pairs in total. For each pair, we compute the cosine similarity between the predicted image semantic shift and the intended textual semantic shift.

Experimental Results. As shown below, the appearance-preserving group achieves higher similarity (0.7126 vs. 0.5915), supporting SSP's sensitivity to appearance drift.

GroupImage–Text Shift Similarity ↑
Appearance-preserving0.7126
Appearance-changing0.5915

Additional Manuscript Details. We have added these results to Supplementary Sec. F (Tab. 13).


2. Clarifying the Construction of Figure 7 and Evaluating SSP Generalization to Held-Out Relation Categories

Figure 7 Visualization Protocol. Regarding Fig. 7, without SSP, we directly compute the endpoint semantic shift from the ground-truth source–target image pair using the image encoder fimgf_{img} (Eq. 9). With SSP, we add noise to the source and target images at t=0.5t=0.5, obtain the model-predicted velocity difference Δvt\Delta v_t, and use SSP to predict the source-to-target image semantic-shift embedding Δimg(t)\Delta_{img}^{(t)} (Eq. 4). We visualize these embeddings using PCA.

Generalization to Held-Out Relations. Beyond the relation categories shown in Fig. 7, SSP's sensitivity to appearance changes also generalizes to independently held-out relation categories. We have added PCA visualizations for held-out relation categories in Supplementary Sec. F (Fig. 20). The consistent clustering patterns across these unseen categories provide further evidence that SSP can identify appearance-related changes and generalize to relation categories unseen during training.

AuthorsOpenReview ↗

Response to Reviewer yPEm (Part 2 / 5)

W1 & Q2: Clarifying CSSA Trajectory Construction

We thank the reviewer for this careful and insightful question, which helps us clarify the distinction between the theoretical formulation and practical implementation of CSSA. We address the concern below by (1) explaining the trajectory approximation, and (2) comparing reverse-trajectory and interpolation-based training.


1. Replacing Reverse Trajectories with Interpolated States for Efficient Training

Trajectory Approximation. Theoretically, CSSA is derived from model-generated reverse trajectories (Eqs. 14–18). However, for training efficiency, we evaluate the velocity difference Δvt\Delta v_t at ground-truth interpolated states ztz_t with matched noise and timesteps, rather than explicitly simulating the full reverse trajectories. Since the endpoint displacement in Eqs. (14–18) represents the accumulated effect of velocity differences along the trajectory, Δvt\Delta v_t at each matched timestep provides a local estimate of its contribution to the overall endpoint shift. This approximation not only provides local supervision for image-semantic shifts but also improves training efficiency.

Exposure-Bias Mitigation. It is worth noting that the approximation error mainly stems from exposure bias [1], which arises because the model is trained on ground-truth interpolated states but may encounter deviated, model-generated states along reverse trajectories. As described in the first paragraph of Sec. 3.3, to mitigate this discrepancy, we first warm up the model for 6,650 steps before activating CSSA. This helps stabilize velocity prediction and provides a more reliable velocity field for CSSA supervision.


2. Comparing Reverse-Trajectory and Interpolation-Based Training

Training Trajectory Comparison Experiment. To evaluate the effectiveness and efficiency of this approximation, we compare two otherwise identical variants using reverse-trajectory states and interpolated states.

Performance and Efficiency. As shown in the table below, interpolation-based training achieves comparable performance while substantially reducing training time compared with reverse-trajectory training.

Training Variantrelation consistency ↑EI ↑HPSv2 ↑ImageReward ↑PickScore ↑Training Time (h) ↓
Reverse-trajectory0.30840.38870.27791.281821.9597122.48
Ours (Interpolation)0.29850.38510.29131.269922.418978.55

Additional Manuscript Details. We have added the explanation of CSSA trajectory construction and the corresponding comparison results in Supplementary Sec. B.3 and Table 4.

AuthorsOpenReview ↗

Response to Reviewer yPEm (Part 3 / 5)

W2 & Q3: Demonstrating Fine-grained Relation Transfer and Validating Evaluation Metrics

We thank the reviewer for this careful and insightful question, which highlights the importance of distinguishing reference-specific fine-grained relation transfer from coarse relation category recognition. We address the reviewer's concerns below by (1) providing additional qualitative evidence of reference-specific fine-grained relation transfer, and (2) validating PViC's sensitivity to fine-grained visual relation differences.


1. Fine-grained Relation Transfer

Fine-Grained Relation Transfer Objective. We clarify that RACE aims to transfer fine-grained visual relation configurations rather than merely recognize coarse relation categories.

Fine-Grained Relation Sensitivity Experiment. Following your suggestion, we conduct an additional experiment in which the source image is fixed while the references depict the same coarse relation category but different fine-grained visual interactions.

Reference-Specific Transfer Results. As shown in the newly appended Fig. 18 (Supplementary Sec. D.7), RACE reproduces reference-specific configurations, including shoulder support vs. overhead hold, gentle push vs. backward pull, and side embrace vs. front embrace. These results demonstrate its ability to transfer fine-grained contact locations and relative poses rather than merely recognize coarse relation categories.


2. PViC Sensitivity to Fine-grained Visual Relation Differences

Our algorithm has the merit that PViC is sensitive to fine-grained visual relation differences.

Experiment on PViC Sensitivity to Fine-Grained Relations. To validate this merit, we construct four controlled groups of reference–target image pairs. Subject and object appearances remain consistent within each pair, while relation categories and fine-grained visual configurations vary.

Experimental Results. The highest PViC score (0.8395) is achieved when both the relation category and visual configuration match (A0), whereas the score decreases to 0.6784 when the relation category remains the same but the visual configuration differs (A1), demonstrating PViC's sensitivity to fine-grained relation differences beyond category-level matching.

GroupRelation CategoryVisual ConfigurationPViC ↑
A0SameSame0.8395
A1SameDifferent0.6784
A2DifferentPartially similar0.6506
A3DifferentDifferent-0.0047

Expert Evaluation of Contact and Pose Consistency. In addition, we ask expert evaluators to assess fine-grained relation consistency by comparing the contact locations and relative poses in the generated images with those in the reference images.

Additional Manuscript Details. We have added this evaluation and its results to Supplementary Sec. B.6 and Table 5.

Comments on the paper

2 comments
Area_Chair_y5xbOpenReview ↗

Discussion phase will end on Nov 18

This is a kind reminder that discussion phase will end on Nov. 18.

Dear Reviewer sSje, thank you for your active engagement in the discussion.

Dear Reviewer yPEm and BPs5, could you please check authors' rebuttal and update your thoughts? Your early feedback would make the discussion more smooth and informative.

Thank you.

AuthorsOpenReview ↗

Summary of Responses and Revisions Across Reviewers

Dear AC,

We sincerely thank all three reviewers (yPEm, BPs5, and sSje) for their insightful evaluations and constructive feedback. We also appreciate their positive assessments of our task's practical relevance (sSje), clear presentation (yPEm, BPs5, sSje), complementary module design (BPs5, sSje), and promising empirical results (yPEm, BPs5, sSje). In response to their questions and suggestions, we have conducted additional experiments, expanded baseline comparisons, and revised the manuscript and supplementary material. The main updates are summarized below.

1. Related Work and Technical Details (yPEm, BPs5, sSje). We expanded discussions of ReVersion and OrionEdit (Main Paper, Sec. 1), SynPS (Sec. 2), and RCEdit-500K and Holistic Order Prediction (Sec. 3.1). These discussions highlight our RACE's distinctive contributions: transferring fine-grained relations from a single reference through RTBA's relation-token bottleneck while using CSSA to suppress relation-irrelevant changes and preserve source content. We also clarified the SSP architecture (existing Supplementary Sec. B.4), mathematical notation (revised Main Paper Secs. 3.1–3.2), and HICORTBench construction (revised Main Paper Sec. 4.1).

2. RTBA Effectiveness (BPs5, sSje). We added dedicated quantitative RTBA ablations (revised Table 2) alongside qualitative ablations (revised Fig. 5). We further compared our RTBA with alternative attention masks, demonstrating superior relation consistency and EI (Supplementary Sec. D.3, existing Table 9). Additional robustness experiments under reference identity and style variations show stable relation transfer with minimal performance changes (Supplementary Sec. D.7, newly-added Fig. 19 and Table 12). These results further support our RTBA's effectiveness in relation transfer.

3. SSP Validation and CSSA Analysis (yPEm, BPs5, sSje). We conducted a held-out evaluation involving 800 image-change pairs, demonstrating our SSP's sensitivity to appearance changes (0.7126 vs. 0.5915). We further added PCA visualizations for unseen relation categories, showing consistent clustering patterns (Supplementary Sec. F, newly-added Table 13 and Fig. 20). In addition, we clarified our CSSA's trajectory approximation and compared interpolation-based training with reverse-trajectory training. The interpolation approach achieves comparable performance while reducing training time from 122.48 to 78.55 hours (Supplementary Sec. B.3, newly-added Table 4). Additional analyses of CSSA loss weights and training steps are provided in Supplementary Sec. D.2, revised Table 8.

4. Fine-Grained Relation Transfer and Evaluation (yPEm). We conducted controlled experiments with fixed source images and references depicting different contact locations and relative poses within the same relation category (Supplementary Sec. D.7, newly-added Fig. 18). We further validated PViC's sensitivity to fine-grained configurations (Supplementary Sec. B.6, newly-added Table 5).

5. Dataset Quality and Evaluation Protocols (yPEm, BPs5). We conducted additional RTD-35K quality evaluations. After filtering, relation consistency improved from 0.4931 to 0.6191, and VLM-R increased from 7.1860 to 8.7388. We further clarified the user-study protocol by providing detailed group assignments and rank normalization (Supplementary Sec. B.6, newly-added Table 6).

6. Expanded Baseline Comparisons (BPs5, sSje). We clarified the reference-image input configurations of comparison methods (Main Paper, Sec. 4.2). We also added the original Bagel model and included our previously evaluated fine-tuned Bagel baseline in the main paper (revised Table 1). Furthermore, we expanded quantitative and qualitative comparisons to include OrionEdit, FreeEvent, ReVersion, GPT Image 2, Nano Banana 2, and Qwen-Image-2.1 (Supplementary Sec. D.6, revised Table 10-11, revised Fig. 10-11). Our RACE achieves the best results across all six objective metrics on HICORTBench.

We believe these additional experiments and revisions substantially strengthen the methodological justification, empirical validation, and reproducibility of our RACE. We appreciate the reviewers' constructive suggestions and hope this summary helps the Area Chair assess the additional evidence and improvements. We have uploaded the revised manuscript, with all changes highlighted in red.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.