acceptodds
Under review as a conference paper at ICLR 2027

Training-Free Conflict-Aware Attribute Forgetting for Customized Video Generation

Abstract

Customized video generation aims to preserve personalized subject characteristics from reference images while generating dynamics aligned with text prompts. Existing methods typically apply reference conditioning uniformly throughout generation, implicitly assuming that all reference attributes remain valid for the target event. When a prompt requires an attribute to change, reference and textual conditions impose incompatible attribute states, resulting in attribute conflict. To address this conflict, we propose AttriForget, a training-free framework for conflict-aware attribute forgetting that constructs an event-conditioned attribute graph to model the validity of reference attributes under the target event. Guided by this graph, AttriForget probes orthogonal attribute-sensitive subspaces in reference features to preserve information associated with persistent attributes while "forgetting" reference constraints on evolving attributes, turning conflicting reference and textual conditions into complementary guidance. Furthermore, AttriForget can be integrated into various Diffusion Transformer architectures without additional training, extending global reference control to fine-grained attribute-level reference control. Experiments on OpenS2V-Eval across four subject-customized video generation backbones show that AttriForget consistently improves total scores and video naturalness while enhancing subject consistency on most evaluated settings, with an average inference latency increase of only 4% across multiple backbones. Code will be made publicly available.

Primary area:
generative models
Keywords:
Customized Video Generation, Diffusion Transformer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_x8HCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates attribute conflicts problem in reference-conditioned video generation and proposes AttriForget, a training-free method. It first uses a vision-language model to distinguish subject attributes that should be preserved from those allowed to change. It then identifies attribute subspaces through Jacobian-based probing of attention responses, preserves directions associated with persistent attributes, and progressively suppresses reference features corresponding to evolving attributes along the video timeline. Experiments across 4 backbone models demonstrate improvements in overall scores and naturalness, with further support from ablation studies and human evaluations.

Critical strengths
  1. The proposed method is well motivated, and the paper is generally clear and easy to follow. The authors formulate conflicts between reference images and text prompts in terms of which subject attributes should be preserved and which should change. They enable selective control by protecting persistent attribute subspaces and suppressing directions associated with evolving attributes. This design offers a useful approach to balancing subject preservation and textual controllability without updating the generative model’s parameters.

  2. Extensive experiments show some improvement and the motivation of the method. For example, table 1 shows improvements in both overall scores and naturalness across all six model/subdomain combinations. The human evaluation results in Table 7 provide further supporting. And Figure 2 show the analysis of attribute conflict.

Critical weaknesses
  1. The existing experiments do not yet fully establish the method’s effectiveness. Although the paper addresses customized video generation, I do not find supplementary video demonstrations available for review. Static frames alone in paper are insufficient to reliably assess the qualitative video results. Moreover, in Table 1, FaceSim decreases from 48.56 to 45.30 for Phantom-1.3B on Open Domain, while Motion Amplitude decreases from 33.42 to 23.50 for Phantom-14B. An improvement in the overall score therefore does not necessarily indicate simultaneous improvements in attribute preservation and control over attribute changes. These points require further clarification.

  2. The discussion of some closely related work could be more detailed. Although the paper briefly discusses TPIGE [1], that work also introduces Prompt Aware Reference Image Enhancement to resolve conflicts with the text prompt. A more detailed discussion of the relationship and differences between that method and the AttriForget would help clarify the contribution. As I am not an expert in this specific area, my knowledge of the relevant work may be incomplete. My current recommendation is therefore a tentative weak accept, and I will give my final assessment after considering the other reviewers’ comments. [1]Identity-Preserving Text-to-Video Generation via Training-Free Prompt, Image, and Guidance Enhancement. ACM MM 25.

Questions for authors
  1. Please address the concerns raised in the Critical Weaknesses section.

  2. Please provide video demonstrations or comparisons during the rebuttal, given that the task is customized video generation.

  3. In the camera example in Fig. 11(b), why do the outputs differ despite the annotation “Evolving: No”? Were the same random seed and otherwise identical inference settings used for both methods?

2 comments

We appreciate your valuable feedback and support of our work. We hope the following clarifications and additional results address your concerns. W = Critical Weaknesses, Q = Questions For Authors.

To Q1 & W1:

These metrics involve trade-offs: improving one metric may come at the expense of another, so overall performance cannot be judged by any single metric’s rise or fall. Reference copying, for example, can inflate FaceSim. Phantom-1.3B shows clear reference copying in Fig. 12 alongside low Motion Amplitude and NaturalScore, illustrating how restricted dynamics can yield misleadingly high similarity. For Phantom-14B, higher optical flow may reflect camera motion or abrupt visual changes, as in the shoe and cup examples in Fig. 5, rather than better subject motion. Thus, decreases in these two metrics alone do not establish poorer generation.

To account for these trade-offs, OpenS2V-Eval uses weighted joint evaluation: NaturalScore 24%, FaceSim and NexusScore 20% each, Aesthetics 16%, GmeScore 12%, Motion Smoothness 6%, and Motion Amplitude 2%, assigning low weights to motion-related scores susceptible to inflation.Crucially, the official GmeScore implementation computes image–text similarities between sampled frames and the full prompt, then averages them across frames. Reordering the same frames leaves the score unchanged, so it cannot fully assess whether attributes evolve correctly over time. Human-rated text alignment improves from 2.67 to 3.01 on VACE-1.3B and from 3.43 to 3.74 on Phantom-14B (Table 7). Evaluation should therefore combine multiple metrics, human judgments, and visual results for a comprehensive assessment.

To Q2 & W1:

We have provided the video of the case presented in the paper in this anonymous repository:https://anonymous.4open.science/r/AttriForget-58A4

To W2:

We have added a detailed discussion in the Related Work section, highlighted in orange in Lines 182–194 of the revised manuscript, to clarify the above connections and differences.

TPIGE attributes generation difficulties to semantic mismatches between reference images and text prompts. Through experiments that control the conflicting attributes, we further find that reference and textual guidance oppose each other when imposing opposite attribute requirements, and tend toward orthogonality after the conflicting description is removed. This provides direct experimental evidence for the attribute-level source of the conflict and reveals the problem of treating all reference attributes as persistently valid constraints.

Regarding the solution, TPIGE’s reference image enhancement module first generates a new reference image that matches the prompt, and then generates a video conditioned on that image. This intermediate generation step may alter identity information in the original reference, and its ablation study also reports a decrease in identity consistency after adding image enhancement. AttriForget operates directly on the original reference features, protects persistent attributes through subspace disentanglement, and progressively removes only the reference constraints on evolving attributes, thereby avoiding the additional risk of information loss introduced by image regeneration while accommodating both subject preservation and attribute evolution.

To Q3:

We thank the reviewer for the careful observation. The “Evolving: No” annotation in Fig. 11(b) was an oversight during figure preparation. This example actually contains evolving attributes, such as knob rotation, and AttriForget applies forgetting to the corresponding reference features during inference, resulting in different outputs. Both methods used the same random seed and otherwise identical inference settings. We have corrected this annotation in the revised manuscript.

Dear Reviewer x8HC,

I hope this message finds you well. As the discussion period is nearing its end with less than two days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work. Thank you for your time and effort in reviewing our paper.

Reviewer_T5A9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AttriForget, a training-free method for customized video generation that addresses conflicts between reference images and text prompts. The method constructs an attribute graph to distinguish persistent from evolving attributes and selectively suppresses reference features associated with attributes that should change. Experiments across several video generation backbones show improvements in overall generation quality.

Critical strengths

The paper addresses an interesting problem in reference-conditioned video generation.
The proposed approach is training-free and applicable to multiple backbones.
The experimental results show promising improvements across several settings.

Critical weaknesses

My expertise is primarily in generative models and representation manipulation, rather than subject-customized video generation. Therefore, some of my concerns are mainly related to experimental validation and methodological clarity.

Attribute graph and computational overhead. The method relies on Qwen3-VL-8B to construct attribute graphs, which is considerably larger than some of the evaluated generation backbones. Although Appendix B.2 compares different Qwen variants, the experiments do not sufficiently isolate the contribution of graph quality from the proposed forgetting mechanism.
Implementation details. The description of the layer-wise feature manipulation could be clearer. In particular, it is not sufficiently explicit which attention layers are modified for each backbone and whether the method is sensitive to this choice. An ablation on layer selection would improve reproducibility.
Robustness to random seeds. Section 4.5 compares the method against random seed search in the baseline. However, this does not establish whether AttriForget itself is robust to different random seeds. Reporting mean performance and standard deviations across multiple seeds would strengthen the conclusions.

Minor Weakness, this does not influence my score on the paper.
The low-rank attribute subspace manipulation in Table 3 appears to be somehow similar to the paper "Video Unlearning via Low-Rank Refusal Vector", Facchiano et al., which also investigates low-rank directions for manipulating video generation. The objectives of this paper clearly differs, but I would consider mentioning this paper in the second subsection of the Related Works.

Limited ablation coverage. The main ablations are conducted exclusively on VACE-1.3B in the Human Domain. It is unclear whether the conclusions about subspace rank, disentanglement, and forgetting schedules generalize to the other evaluated backbones.

Questions for authors

What is the end-to-end inference overhead when attribute graph construction is included?
Which attention layers are modified, and how sensitive are the results to their selection?
How consistent are the improvements across different random seeds?

5 comments

We appreciate your valuable feedback. We hope the following clarifications and additional results address your concerns. W = Critical Weaknesses, Q = Questions For Authors.

To Q1 & W1:

  1. We have added the following details in blue text in Lines 1011–1020 of the revised manuscript. Qwen3-VL-8B takes approximately 10 s to construct an attribute graph on a single H20 GPU. This cost is largely independent of the video backbone size, but its relative impact varies: it amounts to only about 0.4% of the baseline inference time for Phantom-14B, which exceeds 2400 s, while accounting for a larger proportion for the 1.3B models. We therefore report this one-time preprocessing cost separately from the backbone-dependent video generation overhead. Attribute graphs can be generated offline in batches. Each reference image–prompt pair requires only one graph construction, after which the cached graph can be reused for subsequent video generation without additional VLM calls.

  2. We examine the contributions of graph quality and the forgetting mechanism through two complementary controlled experiments. Appendix B.2 fixes the forgetting mechanism and varies only the graph generator to assess the effect of graph quality. Table 3 reuses exactly the same cached graphs to compare the full method with variants without subspace disentanglement or with immediate forgetting. Given identical attribute information, these two variants reduce TotalScore from 58.18% to 56.53% and 56.98%, respectively. These results show that graph quality alone cannot explain the performance gains: subspace disentanglement and progressive forgetting provide additional benefits under identical graph conditions.

To Q2 & W2:

  1. AttriForget enables the model to selectively forget attribute features from reference images and thus operates wherever video tokens attend to reference features. It modifies reference features before K/V projection, not the attention operator itself, leaving the original attention computation and model parameters unchanged. For VACE-1.3B, it covers all 15 conditioning-branch layers at even backbone indices (0, 2, …, 28).

  2. Following your suggestion, we examined different network depths. As shown in the table, these partial-layer configurations yield overall scores close to the baseline, losing the gains of the full configuration. This is consistent with our mechanism: video tokens can still attend to complete, unmodified reference features in the remaining layers, allowing conflicting attribute constraints to persist. These results support applying AttriForget across all relevant interaction layers.

Table 1. Ablation Study on Layer Selection

Selected LayersTotal ↑Aes ↑M.Som. ↑M.Amp. ↑FaceSim ↑Gme ↑Nexus ↑Natural ↑
0–855.12%53.96%96.09%7.70%25.76%73.06%–63.75%
10–1855.24%53.97%96.17%7.59%25.29%72.99%–64.58%
20–2855.20%53.87%95.74%7.62%25.32%73.03%–64.58%

To Q3 & W3:

We evaluated AttriForget with 10 different random seeds and report the mean and standard deviation of each metric in the table below. Most metrics exhibit small variations across seeds, supporting the robustness of AttriForget to random seeds.

Table 2. Random Seed Experiment

SettingTotal ↑Aes ↑M.Som. ↑M.Amp. ↑FaceSim ↑Gme ↑Nexus ↑Natural ↑
Mean ± Std58.08 ± 0.42%53.27 ± 0.39%98.87 ± 0.87%17.74 ± 2.78%26.80 ± 1.17%73.07 ± 0.36%–71.33 ± 0.83%

To W4: We thank the reviewer for recommending this relevant work. We have added a citation and discussion in the second subsection of Related Work, highlighted in red in Lines 180–183 of the revised manuscript.

To W5:

We conducted additional ablations on subspace rank, disentanglement, and forgetting schedules using Kim et al. (5B). As shown in the table, the results follow the same trends as on VACE-1.3B. These results support the generalizability of our design conclusions across backbones.

Table 3. Ablation Study on Kim et al. (5B)

MethodTotal ↑Aes ↑M.Som. ↑M.Amp. ↑FaceSim ↑Gme ↑Nexus ↑Natural ↑
Immediate forgetting46.14%35.67%81.61%21.05%27.91%64.89%44.26%53.73%
w/o disentanglement45.84%35.26%82.44%21.53%24.48%65.04%44.27%55.28%
Rank 145.97%35.62%82.54%17.65%28.31%64.73%44.02%53.04%
Rank 246.62%35.41%83.08%20.69%28.46%64.70%44.52%54.98%
Rank 846.38%35.49%82.70%18.71%28.32%65.24%44.65%53.94%
AttriForget47.08%35.89%82.90%21.89%28.55%65.07%44.97%55.90%

Dear Reviewer T5A9,

I hope this message finds you well. As the discussion period is nearing its end with less than four days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

AuthorsOpenReview ↗

Have we addressed your concerns?

Dear Reviewer T5A9,

We hope this revision addresses your concerns. Please let us know if there is anything further we can clarify; we would be happy to discuss any remaining issues.

Reviewer_TEMGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper tackles subject-to-video generation, where a reference image and a text prompt can disagree about an attribute (e.g. the prompt asks for blonde hair but the reference shows red hair). Existing methods treat the whole reference as a fixed constraint, so the two signals fight. The proposed AttriForget is training-free: a frozen VLM builds an "attribute graph" that tags each subject attribute as persistent or evolving for the given prompt; then, inside DiT attention, the method locates a low-rank subspace for each attribute by perturbing the reference features and taking the top singular vectors of the attention-response Jacobian, orthogonalizes evolving subspaces against persistent ones, and progressively projects the evolving components out of the reference features over the first 30% of frames. On OpenS2V-Eval with four backbones, it consistently improves NaturalScore and TotalScore with about 4% extra latency, supported by a user study and ablations.

Critical strengths
  • Clear problem with a simple diagnostic. The toy experiment showing that reference and text guidance clash only on the contradicted attribute is convincing and easy to understand.
  • Principled, training-free mechanism. Finding attribute subspaces from the model's own attention response (Jacobian + SVD) and orthogonalizing them against persistent attributes is more grounded than hand-made masks. Works on four backbones of different sizes with no weight changes.
  • Good ablations. The seed-search and global-reference-strength baselines directly rule out the obvious "just a lucky seed / weaker reference" explanation. Disentanglement and the progressive schedule are both shown to matter.
Critical weaknesses
  • Gains are almost entirely from NaturalScore. Text alignment (GmeScore) is flat or slightly down in every setting, which is odd if the point is to let text control the evolving attributes. Several metrics also get worse and are not discussed: Phantom-1.3B loses FaceSim (48.6 to 45.3) and Phantom-14B loses a lot of Motion Amplitude (33.4 to 23.5).
  • No robustness analysis of the attribute graph. Everything depends on the VLM correctly labeling attributes as persistent vs evolving, but the paper never shows what happens to the video when the graph is wrong.
  • Key details and hyper-parameters are thin. The forgetting horizon (30%) and rank (4) are tuned on one backbone and reused; the main text does not say at which timestep the Jacobian is computed or how heads are aggregated. The 4% latency also excludes the 8B VLM call.
Questions for authors
  1. Why does GmeScore not improve, and what causes the FaceSim / Motion Amplitude drops on the Phantom models?
  2. How do results change when the attribute graph is wrong (labels flipped, or a weaker VLM used end to end)?
  3. At which timestep(s) is the probe computed, and does the fixed 0 to 30% forgetting ramp hurt prompts where the change should happen late in the video?
2 comments

We appreciate your valuable feedback and support of our work. We hope the following clarifications and additional results address your concerns. W = Critical Weaknesses, Q = Questions For Authors.

To Q1 & W1:

The gains of AttriForget extend beyond NaturalScore, which contributes only ~46% of the TotalScore improvement on Phantom-14B, versus ~54% from other metrics. These metrics involve trade-offs. Improving one metric may come at the expense of another, so overall performance cannot be judged by the rise or fall of any single metric. Reference copying inflates FaceSim on Phantom-1.3B despite low Motion Amplitude and NaturalScore (Fig. 12), while higher optical flow on Phantom-14B may reflect camera motion or abrupt visual changes rather than better subject motion, as illustrated by the shoe and cup examples (Fig. 5).

OpenS2V-Eval accounts for these trade-offs through weighted evaluation, assigning 24% to NaturalScore, 20% each to FaceSim and NexusScore, 16% to Aesthetics, 12% to GmeScore, 6% to Motion Smoothness, and 2% to Motion Amplitude, with low weights for motion metrics susceptible to inflation. Moreover, the official GmeScore averages similarities between sampled frames and the full prompt, making it invariant to frame order and unable to fully assess temporal attribute evolution. Human-rated text alignment improves from 2.67 to 3.01 on VACE-1.3B and from 3.43 to 3.74 on Phantom-14B (Table 7). Evaluation should therefore combine multiple metrics, human judgments, and visual results for a comprehensive assessment.

To Q2 & W2:

Appendix B.2 already provides an end-to-end robustness analysis of the attribute graph using weaker VLMs (Tables 4–5). On 60 Human-Domain cases, we generate graphs with Qwen2-VL-7B, Qwen2.5-VL-7B, and Qwen3-VL-8B and use them with VACE-1.3B. Graph quality increases from 4.07 to 4.17 to 4.43, with corresponding TotalScores of 57.21, 57.85, and 58.18. Even with the weakest graph generator, AttriForget outperforms the original baseline (54.90). Thus, the paper already examines the relationship between graph quality and final video quality.

The effects of incorrect labels can also be understood from Eq. (8). When all attributes are labeled persistent, no forgetting is applied, and AttriForget exactly reduces to the original backbone, as illustrated in Appendix C. Mislabeling a persistent attribute as evolving can remove valid reference constraints and cause identity drift. Conversely, mislabeling an evolving attribute as persistent retains conflicting reference constraints and may cause reference copying or failure to perform the requested attribute changes. These are precisely the failure modes that AttriForget aims to address.

To Q3 & W3:

We have added the following implementation details and experimental results in blue text in Lines 920–944 and 1011–1020 of the revised manuscript.

  1. We specified that probing occurs before K/V projection during the first forward pass conditioned on text and reference images at the first denoising step. Each DiT layer is probed once, and the result is cached. We select P=min⁡(H,2k)P=\min(H,2k) evenly distributed attention heads, stack their score gradients into a P×DP\times D Jacobian, and retain its top kk right singular vectors; k=4k=4 uses eight heads.

  2. We have added an ablation of the 0–60% forgetting interval (table below). The results support 0–30% as an empirical compromise among the tested settings: immediate forgetting may remove useful initial-state information too early, whereas slower release may prolong reference–text conflicts. The ramp controls the release of reference constraints; it does not require attribute changes to finish within the first 30% of the video. Late changes may need different intervals, and your comment motivates future work on adaptive forgetting intervals based on event timing.

  3. The reported ~4% measures AttriForget’s average runtime overhead during video generation with cached attribute graphs. Generating a graph with the 8B VLM takes ~10 s on one H20. This cost is largely independent of backbone size, but its relative impact differs: only ~0.4% for Phantom-14B, whose baseline inference exceeds 2400 s, versus a higher fraction for 1.3B models. We therefore report this preprocessing cost separately from the video generation overhead of each backbone. Graphs can be generated in offline batches and cached: each reference–prompt pair requires only one graph, and subsequent video generation needs no VLM call.

Table 1. Ablation Study on the Forgetting Interval

Forgetting IntervalTotal ↑Aes ↑M.Som. ↑M.Amp. ↑FaceSim ↑Gme ↑Nexus ↑Natural ↑
056.98%53.12%97.32%16.35%27.50%72.77%-67.91%
0–30%58.18%53.08%99.28%17.53%27.56%72.79%-71.19%
0–60%57.37%53.91%98.07%8.83%27.66%72.75%-69.17%

Dear Reviewer TEMG,

I hope this message finds you well. As the discussion period is nearing its end with less than two days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work. Thank you for your time and effort in reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.