acceptodds
Under review as a conference paper at ICLR 2027

Why Do Text-to-Video Models Struggle with Spatial Relations?

Abstract

Text-to-video (T2V) models have achieved impressive generation quality, yet they remain unreliable in establishing spatial relations among multiple objects. In this work, we investigate the sources of spatial-relation failures in transformer-based T2V models. Our analysis identifies two key limitations. First, the frozen text encoder weakly represents the implicit complementary spatial role of reference objects. Second, joint conditioning produces ambiguous object grounding during early denoising. Based on these observations, we propose Spatial Patching and Anchored Relational Conditioning (SPARC), a training-free approach for improving spatial controllability directly from text prompts. Our method first applies Complementary Spatial Token Patching to strengthen the spatial representation of reference objects. It then uses Anchor-to-Graph Denoising to progressively establish relations around a shared spatial anchor. Both components operate at inference time with frozen model parameters and do not require additional spatial inputs. Experiments across two T2V backbones and multiple spatial benchmarks demonstrate consistent improvements in spatial controllability. As a training-free method, SPARC achieves a 12.5% relative improvement in the T2V-CompBench spatial score over the second-best method.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Text-to-video generation, Spatial relations, Training-free control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hWTGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies spatial-relation failures in text-to-video generation and proposes SPARC, a training-free inference framework. The authors motivate the method through analyses of asymmetric spatial-role representations in text embeddings and ambiguous relational grounding during early denoising.

SPARC parses the prompt into a fixed scene graph and introduces two components. Complementary Spatial Token Patching replaces reference-object token representations with representations obtained from inverse prompts that explicitly state the complementary spatial roles. Anchor-to-Graph Denoising selects an anchor object, performs a short anchor-focused initialization, and sequentially introduces relational conditioning during the first few denoising steps before continuing with the complete patched context. The text encoder and video generator remain frozen.

The method is evaluated on Wan 2.2 and Kandinsky using spatial prompts from T2V-CompBench and VBench, together with a constructed multi-relation setting. The paper reports improvements in spatial evaluation scores and provides diagnostic analyses, component ablations, and qualitative examples.

Critical strengths
  1. The paper provides meaningful within-backbone evidence of improved spatial controllability. On Wan 2.2, the T2V-CompBench spatial score increases from 63.90 to 71.87 in Table 2, corresponding to a 12.5% relative improvement. In the multi-relation setting, Wan 2.2 improves across all four reported metrics. Results on Kandinsky provide additional evidence that the method can benefit more than one backbone. These comparisons support the practical usefulness of the proposed conditioning interventions.

  2. The approach is practically appealing because it improves control without additional model training or user-provided spatial annotations. Both the text encoder and video generator remain frozen, and the method operates from a natural-language prompt without requiring user-specified boxes, masks, or trajectories. The 41-step baseline in Table 4 is also a useful control: it shows that simply adding one denoising step does not explain the reported gains. These properties make the method a potentially useful addition to compatible pretrained generators.

Critical weaknesses
  1. The inability to explicitly model and control time-varying spatial relations substantially limits the paper’s contribution to video generation. SPARC constructs a fixed scene graph without explicitly assigning temporal validity to relation edges or representing transitions between different relations. A2G progressively introduces conditions during denoising, but this computational progression does not specify how relations should evolve along the generated video’s playback timeline. Consequently, the proposed mechanism does not address instructions such as “a person initially stands to the left of a car and then walks to its right.” Changes of this kind arise naturally in events involving crossing, overtaking, or movement around another object, making them an important part of spatial reasoning in video. Although the authors acknowledge this limitation, it remains a substantial restriction on the significance of the contribution: the method primarily improves generation under fixed relational constraints, while leaving the control of temporally evolving relations unaddressed. This also raises the question of what the method contributes specifically to video beyond spatial-conditioning improvements that could plausibly apply to image generation.

  2. The architectural scope is limited, and transfer beyond explicit text cross-attention is insufficiently established. Both evaluated backbone families use explicit text-conditioning cross-attention, and the A2G procedure in Section 4.2 relies on sequential updates through this interface. The main-text description does not explain how the same procedure would operate in joint-attention or token-concatenation architectures, where selecting relation-specific text tokens can affect shared attention normalization and the evolution of both visual and textual states. CSTP appears less dependent on this particular structure because it modifies contextual text representations, and its requirements should be distinguished from those of A2G. The existing results demonstrate usefulness on the tested backbones, but provide limited evidence of compatibility across substantially different conditioning mechanisms. The paper should specify the architectural prerequisites and model-specific adaptations, rather than leaving the extent of transfer implicit.

Questions for authors

Q1. Scope of the video contribution: Given that SPARC uses a fixed scene graph and does not explicitly control time-varying spatial relations, what specific advance does it make in video spatial reasoning beyond improving fixed spatial configurations? Are the proposed failure mechanisms and interventions fundamentally specific to video, or are they general spatial-conditioning mechanisms evaluated on video backbones? Please clarify this distinction and the evidence supporting it. A suitable image-generation comparison or an evaluation of explicitly requested relation transitions would help establish the scope of the contribution.

Q2. Transfer to different conditioning mechanisms: Is A2G currently supported for joint-attention or token-concatenation transformers, or would these require a modified algorithm? If such transfer is supported, please explain the masking and state-update rules, including how visual and textual states are handled between relation-edge operations. If it is an untested extension, please state that boundary explicitly and distinguish it from the compatibility demonstrated by the current experiments.

1 comment

Dear Reviewer,

Thank you for your review. The authors have submitted their rebuttal. Could you please take a moment to review their responses?

Thank you for your time and contribution!

Best,

AC

Reviewer_vMiqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies two potential causes of spatial-relation failures in T2V models: asymmetric representations of spatial roles in text embeddings and interference among objects during early denoising. To address these issues, the authors propose SPARC, a training-free framework that enhances complementary spatial representations and progressively introduces spatial relations through scene-graph-guided denoising. Experiments demonstrate improved spatial controllability across different T2V backbones.

Critical strengths

SPARC improves spatial-relation accuracy without requiring additional training. The authors also provide preliminary analyses of the potential causes of spatial-relation failures in T2V models.

Critical weaknesses
  1. The most complex scenarios considered in the paper contain only 5 objects and 4 spatial relations (Appendix C.1). This relatively limited complexity raises concerns about whether the proposed method can effectively generalize to more challenging multi-object scenarios.

  2. The multi-relation benchmark contains only 45 distinct prompts (Appendix C.1). Based on our experience, evaluating visual generation methods typically requires at least several hundred diverse prompts to obtain reasonably stable and convincing results. Although the authors evaluate multiple random seeds, this does not compensate for the limited prompt diversity. Therefore, the current evaluation raises concerns about the statistical reliability of the reported improvements. Furthermore, sufficient prompt diversity should also be ensured to support the generalizability of the conclusions.

Questions for authors

My main concerns with this paper are the evaluation issues raised in the Weaknesses section. If the authors can provide convincing and well-supported explanations addressing these concerns, I would consider raising my score.

In addition, I would appreciate the authors' responses to the following questions:

  1. In Sec. 3.3 (Fig. 2), the computation of the signed relation margin is not clearly specified. Could the authors clarify its mathematical definition, the object localization method used for intermediate denoising predictions, and how detection failures are handled?

  2. In Algorithm 1 (Appendix B.2), does A2G increase cross-attention calls relative to self-attention, and how are residual connections and normalization handled? Since this changes the pretrained computation despite frozen parameters, could it introduce unexpected hidden-state distributions and negatively affect generation quality? I would appreciate more implementation details and clarification on these concerns.

  3. Sec. 4 states that the method uses Schuster et al. (2015), while Appendix D.3 explicitly distinguishes this parser from the regex-based parser used in the main experiments. Could the authors clarify this inconsistency and provide more implementation details? Additionally, I am concerned about the flexibility of such a rule-based approach for complex natural-language prompts and wonder whether more modern parsing methods could offer better generalization.

2 comments

Dear Reviewer,

Thank you for your review. The authors have submitted their rebuttal. Could you please take a moment to review their responses?

Thank you for your time and contribution!

Best,

AC

Thank you for the detailed responses and the additional experiments. I appreciate the authors' efforts to address my concerns. I have two remaining questions:

Q1. Compatibility with the pretrained DiT computation. Although the authors have clarified the implementation of sequential cross-attention and residual updates, I still do not fully understand why this modified computation can remain stable without additional training. Since the pretrained DiT was not optimized for repeatedly applying cross-attention layers, could the authors explain why this modification does not lead to accumulated activation shifts or unexpected hidden-state distributions?

Q2. Generality of LLM-based scene-graph parsing. The newly added Qwen2.5-7B parser experiment is conducted only with Kandinsky. Could the authors also provide results using Wan 2.2 to further verify the compatibility of LLM-based parsing across different T2V backbones?

Reviewer_ipnaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies why text-to-video models struggle to generate correct spatial relations. It reports two findings: object-token embeddings represent explicitly stated spatial roles more reliably than the implicit complementary roles of reference objects, and joint conditioning on multiple objects is associated with ambiguous grounding during early denoising. Based on these observations, the authors propose SPARC, a training-free method with two components. Complementary Spatial Token Patching replaces reference-object embeddings with embeddings that explicitly encode their complementary roles. Anchor-to-Graph Denoising first establishes an anchor and then sequentially introduces relation-specific conditions during early denoising. Experiments on two video backbones show improved spatial scores on two-object benchmarks and a composed multi-relation test set.

Critical strengths
  • The motivation is good. Investigating the sources of spatial-generation failures is useful for improving video generators.
  • The analysis of explicit and implicit spatial roles is interesting. The probe results and role-conditioned embedding intervention provide evidence that reference-object representations can affect spatial generation.
  • SPARC improves spatial scores over the corresponding base models without additional training or user-provided spatial annotations. Component ablations and a runtime-matched baseline support its effectiveness within the evaluated setting.
Critical weaknesses
  • The method relies on several assumptions whose robustness is insufficiently tested: predefined relations, direct replacement of contextualized object embeddings, and sequential conditioning around a selected anchor. It is unclear whether embedding replacement preserves other attributes and actions.
  • The core role-probe analysis is limited to a specific text encoder, UMT5, and low linear-probe accuracy for individual object tokens does not establish that the full text representation lacks the relevant spatial information. Applying SPARC to two video backbones does not resolve this concern or show that the same failure mechanism holds across text encoders.
  • Evidence of broader practical applicability is limited. Evaluation covers six directional relations, and the multi-relation set contains only 45 composed prompts with 3–5 objects. These experiments mainly test fixed spatial arrangements, leaving performance on complex natural descriptions and changing relations unclear.
Questions for authors
  • How sensitive is SPARC to anchor selection and relation-processing order? Does token replacement preserve object attributes and actions in more complex prompts?
  • Does the implicit-role asymmetry persist across different text encoders, and does its severity predict the benefit of CSP?
  • Please provide a clearer description of how CSP and A2G are implemented and integrated into the generation process. Figure 3 provides limited implementation detail and remains rather abstract.
3 comments
AuthorsOpenReview ↗

Response to Reviewer ipna

Dear Reviewer ipna,

Thank you very much for taking the time to review our manuscript and for providing constructive comments and suggestions to improve our work.

We sincerely appreciate your recognition of our motivation, relational-role asymmetry analysis, and the effectiveness of SPARC in improving spatial controllability without additional training.

We address your concerns about methodological robustness, generalizability, and implementation clarity point by point below.


[W1, Q1] The method relies on several assumptions whose robustness is insufficiently tested: predefined relations, direct replacement of contextualized object embeddings, and sequential conditioning around a selected anchor. It is unclear whether embedding replacement preserves other attributes and actions.

Thank you for your comments regarding methodological robustness. We appreciate your suggestions, which help us strengthen the robustness analysis and evaluate the preservation of object attributes and actions. We address both aspects below with additional experiments and implementation clarifications.

1. How sensitive is SPARC to anchor selection and relation-processing order?

To address this concern, we conduct additional ablations on the multi-relation benchmark in Table 3, evaluating different anchor-selection strategies and relation-processing orders. By default, we select the object with the highest node degree as the anchor because it participates in the most spatial relations. We then process the relation edges sequentially in the order returned by the scene-graph parser. For comparison, we test random and lowest-degree anchor selection. We also evaluate reversed and random relation-processing orders. We keep all other experimental settings identical to ensure a fair comparison. The results are summarized below.

Method / VariantT2V-C ↑VBench ↑VQA ↑VISOR ↑
Wan 2.2 (Baseline)67.4510.3432.9960.09
SPARC (Ours)73.2714.9645.2165.15
SPARC (Lowest-degree anchor)51.617.8716.1731.37
SPARC (Random anchor)60.207.7532.4845.45
SPARC (Reverse order)73.0714.6239.1465.07
SPARC (Random order)71.4810.0538.6464.34

The additional results confirm that our default configuration achieves the highest spatial accuracy. Different relation-processing orders degrade the performance of SPARC. Anchor selection has a more substantial impact, with the highest-degree anchor consistently achieving the best results.

2. Does token replacement preserve object attributes and actions?

We appreciate this question and clarify that CSP replaces only the contextualized reference-object embeddings, while leaving all other token embeddings unchanged. For example, in Fig. 3, "a walking cat is to the left of a standing dog" is inverted to "a standing dog is to the right of a walking cat." Only the contextualized embeddings corresponding to "dog" are replaced. The original attribute and action tokens, including "walking" and "standing," remain unchanged. The auxiliary prompt also retains these modifiers.

To evaluate attribute preservation and action alignment, we conduct additional experiments on the multi-object benchmark used in Table 3. We report CLIPScore to measure overall text-video semantic consistency (including object attributes), and the Action Score from T2V-CompBench to assess how well generated actions follow the prompt.

Method / VariantCLIPScore ↑Action Score ↑
Wan 2.2 (Baseline)27.8114.67
SPARC (Ours)27.7616.04

The results demonstrate that SPARC improves Action Score by 1.37 points (+9.34%) over Wan 2.2, while maintaining a nearly identical CLIPScore (27.76 vs. 27.81, a difference of only 0.05). These findings suggest that our spatial control mechanism improves action alignment without substantially compromising overall text-video semantic consistency. We will also clarify the token-replacement procedure in the revised manuscript.

Dear Reviewer,

Thank you for your review. The authors have submitted their rebuttal. Could you please take a moment to review their responses?

Thank you for your time and contribution!

Best,

AC

Reviewer_ipnaOpenReview ↗

Official Comment by Reviewer ipna

Thank the authors for the additional experiments and clarifications. These results and discussions improve the empirical support and make the method easier to understand. However, I remain concerned about its generality and practical value. The added evaluation still focuses on fixed directional relations, and it remains unclear whether the findings and designs extend to other modern architectures, including advanced VLM-based models, and even more complex, and multi-shot dynamic video scenarios. While I recognize the value of the observations in their context (Wan 2.2), the relevance beyond the evaluated setting remains insufficiently established, given the rapid evolution of video generation community nowadays. I therefore maintain my weak reject rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.