acceptodds
Under review as a conference paper at ICLR 2027

Structura: Intrinsic Scoring and Inference-Time Refinement against Structural Hallucinations in Video Generation

Abstract

Modern video generators produce visually compelling clips, yet objects can develop impossible structures as they move: limbs fuse or multiply, parts detach, and geometry changes inconsistently across frames. Mitigating these spatiotemporal structural hallucinations requires feedback that identifies emerging failures and guides correction during sampling. We introduce Structura, a training-free framework that obtains this feedback from the generator’s own reconstruction dynamics. We slightly re-noise each intermediate clean prediction and reconstruct it, using low-noise reconstruction as a local geometric probe. Unreliable regions tend to change more under this probe, and large changes recur at the same locations when a structural failure persists across sampling steps. Structura accumulates the magnitude of these changes to stabilize spatiotemporal localization, while retaining their signed displacement to guide stochastic local resampling. We introduce Structura Bench, a collection of hallucination-prone cases for evaluating video structure. Experiments on Dynamic-Bench and Structura Bench show improved structural and anatomical correctness across video generators. Applications to video object removal and world action models further demonstrate the use of reconstruction feedback for temporal refinement and action completion.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Diffusion, Video Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_28WbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Structura, a training-free inference-time refinement method to reduce structural and anatomical hallucinations (eg, distorted limbs, morphing objects) in flow-matching video generators.

Critical strengths

(1) this paper presents a interesting analysis of decoupling of energy magnitude and signed direction. Accumulating scalar energy B^n\hat{B}_n separately from vector displacement A^n\hat{A}_n is a nice design if a region oscillates in direction across steps, A^n\hat{A}_n cancels out (falling back to isotropic exploration) while B^n\hat{B}_n remains high to keep flagging the region as structurally unstable.

(2) connecting low-noise reconstruction residuals to local manifold projection and deriving an unbiased variance-corrected energy estimator is mathematically clean and well-motivated.

Critical weaknesses

(1) table 1 shows the method severely reduces the dynamic degree compared to baselines, can authors elaborate more on this effect? How do this method prevent the probe from mistaking legitimate fast motion for structural hallucinations?

(2) how is the 10,000-video editing dataset collected and processed, and why are 10000 samples needed if Structura is purely training-free and only evaluated on 100 samples?

(3) What are the exact values of K,mi,β,τ,γ,ηK, m_i, \beta, \tau, \gamma, \eta, total NFEs, and wall-clock runtime per video for Structura vs. SRVS, NFE×2, and Best-of-4 in Table 1? How does Structura perform against Best-of-N under a strictly matched NFE budget?

Questions for authors

see above, I'm willing to discuss and raise my score

2 comments
AuthorsOpenReview ↗

Response to 28Wb: motion, paired editing data and inference cost

Thank you for highlighting the separate energy/direction memories and the geometric motivation.

1. Dynamic Degree and rapid motion. Structura lowers Dynamic Degree relative to some baselines; SRVS shows the same refinement trade-off. When a backbone struggles to reconstruct complex motion consistently, repeated refinement can favor more conservative movement. Dynamic Degree measures motion prevalence, which can also reflect jitter and inconsistent deformation. Relative to SRVS, Structura improves both motion prevalence and structure in original Table 1:

Dynamic-Bench

MethodAutoEval: Dyn. ↑VLM: Struct. ↑
NFE×20.73332.66
SRVS0.55003.06
Structura0.60003.24

Structura Bench

MethodAutoEval: Dyn. ↑VLM: Struct. ↑
NFE×20.93262.54
SRVS0.78282.85
Structura0.83902.94

Our probe measures reconstruction response rather than motion speed. Figure 8 juxtaposes a turning dancer, inter-frame motion and accumulated response: an extra limb with relatively little displacement still elicits response during clear body motion. The example shows response to a structural anomaly during substantial body motion. See Appendix E, p. 22, Table 26, Figure 8.

2. The approximately 10k editing resource. We carefully curate an internal collection of high-quality paired video-editing data involving hand–object contact, occlusion and manipulation. It contains 10,634 video pairs and 20,987 editing instructions, with object references and temporal target localization. The full collection is a reusable resource; Structura uses frozen models at inference time. Own-100 is a subset covering every domain in the collection. Its 100 cases retain that coverage at an evaluation scale comparable to our DAVIS-2017 protocol (61 validation object instances). We will release the data and its processing and evaluation protocols. Appendix F, pp. 23–25, Tables 27–29, Figures 9/10 documents the construction and statistics.

3. Cost of the original configurations. The complete measured mean NFE list for original Table 1 is:

MethodMean NFE ↓
Structura196.00
SRVS145.80
NFE×2160.00
Best-of-4320.00
FlowMo120.00
CFG-Zero80.00
CachedSearch250.27

Appendix A, p. 13, Table 6 records the submitted configurations; the matched-budget experiment is reported separately.

For the strict-budget question, our general response compares Lite160, Native80 and Best-of-2 at exactly 160 denoiser NFEs. Lite160 improves structure/anatomy on both cohorts, with wall time close to Native80 and below Best-of-2. Both Lite configurations use K=2; their exact refinement schedules and measured runtimes are given in Appendix A. The K ablation is in Appendix D.

AuthorsOpenReview ↗

Gentle follow-up on our discussion and revisions

Dear Reviewer 28Wb,

Thank you again for your insightful feedback and your encouraging remark that you are "willing to discuss and raise your score."

With the discussion deadline approaching, we wanted to kindly check in and see if our detailed responses and additional experiments have resolved your concerns—particularly regarding the analysis on dynamic degree/fast motion, the clarification on the 10k dataset, and the strictly matched NFE/runtime comparisons against Best-of-N.

If these clarifications and new results satisfactorily address your points, we kindly invite you to consider adjusting your rating accordingly. We remain fully available and eager to address any further thoughts you might have during the remaining time!

Best regards, The Authors

Reviewer_WDmDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a training-free method for guiding flow matching sampling to prevent structural hallucination. It first provides insights and mathematical proof on how probing an intermediate sample could provide magnitude and direction to a data-supported geometry. Then, it introduces feedback refinement that steers the refinement process according to the direction of the probing response. The paper is mathematically sound, insightful, and the claims are well-supported by experimental results.

Critical strengths

S1. The observation on how low-noise reconstruction could provide magnitude and direction for guidance is insightful and mathematically proven in the appendix.

S2. The proposed feedback-guided refinement is simple, well-motivated, and empirically validated to produce good results across different benchmark

Critical weaknesses

W1. As the correction step pulls intermediate prediction onto data-supported geometry, it could be at the cost of lower diversity yet not mentioned or discussed within the paper. An experiment that measures diversity of the generated samples would be appreciated.

W2. From equation (6) and (7), the correctness of the response increases as K increases, at the expense of higher NFE and thus the efficiency of the method. Line 710 seems to suggest K is fixed to 2 with no empirical support. A small experiment to study the effect of K and its resulting efficiency would be insightful.

Questions for authors

Suggestion: Although interesting upon careful inspection, the observation in figure 3 lacks sufficient experiment details and settings that could cause confusion while reading. Please consider adding more details.

4 comments
AuthorsOpenReview ↗

Response to WDmD: generation diversity, K and Figure 3 details

Thank you for recognizing the geometric insight and feedback-guided refinement.

W1. Generation diversity. We evaluate 81 prompts × 4 seeds × 3 methods = 972 videos, using the same Wan2.1-1.3B backbone, Euler sampler and seeds. We average the six cross-seed distances per prompt, then equally across prompts. Larger values indicate greater diversity.

Dynamic40

MethodLPIPSVGG contentVGG style ×1000ViCLIP distance
Vanilla0.62582.09898.87020.2618
SRVS0.61291.76326.86710.2459
Structura-Full0.61541.79776.70150.2587

Structura41

MethodLPIPSVGG contentVGG style ×1000ViCLIP distance
Vanilla0.60532.35126.82300.2219
SRVS0.58462.12105.17690.2042
Structura-Full0.59622.14065.35460.2110

Structura retains 98.3%/98.5% of Vanilla LPIPS diversity and has higher mean LPIPS and ViCLIP diversity than SRVS on both sets. VGG content/style retention is 85.6%/75.5% on Dynamic40 and 91.0%/78.5% on Structura41. Correction largely preserves cross-seed perceptual diversity while reducing content/style variation.

LPIPS uses 16 matched temporal positions; VGG content/style use all 81 frames; ViCLIP uses video-embedding cosine distance. Each method uses 40 outer steps, with measured mean NFE approximately 80/120/250. Appendix D, p. 20, Tables 21–22 contains the protocol and prompt-bootstrap intervals; the main paper points to this discussion.

W2. Probe count K. K controls the precision of the response estimate: under independent probes, mean-response covariance scales as 1/K. Equation (7) is already unbiased for K≥2 under its stated assumptions. Accumulation combines feedback across rounds. We evaluate 30 Dynamic and 42 Structura videos for each K:

Tables separate AutoEval and VLM as Judge (Gemini). Bold marks the highest value for each metric within a benchmark, including ties.

Dynamic-Bench (N=30)

MetricK=2K=4K=8
AutoEval ↑
Smooth0.98900.98940.9878
Dyn.0.63330.46670.4333
Flicker0.98440.98460.9833
SA3.16673.00003.0000
PC3.86673.86673.9000
Joint (%)23.333330.000023.3333
VLM as Judge ↑
Physical2.63332.60002.6667
Structural2.93332.96673.0000
Anatomical3.23333.26673.2333

Structura Bench (N=42)

MetricK=2K=4K=8
AutoEval ↑
Smooth0.98790.98820.9876
Dyn.0.78570.78570.7857
Flicker0.97980.98030.9788
SA3.57143.59523.5238
PC3.90483.97623.9524
Joint (%)42.857152.381050.0000
VLM as Judge ↑
Physical2.69052.73812.6429
Structural2.80952.78572.8571
Anatomical2.90483.04763.0952

The following cost summary is aggregated by K across the two cohorts (lower is better):

KTotal NFE (equiv.)Seconds/video
2196.00639.37
4277.54893.50
8440.611405.17

K=2 is the smallest valid variance-correction setting. K=8 gives modest structural gains (+0.07/+0.05) at 2.20 times the runtime; it improves Structura anatomy, while Dynamic Degree falls from 0.63 to 0.43 on Dynamic. These results support K=2 as a practical quality–cost default. Appendix D, p. 21, Tables 23–25 reports the full ablation and costs.

Figure 3 settings. We add explicit experimental details in Appendix B. The geometric test uses known local geometry; its extension covers six curve families, dimensions 32/64/128, two decoder gains and eight seeds per family, with 84 endpoint calls per equal-budget method. Figure 7 shows all six families. The original trajectory panel follows five prompts × six seeds, with 40 sampling steps per trajectory. The horizontal axis indexes solver steps, and the response energy is defined by Eq. (6).

AuthorsOpenReview ↗

Gentle follow-up on our response and added experiments

Dear Reviewer WDmD,

Thank you once again for your positive assessment of our work's mathematical soundness and insights.

As the discussion phase nears its end, we wanted to follow up and ensure that our recent revisions—specifically the new experiments measuring sample diversity, the ablation study on K K vs. efficiency (NFE), and the expanded details for Figure 3—have adequately addressed your valuable suggestions.

We hope these additions further strengthen the paper in your view, and we would greatly appreciate any feedback or updates to your assessment. Please let us know if there is anything else we can clarify in the remaining time!

Best regards, The Authors

Reviewer_WDmDOpenReview ↗

Thank you for the clarifications

Thank the author for the detailed rebuttal. The diversity experiment addresses my concern. The study on the effect of K is quite inconclusive, as increasing K does not seem to influence the results significantly (and sometimes K=8 results in worse metrics than K=2) despite the massive increase in computation, which is quite counter-intuitive. Additional qualitative results that highlight the difference of different K under the same setting that could help visualize improvements that cannot be captured with metrics. I will keep my original score in favor of acceptance.

AuthorsOpenReview ↗

Added qualitative comparisons across K

Dear Reviewer WDmD,

Thank you for maintaining your positive rating in favor of acceptance and for suggesting a closer qualitative comparison across KK.

We have added side-by-side response-map examples as Figure R1 at the beginning of Appendix D (p. 20). Across K=2,4,K=2, 4, and 88, isolated pink responses become noticeably less prominent as KK increases, while the primary structural response regions remain consistently visible.

Regarding the quantitative ablation, increasing KK does not yield a strictly monotonic final-quality gain for three reasons:

  1. Monte Carlo precision vs. unbiasedness: KK controls the single-step Monte Carlo precision of the feedback. Under independent probes at a fixed state, the mean-response covariance scales as 1/K1/K, whereas Eq. (7) is already unbiased for every K≥2K \ge 2. Increasing KK reduces random probe variance (as reflected in Figure R1), but it does not remove the frozen generator's systematic reconstruction error.
  2. Trajectory accumulation: Our framework already accumulates feedback across sampling rounds, which naturally smooths single-step estimation variance over the trajectory. Moreover, a slightly more precise local estimate often leaves the discrete refinement decision unchanged.
  3. Local changes vs. whole-video metrics: The visible differences in Figure R1 concern small, localized regions of the response maps. Whole-video rubric scoring and averaging across prompts naturally attenuate the contribution of such fine-grained local changes. Consequently, more precise local feedback need not produce a monotonic increase in every video-level metric.

Together, Figure R1 illustrates the expected local variance-reduction behavior of the feedback estimator, while the final-quality and runtime results remain those in Tables 23–25, where K=8K=8 requires 2.20×2.20\times the runtime of K=2K=2. We therefore retain K=2K=2 as the practical default.

Thank you again for your constructive feedback and support!

Best regards,
The Authors

Reviewer_tx6LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces structura, a training free inference framework for reducing structrual hallucinations in video generation. It probes intermediate clean predictions by slightly renoising and reconstructing them with the frozen generator. Variance corrected residual energy is accumulated across sampling steps to localize persistent disagreement. And the accumulated signed residuals will guide the directional renoising. Candidate updates are accepted through paired reconstruction comparisons using shared perturbations.

Critical strengths
  1. The paper proposes a meaningful extension of generator-native refinement
  2. The feedback estimator and paired comparison are technically well motivated.
  3. The component ablations support the usefulness of trajectory accumulation and directional steering.
Critical weaknesses
  1. Performance gains are not consistent over different backbones.
  2. The method requires several times of renoising and multiple low noise reconstruction, but the extra inference cost is not well discussed.
  3. The central structural scoring hypothesis lacks independent validation.
  4. Proprio is a quite similar work and discussed in the related works but not compared in the experiment.
Questions for authors
  1. what would happen in the comparison if the computational cost are fixed for each method?

Would be happy to raise score if all the concerns are resolved.

3 comments
AuthorsOpenReview ↗

Response to tx6L (1/2): backbone variation and independent validation

Thank you for recognizing the generator-native refinement extension and the motivation for our estimator and paired comparison.

1. Variation across backbones. Structura improves structural scores across the tested backbones, with different gain magnitudes. The table reproduces the SRVS comparison from original Tables 1 and 4.

Backbone / benchmarkSRVSStructuraGain
Wan2.1-1.3B / Dynamic3.063.24+0.18
Wan2.1-1.3B / Structura2.852.94+0.09
Wan2.2-5B3.413.59+0.18
CogVideoX-2B2.762.77+0.01
CogVideoX-5B2.983.20+0.22

Backbone training, noise schedules, timestep parameterizations and samplers affect the clean prediction and reconstruction feedback. For CogVideoX-2B, limited structural modeling may constrain both the samples that can be corrected and the discrimination of its own feedback: when coarse failures dominate, local changes may have little effect on a video-level score. VideoPhy's independent human evaluation reports physical-commonsense pass rates of 34.1%/53.0% for CogVideoX-2B/5B, consistent with this capability gap (VideoPhy results). Tables 2/31 and 5 additionally evaluate the feedback mechanism in World Action Models and video editing.

2. Independent validation of structural feedback. We add complementary geometric and video-region evidence in Appendix B. The controlled reconstruction experiment covers six curve families, dimensions 32/64/128, two decoder gains and eight seeds per family: 288 paired settings. With known local geometry and the same 84 endpoint calls, Structura has lower reconstruction MSE than equal-budget Self-Refine in every setting; the geometric mean paired MSE ratio is 0.0724. Appendix B, pp. 15–16, Tables 10–11 specifies the analytic endpoint, seeds, dimensions and controls, with all six families in Figure 7.

For video localization, Gemini independently labels polygons in 38 videos at the preselected frames 20/40/60 (114 frames), viewing only the original frame and prompt. In the category-sampled primary group, 8 videos/15 frames contain both positive and negative reviewed regions suitable for AP. Comparing cumulative response maps from the first and last saved sampling observations against the same final frames, mean AP rises from 0.191 to 0.383, with improvement in all 8 videos. These results quantify alignment with independently annotated structural defects; Appendix B, pp. 16–17, Tables 12–16 reports coverage, eligibility and the coarse-label protocol.

3. Fixed computation. Please see our general response and Appendix A: at exactly 160 denoiser NFEs, Structura-Lite improves structural and anatomical scores over both longer native sampling and Best-of-2 on the two evaluation cohorts.

AuthorsOpenReview ↗

Response to tx6L (2/2): Proprio and the geometric feedback construction

4. Relation to Proprio and added experiments. We appreciate the reviewer’s careful reading and this valuable pointer. Proprio is an insightful and meaningful concurrent study. Our core distinctions lie in both the mathematical construction of the scoring function and the overall architectural design. Proprio uses flow-matching regression error as an internal proxy for physical plausibility: it compares predicted velocity with the known perturbation target (ε−z₀ for generated latent z₀ under a linear flow path), aggregates squared residuals into a score, and uses it for selection or gradient-based refinement. Its refinement branch backpropagates through the frozen sampler to optimize the initial noise.

Structura's core insight is geometric. Under the local regularity assumptions in Section 3.2, low-noise reconstruction approximates projection onto the conditional data manifold, and its ideal mean residual satisfies

μp∗(x)=x−ΠMc(x)+O(p2).\mu_p^*(x)=x-\Pi_{\mathcal M_c}(x)+O(p^2).

The magnitude measures local deviation, while the negative direction points toward recovery. Structura estimates squared mean displacement with variance correction, accumulates energy and signed displacement separately across sampling steps, and uses directional re-noising with paired forward reconstruction to select local updates.

AspectProprioStructura
Core insightLearned-flow consistency as a plausibility proxyLow-noise reconstruction as geometric projection
FeedbackAggregated sample scoreLocal deviation magnitude and recovery direction
Spatial emphasisMotion-based weightingPersistent local revision demand
AccumulationPerturbations/noise levels for a generated sampleEnergy and direction across evolving sampling states
RefinementBackpropagate to initial-noise parametersDirectional re-noising and paired forward selection

We reproduce Proprio on TurboWan2.1 1.3B, evaluating 30 Dynamic and 134 Structura videos with AutoEval and the original Gemini rubric. Gemini structural scores are 2.50/2.41 and anatomical scores are 3.10/2.66; AutoEval PC is 3.60/3.93 and Joint is 20.00%/46.27%. Relative to the Wan2.1-1.3B NFE×2 reference in Table 1, Proprio has higher AutoEval PC (3.60/3.93 vs. 3.48/3.77), while Structura retains higher structural scores (3.24/2.94 vs. 2.50/2.41). Across these separately evaluated backbones and cohorts, the pattern is consistent with Proprio's physical-plausibility objective and Structura's emphasis on structural correction.

The implementation uses a four-step stochastic-consistency sampler, ten optimization iterations and four perturbations at each of four noise levels; the output minimizes the unregularized hybrid score among the evaluated candidates. The measured mean denoiser NFE is 207.30, including forward recomputation; the recorded runtime also includes backpropagation. Appendix C, pp. 18–19, Tables 17–20 provides the quality metrics, configuration and runtime, and relates the construction to Proprio Sections 3.1–3.3 and Algorithm 1.

AuthorsOpenReview ↗

Gentle follow-up on our response and new experiments

Dear Reviewer tx6L,

Thank you again for your constructive review and for generously noting that you "would be happy to raise the score if all the concerns are resolved."

As the discussion period is drawing to a close, we wanted to gently follow up to see if our response and the newly added experiments—specifically the fixed-computation comparisons, inference cost analysis, comparisons with Proprio, and independent validation of our scoring hypothesis—have fully addressed your concerns.

If you find that our revisions have resolved your questions, we would be deeply grateful if you could consider updating your score to reflect these improvements. Should you have any remaining questions, we are standing by and happy to discuss further before the deadline!

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: additional evidence and experiments

We thank the reviewers for their careful assessment. The new evidence is located in the revised PDF as follows:

  • Inference cost and matched budgets: Appendix A, pp. 13–14, Tables 6–9.
  • Geometric reconstruction and independent localization: Appendix B, pp. 15–17, Tables 10–16, Figure 7.
  • Proprio construction and reproduction: Appendix C, pp. 18–19, Tables 17–20.
  • Generation diversity and K: Appendix D, pp. 20–21, Tables 21–25.
  • Motion and structural response: Appendix E, p. 22, Table 26, Figure 8.
  • Paired editing data: Appendix F, pp. 23–25, Tables 27–29, Figures 9/10.

Matched denoiser budgets. We evaluate 30 Dynamic and 134 Structura videos per method using the original AutoEval dimensions and Gemini rubric. Structura-Lite reduces refinement rounds while retaining the core method. The complete results are grouped into AutoEval, VLM as Judge (Gemini), and computation/runtime. Bold marks the best quality or lowest time within each benchmark and budget, including ties; NFE is unhighlighted. Compare SRVS120/Lite120 at the 120-NFE cap and Native80/Best-of-2/Lite160 at 160 NFE.

Dynamic-Bench (N=30)

MetricSRVS120Lite120Native80Best-of-2Lite160
AutoEval ↑
Smooth0.98990.98880.98050.97890.9904
Dyn.0.56670.73330.70000.73330.5667
Flicker0.98550.98170.96900.96860.9856
SA3.06673.20003.33333.33333.1667
PC3.73333.70003.43333.46673.8333
Joint (%)26.666723.333333.333333.333326.6667
VLM as Judge ↑
Physical2.56672.56672.53332.56672.7333
Structural2.83332.86672.66672.66673.0000
Anatomical3.23333.16672.96672.93333.3667
Computation / runtime
Mean NFE120.0000120.0000160.0000160.0000160.0000
NFE range120–120120–120160–160160–160160–160
Generation ↓384.90387.00510.53519.05515.80
Wall ↓398.90400.96524.57552.74529.94

Structura Bench (N=134)

MetricSRVS120Lite120Native80Best-of-2Lite160
AutoEval ↑
Smooth0.98570.98270.97480.97270.9857
Dyn.0.80600.85820.94780.94030.8134
Flicker0.97730.97120.95900.95680.9764
SA3.56723.56723.61943.58213.5896
PC3.98513.93283.84333.83584.0672
Joint (%)49.253743.283647.014943.283652.2388
VLM as Judge ↑
Physical2.64932.46272.44032.41042.7015
Structural2.43282.67912.42542.27612.7313
Anatomical2.67162.85072.65672.58962.9552
Computation / runtime
Mean NFE120.0000119.9104160.0000160.0000160.0000
NFE range120–120108–120160–160160–160160–160
Generation ↓390.55388.76514.39528.23521.24
Wall ↓402.99401.52526.90559.56533.93

All runs in the 160-NFE comparison use exactly 160 denoiser evaluations. Native80 uses 80 native UniPC steps; Best-of-2 uses two 40-step candidates and Cosmos-Reason1-7B selection, whose cost is additional to denoising. Lite160 improves structure and anatomy on both benchmarks. Its wall time is 1.02%/1.33% above Native80 and 4.12%/4.58% below Best-of-2 on Dynamic/Structura. Structural benefits also appear at the 120-NFE cap.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.