acceptodds
Under review as a conference paper at ICLR 2027

Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generation

19 views

Abstract

Long-form agentic video generation must turn a screenplay into shots that remain spatially and temporally consistent across nonconsecutive requests. Passing prompts or recent frames between shots does not provide a shared spatial frame for revisiting a scene or a time-indexed record for replaying an earlier event. We present Any Time, Any Space (ATAS), which compiles a screenplay into shot requests with explicit location, camera, and event-time references. Before generation, it retrieves relevant evidence from a scene memory of registered static geometry and an event memory of dynamic observations indexed by scene, event, and time. After generation, ATAS reconstructs and registers the clip. It updates memory only when review and state checks accept the observation, while film delivery remains independent of memory admission. Across four video generators, ATAS improves all reported cross-shot consistency measures and achieves the highest aggregate benchmark score for each backbone. It also outperforms the compared agents and 3D-guided methods on target-image spatial revisit, leads three event-recapture criteria, and receives the highest human ratings on all six criteria. Video results are available at https://anonymous-submission-1186.github.io/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long Video Generation, Agentic Framework, Multi-shot Video Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_h7ukOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ATAS, a world-state-centric framework for long-form video generation. It maintains separate scene and event memories for static 3D information and temporally indexed dynamic observations, respectively, and uses state-update validation to control which generated observations are incorporated into persistent memory. The authors evaluate the framework through benchmark experiments, spatial revisiting, event recapture, ablations, and human evaluations across multiple video-generation backbones. While the problem is interesting, the writing and logical flow significantly hinder the understanding of the method and its main contributions.

Critical strengths
  1. The paper addresses an interesting problem in long-form video generation. Separating static scene memory from temporal event memory is intuitive and reasonably motivated.

  2. Separating video delivery from state updates is a useful design choice for preventing unreliable observations from contaminating subsequent generations.

  3. The experimental evaluation covers multiple video-generation backbones, spatial revisiting, event recapture, ablations, and human evaluations.

Critical weaknesses
  1. The writing and logical flow are poor, significantly affecting readability. The motivation, method design, and claimed contributions are not presented clearly enough. The title emphasizes the term world-state-centric, but the abstract does not explicitly define this term or explain how it distinguishes ATAS from conventional video-memory frameworks. Readers must infer the central concept from the subsequent introduction.

  2. The figure captions do not adequately explain the figures. For example, Figure 1 contains multiple video frames, a scene representation, and colored connections, but its caption mainly restates the high-level idea of the paper without explaining the visual elements or how they relate to the proposed pipeline. This significantly reduces the usefulness of the figure. The paper would benefit from a clearer narrative connecting the problem formulation, design choices, and overall workflow.

  3. The novelty of ATAS compared with existing video-memory and 3D scene-memory methods is insufficiently established. The individual contributions of event indexing and validation-controlled state updates remain unclear.

  4. Important implementation details and evaluation protocols need clarification, including the models used for reconstruction and registration, the state-update rules, the definition of the reported aggregate score, and the statistical uncertainty of the results.

Questions for authors
  1. What exactly does world-state-centric mean operationally, and how does it distinguish ATAS from existing persistent-memory frameworks?

  2. Can the authors improve the figures and their captions to explain the visual elements and the information flow through the framework?

  3. Can the authors clarify the implementation details and evaluation protocols to make the method and reported results easier to understand and reproduce?

6 comments
AuthorsOpenReview ↗

Response to Reviewer h7uk (1/3)

We thank Reviewer h7uk for the careful evaluation and constructive suggestions. We appreciate your recognition of the memory design as “intuitive and reasonably motivated” and of separating video delivery from state updates as “a useful design choice.” We have carefully considered all comments and provide our point-by-point responses below.

W1. Writing and logical flow

Thank you for the feedback. We have revised the abstract, introduction, and method description to make the central concept explicit and clarify the connection between our motivation, design choices, and contributions.

Core definition. The world state is the persistent record held in scene memory and event memory. It contains current and historical 3D scene geometry with validity intervals, and dynamic observations indexed by scene, event, and time. Agents query this record before generating shots and update it only with validated observations.

Motivation and flow. Scene revisits require shared geometry. Historical recapture requires selecting an earlier event and the scene geometry valid at that time. Validation determines which new observations can update memory.

We have made the following specific revisions.

  1. Abstract, lines 36–43. We introduce ATAS as a world-state-centric agentic framework and define world state through registered scene geometry, geometry validity intervals, and time-indexed event observations.
  2. Introduction, lines 53–70. We define spatial revisit and historical event recapture, then explain the gap relative to video-memory, 3D scene-memory, and camera-controlled generation methods.
  3. Introduction, lines 71–80. We explain how agents query scene geometry and event observations in shared scene coordinates and render them at the requested camera and time.
  4. Introduction, lines 81–101. We explain how validated observations update changed regions while preserving historical geometry, and state our contributions in scene revisits, event retrieval, and validated memory updates.
  5. Method overview, lines 151–161 and 177–188, and Secs. 3.2–3.5. We explain how shot requests specify scene coordinates and event time, how memory retrieval supplies rendered references, and how validation controls memory updates.

W2. Figure captions and information flow

We have redrawn Fig. 1 and rewritten the captions of Figs. 1 and 2 (lines 22–28 and 170–175) to explain their visual elements and information flow.

Fig. 1. The redrawn teaser shows Shot 1 and Shot 2 alongside the updated world state. Orange silhouettes indicate different times of the walking event, and camera icons identify the queried viewpoints. The two bottom panels separate event recapture and scene revisit, each showing the geometric projection and the generated sequence.

Fig. 2. The caption explains the sequence from shot planning and memory retrieval to rendering, generation, and memory updates. It also clarifies that a clip can enter the film even when its reconstruction is rejected.


Q1. Operational meaning of world-state-centric

World-state-centric means that agents plan and generate shots by querying and updating a shared persistent record of registered 3D scene geometry and past events, stored in scene memory and event memory.

Operational meaning. Agents turn screenplay instructions into requests specifying the scene location, target camera, and, when needed, a historical event interval. To replay an earlier action, they retrieve the matching event observations and scene geometry valid at that time, then render both from the requested camera to guide generation. Only validated observations update memory.

Distinction from existing frameworks. Compared with entity-reference memory in VideoMemory, ATAS provides geometric camera control and retrieval of specific past events and time intervals. Compared with Spatia and SPMem, it uses an agentic workflow to turn screenplay instructions into memory queries, rendering, generation, and validated updates. We clarify this distinction in the abstract (lines 36–43), Introduction (lines 71–88), and Sec. 3.1.


Q2. Figure clarity and information flow

We have redrawn Fig. 1 and rewritten the captions of Figs. 1 and 2 to explain their visual elements and information flow. Please see our response to W2 and the corresponding figures and captions in the revised PDF.

(see next comments for more response, thanks)

AuthorsOpenReview ↗

Response to Reviewer h7uk (2/3)

W3. Novelty and component contributions

We separate the agentic framework from the contributions of event indexing and validated memory updates.

1. Novelty. Our contribution is an agentic workflow that connects shot planning with registered scene geometry, retrieval of specific past events, and validated memory updates.

MethodAgentic planningExplicit 3D/4D geometryGeometric camera controlDynamic reshootingTraining-free integration
OneStory✗✗✗✗✗
StoryMem✗✗✗✗✗
VideoMemory✓✗✗✗✓
SPMem✗✓✓✗✗
Spatia✗✓✓✗✗
GEN3C✗✓✓✓✗
Vista4D✗✓✓✓✗
ATAS✓✓✓✓✓

Training-free integration means using existing generators without method-specific fine-tuning. GEN3C and Vista4D already support dynamic reshooting. ATAS connects this capability to agentic planning, retrieval of specific past events, and validated memory updates across nonconsecutive shots. Recapture uses the scene geometry valid at the requested event time. The capability comparison is also provided in Appendix Table F.

2. Component contributions. We evaluate event indexing and validated memory updates with controlled Wan 3.0 ablations.

Event indexing. Adding indexed event memory to updated scene memory raises C-Top10 from 0.658 to 0.703 and Event score from 0.585 to 0.728, while LPIPS_C changes from 0.286 to 0.285 in Table 8. Below, we compare retrieval of similar events from the same scene with the RGB-D history, renderer, and reference budget fixed. Most recent selects the latest event, Semantic similarity matches by content, and ATAS uses the requested event identifier and time interval. The two accuracy metrics measure correct event and interval selection.

Retrieval ruleEvent retrieval accuracy (%) ↑Interval retrieval accuracy (%) ↑Event score ↑
Most recent event47.329.60.683
Semantic similarity84.179.80.722
Event and time indexing (ATAS)100.0100.00.737

Event and time indexing performs best on all three metrics in Table 5, retrieving the requested event and interval even when several events look similar.

Validated memory updates. Memory updates require two checks. Observation validation checks geometric alignment against held-out historical frames and static correspondences. The state-update check verifies that the reconstruction comes from the reviewed clip, belongs to the requested scene, and uses the current memory version. A clip can enter the film without passing these memory checks. The base row uses recursive compilation without either check. The next two rows add these checks cumulatively under identical requests, cameras, candidate budgets, and retry policies.

ConfigurationC-Top10 ↑LPIPS_C ↓Event score ↑
Base pipeline with recursive compilation0.6840.2880.711
+ observation validation0.7020.2860.726
+ state-update check (full ATAS)0.7190.2840.739

Table 7 shows that each validation stage improves all three metrics, and full ATAS performs best, supporting validation before generated observations are reused in memory.

(see next comments for more response, thanks)

AuthorsOpenReview ↗

Response to Reviewer h7uk (3/3)

W4. Implementation details and evaluation protocols

Thank you for the suggestion. We have clarified the implementation in Secs. 3.3 and 3.5 (lines 252–263 and 283–304) and Appendix C, and the evaluation protocols in Appendix A.6–A.10.

Reconstruction and registration. Pi3X estimates depth and camera poses, and SAM 3 masks dynamic regions. Previously registered frames provide depth, camera poses, and camera intrinsics to constrain reconstruction. We align the result to the scene coordinate frame and validate it on held-out historical frames.

Memory updates. The planner marks regions expected to change before generation. Validated observations update these regions while preserving earlier geometry for historical queries. Ongoing motion is stored in event memory. Before each update, we also check that the reconstruction comes from the reviewed clip, belongs to the requested scene, and uses the current memory version.

Score aggregation. Avg. is the unweighted mean of VBench, ST-Bench, ViStory-Self, and ViStory-Cross. ST-Bench averages Aesthetic, PF-G, PF-S, C-All, and C-Top10. Event Recapture Score, abbreviated as Event score, averages five normalized scores for event identity, action and state preservation, temporal order, character and object consistency, and viewpoint realization within each request, then averages across requests.

Statistical uncertainty. The ± values denote standard errors, calculated as the sample standard deviation of evaluation scores divided by the square root of the total number of evaluated samples. We now state this computation in Appendix A.6 (lines 995–998) and identify the uncertainty as standard error in Table 1.


Q3. Implementation and evaluation reproducibility

For implementation details, score aggregation, and statistical uncertainty, please see our response to W4. Comparisons within each backbone share the generator, prompt package, duration budget, and evaluation protocol. Please also refer to Secs. 3.3 and 3.5, Appendix A.6–A.10, and Appendix C in the revised PDF.

We thank you again for your constructive feedback on presentation and methodology. We hope the clearer definition, redrawn teaser, revised captions, component analyses, and implementation details make the agentic contribution and supporting evidence easier to assess. We welcome any further comments or suggestions.

Reviewer_TRVJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ATAS, an agentic framework for multi-shot long video generation that maintains consistency across non-consecutive shots. Instead of passing recent frames or text summaries, ATAS compiles a screenplay into bounded shot requests with explicit scene, camera, and time references, backed by a two-stage memory: a Scene Memory (registered static 3D point clouds) for revisiting locations, and an Event Memory (time-indexed dynamic RGB-D/point clouds) for replaying past events from new camera views. Generated clips are reconstructed and checked before updating the memory, while decoupling film acceptance from memory admission so failed 3D registrations do not block video delivery.

Critical strengths

(1) this paper decouples the static/dynamic memory and this help to distinguish the role and contribution of different contents/components in the video

(2) the experiments and analysis show that the method can be robust across different models

Critical weaknesses

(1) there are many dynamic events in the video is actually classified as "static" given current paper's claim (if I understand correctly). For example, in the demo page, the bathtub filling up and the fire grouwing larger. Can this point clouds track those state change? If this memory can only track events like human movement and actions, then is this heavy pipeline really necessary?

(2) because each event clip is captured from a single camera trajectory (at least for the first-time cold start), its dynamic point cloud in Et\mathcal{E}_t only covers the visible front surface (a 2.5D shell). When replaying an event from a substantially different viewpoint (eg, the "opposite side" camera c3c_3 in Fig. A), the re-rendered reference CteventC_t^{\mathrm{event}} will miss the back/side geometry entirely and contain large disocclusion holes. How do you prevent front-surface points from bleeding through when rendering from the back, and how do off-the-shelf generators (Wan, HappyHorse) consume these incomplete point-cloud renders without severe artifacts?

Questions for authors

see above weakness, I would like to choose boardline if there is an option but I'm willing to discuss and raise my score

3 comments
AuthorsOpenReview ↗

Response to Reviewer TRVJ (1/2)

We thank Reviewer TRVJ for the careful evaluation and constructive feedback. We appreciate your recognition that separating scene and event memory helps “distinguish the role and contribution” of different components, and that the method can be “robust across different models.” We have carefully considered all comments and provide our point-by-point responses below.

W1. Environmental state changes

ATAS records environmental changes as well as human and object motion.

1. Environmental changes. Event memory stores changing water surfaces and flames as RGB frames with reconstructed depth, indexed by event and time. Stable consequences can update the scene geometry after validation. The planner marks affected regions before generation. Memory records what has been observed, while the generator produces changes between those observations. We clarify this distinction in Sec. 3.3 (lines 235–245 and 258–263) and Appendix C (lines 1207–1212).

2. Why the memory is useful. Under matched Wan 3.0 settings, we compare three memory configurations on bathtub-filling and growing-fire sequences. Historical RGB references supply past frames directly, without geometric reprojection. Static scene memory supplies rendered scene geometry without dynamic event evidence. Full ATAS uses both scene and event memory to render references at the requested viewpoint and event time. The metrics assess action and state preservation, temporal order, and overall recapture quality. Results also appear in Appendix Table C.

SubsetMemory settingAction and state preservation ↑Temporal order ↑Event score ↑
Bathtub fillingHistorical RGB references0.8930.7420.828
Bathtub fillingStatic scene memory0.8870.7860.791
Bathtub fillingFull ATAS0.9410.8080.854
Growing fireHistorical RGB references0.5470.5380.529
Growing fireStatic scene memory0.4460.3610.418
Growing fireFull ATAS0.7460.6930.768

Full ATAS performs best on both subsets across all three metrics, supporting the benefit of combining scene and event memory for environmental changes. Our qualitative examples also show sink and bathtub filling together with updated scene point clouds.

https://anonymous-submission-1186.github.io/anonymous-submission-1186-rebuttal/

(see next comments for more response, thanks)

AuthorsOpenReview ↗

Response to Reviewer TRVJ (2/2)

W2. Large viewpoint changes and incomplete references

Point-cloud renders provide partial geometric guidance, while the video generator completes missing visual content.

1. Back-side visibility. A shared depth buffer resolves occlusion among observed points. A single camera trajectory leaves some surfaces unseen, so correct back-side visibility is not guaranteed. Uncovered pixels remain neutral gray. We do not complete the missing 3D geometry.

2. Using incomplete references. Wan and HappyHorse receive RGB videos rendered from scene and event memory, source images from the original viewpoint, and text. The rendered videos guide the target camera and event timing, while source images provide appearance cues. Generated clips update memory only after reconstruction and validation. We clarify this interface and its visibility limits in Sec. 3.4 (lines 270–275) and Appendix C (lines 1271–1279).

The comparison uses viewpoint changes above 90° under the same requests, prompts, and generation settings within each backbone. Only the visual reference combination changes. Viewpoint realization measures agreement with the requested camera. Visible artifact rate is the percentage of clips with projection or anatomical artifacts identified by Qwen3-VL-plus.

BackboneConditioningAesthetic ↑Viewpoint realization ↑Visible artifact rate (%) ↓Event score ↑
Wan 3.0Source images only0.6040.38113.40.668
Rendered RGB video only0.5790.54730.60.651
ATAS with both references0.5980.56017.80.697
HappyHorse 1.0Source images only0.6090.3288.00.647
Rendered RGB video only0.5690.42626.20.614
ATAS with both references0.5810.43816.30.658

ATAS achieves the highest viewpoint realization and event score on both backbones. Compared with rendered RGB video alone, adding source images reduces visible artifact rates from 30.6% to 17.8% on Wan and from 26.2% to 16.3% on HappyHorse.

Viewpoint breakdown. We group Wan 3.0 ATAS recaptures by source-to-target viewing angle, using both rendered RGB videos and source images throughout. Coverage measures how much of the target view is supported by the projected point cloud.

Viewpoint changePoint-cloud coverage (%)Visible artifact rate (%) ↓Viewpoint realization ↑Event score ↑
0–45°71.60.00.6610.728
45–90°48.49.10.6240.721
90–135°33.216.40.5640.698
135–180°26.821.80.5480.694

As viewpoint changes grow, point-cloud coverage and viewpoint realization decline and artifacts increase, while Event score remains between 0.694 and 0.728. These comparisons are also reported in Appendix A.3, Tables D and E.

We appreciate your careful examination of environmental changes and visibility limitations. We hope the new environmental examples, memory comparisons, and analysis of large viewpoint changes clarify both the benefits and the remaining limitations of ATAS. We welcome your further feedback.

Dear Reviewer TRVJ,

We would like to follow up and ask whether our responses have addressed your concerns. We would be happy to discuss any remaining issues. If our responses have resolved your concerns, we would be grateful if you could update your assessment accordingly.

Thank you for your time!

Best regards,

Authors of Submission #1186

Reviewer_xpzHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ATAS, a persistent world-state framework for multi-shot video generation that aims to improve scene revisits and historical event recapture across nonconsecutive shots. The method decomposes a screenplay into shot requests with explicit scene, camera, and temporal information, and conditions generation on static scene memory and dynamic event memory. After generation, it performs reconstruction, registration, and state checks, distinguishing whether a clip can be included in the final video from whether it can be added to memory. Experiments across four video generation backbones, comparisons with multiple baselines, memory ablations, and human evaluations show improvements in cross-shot consistency and spatial revisits.

Critical strengths
  1. The paper addresses an important problem with a practically useful system design. Scene revisits and historical event recapture are challenging aspects of long-form video generation. Storing static scenes and dynamic events separately, while distinguishing acceptance for the final video from admission to memory, is a reasonable design.
  2. The experimental design is appropriate. The experiments cover multiple generation backbones and examine memory type, capacity, update frequency, and admission policies. The finding that event memory primarily improves event recapture, while scene memory mainly improves spatial consistency, supports the complementary roles of these components.
Critical weaknesses
  1. It is unclear how scene memory handles lasting changes to the environment. The paper does not explain how outdated evidence is invalidated and conflicts are resolved when objects move, doors open or close, or furniture is rearranged. Simply accumulating observations could retain duplicate or contradictory geometry, affecting subsequent scene revisits. The final video may also become inconsistent with the internal memory. Excluding a clip from memory when it fails the admission checks can prevent state contamination, but it may also cause subsequent generation to overlook changes that have already appeared in the video.
  2. Key implementation details are missing. Cross-clip registration, dynamic region separation, point-cloud fusion, and event reprojection are not described in sufficient detail. The appendix mainly provides schematic overviews of the workflow.
Questions for authors
  1. What criteria determine whether registration succeeds and whether an observation is admitted to the persistent state?
  2. How does the system remove or replace outdated geometry after an object moves or the scene layout changes? How does it distinguish temporary occlusion from an object actually leaving its previous location?
  3. When recapturing an earlier event, does the system use the current scene memory or the corresponding historical version? If the environment has changed since the event, how does it avoid a temporal mismatch between the dynamic event and the static background?
  4. Please clarify how the reported ViStory scores correspond to the official metrics. Does the ST-Bench column in the main table report only C-Top10?
  5. I appreciate that the paper reports the increased total runtime, reduced throughput, and storage growth associated with the full-state configuration. However, the analysis covers relatively short sequences. Could the authors further examine how inference overhead scales with sequence length?

I would raise my score if the authors satisfactorily address these concerns.

6 comments
AuthorsOpenReview ↗

Response to Reviewer xpzH (1/3)

We thank Reviewer xpzH for the thorough evaluation and constructive suggestions. We appreciate your recognition of “an important problem with a practically useful system design” and your assessment that “The experimental design is appropriate.” We have carefully considered all comments and provide our point-by-point responses below.

W1. Lasting changes to the environment

ATAS preserves environmental changes through scene updates and image or video references from delivered clips. We clarify both mechanisms in Secs. 3.3 and 3.5 (lines 235–245 and 301–304) and Appendix C.

1. Scene updates. Before each clip, the planner marks point-cloud regions for objects expected to change. After validation, ATAS updates these regions and retains earlier geometry for historical queries. Each query uses geometry valid at the requested time. The append-only baseline adds new observations without invalidating outdated geometry. We compare it with region updates under the same generation settings. The video demonstrations show furniture relocation, door opening, water accumulation, and futon folding alongside updated scene point clouds.
https://anonymous-submission-1186.github.io/anonymous-submission-1186-rebuttal/

Scene-memory write policyC-Top10 ↑PSNR_C ↑SSIM_C ↑LPIPS_C ↓
Append-only0.71115.4100.3770.296
Planner-guided region updates (ATAS)0.71915.9680.3850.284

Planner-guided updates outperform append-only fusion across all four metrics in Table 4, supporting more consistent scene revisits after environmental changes.

2. When memory cannot be updated. We test later shots that depend on a change shown in a delivered clip whose reconstruction fails validation. Both rows use the same last validated scene geometry and Wan 3.0 settings. The second adds images or video from that clip as references, without adding its geometry to memory. The percentages measure preservation of the visible change and contradictions in later shots.

ConditioningEnvironmental change preservation (%) ↑Contradiction rate (%) ↓C-Top10 ↑Extra time (s/shot) ↓
Last validated scene geometry only6.476.10.7020
+ References from delivered clips (ATAS)62.928.60.7085.3

Appendix Table A shows that these references improve environmental change preservation and reduce later contradictions, with an additional 5.3 seconds per shot.


W2. Implementation details

The four operations are implemented as follows.

1. Cross-clip registration. Pi3X[1] reconstructs each clip using historical depth and camera priors. A similarity transform aligns it to the scene frame, with held-out historical frames and independent static correspondences used for validation.

2. Dynamic-region separation. SAM 3 generates and propagates object masks within each clip. Event memory stores dynamic regions as RGB frames with reconstructed depth, called RGB-D observations, together with their masks. Stable geometry updates the scene point cloud after validation, as described in W1.

3. Point-cloud fusion. New static points are checked for duplicates, agreement with historical views, and consistency over time before fusion. Unchanged geometry is preserved, while affected regions are updated as described in W1.

4. Event reprojection. Retrieved RGB-D observations of dynamic regions are unprojected into scene coordinates and rendered with the scene geometry valid at the event time. A shared depth buffer resolves visibility among observed surfaces, producing a rendered RGB video at the requested camera.

We have added these implementation details to Secs. 3.3 and 3.5 (lines 252–263 and 283–286) and Appendix C.

[1] π³: Permutation-Equivariant Visual Geometry Learning (ICLR 2026)

(see next comments for more response, thanks)

AuthorsOpenReview ↗

Response to Reviewer xpzH (2/3)

Q1. Registration and state-admission criteria

An observation updates memory only after passing film review, registration validation, and state-update checks.

Geometric registration. We check alignment on held-out historical frames and independent static correspondences, using 3D error relative to scene depth and the fraction of consistent matches. We also check agreement across reconstruction windows and camera projection accuracy. Appendix C (lines 1231–1237) gives the criteria.

State-update checks. The reconstruction must come from the reviewed clip, belong to the requested scene, and use the current memory version. If memory has changed since reconstruction, we register the candidate again before adding it.

Memory acceptance ablation. Permissive and Strict relax and tighten the Default acceptance criteria, respectively. All variants share the generator, requests, candidate budget, and retry policy. State-update rate reports the resulting percentage of candidate observations accepted into memory.

State-update settingState-update rate (%)C-Top10 ↑LPIPS_C ↓Event score ↑
Permissive90.00.6960.2930.708
Default67.40.7190.2840.739
Strict36.00.6680.2880.682

The default setting achieves the best performance across all three quality metrics, outperforming both permissive and strict admission settings.

We have clarified these in Sec. 3.5 (lines 283–304) and Appendix C. The sensitivity analysis is reported in Appendix D, Table N.


Q2. Outdated geometry and temporary occlusion

ATAS distinguishes environmental changes from temporary occlusion using planner hints and validated observations.

1. Outdated geometry. Validated observations update the affected regions of the scene point cloud, while earlier geometry is retained for historical queries. Each query selects the geometry valid at its requested time. In the armchair demo, the updated scene point cloud places the chair near the window and leaves its previous location beside the sofa unoccupied.
https://anonymous-submission-1186.github.io/anonymous-submission-1186-rebuttal/#armchair

2. Temporary occlusion. An object disappearing from a view does not by itself invalidate its stored geometry. An update requires a change in object location or configuration identified by the planner and confirmed in the generated clip.

We have clarified this distinction in Sec. 3.3 (lines 235–239) and Appendix C (lines 1207–1212).


Q3. Historical scene memory for event recapture

Historical recapture uses the scene geometry valid at the requested event time.

1. Scene memory. We maintain a shared scene point cloud and update regions affected by validated object changes. Earlier points are retained, so historical recapture selects the scene geometry valid at the requested event time.

2. Temporal alignment. We render this historical scene geometry together with the matching event observations at the target camera, excluding later changes to the scene.

Appendix Table B compares the two selections on events followed by environmental changes, keeping event evidence, cameras, and generation settings fixed.

Scene geometry used for recaptureHistorical scene accuracy (%) ↑Event score ↑
Latest scene geometry81.80.687
Geometry at event time (ATAS)100.00.722

ATAS achieves the best performance on both metrics by selecting geometry from the requested event time.

Historical scene accuracy is the percentage of checks on object locations and configurations that match the source scene. We clarify this geometry selection in Sec. 3.3 (lines 243–246) and Appendix C (lines 1214–1221).

(see next comments for more response, thanks)

AuthorsOpenReview ↗

Response to Reviewer xpzH (3/3)

Q4. ViStory and ST-Bench metric definitions

Thank you for asking us to clarify the reported metrics.

1. ViStory. Following ReCA [2], we use the labels ViStory-Self and ViStory-Cross. Our Self score averages normalized CIDS-Self, CSD-Self, and temporal self-consistency, with temporal scores averaged over time. Cross averages normalized CIDS-Cross, CSD-Cross, and Prompt Alignment–Scene against the prescribed character, style, and scene conditions. Each aggregate is the arithmetic mean of its three components. We have clarified these definitions and aggregation in Sec. 4.1 (lines 360–363) and Appendix A.10 (lines 1116–1122).

2. ST-Bench. We replace C-Top10 with the mean of all five metrics in Table 1. Within each backbone, methods share generation and evaluation settings. Aesthetic measures visual quality, PF-G and PF-S measure global and single-shot prompt following, and C-All and C-Top10 measure cross-shot consistency. The complete results below also appear in Appendix A.7, Table G.

BackboneMethodAesthetic ↑PF-G ↑PF-S ↑C-All ↑C-Top10 ↑Mean ↑
MiniMax-H3I2V Extension0.6640.2190.2190.5360.5720.442
VGoT0.6720.2290.2240.5780.6120.463
Mora0.6780.2270.2270.5800.5980.462
MovieAgent0.6910.2310.2290.5740.6080.467
ReCA0.6840.2340.2250.6030.6280.475
ATAS0.6950.2350.2310.6530.6910.501
Wan 2.7I2V Extension0.6170.2040.2070.4490.4860.393
VGoT0.6240.2120.2110.5080.5360.418
Mora0.6320.2100.2130.5110.5280.419
MovieAgent0.6350.2130.2140.5150.5420.424
ReCA0.6560.2160.2090.5440.5680.439
ATAS0.6390.2170.2150.5980.6250.459
HappyHorse 1.0I2V Extension0.6570.2140.2140.5290.5580.434
VGoT0.6440.2220.2180.5600.5880.446
Mora0.6510.2200.2210.5650.5740.446
MovieAgent0.6760.2250.2230.5690.5980.458
ReCA0.6620.2270.2170.6080.6210.467
ATAS0.6790.2290.2240.6620.6680.492
Wan 3.0I2V Extension0.6410.2170.2220.5100.5480.428
VGoT0.6360.2370.2260.5410.5710.442
Mora0.6570.2320.2300.5460.5620.445
MovieAgent0.6700.2390.2330.5440.5780.453
ReCA0.6520.2220.2190.5750.6020.454
ATAS0.6730.2420.2360.6840.7190.511

ATAS retains the highest aggregate score under every backbone when all five ST-Bench metrics are included.

[2] ReCA: Multi-Shot Long Video Extrapolation via Recursive Context Allocation (arXiv 2026)


Q5. Inference overhead with longer sequences

We continue the same stories to 10, 20, 40, and 80 shots under unchanged Wan 3.0 settings. Timings in seconds per shot cover reference retrieval and rendering, reconstruction and registration, and memory updates. Storage is the total size of scene and event memory in GB. The results also appear in Table 6 and Appendix A.4.

ShotsQuery and renderReconstruct and registerState updateMemory storage (GB)C-Top10 ↑Event score ↑
1018.294.787.42.910.7150.734
2018.797.189.83.940.7120.728
4022.199.891.66.280.7040.721
8024.8106.498.38.610.6990.718

From 10 to 80 shots, query and render time per shot increases by 36.3%, while the other two timed stages increase by about 12%. Storage grows approximately threefold, while C-Top10 and Event score each decrease by 0.016. These results show moderate per-shot cost growth and limited quality degradation over the tested range.

We thank you again for the detailed suggestions. We hope the added comparisons of scene updates and historical recapture, clarified evaluation metrics, and analysis of longer sequences address your concerns. We welcome any further questions or feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.