acceptodds
Under review as a conference paper at ICLR 2027

SceneQuery: Target-Queried Scene Memory for Visuospatial Video Generation

Abstract

Visuospatial video generation requires synthesizing videos that satisfy target-object or camera-trajectory instructions while remaining consistent with an observed environment. Although existing methods condition generation on scene observations, how to retrieve task-relevant scene evidence for individual output positions remains less explored. We introduce SceneQuery, a task-conditioned scene retrieval framework in which task instructions guide not only what to generate, but also what scene evidence to retrieve. SceneQuery constructs global and spatial memory from RGB observations and derives a query for each output token using target-object semantics and token coordinates for grounding, or the corresponding target-camera ray for navigation. Unlike queries derived from evolving target features, these task-derived queries remain fixed throughout denoising, providing a stable, output-aligned retrieval signal. The retrieved evidence conditions video generation, while the memory and retrieval modules are jointly optimized through the generation objective without additional memory supervision or explicit 3D reconstruction. Controlled query-source comparisons support task-derived retrieval across both tasks. On ScanNet++, selected configurations improve target-view reconstruction for navigation and reduce geometric error for grounding relative to a matched context-guided baseline, while the primary joint grounding-success rate remains unchanged.

Primary area:
generative models
Keywords:
video generaion, object grounding, scene navagation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_gxwDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SceneQuery constructs scene memory from RGB observations and uses queries derived from task instructions and output positions to retrieve information for video generation. The queries remain fixed throughout denoising. The method is evaluated on object grounding and scene navigation on ScanNet++, with improvements in navigation reconstruction and grounding geometry, but no improvement in the primary joint grounding-success rate.

Critical strengths

The query-source comparison in Table 3(b) keeps the memory organization fixed and disables the task adapter. It provides useful evidence that task-derived queries outperform noisy-state queries. The navigation results also show a meaningful reconstruction improvement over the matched Video4Spatial baseline, suggesting that the proposed design has practical value.

Critical weaknesses
  1. The code is not open sourced. Figure 3 refers to supplementary videos, but I could not find any supplementary files or code in the submission. Were these materials omitted? Do the authors plan to release the implementation and evaluation scripts?
  2. The evaluation needs clarification. The paper explicitly states that object-grounding evaluation coverage remains unresolved, but does not report valid sample counts or how evaluation failures are handled. With 169 context frames, Table 2 reports an SD of 0.2421, whereas Tables 3 and 4 report 0.1764 for the selected full model. Which checkpoints or settings account for this difference? These unresolved issues undermine confidence in the reported comparisons.
  3. Comparison with related memory methods is insufficient. The main contribution concerns how scene information is retrieved, yet the experiments omit relevant methods such as MemLearner [1], which learns to query context memory, and representative retrieval approaches such as Context-as-Memory [2] and VMem [3]. Table 3(b) supports task-derived queries over noisy-state queries, but does not establish an advantage over these existing approaches. Comparisons with their retrieval strategies under matched conditions, or a clear explanation of why such comparisons are unsuitable, would help establish the contribution.

[1] Yu et al. MemLearner: Learning to Query Context Memory for Video World Models. ECCV, 2026.
[2] Yu et al. Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval. SIGGRAPH Asia, 2025.
[3] Li et al. VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memory. ICCV, 2025.


updates in 2026.11.17

The authors have uploaded the videos for visualization this time, making the work more convincing. The replys address my concerns mostly. Therefore, I'm willing to improve the score from 2 to 3. Good luck.

Questions for authors
  • Please clarify the valid sample counts and failure handling for object grounding. Are all methods evaluated on the same samples with the same denominator?
  • What checkpoints or settings explain the SD difference between Table 2 (0.2421) and Tables 3/4 (0.1764) at 169 context frames?
  • Are there existing comparisons with alternative memory-retrieval strategies beyond noisy-state queries? If so, how were their inputs and training budgets matched?
1 comment

We sincerely appreciate your comments and valuable suggestions. We provide the following responses to address your concerns and answer your questions point by point.


W1. The code and supplementary files are not available, although the paper refers to supplementary videos.

The appendix documents memory size, training hyperparameters, inference defaults, and evaluation settings. We also provide a supplementary package with visual examples, and we will release the code upon acceptance.

W2 & Q1 & Q2. The object-grounding coverage issue and the SD discrepancy between Table 2 and Tables 3/4 undermine confidence in the comparisons. What checkpoints or settings explain the SD difference between Table 2 (0.2421) and Tables 3/4 (0.1764) at 169 context frames? Are all methods evaluated on the same samples with the same denominator?

Both issues have different causes: the evaluation set and denominator are shared across methods, while the SD difference comes from the registration anchor. The remaining OG coverage issue is metric coverage: strict IF combines target presence with a centering requirement, so we report presence separately.

For coverage, every method uses the same 128 task instances from 20 unseen ScanNet++ scenes, the same seeds, evaluator, and failure-handling rules. There is no method-specific exclusion, so every rate uses denominator 128; all current runs produce 128 valid outputs, and any invalid evaluator output is counted as a failure. Following Video4Spatial, IF requires the target to be present and its predicted box center to be within a relative tolerance γ=0.6\gamma=0.6 of the box width/height; IF$_{0.2}$ additionally requires SD$<0.2$. Many strict-IF failures are cases where the object is present but slightly off-center:

Configuration (128 tasks, 20 unseen scenes)Target presentPresence rate
Noisy-state query9271.88%
Dynamic state + task + position8364.84%
MemLearner-style query8566.41%
SceneQuery10682.81%

For SceneQuery, the target is present in 106/128 cases (82.81%); the lower strict-IF rate mainly reflects the centering requirement rather than missing targets.

For SD, we apologize for putting a local diagnostic value into Table 2. During development, we used context-anchored SD to monitor whether generated scenes stayed consistent with the observed context. The benchmark evaluation follows the Video4Spatial protocol, which measures alignment to 40 fixed target frames. Thus, at 169 context frames, 0.2421 and 0.1764 differ because of the registration anchor, not the checkpoint. The context-length results, previously misplaced in the main text, are now:

Context framesContext-anchored SD ↓GT-target-anchored SD ↓
450.26600.2326
850.24810.2107
1690.24210.1764
3370.25550.2195
6730.29570.2802

W3 & Q3. The paper does not compare against MemLearner, Context-as-Memory, or VMem. Are there existing comparisons with alternative memory-retrieval strategies beyond noisy-state queries?

The comparison must keep the RGB-only input assumption fixed. Context-as-Memory and VMem require explicit camera poses or 3D indexing, so they are not directly comparable here. MemLearner is the closest compatible strategy, so we added a matched comparison using the same backbone, memory budget, samples, and evaluator:

MethodOG SD ↓OG presence ↑SN LPIPS ↓SN Rotation error ↓SN translation error ↓
MemLearner-style query0.268885/128 (66.41%)0.655625.79400.1105
SceneQuery0.1764106/128 (82.81%)0.52138.98660.0833

Under the same RGB-only setting, SceneQuery has higher target presence, better reconstruction, and lower rotation and translation errors than the MemLearner-style query.


Thank you again for your comments. We revised the paper and supplementary visualizations accordingly. We look forward to further comments!

Reviewer_T6PGOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes scenequery, a scene memory framework for visuospatial video generation under the video4spatial setting, where the model is given sparse context frames and needs to either approach a text-specified object or follow a target camera trajectory. The context is written into a global memory and a pooled spatial memory, together with an initial-view reference. Instead of querying the memory with noisy target features, each target token uses a fixed task query from text and position or plucker ray, and the retrieved evidence is injected into every block by a gated residual and a FiLM adapter. Experiments on scannet++ report improvements over video4spatial on both tasks.

Critical strengths
  • Using task-derived query instead of noisy-state query is intuitive, and the comparison in Tab. 3 is controlled.

  • The paper is honest about its limitations.

Critical weaknesses
  • My main concern is the results do not really support the method. For OG, the primary metric IF0.2 is exactly the same as video4spatial, and IF even drops, only SD gets better. The paper itself says the joint success rate remains unchanged. For SN, PSNR is improved but the absolute value is only around 12 for all methods, and there is no camera accuracy metric, so it is hard to tell whether the model really follows the trajectory. The evaluation also only has 20 scenes with a single run.

  • Method feels sensitive to the scales. The full model uses different (r, f) for OG and SN, and in Tab. 4 only the selected config reaches the baseline IF0.2 on OG, all other settings are below it. Also the memory-only variant has much higher IF than the full model, so the task adapter is more like a trade-off between IF and SD than a clear improvement.

  • Novelty is limited. learnable memory slots, cross-attention readout and FiLM injection are all standard components, and the model is built directly on video4spatial. Query-based memory readout is also close to memlearner. The main new point is replacing the noisy-state query with a fixed task query, which feels a bit thin for a full paper.

  • The query design is not very convincing. For OG, the query is just a shared text feature plus grid coordinates, which has nothing to do with the scene geometry, so its unclear how it retrieves target-specific evidence. For SN, the memory has no context poses or 3D information, so its also unclear how a ray query can find the corresponding region. There is no attention visualization or analysis on what is actually retrieved.

  • Some numbers in the tables seem inconsistent. For example, w/o memory in Tab. 3 should be video4spatial, but its OG IF is 41.73 while Tab. 1 reports 48.62. Some rows in Tab. 3 also look copied from other rows. A clarification would be helpful.

  • Many details are missing and there is no appendix, e.g., memory size, training steps and inference cost. The writing is also heavily hedged, which makes it hard to understand what the actual contribution is

Questions for authors

See weakness

4 comments

We sincerely appreciate your comments and valuable suggestions. We provide the following responses to address your concerns and answer your questions point by point.


W1. The overall gains are limited: OG IF$_{0.2}$ is unchanged, IF decreases, navigation lacks a trajectory metric, and the evaluation uses only 20 scenes with a single run.

The gains are metric-dependent. On OG, SceneQuery reduces SD from 0.22 to 0.1764 and matches Video4Spatial on IF$_{0.2}$ (35.94%), while raw IF decreases from 48.44% to 42.97%; the decrease is consistent with a centering limitation, although target presence is also slightly lower than the baseline (presence 82.81%, 106/128). On SN, we added a direct trajectory metric and improved both target-view reconstruction and pose accuracy:

MetricVideo4SpatialSceneQuery
OG SD ↓0.220.1764
OG IF ↑48.44%42.97%
OG IF$_{0.2}$ ↑35.94%35.94%
SN LPIPS ↓0.61490.5213
SN Rotation error ↓17.11398.9866
SN translation error ↓0.09920.0833

All comparisons use the same 128 task instances from 20 unseen ScanNet++ scenes, the same seeds and evaluator. We therefore claim improved geometry, reconstruction, and measured trajectory consistency, not uniform improvement on every metric.

W2. The full model is sensitive to the integration scales. The task adapter improves geometry but reduces IF, and only the selected configuration reaches the baseline IF$_{0.2}$.

We treat the scales as task-specific operating points, selected to balance geometry and instruction following rather than maximize any single metric:

TaskSelected scales (r,f)(r,f)Selected result
OG(1,0.25)(1,0.25)SD 0.1764, IF 42.97%, IF$_{0.2}$ 35.94%
SN(0.25,1)(0.25,1)PSNR 12.6297, SSIM 0.4420, LPIPS 0.5213

For OG, reducing adapter scale favors geometry while keeping IF$_{0.2}$ at baseline; for SN, reducing memory-injection scale improves reconstruction. The tasks stress different failure modes, so the scales are task-specific trade-offs; the full sweep is in the appendix.

W3. Learnable slots, cross-attention, and FiLM are standard components. The main new point seems to be replacing the noisy-state query with a fixed task query.

We agree that the individual modules are standard and do not claim them as novel. Our contribution is a unified task-conditioned retrieval interface: output tokens address RGB memory using task conditions and position, independently of denoising state. It serves OG and SN from sparse RGB context without poses, depth, or explicit 3D reconstruction. The query ablation isolates this decoupling of memory addressing from the evolving latent.

W4. The OG query has no explicit scene geometry, and the SN ray query has no context poses or 3D information. There is no analysis of what is actually retrieved.

The query is a learned address into RGB feature memory, not an analytic 3D intersection, so we do not claim explicit geometry. With the adapter disabled, fixed 128 samples, and no retraining, we added three interventions: query shuffling tests output alignment, donor-memory swapping tests scene specificity, and zeroing the memory readout tests retrieval necessity:

Task / metricCorrectQuery shuffleMemory swapZero memory
OG SD ↓0.18890.23230.22950.3172
OG IF ↑58.59%54.69%57.81%48.44%
OG IF$_{0.2}$ ↑35.94%32.81%35.16%32.03%
SN LPIPS ↓0.52780.51610.51480.5173
SN PSNR ↑12.338212.297212.334912.2195

Zeroing memory degrades OG geometry/IF and lowers SN PSNR relative to the correct pairing (12.3382→12.2195), while SN LPIPS changes from 0.5278 to 0.5173; the effects are metric-dependent. Shuffling hurts alignment and SN PSNR, while donor-memory replacement changes scene attention (cosine 0.480 SN, 0.582 OG). Attention is stable across denoising (JS divergence 0), and shuffling alters SN attention more than OG (JS/cosine 0.0282/0.943 vs. 0.00043/0.997). These support scene-dependent, output-aligned retrieval without proving exact token-to-region geometry.

W5. Some numbers in Tab. 3 appear inconsistent, and important implementation details (memory size, training steps, inference cost) and an appendix are missing.

Two issues, different in kind: one is a comparison that should not be made, and the other was a mislabeled row. (1)Video4Spatial removes both scene memory and the task adapter, whereas Table 3's w/o memory removes only scene memory and retains task adaptation. It does not represent Video4Spatial, so its 41.41% should not be compared with the 48.44% of the Video4Spatial baseline we ran under the same evaluator in Table 1. We clarify in the text that w/o memory retains task adaptation and is not the matched Video4Spatial baseline. (2)The row formerly labelled w/o task adaptation contains the values for memory only (task query plus scene memory, adapter disabled), the same forward configuration as the Task query row, hence the identical values. It is now named memory only, with an explicit Task query/memory only correspondence.

The appendix documents memory size, training hyperparameters, inference defaults, and evaluation settings.


Thank you again for your comments. We revised the paper and supplementary visualizations accordingly. We look forward to further comments!

Thank you for the response with additional analysis. The new pose metric for SN and the clarification of tab. 3 address part of my concerns and I will raise my score to weak reject. I still lean towards rejection due to 1) on OG the primary metric IF0.2 is unchanged and IF drops 2) in the new intervention study, swapping or shuffling the memory barely changes SN results, which does not show that the query retrieves scene-specific evidence. 3) the novelty is limited, as the authors also agree the components are standard. I am happy to hear if the authors have further clarifications.

Thank you for your thoughtful follow-up and for raising your score. We appreciate the opportunity to address your three remaining concerns below.

1. OG metrics (“IF0.2 is unchanged and IF drops”). This trade-off remains, but SceneQuery reduces geometric error by about 20% (SD: 0.22 → 0.1764; Tables 1(a) and 3(d)). In the memory-only intervention, query shuffling and memory swapping raise SD from 0.1889 to 0.2323 and 0.2295, respectively (Appendix Table 8). The correct scene memory and output-aligned queries therefore have a measurable effect on OG, even though joint success is unchanged.

2. SN interventions (“barely changes SN results”). Memory swapping and query shuffling change the final SN metrics only slightly, but a donor-memory swap substantially changes scene-bank attention (cosine similarity 0.480; Appendix retrieval analysis). This shows that the readout responds to the memory bank, while the small output changes limit what this intervention can establish about scene-specific evidence. The task results provide stronger support for the query design: with task and position inputs held constant, fixed queries lower rotation error from 12.08 to 9.75 and translation error from 0.0865 to 0.0834 relative to dynamic queries (Appendix Table 9). The full model also lowers both pose errors relative to Video4Spatial (17.11 to 8.99; 0.0992 to 0.0833; Table 1(b)).

3. Novelty (“the components are standard”). Our contribution is a task-conditioned retrieval interface: the task condition determines both the generation target and what scene evidence each output token retrieves. OG queries combine target semantics with token position; SN queries encode target-camera rays. These queries address RGB scene memory and remain fixed throughout denoising, without requiring context-camera poses or explicit 3D reconstruction. Appendix Table 9 isolates the benefit of fixed addressing. Under a matched RGB-only setting, a MemLearner-style query also performs worse on OG SD (0.2688 vs. 0.1764) and SN rotation error (25.79 vs. 8.99; Appendix Table 10).

Thank you again for your comments. We hope these could address your concerns. We look forward to further comments!

Reviewer_FzQyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SceneQuery introduces a task-conditioned scene retrieval framework for visuospatial video generation, designed to handle object grounding (OG) and scene navigation (SN) without explicit 3D reconstruction. Its core innovation lies in constructing fixed, task-derived queries for each output token—using target semantics and spatial coordinates for grounding, or target camera rays for navigation. Unlike queries derived from evolving target features, these remain stable throughout denoising to retrieve output-aligned evidence from global and spatial scene memories, as well as an initial-view reference. The entire memory and retrieval pipeline is jointly optimized via the generation loss. Evaluated on 20 unseen ScanNet++ scenes (128 samples per task configuration), SceneQuery demonstrates clear improvements over the matched Video4Spatial baseline: navigation PSNR increases from 11.08 to 12.63 and LPIPS drops from 0.6149 to 0.5213, while object grounding reduces geometric error by ~21.2%. However, the primary joint success rate for grounding remains unchanged, and instruction-following performance actually declines. While controlled ablations robustly validate task-derived queries over noisy-state alternatives, the overall gains remain heavily task- and configuration-dependent. Furthermore, the limited evaluation scope and unresolved coverage issues in object grounding leave several results open to interpretation.

Critical strengths
  1. The task-driven query design is clear and well-backed. By building queries from task conditions and output positions that stay fixed during denoising, and testing them with fixed memory and a disabled task adapter, the authors show a clear win over noisy-state queries on both tasks. This direct empirical support is a solid, substantive strength of the paper.

  2. On scene navigation, SceneQuery delivers consistent, across-the-board gains. Against the matched Video4Spatial baseline, PSNR jumps from 11.08 to 12.63, SSIM goes from 0.4049 to 0.4420, and LPIPS drops from 0.6149 to 0.5213. While these numbers validate the practical utility of the memory-retrieval framework, they don't automatically mean the model has achieved deeper 3D understanding or tighter trajectory adherence.

Critical weaknesses

The overall task gains are limited and are not yet sufficient to demonstrate that the method has a stable advantage. The main joint success rate for object localization did not improve, and instruction following dropped from 48.62 to 43.12; the navigation gains are mainly reflected in image reconstruction metrics, lacking independent validation of trajectory accuracy. In addition, the experiments cover only 20 scenes from a single dataset, and the current evidence is insufficient to support the conclusion that the method can stably improve the two spatial tasks.

Questions for authors

What exactly is the coverage issue in the object-grounding evaluation? Are the numbers of valid samples, exclusion criteria, and evaluated sample sets consistent across methods?

1 comment

We sincerely appreciate your comments and valuable suggestions. We provide the following responses to address your concerns and answer your questions point by point.


W1. The main joint success rate for object localization did not improve, and instruction following dropped from 48.44 to 42.97.

A high IF does not ensure spatial consistency: Veo 3 reaches IF 79.69% but SD 0.41, so we aim to improve grounding on top of scene geometry. Following Video4Spatial, IF first checks whether the target appears in the last frame, then requires the predicted box center to satisfy a relative tolerance γ=0.6\gamma=0.6 on box width/height; IF$_{0.2}$ additionally requires SD$<0.2$. Under this definition, many "IF failures" are cases where the object is present but slightly off-center.

Method (128 tasks, 20 unseen scenes)Presence ↑IF ↑IF$_{0.2}$ ↑
Wan2.2-5B95.31%75.78%10.94%
Veo 399.22%79.69%28.91%
FramePack64.84%16.41%3.13%
Video4Spatial85.16%48.44%35.94%
SceneQuery (ours)82.81%42.97%35.94%

SceneQuery finds the target in 106/128 cases (82.81%), and its IF$_{0.2}$ matches Video4Spatial (35.94%) with lower SD (0.1764 vs. 0.22), so grounding is not degraded. Its lower raw IF (42.97% vs. 48.44%) reflects weaker centering, which we acknowledge. We therefore claim improved scene geometry and grounding consistency, while centering remains a limitation.

W2. The navigation gains are mainly reflected in image reconstruction metrics, lacking independent validation of trajectory accuracy.

PSNR/SSIM/LPIPS alone are reconstruction metrics. We added Relative Pose Error (RPE) from a fixed pose evaluator to measure camera motion directly:

MethodRPE-R ↓RPE-T ↓Valid rate
VolSplat16.06330.0915128/128
AnySplat11.46450.0713128/128
TrajectoryCrafter12.16810.0765128/128
Gen3C17.11380.0967128/128
Video4Spatial17.11390.0992128/128
SceneQuery (ours)8.98660.0833128/128

All methods are evaluated on the same 128 task instances with the same pose evaluator. VolSplat, AnySplat, TrajectoryCrafter, and Gen3C use their documented extra pose/depth inputs, whereas Video4Spatial and SceneQuery use RGB context. SceneQuery has the lowest rotation error, while AnySplat has the lowest translation error, so the navigation gain is not only an image-reconstruction gain.

W3. The experiments cover only 20 scenes from a single dataset.

Although the evaluation uses 20 unseen scenes, it contains 128 task instances.

W4. The overall task gains are limited and are not sufficient to demonstrate a stable advantage across the two spatial tasks.

The gains are not uniform on every metric, but they are measurable on both tasks. For object grounding, SceneQuery finds the target in 106/128 cases (presence 82.81%), and its IF$_{0.2}$ matches Video4Spatial (35.94%) with lower SD (0.1764 vs. 0.22); the lower raw IF is consistent with a centering limitation, although target presence is also slightly lower than the baseline. For navigation, SceneQuery has the lowest rotation error (RPE-R 8.9866 vs. 17.1139 for Video4Spatial) on the same 128 task instances. We therefore claim improved scene geometry and trajectory consistency, with centering as an acknowledged limitation, on 128 task instances in 20 unseen scenes of one dataset.

Q1. What exactly is the coverage issue in the object-grounding evaluation? Are the numbers of valid samples, exclusion criteria, and evaluated sample sets consistent across methods?

Yes. All methods use the same 128 task instances (20 unseen scenes), the same seeds, and the same evaluator. Every method produces 128 valid outputs, with no method-specific exclusion, so the denominator is 128 for all methods. The remaining coverage issue is metric coverage: strict IF combines target presence with centering, so it is not a pure measure of whether the target appears.

Following Video4Spatial, IF requires the target object to be present and its predicted box center to be within a relative tolerance γ=0.6\gamma=0.6 of the box width/height. Many "failures" are cases where the object is present but slightly off-center. Presence and IF are therefore different quantities:

Configuration (128 tasks, 20 unseen scenes)Target presentPresence rate
Noisy-state query9271.88%
Dynamic state + task + position8364.84%
MemLearner-style query8566.41%
SceneQuery10682.81%

For SceneQuery, the target is present in 106/128 cases (82.81%), while strict IF is lower because of centering rather than missing targets.


Thank you again for your comments. We revised the paper and supplementary visualizations accordingly. We look forward to further comments!

Reviewer_exT3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SceneQuery. While generating videos of a scene that either approach an object or follow a camera trajectory, it retrieves relevant context using task-specific queries. This retrieved information guides the video generation.

Critical strengths
  1. Retrieving the necessary context depending on the task is a principled and well-motivated way to improve consistency.

  2. The ablation studies with with different memory designs, query types, and conditioning mechanisms are very useful.

Critical weaknesses
  1. Table 1 reports an SD of 0.18 for SceneQuery, consistent with the full model result of 0.1764 in Table 3(d). But Table 2 reports an SD of 0.2421 at 169 context frames, which is the training context length. It is unclear why the same configuration has different performances. The authors should clarify these differences. If this is due to variance across independent runs, a confidence interval should be specified.

  2. Table 3(d) shows that removing the task adapter improves object-grounding instruction following from 43.12% to 58.71%. Also, for object grounding, IF_0.2 remains unchanged at 36%, while instruction following actually decreases from 48.62% to 43.12%. This suggests that the full architecture introduces significant trade-offs rather than consistently improving performance. Although the authors acknowledge this, a more detailed analysis of this trade-off and the rationale for selecting the task adapter configuration would strengthen the evaluation.

  3. The comparison in Table 3(b) changes both the query and whether it is fixed simultaneously. An ablation comparing fixed and dynamically updated queries with the same task information would help isolate the source of the improvement.

Questions for authors
  1. The scene-navigation task aims to generate videos that follow specified camera trajectories, but the evaluation uses image reconstruction metrics (PSNR, SSIM, and LPIPS). Could the authors provide a more direct evaluation of camera-motion accuracy, such as estimated camera-pose errors or multi-view geometric consistency? Or, justify why the current metrics are sufficient? I imagine an image could be visually similar while not following the specified trajectory.

  2. The paper says 'Combining global and spatial memory gives the strongest OG geometric consistency', but in Table 3a the Global + Spatial configuration does not win on any of the object grounding metrics. What was the rationale for picking the best configuration? Or was this a typo?

3 comments

We sincerely appreciate your comments and valuable suggestions. We provide the following responses to address your concerns and answer your questions point by point.


W1. Table 2 reports an OG SD of 0.2421 at 169 context frames, whereas Tables 1, 3, and 4 report approximately 0.1764 for the selected model. It is unclear why the same configuration gives different results.

We apologize for putting a local diagnostic value into Table 2. During development, we used context-anchored SD to monitor whether generated scenes stayed consistent with the observed context. The benchmark evaluation follows the Video4Spatial protocol, which measures alignment to 40 fixed target frames. The GT-target-anchored column follows the Video4Spatial protocol (40 fixed target frames). The context-length results, previously misplaced in the main text, are now:

Context framesContext-anchored SD ↓GT-target-anchored SD ↓
450.26600.2326
850.24810.2107
1690.24210.1764
3370.25550.2195
6730.29570.2802

The two values at 169 context frames differ because they use different registration anchors, not because the checkpoint changed. The GT-target-anchored value 0.1764 matches Table 1, confirming the selected checkpoint is consistent across evaluations. We provide both protocol sweeps in Table 7 of the appendix.

W2. Removing the task adapter improves OG instruction following from 42.97% to 58.59%, while the full model improves geometry. This suggests a trade-off rather than a uniform improvement.

The adapter trades raw IF for geometry. Comparing the same memory-only configuration with the full model:

ConfigurationOG SD ↓OG IF ↑OG IF$_{0.2}$ ↑
Memory only (adapter disabled)0.188958.59%35.94%
Full model0.176442.97%35.94%

The adapter lowers SD from 0.1889 to 0.1764 but drops raw IF from 58.59% to 42.97%; the joint success rate IF$_{0.2}$ stays at 35.94%. We keep it for geometric consistency and target-view reconstruction, instead of claiming the full model is uniformly better.

W3. The query comparison changes both the query and whether it is fixed. An ablation with the same task information but dynamically updated queries would isolate the source of improvement.

To isolate the effect of keeping the retrieval address fixed, we added the following hybrid control:

qihyb(s)=LN(Ph(hi(s))+Ptask(cτ)+Ppos(ui)).q_i^{\mathrm{hyb}}(s)=\mathrm{LN}\left(P_h(h_i(s))+P_{\mathrm{task}}(c_\tau)+P_{\mathrm{pos}}(u_i)\right).

QueryOG SD ↓SN LPIPS ↓SN Rotation error ↓SN translation error ↓
Noisy-state query0.20240.571711.40170.0893
Dynamic state + task + position0.19220.525812.08030.0865
Fixed task + position (ablation)0.18890.52789.75120.0834

The dynamic query slightly improves LPIPS (0.5258 vs. 0.5278), but the fixed query improves OG SD and both pose errors. Since the two queries carry the same task and position information and differ only in whether the memory address is updated from the evolving latent, this isolates the benefit of keeping the retrieval address independent of the denoising state: our fixed task query gives more stable spatial control, with only a marginal LPIPS drop.

Q1. The navigation evaluation uses PSNR, SSIM, and LPIPS, but does not directly measure camera-motion accuracy.

We added Relative Pose Error from a fixed pose evaluator to measure camera motion directly:

MethodRotation error ↓Translation error ↓
VolSplat16.06330.0915
AnySplat11.46450.0713
TrajectoryCrafter12.16810.0765
Gen3C17.11380.0967
Video4Spatial17.11390.0992
SceneQuery8.98660.0833

All methods are evaluated on the same 128 task instances with the same pose evaluator. VolSplat, AnySplat, TrajectoryCrafter, and Gen3C use their documented extra pose/depth inputs, whereas Video4Spatial and SceneQuery use RGB context. SceneQuery has the lowest rotation error, while AnySplat has the lowest translation error. The results therefore provide direct camera-motion evidence without claiming SceneQuery is best on every pose metric.

Q2. The paper claims that global+spatial memory gives the strongest OG geometric consistency, but Table 3(a) reports a lower SD for spatial-only memory.

We corrected the text on memory organization: spatial-only actually has SD 0.1855, while global+spatial has SD 0.1888. We still use global+spatial because global memory preserves overall layout while spatial memory helps center the target object; it is a balanced, task-dependent design choice, not uniformly best for SD.


Thank you again for your comments. We revised the paper and supplementary visualizations accordingly. We look forward to further comments!

Thank you for the detailed response and the additional experiments.

My concerns about the discrepancy in the tables, evaluating the camera motion accuracy and selecting global+spatial memory have been addressed.

I am still not convinced about the task adapter trade-off. Part of the justification is SN reconstruction, but the paper trains separate OG and SN models, so this doesn’t justify using the adapter in the OG model. In fact for OG, memory-only gives the same IF0.2 and much higher IF, with only slightly worse SD (0.1889 vs. 0.1764). In the experiment for fixed vs. dynamic queries, the dynamic variant also adds noisy-state features, so it differs in more ways than just whether the query is updated. Also the results are mixed and without confidence intervals it is hard to tell if they are meaningful.

I will be maintaining my current score.

Thank you for your thoughtful follow-up. We address your remaining concerns below.


(1) Task adapter. Our scene retrieval improves OG geometry and instruction following even without the adapter. Compared with Video4Spatial, memory-only lowers SD from 0.22 to 0.1886 and raises IF from 48.44% to 58.59%, while matching IF₀.₂.

The adapter further lowers SD by 6.5%, to 0.1764, at the cost of lower IF. We retain it in the full OG model for this additional geometric improvement. Its reconstruction benefit in SN supports its use in SN separately.

(2) Query comparison. Our method queries scene memory directly with task and position information, without using evolving noisy-target features. We have added scene-level 95% confidence intervals to assess the differences between query designs.

The added Dynamic state + task + position row gives the noisy-state baseline the same task and position conditions as ours. Here, “dynamic state” refers to the same noisy-target features used by the original noisy-state query.

Comparing the two task-conditioned rows, ours lowers rotation error by 19.3% (paired difference: −2.3291°, 95% CI [−4.0505°, −0.9625°]). The hybrid’s 0.0020 LPIPS advantage is not statistically significant. Compared with the original noisy-state query, ours also lowers LPIPS by 0.0439 (95% CI [−0.0572, −0.0306]).

Thus, the benefit of our query design remains after the dynamic baseline receives the same task and position information: ours achieves better rotation accuracy without relying on noisy-target features.

QueryOG SD ↓SN LPIPS ↓Rotation error (°) ↓Translation error ↓
Noisy-state query0.2024 [0.1767, 0.2260]0.5717 [0.5476, 0.5924]11.4017 [9.7645, 13.0460]0.0893 [0.0754, 0.1037]
Dynamic state + task + position (noisy-state hybrid)0.1922 [0.1487, 0.2492]0.5258 [0.4944, 0.5534]12.0803 [9.7542, 14.9305]0.0865 [0.0725, 0.1009]
Fixed task + position (ours)0.1886 [0.1553, 0.2293]0.5278 [0.4991, 0.5545]9.7512 [8.1322, 11.7509]0.0834 [0.0697, 0.0965]
Δ: ours − hybrid−0.0025 [−0.0389, +0.0354]+0.0020 [−0.0067, +0.0114]−2.3291 [−4.0505, −0.9625]−0.0031 [−0.0096, +0.0038]
Δ: ours − noisy-state query−0.0140 [−0.0391, +0.0164]−0.0439 [−0.0572, −0.0306]−1.6504 [−3.1097, +0.1479]−0.0059 [−0.0121, +0.0001]

Values are means with 95% CIs from 100,000 scene-level bootstrap resamples, without multiple-comparison correction. Paired differences use videos shared by each pair, so they may differ from differences between the overall means. Negative differences favor ours.


Thank you again for considering these clarifications.

Reviewer_75QDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds a fixed retrieval query from task conditions instead of evolving generative features, and plugs it into a spatial video generation pipeline covering object localization and navigation reconstruction. The design is plausible, but its necessity relative to existing alternatives remains unargued, and the "fixed, task-driven query" is not isolated as the real source of the reported gains.

Critical strengths

The paper presents a clear and intuitive framework that uses task conditions to retrieve scene evidence for individual output tokens. Controlled query-source comparisons support the design, and the method improves navigation reconstruction and grounding geometry over the matched baseline.

Critical weaknesses
  • Underdeveloped motivation: The paper does not establish that evolving generation features undermine retrieval or explain why retrieval needs should remain fixed throughout denoising.
  • Confounded ablation: The query comparison changes task information, output position, and generation-state dependence simultaneously. A key missing control combines current features with task conditions and output positions.
  • Unverified retrieval mechanism: Retrieval stability and relevance are not directly evaluated. Intervention experiments are needed to distinguish task-relevant retrieval from an additional conditioning adapter.
  • Mixed empirical evidence: Grounding geometry improves, but instruction following declines.
Questions for authors

See weaknesses.

1 comment

We sincerely appreciate your comments and valuable suggestions. We provide the following responses to address your concerns and answer your questions point by point.


W1. The motivation for using fixed, task-driven queries is underdeveloped. The paper does not establish why evolving generation features undermine retrieval or why retrieval should remain fixed during denoising.

The required scene evidence (target object and camera ray) does not change during generation, while the noisy generation state does. Our query is not one global condition; it varies per token across time and image location, but stays fixed across denoising steps ss. This decouples memory retrieval from the initially unreliable generation state: a noisy-state query would change with the evolving latent and rely on an unstable early denoising state. Fixed queries thus stabilize memory access without freezing the generator.

W2. The query comparison changes task information, output position, and generation-state dependence simultaneously. A key missing control combines current features with task conditions and output positions.

We added a control that combines current generation features with the task condition and output position: qihyb(s)=LN(Ph(hi(s))+Ptask(cτ)+Ppos(ui)).q_i^{\mathrm{hyb}}(s)=\mathrm{LN}\left(P_h(h_i(s))+P_{\mathrm{task}}(c_\tau)+P_{\mathrm{pos}}(u_i)\right).

QueryOG SD ↓SN LPIPS ↓SN Rotation error ↓SN translation error ↓
Noisy-state query0.20240.571711.40170.0893
Dynamic state + task + position (new)0.19220.525812.08030.0865
Fixed task + position (ours)0.18890.52789.75120.0834

Our fixed task query improves OG SD and both pose errors over the dynamic control, whose only advantage is a marginal LPIPS gain—expected, since a dynamic query inherits the instability of the early denoising state (W1).

W3. Retrieval stability and relevance are not directly evaluated. Intervention experiments are needed to distinguish task-relevant retrieval from an additional conditioning adapter.

Retrieval intervention. We added inference-time interventions using the same memory_only checkpoint with the task adapter disabled, fixed 128 samples, and no retraining. Each intervention targets one concern: query shuffling tests output alignment, donor-memory swapping tests scene specificity, and zeroing the memory readout tests retrieval necessity.

Task / metricCorrectQuery shuffleMemory swapZero memory
OG SD ↓0.18890.23230.22950.3172
OG IF ↑58.59%54.69%57.81%48.44%
OG IF$_{0.2}$ ↑35.94%32.81%35.16%32.03%
SN LPIPS ↓0.52780.51610.51480.5173
SN PSNR ↑12.338212.297212.334912.2195

Zeroing memory degrades OG geometry and instruction following (IF), and lowers SN PSNR relative to the correct pairing (12.3382→12.2195), while its SN LPIPS changes from 0.5278 to 0.5173. Query shuffling and donor-memory swapping similarly lower LPIPS but worsen PSNR relative to the correct pairing, so the effects are metric-dependent; donor-memory swapping shifts scene-bank attention (cosine 0.480 SN, 0.582 OG). Combining both gives OG SD 0.2813 and SN LPIPS 0.5182.

Attention analysis further shows retrieval is perfectly stable across denoising steps (JS divergence 0), and query shuffling alters SN attention more than OG (JS/cosine 0.0282/0.943 vs. 0.00043/0.997). Together these interventions support functional, scene-dependent, output-aligned retrieval, without claiming exact token-to-region correspondence.

W4. Grounding geometry improves, but instruction following declines.

A high IF does not ensure spatial consistency: Veo 3 reaches IF 79.69% but SD 0.41, so we aim to improve grounding on top of scene geometry. Following Video4Spatial, IF first checks whether the target appears in the last frame, then requires the predicted box center to satisfy a relative tolerance γ=0.6\gamma=0.6 on box width/height; IF$_{0.2}$ additionally requires SD$<0.2$.

Method (128 instances, 20 unseen scenes)Presence ↑IF ↑IF$_{0.2}$ ↑
Wan2.2-5B95.31%75.78%10.94%
Veo 399.22%79.69%28.91%
FramePack64.84%16.41%3.13%
Video4Spatial85.16%48.44%35.94%
SceneQuery (ours)82.81%42.97%35.94%

SceneQuery finds the target in 106/128 cases (82.81%), and its IF$_{0.2}$ matches Video4Spatial (35.94%) with lower SD (0.1764 vs. 0.22). Its lower raw IF (42.97% vs. 48.44%) reflects weaker centering, which we acknowledge; higher presence alone (e.g., Veo 3) does not imply better spatial grounding. We therefore claim improved scene geometry and grounding consistency, while centering remains a limitation.


Thank you again for your comments. We revised the paper and supplementary visualizations accordingly. We look forward to further comments!

Comments on the paper

2 comments

Dear reviewers,

As the rebuttal period is coming to an end, I kindly remind you to check the authors' responses. If you have not yet responded, please indicate whether your concerns have been adequately addressed. Many thanks!

Best, AC

We sincerely appreciate the time and care you have devoted to reviewing our work. Your constructive feedback and suggestions have been valuable in helping us refine the manuscript and communicate our findings more clearly.

With the discussion period ending soon, we wanted to briefly follow up on our responses and ask whether there are any remaining concerns we could address. If any points would benefit from further explanation, we would be glad to clarify them before the deadline.

Thank you once again for your careful consideration and engagement in the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.