acceptodds
Under review as a conference paper at ICLR 2027

Honeycomb: Constant-Size Scene Memory Representation for Video World Models

27 views

Abstract

Video world models require persistent scene memory to maintain consistency during long-horizon video generation. Existing spatial memories accumulate RGB observations or latent features, increasing storage requirements as generation proceeds. We introduce Honeycomb, a video world model built on HexMemory, our proposed low-rank representation for storing scene features in a fixed-size memory with a total of six spatial and spatiotemporal planes. A feed-forward writer maps each generated chunk into new plane features. As the spatial coverage or temporal range expands, we warp the previous planes while preserving their dimensions, then fuse them with the new features through confidence-weighted pooling and a learned residual correction. A reader retrieves latents from HexMemory to condition subsequent video generation. The writer processes only observations from the new chunk, avoiding per-scene optimization and repeated processing of the full history. Experiments on WorldScore and RealEstate10K demonstrate strong video generation quality and robust revisit consistency while keeping HexMemory feature storage constant throughout generation. Code and additional visualizations are available on our project page: https://anonymous-submission-20.github.io/Honeycomb.github.io/.

Primary area:
generative models
Keywords:
World Models, Memory, Video Generative Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hKpKOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a video world model (Honeycomb) that stores scene information in six fixed-size feature planes called HexMemory. The model updates the memory with each new video chunk and reads it to guide the next chunk. The model reduces feature storage and memory update costs while achieving good results in novel-view synthesis and revisit consistency.

Critical strengths
  1. HexMemory requires limited feature storage, which remains constant as the video length increases. The paper also shows that further reducing the memory size leads to only limited performance degradation. And the model achieves strong results in novel-view synthesis and revisit consistency, indicating that this compressed memory can retain useful scene information.

  2. Memory updates are efficient. Each update processes only the newly generated chunk, without reprocessing the entire history or requiring per-scene optimization.

Critical weaknesses
  1. The effective temporal resolution of HexMemory decreases as the temporal range expands, which may affect long memory accuracy.

  2. HexMemory appears to provide only static scene memory and cannot handle dynamic object state updates. In this case, is it necessary to retain temporal information? For example, comparing the three spatial planes (XY, XZ, and YZ) with the proposed six-plane representation would help assess the improvements brought by the spatiotemporal planes.

Questions for authors

How does memory accuracy change as the spatial and temporal ranges expand? Could the authors evaluate longer sequences and larger scenes to assess the effect of decreasing effective resolution?

3 comments
AuthorsOpenReview ↗

Author's Response - Thank you for your comments!

We greatly appreciate the reviewer's valuable comments, which help to improve the rigor of the paper. Following your feedback, we have added new experiments and results. We present our responses and findings below.

W1: The effective temporal resolution of HexMemory decreases as the temporal range expands, which may affect long memory accuracy.

We thank the reviewer for the comment. We agree that expanding the temporal range under fixed plane dimensions reduces effective temporal resolution and may affect memory accuracy. We have conducted memory-retention and longer-horizon generation experiments to characterize this trade-off. Please refer to our response to Q1 below for the results and discussion.

W2: HexMemory appears to provide only static scene memory and cannot handle dynamic object state updates. In this case, is it necessary to retain temporal information? For example, comparing the three spatial planes (XY, XZ, and YZ) with the proposed six-plane representation would help assess the improvements brought by the spatiotemporal planes.

We thank the reviewer for this valuable suggestion. The spatiotemporal planes form part of our low-rank feature factorization and can indeed contribute spatial detail even in static scenes, which makes retaining the temporal information important too. We have conducted an ablation comparing the three spatial planes with the current six-plane representation on the full RE10K NVS and WorldScore closed-loop evaluations.

On full RE10K NVS, the six-plane representation achieves better mean performance, as follows:

RepresentationPSNR ↑SSIM ↑LPIPS ↓
Tri-plane (XY, XZ, YZ)18.01930.67130.2747
Six-plane (ours)18.45100.67410.2740

On full WorldScore closed-loop evaluation, the advantage is more pronounced, as follows:

RepresentationPSNR ↑SSIM ↑LPIPS ↓
Tri-plane (XY, XZ, YZ)16.02770.48270.3352
Six-plane (ours)17.22030.50390.3106

We also evaluate memory retention on 100 real RE10K clips. After writing multiple autoregressive chunks, we retrieve the same first-chunk features and measure their reconstruction MSE. The six-plane representation achieves lower reconstruction error at every tested history length, indicating more accurate recall of earlier content as observations accumulate. After three chunks, its error is approximately 6.2× lower than tri-plane, as follows:

Representation1 chunk: 33 frames ↓2 chunks: 65 frames ↓3 chunks: 97 frames ↓
Tri-plane (XY, XZ, YZ)0.96340.79550.7848
Six-plane (ours)0.05980.09100.1275

These results show that the additional spatiotemporal planes contribute to static-scene memory accuracy and generation consistency in our trained model.

Regarding dynamic object state updates, we have now also included a new figure (anonymous link here: https://github.com/Anonymous-Submission-20/Honeycomb/blob/main/rebuttal_figures/rebuttal_figureB.pdf), which provides a qualitative example of ducks moving coherently across successive generated chunks as the camera pans back. Compared to Spatia, the ducks incoherently flew backwards as the camera pans back. Nonetheless, dynamic fidelity depends on the quality of underlying video foundation model. While this example illustrates cross-chunk dynamic motion coherence, rigorous evaluation of dynamic-state tracking and the memory’s contribution remains as potential future work.

Q1: How does memory accuracy change as the spatial and temporal ranges expand? Could the authors evaluate longer sequences and larger scenes to assess the effect of decreasing effective resolution?

We thank the reviewer for the insightful feedback. We evaluate memory retention on 100 RE10K clips by sequentially writing video observations while keeping the plane dimensions fixed and allowing spatial and temporal bounds to expand. After each update, we retrieve the same first-chunk features and measure reconstruction MSE. Error increases from 0.0598 at 33 frames to 0.2006 at 129 frames, indicating reduced recall accuracy as history accumulates. These measurements reflect the combined effects of range expansion and repeated memory updates.

For longer sequences and larger scenes, we additionally evaluate closed-loop generation on WorldScore scenes using 2, 4, and 6 chunks, following the same protocol as Spatia. Each trajectory moves outward for half the chunks before returning. Keeping the per-frame camera motion and memory-plane dimensions fixed, longer trajectories explore farther and revisit earlier content after longer histories (i.e., both spatial and temporal ranges expand). Revisit quality decreases as the trajectory length increases, demonstrating the practical coverage–accuracy trade-off under a fixed feature-memory budget. Note that these generation results also include accumulated autoregressive errors (which may be improved by self-forcing approaches in future work), complementing the direct memory-retention measurements above.

Each cell reports PSNR ↑ / SSIM ↑ / LPIPS ↓, as follows:

Plane resolutionHexMemory (MB)2 chunks4 chunks6 chunks
PSNR ↑ / SSIM ↑ / LPIPS ↓PSNR ↑ / SSIM ↑ / LPIPS ↓PSNR ↑ / SSIM ↑ / LPIPS ↓
51273.917.22 / 0.504 / 0.31115.61 / 0.453 / 0.47113.68 / 0.400 / 0.590
38442.617.14 / 0.503 / 0.31315.53 / 0.449 / 0.47713.44 / 0.393 / 0.602
25619.817.10 / 0.500 / 0.31915.39 / 0.444 / 0.49013.53 / 0.393 / 0.616
1285.816.77 / 0.490 / 0.33815.09 / 0.433 / 0.51913.28 / 0.384 / 0.643

We will now explicitly discuss this in the updated manuscript.

We sincerely thank the reviewers for the helpful feedback, and hope these address your concerns.

AuthorsOpenReview ↗

Summary of Response - Sincerely appreciate the feedback and looking forward to further engagement!

We thank the reviewer again for the constructive feedback. As the discussion period is coming to an end, we would like to follow up on our rebuttal and highlight the additional experiments addressing the effects of expanding coverage and the role of spatiotemporal planes. We summarize the key findings below and indicate where they are discussed in the revised manuscript.

Feedback raisedOur response to feedbackLocation
W1 & Q1. How does HexMemory accuracy change as spatial and temporal coverage expands under a fixed memory budget?We evaluated 2-, 4-, and 6-chunk WorldScore closed-loop trajectories at plane resolutions of 128–512, keeping feature storage fixed at each resolution. Reducing the resolution from 512 to 256 saves approximately 73% of feature storage (73.9 MB to 19.8 MB), with only a 0.12–0.22 dB PSNR decrease across trajectory lengths. Revisit quality declines over longer trajectories, reflecting both expanding coverage and accumulated autoregressive errors.Section 4.5, Table 4
W2. Are the spatiotemporal planes useful, and can the model preserve dynamic motion?We compared spatial-only tri-plane memory against the six-plane representation. The six-plane model improves RE10K NVS PSNR from 18.02 to 18.45 dB and WorldScore closed-loop PSNR from 16.03 to 17.22 dB, with improved SSIM and LPIPS. Our additional retention experiment also shows approximately 6.2× lower first-chunk feature MSE after 97 frames (0.1275 vs. 0.7848). We further added a qualitative example of coherent dynamic subject across generated chunks, while acknowledging that rigorous dynamic-state tracking remains future work.Section 4.5, Table 5; Section 4.4, Figure 7

We hope these additional experiments and clarifications address your concerns. We would greatly appreciate any further feedback. Thank you again for your time and consideration!

Reviewer_mQZBOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Honeycomb is a camera-controlled video world model with HexMemory, a fixed-size representation of scene features using three spatial and three spatiotemporal planes. A shared feed-forward writer maps newly generated observations into plane features. When spatial or temporal bounds expand, existing planes are warped into the updated coordinate range and fused with new features through weighted pooling and a learned residual correction. A reader reconstructs latent conditioning by querying the memory at projected historical points and their source write times. The model generates successive video chunks and writes new observations back into memory. Experiments on WorldScore and RealEstate10K evaluate generation quality, revisit consistency, and the storage and write-time trade-offs of HexMemory.

Critical strengths

1. HexMemory offers a promising way to maintain high-dimensional scene features with fixed-size feature storage. It compresses observations into shared spatial and spatiotemporal planes while retaining the ability to query features at world-space locations. The warping mechanism accommodates expanding spatial bounds and write-time ranges while preserving the plane dimensions. This provides a concrete approach to incorporating new observations without continually enlarging the feature memory.

2. The experiments support the method’s effectiveness and efficient writing. Table 2 shows improved novel-view and revisit quality. In Table 4, recurrent writing takes approximately 13 ms across the tested stages, with quality comparable to rebuilding memory from the full history.

Critical weaknesses

1. The total storage and readout cost over long sequences are not reported. Although the feature planes have fixed dimensions, the system still retains historical point clouds and projects them into each target view before querying HexMemory. The paper does not explain how this accumulated geometry or its projection cost is bounded. Tables 3 and 4 report feature-plane storage and writing time, but leave out the total storage and readout cost as the sequence grows. These costs are essential to assessing the scalability of the complete system.

2. It is unclear to me what HexMemory adds over direct point-cloud conditioning. Since the historical point clouds are already retained and projected, why not remove HexMemory and directly use colored point-cloud renderings to condition generation? I would like to see an ablation comparing RGB point-cloud projection, direct latent-feature projection, and HexMemory with the same backbone and training setup. This would show whether storing richer features helps, and whether the proposed storage and readout design adds anything beyond those features. Since the six-plane representation itself comes from HexPlane, these gains are central to establishing the paper’s contribution, but the current experiments do not isolate them.

3. Comparisons with recent geometry-guided generation methods are missing. TrajectoryCrafter, NeoVerse, and InSpatio-World show promising results using geometric renderings to guide generation. Comparisons under compatible settings would help establish whether Honeycomb’s learned feature memory brings a practical advantage over these approaches.

Questions for authors
  1. Do you keep all historical 3D points throughout generation, or is there a pruning mechanism? How do total storage and readout time, including point-cloud projection, grow over long sequences?

  2. What happens if HexMemory is removed and the generator is conditioned directly on RGB or latent-feature point-cloud projections? With the same backbone and training setup, how much does HexMemory improve generation quality, and at what storage and readout cost?

  3. For static scenes, why is a write-time axis needed in the memory, and what does it capture beyond the spatial planes? A comparison with a spatial-only tri-plane memory at comparable capacity would help clarify this. Please also include dynamic-scene demos showing how the memory handles moving objects over successive chunks.

7 comments
AuthorsOpenReview ↗

Author's Response - Thank you for your suggestion!

We would like to sincerely thank the reviewer for the valuable suggestion provided. We are committed to improving the paper based on your feedback. As per the reviewer's comments, we have added additional experiments and results, which we detail below.

W1: The total storage and readout cost over long sequences are not reported. Although the feature planes have fixed dimensions, the system still retains historical point clouds and projects them into each target view before querying HexMemory. The paper does not explain how this accumulated geometry or its projection cost is bounded. Tables 3 and 4 report feature-plane storage and writing time, but leave out the total storage and readout cost as the sequence grows. These costs are essential to assessing the scalability of the complete system.

We thank the reviewer for this comment, and we agree that the costs are essential to assess scalability of the system. Our point cache retains only 3D positions, validity masks, and write timestamps (please refer to Q1 for more details). We evaluate total storage over progressively longer histories using 33-frame chunks with a one-frame overlap between consecutive chunks.

Total storage (in MB) over long sequences, corresponding to 65, 129, 193, and 257 frames, as follows:

Component2 chunks4 chunks6 chunks8 chunks
HexMemory21.1321.1321.1321.13
Geometry + metadata1.021.982.933.89
Honeycomb (ours)22.1423.1024.0625.02
Dense RGB points245.88477.30708.72940.14
Direct latent points12.2723.8135.3646.90

Readout time (in ms) over long sequences, as follows:

Method2 chunks4 chunks6 chunks8 chunks
Honeycomb (ours)7.6613.4619.3125.38
Dense RGB points2,969.226,054.359,099.0812,068.06
Direct latent points6.1711.9517.6623.47

From 2 to 8 chunks, Honeycomb’s memory (backend) adds 2.87 MB, compared with 34.64 MB for direct latent points i.e., approximately 92% less additional backend storage. At 8 chunks, readout takes 25.38 ms per view, slightly higher compared with 23.47 ms for latent points and 12,068.06 ms for RGB points. We will include these measurements in the updated manuscript.

W2: It is unclear to me what HexMemory adds over direct point-cloud conditioning. Since the historical point clouds are already retained and projected, why not remove HexMemory and directly use colored point-cloud renderings to condition generation? I would like to see an ablation comparing RGB point-cloud projection, direct latent-feature projection, and HexMemory with the same backbone and training setup. This would show whether storing richer features helps, and whether the proposed storage and readout design adds anything beyond those features. Since the six-plane representation itself comes from HexPlane, these gains are central to establishing the paper’s contribution, but the current experiments do not isolate them.

We thank the reviewer for this helpful suggestion. Indeed point-cloud conditioning is a viable alternative. However, rendering RGB observations and encoding them back into diffusion latents introduces additional computation and can discard latent feature information. This round-trip issue can be addressed by storing and projecting latent features instead.

Our HexMemory further addresses the growing storage cost of retaining per-point latent features by consolidating them into a fixed-size learned feature representation. Our NVS ablation shows that this representation outperforms RGB point-cloud conditioning while closely matching direct latent-feature projection.

We compare the three memory representations on RE10K NVS clips, using the same trained generator and inference settings and ablating the memory representation at inference, as follows:

Memory representationPSNR ↑SSIM ↑LPIPS ↓
RGB point-cloud projection15.80700.58470.4460
Latent-feature projection18.44960.67430.2735
HexMemory (ours)18.45100.67410.2740

We also evaluate the three memory representations on WorldScore closed-loop scenes:

Memory representationPSNR ↑SSIM ↑LPIPS ↓
RGB point-cloud projection17.270.5020.318
Latent-feature projection17.310.5100.300
HexMemory (ours)17.220.5040.311

HexMemory improves NVS PSNR by 2.64 dB over RGB conditioning, with better SSIM and LPIPS, while closely matching direct latent projection. In closed-loop evaluation, it achieves comparable PSNR to RGB conditioning and better SSIM and LPIPS, and latent projection leads by 0.09 dB. These results support competitive quality with fixed-size feature storage and slower memory-backend growth (W1). We will add these ablations to the manuscript.

W3: Comparisons with recent geometry-guided generation methods are missing. TrajectoryCrafter, NeoVerse, and InSpatio-World show promising results using geometric renderings to guide generation. Comparisons under compatible settings would help establish whether Honeycomb’s learned feature memory brings a practical advantage over these approaches.

Thank you for the suggestion. These methods provide valuable additional comparisons, although their architectures and training regimes differ substantially from Honeycomb, Spatia, and LSM. That being said, when the 3 models evaluated using the closed-loop samples, the revisited parts are hallucinated.

We have evaluated their models against Honeycomb for NVS, where all the geometry-guided models performed comparably with Honeycomb. Our closed-loop evaluation reveals severe revisit failures for the geometry-guided models i.e., previously observed regions are replaced with newly hallucinated content upon return. Please kindly refer to this anonymous link for the figure (in comparison to our Figure 1): https://github.com/Anonymous-Submission-20/Honeycomb/blob/main/rebuttal_figures/rebuttal_figureA.pdf

We examine only the revisited scene upon panning back the camera for the closed-loop, as follows:

MethodPSNR ↑SSIM ↑LPIPS ↓
TrajectoryCrafter12.66330.33750.5562
NeoVerse11.87520.31480.6293
InSpatio-World12.35890.29470.5659
Honeycomb (ours)18.89060.50520.2308

For NVS results, as follows:

MethodPSNR ↑SSIM ↑LPIPS ↓
TrajectoryCrafter18.35480.68680.3190
NeoVerse18.52920.66480.3353
InSpatio-World18.87320.68200.2611
Honeycomb (ours)18.45100.67410.2740

In essence, this shows that Honeycomb achieves competitive NVS performance while substantially improving consistency in revisited regions, exceeding the strongest geometry-guided baseline by 6.23 dB PSNR. We will include these comparisons in the updated manuscript.

Q1: Do you keep all historical 3D points throughout generation, or is there a pruning mechanism? How do total storage and readout time, including point-cloud projection, grow over long sequences?

We do not keep all historical 3D points throughout generation. Historical per-point feature vectors are discarded after being written into HexMemory. Write-camera directions and centres are also discarded, as later incremental writes do not use them. We retain only the historical points’ 3D positions, validity masks, and write timestamps which are necessary for visibility and memory querying. We will now explicitly mention this in the updated manuscript.

Q2: What happens if HexMemory is removed and the generator is conditioned directly on RGB or latent-feature point-cloud projections? With the same backbone and training setup, how much does HexMemory improve generation quality, and at what storage and readout cost?

We have ablated HexMemory by conditioning Honeycomb directly on RGB or latent points. Our model achieved competitive NVS/closed-loop results compared to RGB/latent points, while maintaining fixed-size HexMemory feature storage and requiring substantially less additional memory-backend storage as the history grows. Please kindly refer to W2 for the generation quality comparison, and W1 for the storage and readout cost comparison.

Q3: For static scenes, why is a write-time axis needed in the memory, and what does it capture beyond the spatial planes? A comparison with a spatial-only tri-plane memory at comparable capacity would help clarify this. Please also include dynamic-scene demos showing how the memory handles moving objects over successive chunks.

The spatiotemporal planes form part of our low-rank feature factorization and can therefore contribute spatial detail even in static scenes, which makes retaining the temporal information important too. Figure 10 in the Appendix shows how the spatiotemporal planes store the features of the scene, which helps in generation/revisit quality.

We have conducted an ablation comparing the three spatial planes (i.e., tri-plane) with the current six-plane representation on the full RE10K NVS and WorldScore closed-loop evaluations.

On RE10K NVS, the six-plane representation achieves better mean performance, as follows:

RepresentationPSNR ↑SSIM ↑LPIPS ↓
Tri-plane (XY, XZ, YZ)18.01930.67130.2747
Six-plane (ours)18.45100.67410.2740

On WorldScore closed-loop evaluation, the advantage is more pronounced, as follows:

RepresentationPSNR ↑SSIM ↑LPIPS ↓
Tri-plane (XY, XZ, YZ)16.02770.48270.3352
Six-plane (ours)17.22030.50390.3106

We have also included a dynamic-scene demo comparison across Spatia, LSM and ours, showing how the memory handles moving objects over successive chunks. In this anonymous figure (https://github.com/Anonymous-Submission-20/Honeycomb/blob/main/rebuttal_figures/rebuttal_figureB.pdf), we provide a qualitative example of ducks continuing to move coherently in a park, across successive generated chunks as the camera pans back. Compared to other models e.g., Spatia, the ducks incoherently flew backwards/appeared out of the ground as the camera pans back. While this demo illustrates how our HexMemory handles moving objects over successive chunks, more rigorous evaluation of dynamic-state tracking and the memory’s contribution remains as interesting future work.

We thank the reviewer for the insightful comments, and we hope these address your concerns.

Reviewer_6DEtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses scene inconsistency across sequentially generated video chunks in world models by introducing an efficient memory mechanism. Its HexPlane-based design maintains a fixed-size feature memory throughout generation. Experiments demonstrate the compactness of this representation and its effectiveness in improving scene consistency.

Critical strengths
  1. Adapting HexPlane representations for persistent scene memory in world models is a novel and well-motivated direction. The fixed-capacity design is also attractive for deployment under constrained hardware resources.

  2. The experiments provide convincing evidence of the proposed method’s effectiveness, including ablations across different memory budgets. The evaluation of feature storage and memory-update efficiency is also relatively comprehensive.

Critical weaknesses
  1. Although fixed-size storage is appealing, the authors acknowledge that expanding the spatial coverage leads to a coarser effective representation. However, the practical implications of this trade-off remain insufficiently explored. Could the authors provide guidelines for selecting plane resolutions and memory budgets according to the intended spatial extent of the generated world? A controlled evaluation that progressively expands the explored region under a fixed memory budget would help characterize the method’s scalability and the resulting degradation in detail preservation and revisit consistency.

  2. Additional ablations on confidence-weighted pooling and the learned residual correction would strengthen the analysis. In particular, comparing uniform averaging against confidence-weighted pooling, both with and without residual correction, would help disentangle their individual contributions to memory retention and scene consistency.

Questions for authors

See the weaknesses above.

2 comments
AuthorsOpenReview ↗

Author's Response - Thank you for your feedback!

We thank the reviewer for the valuable feedback. Following your suggestions, we have conducted additional experiments and incorporated new results to strengthen the paper. We present our responses and findings below.

W1: Although fixed-size storage is appealing, the authors acknowledge that expanding the spatial coverage leads to a coarser effective representation. However, the practical implications of this trade-off remain insufficiently explored. Could the authors provide guidelines for selecting plane resolutions and memory budgets according to the intended spatial extent of the generated world? A controlled evaluation that progressively expands the explored region under a fixed memory budget would help characterize the method’s scalability and the resulting degradation in detail preservation and revisit consistency.

We thank the reviewer for this insightful suggestion. We now evaluate closed-loop generation on WorldScore scenes using 2, 4, and 6 autoregressive chunks, with each trajectory moving outward for half the total chunks before returning. This doubles and triples the outward trajectory length relative to the two-chunk setting while keeping per-frame camera motion unchanged. We repeat this experiment at plane resolutions 128, 256, 384, and 512 across expanding spatial coverage at each resolution. The HexMemory storage is constant throughout per resolution.

Each cell reports PSNR ↑ / SSIM ↑ / LPIPS ↓, as follows:

Plane resolutionHexMemory (MB)2 chunks4 chunks6 chunks
PSNR ↑ / SSIM ↑ / LPIPS ↓PSNR ↑ / SSIM ↑ / LPIPS ↓PSNR ↑ / SSIM ↑ / LPIPS ↓
51273.917.22 / 0.504 / 0.31115.61 / 0.453 / 0.47113.68 / 0.400 / 0.590
38442.617.14 / 0.503 / 0.31315.53 / 0.449 / 0.47713.44 / 0.393 / 0.602
25619.817.10 / 0.500 / 0.31915.39 / 0.444 / 0.49013.53 / 0.393 / 0.616
1285.816.77 / 0.490 / 0.33815.09 / 0.433 / 0.51913.28 / 0.384 / 0.643

Reducing resolution from 512 to 256 produces a slight degradation i.e., approximately 0.12–0.22 dB PSNR across the tested lengths, with a larger drop when reducing further to 128. With the storage measurements, these results suggest 256 as a practical memory-quality compromise, while higher resolutions favor revisit fidelity.

Longer trajectories produce a relatively larger decline in revisit quality. Do note that these measurements also include autoregressive errors, so the decline cannot be attributed solely to coarser memory resolution (future work can consider self-forcing approaches to improve on this). We will include these results in the updated manuscript.

W2: Additional ablations on confidence-weighted pooling and the learned residual correction would strengthen the analysis. In particular, comparing uniform averaging against confidence-weighted pooling, both with and without residual correction, would help disentangle their individual contributions to memory retention and scene consistency.

We thank the reviewer for this feedback. We conduct the 2×2 ablation comparing uniform and confidence-weighted pooling, each with and without residual correction. For memory retention, we sequentially write video chunks into HexMemory and measure the reconstruction MSE of the same first-chunk features after each autoregressive update (which will be affected by the warping). One, two, and three chunks correspond to 33, 65, and 97 frames.

Result for the reconstruction MSE of first-chunk features:

PoolingResidual1 chunk ↓2 chunks ↓3 chunks ↓
UniformNo0.07150.11900.1580
UniformYes0.06300.10050.2738
ConfidenceNo0.06610.10510.1296
ConfidenceYes (ours)0.05980.09100.1275

For scene consistency, we evaluate closed-loop generation on WorldScore scenes:

PoolingResidualPSNR ↑SSIM ↑LPIPS ↓
UniformNo17.200.5030.311
UniformYes17.210.5050.310
ConfidenceNo17.160.5030.313
ConfidenceYes (ours)17.220.5040.311

The benefits are clearest in memory retention, where both components independently improve recall accuracy, and their combination yields the lowest error at all three evaluated horizons. Closed-loop improvements are smaller, with the full model achieving comparable PSNR, SSIM and LPIPS. Note that confidence-weighted pooling and residual correction are just a small addition to improve the HexMemory performance slightly. We will include these results and observations in the updated manuscript.

We sincerely thank the reviewer for the suggestions and we hope these address your concerns.

AuthorsOpenReview ↗

Summary of Response - Hugely appreciate the suggestions and looking forward to engaging further!

We thank the reviewer again for the constructive feedback. As the discussion period draws to a close, we would like to follow up on our rebuttal and highlight the additional experiments conducted in response to your concerns. Specifically, we evaluated the scalability of HexMemory under fixed memory budgets and disentangled the contributions of confidence-weighted pooling and residual correction. We summarize the key findings below and indicate where they appear in the revised manuscript.

Feedback raisedOur response to feedbackLocation
W1. How does HexMemory scale with expanding spatial coverage under a fixed memory budget?We evaluated WorldScore closed-loop generation over 2, 4, and 6 chunks at plane resolutions of 128, 256, 384, and 512, keeping memory storage fixed at each resolution while progressively expanding the explored region. Reducing resolution from 512 to 256 saves approximately 73% of feature storage (73.9 MB to 19.8 MB), with only a 0.12–0.22 dB PSNR decrease across trajectory lengths, suggesting a practical memory-quality trade-off. Longer trajectories exhibit greater revisit degradation, partly due to accumulated autoregressive errors as well.Section 4.5, Table 4
W2. What are the individual contributions of confidence-weighted pooling and residual correction?We conducted a 2×2 ablation comparing uniform and confidence-weighted pooling, with and without residual correction. The full model achieves the lowest memory reconstruction MSE at all three evaluated horizons (0.0598, 0.0910, and 0.1275), demonstrating improved retention of earlier features across successive memory updates. Closed-loop improvements are smaller, with the full model achieving the highest PSNR (17.22 dB) and comparable SSIM and LPIPS.Appendix B, Table 7

We hope these additional experiments and clarifications address the concerns raised. We would greatly appreciate any further feedback from the reviewer and thank you again for your time and consideration!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief summary and huge thank you to all reviewers!

The authors of Submission 5262 would like to once again thank all the reviewers for the fruitful and insightful discussion. We are very pleased that all three reviewers recognize the value of our approach and all have given positive ratings.

  • Reviewer 6DEt highlighted the novelty of adapting HexPlane for persistent scene memory and its practical storage benefits. In response to questions about expanding spatial coverage and memory updates, we added evaluations across trajectory lengths and memory budgets, together with ablations of confidence-weighted pooling and residual correction.
  • Reviewer mQZB recognized the value of efficient latent feature storage. We added total storage and readout measurements, comparisons with RGB and latent-feature conditioning, and revised geometry-guided comparisons that reuse generated history through each baseline’s supported mechanisms. The reviewer also kindly confirmed that the updated results and clarifications resolved their remaining question.
  • Reviewer hKpK highlighted the compact feature storage and efficient memory updates. We evaluated memory retention over longer histories and clarified how spatiotemporal planes contribute spatial information even in static scenes. We also clarified that disabling these planes is an inference-time component ablation. A separately trained tri-plane comparison remains an additional suggestion that we will explore.

We have incorporated all these results and clarifications into the revised manuscript and greatly appreciate the reviewers’ time and contributions to strengthening the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.