- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes CoSTA4D, a streaming feedforward 4D human body reconstruction framework designed for sparse, uncalibrated multi-view video. The method leverages the time invariance of the capturing cameras to stabilize current geometric inferences using historical observations, iteratively optimize shared camera states, and enhance cross-frame representational continuity through Gaussian propagation. The authors evaluated the method on DNA-Rendering and self-recorded videos in terms of new-view reconstruction, camera estimation, and temporal consistency, without the need for optimization for individual sequences.
Critical strengths
1.The overall experimental evaluation is relatively comprehensive, covering reconstruction quality, temporal consistency, camera estimation, efficiency, and comparisons on both benchmark and self-captured videos.
2.The paper addresses a meaningful problem in 4D human body reconstruction using streaming data, and its core motivation is well-founded: while the camera state remains stable over time, the human body’s geometry continuously changes. Leveraging this contrast in temporal persistence to organize historical information provides a reasonable and somewhat novel approach to sparse, uncalibrated multi-view reconstruction.
Critical weaknesses
1.The paper’s central claim regarding temporal persistence goes beyond what has been directly demonstrated to date. The authors emphasize that historical states should be organized according to the physical persistence of different types of information, treating long-term, stable camera states differently from recent, dynamic human information. Existing experiments have demonstrated that historical information is generally useful, but have not yet directly proven that this persistence-aware differentiated organization offers clear advantages over a simpler, unified temporal memory.
2.The method presents a meaningful degree of novelty, but the overall innovation is relatively limited. The paper organizes historical information in streaming 4D human reconstruction from the perspective of temporal persistence, which is a meaningful viewpoint. However, the concrete implementations, such as temporal aggregation and reliability-aware recursive camera refinement, are relatively straightforward, making the overall methodological contribution somewhat less distinctive.
Questions for authors
1.Could the authors provide a more direct comparison between the proposed persistence-aware history organization and a simpler unified temporal memory? In particular, it would be helpful to clarify whether the persistent anchor and recent context contribute differently, rather than only showing the gain of temporal aggregation as a whole.
2.Could the author further clarify the specific role of correspondence reliability in Gaussian propagation? It is currently unclear whether the improvement in temporal consistency stems primarily from the propagation of historical Gaussians or from the proposed reliability-guided selection/gating mechanism.
3.How sensitive is CoSTA4D to the amount and quality of historical information? Since the method continuously accumulates past observations, it would be useful to clarify how its performance changes when only limited history is available or when historical observations are affected by noise or occlusion.
2 comments
Thank you for the additional experiments. The comparison with unified history addresses my main concern and clarifies what the anchor adds beyond recent frames. I will keep my rating at 3. In the final version of the paper, I would still like to see an ablation that isolates the effect of reliability gating in Gaussian propagation.
Response to Reviewer BtoM
We thank the reviewer for recognizing our motivation and evaluation. All ablations below use the DNA-Rendering test set.
Q1: Anchor–Context vs. Unified History. We compare five memory configurations, including Persistent History, which uses 1024 learnable query tokens to aggregate all past features without separating long-term and recent information (Appendix A.1, Tables 5–6).
Table R1. Camera and depth geometry errors (lower is better).
Table R2. Depth temporal consistency and Gaussian scale.
Recent Context provides most gains in dynamic geometry and temporal consistency, while the Anchor further improves FoV, scale stability, and depth accuracy. Compared with Persistent History, our design reduces rotation error by 29.8% and depth RMSE by 25.8%. Although Persistent History achieves slightly better depth tLP, the overall results support separating persistent references from recent dynamics rather than uniformly accumulating history.
Q2: Role of Correspondence Reliability. Gaussian Propagation establishes cross-frame continuity by reusing the final previous Gaussian state, while Correspondence Reliability determines which historical attributes should be retained. We propagate historical Gaussians at reliable backward 3D correspondences and retain full current-frame predictions elsewhere, limiting the accumulation of outdated or incorrect attributes. Thus, propagation maintains temporal continuity, while reliability gating prevents unreliable history from persisting. Their combined effectiveness is supported by Table 4 and Fig. 6 (Sec. 3.4; Appendix A.4).
Q3: History Length and Quality. We evaluate recent-context lengths L={0,1,3,6,9}, where L=0 retains only the Anchor and L=3 is the default (Appendix A.1, Table 7).
Table R3. Recent-context length sensitivity.
L=3 performs best, while larger windows (6/9) degrade reconstruction quality, indicating that more history is not necessarily beneficial.
We also inject zero-mean Gaussian noise only into historical attention K/V caches at inference, keeping current inputs unchanged. Noise follows
α × RMS(K/V)per cache entry, view, and attention head across all 24 layers (Appendix A.1, Table 8).Table R4. Historical K/V noise sensitivity.
Mild corruption reduces PSNR by 0.32 dB and increases tOF by 4.7%, while severe corruption substantially increases temporal errors.
We further evaluate recovery from a single corrupted input frame by replacing an intermediate frame with an all-black image and then resuming normal inputs. The mean PSNR over the following five frames decreases by 0.99 dB, but reconstruction quality recovers to approximately the normal-input level by the third frame (Appendix A.1, Table 9 and Fig. 8).
Together, these results support bounded recent context and demonstrate the importance of historical feature quality, while showing that CoSTA4D can recover reconstruction quality after an isolated input failure.