acceptodds
Under review as a conference paper at ICLR 2027

CoSTA4D: Consistent Spatio-Temporal Aggregation for Streaming 4D Human Reconstruction from Uncalibrated Sparse-View Videos

Abstract

Streaming human reconstruction from sparse, uncalibrated multi-view videos requires both geometric accuracy and temporal consistency. While human motion and visibility vary over time, the capture cameras remain fixed; however, changing visual evidence can still cause unstable camera and depth predictions, leading to geometric jitter and rendering flicker. This suggests that historical observations should not only provide temporal context for current reconstruction, but also accumulate evidence about the invariant capture geometry. We present CoSTA4D, a streaming feed-forward framework built on this principle. CoSTA4D combines anchor-context temporal aggregation with reliability-aware camera refinement, enabling historical context to stabilize geometry inference while repeated camera observations progressively improve the shared camera estimates and back-projected geometry. Correspondence-guided Gaussian propagation further encourages continuity of the dynamic representation across frames. The framework uses only current and past observations without per-sequence optimization. Experiments on DNA-Rendering and self-captured videos demonstrate state-of-the-art reconstruction quality and improved temporal consistency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dynamic Human Reconstruction, Freeview synthesis, Multi-view Geometry

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BtoMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CoSTA4D, a streaming feedforward 4D human body reconstruction framework designed for sparse, uncalibrated multi-view video. The method leverages the time invariance of the capturing cameras to stabilize current geometric inferences using historical observations, iteratively optimize shared camera states, and enhance cross-frame representational continuity through Gaussian propagation. The authors evaluated the method on DNA-Rendering and self-recorded videos in terms of new-view reconstruction, camera estimation, and temporal consistency, without the need for optimization for individual sequences.

Critical strengths

1.The overall experimental evaluation is relatively comprehensive, covering reconstruction quality, temporal consistency, camera estimation, efficiency, and comparisons on both benchmark and self-captured videos.

2.The paper addresses a meaningful problem in 4D human body reconstruction using streaming data, and its core motivation is well-founded: while the camera state remains stable over time, the human body’s geometry continuously changes. Leveraging this contrast in temporal persistence to organize historical information provides a reasonable and somewhat novel approach to sparse, uncalibrated multi-view reconstruction.

Critical weaknesses

1.The paper’s central claim regarding temporal persistence goes beyond what has been directly demonstrated to date. The authors emphasize that historical states should be organized according to the physical persistence of different types of information, treating long-term, stable camera states differently from recent, dynamic human information. Existing experiments have demonstrated that historical information is generally useful, but have not yet directly proven that this persistence-aware differentiated organization offers clear advantages over a simpler, unified temporal memory.

2.The method presents a meaningful degree of novelty, but the overall innovation is relatively limited. The paper organizes historical information in streaming 4D human reconstruction from the perspective of temporal persistence, which is a meaningful viewpoint. However, the concrete implementations, such as temporal aggregation and reliability-aware recursive camera refinement, are relatively straightforward, making the overall methodological contribution somewhat less distinctive.

Questions for authors

1.Could the authors provide a more direct comparison between the proposed persistence-aware history organization and a simpler unified temporal memory? In particular, it would be helpful to clarify whether the persistent anchor and recent context contribute differently, rather than only showing the gain of temporal aggregation as a whole.

2.Could the author further clarify the specific role of correspondence reliability in Gaussian propagation? It is currently unclear whether the improvement in temporal consistency stems primarily from the propagation of historical Gaussians or from the proposed reliability-guided selection/gating mechanism.

3.How sensitive is CoSTA4D to the amount and quality of historical information? Since the method continuously accumulates past observations, it would be useful to clarify how its performance changes when only limited history is available or when historical observations are affected by noise or occlusion.

2 comments

Response to Reviewer BtoM

We thank the reviewer for recognizing our motivation and evaluation. All ablations below use the DNA-Rendering test set.

Q1: Anchor–Context vs. Unified History. We compare five memory configurations, including Persistent History, which uses 1024 learnable query tokens to aggregate all past features without separating long-term and recent information (Appendix A.1, Tables 5–6).

Table R1. Camera and depth geometry errors (lower is better).

HistoryRot. (°)Center/radiusFoV (rad)Scale CVDepth RMSE
None0.56690.0090230.0026410.03120.007489
Anchor Only0.56570.0089180.0026700.02640.007280
Recent Only0.49550.0082660.0024210.02830.007256
Persistent History0.70420.0106010.0036300.03580.009741
Anchor + Recent0.49460.0082890.0022830.02580.007225

Table R2. Depth temporal consistency and Gaussian scale.

HistoryDepth tOF↓Depth tLP↓Gaussian Scale
None0.4540810.0012520.000827
Anchor Only0.3543540.0010680.000824
Recent Only0.2541970.0009510.000816
Persistent History0.3702480.0009430.001018
Anchor + Recent0.2541220.0009470.000807

Recent Context provides most gains in dynamic geometry and temporal consistency, while the Anchor further improves FoV, scale stability, and depth accuracy. Compared with Persistent History, our design reduces rotation error by 29.8% and depth RMSE by 25.8%. Although Persistent History achieves slightly better depth tLP, the overall results support separating persistent references from recent dynamics rather than uniformly accumulating history.

Q2: Role of Correspondence Reliability. Gaussian Propagation establishes cross-frame continuity by reusing the final previous Gaussian state, while Correspondence Reliability determines which historical attributes should be retained. We propagate historical Gaussians at reliable backward 3D correspondences and retain full current-frame predictions elsewhere, limiting the accumulation of outdated or incorrect attributes. Thus, propagation maintains temporal continuity, while reliability gating prevents unreliable history from persisting. Their combined effectiveness is supported by Table 4 and Fig. 6 (Sec. 3.4; Appendix A.4).

Q3: History Length and Quality. We evaluate recent-context lengths L={0,1,3,6,9}, where L=0 retains only the Anchor and L=3 is the default (Appendix A.1, Table 7).

Table R3. Recent-context length sensitivity.

LPSNR↑SSIM↑LPIPS↓
031.22850.97550.0301
131.34060.97780.0297
3 (default)31.34290.97810.0289
631.13320.97620.0304
930.93620.97470.0317

L=3 performs best, while larger windows (6/9) degrade reconstruction quality, indicating that more history is not necessarily beneficial.

We also inject zero-mean Gaussian noise only into historical attention K/V caches at inference, keeping current inputs unchanged. Noise follows α × RMS(K/V) per cache entry, view, and attention head across all 24 layers (Appendix A.1, Table 8).

Table R4. Historical K/V noise sensitivity.

NoiseScalePSNR↑SSIM↑LPIPS↓tOF↓tLP×100↓
None031.34290.97810.02891.79540.1405
Mild0.05 × RMS31.01870.97740.02941.87900.2019
Severe0.50 × RMS30.81740.97690.03092.78650.4315

Mild corruption reduces PSNR by 0.32 dB and increases tOF by 4.7%, while severe corruption substantially increases temporal errors.

We further evaluate recovery from a single corrupted input frame by replacing an intermediate frame with an all-black image and then resuming normal inputs. The mean PSNR over the following five frames decreases by 0.99 dB, but reconstruction quality recovers to approximately the normal-input level by the third frame (Appendix A.1, Table 9 and Fig. 8).

Together, these results support bounded recent context and demonstrate the importance of historical feature quality, while showing that CoSTA4D can recover reconstruction quality after an isolated input failure.

Thank you for the additional experiments. The comparison with unified history addresses my main concern and clarifies what the anchor adds beyond recent frames. I will keep my rating at 3. In the final version of the paper, I would still like to see an ablation that isolates the effect of reliability gating in Gaussian propagation.

Reviewer_FyPXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses causal 4D human reconstruction from four fixed, sparse, uncalibrated multi-view video streams. The cameras remain fixed while human pose, visibility, and geometry change. CoSTA4D exploits this distinction through three components. Anchor-Context Temporal Aggregation combines persistent first-frame tokens with recent per-view context to predict depth and cameras. Reliability-Aware Camera Refinement treats timestamp-level camera estimates as repeated observations of one fixed camera state and recursively fuses rotation, translation, and FoV for depth back-projection. Correspondence-Guided Gaussian Propagation uses backward 3D motion and confidence to select historical Gaussians or retain current predictions. Training first optimizes geometry, camera, depth, and reliability heads; it then adds Gaussian rendering and motion objectives, with SEA-RAFT used only as a training-time teacher. On DNA-Rendering and two self-captured sequences, CoSTA4D reports the best PSNR, SSIM, LPIPS, tOF, and tLP among the compared methods, while camera errors decrease as observations accumulate. Ablations attribute spatial gains mainly to temporal aggregation and camera refinement, and temporal gains to Gaussian propagation.

Critical strengths
  1. The state decomposition is physically well motivated. The distinction between transient human geometry and fixed capture cameras is clear and directly motivates separate temporal treatment for dynamic context and long-lived camera state.

  2. Camera refinement affects geometry rather than only smoothing outputs. Reliability-aware fusion of rotation, translation, and FoV is used for depth back-projection during training and inference. Section 4.3 reports final reductions of 17.6%, 22.0%, and 12.0% in rotation, camera-center, and FoV errors relative to the direct CameraHead.

  3. The evaluation covers complementary quality dimensions. The paper reports spatial rendering metrics, temporal metrics, progressive camera errors, component ablations, and self-captured generalization. Table 4 helps separate the roles of temporal aggregation, camera refinement, and Gaussian propagation.

Critical weaknesses
  1. The temporal and state protocol underlying the main quality tables is under-specified. Section 4.1 states that CoSTA4D maintains historical states continuously over each test sequence and that metrics are averaged per sequence, but the paper does not explicitly state whether PSNR/SSIM/LPIPS are computed over all timestamps, endpoint frames, or sampled frames. It also does not specify how much history has been processed before the evaluated outputs or whether those outputs use the refined shared cameras and recursively propagated Gaussians. Because CoSTA4D is streaming while some baselines are primarily frame-wise, the current description makes it difficult to determine whether reported gains come from the architecture, accumulated history, camera refinement, or Gaussian propagation. The authors should report the exact frame selection and state protocol and provide time-indexed or first-frame-versus-later-frame results.

  2. Baseline coverage is insufficient for both streaming and feed-forward human reconstruction. Although Forge4D is included and 4DGT is reported as a posed-video reference, the paper omits recent streaming dynamic-Gaussian systems such as StreamSplat [1] and Instant Gaussian Stream [2], as well as the recent calibrated human-centric method PointSplat [3]. These omissions matter for two claims: the overall SOTA comparison and the advantage of correspondence-guided Gaussian propagation over prior online propagation/fusion designs. The protocols are not identical—PointSplat uses calibrated inputs, while StreamSplat and Instant Gaussian Stream target different streaming setups—so matched adaptations or clearly delimited upper-bound comparisons may be necessary. This is compatible with the current table, which already annotates different camera-input requirements and includes a posed-video reference.

  3. Temporal stability under simultaneous viewpoint and time changes is not adequately evaluated. The tOF and tLP metrics are computed from fixed held-out viewpoints, and the qualitative figures are static image comparisons. Thus, the submission does not quantitatively demonstrate stability when both the rendering camera and the dynamic human change over time, which is a core free-viewpoint-video setting. In the supplementary video, several fixed-view segments also appear to exhibit visible high-frequency fluctuations, although no metric isolates this behavior. The authors should provide clearly labeled camera-trajectory results in which the camera moves while time advances, together with baseline comparisons and frame-indexed temporal-error curves. Separate low- and high-frequency analyses would help distinguish genuine human motion from flicker.

[1] Zike Wu, Qi Yan, Xuanyu Yi, Lele Wang, and Renjie Liao. StreamSplat: Towards Online Dynamic 3D Reconstruction from Uncalibrated Video Streams. In International Conference on Learning Representations, 2026. arXiv:2506.08862.

[2] Jinbo Yan, Rui Peng, Zhiyan Wang, Luyang Tang, Jiayu Yang, Jie Liang, Jiahao Wu, and Ronggang Wang. Instant Gaussian Stream: Fast and Generalizable Streaming of Dynamic Scene Reconstruction via Gaussian Splatting. arXiv:2503.16979, 2025.

[3] Yujie Guo, Yudong Jin, Lingteng Qiu, Zehong Shen, Zhen Xu, Jing Zhang, Xianchao Shen, Hujun Bao, Sida Peng, and Xiaowei Zhou. PointSplat: Compact Gaussian Splatting via Human-Centric Prediction. arXiv:2606.32036, 2026.

Questions for authors
  1. For Tables 1 and 2, are PSNR/SSIM/LPIPS computed over all timestamps, endpoint frames, or sampled frames? How much history is processed before evaluation, and do the reported outputs use refined cameras and propagated Gaussians?

  2. How do the per-frame baselines process each test sequence? In particular, do they receive only the current timestamp, or are they adapted to use any temporal history?

  3. Could the authors compare with or adapt recent streaming/feed-forward baselines? If direct comparison is unsuitable, could calibrated or streaming upper-bound protocols be reported separately?

  4. Table 4 shows that Gaussian propagation improves temporal metrics but slightly lowers PSNR relative to Temporal + Camera. Is this trade-off consistent across sequences, and what is the corresponding per-sequence variance or statistical significance?

  5. Can the authors provide camera-trajectory renders where both viewpoint and time advance, together with frame-indexed low- and high-frequency temporal-error analyses? This would help assess the apparent fixed-view fluctuations and separate genuine motion from flicker.

1 comment

Response to Reviewer FyPX

Thank you for the constructive review. We clarify the protocols and add temporal analyses, baselines and statistics.

W1/Q1: Temporal Evaluation Protocol. PSNR/SSIM/LPIPS in Tables 1–2 use all timestamps. Each sequence starts with empty history and no warm-up; the first frame initializes memory, cameras, and Gaussians. Later outputs use only current/past observations, refined shared cameras, and propagated Gaussians (Sec. 4.1).

Table R1. Frame-indexed results on DNA-Rendering.

FramePSNR↑SSIM↑LPIPS↓
130.85140.97700.0296
5031.28330.97740.0287
10031.48290.97840.0274
15031.76730.97930.0282

PSNR increases from 30.85 to 31.77 dB as the sequence progresses, indicating an overall improvement in reconstruction quality (Appendix A.8, Table 13). The contributions of individual components are further analyzed in the ablations (Table 4).

W1/Q2: Baseline History. Baselines follow released implementations and native training/inference protocols, with fine-tuning on the corresponding training split. Frame-wise methods use only the current timestamp; temporal methods retain native history mechanisms. Evaluation metrics and foreground masks are shared; camera requirements and output resolutions remain method-specific (Sec. 4.1).

W2/Q3: Additional Baselines. We include PointSplat, Instant Gaussian Stream (IGS), and StreamSplat in our comparisons, following their original implementations and training protocols (Sec. 4.1). The results are integrated into Sec. 4.2, Tables 1–2.

Table R2. DNA-Rendering.

MethodCam.PSNR↑SSIM↑LPIPS↓tOF↓tLP×100↓
PointSplatK,T29.11720.97290.03642.72110.4550
IGSK,T26.49020.95600.05774.54240.8315
StreamSplat∅25.27170.96160.05354.53410.8003
Ours∅31.34290.97810.02891.79540.1405

Table R3. Self-captured videos.

MethodCam.PSNR↑SSIM↑LPIPS↓tOF↓tLP×100↓
PointSplatK,T29.52130.96130.06071.02720.4437
IGSK,T27.46150.96820.05481.26560.9881
StreamSplat∅26.31420.95610.07181.42840.9447
Ours∅30.14720.97420.03510.66370.2795

CoSTA4D achieves the best results across all metrics on both datasets.

Q4: Statistical Analysis of Gaussian Propagation. We compare Temporal + Camera (without propagation) and Full Model (with propagation) across 30 DNA-Rendering sequences using paired statistics (Sec. 4.5, Table 4; Appendix A.8, Table 14). Differences are defined as without minus with propagation.

MetricWithoutWithMean Diff.SD95% CI
PSNR↑31.428631.3429+0.08570.7952[-0.2112,0.3826]
tOF↓2.01211.7954+0.21670.5324[0.0179,0.4155]
tLP×100↓0.19680.1405+0.05630.1434[0.0028,0.1098]

Propagation reduces mean tOF/tLP by 10.8%/28.6% (paired t-tests, unadjusted p=0.034/0.040); the 0.086 dB mean PSNR decrease is not significant (p=0.560).

W3/Q5: Moving-View Temporal Consistency. In Mongolian Dance, fluctuations occur mainly near multi-view stitching boundaries and coincide with inconsistent masks, particularly around white clothing and complex backgrounds. An additional single-view Gaussian rendering video shows largely stable per-view reconstructions and mask inaccuracies. Improving temporal and cross-view segmentation consistency remains future work.

(1) Moving-view evaluation. We evaluate reconstruction under simultaneous changes in target viewpoint and human motion on 4D-DRESS, using the same four-view input setting as Forge4D (Appendix A.8, Table 15).

MethodViewsPSNR↑SSIM↑LPIPS↓tLP×100↓tOF(px)↓
Ours425.8060.960810.031500.16300.4674
Forge4D423.9300.942210.058710.57380.7358

Compared with Forge4D, CoSTA4D achieves higher spatial reconstruction quality while reducing tLP and tOF by 71.6% and 36.5%, respectively.

(2) Frame-indexed frequency analysis. Appendix A.8, Fig. 9 shows frame-indexed tLP/tOF RMS errors and their low-frequency (≤2 Hz) and high-frequency (>2 Hz) components. CoSTA4D exhibits lower overall errors and reduced high-frequency fluctuations, including transient tOF peaks. The Supplementary Material (SM) includes a 4D-DRESS orbiting-view video with simultaneous viewpoint and human motion.

Reviewer_MQCyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoSTA4D, a streaming framework for 4D human reconstruction from sparse, uncalibrated multi-view videos. It combines temporal aggregation, reliability-aware camera refinement, and Gaussian propagation to improve reconstruction quality and temporal consistency. Experiments demonstrate superior performance over existing methods.

Critical strengths

The paper addresses a meaningful and well-motivated problem of achieving temporally consistent 4D human reconstruction from sparse, uncalibrated multi-view videos. The experimental results are promising, showing improvements in both reconstruction quality and temporal consistency across different datasets. The qualitative results also demonstrate visually convincing reconstructions.

Critical weaknesses
  • W1. Limited Methodological Novelty. The methodological novelty of the proposed framework appears somewhat limited. The three main components—temporal aggregation, reliability-aware camera refinement, and Gaussian propagation—are closely related to techniques explored in prior work. In particular, the technical distinctions between the proposed method and existing streaming reconstruction approaches, such as Forge4D, are not sufficiently clarified.
  • W2. Insufficient Evaluation of Streaming Efficiency. Although the paper emphasizes streaming 4D reconstruction, the evaluation of computational efficiency is insufficient. Important metrics such as inference latency, FPS, and GPU memory usage are not reported. Without these results, it is difficult to assess the practicality and scalability of the proposed framework in real-world streaming scenarios.
  • W3. Limited Generalization to Diverse Capture Settings. The proposed method assumes a relatively restrictive setup with four synchronized, fixed cameras. Furthermore, experiments with diverse camera configurations are limited, and generalization to unseen environments is evaluated on only two self-captured sequences. Therefore, it remains unclear whether the proposed method can maintain robust performance under different camera arrangements and capture conditions.
  • W4. Insufficient Ablation and Qualitative Analysis. The analysis of the individual contributions of each component could be more comprehensive. Although Table 4 provides quantitative ablation results, the limited qualitative comparisons in Figure 6 make it difficult to fully understand how each component affects reconstruction quality and temporal consistency. More detailed visualizations and analyses would help clarify which types of errors are specifically addressed by camera refinement and Gaussian propagation.
Questions for authors

I would appreciate it if the authors could address the concerns raised in the Weaknesses section, particularly regarding the limited methodological novelty, the lack of streaming efficiency evaluation, the generalization to diverse camera configurations, and the insufficient ablation and qualitative analysis.

1 comment

Response to Reviewer MQCy

Thank you for the constructive review. We address the four concerns below.

W1: Methodological Novelty. Our insight is to retain and update history according to each component's temporal validity: human poses and occlusions change, whereas capture cameras remain fixed.

Forge4D/CUT3R recursively integrate past observations, while StreamVGGT caches historical features. Our first-frame Anchor + bounded Recent Context separates persistent references from recent dynamics. Compared with Forge4D's recursive history integration, we limit explicit dynamic context to the most recent L frames to reduce potential interference from outdated poses and occlusions. Unlike the sequential image-stream formulations of CUT3R/StreamVGGT, we combine within-timestamp multi-view interaction with same-camera temporal aggregation. Recent context mainly improves geometry, while the Anchor further benefits camera/depth estimation; unified history and larger windows yield no consistent gains (Appendix A.1, Tables 5–7). We also recursively refine fixed cameras using reliable observations for depth back-projection, beyond cross-frame scale consistency (Table 3, Fig. 5). For dynamic Gaussians, we selectively propagate reliable historical attributes at observed timestamps, whereas Forge4D uses fusion for intermediate-time interpolation (Table 4, Fig. 6).

W2: Streaming Efficiency. Efficiency was reported in the original Appendix A.6, Table 5 (now Appendix A.7, Table 10). With four views on an A100, endpoint inference takes 212.75 ms, achieving 4.61 timestamp FPS with 6.14 GiB peak memory. The full pipeline, including motion prediction and Gaussian propagation, achieves 4.13 FPS and 6.50 GiB. An RTX PRO 6000 Blackwell test reaches 22.43 FPS in the full streaming configuration. Latency measures inference only; timestamp FPS includes inference and Gaussian rendering.

W3: Capture Generalization. Four surrounding cameras provide approximate 360° human coverage without requiring exact 90° spacing. We train one variable-view model (4–8 inputs) to test different camera counts and layouts (Appendix A.8, Table 11).

Table R1. DNA-Rendering. Perturbed layouts use approximately uniform surrounding cameras with angular perturbations; the highly non-uniform six-view layout adds two cameras to the original four-view arrangement.

InputsPSNR↑SSIM↑LPIPS↓tOF↓tLP×100↓
6, highly non-uniform31.74810.97880.02861.63390.1534
6, perturbed31.87380.98180.02961.57560.1443
8, perturbed32.05630.98340.02711.55310.1401

We also test CoSTA4D trained on DNA-Rendering on ZJU-MoCap without fine-tuning (Appendix A.8, Table 12).

Table R2. Cross-dataset generalization on ZJU-MoCap.

MethodPSNR↑SSIM↑LPIPS↓tOF↓tLP×100↓
GPS-Gaussian25.15320.97100.02561.24890.2756
4DGT22.32420.93520.05723.17830.6618
AnySplat26.27050.96670.03202.33120.4714
Depth Anything 324.94160.94480.05213.54130.4965
NoPoSplat24.59500.96410.03312.54010.5254
StreamSplat26.85320.96640.03602.34580.7291
IGS23.66720.95020.04662.98470.5888
HiReFF23.09480.96310.03301.90440.3319
Forge4D23.10480.95060.05431.73750.4764
PointSplat25.88220.96960.02761.20100.2622
Ours27.80020.97430.02311.04600.1467

CoSTA4D ranks first on all five metrics on ZJU-MoCap. Together, these results support generalization across camera counts, non-uniform layouts, and unseen captures.

W4: Ablations and Qualitative Analysis. Revised Sec. 4.5 jointly analyzes Table 4 and Figs. 5–7. Temporal Aggregation improves point-map geometry and facial/clothing details (Fig. 7). Camera Refinement reduces camera-estimation errors (Table 3, Fig. 5), and the refined cameras are used for depth back-projection (Sec. 3.3). New Fig. 6 shows reduced clothing-boundary artifacts and appearance fluctuations with Gaussian Propagation. Table 4 supports their complementary roles: temporal aggregation and camera refinement improve geometric inference, while Gaussian propagation improves temporal consistency with similar spatial quality.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.