acceptodds
Withdrawn from ICLR 2027

CoHSPAD: Continuous Human Gaussian Reconstruction from Raw Monocular 1-bit SPAD Streams

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Monocular human reconstruction from conventional RGB video is limited by finite frame rates and exposure time. Rapid motion causes severe blur, degrading the image quality needed for reliable pose estimation and appearance recovery. SPAD cameras offer an alternative: binary photon-arrival observations at very high temporal resolution, which remain informative under low light and rapid motion. However, a single 1-bit frame is extremely sparse, and independent per-frame optimization of SMPL parameters and 3D Gaussians is easily destabilized by noisy photon gradients. We present CoHSPAD, a monocular framework for 3D Gaussian reconstruction of dynamic humans and static scenes with continuous human motion recovered directly from raw SPAD streams. Built on a differentiable SPAD imaging model, we supervise rendering directly in the 1-bit raw sensor domain. Instead of independent per-frame poses, we model motion as a continuous SMPL trajectory with sparse temporal knots that share supervision across the stream and stabilize pose gradients. We introduce a Multi-Scale Temporal Fusion Network (MTF-Net) to aggregate long and short streams per-timestamp, providing a more stable signal than a single binary frame while avoiding blur from naive temporal averaging. Experiments on our newly collected CoH-Synthesis and CoH-Real show that CoHSPAD remains robust and temporally consistent under inaccurate initial SMPL poses and rapid motion.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
SPAD Imaging, Human Reconstruction, 3D Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YKvNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoHSPAD reconstructs a dynamic human and static scene as 3D Gaussians from a raw monocular 1-bit SPAD stream. A differentiable Bayer/Poisson/Bernoulli imaging model supervises Gaussian rendering directly with binary photon observations. Instead of independent per-timestamp poses, SMPL motion is represented by sparse B-spline controls, a shared pose residual MLP, and post-LBS Gaussian deformation, allowing observations across time to constrain one continuous trajectory. MTF-Net fuses ordered long- and short-term photon contexts to produce target-aligned human supervision without fixed-window averaging, while SPAD-aware densification separately handles noisy human and static-scene gradients. The authors introduce CoH-Synthesis and CoH-Real and compare against adapted HUGS and MAD-Avatar, with Restormer used for pseudo-RGB preprocessing. CoHSPAD reports 30.646 average human PSNR, improves held-out PSNR from 26.542 to 28.795, and reduces average SMPL MPJPE from 30.75 to 27.38.

Critical strengths
  1. Underexplored sensing gap and physically grounded raw-domain supervision. RGB video suffers motion blur, while individual SPAD frames are sparse; direct Bayer/Poisson/Bernoulli supervision is physically aligned with the sensor.

  2. Continuous SMPL trajectory stabilizes sparse timestamp supervision. Shared B-spline controls, pose residuals, and post-LBS deformation let photon observations across time jointly constrain motion.

  3. A complete system evaluated beyond synthetic fitting. The paper combines human-scene Gaussians, MTF-Net, SPAD-aware densification, real captures, held-out timestamps, pose refinement, and robustness analysis.

Critical weaknesses
  1. The experimental validation is too self-contained to establish the claimed robustness and superiority. The method is evaluated only on the authors' newly introduced CoH-Synthesis and CoH-Real datasets, with five synthetic and five real sequences in total. The synthetic observations are generated using the same Poisson/Bernoulli image-formation model that the optimizer assumes, which introduces a potential model-match bias. The real captures have no independent clean RGB ground truth; the real-data silhouette evaluation references Cutie masks that are also used for supervision, so the authors themselves note that it measures consistency with training masks rather than independent reconstruction accuracy. In addition, the quantitative comparison is limited mainly to adapted HUGS and MAD-Avatar variants, without direct photon-, spike-, event-, or continuous-motion reconstruction baselines or evaluation on independent public benchmarks. Consequently, the reported experiments do not sufficiently demonstrate generalization beyond the authors' own data protocol.

  2. Novel-view and geometric reconstruction quality are not evaluated, despite the paper's 3D reconstruction and view-synthesis motivation. The introduction frames monocular dynamic human reconstruction as supporting novel-view synthesis and human animation, but the quantitative image metrics are computed at the fixed input viewpoint. The paper reports PSNR, SSIM, and LPIPS against temporally aligned references, plus SMPL pose errors, but it does not evaluate held-out novel camera views, cross-view consistency, depth accuracy, or independent geometry metrics such as point-to-surface or Chamfer errors. Since the input is monocular and the method relies on SMPL, masks, and Gaussian regularization, high fixed-view rendering quality does not by itself establish a correct 3D human-scene decomposition or a view-generalizable representation. Novel-view and geometric evaluations are therefore essential but missing.

  3. The method depends heavily on external priors, while the marginal gains of some proposed components are limited. The claim of reconstruction directly from raw SPAD streams should be interpreted with caution: the pipeline requires GVHMR SMPL initialization, synthetic or Cutie human masks, known or externally estimated fixed camera parameters, human-centered crops, and sensor calibration. The initialization-robustness study perturbs joint rotations only up to five degrees, while the optimized trajectory still has substantial errors, including an average MPJPE of 27.38 and much larger errors on difficult scenes. Moreover, the most distinctive temporal-fusion component improves held-out PSNR by only 0.123 dB over continuous SMPL alone, and B-spline interpolation yields only small gains over linear or SLERP alternatives. Without comparisons to simpler trajectory regularization, alternative continuous-motion parameterizations, and other multi-scale temporal-fusion designs, the advantages of the proposed components over less specialized alternatives are not adequately isolated.

Questions for authors
  1. How would CoHSPAD perform on independent public SPAD datasets, or with synthetic observations generated by a simulator whose imaging assumptions differ from the paper’s Poisson/Bernoulli model?

  2. Can the authors provide held-out novel-view rendering, cross-view consistency, depth, Chamfer, or point-to-surface evaluations to substantiate the 3D reconstruction claim beyond fixed-view rendering?

  3. How sensitive is the method to larger failures in GVHMR initialization, Cutie masks, camera intrinsics, fixed camera pose, and SPAD calibration?

  4. Since MTF-Net adds only 0.123 dB over continuous SMPL alone in the core ablation, how does it compare with fixed-window accumulation, temporal smoothing, trajectory regularization, and the multi-scale design of USP-Gaussian under matched settings?

  5. For CoH-Real, can the authors evaluate against independent masks, multi-view references, depth sensors, or manual annotations rather than Cutie masks already used for supervision?

Reviewer_b1sEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses SPAD to do human reconstruction. The authors model motion as a continuous SMPL trajectory, and fuse sparse photon-arrival observations from different timestamps to optimize reconstruction. To achieve it, they propose a multi-scale temporal fusion network, and designed the loss based on bernoulli reconstruction loss, human opacity, and pose controls. They evaluate their method on synthetic (CoH-Synthesis) and real (CoH-Real) SPAD data. The supplementary video shows the method improves background blur compared with baseline HUGS. The authors study the ablations on continuous trajectory, temporal fusion, and reconstruction refinements.

Critical strengths

This paper leveraged SPAD Streams to study the human reconstruction problems, It is a technically solid way to address the issues of traditinoal rgb based methods, such as motion blur.

To address the sparse information of single SPAD Stream, the authors don't directly merge multi frames, which leads to accumulated errors. Instead, they propose a line of solutions, including continuous articulated representation, multi-scale fusion network, mtf-net optimization, and bernoulli optimization for SPAD.

The method gets clear gains in experiments, and the authors recorded real video to further valid the performance of theri method.

Critical weaknesses

The gaussian optimization module for SPAD already exists in [1].

The temporal fusion module is not new in reconstruction. Also, as showed in table 2, the gain from MTF-Net is minor.

The 1-bit frame photon-arrival solution also exists in [2].

[1] PhotonSplat: 3D Scene Reconstruction and Colorization from SPAD Sensors. ICCP 2025.

[2] bit2bit: 1-bit quanta video reconstruction via self-supervised photon prediction. NeurIPS 2024.

Questions for authors

What happens if there is occlusion? Does the method support moving camera?

Reviewer_b7d5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper proposes method for 4D human Gaussian reconstruction from SPAD sensor data. Method combines multiple ideas from different papers:

  • Human/scene decomposition from HUGS
  • SMPL pose interpolation via splines from MAD-Avatar adapted to full scene motion trajectory
  • Direct gaussian supervision with SPAD raw data via differentiable photon simulation from PhotonSplat

Then author introduce some novel techniques like MTF-Net and SPAD-aware densification.

Method is compared with baselines on on CoH-Synthis quantitatively and qualitatively on CoH-Real

Critical strengths

Authors produce comprehensive framework for SPAD-based Human+static background reconstruction and demonstrate a good reconstruction perfomance.

Critical weaknesses
  • Quality improvements from key contributions of the paper are not conclusively demonstrated
    • According to table 2, MTF-Net improves PSNR only by 0.1 and 0.2 dB compared to respective baselines without it, with run-to-run variance not shown.
    • SPAD-aware densification for scenes ablated only qualitatively on one scene. Densification for human gaussians is not ablated.
    • Second-best method, MAD-Avatar with with SPAD-reconstructed RGB uses general purpose Restormer denoiser instead of specialized ones like QUIVER, despite authors being aware of latter
Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.