acceptodds
Withdrawn from ICLR 2027

VideoPort: Streaming Photorealistic Human Telepresence from a Single View via Real-Time Video Diffusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Human telepresence aims to enable remote viewers experience a live performance from freely chosen viewpoints, however, achieving this from a single RGB camera requires inferring surfaces and appearance that the camera cannot observe. Exist- ing approaches face a fidelity–realism tradeoff. Feedforward 3D reconstruction is geometrically consistent, but often produces coarse appearance. Video diffu- sion synthesizes realistic detail, but when conditioned on its own history, drifts from the person being captured. We present VideoPort, a two-stage framework fixes geometry via feedforward 3D reconstruction, then injects realistic detail us- ing causal video diffusion refinement. Reconstructed geometry guides novel-view synthesis, while surface-aligned observations provide additional appearance con- ditioning during training. To address error accumulation through generated his- tory, we propose deep self-rollout supervision (DSR) with paired performance targets to train the refiner to recover from its own imperfect predictions. As a result, VideoPort preserves performance-specific appearance throughout minutes- long sequences and achieves state-of-the-art reconstruction quality on 3 bench- marks. We further deploy it as a live system that performs continuous, real-time novel-view synthesis from a single consumer RGB camera.

Primary area:
generative models
Keywords:
Human telepresence, video diffusion models, real-time generation, single-view synthesis, 3D Human Avatar

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CGYTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VideoPort, a two-stage method for monocular video to streamable auto-regressive generated video, benefits human reconstruction and telepresence. The first stage is 3D human reconstruction. The proposed method recieves a monocular RGB video input, use DINO V3 to extract feature, and trained a reconstruction transformer to predict 3D Gaussian representation of the person. In the second stage, a video to video refine model recieve the coarse renderings from the 3D Gaussian representation and refine it with a causal wan2.2 model fine-tuned with LoRA. The results show that the proposed method achieves better results than exisiting works.

Critical strengths
  1. The overall presentation is clear. The paper features a detailed supplementary PDF and a video for results demonstration.

  2. The idea of reconstruction with feed-forward model and refine with video model is reasonable. Motivation is clear.

  3. Results are sufficient and convincing.

Critical weaknesses
  1. I think the main problem of this paper is that the method is not new. People has been doing reconstruction with feed-forward model and refining with video model for a while, including MVSplat 360, and some camera control works like TrajectoryCrafter. The novelty or contribution is questionable.

  2. It would be nice to not have the video refinement stage, and the feed-forward model can directly predicts clear 3D human representation, like LHM and LHM++. I wonder what is the performance of stage I alone, and with if we replace stage I with LHM++?

  3. The visual results are still blurry, especially for the back of the person. I would expect the video model can fix that. Meanwhile it will be good to enlarge some parts of the image for a clearer visualization.

[1] Chen, Yuedong, et al. "Mvsplat360: Feed-forward 360 scene synthesis from sparse views." Advances in Neural Information Processing Systems 37 (2024): 107064-107086.

[2] Yu, Mark, et al. "Trajectorycrafter: Redirecting camera trajectory for monocular videos via diffusion models." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

Questions for authors
  1. Given that Wan 2.2 14B is a MoE model, what issues you have to resolve to make it auto-regressive? I would expect using models like Wan 2.2 5B or Wan 2.1 14B will get similar performance with faster speed.
Reviewer_CN1tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

VideoPort introduces a real-time digital-avatar framework, leveraging only a monocular camera. VideoPort is build around two components: a ff-3d reconstruction, which is geometrically consistence but coarse, and a video diffusion model, which produces the details.

Critical strengths

Real-time human avatar generation is a highly relevant. Using a single off-the-shelf monocular camera is highly applicable, especially when contrasted with many of the related works, which require more involved pre-processing stages. Building an end-to-end streaming system is technically challenging.

Critical weaknesses

My major concern for the method is quality - in the supplementary video and in the qualitative results (Figure 4, Figure 9, Figure 10) VideoPort results look significantly more blurry than competitors, i.e. IDOL. This also to some degree explains why the method has such an unusual gap PSNR and LPIPS in Table 1, i.e. does not perform competitively on LPIPS but highly competitive on PSNR. The bad results of IDOL, which in the supplementary video looks much sharper, also indicate that the current evaluation might value “pixel-alignment” over texture, which is why VideoPort can beat IDOL on LPIPS in Table 1. The poor video quality is also measured by FVD. I wonder if DSR is too weak, i.e. in Eq 9 only an L2 loss is utilized, no distribution matching or even feature loss. Could a perceptual loss be used in either Eq 5 or Eq 9?

The method currently seems to be relying on a fixed (?) distance between camera and person - what happens if the person steps towards the camera or away from it? How brittle is this parameter? This parameter should have been ablated.

I also suggest to include a test with held-out frames, i.e. as has been done in IDOL (on THuman2.1) and 4DAnyone.

Questions for authors

Why Dinov3 and not Sapiens2? As the focus is on Humans, would a human-centric model not produce better features? Also, Sapiens2 already has a depth-estimation and a (2d) human key point estimation head which could have been leveraged.

Can the authors comment on what exact protocol they used for 4DAnyone and how they obtained the multi-view gt? How would VideoPort score under 4DAnyone’s protocol?

For the In-the-wild samples (Fig 10) - how are intrinsic and extrinsic camera params, and distance to camera calculated?

Reviewer_NsxgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a 4D reconstruction model, specifically for human body. The task setting is limited to monocular input and the method contains coarse reconstruction and video-gen for refinement. It is compared with some human-body reconstruction baselines in the experiments with qualitative results.

Critical strengths

The illustration of the method is easy to follow.

Critical weaknesses
  1. In the field of 4d reconstrution along with refinement, there are a lot of widely-used general-purpose methods, such as CUT3R, StreamVGGT, and they are not restricted to monocular input. The novelty of a human-specific method would not be adequate.
  2. Modest performance improvement. VideoPort does not surpass the baselines on majority of the metrics in Table 1, neither in efficiency. Hence, the effectiveness of the method lacks a strong proof.
  3. Though in Table 2 VideoPort is the best, the evaluation is based on an intermediate output of coarse reconstruction. The method is still not supported on the task of the paper's main claim.
  4. Comparison unfairness. It is unclear whether the baselines are retrained or fine-tuned on the same training data, or evaluated with their public checkpoints. There is a concern that VideoPort benefits from target-domain training data, which fails to isolate the contribution of this method.
Questions for authors
  1. As language models are more and more capable of reconstruction, why should people choose to use this model? If livestream is a claim, the 7 RTX6000 GPU workstation does not looks like a portable device. In fact, even general-purpose reconstruction models have to compete with lauguage models.

  2. Please specify the training & finetuning datasets for your method and baselines in experiments, and whether in different benchmarks your method received any adaptation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.