acceptodds
Under review as a conference paper at ICLR 2027

Shot3R: Decoupled Multi-Shot Reasoning for Streaming 4D Human-Scene Reconstruction

10 views

Abstract

Multi-shot video splices shots taken at different viewpoints over a continuous timeline. A shot change jumps the viewpoint, breaking identities and spatial reference at the shot change. Reconstructing people and scene from such video requires spatial and identity consistency across the changes. Most existing methods resolve cross-shot ambiguities by optimizing all shots and the scene jointly, once the complete sequence has been reconstructed. Unlike such offline methods, streaming reconstruction can only infer such relations from current and past observations, with no global information and only a history from the previous viewpoint. To address this problem, we present Shot3R, a streaming framework for 4D reconstruction of multiple people and scene from monocular video. At each shot change, Shot3R infers the transform between consecutive shots in latent space, from the boundary frame's features and the accumulated history, placing the new shot in the shared world frame. Identities are associated across the shot change. The transform is applied to the whole new shot, freeing it from the influence of the previous shot. On extensive multi-shot data, Shot3R delivers marked improvements in human reconstruction, camera pose estimation, and cross-shot identity consistency, largest under large viewpoint changes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
streaming 4D reconstruction, multi-shot monocular video, multi-person reconstruction, human–scene reconstruction, cross-shot identity association, recurrent state propagation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UYJXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Shot3R addresses causal human-scene reconstruction across viewpoint cuts. At each boundary, a history-conditioned pass estimates the relation to the preceding shot, while a reset pass reconstructs the incoming shot independently. Learned relation tokens refine the camera estimate; geometric identity association and a shared translation correction integrate people, cameras, and scene points. Experiments on multi-shot streams constructed from EgoBody, EgoHumans, and Harmony4D evaluate reconstruction, camera placement, identity continuity, and state propagation. The principal contribution is the separation of history used for registration from history used for subsequent reconstruction.

Critical strengths
  1. A well-motivated separation supported by a direct control. The two-pass design addresses a concrete tension: retaining state can aid cross-shot alignment while corrupting reconstruction after a large viewpoint change. Appendix D.2 holds the pretrained checkpoint and post-cut inputs fixed with Shot3R modules disabled. At 16 offsets, resetting reduces translation drift from 1.124 to 0.026 m and rotation drift from 14.578 to 0.200 degrees (Table 13), with paired capture-level intervals excluding zero. This supports the central design choice more directly than aggregate performance alone.

  2. Meaningful improvements over the same reconstruction backbone. Table 1 reports EgoBody W-MPJPE decreasing from 321.1 to 258.8 mm and IDF1 increasing from 0.870 to 0.985; EgoHumans W-MPJPE decreases from 1005.1 to 866.8 mm and IDF1 increases from 0.471 to 0.571. The registration controls in Tables 3 and 12 are informative: simple human-joint fitting is competitive in human error, whereas Shot3R also improves camera and identity point estimates. The contribution is therefore best understood as coherent integration of alignment and identity handling.

Critical weaknesses
  1. The headline metrics only partly establish shared-world consistency. Appendix A.2 uses separate per-identity similarity fits for W/WA on EgoHumans and Harmony4D, which can absorb relative placement errors between people. Camera ATE measures aligned camera centers and does not by itself establish correct camera orientation or metric scale. The shared-world diagnostic in Table 8 is valuable and directly addresses this issue, but its human-support sets differ from Human3R (Table 9), and residual errors remain substantial: Shot3R reports 2.69 m root error and 1.49 m pair-vector error. Moreover, EgoHumans Coverage falls from 0.639 to 0.616 in Table 1. A comparison on common supported observations is needed to establish the extent of the joint geometric improvement; the current human/camera metrics also do not quantify dense-scene accuracy.

  2. Validation of repeated streaming transitions is limited. The primary evaluation uses two-shot clips, and Appendix B.2's first-threshold-crossing wrapper does not measure repeated false resets. The four-clip three-shot test improves human error and identity but increases camera ATE from 1.618 to 1.700 m (Table 11). The absence of explicit match rejection also makes association vulnerable when visible identities change. Appendix F acknowledges these limits, which I appreciate. They constrain the demonstrated setting to short streams with temporally adjacent cuts and continuing human visibility, rather than establishing robustness to arbitrary edited video.

Questions for authors
  1. Can the Table 8 shared-world metrics be recomputed on the intersection of supported captures and person observations, using the same single-alignment protocol for every method? Please retain coverage information so that improved geometry can be distinguished from changes in which people are evaluated.

  2. Appendix D.1 reports W-MPJPE difference intervals between human-joint fitting and Shot3R that include zero on both datasets. Under matched boundary inputs and identity/refinement settings, what improvement specifically depends on learned relation tokens? Existing ablations or paired camera/identity results would help establish their incremental value.

  3. For the existing three-shot clips, how do detector events, associations, and camera error evolve at each boundary? This limited diagnostic would help explain the ATE regression and determine whether the abstract and conclusion should further qualify the streaming claim.

6 comments
AuthorsOpenReview ↗

Response to Reviewer UYJX [Part 1/4]

W1. The headline metrics only partly establish shared-world consistency. Appendix A.2 uses separate per-identity similarity fits for W/WA on EgoHumans and Harmony4D, which can absorb relative placement errors between people. Camera ATE measures aligned camera centers and does not by itself establish correct camera orientation or metric scale. The shared-world diagnostic in Table 8 is valuable and directly addresses this issue, but its human-support sets differ from Human3R (Table 9), and residual errors remain substantial: Shot3R reports 2.69 m root error and 1.49 m pair-vector error. Moreover, EgoHumans Coverage falls from 0.639 to 0.616 in Table 1. A comparison on common supported observations is needed to establish the extent of the joint geometric improvement; the current human/camera metrics also do not quantify dense-scene accuracy.

Q1. Can the Table 8 shared-world metrics be recomputed on the intersection of supported captures and person observations, using the same single-alignment protocol for every method? Please retain coverage information so that improved geometry can be distinguished from changes in which people are evaluated.

A1. We thank you for this helpful suggestion. To address it we recomputed the shared-world metrics with a consistent evaluation protocol across all methods. Each method fits one initial Sim(3) from the same two-frame reference, and that single transform is applied to all people, cameras and shots of the clip, with no fit per identity and none per later shot. Post-cut root errors use common person-frame observations, pair-vector errors use common valid person pairs, and camera errors are taken at the first post-cut frame, averaged equally over captures. Geometric errors are therefore evaluated on the intersection of supported observations, taken per metric, so the number of captures behind each column differs; coverage is reported separately for each method, because it reflects that method's own prediction support rather than the common set.

SettingRoot ↓ mPair vector ↓ mCamera position ↓ mCamera rotation ↓ °Coverage ↑ %
Human3R2.6721.5355.03333.8258.99
Reset4.2703.9969.647100.6156.53
Alignment3.3181.3325.18623.1156.53
Shot3R2.6421.3324.58723.1156.53

Magnitude and significance. Against Human3R the first-post-cut rotation difference has a paired 95% capture-bootstrap interval of [−16.68∘,−4.36∘][-16.68^{\circ},-4.36^{\circ}], excluding zero. Root error is essentially unchanged (2.672 → 2.642 m), while the pair-vector error decreases from 1.535 to 1.332 m and the camera rotation from 33.82° to 23.11°. The two stages also separate in the table. From alignment alone to Shot3R, root error falls from 3.318 to 2.642 m on [−1.202,−0.299][-1.202,-0.299] and camera position from 5.186 to 4.587 m on [−1.028,−0.211][-1.028,-0.211], while the pair vector and the camera orientation are identical in the two rows — as expected, since a shared translation satisfies (pi+Δ)−(pj+Δ)=pi−pj(p_i+\Delta)-(p_j+\Delta)=p_i-p_j and moves positions without touching relative configuration. Reset produces the largest errors across the four geometric metrics.

Coverage. Coverage is retained as requested. It is computed on the captures that admit a single initial fit, which is why these values differ from the main table. On EgoHumans it is 58.99% for Human3R and 56.53% for Shot3R. We additionally report common-support results on EgoBody, where both methods achieve identical coverage (98.06%). Shot3R reduces W/WA-MPJPE from 354.08/249.02 to 260.31/203.50 mm on [−153.50,−41.47][-153.50,-41.47] and [−71.29,−22.99][-71.29,-22.99].

Relation to the submitted table. In the submitted table the supports differ by metric — root 26 captures for Human3R against 25 for the other settings, pair vector 20 against 19, and 27 for the camera columns in all methods. The values above are recomputed on the common support of each metric rather than re-reported from that table. A post-hoc Sim(3) alignment does not establish the metric-scale accuracy of the original predictions, and the quantitative claims of the paper concern human and camera geometry and identity consistency; dense-scene accuracy is not evaluated against an independent reference.

AuthorsOpenReview ↗

Response to Reviewer UYJX [Part 2/4]

Q2. Appendix D.1 reports W-MPJPE difference intervals between human-joint fitting and Shot3R that include zero on both datasets. Under matched boundary inputs and identity/refinement settings, what improvement specifically depends on learned relation tokens? Existing ablations or paired camera/identity results would help establish their incremental value.

A2. We agree that the comparison in Appendix D.1 does not isolate the contribution of the relation estimator, as identity association and refinement also differ between the methods. We therefore reran it with everything except the estimator held fixed — the same clean-reset predictions, known boundaries, predicted correspondences from the full model, identity inheritance and the shared translation refinement — replacing only the relation estimate by Kabsch, which fits a unit-scale rigid transform to predicted joints without ground-truth correspondences. Because the correspondences are taken from the full model, this is a conditional module ablation rather than an independent geometric baseline. It isolates the estimator given Shot3R's own correspondences, and we state the comparison on those terms. Camera figures use each dataset's main protocol, except Harmony4D, which uses the multi-cut diagnostic clips.

DatasetCamera error ↓ m (Kabsch → Shot3R)
EgoBody0.0186 → 0.0165
EgoHumans1.2843 → 1.5479
Harmony4D3.0231 → 1.9570

Estimator results. Learned relation estimation improves camera alignment on EgoBody and Harmony4D. The paired interval for the main ATE is [−0.0042,−0.0002][-0.0042,-0.0002] m, with a supplementary SE(3) ATE moving from 0.3059 to 0.2113 m on [−0.1685,−0.0211][-0.1685,-0.0211], and on Harmony4D 3.0231 → 1.9570 m (−35.3%) on [−1.8522,−0.5440][-1.8522,-0.5440], excluding zero. Kabsch fits the predicted joints directly, whereas the learned path estimates the camera relation from the observation and the accumulated history; the two do not optimise the same quantity.

Token ablation. Within the learned estimator the three relation tokens are assessed separately on a larger set of 1,000 four-frame confirmation events.

SettingRel. rot. ↓ °Rel. trans. ↓ mRoot res. ↓ mmJoint res. ↓ mm
Shot3R5.1340.226191.63209.04
w/o semantic7.6820.291221.65239.49
w/o camera5.2280.274198.71212.69
w/o temporal5.3260.265184.80211.51

Removing any single token raises at least three of the four measures, and the relative translation error rises by 28.8%, 21.2% and 17.3% for the semantic, camera and temporal token.

Paper changes. The following places are revised, and every addition or edit is set in blue. Appendix D.3 adds the relation-token ablation reported above (Table 19, Q2).

AuthorsOpenReview ↗

Response to Reviewer UYJX [Part 3/4]

W2. Validation of repeated streaming transitions is limited. The primary evaluation uses two-shot clips, and Appendix B.2's first-threshold-crossing wrapper does not measure repeated false resets. The four-clip three-shot test improves human error and identity but increases camera ATE from 1.618 to 1.700 m (Table 11). The absence of explicit match rejection also makes association vulnerable when visible identities change. Appendix F acknowledges these limits, which I appreciate. They constrain the demonstrated setting to short streams with temporally adjacent cuts and continuing human visibility, rather than establishing robustness to arbitrary edited video.

Q3. For the existing three-shot clips, how do detector events, associations, and camera error evolve at each boundary? This limited diagnostic would help explain the ATE regression and determine whether the abstract and conclusion should further qualify the streaming claim.

A3. To evaluate repeated detection beyond the first-threshold-crossing wrapper, we ran the detector over all frame pairs. All eight annotated cuts were detected, with no additional positive detections among the 588 non-cut frame pairs. Reconstruction was evaluated separately, using ground-truth cut locations at frames 50 and 100.

ClipCut frameRotation error ↓ °Matches correct / wrong / unassessableRoot: shared translation off → on ↓ mCamera position: shared translation off → on ↓ m
karate25014.012 / 0 / 13.561 → 1.8252.435 → 0.876
karate2100171.610 / 2 / 04.125 → 2.9882.556 → 3.616
karate35015.222 / 0 / 03.585 → 1.9482.425 → 0.851
karate3100139.750 / 2 / 06.663 → 4.0683.372 → 5.098
mma3508.111 / 0 / 33.258 → 0.9241.711 → 0.721
mma3100158.850 / 1 / 34.836 → 2.8215.861 → 8.717
mma4508.202 / 0 / 42.120 → 0.8323.191 → 0.464
mma410065.310 / 0 / 62.882 → 0.6226.382 → 5.555

Reading the boundary table. The last two columns compare the same boundary with the shared translation removed locally and restored, with the upstream predictions and the initial alignment fixed. The first boundary shows relatively low rotation errors (8.11–15.22°) and correct associations for all assessable matches; the second boundary exhibits substantially larger rotation errors (65.31–171.61°), accompanied by incorrect or unassessable associations. These observations suggest that errors can accumulate across repeated transitions. With only eight boundaries we cannot distinguish accumulated alignment error from errors in the current relation estimation and identity association; the root error decreases at all eight boundaries and camera position at five.

Reviewer_rSWiOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper claims that

  • it defines a new problem and;
  • introduces the first framework for it.

Problem defined

  • Input:
    • a monocular video,
    • but made of multiple shots taken from different viewpoints
    • in a streaming way.
  • Output: reconstructed humans and scene like Human3R.

It actually consists of

  • Frozen Base model: Human3R.

  • Three added components, which could be considered to be

    • compared with the current SOTA in the introduction section,
    • designed,
    • trained and,
    • evaluated separately.

    The three parts are:

    1. Detection of the cuts (changes of the actual camera shot/view).
    2. Extrinsic estimation during "1."
    3. Cross-shot identity association. However, the paper describes it as a new full pipeline.

The paper is neither convincing nor sound because:

  1. The problem setting is not justified. in terms of "why online", "potential applications" and "Value of the problem".
  2. The novelty and contribution of each technical component is not correctly introduced with the actual SOTA, and is unclear and not well-evaluated
  3. The evaluation setting does not support the contribution. See "weakness" for details.
Critical strengths

This pipeline is clearly described.

Critical weaknesses

1. The "new" problem is not clear

The new problem is not clearly introduced, so that its significance and the logic in the introduction section are vague.

I think: A new problem is not acceptable just because it is well-formed by construction. Otherwise, settings could be combined without end, many of them without practical value. When the main claim of a paper is a new problem, I weigh the value of that problem as much as the method.

The problem definition itself is unclear. In detail, the combination is monocular × streaming × from multi-shot, but its properties are left implicit. In particular, whether the camera may move or zoom within a shot is never defined. The only related statement is that each EgoHumans shot is recorded by a static camera (App. D.2).

Beyond the definition, the motivation is not clear either. This shows in three aspects:

  • "Why online" is not clear

    • The paper's description of why online processing is necessary is factually wrong.

      • The only motivation is one sentence (p2 L62–66): "Offline multi-shot methods resolve cross-shot relations over the whole sequence, so their computational cost grows with the length of the video."
      • This is equally true of streaming methods, so it is not an argument for online processing. Any method that processes every frame has a total cost that grows at least linearly with the video length.
      • What would matter is superlinear growth, e.g., global attention over all frames, whose cost grows quadratically with the number of frames. A fixed-window offline method or an online method with a bounded delay keeps the per-frame cost constant, so its total cost is linear, the same order as strict streaming. It can still use the frames after a cut (see below).
    • Apart from computational cost, the paper never states what online processing is for.

      • If the reason is latency, real-time performance must be shown. It is neither claimed nor reported.
      • Table 15 of the paper gives 0.894 FPS for Shot3R, but its timing includes process launch and model loading. In the same table Human3R reaches only 1.246 FPS, whereas the Human3R paper reports 15 FPS on an RTX 4090.

      • The abstract mentions streaming only as a restriction on the available observations.
    • The strictly online setting discards information that even a short delay or a fixed window would provide.

      • The frames after a cut provide many more observations of the new shot than the boundary frame alone. With a moving camera, even the simplest structure-from-motion, SLAM-based methods, or learned multi-view models can exploit their parallax. With a static camera, the people are still observed over many frames, which helps both the alignment and the identity association.
      • On the new-shot side, Shot3R uses only the boundary frame, and the resulting transform stays fixed for the rest of the shot.
      • The paper does not measure how much accuracy this restriction loses, which concerns me.

I personally think that, maybe, the method appears to be online only because its base model is online, since online processing is never tied to a need of the problem.

  • The potential applications do not support the setting

    • Whatever the application, online processing matters only if results are needed while the video is being captured, i.e., in real time.
    • The paper does not demonstrate real-time operation (see above). Hence none of the applications it names (films, broadcast sports, edited online content; p2 L56–58) benefits from the online setting. Films and edited online content are recorded and can simply be processed offline, and live broadcast would need exactly the real-time operation that is not shown.
  • The value of the problem is not demonstrated

    • Even if real-time operation were shown, the scenarios named in the paper differ from the setting, which assumes hard cuts over a continuous timeline (p3 L150–151) and a single available view (App. A.1):

      ScenarioGap to the setting
      Films, edited online contenttime ellipses, shot/reverse-shot, scene changes, moving cameras
      Broadcast sportsreplays and slow motion; cameras that pan and zoom
      Live broadcastonly partly continuous, since replays are inserted; the director has all camera feeds
      Human3R's own online applications (humanoid policy learning, autonomous navigation, human-robot interaction)continuous camera motion, no hard cuts
    • The paper should therefore analyze and quantify how much real footage matches the setting, e.g., statistics of cut types, timeline continuity, and camera motion within shots, and ideally contribute a dataset of such footage.

Missing related work. Pavlakos et al., "The One Where They Reconstructed 3D Humans and Environments in TV Shows", ECCV 2022, is not discussed. It reconstructs humans, environments, and cameras across shot changes in real TV footage, and evaluates quantitatively on that footage.

2. The whole-pipeline framing is not valid

Framing the contribution as a whole pipeline invalidates the gap analysis in the introduction.

Taken apart, the method is the frozen Human3R plus three separable components: shot-change detection, extrinsic estimation at the cut, and cross-shot identity association, each an existing subproblem with its own literature. Arguing for them as one system breaks the introduction in four places.

  • The gap is argued at the wrong level. The introduction compares system classes (offline vs. streaming) and does not position each component against the state of the art of its subproblem (shot boundary detection, cross-view registration / relocalization, cross-shot re-identification); shot boundary detection, for example, is not discussed at all. Its claim that streaming models do not treat shot changes also conflicts with CUT3R, which accepts unordered photo collections, and with PHALP, which handles identity across shot changes online.
  • The dichotomy does not hold. Independent reconstruction leaves the shots related once external registration and association are attached to it. The paper's own registration control already matches Shot3R in W-MPJPE on EgoBody (Table 3).
  • The key insight is an insight only at the pipeline level. At the module level it reduces to resetting the state at a cut and estimating one extrinsic. Once the extrinsic is known, resetting the state at an abrupt change follows directly.
  • The components are weak. The detector is too naive to support a technical contribution, there is no evidence that the learned query exploits the history, and geometric cross-shot association already exists (the TV-show work above). The components are also independent, although the paper says they are "trained together" (Sec. 3.3).

3. The experimental setting does not support the contribution

The experimental setting inherits the flawed framing of the introduction. It shows that the whole system outperforms other systems, whereas the actual contribution requires module-level evidence.

  • Each module addresses a real problem that has standard evaluation and a state of the art.

    • Shot-change detection is shot boundary detection, evaluated by detection F1 on standard benchmarks.
    • Extrinsic estimation at the cut is relative pose estimation / relocalization between views with little overlap, evaluated by rotation and translation errors.
    • Cross-shot identity association is person re-identification and tracking association, evaluated by identity metrics such as IDF1.

    None of the modules is evaluated in this way against the state of the art of its problem. The detector is evaluated only on constructed frame pairs, the extrinsics only against two simple registration baselines (Table 3), and the association only inside the full system. Because the modules are small, the comparison should also cover classical methods, e.g., threshold-based shot detection, or feature matching with RANSAC for the extrinsics.

  • Wrong evaluation paradigm. The system comparisons and the in-pipeline ablations show what the modules add to Human3R. They do not show whether the modules are better than existing solutions to the same problems, or what they bring to other base methods.

  • The naive route is a side control, and an unfair one. It is not stated which identity association it uses or where its person correspondences come from.

  • Synthetic cuts only. All quantitative results use cuts made by switching between synchronized cameras. Real edited footage is shown only qualitatively, although the TV-show work above shows that quantitative evaluation on real footage is feasible.

Presenting a simple-and-effective module as a large pipeline therefore misdirects both the analysis and the experiments away from the actual contribution.

Suggestions.

Two directions could bring the work to the ICLR level:

  1. Analyze why existing data are insufficient, propose a data-generation method, release a new dataset, and provide analysis and baselines. Pavlakos et al., "Human Mesh Recovery from Multiple Shots" (CVPR 2022), already cited by the paper, follows this route: it builds pseudo-ground truth from movies and uses it to train other models.
  2. Keep the new problem, but justify its value, and analyze and compare against existing pipelines, including the naive combinations.
Questions for authors

I have not listed detailed questions about the method, because my concerns about the problem setting, introduction logic and experiments setting. If these could really be addressed, I will raise more specific questions about the method during the discussion.

24 comments
AuthorsOpenReview ↗

Response to Reviewer rSWi [Part 1/7]

Suggestions. Two directions could bring the work to the ICLR level: build and release a dataset of such footage, or keep the new problem, justify its value, and compare against existing pipelines, including the naive combinations.

A1. We thank you for this exhaustive review, which covers the problem definition, the setting, the motivation, the applications, the related work, the framing of the contribution, the evaluation protocol and the data, and for these two suggestions on how to bring the work to the ICLR level. We have tried to act on both.

Suggestion 1, the dataset. Footage of this kind is genuinely hard to obtain as a training set, and we intend to build such a resource in future work. Time in this cycle was limited, so we prepared a first version ourselves. It consists of 2,500 frames of multi-person basketball recorded by 75 synchronized cameras, with calibration for every view and fitted SMPL-X reference tracks for the people; the quantitative test uses a 100-frame segment of it, with frozen Shot3R and Human3R checkpoints on six two-shot and four five-shot schedules fixed before evaluation, one view per timestamp and no basketball training or checkpoint selection. The full 2,500-frame capture exceeds the size limit for the supplementary material, so we include a compressed 100-frame version with this submission, and we will release the complete capture together with the camera-ready version.

SchedulesCut-relative camera rotation ↓ ° (Human3R → Shot3R)Fixed-initial shared-world human MPJPE ↓ mm (Human3R → Shot3R)
Two-shot19.78 → 4.531282.73 → 1298.87
Five-shot45.84 → 10.202703.98 → 2174.14
Equal-weight mean of the two groups32.81 → 7.371993.36 → 1736.51

Basketball results. Cut-relative camera rotation error falls by about 77% in both groups; human error improves on the five-shot group and is slightly higher on the two-shot group. The shot schedules are constructed from synchronized views and the human reference is a fitted SMPL-X track.

Suggestion 2, the value of the problem. We keep the new problem and consider it worth solving, and the detailed answer follows.

AuthorsOpenReview ↗

Response to Reviewer rSWi [Part 2/7]

W1. The "new" problem is not clear … A new problem is not acceptable just because it is well-formed by construction … the combination is monocular × streaming × from multi-shot, but its properties are left implicit. In particular, whether the camera may move or zoom within a shot is never defined … "Why online" is not clear … The potential applications do not support the setting … The value of the problem is not demonstrated … Missing related work: Pavlakos et al., "The One Where They Reconstructed 3D Humans and Environments in TV Shows", ECCV 2022, is not discussed.

A2. Several points appear to rest on a reading of the paper that differs from what we intended, so we first restate the setting and the design and then answer the points one by one.

Core challenge. The key challenge comes from the conflicting roles of historical information. History is useful for estimating cross-shot relations, whereas directly propagating the recurrent reconstruction state across a transition introduces substantial errors. Shot3R addresses this by decoupling cross-shot alignment from within-shot recurrent reconstruction.

Setting. Inference sees only the observed prefix, never revises an emitted frame, and keeps bounded memory. The relevant case is incremental reconstruction from a single composed video stream, where observations arrive sequentially and the reconstruction is updated without future frames or revisiting previous outputs. Three cases share this condition. In broadcast sport, whoever reworks a match has the feed that was broadcast and not the camera bank behind it, yet wants per-shot world-frame multi-person reconstruction; in film and edited online content, a finished programme is assembled from several cameras and has to be put into one scene coordinate system; for archive material, only the edited programme survives, and the source cameras and their calibration are usually unavailable. What the setting covers is a change of viewpoint on a continuous timeline; edited material also changes the timeline itself, through replays, slow motion and temporal jumps, which is a different problem and outside it.

Data and motivation. Our synchronized multi-camera data is a construction device that supplies exact ground truth and an exact shot change. We agree that the cost of offline methods does not justify the online setting, since any method that processes every frame grows with the video; the revised introduction replaces that motivation with the applications, since the reason is what the user has rather than a complexity advantage.

The reset control. Under this constraint, abrupt shot transitions disrupt recurrent reconstruction, as our controlled reset experiment shows. With the checkpoint and post-cut inputs fixed, resetting the state lowers translation drift over the following 16 frames from 1.124 to 0.026 m and rotation from 14.578° to 0.200° (paired, one-sided Wilcoxon p=7.45×10−9p=7.45\times10^{-9}).

Within-shot camera motion. The formulation permits it — Ctw=GsCt(s)C_t^{w}=G_sC_t^{(s)} fixes the shot-to-world mapping while the camera pose changes at every frame, and the conversion holds for time-varying intrinsics. We tested limited perturbations, with digital zoom from 1.0 to 1.3× and synthetic pure rotation from 0 to 6° on 28 four-frame events, with both methods receiving the same transformed calibrated intrinsics in this controlled test.

PerturbationShot3R: none → applied, MPJPE ↓ mmHuman3R, MPJPE ↓ mmCamera rotation: Human3R → Shot3R ↓ °
Digital zoom535.03 → 533.93551.4922.79 → 15.48
Pure rotation532.00 → 774.07847.8331.14 → 21.24

What the perturbations show. Digital zoom leaves Shot3R's human error essentially unchanged. Pure rotation increases its own error as well, from 532.00 to 774.07 mm, but the perturbed human and camera errors remain below Human3R's. These are bounded perturbation tests, not a validation of general camera motion or optical zoom.

AuthorsOpenReview ↗

Response to Reviewer rSWi [Part 3/7]

Runtime. For a controlled comparison we re-timed the reconstruction stage alone with the same ViT-L/896 weights, FP32, identical 100-frame inputs and one A800.

ConfigurationReconstruction FPS
Human3R2.7680
Shot3R, no cut2.7362
Shot3R, one cut2.6630

Reading the runtime numbers. The additional shot-transition mechanism therefore costs about 1–4% in the reconstruction stage. The full pipeline reaches 1.49–1.56 FPS, with a first output at a median of 0.320 s, a mean per-frame output latency of 651.93 ms (P95 951.90) and 734.66 ms for cut frames (P95 833.50); we do not claim real-time performance. The 0.894 FPS in the complete-pipeline table is a measurement on an L20 that includes process launch and model loading, so it is not comparable to per-frame inference reported elsewhere. The 15 FPS you cite is Human3R's ViT-S/672 on an RTX 4090, whereas the checkpoint we build on and time here is ViT-L/896, for which the same paper reports about 5.1–5.4 FPS on that GPU. Model size, hardware and timing scope all differ, and we do not attribute the gap to a single one of them.

Lookahead. We quantified the trade-off for k=0k=0 and 22 future frames on both datasets and for k=5k=5 and 1010 on Harmony4D, with the two reconstruction paths and predicted correspondences fixed and identity not re-estimated.

DatasetLookahead kkObservation waitPost-cut root error ↓ mBoundary-frame camera rotation error ↓ °
Harmony4D002.106112.48
Harmony4D266.7 ms1.80748.94
Harmony4D5166.7 ms2.677—
Harmony4D10333.3 ms2.896—
EgoHumans005.54563.38
EgoHumans2100.0 ms4.63467.58

Lookahead results. A two-frame lookahead reduces root error on both datasets, although the orientation results are mixed; longer lookahead does not consistently improve reconstruction. This sweep is a non-causal diagnostic that grants access to future frames, not the default streaming setting, and the observation wait is kk divided by the frame rate, excluding computation.

Related work. The omission is genuine. The TV-show work aggregates a whole season into a 3D model of the environment and uses the recovered camera, scene structure and body scale to guide human pose and position across shots, with re-identification as one of its applications. Section 2 now cites and discusses it and distinguishes it from the already-cited Human Mesh Recovery from Multiple Shots, which uses the shots as complementary views of one moment.

Paper changes. The following places are revised, and every addition or edit is set in blue. Section 2 now states that CUT3R outputs neither human models nor identities, cites Pavlakos et al. (ECCV 2022) for reconstructing cameras and scene geometry from television footage and associating people across shot boundaries with geometric and anthropometric constraints, and cites PHALP as online tracking by 3D appearance, location and pose; the earlier taxonomy of multi-view versus tracking problems and the claim that cross-shot identity association relies on appearance are removed, and both references are added to the bibliography (W1, W2). Section 3.3 replaces "trained together" with an explicit list of what shares one training stage, with the detector trained separately and the assignment and shared translation noted as deterministic operations (W2). Appendix D.3 adds the history intervention reported above, where only the scene state and pose memory that the relation module reads are zeroed (Table 20, W2). The three appendix tables answer the shot-level questions of W1, namely the additional held-out basketball capture used for the two- and five-shot test (Appendix A.1, Table 5), bounded digital zoom and synthetic rotation applied inside a shot (Appendix C.2, Table 9), and the reconstruction stage timed on its own (Appendix E.1, Table 22).

Reviewer_7w9pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Shot3R, a streaming framework for 4D human-scene reconstruction from multi-shot monocular videos. The proposed method decouples cross-shot spatial alignment from recurrent reconstruction by resetting the model state at shot boundaries while using historical information to estimate inter-shot camera transformations. It further incorporates geometry-based identity association to maintain consistency across shots. Experiments on EgoBody, EgoHumans, and Harmony4D demonstrate improvements in camera pose estimation, human reconstruction, and identity consistency over existing methods.

Critical strengths
  • The paper is generally well organized, and the proposed framework is clearly motivated and presented.
  • The experiments demonstrate improvements over the Human3R baseline, particularly in camera pose estimation and identity consistency. The ablation studies provide useful evidence for the contributions of individual components.
Critical weaknesses
  • The problem is underconstrained. Without synchronized multi-view observations or reliable geometric correspondences, the relative camera transformations across shots cannot generally be determined uniquely. The proposed method relies heavily on learned priors to infer these transformations, but the paper does not sufficiently explain what information makes this inference reliable. Its effectiveness in ambiguous or unseen configurations remains questionable.
  • The practicalness of the problem setting is not well justified. Most real-world applications of 4D human-scene reconstruction involve continuous video streams or synchronized multi-camera recordings, rather than edited monocular videos with abrupt viewpoint changes. The paper does not convincingly establish why streaming reconstruction of multi-shot videos is an important or widely needed application.
  • The experimental validation is limited. Most evaluations involve only two shots, while real edited videos may contain numerous shot changes, occlusions, and long-term identity ambiguities.
Questions for authors
  • What are the primary real-world applications that require streaming 4D reconstruction from edited multi-shot monocular videos, rather than continuous or synchronized multi-camera recordings?
  • How does the method perform on longer real-world edited videos with many shot transitions, significant occlusions, and people disappearing and reappearing?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 7w9p [Part 1/3]

W1. The problem is underconstrained. Without synchronized multi-view observations or reliable geometric correspondences, the relative camera transformations across shots cannot generally be determined uniquely. The proposed method relies heavily on learned priors to infer these transformations, but the paper does not sufficiently explain what information makes this inference reliable. Its effectiveness in ambiguous or unseen configurations remains questionable.

A1. We thank you for this important comment. With monocular views alone, the relative camera pose between two shots cannot be determined uniquely, and Shot3R does not remove that ambiguity. It estimates the pose of the boundary frame twice, once from the state of the previous shot and once on the reset path in the new shot's own coordinates. Because both estimates describe the same frame, the initial mapping of the new shot into the shared world frame follows from them,

Gs0=Gs−1 C~b(s−1) (Cb(s))−1G_s^{0}=G_{s-1}\,\widetilde{C}_b^{(s-1)}\,(C_b^{(s)})^{-1}

without a cross-shot correspondence search, and its error is inherited from those two estimates. Identity association uses predicted body geometry, and its matches are what the shared translation applies afterwards to refine the final mapping GsG_s. At inference every timestamp is a single view; synchronized cameras only construct the evaluation inputs. What couples the two estimates is the state carried over from the previous shot and three relation tokens computed inside the new shot, which read its image and camera tokens and the cached memory. The experiments below measure the accuracy of this initial mapping.

Matched-training control. We trained a control under the same training configuration that disables the relation path, and evaluated the three models on the same 200 four-frame confirmation events, 160 of which contain a shot change. The errors below are computed on those 160 cut events, where the human support is common to all three models.

SettingRel. rot. ↓ °Rel. trans. ↓ mRoot res. ↓ mmJoint res. ↓ mm
Human3R66.592.082647.80717.75
No relation path49.051.4401170.261193.60
Shot3R7.910.259222.35242.33

Shot3R reaches 7.91° and 0.259 m here, and against that control the four errors drop by 83.9%, 82.0%, 81.0% and 79.7%, so the improvement comes from the relation path rather than from further training.

History intervention. On a separate, smaller sample of 20 boundary events from four captures, with the weights and the current input fixed, we zeroed only the scene state and pose memory that the relation module reads.

SettingRel. rot. ↓ °Rel. trans. ↓ mRoot res. ↓ mmJoint res. ↓ mm
Original history36.690.9101006.361030.52
History contents zeroed36.971.2381440.321450.60

On these 20 events, zeroing the history raises the translation error by 36% and the root and joint residuals by 43% and 41%, while rotation stays essentially the same. The three relation tokens are individually useful as well. Removing any single one of them degrades at least three of the four measures.

Whole clips. We also recomputed the shared-world metrics on the common support of 90 EgoHumans clips. Each method fits one initial Sim(3) from the same two-frame reference and applies it to every person, camera and shot, so that fitting individual people cannot absorb how people are placed relative to each other; root error uses common person frames, the pair vector common person pairs, and camera errors are taken at the first post-cut frame, averaged equally over captures.

SettingRoot ↓ mPair vec. ↓ mCam trans. ↓ mCam rot. ↓ °
Human3R2.6721.5355.03333.82
Reset4.2703.9969.647100.61
Alignment3.3181.3325.18623.11
Shot3R2.6421.3324.58723.11

The first-post-cut camera-rotation difference against Human3R has a paired 95% capture-bootstrap interval of [−16.68∘,−4.36∘][-16.68^{\circ},-4.36^{\circ}], which excludes zero, and the last two rows differ only where the shared translation acts.

Unseen capture and viewpoint change. On a capture held out from module training, relative rotation error falls from 97.82° for Human3R to 10.92°, over the 200 cut events of a 250-event held-out set. Across the four EgoHumans viewpoint-change strata, IDF1 exceeds the continuous-state reference everywhere and the geometric errors are about 30% lower at the largest change, where the cut is most abrupt.

AuthorsOpenReview ↗

Response to Reviewer 7w9p [Part 2/3]

W2. The practicalness of the problem setting is not well justified. Most real-world applications of 4D human-scene reconstruction involve continuous video streams or synchronized multi-camera recordings, rather than edited monocular videos with abrupt viewpoint changes. The paper does not convincingly establish why streaming reconstruction of multi-shot videos is an important or widely needed application.

Q1. What are the primary real-world applications that require streaming 4D reconstruction from edited multi-shot monocular video, rather than continuous or synchronized multi-camera recordings?

A2. W2 and Q1 concern the same point, and we answer them together.

The applications we have in mind share one input condition. The user has only the published, edited video, one camera at a time. In broadcast sport that user also needs the state while the stream arrives — whoever reworks a match sees the single feed that was broadcast and not the camera bank behind it, and wants per-shot world-frame multi-person reconstruction as the frames come in. In film and edited online content, a finished programme is assembled from several cameras and has to be put into one scene coordinate system for reconstruction or for visual effects. Archive material makes the point plainest. Only the edited programme survives, the source cameras and their calibration are frequently unavailable, so a multi-view method cannot be rerun on it at all.

The two alternatives. Neither alternative covers this user's case. For someone with only the broadcast feed, the source views a synchronized multi-camera method needs are not available. Continuous single-camera reconstruction does have that feed, but it assumes what the edit removes, namely an unbroken camera motion. With the checkpoint and the post-cut inputs fixed, resetting the state lowers translation drift over the following 16 frames from 1.124 to 0.026 m and rotation from 14.578° to 0.200° (paired, one-sided Wilcoxon p=7.45×10−9p=7.45\times10^{-9}), which isolates the effect of carrying the state across the cut.

What defines the setting is therefore what the model is allowed to see. Frames up to the present only, no revising of output already emitted, bounded memory. Our synchronized captures serve to construct controlled cuts and to supply reliable ground truth.

Paper changes. The following places are revised, and every addition or edit is set in blue. The second paragraph of the introduction is rewritten (W2/Q1). It no longer motivates streaming by the cost of offline methods, it states that a receiver holds only the transmitted single view and reconstructs incrementally from the observed prefix without revising emitted frames, it drops the claim that streaming models do not treat a shot change, and it closes on the requirement that the boundary step preserve both spatial and identity reference. Appendix D.3 adds the trained side control of this reply (Table 18, W1). Appendix C.5 adds the six-shot sequences (Table 14, W3/Q2, and Reviewer 3's W3).

AuthorsOpenReview ↗

Response to Reviewer 7w9p [Part 3/3]

W3. The experimental validation is limited. Most evaluations involve only two shots, while real edited videos may contain numerous shot changes, occlusions, and long-term identity ambiguities.

Q2. How does the method perform on longer real-world edited videos with many shot transitions, significant occlusions, and people disappearing and reappearing?

A3. W3 and Q2 concern the same evidence, and we answer them together.

The two-shot protocols (129 / 90 / 88 clips) have one cut per clip and exact ground truth, so no earlier cut can contribute accumulated error. That isolates the boundary step, but it cannot show how error develops over many cuts. We therefore evaluated 12 six-shot sequences with five cuts each, that is, 60 transitions. Seven of 300 frames and five of 150–289 frames, built by chaining shots taken from the same synchronized captures, which gives long chains of transitions on real imagery.

GroupnW ↓ mmWA ↓ mmATE ↓ mIDF1 ↑
EgoHumans / 30063200.60 → 2927.67853.92 → 927.685.3819 → 5.45800.2361 → 0.2619
EgoHumans / 210–25622269.36 → 2200.741001.23 → 758.552.3959 → 1.49910.3654 → 0.4919
Harmony4D / 30012972.83 → 1724.48987.57 → 1094.012.9097 → 3.18710.3666 → 0.5273
Harmony4D / 15012685.28 → 2556.37560.94 → 472.203.7216 → 2.30570.5642 → 0.5831
Harmony4D / 263–28921563.75 → 2211.26695.03 → 791.961.5156 → 3.22450.1779 → 0.2754

Human3R → Shot3R, W/WA on common support. IDF1 improves in every stratum and W-MPJPE in four of the five; in the EgoHumans 210–256 and Harmony4D 150 groups all four measures improve. Turning on identity inheritance over the same anonymous correspondences raises whole-sequence IDF1 by 11.30 to 44.21 points in all five strata, and leaves the geometry unchanged.

Real footage. Real edited footage is covered qualitatively. Figure 5 (Section 4) reconstructs a frontal-to-rear shot change in film footage with both people and both camera groups in one world frame, and Figure 11 (Appendix G) shows the same for an outdoor instructional video.

Occlusion and reappearance. For occlusion the annotations provide visibility gaps rather than physical occlusion labels. We examined three events in which a person leaves the view and returns, with gaps of 13, 40 and 62 frames. Shot3R re-detects the returning person in two and preserves the original identity in one.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Revisions and Additional Supplementary Material

We thank you for the reviews and for the specific suggestions.

Revised manuscript. We have clarified the problem formulation and the causal reconstruction setting in the introduction, stated what CUT3R does not output and added the television-footage reconstruction and PHALP in Section 2, and made the training description explicit in Section 3.3. The additional controlled experiments appear in new appendix tables, covering the held-out basketball capture in A.1, camera changes inside a shot in C.2, the six-shot sequences in C.5, the trained side control, relation-token ablation and history intervention in D.3, and reconstruction-stage timing in E.1. All additions and edits are marked in blue.

Supplementary material. We have uploaded an anonymized ZIP with a compressed sample of the multi-person basketball capture. It contains 100 frames from all 75 views, with RGB video, per-person masks and boxes, the per-view calibration, and the fitted SMPL-X tracks that back Table 5.

AuthorsOpenReview ↗

Summary of Revisions - 2

We thank the reviewers for the continued discussion.

Revised manuscript. The introduction now motivates the setting by real-time use, the setup states that the camera is static within a shot in the three evaluation sets, and Section 2 discusses map-based relocalization for cameras that also translate within a shot. New appendix material covers the comparison with four feed-forward geometry models and with system-level baselines in D.4, an association ablation with geometric, appearance and ground-truth variants in D.5, both methods at two backbone sizes in D.6, and the two application sequences repeated at the real-time configuration in G.3. Blue marks every addition and edit of the first revision relative to the submitted version, and teal marks what has been added in this round.

AuthorsOpenReview ↗

Summary of Revisions - 3

Revised manuscript. In this round the introduction narrows the pace requirement to footage that arrives as a stream, the setup records that the two application sequences do move the camera within a shot and are reported qualitatively rather than through the metrics, and the sentence at the end of Section 4 points to the per-frame cost by stage. New appendix material gives the protocol behind the comparison with the four feed-forward geometry models in D.4 and repeats that comparison with each model given more of its history, the weights, input sizes, crop and cost combination of the appearance association in D.5, and the cost of the boundary step against a feed-forward model in its place in E.3. Blue marks every addition and edit of the first revision relative to the submitted version, teal marks what the second revision added, and purple marks what this round adds.

Supplementary material. We have uploaded the four recordings of the two clips of Figures 5 and 11, the real-time configuration and the full model on each, rendered in the camera view of the footage. As the supplementary material also carries the captured data, the size limit meant the recordings are provided as a compressed archive.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.