- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Shot3R addresses causal human-scene reconstruction across viewpoint cuts. At each boundary, a history-conditioned pass estimates the relation to the preceding shot, while a reset pass reconstructs the incoming shot independently. Learned relation tokens refine the camera estimate; geometric identity association and a shared translation correction integrate people, cameras, and scene points. Experiments on multi-shot streams constructed from EgoBody, EgoHumans, and Harmony4D evaluate reconstruction, camera placement, identity continuity, and state propagation. The principal contribution is the separation of history used for registration from history used for subsequent reconstruction.
Critical strengths
-
A well-motivated separation supported by a direct control. The two-pass design addresses a concrete tension: retaining state can aid cross-shot alignment while corrupting reconstruction after a large viewpoint change. Appendix D.2 holds the pretrained checkpoint and post-cut inputs fixed with Shot3R modules disabled. At 16 offsets, resetting reduces translation drift from 1.124 to 0.026 m and rotation drift from 14.578 to 0.200 degrees (Table 13), with paired capture-level intervals excluding zero. This supports the central design choice more directly than aggregate performance alone.
-
Meaningful improvements over the same reconstruction backbone. Table 1 reports EgoBody W-MPJPE decreasing from 321.1 to 258.8 mm and IDF1 increasing from 0.870 to 0.985; EgoHumans W-MPJPE decreases from 1005.1 to 866.8 mm and IDF1 increases from 0.471 to 0.571. The registration controls in Tables 3 and 12 are informative: simple human-joint fitting is competitive in human error, whereas Shot3R also improves camera and identity point estimates. The contribution is therefore best understood as coherent integration of alignment and identity handling.
Critical weaknesses
-
The headline metrics only partly establish shared-world consistency. Appendix A.2 uses separate per-identity similarity fits for W/WA on EgoHumans and Harmony4D, which can absorb relative placement errors between people. Camera ATE measures aligned camera centers and does not by itself establish correct camera orientation or metric scale. The shared-world diagnostic in Table 8 is valuable and directly addresses this issue, but its human-support sets differ from Human3R (Table 9), and residual errors remain substantial: Shot3R reports 2.69 m root error and 1.49 m pair-vector error. Moreover, EgoHumans Coverage falls from 0.639 to 0.616 in Table 1. A comparison on common supported observations is needed to establish the extent of the joint geometric improvement; the current human/camera metrics also do not quantify dense-scene accuracy.
-
Validation of repeated streaming transitions is limited. The primary evaluation uses two-shot clips, and Appendix B.2's first-threshold-crossing wrapper does not measure repeated false resets. The four-clip three-shot test improves human error and identity but increases camera ATE from 1.618 to 1.700 m (Table 11). The absence of explicit match rejection also makes association vulnerable when visible identities change. Appendix F acknowledges these limits, which I appreciate. They constrain the demonstrated setting to short streams with temporally adjacent cuts and continuing human visibility, rather than establishing robustness to arbitrary edited video.
Questions for authors
-
Can the Table 8 shared-world metrics be recomputed on the intersection of supported captures and person observations, using the same single-alignment protocol for every method? Please retain coverage information so that improved geometry can be distinguished from changes in which people are evaluated.
-
Appendix D.1 reports W-MPJPE difference intervals between human-joint fitting and Shot3R that include zero on both datasets. Under matched boundary inputs and identity/refinement settings, what improvement specifically depends on learned relation tokens? Existing ablations or paired camera/identity results would help establish their incremental value.
-
For the existing three-shot clips, how do detector events, associations, and camera error evolve at each boundary? This limited diagnostic would help explain the ATE regression and determine whether the abstract and conclusion should further qualify the streaming claim.
6 comments
Response to Reviewer UYJX [Part 2/4]
Q2. Appendix D.1 reports W-MPJPE difference intervals between human-joint fitting and Shot3R that include zero on both datasets. Under matched boundary inputs and identity/refinement settings, what improvement specifically depends on learned relation tokens? Existing ablations or paired camera/identity results would help establish their incremental value.
A2. We agree that the comparison in Appendix D.1 does not isolate the contribution of the relation estimator, as identity association and refinement also differ between the methods. We therefore reran it with everything except the estimator held fixed — the same clean-reset predictions, known boundaries, predicted correspondences from the full model, identity inheritance and the shared translation refinement — replacing only the relation estimate by Kabsch, which fits a unit-scale rigid transform to predicted joints without ground-truth correspondences. Because the correspondences are taken from the full model, this is a conditional module ablation rather than an independent geometric baseline. It isolates the estimator given Shot3R's own correspondences, and we state the comparison on those terms. Camera figures use each dataset's main protocol, except Harmony4D, which uses the multi-cut diagnostic clips.
| Dataset | Camera error ↓ m (Kabsch → Shot3R) |
|---|---|
| EgoBody | 0.0186 → 0.0165 |
| EgoHumans | 1.2843 → 1.5479 |
| Harmony4D | 3.0231 → 1.9570 |
Estimator results. Learned relation estimation improves camera alignment on EgoBody and Harmony4D. The paired interval for the main ATE is m, with a supplementary SE(3) ATE moving from 0.3059 to 0.2113 m on , and on Harmony4D 3.0231 → 1.9570 m (−35.3%) on , excluding zero. Kabsch fits the predicted joints directly, whereas the learned path estimates the camera relation from the observation and the accumulated history; the two do not optimise the same quantity.
Token ablation. Within the learned estimator the three relation tokens are assessed separately on a larger set of 1,000 four-frame confirmation events.
| Setting | Rel. rot. ↓ ° | Rel. trans. ↓ m | Root res. ↓ mm | Joint res. ↓ mm |
|---|---|---|---|---|
| Shot3R | 5.134 | 0.226 | 191.63 | 209.04 |
| w/o semantic | 7.682 | 0.291 | 221.65 | 239.49 |
| w/o camera | 5.228 | 0.274 | 198.71 | 212.69 |
| w/o temporal | 5.326 | 0.265 | 184.80 | 211.51 |
Removing any single token raises at least three of the four measures, and the relative translation error rises by 28.8%, 21.2% and 17.3% for the semantic, camera and temporal token.
Paper changes. The following places are revised, and every addition or edit is set in blue. Appendix D.3 adds the relation-token ablation reported above (Table 19, Q2).
Response to Reviewer UYJX [Part 3/4]
W2. Validation of repeated streaming transitions is limited. The primary evaluation uses two-shot clips, and Appendix B.2's first-threshold-crossing wrapper does not measure repeated false resets. The four-clip three-shot test improves human error and identity but increases camera ATE from 1.618 to 1.700 m (Table 11). The absence of explicit match rejection also makes association vulnerable when visible identities change. Appendix F acknowledges these limits, which I appreciate. They constrain the demonstrated setting to short streams with temporally adjacent cuts and continuing human visibility, rather than establishing robustness to arbitrary edited video.
Q3. For the existing three-shot clips, how do detector events, associations, and camera error evolve at each boundary? This limited diagnostic would help explain the ATE regression and determine whether the abstract and conclusion should further qualify the streaming claim.
A3. To evaluate repeated detection beyond the first-threshold-crossing wrapper, we ran the detector over all frame pairs. All eight annotated cuts were detected, with no additional positive detections among the 588 non-cut frame pairs. Reconstruction was evaluated separately, using ground-truth cut locations at frames 50 and 100.
| Clip | Cut frame | Rotation error ↓ ° | Matches correct / wrong / unassessable | Root: shared translation off → on ↓ m | Camera position: shared translation off → on ↓ m |
|---|---|---|---|---|---|
| karate2 | 50 | 14.01 | 2 / 0 / 1 | 3.561 → 1.825 | 2.435 → 0.876 |
| karate2 | 100 | 171.61 | 0 / 2 / 0 | 4.125 → 2.988 | 2.556 → 3.616 |
| karate3 | 50 | 15.22 | 2 / 0 / 0 | 3.585 → 1.948 | 2.425 → 0.851 |
| karate3 | 100 | 139.75 | 0 / 2 / 0 | 6.663 → 4.068 | 3.372 → 5.098 |
| mma3 | 50 | 8.11 | 1 / 0 / 3 | 3.258 → 0.924 | 1.711 → 0.721 |
| mma3 | 100 | 158.85 | 0 / 1 / 3 | 4.836 → 2.821 | 5.861 → 8.717 |
| mma4 | 50 | 8.20 | 2 / 0 / 4 | 2.120 → 0.832 | 3.191 → 0.464 |
| mma4 | 100 | 65.31 | 0 / 0 / 6 | 2.882 → 0.622 | 6.382 → 5.555 |
Reading the boundary table. The last two columns compare the same boundary with the shared translation removed locally and restored, with the upstream predictions and the initial alignment fixed. The first boundary shows relatively low rotation errors (8.11–15.22°) and correct associations for all assessable matches; the second boundary exhibits substantially larger rotation errors (65.31–171.61°), accompanied by incorrect or unassessable associations. These observations suggest that errors can accumulate across repeated transitions. With only eight boundaries we cannot distinguish accumulated alignment error from errors in the current relation estimation and identity association; the root error decreases at all eight boundaries and camera position at five.
Response to Reviewer UYJX [Part 1/4]
A1. We thank you for this helpful suggestion. To address it we recomputed the shared-world metrics with a consistent evaluation protocol across all methods. Each method fits one initial Sim(3) from the same two-frame reference, and that single transform is applied to all people, cameras and shots of the clip, with no fit per identity and none per later shot. Post-cut root errors use common person-frame observations, pair-vector errors use common valid person pairs, and camera errors are taken at the first post-cut frame, averaged equally over captures. Geometric errors are therefore evaluated on the intersection of supported observations, taken per metric, so the number of captures behind each column differs; coverage is reported separately for each method, because it reflects that method's own prediction support rather than the common set.
Magnitude and significance. Against Human3R the first-post-cut rotation difference has a paired 95% capture-bootstrap interval of [−16.68∘,−4.36∘], excluding zero. Root error is essentially unchanged (2.672 → 2.642 m), while the pair-vector error decreases from 1.535 to 1.332 m and the camera rotation from 33.82° to 23.11°. The two stages also separate in the table. From alignment alone to Shot3R, root error falls from 3.318 to 2.642 m on [−1.202,−0.299] and camera position from 5.186 to 4.587 m on [−1.028,−0.211], while the pair vector and the camera orientation are identical in the two rows — as expected, since a shared translation satisfies (pi+Δ)−(pj+Δ)=pi−pj and moves positions without touching relative configuration. Reset produces the largest errors across the four geometric metrics.
Coverage. Coverage is retained as requested. It is computed on the captures that admit a single initial fit, which is why these values differ from the main table. On EgoHumans it is 58.99% for Human3R and 56.53% for Shot3R. We additionally report common-support results on EgoBody, where both methods achieve identical coverage (98.06%). Shot3R reduces W/WA-MPJPE from 354.08/249.02 to 260.31/203.50 mm on [−153.50,−41.47] and [−71.29,−22.99].
Relation to the submitted table. In the submitted table the supports differ by metric — root 26 captures for Human3R against 25 for the other settings, pair vector 20 against 19, and 27 for the camera columns in all methods. The values above are recomputed on the common support of each metric rather than re-reported from that table. A post-hoc Sim(3) alignment does not establish the metric-scale accuracy of the original predictions, and the quantitative claims of the paper concern human and camera geometry and identity consistency; dense-scene accuracy is not evaluated against an independent reference.