acceptodds
Under review as a conference paper at ICLR 2027

SCOPE-4D: Endoscopic 4D Geometry Foundation Models

Abstract

Geometric understanding supports endoscopic navigation and robotic assistance, but learning reliable endoscopic geometry faces two challenges: scarce geometric annotations and ambiguity between camera motion and tissue deformation. We present SCOPE-4D, an endoscopic 4D geometry foundation model that jointly predicts camera parameters, dense geometry, and 3D tissue trajectories from monocular RGB video in a single forward pass. Our curation and annotation pipeline constructs SCOPE-5K, a collection of approximately 5,000 clips spanning real and synthetic gastrointestinal endoscopy and laparoscopy. The collection provides rich geometric supervision and includes newly collected phantom and real-colonoscopy evaluation sets. Geometric supervised fine-tuning on SCOPE-5K learns endoscopic priors that improve camera and depth estimation. Common–Residual Motion (CRM) further constrains local deformation relative to common tissue movement. Together with geometric supervision, CRM and trajectory supervision further improve camera and depth estimation over geometric fine-tuning alone while enabling dense 3D tissue tracking. Evaluations on public and newly collected benchmarks demonstrate strong in-domain and out-of-domain geometry, superior 3D tracking, and more stable long-sequence colon reconstruction. A blinded user study further supports the perceived reconstruction quality on real clinical video. Together, these results demonstrate the value of large-scale endoscopic supervision and motion constraints for joint geometry estimation and tissue tracking.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Bio Medical AI, 3D Computer Vision

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PQeeOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SCOPE-4D is a feed-forward endoscopic 4D geometry model for joint RGB point cloud, camera params and depth map prediction. This paper also builds a SCOPE-5K (~4,855 in-domain + 55 OOD clips) dataset with a novel pipeline (VLM screening, track-assisted SfM, MapAnything-conditioned dense depth, stereo SAV depth on StereoMIS). The paper also proposes a Common–Residual Motion (CRM) module with a shared SE(3) tissue transform plus point-dependent residual twists to reduce camera–deformation ambiguity and enable tracking. Anyway, this paper is good but not perfect.

Critical strengths

Strengths:

  1. This paper provides a clear problem formulation that focuses on the scaling and deformation ambiguity problem.
  2. This paper also provides a solid contribution to the endoscopic dataset, with detailed construction steps. The quality control also provides a robust way to ensure geometric accuracy.
  3. The method is also novel with the CRM design, which separates local and global deformation.
  4. The experiments also demonstrate that the model achieves SOTA performance on both qualitative and quantitative results.
Critical weaknesses

Weaknesses:

  1. The use of MapAnything directly for annotating the depth and point map is limited in accuracy. The authors should provide quantitative comparisons between MapAnything and other methods and state clearly why this model was chosen instead of others.
  2. Instead of using the original version of MapAnything, it would be better to use a fine-tuned version. The author may provide clear quantitative results with the MapAnything fine-tuned on existing datasets with depth GT (C3VD, SCARED, Hamlyn, StereoMIS). In this way, I think other experimental results may also be improved.
Questions for authors

Any statement for release/open source?

4 comments
AuthorsOpenReview ↗

Response to Reviewer PQee

Thank you for the review and for the two suggestions on the annotation model. We answer the first with a new measurement against ground truth, and for the second we explain why the pipeline keeps the annotator pretrained.

W1. Accuracy of MapAnything, and why we chose it

We chose MapAnything because it accepts calibrated intrinsics and camera poses as inputs. The pipeline never uses its image-only output, which Table 1 and Tables 11–15 compare with the other methods (in-domain AbsRel 0.141). It gives MapAnything the cameras recovered by MFT-assisted SfM, after these are cross-checked against two other pose sources, so that the dense depth of a clip is tied to verified multi-view geometry.

To measure what this buys, we ran the pipeline unchanged, before its quality gates, on the 11 C3VD test sequences, which have ground-truth depth, poses and calibration. Both rows use the same frames and the same MapAnything checkpoint, and the depth and pointmap errors follow the MapAnything benchmark protocol (new):

C3VD, 11 sequencesDepth [email protected] ↑Pointmap AbsRel ↓Ray direction error (°) ↓Translation error per frame (mm) ↓Rotation error per frame (°) ↓
Image-only MapAnything8.20.24210.930.8011.029
Annotation pipeline18.10.1410.910.2790.586
Gain119.8%41.8%91.7%65.2%43.1%

The camera constraints more than double the share of depth inliers, lower the pointmap error by 42% and reduce the ray direction error by an order of magnitude (lower on all 11); frame-to-frame translation and rotation errors fall by 65% and 43%. This is why we use MapAnything as a camera-conditioned module and not as a standalone predictor. The clips that enter SCOPE-5K additionally pass the quality gates and human verification (Appendix A.4).

The role of the labels is to supply multi-view supervision on clinical colonoscopy, where no depth or pose reference exists, and the test of that role is transfer to independent ground truth (Table 3(a)). Adding the annotated EndoMapper-Real clips to the five existing datasets improves Hamlyn AbsRel from 0.143 to 0.128 and δ1.25 from 80.93 to 83.04, which is also better than pretrained VGGT-Ω (0.135, 80.82), and improves phantom RTE from 11.28 to 10.37.

W2. Annotating with a MapAnything fine-tuned on datasets with depth ground truth

We keep the annotator pretrained by design. The datasets with depth ground truth are small and far from the annotation target, which is clinical colonoscopy: C3VD is a phantom and SCARED is porcine. Hamlyn is our out-of-domain test set, so the annotator cannot be trained on it without giving up that test, and StereoMIS has no depth ground truth (its depth is one of our annotations). Fine-tuning a foundation model on such small datasets fits them without improving its generalization to unseen data. Table 3(a) measures this for VGGT-Ω, the backbone of our model:

VGGT-Ω, geometry-only trainingHamlyn AbsRel ↓Phantom RTE (mm) ↓
Pretrained0.13510.79
Fine-tuned on five datasets with ground truth0.14311.28
+ clinical clips annotated by our pipeline0.12810.37

Fine-tuning on ground truth lowers the in-domain error (AbsRel 0.077 → 0.031) but makes both unseen benchmarks worse than the pretrained model. Transfer improves only when the annotated clinical clips are added (last row). An annotator fine-tuned on the same small datasets carries the same risk on clinical video.

The pipeline improves the pretrained model in a different way, through verified camera inputs, cross-checks between pose sources and human verification. The C3VD measurement above quantifies the first of these: pointmap AbsRel falls from 0.242 to 0.141 and ray direction error from 10.9° to 0.9° without any fine-tuning.

Q1. Release

Yes. We will release the code, the trained models and the annotation pipeline on acceptance, together with the SCOPE-5K annotations for the public source datasets.

Dear Reviewer PQee,

Since the authors have procvided their feedback, please discuss with the authors. Thanks.

Best regards,

AC

I appreciate the response from the author. I think the generalization problem always exists, so the use of MapAnything is reasonable. But since the amount of annotated data will be greater in the future, I believe the choice of a better solution will still be fine-tuning on the surgical data.

AuthorsOpenReview ↗

Response to Reviewer PQee

Thank you for the reply, and for confirming that the use of MapAnything is reasonable.

We agree that, once enough representative annotated surgical data is available, fine-tuning the annotator on it could further improve annotation quality. Such data is scarce today because a clinical endoscope has no depth sensor and its pose is not tracked inside the patient. We will release the pipeline and the SCOPE-5K annotations for the public source datasets, which we hope are a step toward the larger annotated data you describe.

Reviewer_KYGyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles camera pose, dense depth, and 3D tissue position from monocular endoscopic video, where weak texture and tissue deformation can look like camera motion. It builds SCOPE-5K, about 5,000 clips, by screening clinical video, reconstructing cameras, and obtaining dense depth. For EndoMapper-Real, that depth comes from MapAnything conditioned on the recovered poses. The model fine-tunes a pretrained VGGT-Ω backbone and adds Common–Residual Motion: one shared SE(3) motion plus residual twists, including a fixed zero twist. Validation covers in-domain geometry, held-out Hamlyn and a new colon phantom, 3D tracking on Sano and StereoMIS, long RealSynCol sequences, and a blinded ranking by six experts on ten real colonoscopy clips.

Critical strengths

The tracking claim is measured against feed-forward 4D trackers, not only against depth networks, and the prose does not treat the Sano end-point-error loss as a win. On StereoMIS the full model reports end-point error 5.022 against 6.266 for 4RC. The ablation separates a zero-motion anchor from the common–residual step. That is the part of the paper that speaks to the stated ambiguity between camera motion and tissue deformation.

Held-out checks are not only the training distribution. The colon phantom uses an external tracker. Hamlyn is an existing laparoscopic set. Six experts rank the same 128-frame outputs on ten real colonoscopy clips that the metric tables cannot score. Read as preference, that study is evidence the tables do not contain.

Critical weaknesses

In-domain depth on EndoMapper-Real is scored against the MapAnything annotations that produced the labels, and that score enters the Table 1 average presented as depth accuracy. The paper does not say the reference is its own annotator. The claim of leading in-domain geometry is not supported for that cell until the average is recomputed on references the model did not train against.

The long-sequence claim has two unreonciled numbers for the same RealSynCol setting: 27.65 mm in Figure 1 and 71.17 mm in Table 4. A reader cannot tell which number the claim uses. Either figure should be explained, or one of them should be withdrawn.

Questions for authors
  1. Is EndoMapper-Real depth in Table 1 and Table 15 computed against the MapAnything labels described in Section 3? If so, what is the in-domain absolute-relative error and δ_1.25 with that set removed from the depth average?
  2. What protocol produces the Figure 1 Colon 2 ATE of 27.65 mm, and why does Table 4 report 71.17 mm for long-sequence RealSynCol? Which number supports the long-sequence claim?
  3. For SCARED, which frames are scored, and are the poses the original kinematic poses? The paper already says depth is propagated under a static-scene assumption. SCARED-C (Han et al., arXiv 2605.16628, row 16 of the retrieved literature) is an arXiv preprint and cannot by itself support rejection. The answer still changes how much weight the SCARED numbers can carry.
  4. Should Section 3.2's "clinical validation" be read as geometric accuracy, or as the visual preference described in Appendix D.3?
3 comments
AuthorsOpenReview ↗

Response to Reviewer KYGy

Thank you for the review. Each answer below says where the paper states the item; we add the recomputed average you asked for, and the revision makes one statement easier to find.

W1 and Q1. EndoMapper-Real and the in-domain depth average

Yes, EndoMapper-Real depth in Tables 1 and 15 is scored against the annotations of our pipeline, and the paper states this source: Section 3 says that the pipeline adds camera and dense geometry annotations to EndoMapper-Real, Section 3.1 describes how, and Table 6 marks these annotations with †. Real colonoscopy video has no dense geometric ground truth. Annotating it with cameras cross-checked across pose sources, geometric quality control and human verification (Appendices A.2 and A.4) is one contribution of SCOPE-5K. So that the source is also visible next to the results, Appendix D.2 (new table) lists the reference of every test set.

The lead in in-domain geometry does not depend on this set. In-domain depth with and without EndoMapper-Real; "without" averages the remaining six sets, 99 sequences (new):

MethodAbsRel ↓ withAbsRel ↓ withoutδ1.25 ↑ withδ1.25 ↑ without
VGGT-Ω0.0930.08391.0793.51
SpatialTrackerV20.1130.09986.9890.12
MapAnything0.1410.11985.0085.65
4RC0.1380.12579.7182.63
SM4RT0.1470.13477.7880.50
SCOPE-4D SFT0.0420.03498.5299.45
SCOPE-4D0.0400.03298.6899.48

Every method improves, because EndoMapper-Real is the hardest set. SCOPE-4D remains first among all 14 methods, and its error remains under half that of the best baseline (0.032 against 0.083). The annotations are also validated on independent ground truth: adding EndoMapper-Real to the training data improves AbsRel on the unseen Hamlyn set from 0.143 to 0.128 (Table 3(a)).

W2 and Q2. 27.65 mm in Figure 1 and 71.17 mm in Table 4

The two numbers are consistent, and both support the long-sequence claim. Figure 1(d) is labelled "RealSynCol · Colon 2 (701 views)": it shows one of the nine RealSynCol clips, on which SCOPE-4D has an ATE of 27.65 mm against 68.29 mm for VGGT-Ω. Table 4 reports the mean ATE over all nine clips: 71.17 mm for SCOPE-4D against 77.87 mm for the best baseline (Endo3R) and 85.92 mm for VGGT-Ω.

Q3. SCARED frames and pose references

We score the eight keyframe sequences of the official test datasets 8 and 9 (training uses datasets 1–7), taking every fifth frame, which gives 62–191 views per sequence. Depth is scored against the structured-light depth, and poses against the dataset's original robot-kinematic poses. This follows the protocol of prior work, including Endo3R (Guo et al., 2025): the same references, and the same depth and relative-pose metrics. All methods are scored on the same frames and references, and Appendix D.2 states this protocol. We use the original SCARED release, not SCARED-C, and "row 16 of the retrieved literature" does not correspond to anything in the submission.

The pose references are the same for every method and favour none of them, and the SCARED results agree with the sets whose references come from a simulator or a registered phantom (pose AUC@5 of SCOPE-4D against the best baseline: 27.89 against 17.87 on SCARED, 59.70 against 6.69 on SimCol3D, 67.35 against 25.32 on C3VD).

Q4. "Clinical validation"

It refers to the blinded expert study of Appendix D.3. That study is a clinical validation of geometry, not a general visual preference: experts rank the camera trajectory, point cloud and depth of each reconstruction on clinical video, where no geometric ground truth exists. Six clinical and biomedical experts ranked anonymised reconstructions of ten clinical clips, using synchronized video, depth maps and an interactive 3D viewer. SCOPE-4D has the best mean rank on all three criteria (1.33, 1.32 and 1.32, against 1.93, 1.95 and 1.98 for VGGT-Ω) and is ranked first in 70% of evaluations, with high agreement among the experts (Kendall's W 0.70–0.78). The study complements the quantitative accuracy measured on sets with ground truth; outside the training domains these are Hamlyn, the tracked phantom and RealSynCol. Section 3.2 keeps the term and now points to Appendix D.3.

Dear Reviewer KYGy,

Since the authors have procvided their feedback, please discuss with the authors. Thanks.

Best regards,

AC

Thank you again for your review. We have answered your four questions: the depth average without EndoMapper-Real, the two RealSynCol numbers, the SCARED protocol, and the expert study in Section 3.2. Please let us know if you have any remaining questions. We would be glad to answer them before the discussion closes. If our replies have resolved your concerns, we would be grateful if you would consider updating your assessment.

Reviewer_T2MJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SCOPE-4D is an endoscopic “4D geometry foundation model” built on VGGT-ℓ. It predicts cameras, depth/pointmaps and dense 3D tissue tracks from monocular video. The paper contributes (i) SCOPE-5K, ~5k clips whose cameras and dense depth are produced by a curation/annotation pipeline plus two new colon benchmarks; (ii) geometric SFT on it; and (iii) Common-Residual Motion (CRM), which composes a shared SE(3) transform with point-wise residuals and is supervised with trajectories. It reports large in-domain gains, better camera stability on Hamlyn, higher tracking APD, and an expert user study.

Critical strengths

SCOPE-4D is a scalable, documented annotation pipeline with cross-source camera checks, human verification of QC, and new phantom and clinical benchmarks.

CRM is a clean formulation with a bound on residual displacement, and the controlled ablation against SoM is informative.

Critical weaknesses

The claimed benefit of CRM for camera and depth is not fully supported. Full SCOPE-4D vs SFT is mixed, and AUC@5 drops on SCARED, EndoMapper-Sim and EndoMapper-Real. Camera stability on Hamlyn is already achieved by SFT, which has no motion model, so the claim that CRM resolves the camera-vs-deformation ambiguity is not demonstrated.

The headline comparisons favour the proposed model. In-domain test splits share scenes with training, and labels for EndoMapper-Real and StereoMIS are pseudo-labels from MapAnything, SAV and MFT. Tracking "ground truth" is MFT correspondences lifted with SAV depth, and baselines (SM4RT, 4RC, a D4RT reproduction) are used zero-shot while SCOPE-4D is trained on these sources. OOD gains from the new data are small.

Questions for authors

Could the authors fine-tune SM4RT on SCOPE-5K with the same tracking supervision, and compare SFT vs full over several seeds with CIs, to isolate what CRM adds?

Why does SFT alone match or beat the full model on Hamlyn stability and on three of five per-dataset pose results? Is there a test with a moving camera and tissue deformation with ground truth, plus a constant-camera baseline?

Are test clips split by patient/recording? How do results change with leave-recording-out, and how are errors in MapAnything/SAV/MFT pseudo-labels bounded?

7 comments
AuthorsOpenReview ↗

Response to Reviewer T2MJ

Thank you for the careful review. The experiments you asked for are below. SFT is the geometry-only fine-tuned model, and "full" is SCOPE-4D with CRM and trajectory supervision.

W1 and Q1 (several seeds). What CRM adds over geometric fine-tuning

CRM enables dense 3D tissue tracking, which the geometry-only SFT model does not provide (Table 2). Beyond tracking, it improves depth, and it improves camera pose where the camera moves over deforming tissue. We trained each model three times with different random seeds (new; Appendix E.4, Table 21). Each cell is the mean ± standard deviation over the three runs, and the last row is the difference with its paired 95% interval.

In-domain average, 7 sets:

MethodAUC@5 ↑AUC@30 ↑ATE ↓AbsRel ↓
SFT30.43 ± 0.2067.91 ± 0.050.01596 ± 0.000050.0423 ± 0.0001
Full30.86 ± 0.0767.45 ± 0.060.01694 ± 0.000090.0406 ± 0.0001
Full − SFT+0.44 [−0.03, 0.91]−0.46 [−0.55, −0.37]+0.00098 [0.00083, 0.00113]−0.0017 [−0.0019, −0.0016]

Out of domain (Hamlyn pose = stability under a stationary camera):

MethodHamlyn ATE ↓Hamlyn ARE ↓Hamlyn AbsRel ↓Hamlyn δ1.25 ↑Phantom ATE ↓Phantom RTE ↓
SFT2.887 ± 0.0401.027 ± 0.0270.1350 ± 0.001382.56 ± 0.288.23 ± 0.0710.55 ± 0.09
Full3.181 ± 0.0771.151 ± 0.0150.1248 ± 0.000285.08 ± 0.078.32 ± 0.1510.84 ± 0.15
Full − SFT+0.295 [0.088, 0.502]+0.124 [0.078, 0.170]−0.0102 [−0.0130, −0.0073]+2.52 [2.00, 3.04]+0.09 [−0.34, 0.51]+0.28 [−0.10, 0.67]

Depth is better with CRM, both in-domain and on Hamlyn (AbsRel 0.135 → 0.125, δ1.25 +2.5 points, about nine times the run-to-run standard deviation); both intervals exclude zero. The pose averages are comparable: in-domain AUC@5 is 0.44 higher and AUC@30 0.46 lower, and the phantom differences lie within their intervals. Per dataset, AUC@5 is higher on SimCol3D and C3VD and lower on the three sets you list. On Hamlyn, both variants (2.89 and 3.18) are below every compared method in Table 1 (SM4RT 3.52, VGGT-Ω 4.50).

Comparable pose averages do not mean that CRM has no effect on pose. CRM separates camera motion from tissue motion, whether the camera is stationary over deforming tissue, moves over rigid tissue, or moves over deforming tissue. Three results show its effect on pose:

  • Moving camera over deforming tissue. Pose improves on both test sets that have such sequences (see Q2).
  • Motion model alone. With the backbone fixed, CRM in place of the SoM motion bases improves StereoMIS tracking and pose (Table 3(b)).
  • Visual results. Figure 3 and the videos on the anonymous project page ("Geometry and camera stability": https://scope4dreview.github.io/scope4d-anonymous/) show a stationary-camera Hamlyn clip with deforming tissue. The cameras predicted by the full model stay tightly clustered, more so than those of SFT, while VGGT-Ω, Endo3R and SM4RT show spurious camera motion.
AuthorsOpenReview ↗

Response to Reviewer T2MJ

Q2. Moving camera with deformation, and a constant-camera baseline

Why SFT matches the full model on Hamlyn. Hamlyn has no pose ground truth, so on its stationary-camera clips ATE and ARE measure how far the predicted camera drifts from the first predicted pose (Appendix D.3). Geometric fine-tuning already removes most of this drift, and a model that always outputs the same camera would score zero. The metric therefore tests whether the camera stays stationary, not whether the tissue motion is explained correctly; at similar stability, the full model has the better Hamlyn depth. Testing whether a model separates camera motion from deformation requires sequences that contain both.

Moving camera over deforming tissue. Two test sets contain such sequences. Sano (simulated cholecystectomy; camera, depth and tracks from the simulator) has windows in which the camera moves (0.5–27.9 mm, 1.1–28.4°) while instruments deform the tissue. StereoMIS (in vivo; poses from robot kinematics) also has moving-camera windows (70–114 mm). The constant-camera row predicts the same pose for every frame; its ATE is undefined after trajectory alignment. SFT and Full are three-run means (new; Appendix E.4, Table 22):

WindowsMethodATE ↓RTE ↓RRE ↓AbsRel ↓
Sano, movingConstant camera–0.1820.326–
SM4RT, zero-shot0.01260.1850.1450.185
SFT0.00200.0900.0820.040
Full0.00190.0860.0770.035
StereoMIS, movingConstant camera–1.5310.369–
SM4RT, zero-shot0.03970.7450.3030.142
SFT0.03930.6390.2650.120
Full0.03250.5670.2260.116

Compared with SFT, the full model lowers RTE by 5% and RRE by 7% on Sano and raises pose AUC@5 from 28.1 to 31.9 (+3.8, paired 95% interval [3.1, 4.6]); on StereoMIS it lowers ATE, RTE and RRE by 17%, 11% and 15%. Both SCOPE-4D variants have less than half the RTE of a constant camera, whereas zero-shot SM4RT is at the constant-camera level on Sano. On the stationary-camera windows of both sets, both variants are stable (ATE 0.0004–0.0005); the gain from CRM appears when the camera moves.

Q1 (fine-tuned SM4RT). SM4RT fine-tuned on SCOPE-5K

We fine-tuned SM4RT on SCOPE-5K with the same supervision as SCOPE-4D, including the tracking supervision, and the same training budget (SM4RT_FT). The comparison is in the revised paper (Appendix E.3, Tables 18–20, Figures 17–19, new) and on the anonymous project page (section "Comparison with fine-tuned SM4RT": https://scope4dreview.github.io/scope4d-anonymous/#sm4rt-ft).

Camera pose and depth (Table 18; ID = in-domain average):

MethodID AUC@5 ↑ID AbsRel ↓ID δ1.25 ↑Hamlyn ATE ↓Hamlyn ARE ↓Hamlyn AbsRel ↓Hamlyn δ1.25 ↑Phantom ATE ↓
SM4RT4.860.14777.783.5171.3500.12083.9414.46
SM4RT_FT3.840.10489.222.5211.7760.12182.6313.40
SCOPE-4D29.860.04098.682.9761.0130.12485.238.29

Tracking and long sequences (Tables 19 and 20; RealSynCol in mm):

MethodSano EPE ↓Sano APD ↑StereoMIS EPE ↓StereoMIS APD ↑RealSynCol ATE ↓RealSynCol RTE ↓
SM4RT0.41195.646.27643.9588.9521.29
SM4RT_FT0.66594.816.10044.4091.6318.55
SCOPE-4D0.44296.115.02253.0871.1710.94

Fine-tuning mainly improves SM4RT's in-domain depth (AbsRel 0.147 → 0.104), which stays far from SCOPE-4D (0.040). The gain does not carry over to unseen data: Hamlyn δ1.25 falls from 83.94 to 82.63, and phantom ATE improves only from 14.46 to 13.40 (SCOPE-4D: 8.29). Tracking improves slightly on StereoMIS (EPE 6.276 → 6.100) and gets worse on Sano (0.411 → 0.665), and long-sequence ATE does not improve (91.63 against 71.17 mm). On Hamlyn, the lower stability ATE of SM4RT_FT comes with a higher ARE (1.776 against 1.013) and a lower δ1.25, and AbsRel is comparable (0.121 and 0.124). On Sano, SCOPE-4D has the highest APD, with EPE comparable to zero-shot SM4RT.

Figure 18 and the tracking videos on the project page explain the tracking result: after fine-tuning, the SoM motion bases of SM4RT give a weak motion that is nearly the same over the whole scene, whereas CRM separates the shared tissue motion from the local tool-induced motion. With the same data, supervision and training budget, SCOPE-4D remains ahead, so its gains come from the CRM formulation and not only from training on SCOPE-5K. Table 3(b) isolates the motion model within one network: replacing the SoM motion bases with CRM lowers StereoMIS EPE from 5.94 to 5.27 and raises APD from 46.58 to 52.63 and pose AUC@5 from 5.67 to 6.96.

AuthorsOpenReview ↗

Response to Reviewer T2MJ

W2. Evidence for the method and the data beyond the in-domain columns

The in-domain columns are one part of the evidence. Four further comparisons support the model and SCOPE-5K, and none of them rests on a training-data advantage or on our own annotations:

  • Same training data. With SM4RT fine-tuned on SCOPE-5K with the same supervision and training budget, SCOPE-4D remains ahead in in-domain pose and depth, phantom pose, StereoMIS tracking and long sequences (Q1). The advantage comes from the model, not only from the data.
  • Data no method was trained on. Against its pretrained backbone VGGT-Ω, SCOPE-4D improves Hamlyn stability (ATE 4.50 → 2.98) and depth (AbsRel 0.135 → 0.124), phantom pose (ATE 8.54 → 8.29) and long-sequence reconstruction on RealSynCol (ATE 85.9 → 71.2 mm, −17.2%; RTE 21.7 → 10.9, −49.5%; Table 4). In the blinded expert study on unseen clinical clips it has the best mean rank for cameras, point clouds and depth (Table 5).
  • The new annotations transfer. Adding EndoMapper-Real to the five existing datasets lowers Hamlyn AbsRel from 0.143 to 0.128 (−10.5%) and phantom RTE from 11.28 to 10.37 (Table 3(a)). Both test sets are unseen, so the clinical annotations improve geometry outside their own domain.
  • Independent references. The lead does not depend on the references we annotated. Without EndoMapper-Real, the in-domain average over the remaining six sets (99 sequences) is AbsRel 0.032 for SCOPE-4D against 0.083 for VGGT-Ω and 0.134 for SM4RT. On Sano, whose tracks are rendered by the simulator, SCOPE-4D has the highest APD (96.11). On StereoMIS, whose track references are MFT correspondences lifted with stereo depth (Appendix A.3), EPE improves by 19.8% over the best compared method.

Q3. Splits, leave-recording-out, and error in the pseudo-labels

No test frame or sequence is used in training. The held-out unit of each in-domain test set:

Test setHeld-out unit
SCAREDRecording: datasets 8 and 9 (official split)
EndoMapper-RealClinical recording: Seq_001, Seq_008
SimCol3DSynthetic trajectory (official split)
C3VDAnatomical region: descending colon
EndoMapper-SimSimulated sequence
StereoMISTemporal segment, ±90 frames excluded
SanoSimulation attempt, separate lighting block

Leave-recording-out. Two in-domain sets are already held out by recording, SCARED (official split) and EndoMapper-Real. On SCARED the full model has pose AUC@5 27.89 against 17.87 for the best baseline, ATE 0.016 against 0.030, and the lowest AbsRel (0.033). The three out-of-domain sets are a stricter test than leave-recording-out: Hamlyn, the tracked phantom and RealSynCol are whole datasets that share no scene, phantom or recording with training, and SCOPE-4D improves on its pretrained backbone on all three (W2).

Error in the pseudo-labels. Clinical video has no independent geometric reference, so label quality is controlled in two ways. First, every accepted clip passes the quality gates: at least 90% of frames registered, agreement between pose sources, agreement between sparse and dense depth, and human review. Second, we measured the pipeline against ground truth by running it unchanged, before the quality gates, on the 11 C3VD test sequences, which have ground-truth depth, poses and calibration. Both rows use the same frames and the same MapAnything checkpoint, and the depth and pointmap errors follow the MapAnything benchmark protocol (new):

C3VD, 11 sequencesDepth [email protected] ↑Pointmap AbsRel ↓Ray direction error (°) ↓Translation error per frame (mm) ↓Rotation error per frame (°) ↓
Image-only MapAnything8.20.24210.930.8011.029
Annotation pipeline18.10.1410.910.2790.586
Gain119.8%41.8%91.7%65.2%43.1%

The camera constraints more than double the share of depth inliers, lower the pointmap error by 42% and reduce the ray direction error by an order of magnitude, to 0.9°, and the frame-to-frame pose error of the labels is 0.28 mm and 0.59°. The labels are also validated by what they teach: adding them to training improves depth and pose on the unseen Hamlyn and phantom sets (W2). Track supervision is tested on Sano, whose reference tracks are rendered by the simulator, and SCOPE-4D has the highest APD there (96.11).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We thank the three reviewers for their careful reading. The reviews mainly raise three questions:

  1. What does Common–Residual Motion (CRM) add over geometric fine-tuning?
  2. Do the gains come from the model or from the training data?
  3. How reliable are the annotations and the evaluation references?

We answer each below with new experiments; the individual replies give the full tables. SFT is the geometry-only fine-tuned model, and "full" is SCOPE-4D with CRM and trajectory supervision.

1. What CRM adds (T2MJ W1, Q1, Q2)

  • Dense 3D tissue tracking. The geometry-only SFT model cannot track tissue; CRM adds this ability (Table 2).
  • Better depth. We trained each model three times with different random seeds. CRM improves depth in-domain and on the unseen Hamlyn set (AbsRel 0.135 → 0.125, δ1.25 82.6 → 85.1); the paired 95% intervals exclude zero.
  • Better pose when the camera moves over deforming tissue. This is the hardest case for separating the two motions. On the moving-camera Sano windows, which have simulator ground truth, pose AUC@5 rises from 28.1 to 31.9. On the moving-camera StereoMIS windows, ATE falls by 17%.
  • The motion model in isolation. With the backbone fixed, replacing the SoM motion bases with CRM improves StereoMIS tracking EPE from 5.94 to 5.27 and pose AUC@5 from 5.67 to 6.96 (Table 3(b)).

The remaining pose averages are comparable between the two models: in-domain AUC@5 is 30.86 against 30.43, and Hamlyn stability ATE is 3.18 against 2.89, both below every compared method in Table 1. Figure 3 and the videos on the anonymous project page show that the full model predicts more stable cameras on deforming tissue.

2. The gains come from the model, not only from the data (T2MJ W2, Q1)

Same training data. We fine-tuned SM4RT on SCOPE-5K with the same supervision, including the tracking supervision, and the same training budget as SCOPE-4D. SCOPE-4D stays ahead in pose, depth, tracking and long sequences:

MethodIn-domain AUC@5 ↑In-domain AbsRel ↓Phantom ATE ↓StereoMIS EPE ↓RealSynCol ATE (mm) ↓
SM4RT, fine-tuned3.840.10413.406.1091.6
SCOPE-4D29.860.0408.295.0271.2

The fine-tuned SM4RT predicts a weak, nearly uniform motion, whereas CRM separates the shared tissue motion from the local tool-induced motion (Figure 18).

Data no method was trained on. Compared with its pretrained backbone VGGT-Ω, SCOPE-4D improves on all three out-of-domain sets: Hamlyn (stability ATE 4.50 → 2.98, AbsRel 0.135 → 0.124), the phantom (ATE 8.54 → 8.29) and RealSynCol (ATE 85.9 → 71.2 mm). It also has the best mean rank for cameras, point clouds and depth in the blinded expert study (Table 5).

3. Annotations and evaluation references (all reviewers)

  • The results do not depend on our own annotations (KYGy W1, Q1). Clinical video has no dense geometric ground truth, and the paper states that our pipeline annotates EndoMapper-Real and StereoMIS (Section 3, Table 6, Appendix A.3). Without EndoMapper-Real, the in-domain AbsRel of SCOPE-4D is 0.032 against 0.083 for the best baseline. On Sano, whose tracks are rendered by the simulator, SCOPE-4D has the highest APD.
  • The annotations are accurate against ground truth (PQee W1, T2MJ Q3). On the 11 C3VD test sequences, giving MapAnything the cameras recovered by our pipeline lowers its pointmap AbsRel from 0.242 to 0.141 and its ray direction error from 10.9° to 0.9°.
  • The annotations improve transfer (PQee W2). Fine-tuning on the five datasets with ground truth alone does not improve the unseen sets over the pretrained model (Hamlyn AbsRel 0.135 → 0.143). Adding the clinical clips annotated by our pipeline does (0.128; phantom RTE 11.28 → 10.37; Table 3(a)).
  • One protocol for all methods (T2MJ Q3, KYGy Q3). No test frame or sequence is used in training. SCARED and EndoMapper-Real are held out by recording, and the three out-of-domain sets share no scene, phantom or recording with training. SCARED is scored on the official test datasets against the references of the original release, following prior work, including Endo3R (Guo et al., 2025).
  • Figure 1(d) and Table 4 are consistent (KYGy W2, Q2). Figure 1(d) shows one RealSynCol clip (Colon 2, 701 views; ATE 27.65 mm). Table 4 reports the mean over all nine clips (71.17 mm, against 77.87 mm for the best baseline).

Changes in the revised PDF (in blue)

  • Appendix E.3 (new; Tables 18–20, Figures 17–19): SM4RT fine-tuned on SCOPE-5K.
  • Appendix E.4 (new; Tables 21–22): SFT against the full model over three runs, and the moving-camera windows with a constant-camera reference.
  • Appendix D.2 (new table): frames and references of every evaluation set.
  • Section 3.2 ("clinical validation", KYGy Q4): pointer to the blinded expert study of Appendix D.3 added.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.