- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes SCOPE-4D is a feed-forward endoscopic 4D geometry model for joint RGB point cloud, camera params and depth map prediction. This paper also builds a SCOPE-5K (~4,855 in-domain + 55 OOD clips) dataset with a novel pipeline (VLM screening, track-assisted SfM, MapAnything-conditioned dense depth, stereo SAV depth on StereoMIS). The paper also proposes a Common–Residual Motion (CRM) module with a shared SE(3) tissue transform plus point-dependent residual twists to reduce camera–deformation ambiguity and enable tracking. Anyway, this paper is good but not perfect.
Critical strengths
Strengths:
- This paper provides a clear problem formulation that focuses on the scaling and deformation ambiguity problem.
- This paper also provides a solid contribution to the endoscopic dataset, with detailed construction steps. The quality control also provides a robust way to ensure geometric accuracy.
- The method is also novel with the CRM design, which separates local and global deformation.
- The experiments also demonstrate that the model achieves SOTA performance on both qualitative and quantitative results.
Critical weaknesses
Weaknesses:
- The use of MapAnything directly for annotating the depth and point map is limited in accuracy. The authors should provide quantitative comparisons between MapAnything and other methods and state clearly why this model was chosen instead of others.
- Instead of using the original version of MapAnything, it would be better to use a fine-tuned version. The author may provide clear quantitative results with the MapAnything fine-tuned on existing datasets with depth GT (C3VD, SCARED, Hamlyn, StereoMIS). In this way, I think other experimental results may also be improved.
Questions for authors
Any statement for release/open source?
4 comments
Dear Reviewer PQee,
Since the authors have procvided their feedback, please discuss with the authors. Thanks.
Best regards,
AC
I appreciate the response from the author. I think the generalization problem always exists, so the use of MapAnything is reasonable. But since the amount of annotated data will be greater in the future, I believe the choice of a better solution will still be fine-tuning on the surgical data.
Response to Reviewer PQee
Thank you for the reply, and for confirming that the use of MapAnything is reasonable.
We agree that, once enough representative annotated surgical data is available, fine-tuning the annotator on it could further improve annotation quality. Such data is scarce today because a clinical endoscope has no depth sensor and its pose is not tracked inside the patient. We will release the pipeline and the SCOPE-5K annotations for the public source datasets, which we hope are a step toward the larger annotated data you describe.
Response to Reviewer PQee
Thank you for the review and for the two suggestions on the annotation model. We answer the first with a new measurement against ground truth, and for the second we explain why the pipeline keeps the annotator pretrained.
W1. Accuracy of MapAnything, and why we chose it
We chose MapAnything because it accepts calibrated intrinsics and camera poses as inputs. The pipeline never uses its image-only output, which Table 1 and Tables 11–15 compare with the other methods (in-domain AbsRel 0.141). It gives MapAnything the cameras recovered by MFT-assisted SfM, after these are cross-checked against two other pose sources, so that the dense depth of a clip is tied to verified multi-view geometry.
To measure what this buys, we ran the pipeline unchanged, before its quality gates, on the 11 C3VD test sequences, which have ground-truth depth, poses and calibration. Both rows use the same frames and the same MapAnything checkpoint, and the depth and pointmap errors follow the MapAnything benchmark protocol (new):
The camera constraints more than double the share of depth inliers, lower the pointmap error by 42% and reduce the ray direction error by an order of magnitude (lower on all 11); frame-to-frame translation and rotation errors fall by 65% and 43%. This is why we use MapAnything as a camera-conditioned module and not as a standalone predictor. The clips that enter SCOPE-5K additionally pass the quality gates and human verification (Appendix A.4).
The role of the labels is to supply multi-view supervision on clinical colonoscopy, where no depth or pose reference exists, and the test of that role is transfer to independent ground truth (Table 3(a)). Adding the annotated EndoMapper-Real clips to the five existing datasets improves Hamlyn AbsRel from 0.143 to 0.128 and δ1.25 from 80.93 to 83.04, which is also better than pretrained VGGT-Ω (0.135, 80.82), and improves phantom RTE from 11.28 to 10.37.
W2. Annotating with a MapAnything fine-tuned on datasets with depth ground truth
We keep the annotator pretrained by design. The datasets with depth ground truth are small and far from the annotation target, which is clinical colonoscopy: C3VD is a phantom and SCARED is porcine. Hamlyn is our out-of-domain test set, so the annotator cannot be trained on it without giving up that test, and StereoMIS has no depth ground truth (its depth is one of our annotations). Fine-tuning a foundation model on such small datasets fits them without improving its generalization to unseen data. Table 3(a) measures this for VGGT-Ω, the backbone of our model:
Fine-tuning on ground truth lowers the in-domain error (AbsRel 0.077 → 0.031) but makes both unseen benchmarks worse than the pretrained model. Transfer improves only when the annotated clinical clips are added (last row). An annotator fine-tuned on the same small datasets carries the same risk on clinical video.
The pipeline improves the pretrained model in a different way, through verified camera inputs, cross-checks between pose sources and human verification. The C3VD measurement above quantifies the first of these: pointmap AbsRel falls from 0.242 to 0.141 and ray direction error from 10.9° to 0.9° without any fine-tuning.
Q1. Release
Yes. We will release the code, the trained models and the annotation pipeline on acceptance, together with the SCOPE-5K annotations for the public source datasets.