- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Endo4DLR, a feed-forward framework for online language-aligned 4D reconstruction from unposed endoscopic videos. It jointly predicts camera parameters and language-embedded Gaussians without per-scene optimization. Adaptive Gaussian queries and Z-order serialization enable local grouping, while persistent anchors retrieve historical features for Gaussian attribute and language embedding prediction. Experiments evaluate reconstruction and language grounding across multiple datasets.
Critical strengths
The problem formulation presented in this study is significant. The paper explores how to achieve online 4D reconstruction—aligned with linguistic information—from endoscopic video lacking camera pose data, without requiring per-scene optimization. By jointly estimating camera poses and constructing Gaussian primitives that incorporate linguistic information, this approach facilitates the understanding of dynamic surgical scenes and lends practical utility to the proposed framework.
The method demonstrates strong rendering performance across multiple datasets. On the five endoscopic datasets evaluated, Endo4DLR consistently outperforms an improved version of the Stream3R baseline model in terms of RGB rendering quality. Furthermore, ablation studies indicate that both adaptive queries and persistent Gaussian anchors contribute to performance gains, providing compelling evidence for the effectiveness of the proposed design.
Critical weaknesses
I think the main problems of this paper are as follows:
1: The paper claims to introduce the first fully feed-forward framework for online language-aligned dynamic scene reconstruction from unposed endoscopic videos. However, the experiments primarily compare against general-purpose online reconstruction methods developed for natural scenes. Although Table 2 includes DGD and 4D LangSplat as per-scene optimization baselines for language grounding, the reconstruction evaluation lacks a comparison with the representative per-scene endoscopic scene understanding, such as SurgTPGS++ [1],DeGenseGS [2] and SurgSemGS [3]. The absence of these analysis and experiments diminishes the contribution of this method to scene understanding.
2: Several key components have clear precedents, including the reconstruction backbone from Stream3R [4], anchor-based Gaussian organization in Scaffold-GS [5] and Z-order serialization in Z-Order Transformer [6]. Language feature compression also follows the basic approach of 4D LangSplat [7]. The paper integrates these ideas but does not sufficiently clarify the substantive differences between these designs and existing mechanisms.
[1] Huang, Y., Bai, L., Cui, B., Yuan, K., Wang, G., Hoque, M. I., ... & Ren, H. (2026). SurgTPGS++: Text-promptable Gaussian Splatting with dense features aggregation for semantic 4D surgical scene understanding. Medical Image Analysis, 104318. [2] Wang, Y., Kang, B., Yang, S., & Jin, Y. (2026, September). DeGenseGS: Geometrically and Semantically Decoupled Surgical Scene Understanding in 4D Gaussian Splatting. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 112-122). Cham: Springer Nature Switzerland. [3] Liu, L., Gu, Q., Feng, Y., Hu, L., Nguyen, A., & Huang, B. SurgSemGS: Semantic and Contrastive Gaussian Splatting for Dynamic Surgical Scene Understanding. In CREATE: Clinical-driven Robotics and Embodied AI TEchnology. [4] Lan, Y., Luo, Y., Hong, F., Zhou, S., Chen, H., Lyu, Z., ... & Pan, X. (2026, April). Stream3r: Scalable sequential 3d reconstruction with causal transformer. In International Conference on Learning Representations (Vol. 2026, pp. 42746-42768). [5] Lu, T., Yu, M., Xu, L., Xiangli, Y., Wang, L., Lin, D., & Dai, B. (2024, June). Scaffold-gs: Structured 3d gaussians for view-adaptive rendering. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 20654-20664). IEEE. [6] Wang, C., Liu, L., Jiang, W., & Xu, D. (2026). Z-Order Transformer for Feed-Forward Gaussian Splatting. arXiv preprint arXiv:2605.13465. [7] Li, W., Zhou, R., Zhou, J., Song, Y., Herter, J., Qin, M., ... & Pfister, H. (2025, June). 4d langsplat: 4d language gaussian splatting via multimodal large language models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 22001-22011). IEEE.
Questions for authors
-
Please clarify the fundamental technical innovations of the proposed method compared to existing methods. For instance, anchor-based Gaussian models and Z-order organization are existing techniques.
-
Please analyze and compare the reconstruction and semantic segmentation quality against "scene-based optimization methods", such as SurgTPGS++, rather than limiting comparisons to feed-forward baseline methods. If such a comparison is not feasible, please explain why.
5 comments
Response to Reviewer Df2q (Part 2)
Q2 / W1: Additional comparisons with surgical per-scene optimization methods
A2: We thank the reviewer for this suggestion. We have added comparisons with SurgTPGS [R5] and SurgTPGS++ [R6] on reconstruction and language grounding. We were unable to obtain publicly available implementations of DeGenseGS and SurgSemGS when conducting these experiments and therefore focus on the two available methods. We evaluate all methods on the same 13 test scenes, following the reconstruction and language-grounding evaluation protocols in the main paper. SurgTPGS and SurgTPGS++ undergo per-scene optimization, while our model processes these unseen scenes without test-time optimization.
Reconstruction quality. Table R8 reports the additional reconstruction results. Our method achieves 26.29 dB PSNR, compared with 18.73 dB for SurgTPGS and 18.84 dB for SurgTPGS++, while reducing LPIPS to 0.185, compared with 0.570 and 0.558. We also obtain more accurate depth predictions, with RMSE of 1.043 cm, compared with 2.50 cm and 2.22 cm. These improvements demonstrate that our framework achieves better appearance reconstruction and depth accuracy on the evaluated surgical scenes without per-scene optimization. We also provide qualitative reconstruction comparisons in Figure R3 (click to view). Our method preserves finer appearance details and produces depth structures that more closely match the reference.
Table R8. Reconstruction comparisons with surgical scene optimization methods.
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | AbsRel ↓ | RMSE (cm) ↓ | δ₁ (%) ↑ |
|---|---|---|---|---|---|---|
| SurgTPGS | 18.734 | 0.546 | 0.570 | 0.253 | 2.497 | 61.571 |
| SurgTPGS++ | 18.837 | 0.552 | 0.558 | 0.226 | 2.219 | 65.525 |
| Endo4DLR (ours) | 26.294 | 0.789 | 0.185 | 0.097 | 1.043 | 88.804 |
Language grounding. Table R9 reports the additional grounding results. Our method achieves higher IoU across all four query levels, with mean IoU of 0.5442, compared with 0.4082 for SurgTPGS and 0.4254 for SurgTPGS++. We also obtain higher mean Dice, precision, and recall. These results demonstrate more accurate localization of the queried regions across different levels of language description. Figure R4 (click to view) provides qualitative language-grounding comparisons. Our predictions more closely match the annotated targets, with fewer missed regions and less background activation in the illustrated examples.
Table R9. Language-grounding comparisons. The four query-level columns report IoU. Mean metrics are averaged over the four query levels.
| Method | Tool | Tool+Act. | Tool+Pos. | Full | mIoU ↑ | mDice ↑ | mPrec. ↑ | mRec. ↑ |
|---|---|---|---|---|---|---|---|---|
| SurgTPGS | 0.4292 | 0.4207 | 0.3976 | 0.3853 | 0.4082 | 0.4473 | 0.5258 | 0.6547 |
| SurgTPGS++ | 0.4416 | 0.4708 | 0.4260 | 0.3632 | 0.4254 | 0.4761 | 0.5426 | 0.5991 |
| Endo4DLR (ours) | 0.5467 | 0.5509 | 0.5415 | 0.5379 | 0.5442 | 0.6393 | 0.6134 | 0.7527 |
These additional comparisons extend our evaluation to surgical-domain, scene-specific optimization methods. We achieve better reconstruction and language-grounding performance on the evaluated scenes while requiring no test-time optimization, further demonstrating the effectiveness of our unified online framework.
[R1] Lan et al. Stream3R: Scalable Sequential 3D Reconstruction with Causal Transformer. ICLR, 2026.
[R2] Lu et al. Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering. CVPR, 2024.
[R3] Wang et al. Z-Order Transformer for Feed-Forward Gaussian Splatting. CVPR, 2026.
[R4] Li et al. 4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models. CVPR, 2025.
[R5] Huang et al. SurgTPGS: Semantic 3D Surgical Scene Understanding with Text Promptable Gaussian Splatting. MICCAI, 2025.
[R6] Huang et al. SurgTPGS++: Text-Promptable Gaussian Splatting with Dense Features Aggregation for Semantic 4D Surgical Scene Understanding. Medical Image Analysis, 2026.
Follow-up on our responses
Dear Reviewer Df2q, as the discussion period is nearing its end, we would appreciate your feedback on our responses.
Following your suggestions, we have added reconstruction and language-grounding comparisons with SurgTPGS and SurgTPGS++, and clarified the technical differences from the prior methods you identified.
We would be happy to address any remaining questions.
Thank you for your time and consideration.
Response to Reviewer Df2q (Part 1)
Dear Reviewer Df2q, we appreciate your careful review and recognition of our problem formulation, rendering performance, and ablation results. Below, we clarify the technical differences from prior work and provide additional comparisons with surgical scene optimization methods.
Q1 / W2: Clarification on technical contributions and differences from existing methods
A1: Thank you for the questions. Endo4DLR targets online, language-queryable dynamic scene reconstruction from unposed endoscopic streams, jointly estimating cameras and the scene representation without per-scene optimization. The central technical contribution is a history-aware Gaussian representation that associates current observations with relevant past scene information, enabling its reuse for both reconstruction and language grounding as motion, occlusion, and viewpoint changes alter scene visibility. Built on Stream3R’s causal reconstruction backbone [R1], this representation introduces three complementary designs: anchors for temporal association and feature reuse, spatial organization for efficient Gaussian grouping and historical retrieval, and a shared language encoder for consistent supervision across scenes. Together, these designs enable historical feature reuse and consistent language supervision within a unified framework for online feed-forward scene understanding.
The different mechanism of anchors. Scaffold-GS [R2] optimizes anchors and local offsets for each scene to position Gaussians, and predicts their attributes from anchor features and viewing direction and distance. We construct anchors from incoming observations and use them to associate current Gaussian groups with historical scene descriptors. Spatial proximity and feature compatibility determine which descriptors are retrieved, and each group shares the retrieved features for Gaussian attribute and language prediction. Our anchors therefore provide a mechanism for reusing observations across time within online inference (Secs. 3.1–3.2 and Appendix B.2). Removing this memory mechanism while retaining the causal backbone reduces PSNR from 26.29 to 22.72 dB and mean IoU from 0.544 to 0.503 (Sec. 4.2, Table 3).
The spatial organization strategy of anchors. We group observations in 3D because neighboring image pixels can lie at different depths. Z-order is an established spatial ordering strategy. It provides approximate spatial neighborhoods for both current grouping and historical retrieval, avoiding repeated exact KNN searches (Secs. 3.1–3.2). Z-Order Transformer [R3] uses this serialization within a multi-view attention and pooling architecture. Our framework uses the ordering strategy for current Gaussian grouping and historical association, rather than adopting that Transformer architecture. Replacing this organization with raster-order grouping and Euclidean nearest-neighbor retrieval reduces PSNR from 26.29 to 24.89 dB and mean IoU from 0.544 to 0.514 (Sec. 4.2, Table 3), supporting this design choice.
Language compression. 4D LangSplat [R4] compresses text embeddings into 6-dimensional features with an autoencoder and fits a language field for each scene. We retain compact language features but train one shared encoder across training scenes for our feed-forward predictor. Independently trained scene-specific encoders can map the same description to different vectors, requiring the predictor to learn inconsistent targets across scenes. Our shared encoder provides consistent supervision and remains fixed when encoding queries for unseen scenes (Appendix B.3). This change improves mean IoU from 0.162 to 0.544 compared with per-scene encoding (Sec. 4.2, Table 4). Appendix C.2 further visualizes the resulting semantic organization across scenes.