acceptodds
Under review as a conference paper at ICLR 2027

Endo4DLR: Online Language-Aligned 4D Reconstruction with Persistent Gaussian Anchors

Abstract

Dynamic intraoperative scene understanding requires jointly recovering evolving 3D geometry and grounding language semantics in reconstructed scenes. Yet performing both tasks online from unposed endoscopic streams remains challenging due to camera motion, tissue deformation, and changing visibility. Existing methods either rely on per-scene optimization or focus primarily on geometric reconstruction, leaving this joint problem largely unexplored. In this work, we introduce , a fully feed-forward framework for online language-aligned 4D reconstruction that jointly predicts camera parameters and language-embedded Gaussians without per-scene optimization. Our key idea is to maintain persistent Gaussian anchors that connect current local observations with reusable historical scene information. In particular, we introduce adaptive observation queries that capture fine-grained details of the current frame and are organized into local 3D Gaussian groups. In addition, we introduce spatial anchors that retrieve and aggregate historical features to predict Gaussian attributes and language embeddings. By combining current geometric estimates with retrieved historical context, Endo4DLR adapts to scene evolution while maintaining temporal consistency in reconstruction and language alignment. Extensive experiments on multiple endoscopic datasets demonstrate that Endo4DLR outperforms state-of-the-art online methods in reconstruction, camera estimation, and language grounding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Surgical Scene Understanding, 4D Online Reconstruction, Language Grounding, Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Df2qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Endo4DLR, a feed-forward framework for online language-aligned 4D reconstruction from unposed endoscopic videos. It jointly predicts camera parameters and language-embedded Gaussians without per-scene optimization. Adaptive Gaussian queries and Z-order serialization enable local grouping, while persistent anchors retrieve historical features for Gaussian attribute and language embedding prediction. Experiments evaluate reconstruction and language grounding across multiple datasets.

Critical strengths

The problem formulation presented in this study is significant. The paper explores how to achieve online 4D reconstruction—aligned with linguistic information—from endoscopic video lacking camera pose data, without requiring per-scene optimization. By jointly estimating camera poses and constructing Gaussian primitives that incorporate linguistic information, this approach facilitates the understanding of dynamic surgical scenes and lends practical utility to the proposed framework.

The method demonstrates strong rendering performance across multiple datasets. On the five endoscopic datasets evaluated, Endo4DLR consistently outperforms an improved version of the Stream3R baseline model in terms of RGB rendering quality. Furthermore, ablation studies indicate that both adaptive queries and persistent Gaussian anchors contribute to performance gains, providing compelling evidence for the effectiveness of the proposed design.

Critical weaknesses

I think the main problems of this paper are as follows:

1: The paper claims to introduce the first fully feed-forward framework for online language-aligned dynamic scene reconstruction from unposed endoscopic videos. However, the experiments primarily compare against general-purpose online reconstruction methods developed for natural scenes. Although Table 2 includes DGD and 4D LangSplat as per-scene optimization baselines for language grounding, the reconstruction evaluation lacks a comparison with the representative per-scene endoscopic scene understanding, such as SurgTPGS++ [1],DeGenseGS [2] and SurgSemGS [3]. The absence of these analysis and experiments diminishes the contribution of this method to scene understanding.

2: Several key components have clear precedents, including the reconstruction backbone from Stream3R [4], anchor-based Gaussian organization in Scaffold-GS [5] and Z-order serialization in Z-Order Transformer [6]. Language feature compression also follows the basic approach of 4D LangSplat [7]. The paper integrates these ideas but does not sufficiently clarify the substantive differences between these designs and existing mechanisms.

[1] Huang, Y., Bai, L., Cui, B., Yuan, K., Wang, G., Hoque, M. I., ... & Ren, H. (2026). SurgTPGS++: Text-promptable Gaussian Splatting with dense features aggregation for semantic 4D surgical scene understanding. Medical Image Analysis, 104318. [2] Wang, Y., Kang, B., Yang, S., & Jin, Y. (2026, September). DeGenseGS: Geometrically and Semantically Decoupled Surgical Scene Understanding in 4D Gaussian Splatting. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 112-122). Cham: Springer Nature Switzerland. [3] Liu, L., Gu, Q., Feng, Y., Hu, L., Nguyen, A., & Huang, B. SurgSemGS: Semantic and Contrastive Gaussian Splatting for Dynamic Surgical Scene Understanding. In CREATE: Clinical-driven Robotics and Embodied AI TEchnology. [4] Lan, Y., Luo, Y., Hong, F., Zhou, S., Chen, H., Lyu, Z., ... & Pan, X. (2026, April). Stream3r: Scalable sequential 3d reconstruction with causal transformer. In International Conference on Learning Representations (Vol. 2026, pp. 42746-42768). [5] Lu, T., Yu, M., Xu, L., Xiangli, Y., Wang, L., Lin, D., & Dai, B. (2024, June). Scaffold-gs: Structured 3d gaussians for view-adaptive rendering. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 20654-20664). IEEE. [6] Wang, C., Liu, L., Jiang, W., & Xu, D. (2026). Z-Order Transformer for Feed-Forward Gaussian Splatting. arXiv preprint arXiv:2605.13465. [7] Li, W., Zhou, R., Zhou, J., Song, Y., Herter, J., Qin, M., ... & Pfister, H. (2025, June). 4d langsplat: 4d language gaussian splatting via multimodal large language models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 22001-22011). IEEE.

Questions for authors
  1. Please clarify the fundamental technical innovations of the proposed method compared to existing methods. For instance, anchor-based Gaussian models and Z-order organization are existing techniques.

  2. Please analyze and compare the reconstruction and semantic segmentation quality against "scene-based optimization methods", such as SurgTPGS++, rather than limiting comparisons to feed-forward baseline methods. If such a comparison is not feasible, please explain why.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Df2q (Part 1)

Dear Reviewer Df2q, we appreciate your careful review and recognition of our problem formulation, rendering performance, and ablation results. Below, we clarify the technical differences from prior work and provide additional comparisons with surgical scene optimization methods.


Q1 / W2: Clarification on technical contributions and differences from existing methods

A1: Thank you for the questions. Endo4DLR targets online, language-queryable dynamic scene reconstruction from unposed endoscopic streams, jointly estimating cameras and the scene representation without per-scene optimization. The central technical contribution is a history-aware Gaussian representation that associates current observations with relevant past scene information, enabling its reuse for both reconstruction and language grounding as motion, occlusion, and viewpoint changes alter scene visibility. Built on Stream3R’s causal reconstruction backbone [R1], this representation introduces three complementary designs: anchors for temporal association and feature reuse, spatial organization for efficient Gaussian grouping and historical retrieval, and a shared language encoder for consistent supervision across scenes. Together, these designs enable historical feature reuse and consistent language supervision within a unified framework for online feed-forward scene understanding.

The different mechanism of anchors. Scaffold-GS [R2] optimizes anchors and local offsets for each scene to position Gaussians, and predicts their attributes from anchor features and viewing direction and distance. We construct anchors from incoming observations and use them to associate current Gaussian groups with historical scene descriptors. Spatial proximity and feature compatibility determine which descriptors are retrieved, and each group shares the retrieved features for Gaussian attribute and language prediction. Our anchors therefore provide a mechanism for reusing observations across time within online inference (Secs. 3.1–3.2 and Appendix B.2). Removing this memory mechanism while retaining the causal backbone reduces PSNR from 26.29 to 22.72 dB and mean IoU from 0.544 to 0.503 (Sec. 4.2, Table 3).

The spatial organization strategy of anchors. We group observations in 3D because neighboring image pixels can lie at different depths. Z-order is an established spatial ordering strategy. It provides approximate spatial neighborhoods for both current grouping and historical retrieval, avoiding repeated exact KNN searches (Secs. 3.1–3.2). Z-Order Transformer [R3] uses this serialization within a multi-view attention and pooling architecture. Our framework uses the ordering strategy for current Gaussian grouping and historical association, rather than adopting that Transformer architecture. Replacing this organization with raster-order grouping and Euclidean nearest-neighbor retrieval reduces PSNR from 26.29 to 24.89 dB and mean IoU from 0.544 to 0.514 (Sec. 4.2, Table 3), supporting this design choice.

Language compression. 4D LangSplat [R4] compresses text embeddings into 6-dimensional features with an autoencoder and fits a language field for each scene. We retain compact language features but train one shared encoder across training scenes for our feed-forward predictor. Independently trained scene-specific encoders can map the same description to different vectors, requiring the predictor to learn inconsistent targets across scenes. Our shared encoder provides consistent supervision and remains fixed when encoding queries for unseen scenes (Appendix B.3). This change improves mean IoU from 0.162 to 0.544 compared with per-scene encoding (Sec. 4.2, Table 4). Appendix C.2 further visualizes the resulting semantic organization across scenes.

AuthorsOpenReview ↗

Response to Reviewer Df2q (Part 2)

Q2 / W1: Additional comparisons with surgical per-scene optimization methods

A2: We thank the reviewer for this suggestion. We have added comparisons with SurgTPGS [R5] and SurgTPGS++ [R6] on reconstruction and language grounding. We were unable to obtain publicly available implementations of DeGenseGS and SurgSemGS when conducting these experiments and therefore focus on the two available methods. We evaluate all methods on the same 13 test scenes, following the reconstruction and language-grounding evaluation protocols in the main paper. SurgTPGS and SurgTPGS++ undergo per-scene optimization, while our model processes these unseen scenes without test-time optimization.

Reconstruction quality. Table R8 reports the additional reconstruction results. Our method achieves 26.29 dB PSNR, compared with 18.73 dB for SurgTPGS and 18.84 dB for SurgTPGS++, while reducing LPIPS to 0.185, compared with 0.570 and 0.558. We also obtain more accurate depth predictions, with RMSE of 1.043 cm, compared with 2.50 cm and 2.22 cm. These improvements demonstrate that our framework achieves better appearance reconstruction and depth accuracy on the evaluated surgical scenes without per-scene optimization. We also provide qualitative reconstruction comparisons in Figure R3 (click to view). Our method preserves finer appearance details and produces depth structures that more closely match the reference.

Table R8. Reconstruction comparisons with surgical scene optimization methods.

MethodPSNR ↑SSIM ↑LPIPS ↓AbsRel ↓RMSE (cm) ↓δ₁ (%) ↑
SurgTPGS18.7340.5460.5700.2532.49761.571
SurgTPGS++18.8370.5520.5580.2262.21965.525
Endo4DLR (ours)26.2940.7890.1850.0971.04388.804

Language grounding. Table R9 reports the additional grounding results. Our method achieves higher IoU across all four query levels, with mean IoU of 0.5442, compared with 0.4082 for SurgTPGS and 0.4254 for SurgTPGS++. We also obtain higher mean Dice, precision, and recall. These results demonstrate more accurate localization of the queried regions across different levels of language description. Figure R4 (click to view) provides qualitative language-grounding comparisons. Our predictions more closely match the annotated targets, with fewer missed regions and less background activation in the illustrated examples.

Table R9. Language-grounding comparisons. The four query-level columns report IoU. Mean metrics are averaged over the four query levels.

MethodToolTool+Act.Tool+Pos.FullmIoU ↑mDice ↑mPrec. ↑mRec. ↑
SurgTPGS0.42920.42070.39760.38530.40820.44730.52580.6547
SurgTPGS++0.44160.47080.42600.36320.42540.47610.54260.5991
Endo4DLR (ours)0.54670.55090.54150.53790.54420.63930.61340.7527

These additional comparisons extend our evaluation to surgical-domain, scene-specific optimization methods. We achieve better reconstruction and language-grounding performance on the evaluated scenes while requiring no test-time optimization, further demonstrating the effectiveness of our unified online framework.


[R1] Lan et al. Stream3R: Scalable Sequential 3D Reconstruction with Causal Transformer. ICLR, 2026.

[R2] Lu et al. Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering. CVPR, 2024.

[R3] Wang et al. Z-Order Transformer for Feed-Forward Gaussian Splatting. CVPR, 2026.

[R4] Li et al. 4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models. CVPR, 2025.

[R5] Huang et al. SurgTPGS: Semantic 3D Surgical Scene Understanding with Text Promptable Gaussian Splatting. MICCAI, 2025.

[R6] Huang et al. SurgTPGS++: Text-Promptable Gaussian Splatting with Dense Features Aggregation for Semantic 4D Surgical Scene Understanding. Medical Image Analysis, 2026.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer Df2q, as the discussion period is nearing its end, we would appreciate your feedback on our responses.

Following your suggestions, we have added reconstruction and language-grounding comparisons with SurgTPGS and SurgTPGS++, and clarified the technical differences from the prior methods you identified.

We would be happy to address any remaining questions.

Thank you for your time and consideration.

Reviewer_dqJPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Endo4DLR builds on Stream3R and adds three things. (i) Tile-based queries with small learned offsets (±3 px) are lifted to 3D via a predicted pointmap. (ii) The queries are grouped into anchors by Z-order (Morton) serialization. (iii) A "persistent anchor" memory stores and is queried by spatial and feature compatibility to enrich Gaussian attribute and language decoding. Language is supervised by regressing a 6-D compressed BioBERT embedding of instrument descriptions. It improves over Stream3R and other online baselines in rendering, depth, pose, and instrument grounding on 13 unseen test scenes.

Critical strengths

Relevant, well-scoped problem with a complete system. It covers camera, geometry, appearance, and language in one causal feed-forward pass. Training details, ablations on group size and density, and per-dataset tables are provided.

The gains over the strongest baseline are large and consistent in appearance and in language IoU.

Critical weaknesses

No temporal-consistency metric is reported, although this is a key contribution. Rendering is evaluated on the input frame itself using only current-frame Gaussians, so the PSNR measures re-encoding of the input, not a persistent 4D scene. The camera head is not connected to the anchor memory, there is no pose ablation, and pose gains are tiny (StereoMIS ATE 1.332→1.306 cm; SCARED RPE_r 0.229→0.230). The evidence does not match the central claims.

The protocol limits attribution. Only Stream3R is a meaningful baseline, and Endo4DLR is initialized from it. It is unclear whether Stream3R was fine-tuned with the same losses and trainable parameters, and no capacity-matched control exists.

Questions for authors

Since temporal consistency is a stated contribution, please report at least one direct metric, e.g. Temporal Alignment Error for depth (as in Video Depth Anything) or warped-mask mIoU (mTC-style, Varghese et al. 2020), and compare with and without the anchor memory.

Was Stream3R fine-tuned on your split with the same losses and trainable parameters?

Queries come from the training vocabulary and a 6-D autoencoder. How does the model handle paraphrased or unseen phrases?

4 comments
AuthorsOpenReview ↗

Response to Reviewer dqJP (Part 1)

Dear Reviewer dqJP, thank you for recognizing our well-scoped problem, complete causal framework, and consistent gains in rendering and grounding. Below, we address your concerns with clarifications and additional experiments.


W1 / Q1: Temporal consistency, scene representation across time, and camera estimation

A1: We thank the reviewer for this important comment. We have added temporal-consistency and camera-pose ablations and clarify how our framework represents the scene across time.

Temporal consistency. Following your suggestion, we evaluate the full model and its counterpart trained without anchor memory using both warped-mask mIoU (mTC-style) [R1] for language grounding and Temporal Alignment Error (TAE) [R2] for depth prediction. We apply the same evaluation protocol to both models. As shown in Table R4, anchor memory increases mTC from 73.329% to 75.354%, with improvements across all four query levels, and reduces TAE from 1.435 to 1.108, showing improved temporal consistency in both grounding and depth prediction. These results complement the grounding IoU improvement from 0.503 to 0.544 in Sec. 4.2, Table 3 of the main paper. The qualitative ablation in the main paper also shows responses to unrelated instruments without anchor memory, whereas the full model more consistently localizes the queried target across successive frames.

Table R4. Temporal consistency with and without anchor memory. The four query-level columns report warped-mask IoU (%), and mTC is their mean.

ModelTool ↑Tool+Act. ↑Tool+Pos. ↑Full ↑mTC (%) ↑TAE ↓
w/o anchor memory72.95272.52773.41074.42773.3291.435
w/ anchor memory74.32676.30574.61576.17175.3541.108

Scene representation across time. Our framework maintains persistent scene information in the anchor states and uses it to predict the Gaussian representation at each time step. As described in Sec. 3.2 and Appendix B.2, these states retain spatial information and feature descriptors across frames, which we retrieve and update as new observations arrive. Using current-frame Gaussians for rendering therefore does not imply that we reconstruct each frame independently. Their attribute and language predictions incorporate historical descriptors retained from earlier observations. Input-view PSNR evaluates how faithfully the resulting Gaussian representation reproduces the observed scene appearance. To evaluate the contribution of the persistent states, we compare models with and without anchor memory. Retaining memory improves PSNR from 22.72 to 26.29 dB in Sec. 4.2, Table 3, while the additional evaluations in Table R4 show improved grounding and depth consistency across frames. These results demonstrate that the historical information maintained by our framework benefits both current-scene reconstruction and temporal consistency.

Camera estimation. We jointly predict cameras and language-embedded Gaussians so that our framework can reconstruct and query scenes from unposed video. As described in Sec. 3, the camera head and Gaussian decoder use shared backbone features, while anchor memory provides historical information for Gaussian attribute and language prediction. To examine its effect within the jointly trained framework, we additionally evaluate camera accuracy for the full model and its separately trained counterpart without anchor memory under the original pose-evaluation protocol. As shown in Table R5, the full model reduces average ATE, translational RPE, and rotational RPE by 15.0%, 17.2%, and 9.2%, respectively, compared with the model trained without anchor memory. This ablation demonstrates that including anchor memory in the jointly trained framework also benefits camera estimation.

Regarding the small gains you noted, the improvements over Stream3R vary across datasets and metrics. On SCARED, we achieve comparable rotational RPE while reducing ATE from 1.502 to 1.229 cm and translational RPE from 0.571 to 0.401 cm (Appendix C.1). These improvements demonstrate the effectiveness of our framework in estimating camera motion while reconstructing language-queryable scenes from unposed video.

Table R5. Camera-pose ablation. Results are averaged over StereoMIS and SCARED. ATE and translational RPE are in cm, and rotational RPE is in degrees.

ModelATE ↓RPE trans. ↓RPE rot. ↓
w/o anchor memory1.4920.4950.185
w/ anchor memory1.2680.4100.168
AuthorsOpenReview ↗

Response to Reviewer dqJP (Part 2)

W2 / Q2: Clarification on training protocol and model capacity

A2: We thank the reviewer for requesting clarification of the training protocol and model capacity.

Training protocol. Yes. We fine-tune Stream3R on the same training split as Endo4DLR, using the same losses and weights for the shared geometry, camera, RGB-rendering, and language-prediction tasks. Both models start from the pretrained Stream3R backbone and follow the same freezing policy: the DINO image encoder is frozen, while the causal Transformer, camera head, geometry heads, and Gaussian/language prediction modules are trained. The optimizer, learning rate, training duration, and rendering-loss schedule are also identical. As described in the Baselines paragraph of the main paper (Sec. 4), the compared Stream3R model includes a DPT-style Gaussian/language head and is fine-tuned with the corresponding supervision. It predicts Gaussians densely per pixel, whereas Endo4DLR uses adaptive queries and anchor-based prediction. The two models use the same training annotations, rasterizer, and language embedding space, with training details provided in Appendix B.3.

Trainable parameters and model capacity. As shown in Table R6, Endo4DLR uses approximately 1.47% fewer trainable parameters than the adapted Stream3R baseline. Both models retain the same reconstruction backbone, with the difference arising from the Gaussian/language prediction head we designed. Our gains over Stream3R therefore do not depend on a larger trainable parameter count. Our ablations also include a comparison at identical parameter capacity. In Sec. 4.2, Table 3 of the main paper, the w/o Z-order variant retains the same learnable modules and loss configuration as the full model, including adaptive queries and anchor memory. It changes only the organization and retrieval strategy, replacing Z-order grouping and retrieval with raster-order grouping and Euclidean nearest-neighbor retrieval. Despite having the same parameter count, this variant reduces PSNR from 26.29 to 24.89 dB and mean IoU from 0.544 to 0.514. This controlled comparison demonstrates that the organization and retrieval strategy contributes to performance independently of parameter count.

Table R6. Trainable parameters of Endo4DLR and the adapted Stream3R baseline.

ModelTrainable parameters (M)
Adapted Stream3R919.433
Endo4DLR905.906

Q3: Clarification on generalization to paraphrased queries

A3: We thank the reviewer for this insightful question. Our model accepts free-form text queries without requiring a match to a predefined vocabulary or sentence list. We encode each query using the frozen BioBERT-based sentence encoder and shared autoencoder, then compare its embedding with the rendered language features. As described in Appendix B.3, we compress the 768-dimensional embeddings into 6-dimensional features to reduce feature storage and rendering costs. This compression does not impose a fixed query format. We evaluate four query levels to assess how the model responds to tool identity, actions, spatial positions, and their combinations (Sec. 4 of the main paper).

Our original evaluation uses test queries that are absent from the training descriptions. To further evaluate robustness to variations in wording, we paraphrase the 'Tool+Act' and 'Full' queries by changing their sentence structure and word order. We evaluate these paraphrases across all 13 test scenes under the same evaluation protocol, changing only the query text. As shown in Table R7, grounding performance remains stable. Tool+Act IoU changes from 0.5509 to 0.5514, while Full IoU changes from 0.5379 to 0.5370. These results demonstrate robustness to the tested variations in sentence structure and word order.

Table R7. Grounding IoU with original and paraphrased test queries.

Query levelOriginalParaphrased
Tool+Act0.55090.5514
Full0.53790.5370

[R1] Varghese et al. Unsupervised Temporal Consistency Metric for Video Segmentation in Highly-Automated Driving. CVPR Workshops, 2020.

[R2] Chen et al. Video Depth Anything: Consistent Depth Estimation for Super-Long Videos. CVPR, 2025.

The authors have addressed most of my concerns and questions, and I'm inclined to keep my weak accept rating. In this round, when the authors removed the historical information, PSNR dropped by 3.5 dB. For single-frame reconstruction, that shows the historical features clearly play a big role. I'd like the authors to explain why in the revised version.

Dear Reviewer dqJP, we sincerely thank you for your continued support and helpful suggestion. We will clarify in Sec. 4.2 why historical features benefit current-view reconstruction, and incorporate the additional experiments and relevant clarifications from our rebuttal into the appendix. We appreciate your constructive feedback!

Reviewer_ZznfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Endo4DLR, a feed-forward framework for online, language-aligned 4D reconstruction from unposed endoscopic videos. It combines current observations with historical features through persistent Gaussian anchors, jointly estimating camera parameters, evolving geometry, and language semantics without per-scene optimization. Experiments show improved reconstruction, camera estimation, and language grounding over existing online methods.

Critical strengths
  • The paper is generally well-written, and the motivation and overall framework are easy to follow.
  • The experiments are extensive, covering multiple endoscopic datasets and evaluating reconstruction, camera estimation, and language grounding.
  • Jointly addressing dynamic reconstruction and language alignment is a valuable direction for intraoperative scene understanding.
  • The persistent Gaussian anchors provide an intuitive way to reuse historical information while incorporating new observations.
Critical weaknesses
  • The novelty needs clearer justification, particularly how the proposed queries and spatial anchors differ from existing memory-based reconstruction and Gaussian prediction methods.
  • Historical feature reuse may introduce stale information under substantial tissue deformation or prolonged occlusion. More targeted evaluation would help establish robustness in these conditions.
  • The practical value of language alignment needs stronger validation, particularly for clinically meaningful queries involving similar-looking tissues or changing anatomical regions.
  • The online setting requires more information on inference speed, memory consumption, and how computational costs grow with sequence length.
  • Detailed ablations are needed to distinguish the contributions of observation queries, spatial anchors, and historical feature aggregation.
Questions for authors
  • How do the proposed observation queries and spatial anchors differ from existing memory-based reconstruction methods?
  • How robust is historical feature reuse under substantial tissue deformation, prolonged occlusion?
  • What are the inference speed and memory requirements, and how do they scale with video length while maintaining reconstruction and language-grounding performance?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Zznf (Part 1)

Dear Reviewer Zznf, thank you for your time and constructive comments. We appreciate your recognition of our clear presentation and of joint reconstruction and language alignment as a “valuable direction for intraoperative scene understanding.” We address your concerns point by point below.


W1 / Q1: Clarification on technical novelty and differences from existing methods

A1: The core novelty is that we are the first to propose a unified online framework capable of jointly estimating cameras and reconstructing language-queryable dynamic scenes from unposed endoscopic streams, without per-scene optimization. In this setting, motion, occlusion, and viewpoint changes affect which scene details are visible at each time step. Reusing earlier observations can complement incomplete current observations, but requires associating historical information with the scene as it changes. We design adaptive observation queries to capture current local detail and persistent spatial anchors to associate these observations with relevant historical features for Gaussian attribute and language prediction.

Adaptive observation queries. Many recent feed-forward Gaussian methods start from pixel-aligned predictions, associating image pixels with Gaussian primitives. For example, Z-Order Transformer [R1] aggregates pixel-level Gaussians through spatial serialization and attention to obtain a more compact representation. We instead use adaptive observation queries to select sampling locations before Gaussian construction, refining their image-space positions using local features to better capture fine structures and boundaries.

Our queries serve as local observation units for constructing the current scene representation. Each query samples a 3D position from the predicted pointmap and the corresponding local features for Gaussian prediction (Sec. 3.1). We therefore construct Gaussians from feature-adaptive observations rather than assigning predictions to every image pixel. Replacing adaptive queries with fixed, uniformly distributed queries while retaining the remaining framework reduces PSNR from 26.29 to 25.64 dB and mean IoU from 0.544 to 0.495 (Sec. 4.2, Table 3), demonstrating the benefit of adapting observation locations for reconstruction and grounding.

The mechanism of spatial anchors. Existing streaming reconstruction methods retain historical information through recurrent states, as in CUT3R [R2], or causal attention over cached features, as in Stream3R [R3]. We design spatial anchors to explicitly associate current Gaussian groups with historical scene descriptors. Because we construct groups from each incoming observation, the same scene region may be represented by different queries and groups across frames. Each anchor summarizes a group’s position and features, which we use to retrieve compatible historical descriptors through spatial proximity and feature similarity. We then share the retrieved context among the group’s queries for Gaussian attribute and language prediction. This makes the anchor a local unit for historical association and context sharing, reducing repeated retrieval across individual queries. We also retain query-specific features in language prediction to preserve semantic distinctions within each group. Sec. 3.2 and Appendix B.2 describe this mechanism and how the persistent descriptors are updated as new observations arrive.

Our spatial anchors also serve a different role from those in Scaffold-GS [R4], a representative anchor-based Gaussian model that optimizes scene-specific anchors and local offsets to generate Gaussians for view-adaptive rendering. We construct anchors from incoming observations and use them to associate current Gaussian groups with persistent scene descriptors during online inference. We evaluate this mechanism by disabling anchor memory while retaining the causal backbone and its temporal features. PSNR decreases from 26.29 to 22.72 dB, and mean IoU decreases from 0.544 to 0.503 (Sec. 4.2, Table 3). These results demonstrate that explicit historical association through spatial anchors improves reconstruction and grounding beyond the temporal information already provided by the backbone.

Overall, we develop a unified framework for online reconstruction and language grounding from unposed endoscopic video, with joint camera estimation. We design adaptive queries to capture current local observations and spatial anchors to associate them with historical scene information for Gaussian prediction. The corresponding ablations demonstrate the effectiveness of these designs for reconstruction and grounding.

AuthorsOpenReview ↗

Response to Reviewer Zznf (Part 2)

W2 / Q2: Robustness of historical feature reuse

A2: Thank you for raising this concern.

Historical feature reuse under scene changes. Historical descriptors do not contribute unconditionally to current predictions. As described in Sec. 3.2 and Appendix B.2, we associate current anchors with stored descriptors using both spatial proximity and feature compatibility. The aggregation weights account for association reliability, reducing the historical contribution for uncertain matches. An unmatched anchor receives zero historical context. We update accepted historical descriptors using current observations, initialize new entries for unmatched current anchors, and prioritize newly observed and recently updated anchors during memory retention. These mechanisms allow current observations to determine which historical information is reused and how strongly it contributes, reducing reliance on descriptors that no longer match the scene.

We have examined the contribution of historical information through the anchor-memory ablation in Sec. 4.2, Table 3. Removing anchor memory while retaining the causal backbone reduces mean grounding IoU from 0.544 to 0.503. The consecutive-frame comparison in Fig. 4 also includes this ablation and shows activations on unrelated instruments without anchor memory, whereas the model with anchor memory localizes the queried targets more accurately. Furthermore, Appendix C.3, Fig. 11 reports frame-wise grounding IoU across unseen sequences, providing a sequence-level assessment as scene configurations and target visibility change.

Targeted evaluation under deformation and prolonged invisibility. To further address your concern, we conduct additional targeted comparisons to evaluate the effectiveness and robustness of historical feature reuse under these conditions. We examine a target reappearing after an extended period outside the field of view, and surgical manipulation intervals involving tissue deformation and target motion during grasping, cutting, and retraction. Table R1 compares grounding IoU with and without anchor memory over these intervals.

  • Target reappearance after prolonged invisibility. On StereoLung P1_5, the stapler leaves the field of view for 105 frames, from frame 106 through 210, before returning at frame 211. Over frames 211–240, immediately following the target’s reappearance, the model with anchor memory achieves 0.752 IoU, compared with 0.015 without memory. Figure R1 (click to view) shows that the model with anchor memory localizes the returning stapler, while the model without memory frequently responds to other instruments and background regions. These results show accurate grounding after an extended interruption in target visibility while the anchor memory continues to operate.

  • Tissue deformation during manipulation. Across the five manipulation intervals, the model with anchor memory achieves higher mean grounding IoU than the model without anchor memory. For tissue retraction on StereoLung P1_5, IoU increases from 0.426 to 0.699. Figure R2 (click to view) illustrates the grasper moving as it retracts and deforms the tissue. The model with anchor memory more accurately follows the queried target, whereas the model without memory more frequently includes unrelated instruments and surrounding tissue. The observed gains extend across changing target positions and surgical interactions, demonstrating more accurate grounding in the evaluated dynamic conditions.

Table R1. Grounding IoU with and without anchor memory after target reappearance and during tissue manipulation. We average IoU over annotated frames within each listed interval.

EventSequenceQueryEvaluated framesw/o anchor memoryw/ anchor memory
Target reappearanceStereoLung P1_5Stapler is idle on the left211–2400.0150.752
Tissue graspingEndoNeRF thinBipolar forceps is grasping tissue10–1200.3490.832
Tissue manipulationEndoVis17 S_08Bipolar forceps is manipulating7–2240.0580.660
Tissue cuttingStereoLung P3_1Monopolar hook is cutting tissue33–2010.5340.787
Tissue retractionStereoLung P1_5Oval grasper is retracting tissue512–7010.4260.699
Tissue graspingStereoLung P3_5Oval grasper is grasping tissue125–4340.5830.745

Overall, anchor memory improves grounding IoU across all six events, with qualitative comparisons showing more accurate target localization. These results demonstrate its effectiveness during tissue deformation and after prolonged target invisibility.

AuthorsOpenReview ↗

Response to Reviewer Zznf (Part 3)

W3: Practical value and scope of language grounding

A3: Thank you for raising the practical value of language alignment. Our query design was informed by consultation with surgeons, with the aim of locating relevant targets through short descriptions as an endoscopic video stream arrives.

Clinically motivated queries and their evaluation. Surgical scenes can contain multiple instruments of the same type or similar appearance, making a tool name alone insufficient to specify the intended target. Descriptions of actions and spatial locations provide additional information to distinguish these targets. We therefore evaluate four query levels, covering tool identity, action, position, and their combinations, to examine whether the reconstructed representation responds to these distinctions. Our framework supports this localization online from unposed endoscopic video, without camera parameters supplied in advance or per-scene optimization.

The original paper evaluates this capability both quantitatively and qualitatively. Table 2 quantitatively evaluates grounding across all four query levels. Fig. 4 illustrates how action and position descriptions distinguish the intended target when multiple instruments are present. Our model more selectively localizes the forceps specified by left/right position and activity, while the compared predictions also activate unrelated instruments. Appendix C.3, Figs. 11–12 further present frame-wise IoU and qualitative comparisons across unseen sequences. These results demonstrate the ability to localize targets using their described actions and spatial relationships as the scene evolves.

Extension to tissues and anatomical structures. We agree that queries involving similar-looking tissues and changing anatomical regions are valuable extensions. As discussed in Appendix D, our current evaluation focuses on instruments and descriptions of their actions and locations because suitable datasets combining dynamic reconstruction data with language annotations for tissues and anatomical structures are still lacking.

Our language-embedded Gaussian representation is not restricted to instrument categories and can seamlessly accommodate these additional semantic targets with corresponding language supervision. Extending this semantic coverage does not require changing the underlying Gaussian representation or text-query mechanism. Evaluating this extension requires appropriately annotated data, particularly to assess distinctions between visually similar tissues and anatomical regions. We identify richer semantic annotations and broader scene-entity coverage as future directions in the paper.


W4 / Q3: Inference efficiency and scaling with sequence length

A4: We additionally evaluate inference cost and prediction quality on a 266-frame StereoLung P1_6 sequence at 518 × 280 resolution, using a single NVIDIA RTX A6000. We retain the original inference configuration, with the KV cache storing the most recent five frames and anchor memory updated as new observations arrive. From frame 10 onward, peak GPU memory at the reported checkpoints remains approximately 6.39 GB, with forward latency between 222.1 and 224.2 ms. From 100 to 266 frames, cumulative PSNR remains around 25.5 dB, depth RMSE ranges from 0.300 to 0.325 cm, and grounding IoU ranges from 0.771 to 0.794. These results show approximately stable per-frame inference cost while maintaining reconstruction and grounding quality over the evaluated sequence.

Table R2. Inference cost and prediction quality at increasing sequence lengths. Latency and peak allocated GPU memory are measured for the forward pass at the NN-th frame, excluding rendering. Quality metrics are averaged over the first NN frames. Grounding IoU uses annotated frames for “Oval grasper is grasping gauze on the left.”

Metric5 frames10 frames50 frames100 frames150 frames200 frames266 frames
Forward latency (ms) ↓201.9222.1222.4222.6224.0223.4224.2
Peak GPU memory (GB) ↓6.106.396.396.396.396.396.39
PSNR (dB) ↑27.3327.1925.8325.6125.5825.4725.50
Depth RMSE (cm) ↓0.3460.3720.3180.3250.3000.3020.312
Grounding IoU ↑0.8180.8180.8180.7850.7880.7940.771

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.