acceptodds
Desk rejected at ICLR 2027

Query Anytime: Find When Before Where for Online 4D Point Cloud Segmentation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Query-driven 4D point cloud segmentation aims to segment target instances in dynamic 3D sequences based on natural language queries. Existing methods largely follow a query-first offline setting, where the query is given in advance and the input sequence is temporally bounded around the queried target. However, in real-world interactions, queries may arrive at arbitrary times while 3D observations continue to evolve. In such an online scenario, the query and its relevant observations are no longer temporally aligned in advance, requiring the model to establish their correspondence dynamically. We summarize this challenge as “when before where”: the model must first identify when the queried target is present in the evolving sequence, before determining where it should be segmented. Based on this principle, we propose QueryAnytime. For when, a Searchable Evidence Tokenization (SET) module is proposed to build a compact, query-independent evidence space for efficient anytime query grounding without revisiting dense history. For where, a TrackState module is designed to update a structured semantic-geometric target state, which maintains target identity and spatial continuity for instance-consistent segmentation over time. We further introduce Online4DKITTI, a dedicated benchmark for online query-driven 4D segmentation with arbitrary query timing and fine-grained 3D instance annotations. Experiments demonstrate that QueryAnytime substantially outperforms existing baselines in segmentation accuracy while maintaining acceptable inference latency. Code and benchmark will be released at: https://anonymous.4open.science/r/QueryAnytime-E979/

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Online 4D Point Cloud Segmentation, Language-Guided 3D Segmentation, Query Grounding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xgJPOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduce an important setting, that the query can arrive at any time, how to conduct 4d point cloud segmentation accordingly. And the method proposed by this paper is to compress the history point cloud representations, and retrieve relevant "3D frame" to conduct segmentation.

Critical strengths

The proposed task setting is interesting and practically meaningful, and the experimental results provide evidence of the framework's effectiveness.

The overall framework is reasonable and well-structured. Image-point features are used for query-driven temporal retrieval, while point-voxel features support fine-grained instance segmentation. This design effectively separates temporal localization from spatial segmentation and achieves a favorable trade-off between accuracy and computational efficiency.

Critical weaknesses

The paper appears to overstate its methodological contributions. The proposed framework essentially follows a retrieval-and-segmentation paradigm, rather than introducing a fundamentally new query-driven segmentation mechanism.

The proposed approach does not fully address the challenge of efficient historical representation management. Although SET compresses visual features for retrieval, the framework still needs to retain dense historical representations for subsequent segmentation.

A more compelling approach would enable query-agnostic compression of historical representations while preserving sufficient information for accurate segmentation upon query arrival.

Questions for authors

What are the specific advantages of the proposed framework over simply augmenting an existing 4D segmentation method with a standard retrieval-and-segmentation pipeline? Are there any architectural adaptations specifically designed for the anytime-query setting beyond the additional retrieval module? A comparison with such a baseline would help clarify the methodological contributions.

Is it possible to develop a more memory-efficient mechanism that compresses historical representations in a query-agnostic manner while preserving sufficient information for future segmentation? Currently, the framework appears to cache dense historical features continuously, which raises concerns about its scalability to long-duration streams and the extent to which it addresses the claimed online processing challenge.

Reviewer_9ZYXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces QueryAnytime for online language-guided 4D point cloud segmentation. Unlike existing methods that assume the query and relevant temporal frames are known beforehand, QueryAnytime allows queries to be prompted at arbitrary times. It first identifies the relevant temporal frames and then locate/segments the object. The authors also introduce the Online4DKITTI benchmark that is built on SemanticKITTI for evaluation.

Critical strengths
  • The problem formulation is clear and well motivated.
  • The reported improvements over existing baselines are substantial.
  • The new benchmark is a potentially useful contribution.
Critical weaknesses
  • The method description is difficult to follow. Figure 3 is cluttered, Equation (2) needs clarification, and Figure 1 contains apparent visual artifacts. The writing should be improved.
  • The baseline ST-IoU scores are extremely low (0.11–1.09%), raising concerns about the fairness and meaningfulness of the comparison.
  • The architecture and three-stage training procedure seem complicated, with limited justification for some design choices such as 32 SET tokens. More controlled ablations and stronger baselines are needed to justify the proposed pipeline.
  • Although the authors mention human verification of generated annotations, no quantitative quality assessment or annotation agreement statistics are provided. More qualitative examples, particularly ambiguous cases, would also be helpful.
  • The authors should include relevant 4D segmentation works such as 4D-StOP, Mask4Former, and 4D-Former.
Questions for authors
  • Why are the baseline ST-IoU scores so low? Could the authors include a stronger, jointly trained temporal-presence baseline rather than only a linear classifier on frozen features?
  • Do all baselines use RGB images? If not, is the comparison fair? Tables should mention the modalities used.
  • Which point and voxel encoders are used? Please specify their architectures and pretrained weights.
  • How was human verification conducted? Could the authors report annotation rejection rates, agreement statistics, and examples of ambiguous queries?
  • Could the authors report performance separately for queries arriving before, during, and after target visibility?
Reviewer_rCyyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies online query-driven 4D point cloud segmentation, where language queries may arrive before, during, or after the target’s appearance. QueryAnytime combines query-independent Searchable Evidence Tokenization (SET) for temporal presence detection with a semantic-geometric TrackState for consistent spatial segmentation. The authors also introduce Online4DKITTI. Experiments report substantial improvements over existing methods and their temporal-classifier adaptations, reaching 12.70% ST-IoU and 52.30% T-IoU.

Critical strengths
  1. The paper addresses a meaningful setting that decouples query arrival from target visibility. The accompanying benchmark provides a useful basis for evaluating historical retrieval and subsequent online segmentation.
  2. The architecture is well aligned with the task and achieves substantial improvements over the evaluated baselines. The TrackState ablation also demonstrates benefits beyond temporal activation alone.
Critical weaknesses
  1. The experiments do not adequately isolate SET’s specific contribution. Removing SET also removes temporal activation, so Table 2 primarily demonstrates the importance of presence detection. The NTC baselines are helpful, but they train only a linear classifier on frozen representations, whereas SET receives dedicated representation learning and distillation. Comparisons using the same spatial backbone and presence supervision, with simpler pooling or dense retrieval alternatives, are needed to establish the value of the proposed design.

  2. The evidence for scalable online operation is incomplete. Section 3.3 retains dense features for every frame alongside SET tokens, so historical storage still grows with sequence length. The latency measurements explicitly exclude pre-query encoding and CPU/disk I/O; these are legitimate accounting choices, but they do not establish the cost of sustained operation. Memory consumption and latency scaling with history length are particularly important given the paper’s emphasis on arbitrary-time historical queries.

Questions for authors
  1. Under matched backbone and supervision, how does SET compare with simpler pooling-based or dense-feature presence classifiers? What is the contribution of evidence distillation alone?
  2. Where are historical dense features stored, and what are the peak memory usage and query-response latency as history length increases? Please also specify the hardware and pre-query encoding cost.
  3. Do different query arrival times produce different final predictions for the same query–instance pair? Are historical and online state-update/reset rules identical? Results separated into before-, during-, and after-target arrivals would clarify this.
  4. Since I-F1 is a conditional point-level F1 rather than an identity-association metric, what evidence directly supports improved identity preservation across occlusions or similar-instance distractors?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.