acceptodds
Under review as a conference paper at ICLR 2027

Just-In-Time Scene Graph Growth: Combating Perceptual Saturation in Long-Horizon Robotics

Abstract

While 3D Scene Graphs (3DSGs) provide crucial structured representations for embodied agents, conventional Ahead-of-Time, ”build-everything-then-filter” pipelines conflict with the real-time, low-latency demands of edge platforms, inducing a perceptual saturation effect via severe observation redundancy. To resolve this, we present JITOMA (Just-In-Time On-demand Memory Activation), a closed-loop framework that unifies task reasoning, perception, and memory into a just-in-time growth process. Instead of exhaustively mapping the entire environment, JITOMA leverages a top-down task heatmap at the frontend to filter continuous observations, routing minimal streams to maintain a global foundation of low-cost, dormant anchors. Upon a cognitive query, the backend Large Language Model (LLM) parses the robotic intent to dynamically awaken task-relevant anchors, triggering expensive semantic operations such as dense node captioning exclusively within the activated local subgraph. To evaluate these dynamic capabilities and study perceptual saturation trade-offs, we introduce JITOMA-Bench, a benchmark for long-horizon task switching and complex intent grounding. Across JITOMA-Bench, JITOMA maintains only 1–6 active semantic objects and 0.25–0.28 s/frame across all tiers, showing that semantic computation remains bounded by current task demand rather than accumulated scene complexity.

Primary area:
applications to robotics, autonomy, planning
Keywords:
3D scene graph, just-in-time scene graph growth, perceptual saturation, long-horizon robotics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DjSsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

JITOMA builds task-conditioned 3D scene graphs by selectively storing observations and delaying caption generation until an object is needed. It maintains dormant anchors with visual and geometric information, retrieves candidates for explicit and implicit task targets, and captions and reranks those candidates. The authors also introduce JITOMA-Bench, based on three Clio scenes, to evaluate single-object grounding, task switching, and multi-object instructions. The experiments report improved grounding accuracy and lower average processing time compared with ConceptGraphs, ReasoningGraph, and Clio.

Critical strengths
  1. Deferring expensive semantic processing until it is needed is a useful direction for online scene graphs. The ablations in Table 2 separate the effects of task-guided observation admission and on-demand captioning. The budget-matched control is helpful: it shows that selecting observations based on task relevance works better than simply admitting fewer observations.

  2. The paper provides evidence for its motivation rather than relying only on illustrative examples. In the controlled distractor study, task-relevant content is held fixed while irrelevant objects are added. Latent-target coverage drops from 68.10% to 56.30%. This supports the argument that additional scene information can hurt the planner’s ability to identify objects implicitly required by a task.

Critical weaknesses
  1. The task-switching protocol does not test whether the system can recover information needed for an unexpected future task. Each Tier 2 task starts when its target first becomes visible and is evaluated ten frames after the target was last visible. The system therefore knows which object matters while observing it. A harder and more relevant case is a request that arrives after the object has left view, when that object was irrelevant to the previous task. Without this evaluation, it is unclear whether task-guided filtering sacrifices future-task recall. The experiments are also limited to grounding on three recorded scenes, so the claims about long-horizon robotic execution go beyond what is demonstrated.

  2. The efficiency evaluation leaves out important costs of on-demand processing. Average time per frame does not show how long the robot waits when a new task triggers captioning and reranking. Similarly, a small active graph does not imply a small total memory footprint: dormant anchors retain visual information and, after activation, captions and other attributes. The paper acknowledges this distinction, but does not report actual memory usage or explain what happens when the 512-node capacity is reached.

Questions for authors
  1. How does performance change when a task arrives after its target has left view and the target was irrelevant to the preceding task? A comparison with Clio and the budget-matched admission control would help establish whether the method preserves information for future requests.

  2. What is the latency from receiving a new task to returning a grounded subgraph, including parsing, captioning, and reranking?

  3. Could you reconcile the default Tier 2 results across Tables 1, 2, and 5? Averaging Table 1 across scenes gives approximately 24.3 IoU and 59.4 mR@1. Table 2 reports 24.3 and 57.5, while Table 5 reports 23.95 and 56.83. Were these obtained using different settings or runs?

  4. The paper describes FOUND-IT as retaining visual memory and resolving representations on demand. What benefit does JITOMA provide over that approach? A direct comparison, or a controlled comparison using shared components, would help clarify the contribution.

7 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/6)

W1: The task-switching protocol does not test whether the system can recover information needed for an unexpected future task. Each Tier 2 task starts when its target first becomes visible and is evaluated ten frames after the target was last visible. The system therefore knows which object matters while observing it. A harder and more relevant case is a request that arrives after the object has left view, when that object was irrelevant to the previous task. Without this evaluation, it is unclear whether task-guided filtering sacrifices future-task recall.

Question: How does performance change when a task arrives after its target has left view and the target was irrelevant to the preceding task? A comparison with Clio and the budget-matched admission control would help establish whether the method preserves information for future requests.

A: We thank the reviewer for raising this important concern. We agree that the original Tier 2 protocol does not fully evaluate whether task-irrelevant observations are preserved for unexpected future requests. To address this, we introduce a delayed task-arrival evaluation and compare JITOMA against Clio and our budget-matched admission control (D†).

Revised evaluation protocol. We preserve the original object-task ordering but delay all task activation and query events, such that each target has already left the field of view when its corresponding task is activated.

ProtocolTemporal sequence
Original Tier 2Activate task → Target observed → Query
Delayed Tier 2Target observed → Target leaves view → Activate task → Query

Thus, the target is observed before its corresponding task arrives and is unrelated to the preceding task. JITOMA must retrieve it from previously accumulated memory without having received its task during observation.

Results. We evaluate Clio, D†, and JITOMA under the delayed protocol across all three scenes. We also include JITOMA's original Tier 2 results for reference.

SceneMethodmIoUmR@1mR@3
CubicleClio (Delayed)22.566.770.4
D† (Delayed)15.347.252.7
JITOMA (Original)22.466.774.1
JITOMA (Delayed)21.963.074.1
OfficeClio (Delayed)18.145.252.4
D† (Delayed)12.931.045.2
JITOMA (Original)29.164.373.8
JITOMA (Delayed)29.061.973.8
ApartmentClio (Delayed)11.325.027.8
D† (Delayed)10.627.836.1
JITOMA (Original)21.447.252.8
JITOMA (Delayed)19.644.452.8

Delayed-arrival robustness. The delayed protocol has little impact on Clio: its average mIoU, mR@1, and mR@3 change from 17.5/43.3/49.8 in the original Tier 2 to 17.3/45.6/50.2. JITOMA experiences a modest reduction from 24.3/59.4/66.9 to 23.5/56.4/66.9, with mR@3 remaining unchanged across all three scenes. This suggests that although delayed activation removes task-conditioned heatmap evidence collected during the target's visibility, JITOMA can still recover historical targets through its persistent visual memory and accumulated evidence.

Comparison with budget-matched admission control. More importantly, JITOMA consistently outperforms D† under the delayed protocol. Averaged across the three scenes, JITOMA improves mIoU, mR@1, and mR@3 by 10.6, 21.1, and 22.2 percentage points, respectively. Although D† matches JITOMA's per-frame observation-admission budget, it uses task-independent admission and does not implement JITOMA's two complementary writing policies: task-guided heatmap selection during active tasks and novelty-preserving conservative writing during task-free intervals. In particular, JITOMA explicitly admits previously unseen tracks while suppressing redundant updates to existing tracks when no task is active. Therefore, matching the admission budget alone is insufficient to preserve useful information for future tasks; how observations are selected and maintained also matters.

AuthorsOpenReview ↗

Rebuttal by Authors (2/6)

W2: The efficiency evaluation leaves out important costs of on-demand processing. Average time per frame does not show how long the robot waits when a new task triggers captioning and reranking.

Question: What is the latency from receiving a new task to returning a grounded subgraph, including parsing, captioning, and reranking?

A: We thank the reviewer for pointing out that per-frame processing time does not fully characterize the latency of on-demand semantic processing. Importantly, JITOMA executes these operations asynchronously rather than as a sequential pipeline: backend captioning overlaps with continuous frontend observation.

Latency breakdown. We report the average processing time of each stage:

StageAverage latencyExecution
Task parsing1.9 sSynchronous upon task arrival
Batch captioning3.2 sAsynchronous, overlapping with frontend observation
LLM reranking2.5 sSynchronous upon query arrival

Consequently, these durations should not be added directly to estimate query latency. The 3.2 s captioning process runs in the background while the frontend continues processing observations. Once a grounding query arrives, the main additional waiting time is the 2.5 s reranking stage. Task parsing incurs a separate 1.9 s overhead upon task arrival.

Semantic richness and resource allocation. Unlike embedding-based approaches such as Clio [1], which primarily rely on visual-language features for semantic matching and clustering, JITOMA additionally provides explicit textual object descriptions and LLM-based semantic reranking. Generating these richer semantic representations inevitably introduces additional computation. However, JITOMA allocates captioning resources selectively: only shortlisted task-relevant candidates undergo expensive caption generation, rather than captioning all observed objects. Combined with asynchronous execution, this design keeps the additional semantic processing manageable without placing the entire captioning cost on the query-time critical path.

We will clarify this execution pipeline and include the stage-wise latency breakdown in the revised manuscript.

Reference:

[1] Maggio, D., Chang, Y., Hughes, N., Trang, M., Griffith, D., Dougherty, C., ... & Carlone, L. (2024). Clio: Real-time task-driven open-set 3d scene graphs. IEEE Robotics and Automation Letters, 9(10), 8921-8928.

AuthorsOpenReview ↗

Rebuttal by Authors (3/6)

W3: Similarly, a small active graph does not imply a small total memory footprint: dormant anchors retain visual information and, after activation, captions and other attributes. The paper acknowledges this distinction, but does not report actual memory usage or explain what happens when the 512-node capacity is reached.

A: We thank the reviewer for pointing out this distinction. We would like to clarify that the active-node count reported in our efficiency evaluation is intended to measure the semantic processing workload, rather than total memory consumption. JITOMA retains dormant anchors, including their visual representations, in memory, while expensive semantic attributes such as captions are generated on demand upon activation. Therefore, our efficiency claim primarily concerns reducing the number of nodes requiring semantic processing, rather than reducing the total memory footprint. We will clarify this distinction in the revised manuscript.

Global memory capacity. JITOMA uses a fixed global capacity of 512 nodes, including both ephemeral hypotheses and persistent dormant anchors. When the capacity is reached, ephemeral hypotheses are first removed in temporal order to reclaim space before evicting persistent anchors. If further eviction is necessary, a sliding-window policy removes the oldest dormant anchors, following the memory management strategy commonly adopted in prior 3D scene graph systems. More sophisticated long-term memory forgetting policies are beyond the scope of this work.

Reviewer_7UwZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents JITOMA, a task-driven framework for just-in-time 3D scene graph construction. It combines task-aware perception, lightweight dormant anchors, and on-demand semantic activation to reduce redundant computation. The authors also introduce JITOMA-Bench to evaluate object grounding, long-horizon task switching, and computational efficiency. Experiments demonstrate a reduced active semantic working set and improved processing efficiency while maintaining competitive grounding performance.

Critical strengths
  1. Practical problem and system design. The paper addresses the computational overhead of continuously maintaining detailed 3D scene graphs, which is relevant to long-horizon robotic perception.

  2. System-level evaluation. The proposed benchmark evaluates task switching and complex intent grounding, and the experiments demonstrate benefits in active semantic-state size and processing efficiency.

Critical weaknesses
  1. Limited methodological novelty. The main components—task-guided perception, selective semantic processing, and on-demand memory activation—are closely related to existing task-driven scene graph approaches, particularly Clio and FOUND-IT. The contribution appears primarily to be a system-level integration, while the distinct algorithmic advance remains limited.

  2. Limited downstream validation. The experiments mainly evaluate object grounding and computational efficiency rather than completed long-horizon robotic tasks. Consequently, the evidence does not fully establish whether the proposed framework improves navigation, manipulation, or overall task execution beyond more efficient scene graph processing.

Questions for authors
  1. Compared with FOUND-IT and Clio, what is the key algorithmic contribution beyond moving task-conditioned filtering and semantic activation earlier in the perception pipeline?

  2. Can the authors provide evidence that the reduced active semantic state improves downstream robotic task performance, rather than primarily reducing semantic processing cost?

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

W1: Limited methodological novelty. The main components—task-guided perception, selective semantic processing, and on-demand memory activation—are closely related to existing task-driven scene graph approaches, particularly Clio and FOUND-IT. The contribution appears primarily to be a system-level integration, while the distinct algorithmic advance remains limited.

Question: Compared with FOUND-IT and Clio, what is the key algorithmic contribution beyond moving task-conditioned filtering and semantic activation earlier in the perception pipeline?

A: We thank the reviewer for raising this important question. Although JITOMA shares the broader principle of task-driven scene representation with Clio [1] and FOUND-IT [2], the key distinction lies in what each method optimizes. Clio and FOUND-IT primarily focus on task-dependent representation granularity, whereas JITOMA focuses on task-conditioned resource allocation across perception, memory formation, and semantic activation.

Comparison with prior methods.

AspectClio [1]FOUND-IT [2]JITOMA
Primary focusTask-driven clusteringOn-demand representation granularityResource allocation
Online perceptionTask-agnostic primitivesTask-agnostic keyframesTask-guided admission
Persistent memory3D primitivesVisual keyframes and object cacheDormant anchors
On-demand processingClusteringObject extractionCaptioning and semantic activation
Explicit captionsNot a core stageNot a core stageGenerated and reused

Key algorithmic contribution. JITOMA connects two complementary decisions through persistent task-relevance evidence:

  • Frontend admission (Eq. (2)). Task-conditioned heatmaps select incoming observations before memory commitment, while task-free conservative writing preserves novel observations and suppresses redundant updates.
  • Backend activation (Eqs. (3)–(4)). Dormant anchors retain lightweight CLIP features. Upon task arrival, retrieval combines CLIP similarity with accumulated heatmap evidence, and only shortlisted candidates undergo DAM captioning and LLM verification. Reusable semantics are distilled into memory (Sec. 4.5).

Thus, task relevance governs both what evidence is retained and when expensive semantic computation is performed.

Experimental comparison with FOUND-IT. We evaluate both methods on JITOMA-Bench. For Tier 3, we report FOUND-IT's Augmented results using our LLM-based task parser to control for differences in task interpretation.

TierMethodmIoUmR@1mR@3TPF (s/frame)
1FOUND-IT17.038.042.80.10
JITOMA19.546.959.50.25
2FOUND-IT25.655.959.90.08
JITOMA24.359.466.90.27
3FOUND-IT (Aug.)11.621.026.70.16
JITOMA18.741.050.30.27

JITOMA achieves higher mR@1 and mR@3 across all three tiers. In Tier 3, it improves mIoU, mR@1, and mR@3 over FOUND-IT (Augmented) by 7.2, 20.0, and 23.7 percentage points, respectively. These gains come with higher processing time due to explicit caption generation and LLM reranking, reflecting JITOMA's emphasis on semantic richness rather than minimizing computation alone.

Clarification. Its contribution is instead a task-conditioned mechanism for allocating observation, memory, and semantic computation across different stages of online scene understanding. We will make this distinction more explicit in Sec. 2 and Sec. 4, particularly in relation to Clio and FOUND-IT.

References

[1] Maggio, D., Chang, Y., Hughes, N., Trang, M., Griffith, D., Dougherty, C., ... & Carlone, L. (2024). Clio: Real-time task-driven open-set 3d scene graphs. IEEE Robotics and Automation Letters, 9(10), 8921-8928.

[2] Maggio, D., Gorlo, N., Hauser, K., & Carlone, L. (2026). FOUND-IT: Foundation-model-first Task-driven 3D Scene Graphs with Granularity on Demand. arXiv preprint arXiv:2605.25371.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W2: Limited downstream validation. The experiments mainly evaluate object grounding and computational efficiency rather than completed long-horizon robotic tasks. Consequently, the evidence does not fully establish whether the proposed framework improves navigation, manipulation, or overall task execution beyond more efficient scene graph processing.

Question: Can the authors provide evidence that the reduced active semantic state improves downstream robotic task performance, rather than primarily reducing semantic processing cost?

A: We thank the reviewer for raising this important concern. We agree that our current evaluation does not directly measure end-to-end robotic execution success. However, 3D scene graph-based robotic applications can generally be viewed as involving two stages: task-relevant subgraph retrieval and downstream reasoning over the retrieved representation. For example, SG-Nav [1] uses online 3D scene graphs to prompt LLM-based reasoning for zero-shot object navigation. Our experiments provide evidence for both stages.

Subgraph retrieval. This is the primary focus of JITOMA-Bench, which evaluates foundational grounding, long-horizon task switching, and complex-intent grounding. Our results demonstrate that JITOMA retrieves task-relevant objects with strong accuracy while reducing active semantic state and processing cost.

Downstream planning. Our perceptual-saturation study (Sec. 3 and Appendix A.2) further evaluates whether the retrieved scene representation supports downstream reasoning. We provide the same LLM planner with either an AOT full graph or a task-relevant JIT subgraph, keeping the task instruction and prompt unchanged. As illustrated in Fig. 1, the full graph containing 18 entities (including 15 irrelevant objects) causes the planner to omit a required containment step, whereas the JIT subgraph containing only three relevant entities enables a complete plan.

We further extend this study to three LLMs under three planning-step settings. The results consistently show reduced planning coverage when irrelevant objects are introduced:

ModelStepsAll-target coverage (0 → 15)Δ (pp)Latent-target coverage (0 → 15)Δ (pp)
Qwen-27BUnrestricted88.97 → 72.56−16.4183.10 → 57.39−25.72
3–493.13 → 83.33−9.8187.27 → 73.67−13.59
695.83 → 80.20−15.6392.36 → 66.35−26.01
Qwen-35B-A3BUnrestricted91.59 → 81.10−10.4985.65 → 71.30−14.35
3–490.82 → 86.81−4.0185.19 → 76.62−8.56
695.14 → 89.93−5.2190.97 → 82.09−8.88
GPT-5.1Unrestricted95.22 → 87.22−8.0092.82 → 80.77−12.06
3–494.60 → 85.64−8.9693.29 → 75.99−17.30
695.22 → 93.08−2.1392.82 → 87.71−5.11

All nine model–step combinations exhibit lower coverage with 15 irrelevant objects, with latent-target coverage consistently experiencing larger degradation. This demonstrates that irrelevant scene semantics can impair downstream planning completeness, particularly the recovery of implicit objects required for task execution.

Taken together, these evaluations cover two complementary stages: JITOMA-Bench assesses the accuracy and efficiency of task-relevant subgraph retrieval, while the planning study examines how the retrieved representations support downstream reasoning. The results suggest that more focused scene representations help LLM planners recover task-required objects and generate more complete plans, beyond merely reducing semantic processing costs. We acknowledge that planning completeness does not directly establish physical execution success; closed-loop navigation and manipulation evaluation remains an important direction for future work.

Reference:

[1] Yin, H., Xu, X., Wu, Z., Zhou, J., & Lu, J. (2024). SG-Nav: Online 3D Scene Graph Prompting for LLM-Based Zero-Shot Object Navigation. Advances in Neural Information Processing Systems, 37, 5285–5307.

Reviewer_jrCeOpenReview ↗
Rating
3
Confidence
3(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JITOMA, a just-in-time 3D scene-graph framework that combines task-guided perception with on-demand semantic-memory activation to reduce redundant computation. The authors introduce JITOMA-Bench and evaluate the framework on single-task grounding, long-horizon task switching, and complex-intent grounding. The experimental results report promising results in grounding accuracy and computational efficiency.

Critical strengths
  1. The paper identifies a meaningful problem in long-horizon robotic perception, and its overall motivation and technical narrative are clear and well organized.
  2. The framework and benchmark represent substantial experimental work. The reported grounding and efficiency results provide encouraging evidence of the approach’s potential.
Critical weaknesses
  1. The scene-graph formulation is unclear. Sec.4 emphasizes ephemeral hypotheses, dormant anchors, CLIP features, and on-demand descriptions, but does not explain how inter-object relational edges are constructed, maintained, or used. It is therefore unclear whether JITOMA builds a 3D scene graph or primarily maintains task-conditioned object memories.
  2. The emprical evidence has limitations in supporting the main claims. The perceptual-saturation analysis uses Qwen3.5-9B and a planner constrained to three or four steps, so the findings are somehow limited. In addition, Tier 2 uses ground-truth target visibility to schedule task activation. Because JITOMA uses task-dependent heatmaps to select observations, this setup may favor the method and limit how broadly the conclusions can be applied.
Questions for authors
  1. Regarding the perceptual saturation and model generality, the Qwen3.5-9B planning examples and controlled experiments suggest that irrelevant global semantics may bias planning, but they do not establish global graph construction as the cause. Could the authors test whether this effect also occurs with open- and closed-source LLMs of different sizes?
  2. Could the authors add a concise summary of the main contributions at the end of the Introduction to help readers identify the paper’s focus?
  3. For scene-graph definition in Sec. 4, does JITOMA construct a 3D scene graph or a task-conditioned collection of object memories? In particular, how are relationships between objects established, updated, and used for downstream reasoning?
  4. Tasks are activated at the target’s first visible frame and evaluated ten frames after its last visible frame. Because JITOMA uses task-dependent heatmaps to select observations, this timing may favor the method. Could the authors provide an ablation in which task arrival times are independent of ground-truth target visibility, including queries made before or after the target is observed?
  5. The shared planner prompt restricts outputs to three or four steps. Could this constraint force the planner to omit required actions and partly explain the observed perceptual-saturation effect?
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/4)

W1: The scene-graph formulation is unclear. Sec. 4 emphasizes ephemeral hypotheses, dormant anchors, CLIP features, and on-demand descriptions, but does not explain how inter-object relational edges are constructed, maintained, or used. It is therefore unclear whether JITOMA builds a 3D scene graph or primarily maintains task-conditioned object memories.

Question: For scene-graph definition in Sec. 4, does JITOMA construct a 3D scene graph or a task-conditioned collection of object memories? In particular, how are relationships between objects established, updated, and used for downstream reasoning?

A: We thank the reviewer for pointing out this ambiguity. We agree that JITOMA primarily maintains task-conditioned object memories, rather than explicitly constructing and maintaining inter-object relational edges. We apologize that the original manuscript did not make this distinction sufficiently clear.

Object-centric representation. JITOMA maintains object-level representations through ephemeral hypotheses and dormant anchors (Eq. 1). During task activation, relevant anchors are retrieved and semantically instantiated based on the task's primary and latent concepts. Thus, the activated representation is a collection of task-relevant object nodes, without explicitly modeling spatial or semantic relations between different objects. This is an intentional design choice: our focus is on identifying which objects are relevant to a task and when their semantic representations should be instantiated. In our target setting, task relevance does not necessarily imply meaningful pairwise spatial relations. For example, "clean the cup" may require grounding a cup, a sponge, and a faucet, whose relevance arises from the task's functional requirements. This object-centric, task-driven design is related in spirit to Clio [1], while JITOMA specifically focuses on task-conditioned observation admission and on-demand semantic activation.

Optional functional refinement. JITOMA nevertheless supports finer-grained functional structures when required by downstream tasks. As described in Sec. 4.4, an optional refinement uses a task-conditioned VLM to predict an interaction-relevant part, which is then localized through segmentation and 3D back-projection. The resulting part node is attached only to its corresponding object anchor. As described in Sec. 4.5, this part-level information can subsequently be distilled into the anchor's auxiliary attributes for reuse. The distinction is summarized below:

RepresentationRelationship modeledPurpose
Core object memoryNo explicit inter-object edgesObject grounding and semantic activation
Optional functional refinementPart-to-object attachmentFine-grained interaction localization
General inter-object relationsNot explicitly modeledBeyond the current scope

Therefore, this optional refinement should not be interpreted as constructing functional edges between different object-level nodes.

We will clarify the representation scope in Sec. 4.1 and the optional refinement in Secs. 4.4–4.5. General inter-object relational modeling and its downstream evaluation remain interesting directions for future work.

Reference: [1] Maggio, D., Chang, Y., Hughes, N., Trang, M., Griffith, D., Dougherty, C., ... & Carlone, L. (2024). Clio: Real-time task-driven open-set 3d scene graphs. IEEE Robotics and Automation Letters, 9(10), 8921-8928.

AuthorsOpenReview ↗

Rebuttal by Authors (2/4)

W2: The empirical evidence has limitations in supporting the main claims. The perceptual-saturation analysis uses Qwen3.5-9B and a planner constrained to three or four steps, so the findings are somewhat limited.

Question 1: Regarding perceptual saturation and model generality, could the authors test whether this effect also occurs with open- and closed-source LLMs of different sizes?

Question 2: The shared planner prompt restricts outputs to three or four steps. Could this constraint force the planner to omit required actions and partly explain the observed perceptual-saturation effect?

A: We thank the reviewer for highlighting these limitations. We agree that evaluating only Qwen3.5-9B under a fixed 3–4-step planning constraint limits the generality of the original observations. To address these concerns, we extended the controlled distractor study to three additional LLMs, covering open- and closed-source models, and examined the effect of different planning-step constraints.

Experimental setup. Following the original protocol in Appendix A.2.2, we evaluate all 25 Tier 3 tasks across three scenes using the same distractor sampling procedure and coverage metrics. For each task, the required objects and their attributes remain unchanged, while the number of irrelevant objects increases from 0 to 15 (k=0 vs. k=15). We evaluate Qwen-27B, Qwen-35B-A3B, and GPT-5.1 under three planning settings: unrestricted steps, strictly 3–4 steps (original), and exactly 6 steps. Other prompt instructions are kept consistent.

Results. The table reports all-target and latent-target coverage (%) under k=0 and k=15, with absolute changes in percentage points (pp).

ModelStepsAll-target coverage (0 → 15)Δ (pp)Latent-target coverage (0 → 15)Δ (pp)
Qwen-27BUnrestricted88.97 → 72.56−16.4183.10 → 57.39−25.72
Qwen-27B3–493.13 → 83.33−9.8187.27 → 73.67−13.59
Qwen-27B695.83 → 80.20−15.6392.36 → 66.35−26.01
Qwen-35B-A3BUnrestricted91.59 → 81.10−10.4985.65 → 71.30−14.35
Qwen-35B-A3B3–490.82 → 86.81−4.0185.19 → 76.62−8.56
Qwen-35B-A3B695.14 → 89.93−5.2190.97 → 82.09−8.88
GPT-5.1Unrestricted95.22 → 87.22−8.0092.82 → 80.77−12.06
GPT-5.13–494.60 → 85.64−8.9693.29 → 75.99−17.30
GPT-5.1695.22 → 93.08−2.1392.82 → 87.71−5.11

Effect of planning-step constraints. The results show that planning length does affect coverage, but no consistent trend emerges across models and step settings. Unrestricted planning does not necessarily improve completeness, as models may generate overly concise plans that omit required actions. Similarly, increasing the required length from 3–4 to 6 steps improves robustness for some models but not others. Thus, simply relaxing or increasing the step budget does not consistently resolve target omissions.

Cross-model consistency. More importantly, all nine model–step combinations exhibit lower coverage at k=15 than at k=0. Even without step restrictions, latent-target coverage drops by 25.72, 14.35, and 12.06 pp for Qwen-27B, Qwen-35B-A3B, and GPT-5.1, respectively. Across all settings, latent targets also experience larger coverage reductions than the overall target set. These observations suggest that irrelevant semantic information consistently interferes with the recovery of task-required objects, particularly implicit dependencies, although the magnitude varies across models and planning constraints.

AuthorsOpenReview ↗

Rebuttal by Authors (3/4)

W3: In addition, Tier 2 uses ground-truth target visibility to schedule task activation. Because JITOMA uses task-dependent heatmaps to select observations, this setup may favor the method and limit how broadly the conclusions can be applied.

Question: Tasks are activated at the target's first visible frame and evaluated ten frames after its last visible frame. Because JITOMA uses task-dependent heatmaps to select observations, this timing may favor the method. Could the authors provide an ablation in which task arrival times are independent of ground-truth target visibility, including queries made before or after the target is observed?

A: We thank the reviewer for identifying this potential bias. To address this concern, we introduce a delayed task-arrival ablation, preserving the original object-task ordering while delaying all task activation and query events until after their corresponding targets have left the field of view.

Revised protocol. The temporal difference between the two settings is illustrated below:

ProtocolTimeline
Original Tier 2Activate task → Target visible → Query
Delayed Tier 2Target visible → Target leaves view → Activate task → Query

Thus, unlike the original setting, the system must retrieve previously observed targets from memory without having received their corresponding task during observation.

Why can JITOMA retain previously irrelevant objects? JITOMA employs two complementary observation-admission policies (Sec. 4.2 and Appendix A.3.2):

  • Task-free: Confidently tracked objects that have not previously contributed observations to memory are admitted, while existing objects are updated only when observations provide sufficient visual improvement.
  • Task-active: JITOMA admits the top 50% of observations ranked by task-conditioned heatmaps. This is a budgeted selection strategy, not an exclusive filter on task-relevant objects; previously unseen, non-target objects can still be admitted.

Admitted observations accumulate as ephemeral hypotheses and are promoted to persistent dormant anchors. These anchors survive task transitions without requiring immediate semantic expansion, allowing future tasks to retrieve historical objects through their stored visual features and accumulated task-relevance evidence.

Results. We compare the original and delayed Tier 2 protocols across all three scenes:

SceneProtocolmIoUmR@1mR@3
CubicleOriginal22.466.774.1
Delayed21.963.074.1
OfficeOriginal29.164.373.8
Delayed29.061.973.8
ApartmentOriginal21.447.252.8
Delayed19.644.452.8
AverageOriginal24.359.466.9
Delayed23.556.466.9

Despite delayed task arrival, average mIoU decreases by only 0.8 pp and mR@1 by 3.0 pp, while mR@3 remains unchanged across all three scenes. This demonstrates that JITOMA can preserve and recover previously observed objects even when their corresponding tasks arrive only after they have left view.

Comments on the paper

1 comment

Dear Reviewer 7UwZ,

Thank you for your follow-up. We noticed that your comment was posted under our response to Reviewer DjSs. To avoid any misunderstanding, could you please double-check the discussion thread and clarify which specific response from us your assessment refers to? We would also appreciate confirmation that our full responses to your concerns have been reviewed and taken into account.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.