- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
JITOMA builds task-conditioned 3D scene graphs by selectively storing observations and delaying caption generation until an object is needed. It maintains dormant anchors with visual and geometric information, retrieves candidates for explicit and implicit task targets, and captions and reranks those candidates. The authors also introduce JITOMA-Bench, based on three Clio scenes, to evaluate single-object grounding, task switching, and multi-object instructions. The experiments report improved grounding accuracy and lower average processing time compared with ConceptGraphs, ReasoningGraph, and Clio.
Critical strengths
-
Deferring expensive semantic processing until it is needed is a useful direction for online scene graphs. The ablations in Table 2 separate the effects of task-guided observation admission and on-demand captioning. The budget-matched control is helpful: it shows that selecting observations based on task relevance works better than simply admitting fewer observations.
-
The paper provides evidence for its motivation rather than relying only on illustrative examples. In the controlled distractor study, task-relevant content is held fixed while irrelevant objects are added. Latent-target coverage drops from 68.10% to 56.30%. This supports the argument that additional scene information can hurt the planner’s ability to identify objects implicitly required by a task.
Critical weaknesses
-
The task-switching protocol does not test whether the system can recover information needed for an unexpected future task. Each Tier 2 task starts when its target first becomes visible and is evaluated ten frames after the target was last visible. The system therefore knows which object matters while observing it. A harder and more relevant case is a request that arrives after the object has left view, when that object was irrelevant to the previous task. Without this evaluation, it is unclear whether task-guided filtering sacrifices future-task recall. The experiments are also limited to grounding on three recorded scenes, so the claims about long-horizon robotic execution go beyond what is demonstrated.
-
The efficiency evaluation leaves out important costs of on-demand processing. Average time per frame does not show how long the robot waits when a new task triggers captioning and reranking. Similarly, a small active graph does not imply a small total memory footprint: dormant anchors retain visual information and, after activation, captions and other attributes. The paper acknowledges this distinction, but does not report actual memory usage or explain what happens when the 512-node capacity is reached.
Questions for authors
-
How does performance change when a task arrives after its target has left view and the target was irrelevant to the preceding task? A comparison with Clio and the budget-matched admission control would help establish whether the method preserves information for future requests.
-
What is the latency from receiving a new task to returning a grounded subgraph, including parsing, captioning, and reranking?
-
Could you reconcile the default Tier 2 results across Tables 1, 2, and 5? Averaging Table 1 across scenes gives approximately 24.3 IoU and 59.4 mR@1. Table 2 reports 24.3 and 57.5, while Table 5 reports 23.95 and 56.83. Were these obtained using different settings or runs?
-
The paper describes FOUND-IT as retaining visual memory and resolving representations on demand. What benefit does JITOMA provide over that approach? A direct comparison, or a controlled comparison using shared components, would help clarify the contribution.
7 comments
Rebuttal by Authors (2/6)
W2: The efficiency evaluation leaves out important costs of on-demand processing. Average time per frame does not show how long the robot waits when a new task triggers captioning and reranking.
Question: What is the latency from receiving a new task to returning a grounded subgraph, including parsing, captioning, and reranking?
A: We thank the reviewer for pointing out that per-frame processing time does not fully characterize the latency of on-demand semantic processing. Importantly, JITOMA executes these operations asynchronously rather than as a sequential pipeline: backend captioning overlaps with continuous frontend observation.
Latency breakdown. We report the average processing time of each stage:
| Stage | Average latency | Execution |
|---|---|---|
| Task parsing | 1.9 s | Synchronous upon task arrival |
| Batch captioning | 3.2 s | Asynchronous, overlapping with frontend observation |
| LLM reranking | 2.5 s | Synchronous upon query arrival |
Consequently, these durations should not be added directly to estimate query latency. The 3.2 s captioning process runs in the background while the frontend continues processing observations. Once a grounding query arrives, the main additional waiting time is the 2.5 s reranking stage. Task parsing incurs a separate 1.9 s overhead upon task arrival.
Semantic richness and resource allocation. Unlike embedding-based approaches such as Clio [1], which primarily rely on visual-language features for semantic matching and clustering, JITOMA additionally provides explicit textual object descriptions and LLM-based semantic reranking. Generating these richer semantic representations inevitably introduces additional computation. However, JITOMA allocates captioning resources selectively: only shortlisted task-relevant candidates undergo expensive caption generation, rather than captioning all observed objects. Combined with asynchronous execution, this design keeps the additional semantic processing manageable without placing the entire captioning cost on the query-time critical path.
We will clarify this execution pipeline and include the stage-wise latency breakdown in the revised manuscript.
Reference:
[1] Maggio, D., Chang, Y., Hughes, N., Trang, M., Griffith, D., Dougherty, C., ... & Carlone, L. (2024). Clio: Real-time task-driven open-set 3d scene graphs. IEEE Robotics and Automation Letters, 9(10), 8921-8928.
Rebuttal by Authors (3/6)
W3: Similarly, a small active graph does not imply a small total memory footprint: dormant anchors retain visual information and, after activation, captions and other attributes. The paper acknowledges this distinction, but does not report actual memory usage or explain what happens when the 512-node capacity is reached.
A: We thank the reviewer for pointing out this distinction. We would like to clarify that the active-node count reported in our efficiency evaluation is intended to measure the semantic processing workload, rather than total memory consumption. JITOMA retains dormant anchors, including their visual representations, in memory, while expensive semantic attributes such as captions are generated on demand upon activation. Therefore, our efficiency claim primarily concerns reducing the number of nodes requiring semantic processing, rather than reducing the total memory footprint. We will clarify this distinction in the revised manuscript.
Global memory capacity. JITOMA uses a fixed global capacity of 512 nodes, including both ephemeral hypotheses and persistent dormant anchors. When the capacity is reached, ephemeral hypotheses are first removed in temporal order to reclaim space before evicting persistent anchors. If further eviction is necessary, a sliding-window policy removes the oldest dormant anchors, following the memory management strategy commonly adopted in prior 3D scene graph systems. More sophisticated long-term memory forgetting policies are beyond the scope of this work.
Rebuttal by Authors (1/6)
A: We thank the reviewer for raising this important concern. We agree that the original Tier 2 protocol does not fully evaluate whether task-irrelevant observations are preserved for unexpected future requests. To address this, we introduce a delayed task-arrival evaluation and compare JITOMA against Clio and our budget-matched admission control (D†).
Revised evaluation protocol. We preserve the original object-task ordering but delay all task activation and query events, such that each target has already left the field of view when its corresponding task is activated.
Thus, the target is observed before its corresponding task arrives and is unrelated to the preceding task. JITOMA must retrieve it from previously accumulated memory without having received its task during observation.
Results. We evaluate Clio, D†, and JITOMA under the delayed protocol across all three scenes. We also include JITOMA's original Tier 2 results for reference.
Delayed-arrival robustness. The delayed protocol has little impact on Clio: its average mIoU, mR@1, and mR@3 change from 17.5/43.3/49.8 in the original Tier 2 to 17.3/45.6/50.2. JITOMA experiences a modest reduction from 24.3/59.4/66.9 to 23.5/56.4/66.9, with mR@3 remaining unchanged across all three scenes. This suggests that although delayed activation removes task-conditioned heatmap evidence collected during the target's visibility, JITOMA can still recover historical targets through its persistent visual memory and accumulated evidence.
Comparison with budget-matched admission control. More importantly, JITOMA consistently outperforms D† under the delayed protocol. Averaged across the three scenes, JITOMA improves mIoU, mR@1, and mR@3 by 10.6, 21.1, and 22.2 percentage points, respectively. Although D† matches JITOMA's per-frame observation-admission budget, it uses task-independent admission and does not implement JITOMA's two complementary writing policies: task-guided heatmap selection during active tasks and novelty-preserving conservative writing during task-free intervals. In particular, JITOMA explicitly admits previously unseen tracks while suppressing redundant updates to existing tracks when no task is active. Therefore, matching the admission budget alone is insufficient to preserve useful information for future tasks; how observations are selected and maintained also matters.