- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces R3D-Bench, with 3,033 quantitative spatial questions across 57 egocentric videos. Its 15 question types cover object comparisons, distances, and volumes. Inputs include RGB-D video and camera poses. It also proposes R3D, an agentic pipeline. It combines SAM3 segmentation, depth-based scene reconstruction, and SAM3D meshes. An LLM answers questions through eight spatial tools, without directly accessing the image inputs.
Critical strengths
- Meaningful benchmark for the community. Measuring distances and container capacities is a meaningful direction for wearable assistants.
- Strong reported improvements. R3D achieves an overall average of 73.5, compared with 61.9 for CuTR+Tools using the same LLM.
Critical weaknesses
- The bibliography contains multiple hallucinated or factually incorrect references, including incorrect author lists.
- The methodological novelty of R3D is limited. Using an LLM to call perception and geometry tools for spatial VQA is already a well-established paradigm, e.g., VADAR, SpaceTools, Think3D, and S-Agent. These methods already combine tools such as object detection, segmentation, depth estimation/reconstruction, 3D lifting, and metric computation. R3D largely follows the same paradigm, while further assuming metric depth and camera pose are directly provided by the sensor, making the problem even easier. The main additional component, SAM3D-based mesh completion for volume estimation, also requires stronger justification, since SAM3D is a generative model, rather than geometry-based reconstruction. Thus it only produces a plausible mesh rather than evidence-faithful geometry.
- The benchmark and method are strongly entangled. Objects are filtered based on SAM3 tracking quality, while R3D itself relies on SAM3. Moreover, the paper does not explain in R3D pipeline how the candidate object list is obtained when constructing the scene database before receiving a question. This raises concerns about selection bias.
- The role of the LLM is poorly motivated. In the reported setting, the LLM does not observe the video; it only queries a precomputed database for object positions, dimensions, distances, and volumes. Most benchmark questions could be answered by deterministic programs over the same database. For example, a distance question can directly query the two relevant objects and compute their distance, without any LLM reasoning. This makes the necessity of the agentic LLM unclear.
- The baseline coverage is incomplete. The paper adapts the image-based SpatialRGPT to video by simple frame-wise aggregation, while omitting stronger video/multi-view successors such as SR-3D (Cheng et al.).
- Some evaluation choices weaken the comparisons. Parse failures are excluded from accuracy, and R3D and CuTR+Tools use different procedures for volume estimation (mesh-based cavity estimation vs. bounding-box volume), making the reported gap difficult to attribute solely to the proposed perception pipeline.
Questions for authors
- Could the authors report a non-LLM baseline that directly answers the questions using deterministic programs over the constructed scene database? This would help clarify how much of the performance gain actually comes from LLM reasoning.
- The benchmark provides a “pointing reference”, i.e., a ground-truth mask for each referred object at its first appearance. However, in the reported R3D setting, the LLM receives no visual inputs, so this reference is not directly used by the LLM. Could the authors clarify whether the pointing reference is used elsewhere in R3D, e.g., to initialize or guide SAM3 tracking? If so, does this introduce privileged ground-truth guidance into the perception pipeline?
- Could the authors clarify the methodological novelty of R3D compared with prior agentic pipelines for spatial reasoning?
- Appendix E reports per-query runtime, but the intended application is an always-on wearable assistant. How does R3D scale in a continuous streaming setting? In particular, how is the object database updated over time, and do tracking or multi-view reconstruction errors accumulate as the sequence becomes longer?
- How is the set of objects in the precomputed scene database determined before a question arrives? Where do the object inventory and semantic names come from?
- Could the authors directly evaluate the accuracy of the constructed object database against the GT object metadata, including object inventory/category names, semantic identities, positions, dimensions, distances, and volumes? This would help disentangle errors from object discovery/grounding and metric estimation, and better assess the coupling between the benchmark and the proposed pipeline.
- SAM3D is a generative shape-completion model rather than a deterministic reconstruction method. Figure 5 also shows that the completed mesh is an imagined typical rubbish bin and can differ substantially from the GT geometry. Could the authors further discuss how hallucinated or prior-driven geometry affects quantitative volume estimation and discuss this choice of SAM3D?
4 comments
Response to YYAS, Part 2 of 2
Q1: We believe the reviewer is asking for an evaluation using the ground-truth answer formula. This evaluation is an unrealistic upper bound on performance when the scene graph contains no errors. The LLM's role is to interpret the question, interpret the scene data (through tools), and give an answer.
With this in mind, we add an experiment in Appendix J (Table 15) to show this upper bound. The short answer is: using GT scene data and GT question-answering programs, we achieve 99.6% accuracy (tools round results to 1cm, hence imperfect results). Using GT scene data with R3D tools, we achieve 80.0%-94.8% accuracy depending on the Qwen variant. We also show results for GT tools (a fixed formula per question type, given the question's type and objects directly, i.e. oracle question parsing) with the R3D scene; this is the non-LLM baseline over the constructed scene that you asked for. In this case, the LLMs outperform the tools overall (57.0 for R3D with Qwen3-VL 235B-A22B Thinking vs. 37.4), because the fixed formula abstains when an object is missing from the scene. On the questions whose objects are all in the scene, the fixed formula scores 79.0 vs. 73.9 for the best LLM. See Appendix J for details.
You might also be interested in the ablations on scene construction in Appendix H.
Q2: The pointing reference is used as a location hint to SAM3, which may or may not successfully recognize the object. It is used nowhere else. Note that other baselines rely on SAM3 tracks as well (see Section 5.1 and Appendix E). For example, CuTR only detects candidate objects and needs SAM3 to select the right candidate boxes, and RGB models use SAM3 masks as a reference overlay for disambiguation. This is privileged information in the sense of a single ground-truth mask on one frame, standing in for the user's pointing gesture; every SAM3-based method receives the same.
Q3: See CW2.
Q4: Our videos are 90-100 s long (Section 3.1), over which object representations are stable. There are many design choices for scaling to hours or days or weeks of context, such as object retention, whether to continue updating the representation of static objects, when to refresh the database, etc. We consider this out of scope, as our focus is to establish a natural egocentric metric Q&A benchmark and establish strong baselines.
Q5: We consider the objects of interest to be given as part of the question; their natural names are mapped from ADT object names (Section 3.2). This is in line with our prompting method for all baselines (giving SAM3 object tracks). There are many options for how a real-world system would work: (a) requiring the user to specify, (b) different modes for different applications, e.g. tracking cooking-related objects while the user is cooking, tracking tools when the user is repairing something, etc. Our focus is instead on the metric-space Q&A when tracking particular object classes.
Q6: We have added this analysis to Appendix G (Table 11): 3D IoU, center, yaw, height and longest-dimension errors, box-volume and functional-volume errors, and object-to-object and distance-from-camera errors, for R3D and CuTR against the ADT ground truth. Regarding the object inventory and semantic identities: each SAM3 track is started from the question's object name and pointing reference, so scene objects carry the names used in the questions, and the inventory question reduces to coverage. 1,447 of the 2,092 object-sequence pairs referenced by the questions (69.2%) are reconstructed (Section 3.3, Table 4); the rest are objects SAM3 never tracked. Table 11 covers the reconstructed pairs. Figure 3a shows the SAM3 tracking IoU of every referenced object, and Appendix F splits the questions by tracking quality.
Q7: See CW2. In addition: SAM3D only shapes the cavity, while the metric scale comes from R3D's depth-lifted box (Equation 1). The per-object functional-volume error has a median of 23.2% but a 90th percentile of 65.9% (Table 11). Using ground-truth masks as SAM3D input gives similar reconstructions (Table 16), so the masks are not the main source of error, and replacing SAM3D volumes with box volumes lowers the volume average from 30.1 to 8.8 (Table 17). Figure 5 shows examples.
Thanks for your very detailed rebuttal. However, my main concern is still not addressed.
- Contribution. The idea of a metric spatial benchmark is not novel at all, having been explored by a bunch of works (VSI, OSI, ...), just converting to a "wearable" setting does not add to that. Importantly, considering this work has no data contribution, but just extracting from existing off-the-shelf dataset metadata or making use of some specialist model annotations (this pipeline, is also not novel and used by many benchmark works) to generate QAs, the contribution of this work seems not established to me.
- The role of the LLM remains unclear. The new experiments show that deterministic programs outperform the LLM when the required objects are available (79.0 vs. 73.9). So that the role of LLM is not reasoning, but just handling object names. I am not convinced that using a heavy LLM is justified for relatively simple questions that can be solved more reliably by deterministic tools.
- The methodological contribution remains limited.
Overall, I appreciate the additional experiments and revisions, but I remain unconvinced of the paper's core contributions.
Response to Comment by YYAS
We thank you for your reply. We provide additional clarification here.
Regarding 1: We discuss the drawbacks of existing metric 3D datasets in Table 1, Page 2, and Page 3. We feel that existing benchmarks do not reflect the real-world challenges of the egocentric case (Fig 3), as in our setting, scanning motions are not used (cannot perform a full reconstruction). We view this as an important contribution for real-world evaluation.
Regarding 2: We clarify the role of the LLM:
I am not convinced that using a heavy LLM is justified for relatively simple questions that can be solved more reliably by deterministic tools.
The role of the LLM is to decide how to answer questions (determining what the user is asking, and deciding how to compose tools to answer them). Using the ground-truth formula is not an option for a real deployment. We provided the ablation to help you understand the ceiling of performance, but such a system could not be reliably deployed, since we don't know the ground-truth formula for what the user will ask.
As an example, imagine I write a program to solve for the hypotenuse of a right triangle using the Pythagorean theorem. Will an LLM capable of mathematical reasoning ever outperform my program? No - there's one ground truth answer to this hypotenuse question, and the ground-truth formula will always achieve it. However, an LLM capable of mathematical reasoning can answer any question, not just the Pythagorean theorem question.
Similarly, our LLM's job is to answer any question, since there's no way of knowing what ground-truth mixture of tools to apply a priori. Our dataset includes 15 question types, but there are endless possibilities of what a user could ask. The LLM will interpret the question, understand what the user is asking, and provide the answer. We included the ground-truth formula ablation to help contextualize our results, but it is not a realistic scenario.
Regarding 3: We emphasize the novelty of our volumetric estimation method, whose efficacy is shown in main evaluations (Table 2) and ablations (Appendices H, K) in scenarios where full reconstruction is not possible due to only viewing objects from a few angles (Fig 5, Fig 3c). Additionally, we believe the empirical evaluation showing that tool calling performs strongly compared to fully latent methods, provides value. While we recognize that other recent works have used similar tool calling setups for non-volumetric questions, we view the insights given by these strong results as critical to building functional metric Q&A systems, and important for establishing baseline results on the dataset.
We thank you again for your reply and the opportunity to discuss.
Response to YYAS, Part 1 of 2
We thank the reviewer for the detailed review and the noted strengths. We address Critical Weaknesses (CW) and Questions (Q) below. We have updated our paper draft. The most significant change is, SAM3 is no longer used to filter the dataset. This has made the dataset significantly more challenging and rebalanced results. Additionally, we found an issue in which some images were dropped in RGB-only models - we have corrected this error. Please see the updated Table 2 (main results) and updated abstract. Our key result is now, R3D achieves the best results (<100B), and R3D achieves the second-best on (>100B), second to closed-source GPT-5.5 assisted by the SAM3 tracker.
We have also added six new sections to the Appendix (Appendices A, F, G, H, I and J) and expanded Appendices B, E and K. We highlight the sections relevant to your questions, but we encourage you to look through all of them to see the new experiments, analyses, and ablations.
CW1: We deeply apologize for these errors. Upon reviewing every entry, we found 13 entries with incorrect metadata, most of which involve incorrect author names. In addition, four citations pointed to the wrong work (e.g. Qwen3 / Qwen3-VL / Qwen3.6 models were cited via the Qwen2.5-VL report). All of these are corrected.
CW2: Our primary contributions are (a) a benchmark for metric Q&A in the egocentric setting, and (b) the observation that reasoning over a scene performs strongly here. The algorithmic novelty compared to recent agentic works is not the highlight, though our volumetric Q&A is novel.
Regarding SAM3D: we have no choice but to use a prior-based method, as we cannot perform full reconstructions in this scenario. Almost none of our objects are viewed from below (Figure 3c), and the viewpoints leave massive gaps that must be filled by priors (Figure 5). That means we can't reconstruct full meshes. The observation that SAM3D can be used to improve volumetric estimation in the face of uncertainty is an interesting contribution from our work. We have a large amount of empirical justification of the efficacy (Table 2; Appendices E, G and K: Tables 7, 11, 16 and 17). We emphasize that other datasets contain unrealistic camera trajectories (scanning motions instead of natural egocentric motion, Figure 2a) and thus can support full 3D reconstruction. By contrast, our goal is to evaluate real-world performance with natural egocentric motion.
CW3: We no longer use SAM3 to filter the dataset. We also emphasize that other baselines rely on SAM3 tracks, see Section 5.1 and Appendices D and E. Regarding scene database construction, we assume the candidate objects are known. This is in line with how we initialize SAM3 tracks, as these serve as a cue that the user is interested in an object. We do this so that our evaluation focuses on metric-space Q&A rather than object disambiguation, for which there are already many excellent datasets. See also Q5 below.
CW4: The purpose of the LLM is to interpret the user's question and decide what to do; whether to answer directly, use tools to help, etc. In real life a deployed system might be asked general-knowledge questions in addition to 3D Q&A. A rules-based system could not be deployed; it would have no way of knowing which functions to call, and how to compose the answers (e.g. querying for distances and summing them to compute path length).
CW5: We have added SR-3D to Table 2. Note that it does not outperform Video-3D LLM in our evaluations (14.0 vs. 16.3), and it also suffers from many parse failures (32.5% of answers; 55.9% for Video-3D LLM). It also scores below SpatialRGPT (20.0). Note that SR-3D normalizes its point maps to [-1, 1], so its inputs carry no metric scale (Appendix E). For these reasons, we believe using an explicit scene called through tools to be a better approach for quantitative questions.
CW6: Our main results now include parse failures, see Table 2. Results excluding parse failures are now in Appendix F (Table 8).
Ablations showing CuTR with SAM3D appear in Appendix E (Table 7): CuTR scores 45.2 with SAM3D volumes vs. 47.7 with its box volumes. With matched volume procedures R3D still leads: 51.3 vs. 47.7 with box volumes, and 55.6 vs. 45.2 with SAM3D volumes (per-object volume error 23.2% vs. 58.1%). We note that CuTR performs worse with SAM3D, because CuTR tends to predict boxes that are too small. This error balances out the overestimation caused by the bounding box including air outside the object, but causes a problem when this overestimation factor is removed. We leave CuTR+boxes in the main table since it performs best, and leave CuTR+SAM3D as an ablation in the appendix.