- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
AgentSTAR presents an agentic analysis-by-synthesis framework for jointly recovering object geometry, kinematic structure, and motion from video. A VLM agent constructs and updates a code-based object model through visual feedback, numerical pose optimization, and temporal diagnostics. The method is evaluated on ARCTIC, HOT3D, and the RGB-D iTACO benchmark. I view its central contribution as an integrated framework connecting agentic 3D reconstruction with kinematic modeling and motion recovery.
Critical strengths
S1. An interesting and reasonably novel integration of agentic reconstruction, kinematic modeling, and motion estimation
The framework connects object geometry, articulated parts, joints, and their motion within a coherent optimization process. I find this integration interesting and reasonably novel.
The structured representation also has meaningful potential for robotics. For example, manipulating an articulated object requires identifying the movable component and understanding how it connects to the rest of the object. The recovered representation could support consistent part-level mask rendering and reasoning about opening or closing such objects. These applications are not directly demonstrated, but they provide a meaningful motivation for recovering geometry and kinematics together.
S2. Encouraging quantitative results across tracking and articulated reconstruction
The reported improvements are substantial under the stated protocols. ARCTIC EPE decreases from 7.65 cm for V-DPM to 5.59 cm, while HOT3D mean translation error decreases from 6.41 cm for SAM3D-Tracker to 3.04 cm. On iTACO, revolute-axis error decreases from 0.32 to 0.08 rad relative to iTACO.
The improvement from 11.26 cm for No Harness to 5.59 cm for the default full system also provides meaningful evidence that the specialized tools and orchestration are useful.
Critical weaknesses
W1. The contribution of the specialized harness and its computational tradeoff need clearer characterization
The No Harness comparison and common ten-hour cap are useful controls, and Section 4.1 suggests that the default GPT backbone is shared. However, Table 2 does not explicitly document each variant’s backbone, host agent, critic configuration, available tools, and actual resource consumption. Fable 5 has no corresponding No Harness result, and its host also changes to Claude Code.
The paper discloses approximately 18M tokens per ARCTIC sequence and an estimated GPT API cost of approximately $300 per sequence, but lacks per-variant measured runtime and token/cost statistics. Consequently, the incremental benefit of the specialized framework at comparable budgets, particularly with the strongest reported model, remains insufficiently characterized. This makes it difficult to distinguish the contributions of the framework, the underlying model, and additional inference-time computation.
W2. The comparison tables insufficiently establish the reconstruction advantage and do not adequately contextualize the offline pose results
Table 3 evaluates pose accuracy, not reconstruction quality. Its translation and rotation errors demonstrate improved pose estimation under the stated protocol, but do not show whether AgentSTAR reconstructs more accurate geometry than the SAM3D-based pipeline. Better pose accuracy alone cannot establish better reconstruction.
The reconstruction comparisons are limited. Table 1 reports Chamfer distance, but compares geometry only against V-DPM, a point-tracking method. This provides limited evidence relative to competitive reconstruction pipelines. Table 4 provides relevant articulated-reconstruction evidence and shows strong part-geometry and kinematic results. However, whole-object CD is 0.02 versus 0.01 m² for iTACO, so the results do not establish a consistent advantage across all aspects of reconstruction quality.
The pose comparison also needs to account for different computational settings. AgentSTAR repeatedly revisits the observed video within a ten-hour optimization budget, whereas FoundationPose is designed for online tracking. The original FoundationPose reports approximately 32 Hz on an RTX 3090; this is a reference speed, not an end-to-end measurement of the FoundationPose + VGGT-Ω pipeline evaluated here. The table should report the actual evaluated pipelines’ runtimes and temporal access alongside accuracy. See FoundationPose, Section 4.5.
The joint contribution therefore needs separate evidence for geometry and kinematic recovery against appropriate reconstruction baselines, and for pose accuracy together with computational cost and access to future frames. Substantial computation could be justified by substantially better structured reconstruction. However, the current pose comparison does not establish that tradeoff, and lower offline pose error alone does not demonstrate a preferable practical tracking system.
W3. Mask provenance and quality are insufficiently characterized
Section 3 permits masks from a video segmentation model or masks provided directly, but does not clearly specify their actual sources, quality, or manual correction for each benchmark.
Since silhouette IoU drives numerical optimization, missing parts, inaccurate boundaries, or incorrect hand masks could bias both reconstructed geometry and estimated poses. Agreement with the input mask does not establish that mask’s accuracy. No dedicated evaluation of mask quality or sensitivity to segmentation errors is reported.
The paper should clarify whether the reported gains hold with realistic predicted masks and what mask information is available to each baseline. This uncertainty affects both practical reliability and interpretation of the comparisons.
Questions for authors
Q1. Model and budget attribution — corresponding to W1
Please specify the backbone, host agent, critic configuration, available tools, and stopping criteria for No Harness and the full system. Can you report their actual runtime, token consumption, API cost, and numerical search effort?
Does the improvement persist with the same model under comparable computational budgets? A matched Fable 5 control on a representative subset, if feasible, would help interpret the strongest reported variant.
Q2. Reconstruction comparisons — corresponding to W2
Can the geometry produced by the SAM3D-based pipeline already evaluated in Table 3 be evaluated alongside AgentSTAR, or can another appropriate reconstruction baseline be included under clearly stated input conditions?
Please clarify whether the geometry evaluation separates shape error from pose error, and identify which reconstruction or kinematic advantages justify the additional optimization cost.
Q3. Offline versus online comparison — corresponding to W2
For Table 3, please specify each method’s temporal access, input assumptions, and measured end-to-end runtime, including the VGGT-Ω stage where applicable.
What quality–cost tradeoff follows from earlier stopping points in the existing optimization traces? How should the offline pose improvements be interpreted relative to online tracking and the intended applications?
Q4. Mask sources and robustness — corresponding to W3
For each benchmark, are object and hand masks ground-truth, automatically predicted, or manually corrected? Which segmentation model and prompting procedure are used, and what mask information is available to each baseline?
Can you quantify mask quality or provide a limited assessment of sensitivity to realistic segmentation errors, such as missing parts or inaccurate boundaries?
5 comments
Thank you for the thoughtful responses and additional experiments. The same-backbone comparison and new reconstruction results strengthen the paper and address important parts of my earlier concerns. I appreciate these improvements.
My main remaining concern is W3. The quantitative evaluations use GT object masks, whereas practical use would generally rely on predictions such as SAM3 masks. High whole-object mask IoU does not establish how segmentation errors affect reconstruction, kinematic estimation, and tracking. Small movable parts may be omitted, regions may be incorrectly split or merged, and coverage may vary across frames. Please also clarify whether any GT part masks or labels are provided as input.
Could you provide a controlled comparison on a representative subset, replacing GT object masks with uncorrected SAM3 predictions while keeping the other inputs, model, and optimization budget fixed? Please report changes in geometry, kinematic, and tracking errors, together with whole-object completeness and part/kinematic correctness, including missing or extra parts and joints. Please specify any part-specific prompting, manual selection, or correction required.
Given the substantial token consumption, monetary cost, and optimization time, it remains important to establish whether the benefits persist under realistic segmentation inputs and justify the additional expense. Please report actual resource consumption on this subset and, where feasible, compare the main baseline under the same predicted-mask conditions.
We thank the reviewer for the insightful experiment suggestion. No part labels were provided as input in all cases.
We repeat our experiment on the entire ARCTIC evaluation split, replacing GT object masks with predictions from the SAM3 video tracker, initialised with the first-frame GT mask to identify the object of interest. All subsequent masks were generated automatically, without part-specific prompting or manual correction, which reflects real-world scenarios. We kept our strongest variant (Fable 5 + our harness), all other inputs, harness and the optimisation budget unchanged.
| Method | Tracking EPE (cm) | Chamfer (cm) |
|---|---|---|
| Fable 5 + Ours (GT masks) | 4.95 | 2.83 |
| Fable 5 + Ours (SAM masks) | 5.47 | 3.58 |
| VDPM | 7.65 | 4.72 |
This remains substantially better than the strongest baseline, V-DPM. Importantly, the recovered kinematics do not miss any annotated degrees of freedom. Occasional additional joints correspond to valid but unannotated articulations (e.g mixer's whisk), and also occur with GT masks.
Experiment budget and different modelling options
We provide compute cost analysis in supplementary A1, with exact configurations available in our supplementary code. Figure 5 shows the compute-accuracy trade-off for GPT-Sol 5.6.
We additionally evaluated Fable 5 without our harness, obtaining 6.50cm tracking error versus 4.95cm with our tooling. Even with a stronger backbone, our specialised framework provides substantial improvement.
Effective use of inference-time compute is itself a key contribution of our harness. It enables continued iterative refinement, as shown in Figure 5. We therefore argue that a common maximum runtime budget is more appropriate than identical compute consumption.
Future VLMs may become proficient at this task through specialised post-training, without requiring a specialised harness. However, we view our central contribution as a general computational formulation for agentic optimisation, which iteratively couples VLM reasoning with traditional numerical optimisation. AgentSTAR showcases this paradigm for shape reconstruction and tracking, but we believe it applies to a broader class of challenging optimisation problems.
Computational trade-offs; offline versus online tracking
We compare methods with different computational budgets to contextualise the benefits of additional inference-time computation and the analysis-by-synthesis paradigm. For example, SAM3D-Tracker, the strongest baseline, takes approximately 40 minutes per sequence.
Regarding online/offline comparison: FoundationPose is an RGB-D tracker. To run on RGB videos, it requires depth estimation via VGGT-Omega, which processes the full sequence and makes our evaluated pipeline non-streaming.
Reconstruction comparison on HOT3D
Pose and reconstruction errors are coupled. To isolate geometry quality, we evaluate Chamfer distance both before and after ICP alignment to ground-truth geometry, using the trajectory-based alignment for initial registration. Our method achieves 2.489 / 0.895 cm (before/after ICP), compared to 4.090 / 1.256 cm for SAM3D-Tracker, demonstrating superior reconstruction quality even after decoupling pose errors.
Mask sources and robustness
We use ground-truth object masks for all methods in quantitative evaluations to isolate tracking and reconstruction performance from segmentation errors. For real-world demos, we use SAM3.
On our ARCTIC set, SAM3 achieves 0.89 mean mask IoU against ground truth, indicating reliable segmentation. Jointly refining masks and reconstruction is possible but would require significant pipeline changes and is beyond the scope of this work.