acceptodds
Under review as a conference paper at ICLR 2027

AgentSTAR: Agentic Shape Tracking and Reconstruction from Monocular Videos

20 views

Abstract

In this work, we present a method for shape reconstruction and tracking from video via agentic analysis-by-synthesis. Unlike prior methods which first estimate dense pixel correspondences and then recover object motion from them, our method infers a structured 3D object model, including its geometry and kinematic structure, and uses this model to optimise object track estimates over time. In our optimisation loop, a Vision-Language Model (VLM) agent iteratively refines shape or generalised pose through a render-and-compare loop, combining coarse visual reasoning with numerical pose optimisation for precise state estimation. This structured formulation enables our method to track through large motion, articulation, and severe occlusion without relying on pixel-matching objectives. Quantitatively, on ARCTIC, our method substantially outperforms state-of-the-art 3D point-tracking baselines for articulated objects, and on HOT3D it outperforms all evaluated rigid-object tracking baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
agentic optimisation, 3D tracking, 3D mapping, agentic vision, 4D reconstruction, VLM, articulated objects

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_y92xOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AgentSTAR presents an agentic analysis-by-synthesis framework for jointly recovering object geometry, kinematic structure, and motion from video. A VLM agent constructs and updates a code-based object model through visual feedback, numerical pose optimization, and temporal diagnostics. The method is evaluated on ARCTIC, HOT3D, and the RGB-D iTACO benchmark. I view its central contribution as an integrated framework connecting agentic 3D reconstruction with kinematic modeling and motion recovery.

Critical strengths

S1. An interesting and reasonably novel integration of agentic reconstruction, kinematic modeling, and motion estimation

The framework connects object geometry, articulated parts, joints, and their motion within a coherent optimization process. I find this integration interesting and reasonably novel.

The structured representation also has meaningful potential for robotics. For example, manipulating an articulated object requires identifying the movable component and understanding how it connects to the rest of the object. The recovered representation could support consistent part-level mask rendering and reasoning about opening or closing such objects. These applications are not directly demonstrated, but they provide a meaningful motivation for recovering geometry and kinematics together.

S2. Encouraging quantitative results across tracking and articulated reconstruction

The reported improvements are substantial under the stated protocols. ARCTIC EPE decreases from 7.65 cm for V-DPM to 5.59 cm, while HOT3D mean translation error decreases from 6.41 cm for SAM3D-Tracker to 3.04 cm. On iTACO, revolute-axis error decreases from 0.32 to 0.08 rad relative to iTACO.

The improvement from 11.26 cm for No Harness to 5.59 cm for the default full system also provides meaningful evidence that the specialized tools and orchestration are useful.

Critical weaknesses

W1. The contribution of the specialized harness and its computational tradeoff need clearer characterization

The No Harness comparison and common ten-hour cap are useful controls, and Section 4.1 suggests that the default GPT backbone is shared. However, Table 2 does not explicitly document each variant’s backbone, host agent, critic configuration, available tools, and actual resource consumption. Fable 5 has no corresponding No Harness result, and its host also changes to Claude Code.

The paper discloses approximately 18M tokens per ARCTIC sequence and an estimated GPT API cost of approximately $300 per sequence, but lacks per-variant measured runtime and token/cost statistics. Consequently, the incremental benefit of the specialized framework at comparable budgets, particularly with the strongest reported model, remains insufficiently characterized. This makes it difficult to distinguish the contributions of the framework, the underlying model, and additional inference-time computation.

W2. The comparison tables insufficiently establish the reconstruction advantage and do not adequately contextualize the offline pose results

Table 3 evaluates pose accuracy, not reconstruction quality. Its translation and rotation errors demonstrate improved pose estimation under the stated protocol, but do not show whether AgentSTAR reconstructs more accurate geometry than the SAM3D-based pipeline. Better pose accuracy alone cannot establish better reconstruction.

The reconstruction comparisons are limited. Table 1 reports Chamfer distance, but compares geometry only against V-DPM, a point-tracking method. This provides limited evidence relative to competitive reconstruction pipelines. Table 4 provides relevant articulated-reconstruction evidence and shows strong part-geometry and kinematic results. However, whole-object CD is 0.02 versus 0.01 m² for iTACO, so the results do not establish a consistent advantage across all aspects of reconstruction quality.

The pose comparison also needs to account for different computational settings. AgentSTAR repeatedly revisits the observed video within a ten-hour optimization budget, whereas FoundationPose is designed for online tracking. The original FoundationPose reports approximately 32 Hz on an RTX 3090; this is a reference speed, not an end-to-end measurement of the FoundationPose + VGGT-Ω pipeline evaluated here. The table should report the actual evaluated pipelines’ runtimes and temporal access alongside accuracy. See FoundationPose, Section 4.5.

The joint contribution therefore needs separate evidence for geometry and kinematic recovery against appropriate reconstruction baselines, and for pose accuracy together with computational cost and access to future frames. Substantial computation could be justified by substantially better structured reconstruction. However, the current pose comparison does not establish that tradeoff, and lower offline pose error alone does not demonstrate a preferable practical tracking system.

W3. Mask provenance and quality are insufficiently characterized

Section 3 permits masks from a video segmentation model or masks provided directly, but does not clearly specify their actual sources, quality, or manual correction for each benchmark.

Since silhouette IoU drives numerical optimization, missing parts, inaccurate boundaries, or incorrect hand masks could bias both reconstructed geometry and estimated poses. Agreement with the input mask does not establish that mask’s accuracy. No dedicated evaluation of mask quality or sensitivity to segmentation errors is reported.

The paper should clarify whether the reported gains hold with realistic predicted masks and what mask information is available to each baseline. This uncertainty affects both practical reliability and interpretation of the comparisons.

Questions for authors

Q1. Model and budget attribution — corresponding to W1

Please specify the backbone, host agent, critic configuration, available tools, and stopping criteria for No Harness and the full system. Can you report their actual runtime, token consumption, API cost, and numerical search effort?

Does the improvement persist with the same model under comparable computational budgets? A matched Fable 5 control on a representative subset, if feasible, would help interpret the strongest reported variant.

Q2. Reconstruction comparisons — corresponding to W2

Can the geometry produced by the SAM3D-based pipeline already evaluated in Table 3 be evaluated alongside AgentSTAR, or can another appropriate reconstruction baseline be included under clearly stated input conditions?

Please clarify whether the geometry evaluation separates shape error from pose error, and identify which reconstruction or kinematic advantages justify the additional optimization cost.

Q3. Offline versus online comparison — corresponding to W2

For Table 3, please specify each method’s temporal access, input assumptions, and measured end-to-end runtime, including the VGGT-Ω stage where applicable.

What quality–cost tradeoff follows from earlier stopping points in the existing optimization traces? How should the offline pose improvements be interpreted relative to online tracking and the intended applications?

Q4. Mask sources and robustness — corresponding to W3

For each benchmark, are object and hand masks ground-truth, automatically predicted, or manually corrected? Which segmentation model and prompting procedure are used, and what mask information is available to each baseline?

Can you quantify mask quality or provide a limited assessment of sensitivity to realistic segmentation errors, such as missing parts or inaccurate boundaries?

5 comments

Experiment budget and different modelling options

We provide compute cost analysis in supplementary A1, with exact configurations available in our supplementary code. Figure 5 shows the compute-accuracy trade-off for GPT-Sol 5.6.

We additionally evaluated Fable 5 without our harness, obtaining 6.50cm tracking error versus 4.95cm with our tooling. Even with a stronger backbone, our specialised framework provides substantial improvement.

Consequently, the incremental benefit of the specialized framework at comparable budgets, particularly with the strongest reported model, remains insufficiently characterized

Effective use of inference-time compute is itself a key contribution of our harness. It enables continued iterative refinement, as shown in Figure 5. We therefore argue that a common maximum runtime budget is more appropriate than identical compute consumption.

Future VLMs may become proficient at this task through specialised post-training, without requiring a specialised harness. However, we view our central contribution as a general computational formulation for agentic optimisation, which iteratively couples VLM reasoning with traditional numerical optimisation. AgentSTAR showcases this paradigm for shape reconstruction and tracking, but we believe it applies to a broader class of challenging optimisation problems.

Computational trade-offs; offline versus online tracking

We compare methods with different computational budgets to contextualise the benefits of additional inference-time computation and the analysis-by-synthesis paradigm. For example, SAM3D-Tracker, the strongest baseline, takes approximately 40 minutes per sequence.

Regarding online/offline comparison: FoundationPose is an RGB-D tracker. To run on RGB videos, it requires depth estimation via VGGT-Omega, which processes the full sequence and makes our evaluated pipeline non-streaming.

Reconstruction comparison on HOT3D

Pose and reconstruction errors are coupled. To isolate geometry quality, we evaluate Chamfer distance both before and after ICP alignment to ground-truth geometry, using the trajectory-based alignment for initial registration. Our method achieves 2.489 / 0.895 cm (before/after ICP), compared to 4.090 / 1.256 cm for SAM3D-Tracker, demonstrating superior reconstruction quality even after decoupling pose errors.

Mask sources and robustness

We use ground-truth object masks for all methods in quantitative evaluations to isolate tracking and reconstruction performance from segmentation errors. For real-world demos, we use SAM3.

On our ARCTIC set, SAM3 achieves 0.89 mean mask IoU against ground truth, indicating reliable segmentation. Jointly refining masks and reconstruction is possible but would require significant pipeline changes and is beyond the scope of this work.

Thank you for the thoughtful responses and additional experiments. The same-backbone comparison and new reconstruction results strengthen the paper and address important parts of my earlier concerns. I appreciate these improvements.

My main remaining concern is W3. The quantitative evaluations use GT object masks, whereas practical use would generally rely on predictions such as SAM3 masks. High whole-object mask IoU does not establish how segmentation errors affect reconstruction, kinematic estimation, and tracking. Small movable parts may be omitted, regions may be incorrectly split or merged, and coverage may vary across frames. Please also clarify whether any GT part masks or labels are provided as input.

Could you provide a controlled comparison on a representative subset, replacing GT object masks with uncorrected SAM3 predictions while keeping the other inputs, model, and optimization budget fixed? Please report changes in geometry, kinematic, and tracking errors, together with whole-object completeness and part/kinematic correctness, including missing or extra parts and joints. Please specify any part-specific prompting, manual selection, or correction required.

Given the substantial token consumption, monetary cost, and optimization time, it remains important to establish whether the benefits persist under realistic segmentation inputs and justify the additional expense. Please report actual resource consumption on this subset and, where feasible, compare the main baseline under the same predicted-mask conditions.

We thank the reviewer for the insightful experiment suggestion. No part labels were provided as input in all cases.

We repeat our experiment on the entire ARCTIC evaluation split, replacing GT object masks with predictions from the SAM3 video tracker, initialised with the first-frame GT mask to identify the object of interest. All subsequent masks were generated automatically, without part-specific prompting or manual correction, which reflects real-world scenarios. We kept our strongest variant (Fable 5 + our harness), all other inputs, harness and the optimisation budget unchanged.

MethodTracking EPE (cm)Chamfer (cm)
Fable 5 + Ours (GT masks)4.952.83
Fable 5 + Ours (SAM masks)5.473.58
VDPM7.654.72

This remains substantially better than the strongest baseline, V-DPM. Importantly, the recovered kinematics do not miss any annotated degrees of freedom. Occasional additional joints correspond to valid but unannotated articulations (e.g mixer's whisk), and also occur with GT masks.

Reviewer_WQoyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AgentSTAR to recover object geometry, articulated structure, and motion trajectories from videos. The method alternates between VLM-driven shape editing and numerical pose optimization. The VLM guides pose searches and selects candidates using visual feedback, while temporal diagnostics help identify inconsistent motion estimates. Experiments show improvements over existing baselines in 3D point tracking and object pose tracking.

Critical strengths
  1. The motivation is reasonable: explicit object structure and articulation constraints complement unreliable visual correspondences under occlusion and large motion, helping improve the stability of 3D tracking.
  2. The division of responsibilities is clear and complementary: the VLM provides coarse reasoning, while numerical optimization performs precise fitting.
  3. The method achieves notable improvements across multiple datasets. The paper also includes tool ablations and analyses of run-to-run stability and computational cost.
Critical weaknesses
  1. Inference is computationally expensive, with a budget of up to 10 hours per optimization run and an average consumption of approximately 18 million tokens per sequence on ARCTIC. This limits the method’s practicality and scalability.
  2. Although the ablations demonstrate the effectiveness of the overall framework, they do not separately isolate the effects of iterative shape refinement, search-region guidance, and visual candidate selection. The individual contributions of these mechanisms to the final performance therefore remain unclear.
Questions for authors
  1. What specific inputs and prompts are used for the initial shape generation? Does the agent construct the initial geometry and articulated structure from scratch, or does it rely on predefined templates or external generative models?
  2. Where do the textures or materials shown in the figures come from? Are they iteratively updated alongside the shape?
  3. When the rendering does not match the observations, how does the agent decide whether to modify the shape or adjust the pose? For example, both object size and distance from the camera affect the projected size. How does the system avoid compensating for pose errors through incorrect shape modifications?
2 comments

Inference complexity

We expect this optimisation process to become faster with better base models. The compute/performance trade off is presented in Figure 5. For practical applications we think 4D reconstructions can be generated with our pipeline to make the next generation of feed-forward models better.

Contribution of search region and visual selection components

We did another ablation where the visual inspection after each optimisation run is not required: best IoU score candidate is chosen automatically. The resulting tracking error is significantly higher, 7.63 cm (compared to 5.59cm).

Search guidance is difficult to ablate in isolation, as the agent may independently reproduce similar search-region guided strategies. We therefore provide an agent-free global optimisation baseline with ground-truth geometry (Table 2, GT mesh + VLM-free opt.) as an alternative comparison.

Inputs and additional tooling used for shape generation

No external templates or generative models were used: 3D model and kinematic structure is authored from images and segmentation masks only. Full pipeline and prompts can be inspected in the attached supplementary code.

Are textures part of the shape iteration?

Yes, they are part of the shape reconstruction and are authored via blender python interface.

Shape-pose ambiguity in a monocular setting

The agent decides, but our representation constrains the scale changes. The canonical shape and its scale are shared across all frames, while estimated poses are per-frame. A shape or scale change therefore affects every keyframe at once and cannot absorb a pose error in a single frame without degrading the others.

Thank you for the detailed responses and additional ablation. The clarifications regarding shape initialization, texture modeling, and shape-pose ambiguity are helpful. I will maintain my original rating.

Reviewer_bHxXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper describes an analysis-by-synthesis system (AgentSTAR) that reconstructs the geometry and tracks the overall and per-part poses of (articulated) objects with Vision-Language Models (VLM). Shape are represented as Blender primitives (python code) and poses are defined as Sim(3) plus per-part SE(3). The VLM is prompted to iterate between shape and pose adjustment to gradually align the reconstruction to the input video based on visual comparison and silhouette Intersection over Union (IoU).

AgentSTAR is compared to feedforward point tracking methods on ARCTIC, feedforward rigid-object tracking methods on HOT3D, and kinematic estimation methods on iTACO, showing consistent superiority in performance.

Critical strengths
  1. This paper forms the first agentic method for articulated object reconstruction and tracking. The general idea of combining visual reasoning and numerical optimization is reasonable. The pipeline is complete and the visual results on the web page are impressing.
  2. The authors mention the time and money cost of the agentic optimization, and provide one use case of such system -- producing data for feedforward methods.
Critical weaknesses
  1. The comparison to feedforward 3d tracking methods are limited to numbers. No qualitative comparison of 3D tracking trajectories are shown to tell if the gap exist in geometry or motion. Meanwhile, the comparison isn't very meaningful as AgenticSTAR is the only optimization-based method that is allowed to fit to a specific scene. Therefore, the most meaningful comparison is on iTACO (with other optimization-based methods) where AgentSTAR leads on every metric except the whole object chamfer distance (CD-w). It is then a bit confusing why AgentSTAR gets the best performance on both static (CD-s) and moving parts (CD-m) but falls behind on CD-w.

  2. Under the idea of combining visual reasoning and numerical optimization, it is not mentioned why the authors chose the Differential Evolution method for pose refinement, which is known to be less efficient than gradient-based optimization. How much does the number of candidate solutions affect the final accuracy and total optimization time? What is the terminal condition for the agent (e.g., time limit, token limit, IoU)?

Questions for authors
  1. An obvious alternative to Differential Evolution could be photometric optimization with differentiable rendering of the silhouette, which potentially requires less VLM verification. More discussion on this part could largely enhance this paper.
3 comments

Qualitative tracking comparison

Qualitative comparison was added to the supplementary, section B.

Fairness of comparisons to feedforward methods

Existing correspondence-based optimisation methods are difficult to apply to ARCTIC due to rapid motion, articulation, and limited visual overlap between keyframes (e.g., a book transitioning between open and closed states). Feed-forward methods are more robust to these conditions but remain less precise, as demonstrated by our evaluations.

Our analysis-by-synthesis approach aims to enable optimisation precisely in these challenging scenarios. In contrast, iTACO's PartNet-based benchmark features more controlled motion and RGB-D observations, making correspondence-based optimisation feasible.

CD-w versus CD-m/CD-s quality

CD-w computes bidirectional nearest-neighbor distances over the entire object, ignoring moving/static labels, whereas CD-m and CD-s restrict matching to the corresponding part. Part-wise scores reflect both geometry and part-assignment accuracy, while CD-w measures overall geometry only.

We additionally measure moving/static label agreement using bidirectional nearest-neighbor matching over the whole object. On moving parts, AgentSTAR achieves 68.6% accuracy versus 63.2% for iTACO. Although this metric still remains coupled to reconstruction quality, we think this provides a good intuition for interpreting these results.

Besides, iTACO also used 24 views uniformly sampled around the object to acquire the initial object shape which gives it upside in acquiring a more accurate surface, because it has access to perfect multi-view depth information.

Choice of Differential Evolution as optimiser

We chose Differential Evolution because it allows us to directly optimise the non-differentiable silhouette IoU objective. Gradient-based alternatives would require a differentiable substitute, such as soft silhouettes, or a different objective, such as photometric error, which introduces sensitivity to lighting and non-Lambertian surfaces.

While differentiable optimisation is certainly possible, it would introduce additional design choices. Our main contribution is demonstrating how VLM-guided search can effectively optimise challenging, non-differentiable objectives without requiring differentiable rendering.

Candidate population size and runtime

Tweaking the population size is left to the agent. Our initial population size budgeting is 8 per active degree of freedom, which in practice results in ~48 candidates on average. Numerical optimisation accounts for only ~10% of pose-iteration runtime, with the remainder dominated by agent reasoning, rendering, and visual inspection. The numerical search itself is not the primary computational bottleneck

Agent termination criteria

Agents may terminate early at their discretion once the silhouette IoU exceeds 0.9. All runs are subject to a maximum wall-clock budget of 10 hours.

We have also included side-by-side video comparisons of tracking and geometry between our method and VDPM on the ARCTIC dataset. We encourage the reviewer to inspect the updated supplementary material.

I thank the authors for the reply and addtional qualitative results. I will maintain my original rating of 3 (Weak Accept).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comments

We thank all reviewers for their thoughtful and constructive feedback. We are encouraged by their positive assessment of our approach and its experimental results. We wish to highlight that our central contribution is a general formulation for agentic optimisation, iteratively coupling VLM reasoning with numerical optimisation. AgentSTAR demonstrates this paradigm for structured 3D reconstruction and tracking under challenging conditions where conventional correspondence-based methods struggle.

In response to the reviews, we strengthened our experimental evidence:

  • Reviewer bHxX: Added qualitative ARCTIC comparisons (both supplementary videos and paper figures) against GT and VDPM, illustrating improvements in both tracking and geometry.
  • Reviewer WQoy: Ablating VLM visual candidate selection increases tracking error from 5.59 to 7.63 cm, demonstrating its importance. Moreover, fully “blind” VLM-free numerical optimisation achieves 14.60 cm even with ground-truth geometry, demonstrating that an optimiser on its own is not enough.
  • Reviewer y92x: Added Fable 5 backbone ablations (6.50 versus ours 4.95 cm EPE) and reconstruction comparisons against SAM3D (1.256 versus ours 0.895 cm chamfer distance after ICP)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.