- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces CRAFT, which uses coding agents to automatically modify and improve the controller of a visual tracker while keeping the visual model fixed. The agent repeatedly changes tracking logic, such as search-region selection, template updates, confidence rules, and recovery, and uses development-set performance to decide which changes are better. Therefore, although CRAFT is presented as code-level optimization, a large part of the process remains similar to automated search or tuning of tracking rules and hyperparameters. The final tracker does not use an LLM at inference time and reports improvements on several tracking benchmarks.
Critical strengths
The idea of using coding agents to automatically hyperparameterize code and improve tracking controllers is interesting and potentially useful.
Critical weaknesses
The novelty, reproducibility, and generality of the agent-driven design are not yet sufficiently established. Several discovered mechanisms appear related to prior visual-search and controller-design approaches, while the overall procedure also resembles an automated search over controller programs based on development-set performance. It is therefore important to clarify what the LLM agent contributes beyond conventional AutoML- or NAS-style search. Moreover, all reported experiments use GPT-5.6-sol and closely related tracker configurations, with no repeated runs or evaluation across different agent families and tracker architectures.
Questions for authors
-
The search-region selection and recovery mechanisms appear related to CPDTrack, “Beyond Accuracy: Tracking More Like Human via Visual Search” (NeurIPS 2024). Please clarify the key conceptual and technical differences, particularly in how search expansion, re-detection, and recovery are designed.
-
(a) How is CRAFT fundamentally different from prior AutoML/NAS-based tracking methods such as LightTrack (CVPR 2021) and CHASE (BMVC 2021), beyond using an LLM agent as the search mechanism? Since CRAFT repeatedly modifies a tracker and selects revisions according to development-set performance before deployment, the distinction from automated parameter or architecture search is important for assessing novelty. Please clarify what capabilities the coding agent provides beyond conventional search. (b) It would also be valuable to discuss how CRAFT’s code-level offline optimization compares with weight-level online adaptation methods, such as ToMP (CVPR 2022) and GOT-Edit (ICLR 2026). In particular, online weight updates appear to offer greater adaptability to dynamic scenarios than offline code optimization.
-
All CRAFT experiments use GPT-5.6-sol. Given that agent behavior can be sensitive to the underlying model and that specific model versions may change or be deprecated over time, could the authors report several independent CRAFT runs and evaluate at least one additional model family or model version? Please report performance variance and whether similar mechanisms are consistently discovered across runs. Evaluating CRAFT on a genuinely different tracker architecture would also strengthen the generality claim.
-
(a) The GOT-10k comparison requires clarification. The base tracker is trained on multiple datasets, while CRAFT additionally uses labeled TNL2K training videos to optimize the controller. Please clarify which baselines follow the strict GOT-10k-only protocol and provide a protocol-matched comparison. (b) Additional evaluation on common and challenging benchmarks such as NfS, OTB-100, AVisT, and an appropriate VOT benchmark would also help assess robustness to occlusion, disappearance, and recovery.
-
Please provide the complete template-cropping details, including the exact template crop factor, the amount of surrounding background, padding, and resizing. The tracker uses four template slots; please also provide an ablation on the number of template slots, or explain why this cannot be varied.
-
Many decisions depend on confidence, including template admission, loss detection, recovery, and SAM3-based search. These values may be sensitive. Please report the exact threshold and calibration settings, together with a sensitivity analysis of the major parameters.
-
Please make the comparison in Table 1 more transparent. For each tracker, I suggest reporting the visual backbone, training data, input resolution, number of templates or memory capacity, parameter count, and inference speed. The compared methods use substantially different backbones and settings, making cross-method comparisons difficult to interpret fairly.
-
I would also appreciate a video demo showing an example agent modification, the corresponding code change, and the resulting tracking behavior.
8 comments
Response to Reviewer k3Wo (2/4): online weight adaptation, repeated runs, other agents and tracker architectures
Q2(b). Code-level offline optimization vs. weight-level online adaptation (ToMP, GOT-Edit).
- They change different parts of the tracker. ToMP predicts the weights of its target model online with a Transformer, from a memory of reference frames, and GOT-Edit edits the model online to inject geometric cues under a null-space constraint. Both adapt the visual model. CRAFT leaves the model unchanged and optimizes the controller, the program logic that decides how the model is used.
- Online adaptation still depends on controller rules. In both ToMP and GOT-Edit, hand-written rules decide which frames the model learns from, when it updates, and when it stops learning during occlusion. A wrong decision lets the model learn from a distractor or the background, a known failure mode of online adaptation. These rules are what CRAFT optimizes, and the pattern it discovers repeatedly, verifying a candidate before it changes the tracker state, helps prevent such updates. The two approaches are therefore potentially complementary.
- Fixed code does not mean fixed behavior. The code of the CRAFT controller is fixed after design, but its behavior adapts online through per-object calibration, motion trust and a quality-gated template memory, all computed from the current sequence.
- The optimization happens once, offline. The deployed CRAFT controller then runs online with no need for further optimization. Weight-level methods instead adapt the model online in every sequence, so they have to pay the cost repeatedly in every sequence.
We have added a brief version of this discussion to Related Work in the revised paper.
Q3. Independent runs, other agent families, and another tracker architecture.
Repeated runs and other agents. The table gives held-out AUC (LaSOT / LaSOT$_\mathrm{ext}$ / TNL2K) of complete CRAFT runs on Seed-B256 with unchanged design spaces, constraints and budget. ΔH is the change in mean held-out AUC over the seed (61.7).
| Controller | LaSOT / LaSOT$_\mathrm{ext}$ / TNL2K | ΔH |
|---|---|---|
| Seed controller | 72.6 / 52.0 / 60.6 | – |
| CRAFT with GPT-5.6-sol, 3 runs | 74.7±0.1 / 54.2±0.3 / 63.7±0.1 | +2.4 |
| CRAFT with Claude Opus 4.8 | 74.6 / 54.4 / 63.5 | +2.4 |
| CRAFT with DeepSeek-V4.1-Flash (open weights) | 74.4 / 53.9 / 63.2 | +2.1 |
- Every run improves every benchmark. The three GPT-5.6-sol runs have a standard deviation of at most 0.3 AUC per benchmark. On Seed-SAM3, three runs give 78.8±0.1 / 65.9±0.4 / 68.3±0.2 against 77.6 / 63.8 / 66.1 for the seed, and again every run improves every benchmark. We also repeated every ablation variant three times. The revised paper reports these means and standard deviations in Table 2 and lists every run in App. F.
- Other agent families reach similar gains. Claude Opus 4.8 matches GPT-5.6-sol, and even DeepSeek-V4.1-Flash, a fast open-weight model, reaches +2.1 (App. F of the revised paper).
- The runs discover the same mechanisms. Every run places the largest scoped gain in template memory, verification before a state change appears in 5/5 runs, and per-object calibration in 4/5.
- Model deprecation. Agents are used only during design. The deployed result is ordinary tracker code that runs without any agent, and the open-weight run shows that the procedure does not depend on a proprietary model either. As coding agents become stronger, the same procedure can use them without change, which we expect to leave further room in this direction.
Tracker architectures. The two trackers in the paper cover the two model families most used in current tracking. Seed-B256 is a one-stream Transformer tracker, and Seed-SAM3 adds SAM3, a segmentation foundation model, to the model set. In particular, Seed-SAM3 starts from a very strong point in the tracking field. Its LaSOT AUC of 77.6 is already above every SAM-based tracker in Table 1, and CRAFT still improves it on every benchmark, to 78.8 / 65.9 / 68.3. We do not claim that CRAFT generalizes to every tracker, but these results suggest that it holds, to some extent, across trackers of different types and strengths.
Response to Reviewer k3Wo (3/4): GOT-10k protocol, more benchmarks, and template details
Q4(a). GOT-10k protocol. The GOT-10k results, which we report in the appendix, are not a restricted-protocol comparison. GOT-10k defines a one-shot protocol in which trackers are trained only on its training split. Some recent trackers instead report GOT-10k after training on multiple datasets, so Table 5 contains entries of both kinds. Our seed belongs to the second group, and CRAFT additionally uses TNL2K-train for controller development. In App. C of the revised paper, we now state this difference explicitly and present our GOT-10k results as an unrestricted-data evaluation, a reference rather than a protocol-matched ranking. The seed-to-CRAFT gain is unaffected, since both rows use identical weights. Our main results remain those on LaSOT, LaSOT$_\mathrm{ext}$ and TNL2K.
Q4(b). More benchmarks. We evaluated the same programs, unchanged, on four more benchmarks (EAO on VOT2022-STB, AUC on the others).
| Tracker | VOT2022-STB (EAO) | AVisT | NfS30 | OTB-100 |
|---|---|---|---|---|
| Seed-B256 | 0.574 | 59.0 | 70.0 | 70.5 |
| CRAFT-B256 | 0.615 | 61.1 | 70.0 | 72.7 |
| Seed-SAM3 | 0.774 | 65.4 | 72.7 | 74.2 |
| CRAFT-SAM3 | 0.788 | 67.8 | 72.5 | 74.5 |
CRAFT improves VOT2022-STB, AVisT and OTB-100 on both trackers and leaves NfS30 nearly unchanged (0.0 and −0.2 AUC). The gains are largest on AVisT (+2.1 and +2.4), whose adverse visibility conditions cause frequent occlusion and appearance change. NfS30 and OTB-100 are small, with 100 sequences each, and largely saturated, so recent trackers differ little on them, and we read them as a consistency check. We have added these results to App. C of the revised paper.
Q5. Template cropping and the number of template slots. Every template is a square crop of side 2.0·√(wh) centered on the box, so the target covers about a quarter of the crop and the rest is surrounding context. Out-of-image pixels are zero-padded, and the crop is resized bilinearly to 128×128 (now stated in App. B). This cropping is the standard choice of recent trackers such as TransT (CVPR 2021), OSTrack (ECCV 2022), ODTrack (AAAI 2024) and SUTrack (AAAI 2025), and CRAFT does not change it.
The tracker uses four template slots. We keep this number fixed for two reasons.
- It belongs to the trained model, not to the controller. The frozen model was trained with four templates, and changing their number changes its inference, which constraint (i) of the paper excludes. Four is also a common choice. ODTrack, for example, feeds four templates at inference, the first frame and three frames sampled from its memory.
- The comparison is matched. The seed and CRAFT use the same four slots, so every reported gain comes from the controller rather than from template capacity. What the controller decides is which observations enter these slots and when.
Response to Reviewer k3Wo (1/4): CPDTrack, and what the coding agent adds beyond AutoML/NAS
Thank you for the detailed review and specific questions. We address each of them below.
Novelty and contribution. CRAFT automates the design of tracking controllers with coding agents, organized by a design procedure that we propose. We agree that, like an automated search, CRAFT selects programs by development-split performance, as Eq. 2 states. CRAFT makes two claims. For visual object tracking, the controller around a frozen model holds accuracy that is still largely untapped, and a procedure can find better controllers automatically and output them as ordinary tracker code. The novelty therefore lies in this optimization target and in how CRAFT organizes the design over it, not in the generate-evaluate-select loop itself. To our knowledge, CRAFT is also the first work to design tracking controllers with coding agents, and with this procedure the resulting controllers improve both trackers on every held-out benchmark.
Q1. CPDTrack. CPDTrack builds human-like visual search into the trained network. It uses a Gaussian-acuity central view and a resized full-image peripheral view, linked by a temporal information query and trained end to end. CRAFT differs in where the mechanism lives (controller code around an unchanged model, with no training), in who designs it (an agent driven by development feedback), and in how it behaves:
We have added CPDTrack to Related Work in the revised paper. Thank you for the suggestion.
Q2(a). Beyond AutoML and NAS.
CRAFT differs from these methods in three ways.
The two lines of work are potentially complementary, since a NAS-found network could serve as the frozen model under a CRAFT controller.