CRAFT: Coding Agents for Visual Tracking Controller Design

A visual tracker couples frozen visual model components with an online tracking controller: ordinary program logic that maintains state and determines how visual components are used over time. Although visual models have advanced substantially, controllers still encode manually designed choices about where to search, how to estimate target state, which templates to retain, and when to recover. We introduce Code Refinement by Agents for Tracking (CRAFT), a computational framework in which coding agents propose and test controller revisions offline under purpose-constrained design tasks while keeping the visual model components and their internal inference procedures fixed. CRAFT organizes exploration around four controller operations: search-region selection, response decoding, template-memory management, and failure recovery. First, candidate revisions are sampled for each operation and selected according to tracking performance on a development split. A composing agent then integrates the selected mechanisms without access to their performance metrics. Guided by evidence from earlier revisions, joint refinement then returns to the coupled controller space to address interactions among the discovered mechanisms. The resulting controller runs as ordinary tracker code, with no language model at inference. Using the TNL2K training split as its sole source of tracking feedback, CRAFT improves performance on every reported held-out benchmark: LaSOT AUC increases from 72.6% to 74.6% for a base-scale one-stream tracker and from 77.6% to 78.8% for its SAM3-augmented counterpart, with gains of 2.1 to 3.0 points on LaSOT${}_{\mathrm{ext}}$ and the TNL2K test split. Code, discovered controllers, and models will be released.