acceptodds
Under review as a conference paper at ICLR 2027

CRAFT: Coding Agents for Visual Tracking Controller Design

Abstract

A visual tracker couples frozen visual model components with an online tracking controller: ordinary program logic that maintains state and determines how visual components are used over time. Although visual models have advanced substantially, controllers still encode manually designed choices about where to search, how to estimate target state, which templates to retain, and when to recover. We introduce Code Refinement by Agents for Tracking (CRAFT), a computational framework in which coding agents propose and test controller revisions offline under purpose-constrained design tasks while keeping the visual model components and their internal inference procedures fixed. CRAFT organizes exploration around four controller operations: search-region selection, response decoding, template-memory management, and failure recovery. First, candidate revisions are sampled for each operation and selected according to tracking performance on a development split. A composing agent then integrates the selected mechanisms without access to their performance metrics. Guided by evidence from earlier revisions, joint refinement then returns to the coupled controller space to address interactions among the discovered mechanisms. The resulting controller runs as ordinary tracker code, with no language model at inference. Using the TNL2K training split as its sole source of tracking feedback, CRAFT improves performance on every reported held-out benchmark: LaSOT AUC increases from 72.6% to 74.6% for a base-scale one-stream tracker and from 77.6% to 78.8% for its SAM3-augmented counterpart, with gains of 2.1 to 3.0 points on LaSOT and the TNL2K test split. Code, discovered controllers, and models will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Object Tracking, Agent-driven Algorithm Design, Tracking Controller

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_k3WoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CRAFT, which uses coding agents to automatically modify and improve the controller of a visual tracker while keeping the visual model fixed. The agent repeatedly changes tracking logic, such as search-region selection, template updates, confidence rules, and recovery, and uses development-set performance to decide which changes are better. Therefore, although CRAFT is presented as code-level optimization, a large part of the process remains similar to automated search or tuning of tracking rules and hyperparameters. The final tracker does not use an LLM at inference time and reports improvements on several tracking benchmarks.

Critical strengths

The idea of using coding agents to automatically hyperparameterize code and improve tracking controllers is interesting and potentially useful.

Critical weaknesses

The novelty, reproducibility, and generality of the agent-driven design are not yet sufficiently established. Several discovered mechanisms appear related to prior visual-search and controller-design approaches, while the overall procedure also resembles an automated search over controller programs based on development-set performance. It is therefore important to clarify what the LLM agent contributes beyond conventional AutoML- or NAS-style search. Moreover, all reported experiments use GPT-5.6-sol and closely related tracker configurations, with no repeated runs or evaluation across different agent families and tracker architectures.

Questions for authors
  1. The search-region selection and recovery mechanisms appear related to CPDTrack, “Beyond Accuracy: Tracking More Like Human via Visual Search” (NeurIPS 2024). Please clarify the key conceptual and technical differences, particularly in how search expansion, re-detection, and recovery are designed.

  2. (a) How is CRAFT fundamentally different from prior AutoML/NAS-based tracking methods such as LightTrack (CVPR 2021) and CHASE (BMVC 2021), beyond using an LLM agent as the search mechanism? Since CRAFT repeatedly modifies a tracker and selects revisions according to development-set performance before deployment, the distinction from automated parameter or architecture search is important for assessing novelty. Please clarify what capabilities the coding agent provides beyond conventional search. (b) It would also be valuable to discuss how CRAFT’s code-level offline optimization compares with weight-level online adaptation methods, such as ToMP (CVPR 2022) and GOT-Edit (ICLR 2026). In particular, online weight updates appear to offer greater adaptability to dynamic scenarios than offline code optimization.

  3. All CRAFT experiments use GPT-5.6-sol. Given that agent behavior can be sensitive to the underlying model and that specific model versions may change or be deprecated over time, could the authors report several independent CRAFT runs and evaluate at least one additional model family or model version? Please report performance variance and whether similar mechanisms are consistently discovered across runs. Evaluating CRAFT on a genuinely different tracker architecture would also strengthen the generality claim.

  4. (a) The GOT-10k comparison requires clarification. The base tracker is trained on multiple datasets, while CRAFT additionally uses labeled TNL2K training videos to optimize the controller. Please clarify which baselines follow the strict GOT-10k-only protocol and provide a protocol-matched comparison. (b) Additional evaluation on common and challenging benchmarks such as NfS, OTB-100, AVisT, and an appropriate VOT benchmark would also help assess robustness to occlusion, disappearance, and recovery.

  5. Please provide the complete template-cropping details, including the exact template crop factor, the amount of surrounding background, padding, and resizing. The tracker uses four template slots; please also provide an ablation on the number of template slots, or explain why this cannot be varied.

  6. Many decisions depend on confidence, including template admission, loss detection, recovery, and SAM3-based search. These values may be sensitive. Please report the exact threshold and calibration settings, together with a sensitivity analysis of the major parameters.

  7. Please make the comparison in Table 1 more transparent. For each tracker, I suggest reporting the visual backbone, training data, input resolution, number of templates or memory capacity, parameter count, and inference speed. The compared methods use substantially different backbones and settings, making cross-method comparisons difficult to interpret fairly.

  8. I would also appreciate a video demo showing an example agent modification, the corresponding code change, and the resulting tracking behavior.

8 comments
AuthorsOpenReview ↗

Response to Reviewer k3Wo (1/4): CPDTrack, and what the coding agent adds beyond AutoML/NAS

Thank you for the detailed review and specific questions. We address each of them below.

Novelty and contribution. CRAFT automates the design of tracking controllers with coding agents, organized by a design procedure that we propose. We agree that, like an automated search, CRAFT selects programs by development-split performance, as Eq. 2 states. CRAFT makes two claims. For visual object tracking, the controller around a frozen model holds accuracy that is still largely untapped, and a procedure can find better controllers automatically and output them as ordinary tracker code. The novelty therefore lies in this optimization target and in how CRAFT organizes the design over it, not in the generate-evaluate-select loop itself. To our knowledge, CRAFT is also the first work to design tracking controllers with coding agents, and with this procedure the resulting controllers improve both trackers on every held-out benchmark.

Q1. CPDTrack. CPDTrack builds human-like visual search into the trained network. It uses a Gaussian-acuity central view and a resized full-image peripheral view, linked by a temporal information query and trained end to end. CRAFT differs in where the mechanism lives (controller code around an unchanged model, with no training), in who designs it (an agent driven by development feedback), and in how it behaves:

  • Normal search has no peripheral stream. The crop shifts ahead along motion that has been consistent over several frames, with per-axis limits, and resets when the target stops or reverses.
  • Re-detection runs only when the tracker state indicates loss. The controller suspects loss when raw confidence stays far below the object's own running trusted level, with hysteresis. The search then widens around the last reliable position and, if loss persists, scans target-scale tiles ordered along the last motion direction.
  • Re-acquisition requires score, sharpness and geometry gates, a second consistent detection, and a backward check: used as a template, the candidate must re-find the target in the last reliable frame.

We have added CPDTrack to Related Work in the revised paper. Thank you for the suggestion.

Q2(a). Beyond AutoML and NAS.

LightTrack / CHASEHyperparameter searchCRAFT
Searched objectModel architecture, trained after the searchConstants of fixed codeSource code of the tracking controller
SpacePredefined operatorsPredefined rangesAny program satisfying the constraints of Sec. 3.1, not enumerated in advance
Search organizationOne-shot supernet (LightTrack) or differentiable cell search (CHASE)Sampler over the rangesScoped exploration, composition, joint refinement
Evidence per candidateScalar scoreScalar scoreAggregate and per-sequence AUC, speed, failure logs, earlier source code, and the development videos and predictions
OutputNetwork, still run by a hand-written controllerConstantsReadable controller code

CRAFT differs from these methods in three ways.

  1. The agent writes program structure that no predefined space contains, such as a trial run of each new template set before it is activated, or the backward check of recovery candidates described in Q1. Of the 180 revisions on the two seed trackers (90 each), 175 (97.2%) change the decision logic of the controller rather than only its constants.
  2. The agent reasons from the full evidence of tracking runs. It reads per-sequence AUC changes, failure logs and earlier code, and inspects the videos and predictions of the development split (Sec. 3.2), rather than receiving a single score per candidate.
  3. The design procedure organizes the agent's work. Scoped exploration makes each evaluation evidence about one operation, metric-blind composition integrates the selected mechanisms, and joint refinement revises their interactions with the scoped archives as evidence. Each stage matters. Over three repeated runs, replacing scoped exploration by a single unscoped campaign costs 1.4 held-out AUC, removing joint refinement costs 0.9, and withholding the scoped archives costs 0.3, and each margin holds in every run (Table 2 of the revised paper).

The two lines of work are potentially complementary, since a NAS-found network could serve as the frozen model under a CRAFT controller.

AuthorsOpenReview ↗

Response to Reviewer k3Wo (2/4): online weight adaptation, repeated runs, other agents and tracker architectures

Q2(b). Code-level offline optimization vs. weight-level online adaptation (ToMP, GOT-Edit).

  • They change different parts of the tracker. ToMP predicts the weights of its target model online with a Transformer, from a memory of reference frames, and GOT-Edit edits the model online to inject geometric cues under a null-space constraint. Both adapt the visual model. CRAFT leaves the model unchanged and optimizes the controller, the program logic that decides how the model is used.
  • Online adaptation still depends on controller rules. In both ToMP and GOT-Edit, hand-written rules decide which frames the model learns from, when it updates, and when it stops learning during occlusion. A wrong decision lets the model learn from a distractor or the background, a known failure mode of online adaptation. These rules are what CRAFT optimizes, and the pattern it discovers repeatedly, verifying a candidate before it changes the tracker state, helps prevent such updates. The two approaches are therefore potentially complementary.
  • Fixed code does not mean fixed behavior. The code of the CRAFT controller is fixed after design, but its behavior adapts online through per-object calibration, motion trust and a quality-gated template memory, all computed from the current sequence.
  • The optimization happens once, offline. The deployed CRAFT controller then runs online with no need for further optimization. Weight-level methods instead adapt the model online in every sequence, so they have to pay the cost repeatedly in every sequence.

We have added a brief version of this discussion to Related Work in the revised paper.

Q3. Independent runs, other agent families, and another tracker architecture.

Repeated runs and other agents. The table gives held-out AUC (LaSOT / LaSOT$_\mathrm{ext}$ / TNL2K) of complete CRAFT runs on Seed-B256 with unchanged design spaces, constraints and budget. ΔH is the change in mean held-out AUC over the seed (61.7).

ControllerLaSOT / LaSOT$_\mathrm{ext}$ / TNL2KΔH
Seed controller72.6 / 52.0 / 60.6–
CRAFT with GPT-5.6-sol, 3 runs74.7±0.1 / 54.2±0.3 / 63.7±0.1+2.4
CRAFT with Claude Opus 4.874.6 / 54.4 / 63.5+2.4
CRAFT with DeepSeek-V4.1-Flash (open weights)74.4 / 53.9 / 63.2+2.1
  • Every run improves every benchmark. The three GPT-5.6-sol runs have a standard deviation of at most 0.3 AUC per benchmark. On Seed-SAM3, three runs give 78.8±0.1 / 65.9±0.4 / 68.3±0.2 against 77.6 / 63.8 / 66.1 for the seed, and again every run improves every benchmark. We also repeated every ablation variant three times. The revised paper reports these means and standard deviations in Table 2 and lists every run in App. F.
  • Other agent families reach similar gains. Claude Opus 4.8 matches GPT-5.6-sol, and even DeepSeek-V4.1-Flash, a fast open-weight model, reaches +2.1 (App. F of the revised paper).
  • The runs discover the same mechanisms. Every run places the largest scoped gain in template memory, verification before a state change appears in 5/5 runs, and per-object calibration in 4/5.
  • Model deprecation. Agents are used only during design. The deployed result is ordinary tracker code that runs without any agent, and the open-weight run shows that the procedure does not depend on a proprietary model either. As coding agents become stronger, the same procedure can use them without change, which we expect to leave further room in this direction.

Tracker architectures. The two trackers in the paper cover the two model families most used in current tracking. Seed-B256 is a one-stream Transformer tracker, and Seed-SAM3 adds SAM3, a segmentation foundation model, to the model set. In particular, Seed-SAM3 starts from a very strong point in the tracking field. Its LaSOT AUC of 77.6 is already above every SAM-based tracker in Table 1, and CRAFT still improves it on every benchmark, to 78.8 / 65.9 / 68.3. We do not claim that CRAFT generalizes to every tracker, but these results suggest that it holds, to some extent, across trackers of different types and strengths.

AuthorsOpenReview ↗

Response to Reviewer k3Wo (3/4): GOT-10k protocol, more benchmarks, and template details

Q4(a). GOT-10k protocol. The GOT-10k results, which we report in the appendix, are not a restricted-protocol comparison. GOT-10k defines a one-shot protocol in which trackers are trained only on its training split. Some recent trackers instead report GOT-10k after training on multiple datasets, so Table 5 contains entries of both kinds. Our seed belongs to the second group, and CRAFT additionally uses TNL2K-train for controller development. In App. C of the revised paper, we now state this difference explicitly and present our GOT-10k results as an unrestricted-data evaluation, a reference rather than a protocol-matched ranking. The seed-to-CRAFT gain is unaffected, since both rows use identical weights. Our main results remain those on LaSOT, LaSOT$_\mathrm{ext}$ and TNL2K.

Q4(b). More benchmarks. We evaluated the same programs, unchanged, on four more benchmarks (EAO on VOT2022-STB, AUC on the others).

TrackerVOT2022-STB (EAO)AVisTNfS30OTB-100
Seed-B2560.57459.070.070.5
CRAFT-B2560.61561.170.072.7
Seed-SAM30.77465.472.774.2
CRAFT-SAM30.78867.872.574.5

CRAFT improves VOT2022-STB, AVisT and OTB-100 on both trackers and leaves NfS30 nearly unchanged (0.0 and −0.2 AUC). The gains are largest on AVisT (+2.1 and +2.4), whose adverse visibility conditions cause frequent occlusion and appearance change. NfS30 and OTB-100 are small, with 100 sequences each, and largely saturated, so recent trackers differ little on them, and we read them as a consistency check. We have added these results to App. C of the revised paper.

Q5. Template cropping and the number of template slots. Every template is a square crop of side 2.0·√(wh) centered on the box, so the target covers about a quarter of the crop and the rest is surrounding context. Out-of-image pixels are zero-padded, and the crop is resized bilinearly to 128×128 (now stated in App. B). This cropping is the standard choice of recent trackers such as TransT (CVPR 2021), OSTrack (ECCV 2022), ODTrack (AAAI 2024) and SUTrack (AAAI 2025), and CRAFT does not change it.

The tracker uses four template slots. We keep this number fixed for two reasons.

  • It belongs to the trained model, not to the controller. The frozen model was trained with four templates, and changing their number changes its inference, which constraint (i) of the paper excludes. Four is also a common choice. ODTrack, for example, feeds four templates at inference, the first frame and three frames sampled from its memory.
  • The comparison is matched. The seed and CRAFT use the same four slots, so every reported gain comes from the controller rather than from template capacity. What the controller decides is which observations enter these slots and when.
Reviewer_5cMZOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

CRAFT formulates the manually designed controller in visual tracking as an offline program-search problem around a frozen vision model. Without modifying model weights or internal inference, a coding agent optimizes four control components: search-region selection, response decoding, template-memory management, and failure recovery. Experiments on a single-stream Transformer tracker and its SAM3-enhanced variant show that the automatically designed controller improves tracking performance across multiple benchmarks with only modest additional inference overhead, highlighting the potential of optimizing temporal decision-making and memory management outside the vision model itself.

Critical strengths
  1. CRAFT treats the traditionally hand-crafted control logic in visual trackers as an optimization target and improves performance while keeping the vision model frozen. This reveals the potential of optimizing components such as search strategy, template management, and failure recovery, offering a complementary direction to model-centric training for improving tracking systems.
  2. Using only the TNL2K training set for search feedback, the resulting controller achieves consistent improvements on the primary metrics across multiple benchmarks with identical code and hyperparameters. Deployment requires no language model, and the base tracker slows only from 50 to 47 FPS, indicating a favorable balance between performance and efficiency.
Critical weaknesses
  1. The coding agent introduces stochasticity into the search process, yet the paper does not report the mean, variance, or confidence intervals over multiple independent runs. Consequently, particularly for the relatively small gains observed in some ablations, it remains unclear whether the improvements reflect a robust advantage of the proposed pipeline or randomness from a single search run.
  2. Each full search requires 90 agent-driven revisions and 91 development-set evaluations, with each evaluation reportedly taking about 25 minutes on four GPUs. Although the paper includes a comparison under the same search budget, it lacks comparisons with stronger hand-crafted controllers or systematic hyperparameter optimization, making it difficult to establish whether the substantial search cost provides clear advantages over simpler alternatives.
  3. The paper lacks intuitive visualizations of tracking results and underlying mechanisms. The effectiveness is mainly demonstrated through quantitative metrics and search curves, without qualitative comparisons between the initial controller and CRAFT on representative video sequences, or visualizations of template updates, search-region adaptation, and failure-recovery processes.
Questions for authors
  1. Can the authors provide qualitative visualizations and failure cases on representative video sequences? In particular, comparisons between the initial controller and CRAFT under occlusion, fast motion, and distractor interference, together with template updates, confidence trajectories, and recovery trigger points, would help clarify which mechanisms drive the performance gains and under what conditions the method still fails.
  2. Are the gains from CRAFT’s search process reliably reproducible? Since the coding agent introduces stochasticity, were multiple independent full searches conducted? Reporting the mean and standard deviation of the final performance and key ablation results would help assess whether the advantage of the complete pipeline over alternative search strategies is statistically stable.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 5cMZ (1/3): reproducibility across independent runs

Thank you for the review and for recognizing controller optimization as a complementary direction to model-centric training. We address each of your concerns below.

W1 and Q2. Are the gains reproducible across independent runs? Tracker evaluation is deterministic, and the coding agent is the only stochastic element, so we repeated the whole design procedure. Every result below comes from three independent runs from scratch, with fresh agent sessions and unchanged settings (run 1 is the paper's). AUC, mean ± std over 3 runs. H is the mean over the three held-out benchmarks, also listed per run.

Final performance. We ran full CRAFT three times on both trackers.

TrackerProgramDevLaSOTLaSOT$_\mathrm{ext}$TNL2KHH per run
Seed-B256Seed controller66.172.652.060.661.7–
CRAFT68.8±0.174.7±0.154.2±0.363.7±0.164.2±0.164.1 / 64.3 / 64.1
Seed-SAM3Seed controller70.277.663.866.169.2–
CRAFT72.5±0.278.8±0.165.9±0.468.3±0.271.0±0.271.0 / 71.2 / 70.8
  • The gain holds in every run. Every run improves every benchmark on both trackers. On average, CRAFT raises held-out AUC by 2.1 to 3.1 points on Seed-B256 and by 1.2 to 2.2 on Seed-SAM3, against a run-to-run std of at most 0.4.
  • The confidence intervals exclude the seed by a wide margin. With three runs, the 95% t-interval of H is [63.9, 64.4] on Seed-B256 and [70.5, 71.5] on Seed-SAM3, against 61.7 and 69.2 for the seeds.

Ablations. We also ran every variant of Table 2 three times on Seed-B256. Rows 4 and 5 reuse the scoped campaigns and the composite, respectively, of the matching full run, so their comparison with the full procedure is paired. Δ averages the change over all four splits as in Table 2, and ΔH over the three held-out benchmarks only.

VariantDevLaSOTLaSOT$_\mathrm{ext}$TNL2KHH per runΔΔH
Full CRAFT68.8±0.174.7±0.154.2±0.363.7±0.164.2±0.164.1 / 64.3 / 64.1––
Single unscoped campaign67.5±0.173.5±0.152.7±0.262.1±0.262.8±0.162.9 / 62.7 / 62.7−1.38−1.40
Sequential scoped campaigns68.6±0.174.5±0.153.8±0.263.0±0.163.8±0.163.8 / 63.7 / 63.8−0.35−0.41
w/o joint refinement68.2±0.174.2±0.152.0±0.263.7±0.263.3±0.163.3 / 63.4 / 63.3−0.79−0.86
w/o scoped archives68.2±0.174.3±0.253.8±0.463.5±0.263.9±0.163.8 / 63.8 / 64.0−0.39−0.31
  • The full procedure is ahead in every run. Every full run exceeds every run of every variant in H. The weakest full run reaches 64.1, and the strongest variant run reaches 64.0.
  • Even the small margins exceed the run-to-run variation. The run-to-run std of H is at most 0.12 for every variant. The largest margins, 1.4 for a single unscoped campaign (row 2) and 0.9 without joint refinement (row 4), are more than six times this std. The smallest, 0.3–0.4 for sequential campaigns (row 3) and refinement without scoped archives (row 5), are more than twice this std under either Δ or ΔH, and they hold in every run, including every pair for row 5. These margins therefore reflect the proposed organization rather than the randomness of a single run.

We have added these means and standard deviations to Table 2 of the revised paper, with every run listed in the new App. F. Thank you for your suggestion.

AuthorsOpenReview ↗

Response to Reviewer 5cMZ (2/3): CRAFT vs. simpler alternatives

W2. Does the search cost pay off against simpler alternatives? We compare CRAFT with systematic hyperparameter optimization, using the Tree-structured Parzen Estimator (TPE), and with a hand-crafted controller built from published mechanisms on Seed-B256. Each alternative receives the same compute time as one CRAFT run. Since it spends no time on agent proposals, this gives it 106 development evaluations, against 91 for CRAFT.

ControllerTuned constantsDevLaSOTLaSOT$_\mathrm{ext}$TNL2KΔH
Seed controller–66.172.652.060.6–
Seed + hyperparameter optimization668.373.052.761.1+0.5
Hand-crafted controller–64.471.345.058.2−3.6
Hand-crafted controller + hyperparameter optimization1467.573.151.560.4−0.1
CRAFT, mean of 3 runs–68.874.754.263.7+2.4
  • Tuning fits the development split but transfers little. Hyperparameter optimization over the six constants of the seed (search factor, template crop factor, cosine window, update threshold, update interval and memory capacity) raises Dev AUC by 2.2 but held-out mean AUC by only 0.5. CRAFT raises them by 2.7 and 2.4. Its gain comes from program structure that no range of constants contains, for example trial-running a new template set before activating it, and verifying a recovery candidate by re-finding the target in the last reliable frame.
  • Adding published mechanisms by hand does not help this seed. This hand-crafted controller adds to the seed a Kalman motion prior for the crop (as in SAMURAI), quality-aware template selection, and confidence-triggered local-to-global re-detection (as in LTMU and GlobalTrack), with 14 tunable constants. Without tuning, it falls 3.6 below the seed in held-out mean AUC. Tuning raises its Dev AUC 1.4 above the seed, but its held-out mean AUC only returns to the level of the seed. The seed controller of recent one-stream trackers is already well refined, and each added mechanism interacts with the others and with the confidence of the model. We do not claim that an expert could not combine these mechanisms into a better controller. The result shows, however, that published mechanisms cannot simply be taken from other trackers and combined into a better one. Making them work together on a given model takes expert time that is hard to measure. CRAFT aims to save such expert time in controller design. We have added this comparison to the new App. I of the revised paper. Thank you again for your suggestions.
AuthorsOpenReview ↗

Response to Reviewer 5cMZ (3/3): qualitative visualizations and failure cases

W3 and Q1. Qualitative visualizations and failure cases. Thank you for these suggestions. Following them, we have added qualitative visualizations and failure cases to App. H of the revised paper, and a video to the supplementary material.

  • Comparison with the initial controller (Fig. 4). On five held-out sequences, with the predictions behind Table 1, the seed switches to another object and CRAFT keeps the target, under occlusion (bottle-12), out-of-view (pool-15), fast motion (pool-7), a similar distractor (raccoon-8), and other pedestrians in thermal infrared (INF_boundaryandfast).
  • Mechanisms (Figs. 5 and 6). For a controlled comparison, each figure compares CRAFT with an ablated rollout that reverts one mechanism of the final CRAFT controller to the corresponding rule of the seed, or removes it if the seed has none, and keeps all other mechanisms unchanged. Any difference in tracking behavior can then be attributed to that mechanism.
    • Template updates. On elephant-16, while both rollouts have lost the calf, they store templates of other regions. The ablated rollout, which selects uniformly as the seed does, activates the newest of them. CRAFT selects the best template from each of three time spans of the memory, keeps templates of the calf and finds it again, while the ablated rollout follows other elephants.
    • Search region. On sheep-9, the camera pans fast. CRAFT shifts its search window ahead along the consistent motion, so the target stays inside. The fixed window of the ablated rollout lags behind and contains another sheep, which it follows.
    • Response decoding. On dancingshoe-5, the network outputs are fixed, and only the response rule differs. The fixed Hann window selects the neighboring shoe, and the adaptive rule of CRAFT selects the target.
    • Confidence and recovery trigger. On pool-15, the figure plots the confidence against the thresholds for entering and leaving a target loss state. When the white target ball leaves the view at frame 262, the confidence falls below the entry threshold. During the loss, CRAFT stops updating its templates and motion estimate, and searches wider regions. When the ball reappears at frame 265, CRAFT finds it again and the confidence exceeds the exit threshold. Without loss handling, the ablated rollout switches to the orange ball.
  • Failure cases (Fig. 4, last two rows). These rows show two representative failures under challenging conditions, a fast shuttlecock only a few pixels wide (badminton-7) and one of hundreds of nearly identical lanterns (lantern-4). CRAFT loses the target in both and does not find it again.

Thank you very much for your constructive comments, which have made the paper stronger. We are glad that you found no substantive soundness concern, and the remaining concerns asked for evidence that we now provide. We would be grateful if you would consider updating your score.

Reviewer_GyW5OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

CRAFT optimizes the program logic surrounding a frozen visual tracker. Coding agents explore four controller operations independently, compose selected revisions, and refine their interactions using development-set feedback. The resulting controllers improve two tracker configurations across several held-out datasets without requiring a language model during deployment.

Critical strengths
  • Keeping visual models fixed makes the controller’s practical contribution easy to assess.
  • Improvements transfer across datasets using one controller and one parameter configuration.
  • The discovered programs remain deployable at nearly the original tracking speed.
  • Archived revisions and stage-wise results provide a useful account of the search process.
Critical weaknesses
  • The procedure comparisons lack independent repetitions. Differences of 0.3–0.4 AUC against sequential search or refinement without scoped archives could reflect campaign variability. Repeat the principal search variants before attributing these margins to the proposed organization.
  • Most gains come from template-memory design. In Table 6, the final B256 controller improves mean held-out AUC by only 0.3 points over the best memory-only revision. An equally budgeted memory-only campaign would test whether the broader orchestration earns its additional cost.
  • The GOT-10k comparison uses different training conditions. Appendix B describes multi-dataset tracker training and TNL2K controller development, unlike standard GOT-10k-only training. Mark these results as unrestricted-data evaluation or provide a comparison under the restricted protocol.
Questions for authors
  • How is preservation of inactive controller operations checked?
  • Does hiding metrics during composition improve the resulting controller?
2 comments
AuthorsOpenReview ↗

Response to Reviewer GyW5 (1/2): repeated search variants and an equal-budget memory-only campaign

Thank you for your precise review and the concrete suggestions. We address each point below.

W1. Independent repetitions of the principal search variants. We ran every Table 2 variant three times from scratch, with fresh agent sessions and the same settings (run 1 is the paper's). Rows 4 and 5 reuse the scoped campaigns and the composite, respectively, of the matching full run, so their comparison with the full procedure is paired. AUC, mean ± std over 3 runs. H is the mean over the three held-out benchmarks, also listed per run. Δ averages the change over all four splits as in Table 2, and ΔH over the three held-out benchmarks only.

VariantDevLaSOTLaSOT$_\mathrm{ext}$TNL2KHH per runΔΔH
Full CRAFT68.8±0.174.7±0.154.2±0.363.7±0.164.2±0.164.1 / 64.3 / 64.1––
Single unscoped campaign67.5±0.173.5±0.152.7±0.262.1±0.262.8±0.162.9 / 62.7 / 62.7−1.38−1.40
Sequential scoped campaigns68.6±0.174.5±0.153.8±0.263.0±0.163.8±0.163.8 / 63.7 / 63.8−0.35−0.41
w/o joint refinement68.2±0.174.2±0.152.0±0.263.7±0.263.3±0.163.3 / 63.4 / 63.3−0.79−0.86
w/o scoped archives68.2±0.174.3±0.253.8±0.463.5±0.263.9±0.163.8 / 63.8 / 64.0−0.39−0.31

Two conclusions follow.

  • The full procedure is ahead in every run. Every full run exceeds every run of every variant in H. The weakest full run reaches 64.1, and the strongest variant run reaches 64.0.
  • The margins exceed the run-to-run variation. The run-to-run std of H is at most 0.12 for every variant. Replacing the decomposition into design spaces by a single unscoped campaign (row 2) and removing joint refinement (row 4) give the largest margins, 1.4 and 0.9, more than six times this std. The 0.3–0.4 margins you pointed to, for sequential campaigns (row 3) and refinement without scoped archives (row 5), are more than twice this std under either Δ or ΔH, and they hold in every run, including every pair for row 5. These margins therefore reflect the proposed organization rather than campaign variability.

We have added these means and standard deviations to Table 2 of the revised paper, with every run listed in the new App. F. Thank you for your suggestion.

W2. An equally budgeted memory-only campaign. We ran three 90-round campaigns from the seed controller on both trackers, each confined to template memory. The table compares them with the 15-round memory revision of Table 6 (Table 7 in the revised paper) and with full CRAFT, which uses the same 90 rounds. AUC, mean ± std over 3 runs. H is the held-out mean.

TrackerProgramDevLaSOTLaSOT$_\mathrm{ext}$TNL2KHH per run
Seed-B256Memory, 15 rounds (Table 7)68.874.354.462.863.8–
Memory only, 90 rounds68.7±0.174.3±0.154.4±0.262.7±0.263.8±0.163.8 / 63.9 / 63.8
Full CRAFT, 90 rounds68.8±0.174.7±0.154.2±0.363.7±0.164.2±0.164.1 / 64.3 / 64.1
Seed-SAM3Memory, 15 rounds (Table 7)71.579.265.167.670.6–
Memory only, 90 rounds71.5±0.278.9±0.365.0±0.267.5±0.170.4±0.270.6 / 70.3 / 70.4
Full CRAFT, 90 rounds72.5±0.278.8±0.165.9±0.468.3±0.271.0±0.271.0 / 71.2 / 70.8

Two observations follow.

  1. Memory-only design saturates early, and the broader procedure is ahead in every run. On both trackers, the 90-round memory-only campaigns are no better than the 15-round memory revision on any split, so the extra 75 memory rounds add no gain. CRAFT spends these rounds on the other operations and on joint refinement instead, and every full run exceeds every memory-only run in H on both trackers. The margin is 0.33 H on Seed-B256 and 0.57 H on Seed-SAM3, close to three times the run-to-run std on both trackers. On Seed-B256 it is close to the 0.3 you noted from Table 6, and it now holds in every run.
  2. Where the headroom lies is known only after scoped exploration, and finding it is what CRAFT does. Before the design procedure runs, it is not known which operation of a given tracker can still improve. On Seed-B256 the answer turned out to be mainly template memory. On Seed-SAM3, part of the headroom moves to response decoding (Table 7). Accordingly, the margin over the memory-only campaign is larger on Seed-SAM3 than on Seed-B256 (0.57 vs 0.33 H). A memory-only campaign therefore presupposes an answer that is available only in hindsight, and scoped exploration is how CRAFT obtains it. Designing without that structure, with all operations open from the first round, is the single unscoped campaign (Table 2, row 2), which trails full CRAFT by 1.4 H (W1). We have added these campaigns to App. D of the revised paper, with every run listed in App. F. Thank you for the helpful suggestion.
AuthorsOpenReview ↗

Response to Reviewer GyW5 (2/2): GOT-10k protocol, preservation of inactive operations, and metric-blind composition

W3. GOT-10k protocol. Thank you for this point. Yes, the GOT-10k results in the appendix are an unrestricted-data evaluation. Earlier trackers followed the one-shot protocol of GOT-10k, in which trackers are trained only on its training split. Some recent trackers instead report GOT-10k after training on multiple datasets, so Table 5 contains entries of both kinds. Our seed tracker belongs to the second group, and CRAFT additionally uses TNL2K-train for controller development. In App. C of the revised paper, we now state this difference explicitly and present our GOT-10k results as a reference rather than a protocol-matched ranking. The seed-to-CRAFT gain is unaffected, since both rows use identical weights. Our main results remain those on LaSOT, LaSOT$_\mathrm{ext}$ and TNL2K.

Q1. How is preservation of inactive controller operations checked? In scoped exploration, a revision changes only its active operation, and the three inactive operations keep their tracking behavior. Two mechanisms enforce this during the run, and for this response we also verified it post hoc.

  • Instruction and declaration. Every scoped round declares its active space and receives the definitions of Table 3 with the instruction to preserve the tracking behavior of the inactive operations, and to change shared code only where the active mechanism needs it and without altering the behavior of the inactive operations.
  • Supervisor review. A separate supervising agent reviewed every revision and was instructed to stop and revert any round that changed an inactive operation. It found none.
  • Diff audit. After the runs, two authors independently labeled every diff hunk of all 120 scoped revisions of both trackers by operation. Of 7,935 changed controller lines, 7,871 (99.19%, with 99.39% on Seed-B256 and 98.92% on Seed-SAM3) lie within the declared operation. The other 64 lines are necessary changes to shared code, for example mapping the box back through the actual crop center after a search-region change, and do not change what any inactive operation does.
  • Behavioral replay. We logged the inputs of the inactive operations under each revision and replayed them through the seed implementation, so that both versions see identical inputs. Their decisions matched in all 120 revisions.

We have added this protocol and the audit statistics to App. A of the revised paper. Thank you for your helpful comments.

Q2. Does hiding metrics during composition improve the resulting controller? Not measurably in our runs. When we composed the same four selected revisions with metrics visible, the composing agent still followed its instruction. It kept all four mechanisms and changed only what integration required, and both the composite and the final program after joint refinement matched the blind setting on the development split (visible vs blind: Dev 68.1 vs 68.1 for the composite, 68.7 vs 68.8 for the final program). This reflects the instruction following of current coding agents. Hiding the metrics is a safeguard rather than a source of accuracy. Composition is meant to integrate the selected mechanisms, not to re-rank the scoped campaigns or rewrite their mechanisms (Sec. 3.3). Selection has already happened within each space, and any decision to revise or remove a mechanism belongs to joint refinement, which sees the full evidence of the scoped archives. Withholding the metrics makes this division hold by construction rather than through the agent's compliance, in the same spirit as the scope checks of your Q1. We have added this discussion to App. D of the revised paper. Thank you for your comments.

Thank you very much for your positive evaluation of our work, and for the suggestions that improved the paper. We hope our responses further support your assessment, and we are happy to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback. We are encouraged that two reviewers gained "a substantially deeper understanding of the problem" from the paper (GyW5, 5cMZ), and that the reviewers found that the work "reveals the potential of optimizing components such as search strategy, template management, and failure recovery, offering a complementary direction to model-centric training" (5cMZ), that the idea is "interesting and potentially useful" (k3Wo), and that the improvements "transfer across datasets using one controller and one parameter configuration", with discovered programs that "remain deployable at nearly the original tracking speed" (GyW5).

Core idea. A visual tracker combines a learned visual model with an online controller, the program logic that decides where to search, how to read the model's response, which templates to keep, and when to recover from loss. Visual models improve through training on large datasets, whereas controllers are still designed by hand, one mechanism at a time.

CRAFT makes the controller an optimization target. Coding agents take on the work of a controller designer. They inspect failures in full tracking runs on TNL2K-train, write new decision logic into the controller's source code, and test it, while the visual model stays frozen. A design procedure organizes this work, exploring each controller operation separately, composing the selected mechanisms, and refining their interactions jointly. The result is ordinary tracker code with no language model at inference. On a one-stream Transformer tracker and a much stronger SAM3-augmented tracker, it improves tracking performance at nearly the original speed.

Our contribution therefore lies in showing that, for visual object tracking, the controller around a frozen model holds accuracy that is still largely untapped, and in a design procedure that finds better controllers automatically.

Two shared questions. The reviews ask (i) whether the gain is reliable and comes from the procedure, rather than from the randomness of a single agentic run or from tuning (all reviewers), and (ii) how far it generalizes beyond one agent and the reported benchmarks (k3Wo). Numbers refer to Seed-B256 unless noted, and held-out mean AUC is the mean over LaSOT, LaSOT$_\mathrm{ext}$ and TNL2K (61.7 for the seed).

(i) The gain is reliable, and it comes from program structure.

  1. Reproducible across independent runs (all). Three independent full CRAFT runs give a held-out mean AUC of 64.2 ± 0.1 on Seed-B256 and 71.0 ± 0.2 on Seed-SAM3, and each run outperforms the seed on every benchmark. The 95% confidence intervals, [63.9, 64.4] and [70.5, 71.5], lie well above the seeds. The Table 2 variants, also repeated three times, stay below the full procedure in all runs.
  2. Not reachable by hyperparameter tuning (5cMZ, k3Wo). With the same compute time, tuning the hyperparameters of the seed with TPE (Tree-structured Parzen Estimator) raises Dev AUC by 2.2 but held-out mean AUC by only 0.5, against 2.7 and 2.4 for CRAFT. A controller that adds published mechanisms by hand reaches only the level of the seed after tuning. Of the 180 revisions in the two runs reported in the paper, 175 (97.2%) change the decision logic of the controller rather than only its constants.
  3. Not memory design alone (GyW5). Most of the gain on Seed-B256 comes from template memory, but this is known only after scoped exploration, and on Seed-SAM3 part of the headroom lies in response decoding instead. Even with this hindsight, equal-budget 90-round campaigns confined to template memory do no better than the 15-round memory revision of CRAFT and trail full CRAFT in every run, by 0.33 held-out mean AUC on Seed-B256 and 0.57 on Seed-SAM3.

(ii) The gain generalizes.

  1. Across agents (k3Wo). Full runs with Claude Opus 4.8 and the open-weight DeepSeek-V4.1-Flash raise held-out mean AUC by 2.4 and 2.1, close to 2.4 for GPT-5.6-sol. The gain therefore does not depend on a single coding agent.
  2. Across more benchmarks (k3Wo). Beyond the five benchmarks of the paper, the same programs, unchanged, also improve VOT2022-STB, AVisT and OTB-100 on both trackers, most on AVisT (+2.1 and +2.4 AUC).

Revisions. We have uploaded a revised paper, with every change marked in blue, and a video in the supplementary material. Table 2 now reports mean and std over three runs. New appendices report the repeated runs and other agents (App. F), the constants of both controllers (App. G), the qualitative analysis (App. H), and the comparison with tuning (App. I). We have also added the preservation audit (App. A), template cropping (App. B), the GOT-10k protocol and more benchmarks (App. C), and the memory-only campaigns (App. D), and Related Work now covers CPDTrack, GOT-Edit, LightTrack and CHASE.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.