- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
VisOrch evaluates agents that use a small labelled adaptation set to build reusable vision programs, which are subsequently frozen for deployment without LLM or VLM calls. It selects nine tasks across four families using a normalized single-tool saturation screen, and evaluates direct prediction, one-shot synthesis, a coding agent, and iterative program search. The benchmark includes task-dependent split safeguards, two ranking metrics per task, and validation-based program selection. Results show that agent-designed programs can exceed the strongest measured single-tool baseline on eight tasks, while the relative performance of the coding agent and iterative search varies substantially across tasks.
Critical strengths
-
The design-once, deploy-many setting is practically relevant and has a concrete evaluation contract. Task-specific splits, including unseen LVIS categories and separate drawing batches for DECIMER, make it harder for adaptation gains to come from memorizing closely related examples. Measuring the delivered executable on held-out inputs is a useful contribution for vision-agent evaluation.
-
The paper documents failure cases that would otherwise make agent results misleading. The NDD20 background controls, metric trade-offs on segmentation, and separation of harness failures from program failures are informative. Appendix F also tests SIFT alone on Recod.ai/LUC: its 26.2 DLI-B and 23.1 Dice are below both TruFor and the combined program's 56.4 and 40.3, supporting a genuine complementary-tool benefit in this case.
Critical weaknesses
-
The saturation screen does not establish that the admitted tasks require multi-tool composition. Appendix F and Table F.1 explicitly show that the delivered AI-TOD, LVIS, and DroneVehicle programs repeatedly invoke one detector with different settings and merge the outputs. LVIS and DroneVehicle are also two of the four tasks where iterative search significantly improves both metrics. Thus, single-tool headroom is useful for task selection, but the central claim that the screen makes scores reflect genuine orchestration is stronger than the evidence. Table G.1 already includes threshold-tuned and tiling/refinement arms. A comparison with comparably flexible single-model parameter search is still needed to interpret how much of the improvement actually depends on composing distinct models.
-
Several descriptions of the evaluation protocol do not match the reported execution. Algorithm 1 selects from the top five candidates per run plus the seed, whereas Appendix F describes validating all 117 AI-TOD candidates and Appendix J describes re-ranking all counting candidates after changing the metric pair. Section 5 also describes one fixed one-shot program, while Appendix G says the one-shot writers generate a program per image without seeing it. These are consequential differences in validation access and deployment behavior, not merely implementation details. The disclosures are useful, but a single canonical protocol is needed before the cross-paradigm results can be interpreted reproducibly. These discrepancies do not by themselves establish test leakage.
Questions for authors
-
Clarification: Which reported gains require multiple distinct learned models, and which can be explained by repeated use of one model? Can the claims about the admission screen be aligned with that distinction using the existing program traces?
-
Clarification: What candidate set was actually validated for each task, when was the ranking rule finalized relative to test evaluation, and was the One-shot column generated once per task or once per test image? Which Table 2 results follow the stated frozen-program protocol exactly?