acceptodds
Withdrawn from ICLR 2027

VisOrch: Benchmarking Agent-Designed Vision Pipelines

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Composing off-the-shelf vision models into task-specific pipelines avoids training from scratch, but manually designing them is costly. While typical vision agents invoke an LLM or VLM at every inference step, we study a deployment-efficient alternative: an agent designs an executable pipeline once during adaptation, which is then frozen and evaluated on unseen inputs with no such model calls at test time. We introduce VisOrch, a benchmark evaluating whether agents can automatically synthesize reusable vision pipelines from few examples. To ensure scores reflect genuine orchestration rather than a single dominant tool, VisOrch admits a task only when its strongest off-the-shelf component leaves substantial headroom. This yields nine tasks across four task families, totalling 540 adaptation, 480 validation, and 2,415 blind test examples; each task provides a fixed budget of 60 labelled images used to tune the tool baselines. On eight of the nine tasks, an agent-designed pipeline outperforms the strongest single tool on both metrics, whereas directly prompted frontier VLMs fall below it on every task they attempt. However, no single designer is universally reliable: iterative program search is ahead on both metrics at on four tasks, a general coding agent leads on three, both improve on DECIMER, and neither improves on both metrics for PlantSeg. Finally, building VisOrch exposed key evaluation pitfalls, including shortcut exploitation and degenerate metrics, which the benchmark protocol now explicitly guards against.

Primary area:
datasets and benchmarks
Keywords:
Vision Agents, LLM Agents, Vision Tool Orchestration, Program Synthesis, Benchmarks, Evaluation Methodology, Vision Foundation Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y6krOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VisOrch evaluates agents that use a small labelled adaptation set to build reusable vision programs, which are subsequently frozen for deployment without LLM or VLM calls. It selects nine tasks across four families using a normalized single-tool saturation screen, and evaluates direct prediction, one-shot synthesis, a coding agent, and iterative program search. The benchmark includes task-dependent split safeguards, two ranking metrics per task, and validation-based program selection. Results show that agent-designed programs can exceed the strongest measured single-tool baseline on eight tasks, while the relative performance of the coding agent and iterative search varies substantially across tasks.

Critical strengths
  1. The design-once, deploy-many setting is practically relevant and has a concrete evaluation contract. Task-specific splits, including unseen LVIS categories and separate drawing batches for DECIMER, make it harder for adaptation gains to come from memorizing closely related examples. Measuring the delivered executable on held-out inputs is a useful contribution for vision-agent evaluation.

  2. The paper documents failure cases that would otherwise make agent results misleading. The NDD20 background controls, metric trade-offs on segmentation, and separation of harness failures from program failures are informative. Appendix F also tests SIFT alone on Recod.ai/LUC: its 26.2 DLI-B and 23.1 Dice are below both TruFor and the combined program's 56.4 and 40.3, supporting a genuine complementary-tool benefit in this case.

Critical weaknesses
  1. The saturation screen does not establish that the admitted tasks require multi-tool composition. Appendix F and Table F.1 explicitly show that the delivered AI-TOD, LVIS, and DroneVehicle programs repeatedly invoke one detector with different settings and merge the outputs. LVIS and DroneVehicle are also two of the four tasks where iterative search significantly improves both metrics. Thus, single-tool headroom is useful for task selection, but the central claim that the screen makes scores reflect genuine orchestration is stronger than the evidence. Table G.1 already includes threshold-tuned and tiling/refinement arms. A comparison with comparably flexible single-model parameter search is still needed to interpret how much of the improvement actually depends on composing distinct models.

  2. Several descriptions of the evaluation protocol do not match the reported execution. Algorithm 1 selects from the top five candidates per run plus the seed, whereas Appendix F describes validating all 117 AI-TOD candidates and Appendix J describes re-ranking all counting candidates after changing the metric pair. Section 5 also describes one fixed one-shot program, while Appendix G says the one-shot writers generate a program per image without seeing it. These are consequential differences in validation access and deployment behavior, not merely implementation details. The disclosures are useful, but a single canonical protocol is needed before the cross-paradigm results can be interpreted reproducibly. These discrepancies do not by themselves establish test leakage.

Questions for authors
  1. Clarification: Which reported gains require multiple distinct learned models, and which can be explained by repeated use of one model? Can the claims about the admission screen be aligned with that distinction using the existing program traces?

  2. Clarification: What candidate set was actually validated for each task, when was the ranking rule finalized relative to test evaluation, and was the One-shot column generated once per task or once per test image? Which Table 2 results follow the stated frozen-program protocol exactly?

Reviewer_QQXwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the construction of executable pipeline that composes off-the-self vision models for solving specific tasks. Since invoking an LLM or a VLM at every inference step is costly, the paper considers the setting where a pipeline is built by an agent once in an adaptation phase during which LLMs/VLMs can be invoked before being frozen for inference without LLM/VLM call.

To isolate the orchestration performance of the agent-designed pipeline, the paper considers only tasks that are not close to be solved by the strongest tool, leaving some headroom for orchestration. It considers 9 tasks belonging to 4 different task families. It compares several such pipelines to the strongest tool and a standalone VLM and provides several interesting observations.

Critical strengths
  • The paper considers an interesting scenario with practical impact: Invoking a VLM or an LLM at every inference step is indeed very costly and an LLM-free pipeline during inference would dramatically reduce inference cost.

  • The paper’s overall experimental design is relatively well thought. The designed criterion for task selection makes sense and is important in isolating the gain of orchestration from the tool performance.

  • The paper experimental protocol reveals interesting evaluation issues that would benefit future effort in this direction.

  • The paper’s writing is clear and easy to understand.

Critical weaknesses

Though well thought, there are some limitations in the paper’s experimental protocol some of which are also discussed in the limitation section:

  • A single LLM is used throughout the experiments. The observations therefore are not established for other models.
  • The validation splits are too small which leads to unstable observation. There are also two few searches in VTOS. These factors should have been assessed even before running the evaluation,
  • The scenario is considered as a low cost alternative to normal visual agent but no comparison to them provided. It would be interesting to see how much performance is covered by the low-cost pipeline.
Questions for authors

The paper would be strengthened if the weaknesses are addressed.

Reviewer_xHmxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studied the use of agent-designed executable pipelines for vision tasks at test time. The authors introduced VisOrch, a benchmark evaluating whether agents can automatically synthesize reusable vision pipelines from few examples. To ensure scores reflect genuine orchestration rather than a single dominant tool, VisOrch admits a task only when its strongest off-the-shelf component leaves substantial headroom. Under this requirement, the VisOrch benchmark yields nine tasks across four task families. Based on the curated VisOrch benchmark, the authors investigated and compared agent-designed pipelines with directly prompted frontier VLMs and evaluated the reliability of agent-designed pipelines.

Critical strengths
  1. Developing benchmarks for agent-designed pipeline for vision tasks is a timely topic.

  2. This paper conducted in-depth and comprehensive tests on their developed VisOrch dataset.

Critical weaknesses
  1. The VisOrch dataset only covers nine tasks in four task families, which may still be limited in representing visual problems for which agent-designed pipelines are useful.

  2. The underlying LLM used for performance evaluation is too limited (based only one 1 LLM), and so the results observed in this paper may not be sufficiently conclusive. The authors need to expand the diversity of the underlying LLMs.

  3. The examples for each task evaluation appear to be limited (40–60) examples. A larger-scale dataset for more comprehensive experiments are necessary.

Questions for authors

How long does it take to expand the benchmark to facilitate larger-scale and more in-depth experiments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.