acceptodds
Under review as a conference paper at ICLR 2027

IRSTD-Agent: Agentic Infrared Small Target Detection via Zoom-Guided Interaction Learning

Abstract

Infrared small-target detection plays an important role in maritime monitoring and aerial surveillance. Although multimodal large language models (MLLMs) offer promising capabilities for visual understanding, existing MLLM-based approaches struggle to precisely localize infrared small targets. In this paper, we propose IRSTD-Agent. an agentic framework for infrared small target detection through dynamic visual search. The framework enables an MLLM to adaptively determine where and at what scale to inspect an image and progressively gather fine-grained visual evidence for precise target localization. Five complementary visual tools (PROPOSAL, ZOOM, DETECT, DROP and REFINE) support object candidate discovery, adaptive observation, target localization, hypothesis rejection, and target extent refinement, together enabling a coordinated search process over original-resolution images. To teach the MLLMs to conduct this search, we introduce Zoom-guided Interaction Learning, which uses annotation-derived interaction trajectories to supervise tool selection and the corresponding arguments. Through extensive experiments on WideIRSTD-Full and IRSTD-1k datasets, we demonstrate that IRSTD-Agent outperforms the evaluated vision-language models and enhances the precise localization capabilities of MLLMs in IRSTD tasks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
infrared small target detection, zoom-guided interaction learning, IRSTD-agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ai3MOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the IRSTD‑Agent agent framework, which formulates infrared small‑target detection as a dynamic visual‑search task. It designs five visual tools including PROPOSAL, ZOOM, DETECT, DROP and REFINE to accomplish candidate generation, adaptive zoom‑in, target localization, disturbance elimination and target‑boundary refinement. It also proposes Zoom‑guided Interaction Learning, which constructs positive and negative interaction trajectories based on ground‑truth annotations to supervise large‑language models in learning tool invocation and parameter selection. Extensive experiments are conducted on the WideIRSTD‑Full and IRSTD‑1k datasets, which effectively improve the localization accuracy of multimodal large‑language models for infrared small‑target detection. Nevertheless, sequential tool invocations bring additional inference overhead, and detecting extremely tiny targets with ambiguous features still poses challenges.

Critical strengths

1.It innovates in task paradigm by transforming infrared small‑target detection into agent‑driven dynamic visual search. It locally zooms in to search for targets directly on original‑resolution images without image compression. 2.It proposes a set of dedicated visual tools with well‑defined complementary responsibilities. Reasonable division‑of‑labor among these tools yields noticeable performance gains when combined. 3.It presents Zoom‑guided Interaction Learning (ZIL) to optimize the agent’s tool‑calling strategy, adopting the trick of joint positive‑and‑negative trajectory supervision.

Critical weaknesses

1.Serial tool reasoning incurs heavy computational overhead. Experiments require multiple rounds of MLLM inference, resulting in high latency. 2.It heavily relies on high‑quality manual annotations for constructing interaction trajectories. Zoom‑guided Interaction Learning requires manually building complete positive and negative interaction trajectories from ground‑truth annotations for SFT fine‑tuning. Valid training trajectories cannot be produced without high‑quality dataset annotations; poor‑quality annotations will directly corrupt the learning of tool‑calling policy and lead to severe performance degradation. 3.The PROPOSAL candidate‑generation tool has inherent flaws. Generating candidates via spatial filtering and FFT‑based frequency‑domain saliency, this hand‑crafted feature is prone to being triggered by heavy clutter and high‑light noise points, producing numerous false candidates. This increases subsequent ZOOM iterations and further raises inference overhead. Moreover, the real target will be missed entirely if it is not selected as a candidate by PROPOSAL.

Questions for authors

This paper compares extensively against general‑purpose multimodal large‑language models and generic active visual‑search baselines, yet it lacks comparisons against domain‑specialized infrared small‑target detection algorithms with comparable‑scale backbones. Does the proposed method outperform these dedicated approaches across all metrics?

1 comment

The author have well responsed to all my concers. My concerns have been addressed

Reviewer_Y6R2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

IRSTD-Agent searches for small infrared targets by zooming into candidate regions and rejecting background clutter. It learns these actions from search sequences built using annotations, while a separate step refines the detected target boundaries. Results on multiple benchmarks show improvements over the evaluated VLMs.

Critical strengths
  1. The learned search approach is interesting, and training from existing target masks avoids additional manual annotations.
  2. The method improves over the fine-tuned RSCoVLM-det by 16.7 F1 points on WideIRSTD and 2.84 points on IRSTD-1k.
Critical weaknesses
  1. Most baselines are not fine-tuned on infrared data, so the gains cannot be attributed to adaptive search alone. A comparison with the same backbone trained on the same data using predefined crops is missing. Disabling zoom in the trained agent does not provide that comparison.
  2. LGNet, MIRSAM and SAIST use vision-language information for infrared detection or segmentation, they are discussed but not evaluated.
  3. All five specialized detectors outperform the agent on full WideIRSTD (Table 4). The agent performs better on the 44-image large-resolution subset. However, without measured latency and memory usage, the practical trade-off of using a 27B model remains unclear.
Questions for authors
  1. What are the average model calls, latency, and peak memory per image? Does the large-image advantage remain against a detector trained for native-resolution patch inference?
  2. What are the exact trajectory-generation rules, sampling proportions, and tool parameters? Please clarify which experiments use a fixed proposal queue and whether the code will be released.
Reviewer_JTCrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets the problem that full-image observation cannot preserve fine details of infrared small targets. It proposes a tool-interaction framework combining candidate generation, adaptive local observation, target localization, and background rejection. The design logic is reasonable: cropping local observations from the original-resolution image and using negative-branch supervision to learn background rejection are practically meaningful, and the current results support the effectiveness of the complete system among the evaluated vision-language models.

Critical strengths
  1. A concrete focus on limited visual evidence for tiny targets in full-image views. Cropping from the original-resolution image and deciding whether to keep inspecting or reject a candidate directly addresses target visibility. Positive/negative branches plus multi-target outputs handle both background interference and localization. This is the main value of the paper as an exploration of MLLM visual perception.
  2. Clear gains of the complete system over the evaluated vision-language baselines. F1 is 46.87% on WideIRSTD-Full vs. 30.17% for the strongest listed baseline, and 77.97% on IRSTD-1k vs. 75.13%. This supports the effectiveness of the complete system in the evaluated settings, though it does not by itself establish the contribution of adaptive search.
Critical weaknesses
  1. The distinction from existing visual-search/tool-learning work is unclear. Active cropping, progressive zooming, and interaction-trajectory supervision are close to ZoomEye, Pixel Reasoner, and ZoomEarth. The current description does not show where the substantive novelty lies—whether in infrared-specific tools and supervision or in a new adaptive decision mechanism—and there is little targeted validation of offset recovery, negative-branch rejection, or multi-target supervision.

  2. The independent contribution of adaptive search is not isolated. Domain SFT, local high-resolution input, sequential tool calls, and post-processing are introduced together, while most baselines do not receive the same infrared supervision, so the gains cannot be attributed directly to adaptive search. The ablations show the issue: removing negative-branch supervision drops F1 to 12.38%, and adding REFINE alone raises it from 66.45% to 77.97%. A comparison of adaptive search vs. fixed multiscale cropping under the same backbone, supervision, post-processing, and compute budget is missing.

  3. No comparison with specialized infrared methods, so practical value is unclear. Table 3 includes neither specialized IRSTD models nor adaptation methods such as IRSAM and SAIST; the method uses a 27B model, multiple interaction rounds, and dataset-specific fine-tuning, without comparing accuracy or inference cost. Also, the IoU in Table 3 is a matched-box IoU rather than full-image pixel-level IoU, and the main text reports no pixel-level IoU/nIoU or Fa—the missing Fa in particular makes DROP's background-rejection ability hard to assess.

  4. The adaptive threshold in REFINE is not reproducible. Appendix A.3 only says it is estimated from local residual statistics, without the formula, statistical region, or parameter-selection rule, so the module cannot be uniquely determined from the paper.

  5. Adaptive vs. fixed local observation is not validated clearly. The fixed-scale ablation in Table 1 is only on 512×512 IRSTD-1k and does not show the effect on large images; the difference between the two datasets also mixes scene, target distribution, and training settings. The sliding-window experiment in the appendix uses different models and training procedures on the two sides, so the advantage cannot be attributed to adaptive observation.

Questions for authors
  1. Under the same splits and comparable protocols, how does the method compare with representative specialized IRSTD models and SAM/vision-language adaptation methods in detection performance and actual inference cost? Is there evidence of advantage in accuracy, false alarms, or generalization?
  2. Under the same backbone, comparable positive/negative supervision, identical post-processing, and matched compute, how much does adaptive search improve over fixed multiscale cropping or independent candidate classification/localization?
  3. What is the substantive difference from ZoomEye, Pixel Reasoner, and ZoomEarth? Is the main innovation infrared-specific tools/supervision or a new adaptive decision mechanism? Can offset recovery, negative-branch rejection, and multi-target supervision be validated specifically?
  4. What is the exact formula, statistical region, and parameter-selection rule for the adaptive threshold in REFINE? Can the module be specified precisely enough to reproduce?
  5. Can the gains from preserving local resolution be separated from those of adaptively choosing observation position and scale, especially on large images? Can the appendix sliding-window comparison use identical models and training procedures?
  6. Does the three-pixel matching criterion use the DETECT center or the post-REFINE box center? Do REFINE and NMS change the number of predictions, and how are boxes built and NMS applied without REFINE? How does this explain the simultaneous precision/recall/F1 gains in Table 1?
  7. Are full-image pixel-level IoU/nIoU and Fa results available, including Appendix B.2? Does recall match the usual Pd in matching rule and aggregation?
  8. How are candidate branches scheduled, how is the next candidate chosen after DETECT/DROP, and when does image-level search stop? Which decisions are made by the model vs. the executor? What are the actual model-call counts and per-image latency under the default setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.