acceptodds
Under review as a conference paper at ICLR 2027

Locate in Any Scene: Agentic Object Localization with Multimodal Large Language Models under Diverse Visual Degradations

Abstract

Object localization in real-world scenarios remains challenging due to diverse image degradations and heterogeneous object distributions, which substantially limit the generalization of existing grounding models. Conventional approaches, including scene-specific representation learning and end-to-end pipeline design, are inherently constrained by predefined conditions and therefore lack the flexibility required for evolving environments. In this paper, we propose LocAS, an agentic framework that formulates object localization as an adaptive decision-making process. Rather than relying on static pipelines, LocAS employs a Multimodal Large Language Model (MLLM) as a central agent to dynamically compose localization workflows by selecting from a toolbox of restoration modules and specialized detectors. Specifically, LocAS consists of two key components: Self-Adaptive Image Restoration, which determines whether and how an input image should be enhanced for downstream detection, and Multi-Expertise Localization, which coordinates multiple domain-specialized detectors and consolidates their predictions through instance-level reasoning. To further improve decision quality under fine-grained conditions, we introduce Experience-Aware Calibration and extend LocAS to LocAS-X. LocAS-X accumulates node-level decision experience from a small set of annotated samples and incorporates it into agent memory to support more informed reasoning during inference. This mechanism enables the system to progressively refine its decision policy and better adapt to diverse and dynamically changing degradation scenarios. Extensive experiments on eight challenging benchmarks demonstrate that LocAS-X significantly outperforms existing MLLM-based grounding models, achieving an average improvement of 30.59% in F1 score. These results highlight the potential of agentic localization and provide a promising foundation for robust object localization in complex and dynamic real-world environments.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Object Localization, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ky4gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents localization under degradation as a tool-composition problem solved by an MLLM agent. The agent selects restorers, super-resolution, and specialist detectors, and arbitrates among their outputs per instance. LocAS-X adds an experience memory of per-profile tool win-rates from a few labeled images. The paper claims localization in "any scene" with "any category".

Critical strengths

Timely framing with a real observation. The paper shows that whether to restore at all is an image-dependent decision (the restored-vs-original selection; App. F shows LocAS-X skips restoration and super-resolution where the rule-based version always applied them). That is a useful empirical point for the restoration-for-recognition literature.

Critical weaknesses
  1. Conceptual novelty is limited and the closest work is not discussed. Tool-using visual agents (VisProg, ViperGPT, HuggingGPT, MM-REACT) and LLM-driven restoration agents (AgenticIR, RestoreAgent, MAIR, 4KAgent, which is cited but not compared) already do "perceive → choose a restoration tool → verify". LocAS combines such an agent with detector routing and a candidate-selection step, but does not compare against an AgenticIR-style restoration agent followed by a strong detector. EAC is effectively a per-profile lookup of best tools learned from 80 labeled images, i.e., a few-shot supervised policy. Calling it "experience-aware calibration" that "progressively refines its decision policy" overstates what is a static retrieval table.
  2. The "any scene / any category" claim is not tested. Four of the eight benchmarks have an expert purpose-built from their own domain, and every degradation type has a matching restorer. There is no held-out evaluation: no leave-one-domain-out run with the matching expert and restorer removed, no unseen degradation types (e.g., snow-only, sandstorm, infrared), and no category lists outside the experts' coverage. Without these, the results show that a well-stocked toolbox matched to the test domains works, not that the agent generalizes.
Questions for authors
  1. Leave-one-domain-out results: remove the matching expert (and restorer) for a dataset and re-evaluate.
  2. Does the LLM agent beat a non-LLM controller, e.g., a CLIP or small CNN degradation classifier plus fixed rules, using the same tools?
  3. How sensitive is LocAS to the choice of agent model (a weaker or stronger MLLM)?
6 comments
AuthorsOpenReview ↗

Our Response to Q1-Q3

Dear Reviewer ky4g,

We sincerely appreciate your careful evaluation of our manuscript and the valuable suggestions provided. Taking into account the feedback from all reviewers, we have performed additional experiments, expanded the technical explanations, and refined the organization of the manuscript to enhance its clarity and completeness. For ease of reference, all revisions are highlighted in blue in the updated manuscript.

Q1:

We appreciate this suggestion and have conducted additional leave-one-domain-out experiments on DarkFace to assess the role of domain-specific restoration and detection tools. Specifically, we evaluate three configurations by removing the expert face detector, the brightness enhancement restorer, and both tools simultaneously.

Brightness EnhancementFace Detector[email protected] (%)
✓41.15
✓7.90
5.45
✓✓44.87

As reported in the above table, the F1-score decreases from 44.87 to 41.15, 7.90, and 5.45, respectively. Notably, removing brightness enhancement leads to a substantially larger performance degradation than removing the face detector alone, suggesting that effective restoration is particularly important for downstream localization in low-light environments. These findings highlight the complementary roles of restoration and detection tools within LocAS and support the importance of their coordinated use in agentic localization. We have added the complete results and discussions to Appendix K-i) of the revised manuscript.

Q2:

A. CLIP-Based Controller.

Thank you for this valuable suggestion. To evaluate LLM-based decision-making against simpler alternatives, we introduce four strategies: Fixed, Ensembling, Random, and Zero-Shot CLIP. In particular, Zero-Shot CLIP employs scene classification with the prompts "A photo of {c} scene" and "A photo with {c} objects" to select the top-K detectors by logit similarity scores. Although this non-LLM baseline performs best among the alternatives, it still trails LocAS-X by 16.97 F1 points on average across three datasets. These results support the benefits of agentic reasoning beyond simple scene-aware tool selection. Please refer to Fig. 5(b) and Lines 504–518 for detailed comparisons.

MethodHazyDetDarkFaceMARIS
Fixed33.1425.1615.28
Ensembling35.4510.4234.25
Random31.8612.2036.30
Zero-Shot CLIP38.9620.2843.07
LocAS-X53.5447.4052.29

B. CNN-Based Controller.

CNN-based degradation classifiers provide another reasonable alternative, but their reliance on predefined degradation categories may limit flexibility under unseen conditions, usually require classifier retraining. In contrast, LocAS can incorporate new restoration and detection tools. We agree that an additional CNN-based comparison would be informative and would consider including it in the camera-ready version, if feasible, or future work.

Q3:

We appreciate this important question. During our exploration, we tested Qwen3-VL-8B-Instruct and Qwen3.5-9B as alternative central agents. While both models exhibit relatively stable scene perception, their top-K detector selections occasionally differ, reflecting model-specific decision preferences. We found that these differences can be alleviated by refining the decision instructions in the system and user prompts.

More importantly, EAC further improves decision consistency by incorporating previously acquired experience into the decision-making process. Instead of relying solely on the inherent reasoning preferences of individual MLLMs, LocAS-X leverages retrieved experience to provide additional guidance for tool selection (see Appendix F). We therefore expect the experience-aware mechanism to reduce model-dependent decision variations and improve the robustness of agentic localization across different central MLLMs.

AuthorsOpenReview ↗

Our Response to W1

W1:

We appreciate the reviewer's concern regarding conceptual novelty. We would like to emphasize that our contribution is not the general concept of tool-using agents, but a localization-oriented agentic framework designed for diverse visual degradations.

A. Comparison with Existing Agentic Frameworks.

We acknowledge the relevance of existing visual and restoration agents, including VisProg, ViperGPT, AgenticIR, and 4KAgent. During our initial investigation, we attempted to build a localization pipeline based on 4KAgent, but encountered two limitations: (1) its iterative restoration process required over 10 minutes per image in our preliminary experiments, and (2) the restored images did not consistently benefit downstream localization, as restoration agents primarily focus on perceptual quality. These findings motivated our development of SAIR, which prioritizes localization-oriented restoration with lower computational overhead, and MEL, which incorporates adaptive expert detector routing and instance-level prediction refinement. Importantly, rather than simply connecting an existing restoration agent with downstream detectors, we redesign the entire agentic workflow around the localization objective, coordinating scene perception, adaptive restoration, expert selection, and instance-level reasoning into a unified decision-making framework.

B. Experience-Aware Calibration (EAC).

We agree that EAC is a lightweight retrieval-based calibration mechanism rather than a continuously evolving policy. We explored more complex alternatives, including decision-trajectory distillation, but found that directly utilizing localization outcome statistics provided more stable end-to-end performance. Unlike a fixed per-scene tool lookup, EAC retrieves similar fine-grained scene profiles and provides outcome-based recommendations at multiple decision nodes, allowing the agent to jointly consider current visual observations and historical experience. As demonstrated in Appendix F, this mechanism effectively guides tool selection and consistently improves localization performance. We have also revised potentially misleading descriptions of policy refinement (e.g., Lines 29–32 and 285–288) to clarify the actual mechanism.

Overall, our research is problem-driven, focusing on robust localization under diverse degradations. We believe the task-specific design and coordination of SAIR, MEL, and EAC offer meaningful advances over existing restoration-oriented agents by jointly addressing image enhancement, expert selection, and localization reasoning.

AuthorsOpenReview ↗

Our Response to W2 and Summary

W2:

We appreciate the reviewer's concern regarding the generalization capabilities of LocAS. We would like to clarify that "Any Scene / Any Category" primarily refers to the flexibility and extensibility of the agentic localization framework, rather than guaranteeing optimal performance of current LocAS-X version under every unseen condition.

1. Generalization and Extensibility of Agentic Localization.

As discussed in Fig. 1, conventional representation learning and fixed pipeline approaches typically require dataset-specific retraining to handle new degradation conditions or object distributions. In contrast, LocAS dynamically coordinates available restoration and detection tools, allowing new domain-specific experts to be incorporated without redesigning the entire localization pipeline. For instance, localization in infrared environments could be supported by simply integrating suitable infrared detectors or enhancement tools. We believe this modular adaptability is an important advantage of agentic localization in diverse and dynamic environments.

2. Additional Leave-One-Out Experiments and Cross-Dataset Generalization.

To further evaluate generalization beyond existing domain-specific tool configurations, we conduct three additional leave-one-out experiments in Appendix K. First, the Leave-One-Domain-Out experiment on DarkFace evaluates the impact of removing its corresponding expert detector and restoration tool (Appendix K-i). Second, the Leave-One-Calibration-Out experiment investigates whether EAC can transfer experience without target-dataset calibration (Appendix K-ii).

More importantly, DAWN and ExDark provide two important examples of cross-dataset generalization, as neither dataset is used for domain-specific expert fine-tuning, and both are evaluated without their corresponding EAC calibration samples in the held-out settings. DAWN contains diverse adverse weather conditions, including fog, rain, snow, and sandstorms, while ExDark covers challenging low-light environments with diverse object categories. Notably, no dedicated restoration tools are available for snow or sandstorms in current LocAS-X.

To further quantify this generalization capability, we conduct a Leave-One-Dataset-Out experiment on DAWN in the following table (Appendix K-iii). When DAWN training data are excluded, YOLOv11-Large suffers a substantial F1-score decline from 45.77 to 12.43. In contrast, LocAS maintains 47.71 without any expert detector fine-tuned on DAWN, while LocAS-X achieves 51.69 without DAWN-specific experience calibration. Together with the strong localization performance on ExDark, these findings suggest that agentic localization can generalize beyond target-dataset-specific training and calibration, rather than merely relying on experts specifically trained for the evaluated datasets.

ModelDataset Data[email protected]
YOLOv11-Large (w/o Training Data)×12.43
YOLOv11-Large (w/ Training Data)✓45.77
LocAS (w/o EAC)×47.71
LocAS-X (EAC w/o DAWN Data)×51.69

While we do not claim universal coverage of every possible scene or category in current version of LocAS-X, we believe these additional experiments, together with the modular extensibility of LocAS, provide meaningful support for robust and generalizable localization under diverse and dynamically changing visual conditions.

Summary

We sincerely appreciate the reviewer's thoughtful comments. We understand that the primary concerns focus on the conceptual novelty of LocAS, its generalization capabilities, and the effectiveness of agentic decision-making. We believe our clarifications and additional experiments have substantially addressed these concerns. First, our contribution lies in redesigning the entire agentic workflow specifically for localization, rather than simply combining existing restoration and detection tools. Second, our leave-one-out experiments, together with the strong results on DAWN and ExDark without dataset-specific expert fine-tuning, provide evidence of the framework's generalization beyond matched training domains. Third, our comparison with alternative decision strategies, particularly Zero-Shot CLIP, demonstrates the substantial benefits of agentic decision-making beyond simple scene classification and tool selection.

Overall, we firmly believe that the proposed framework offers a meaningful and practical direction for robust localization under diverse visual degradations. We sincerely thank the reviewer for helping us strengthen the manuscript and welcome any further questions or suggestions for additional experiments, which we would be happy to address whenever feasible.

Reviewer_xaotOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an agentic framework, LocAS, for object localization under diverse visual degradations. In order to achieve that, they take the Self-Adaptive Image(SAIR) to select restoration operations and determines whether the restored image is preferable to the original, and Multi-Expertise Localization (MEL) to coordinate multiple specialized detectors and consolidates their predictions through instance-level reasoning. They also present an extended framework, LocAS-X, incorporates experience-based memory retrieval (EAC) to guide localization decisions using experiences collected from a small number of annotated training examples.

The paper evaluates the proposed framework on eight benchmarks covering different visual conditions and application scenarios. LocAS achieves an average F1 score of 49.29%, while LocAS-X reaches 53.94%, compared with 23.35% for Qwen3-VL-8B. The paper also compares LocAS-X with YOLOv11-Large and reports ablation studies on the number of stored experiences and restoration-selection behavior. These results suggest that combining restoration selection, expert coordination, and experience retrieval can improve localization performance under visual degradations. However, the evaluation provides limited comparisons with alternative detection methods and simpler pipelines, leaving the contribution of the proposed agentic design and its generalizability insufficiently established.

Critical strengths
  1. LocAS-X reports consistent improvements over LocAS across the eight evaluated datasets, with the average F1 score increasing from 49.29% to 53.94%. The method also performs substantially better than the evaluated MLLM baselines on several challenging degraded datasets. These results indicate that a carefully coordinated combination of restoration and specialized localization tools can be practically effective.

  2. The separation between restoration selection, expert localization, and experience-guided calibration provides a clear engineering structure. The framework can incorporate different restoration modules and detectors without requiring the entire system to be trained end to end. This modularity is potentially useful for applications involving heterogeneous visual conditions.

Critical weaknesses

-The core framework combines existing image restoration models, specialized detectors, and an MLLM that selects tools and consolidates predictions. The EAC component stores empirical localization outcomes associated with scene profiles and retrieves similar experiences to guide later decisions. Although this combination is useful, the paper does not sufficiently establish a novel decision-making principle or learning mechanism beyond the integration of existing components.

-In particular, the paper does not clearly explain why the proposed experience representation and retrieval procedure are preferable to simpler alternatives, such as a lookup table indexed by degradation type, a lightweight learned tool-selection policy, or direct evaluation of a small number of candidate configurations. The existing ablations establish that the complete system is effective, but they do not adequately establish that the specific agentic formulation is necessary. This limits the conceptual contribution relative to the paper's broad claims about adaptive agentic localization.

-The experience memory is initialized using annotated examples sampled from the training split of each dataset, and the method is evaluated across those same benchmark domains. This setup demonstrates that limited labeled experience can improve performance within the evaluated domains, but it does not by itself establish generalization to previously unseen degradation types, object distributions, or datasets.

-The experience-harvesting procedure evaluates alternative decision configurations using ground-truth localization performance. The paper should more clearly specify the scope of this exploration, its computational and annotation costs, and how the resulting experience is prevented from encoding benchmark-specific information that would not be available in a genuinely unseen deployment environment. Without stronger cross-dataset or held-out-domain evaluation, the claim of broad adaptation to dynamically changing environments remains insufficiently supported.

Questions for authors
  1. Are the experience memories constructed independently for each benchmark, and are all annotated samples used to build them strictly separated from the evaluation samples? More importantly, can LocAS-X transfer to a target dataset whose annotations have not been used to construct or calibrate its experience memory? A cross-dataset evaluation with the memory built only from source-domain data would materially clarify the generalization claim.
  2. How does EAC compare with simpler alternatives, including a degradation-category lookup table, a non-agentic tool-selection policy, and exhaustive or fixed-rule selection among candidate configurations? Can the authors provide controlled comparisons that isolate the benefit of experience-aware reasoning from the benefit of using multiple restoration and detection models?
  3. Cost of experience harvesting and inference. How many tool configurations are evaluated per annotated sample when constructing the experience memory, and what are the resulting annotation, inference, and computational costs? Since the framework invokes multiple restoration and detection models and performs LLM-based reasoning, a more complete cost analysis is needed to substantiate its practical scalability.
  4. What is the underlying rationale for the proposed agentic decision-making strategy? Can the authors provide a more principled analysis of when restoration selection, expert coordination, and experience retrieval are expected to improve localization?
  5. Can the authors provide complete implementation details, including the decision rules, prompt configurations, memory construction procedure, retrieval settings, and computational costs, so that the full pipeline can be independently reproduced?
  6. How does the proposed method compare with stronger conventional baselines and simpler multi-candidate strategies under comparable computational budgets? Can the authors isolate the contribution of each component and demonstrate that the improvements justify the additional complexity?
  7. The experimental evaluation appears to be limited to comparisons with YOLOv11-L, which is insufficient to fully demonstrate the effectiveness and competitiveness of the proposed method. Notably, the proposed method exhibits even poorer performance on the HazyDet, B-Night, and B-Rainy datasets compared with YOLOv11-L. The authors should provide a more thorough analysis of the underlying reasons for these performance degradations, particularly in terms of the challenges posed by adverse environmental conditions. Furthermore, the experimental comparisons should be extended to include more recent state-of-the-art YOLO models, such as YOLOv13 and YOLO26, to establish whether the proposed method offers consistent advantages over stronger and more up-to-date baselines.
8 comments
AuthorsOpenReview ↗

Our Response to Q1 & Q2

Dear Reviewer xaot,

We sincerely thank you for your thorough review and constructive feedback. In response to the comments and suggestions from all reviewers, we have conducted additional experiments, provided further technical clarifications, and improved the structure and presentation of our manuscript. To facilitate the review process, all modifications are highlighted in blue in the revised manuscript.

Q1:

A. Calibration Samples and Data Separation.

For EAC, we select 10 annotated calibration images from the training split of each benchmark, resulting in 80 calibration images across eight datasets. These samples are strictly separated from the evaluation data to prevent data leakage. The collected experiences are aggregated into a shared experience memory, which is used for the main LocAS-X evaluations.

B. Cross-Dataset Generalization of Experience Memory.

To directly address the reviewer's concern, we conduct an additional Leave-One-Calibration-Out experiment to evaluate whether EAC can transfer knowledge to a target dataset without using its annotations. Specifically, we remove all 10 DAWN calibration images from the experience memory and evaluate LocAS-X using only experiences collected from the other seven benchmarks. As shown in the following table, LocAS-X achieves 51.69 [email protected] on DAWN without any DAWN-specific calibration, outperforming LocAS without EAC (47.71) by 3.98 F1 points. Importantly, DAWN contains diverse adverse weather conditions, including fog, rain, snow, and sandstorms. Snow and sandstorm conditions are not represented in the remaining calibration samples, and none of the expert detectors are specifically fine-tuned on DAWN. Nevertheless, EAC continues to provide substantial improvements, demonstrating its potential to transfer experience across datasets and previously unseen degradation conditions.

MethodDAWN Calibration Data (EAC)[email protected] (%)
LocASN/A47.71
LocAS-XNo51.69
LocAS-XYes52.44

We have incorporated the complete experimental results and analysis into Appendix K-ii) of the revised manuscript. Additionally, the dataset processing scripts are provided in the supplementary code to facilitate reproducibility and verification of the data preparation and evaluation protocols.

Q2:

A. Contribution of Experience-Aware Calibration.

We appreciate the reviewer's suggestion to distinguish the benefits of experience-aware reasoning from those of using multiple restoration and detection tools. In fact, the comparison between LocAS and LocAS-X in Table 1 provides a controlled evaluation of EAC, as both frameworks share the same restoration and detection tool pools. Without EAC, LocAS relies on MLLM-guided decisions based on the perceived scene characteristics and predefined decision rules. By incorporating experience memory, LocAS-X achieves an average improvement of 4.65 F1 points across eight benchmarks, demonstrating the additional benefits of experience-aware calibration beyond the use of multiple expert tools.

B. Comparison with Simpler Tool-Selection Strategies.

We further introduce four alternative decision strategies: Fixed, Ensembling (all-detector execution), Random, and Zero-Shot CLIP. Fixed employs a predefined restoration and detection pipeline, Ensembling combines predictions from all six expert detectors, Random replaces agent decisions with random choices, and Zero-Shot CLIP performs scene-aware detector selection using CLIP zero shot classification. As shown in the following table, even the strongest alternative, Zero-Shot CLIP, underperforms LocAS-X by 16.97 F1 points on average across three datasets.

MethodHazyDetDarkFaceMARIS
Fixed33.1425.1615.28
Ensembling35.4510.4234.25
Random31.8612.2036.30
Zero-Shot CLIP38.9620.2843.07
LocAS-X53.5447.4052.29

These results provide complementary evidence that both experience-aware calibration and agentic decision-making contribute to localization performance, rather than the improvements arising solely from the availability of multiple expert models. Detailed experimental settings and analyses have been added to Lines 504–518 and Fig. 5(b) of the revised manuscript.

AuthorsOpenReview ↗

Our Response to Q3-Q5

Q3:

A. Computational Cost of Framework Construction.

We appreciate the reviewer's concern regarding the computational scalability of LocAS-X. During EAC, we evaluate three restoration tools and four expert detectors for each annotated calibration image to collect localization experiences. We use only 10 annotated images per benchmark (80 in total). Experience acquisition requires approximately 5–10 minutes per image, resulting in around 8 hours for the complete offline calibration.

Additionally, domain-specific expert detector fine-tuning requires approximately 0.5–3.0 GPU-hours per model, while the central decision-making MLLM requires no additional training. Importantly, experience acquisition is performed only once offline, and the resulting memory can be reused during inference. We have added Appendix L.3 to provide a more detailed discussion of these computational costs.

B. Inference Latency and Computational Efficiency.

We provide an inference latency analysis in Appendix G. LocAS employs relatively lightweight restoration tools, while SAIR helps reduce false-positive predictions and downstream MLLM decoding overhead. In our DarkFace latency evaluation, the overall inference time is approximately 19% higher than that of a standalone MLLM detector, indicating a moderate overhead despite the integration of multiple tools. Furthermore, LocAS-X achieves a substantially higher average F1-score than Qwen3.8-27B (53.94% vs. 36.75%), while requiring fewer estimated FLOPs. These findings suggest that the performance improvements are not simply attributable to greater computational resources, but also reflect the effectiveness of coordinated agentic decision-making.

Q4:

The underlying rationale of LocAS-X is that the effectiveness of restoration and detection tools depends on the specific visual characteristics of each input image. Therefore, adaptive localization requires decisions informed by both current visual perception and previously observed localization outcomes. Specifically, we first develop LocAS with two complementary components. SAIR selects restoration operations according to their expected benefits for downstream localization rather than perceptual image quality, since unnecessary restoration may introduce artifacts and degrade detection performance, particularly in mildly degraded scenes. Meanwhile, MEL dynamically coordinates specialized detectors according to scene characteristics and object requirements, enabling more appropriate expert selection and prediction refinement.

Although this perception-guided framework already achieves substantial improvements, we observe that predefined decision rules cannot always capture the fine-grained effects of restoration, SR, and detector selection. This motivates EAC, which retrieves relevant experiences from annotated calibration samples and incorporates their observed localization outcomes into the decision-making process. Consequently, LocAS-X can make more informed, image-specific decisions rather than relying solely on coarse scene-level heuristics. Importantly, our tool-selection analysis in Appendix F shows that LocAS-X exhibits more diverse and adaptive selection behaviors than LocAS. Together with the consistent performance improvements reported in Table 1, these findings support our design rationale: localization-oriented tool coordination, further guided by empirical experience, enables more effective adaptation to diverse visual degradations.

Q5:

A. Implementation Details.

We have provided the complete decision rules, memory construction procedures, and system/user prompts in Appendix I. Additionally, Sec. 3.4 and Algorithm 1 have been added to illustrate the complete execution pipeline. The supplementary code contains the full implementation, including instructions for environment setup, dataset preprocessing, and reproducing the main results.

B. Experience Retrieval Settings.

We have added Appendix J to explicitly define the retrieval similarity metric, which combines four fine-grained scene attributes: object scale, object density, visibility, and illumination. Detailed similarity formulations and retrieval settings are provided for reproducibility.

C. Computational Costs.

As detailed in our response to Q3, Appendix L.3 summarizes the computational costs of experience acquisition and expert detector training, while Appendix G provides the inference latency analysis.

We believe these additions substantially improve the transparency and reproducibility of LocAS-X, enabling independent implementation and evaluation of the proposed framework.

AuthorsOpenReview ↗

Our Response to Q6 & Q7

Q6:

A. Comparison with Stronger Baselines and Simpler Strategies.

We appreciate the reviewer's concern regarding baseline strength and computational fairness. We have compared LocAS-X with recent localization approaches, including LocateAnything (CVPR '26), and further introduced the stronger Qwen3.8-27B baseline in Table 1. Despite requiring fewer estimated FLOPs, LocAS-X achieves a substantially higher average F1-score than Qwen3.8-27B (53.94% vs. 36.75%). We also investigated CoT-based localization strategies and observed failure cases when generating predictions for scenes containing numerous objects (see Appendix F). Furthermore, to evaluate simpler multi-candidate strategies, we introduce Fixed, Ensembling, Random, and Zero-Shot CLIP baselines in Fig. 5(b). These comparisons provide additional evidence that the improvements arise from effective agentic decision-making rather than merely employing multiple expert models.

B. Component Contributions and Computational Efficiency.

In Fig. 5(a), we conduct component-wise ablations to evaluate the contributions of SAIR, MEL, and EAC. The results demonstrate that localization-oriented restoration, adaptive multi-expert coordination, and experience-aware calibration each contribute to the overall performance improvements.

Importantly, the additional computational overhead remains moderate. As analyzed in Appendix G, LocAS incurs approximately 19% additional inference latency compared with a standalone MLLM detector in our DarkFace evaluation. Together with the substantial localization improvements and the FLOPs comparison above, these results suggest that the benefits of the proposed components justify their additional computational complexity.

Q7:

A. Comparison with Stronger YOLO Baselines.

We appreciate the reviewer's suggestion to include stronger and more recent conventional detectors. Accordingly, we additionally evaluate YOLO26-L in the revised manuscript (Appendix C). Its performance is generally comparable to YOLOv11-Large, while LocAS-X maintains a substantial advantage in average F1-score across all eight benchmarks. We acknowledge that YOLO-based detectors achieve better performance on certain datasets, including HazyDet, B-Night, and B-Rainy. One important distinction is that these conventional detectors are trained on aggregated in-domain data, allowing them to learn degradation-specific features and object distributions. In contrast, LocAS relies on general-purpose and specialized open-vocabulary experts without requiring detector fine-tuning for every evaluated degradation. Therefore, our goal is not to outperform fully supervised detectors on every individual dataset, but to provide a more transferable localization framework across diverse visual conditions.

B. Cross-Dataset Generalization Beyond In-Domain Training.

To further demonstrate this distinction, we introduce a Leave-One-Dataset-Out experiment on DAWN (Appendix K-iii). When DAWN training samples are excluded, YOLOv11-Large suffers a substantial F1-score decline from 45.77 to 12.43. In contrast, LocAS achieves 47.71, while LocAS-X reaches 51.69 without DAWN-specific experience calibration. Importantly, none of the expert detectors are fine-tuned on DAWN, which also includes challenging weather conditions such as snow and sandstorms.

ModelDataset Data[email protected]
YOLOv11-Large (w/o Training Data)×12.43
YOLOv11-Large (w/ Training Data)✓45.77
LocAS (w/o EAC)×47.71
LocAS-X (EAC w/o DAWN Data)×51.69

These results suggest that conventional detectors can achieve strong in-domain accuracy through supervised representation learning, whereas agentic localization offers greater flexibility when transferring to new domains without target-specific detector training. Together with the additional YOLO26-L comparison, these experiments further clarify the practical advantages and limitations of LocAS-X.

Reviewer_La9BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LocAS, an agentic framework for object localization under diverse visual degradations. The framework combines scene-aware image restoration and enhancement, selection of domain-specific object detectors, and multimodal reasoning for candidate selection. Experiments are conducted on eight datasets covering different visual conditions. The results show that LocAS and LocAS-X outperform the evaluated multimodal large language models and existing localization methods in terms of F1 score at an IoU threshold of 0.5.

Critical strengths

The paper addresses object localization under diverse visual degradations, including haze, low illumination, and adverse weather. Evaluating the framework across eight datasets provides a broader assessment than experiments limited to a single degradation type. The experimental results show substantial improvements over the evaluated baselines.

Critical weaknesses
  1. The paper compares LocAS and LocAS-X with several existing models and localization methods. However, the comparison does not establish whether the proposed agentic strategy is more effective than simpler alternatives using the same restoration tools and expert detectors. Comparisons against fixed tool selection, random expert selection, or exhaustive evaluation of available configurations would help determine whether the gains arise from the decision-making mechanism itself or primarily from the expanded toolset.

  2. The baseline methods include individual multimodal models, whereas LocAS can use multiple domain-specific detectors and image enhancement tools. The reported performance advantage therefore reflects the complete system configuration rather than a controlled comparison of the underlying localization or reasoning methods.

  3. The paper describes LocAS-X as not requiring dataset-specific training, but the framework relies on domain-specific expert detectors. The experience-aware calibration module also uses annotated examples from the target datasets. The authors should clarify what they mean by avoiding dataset-specific training and distinguish this claim from dependence on domain-specific training data and annotated calibration samples.

Questions for authors
  1. The paper compares LocAS with several multimodal models and existing localization methods. Could the authors compare the proposed agentic decision-making strategy with fixed tool selection, random expert selection, and exhaustive configuration evaluation while keeping the available restoration tools and expert detectors unchanged? Also, please compare the proposed method with recent SOTA detectors.

  2. How are the object detectors used by LocAS trained, and what training data are available to each detector? Could the authors report the additional training data, model resources, and computational costs required by LocAS, and explain how these differ from those of the baseline methods?

  3. The paper states that LocAS-X does not require dataset-specific training, while the system relies on domain-specific expert detectors and annotated calibration examples. Could the authors clarify the scope of this claim and specify which components require dataset-specific training or annotations?

  4. Does the experience-aware calibration mechanism generalize to a dataset or degradation type that is not represented in its calibration memory? Could the authors evaluate cross-dataset transfer or test the framework on previously unseen degradation conditions?

6 comments
AuthorsOpenReview ↗

Our Response to Q1 & Q2

Dear Reviewer La9B,

Thank you for your thoughtful review and constructive feedback. In response to the reviewers' comments, we have conducted additional experiments, clarified technical details, and reorganized the manuscript to improve its clarity and completeness. All changes are highlighted in blue in the revised manuscript.

Q1:

A. Effectiveness of Agentic Decision-Making.

To address this concern, we introduce four alternative strategies: Fixed, Ensembling (all-detector execution), Random, and Zero-Shot CLIP. These variants evaluate different tool-selection strategies using the existing tool pools, ranging from a predefined pipeline to exhaustive detector execution and scene-guided selection. The results are summarized below ([email protected], %):

MethodHazyDetDarkFaceMARIS
Fixed33.1425.1615.28
Ensembling35.4510.4234.25
Random31.8612.2036.30
Zero-Shot CLIP38.9620.2843.07
LocAS-X53.5447.4052.29

LocAS-X consistently outperforms all alternatives, exceeding the strongest competing strategy by 16.97 F1 points on average. Notably, exhaustive detector execution introduces substantial false positives, while scene-guided CLIP selection remains inferior to agentic reasoning. These findings highlight the importance of adaptive decision-making beyond simply combining expert tools. Detailed settings and analyses are provided in Lines 504–518 and Fig. 5(b) of the revised manuscript.

B. Comparison with Recent SOTA Detectors.

We further strengthen our comparisons with recent localization approaches. As shown in Table 1, LocateAnything (CVPR '26) suffers substantial performance degradation under diverse visual conditions, achieving an average F1-score of 16.63%, compared with 53.94% for LocAS-X. We additionally introduce Qwen3.8-27B, a more advanced MLLM with higher estimated FLOPs than the entire LocAS-X pipeline, yet its average F1-score remains 17.19 points lower. For CoT-based referring methods such as Rex-Thinker, we discuss the challenges of dense multi-object localization and provide a representative CoT failure case in Appendix E. Finally, we include YOLO26-L as an additional closed-vocabulary detector baseline in Appendix C. These comparisons further support the advantages of agentic localization over conventional detector scaling and static localization approaches under challenging visual degradations.

Q2:

A. Training Data.

The four expert detectors in LocAS are fine-tuned on the corresponding training splits, primarily using non-degraded samples to avoid direct adaptation to target-specific degradation conditions. In contrast, baseline MLLMs are directly fine-tuned on the corresponding degraded-scene training subsets. All evaluations are conducted on strictly disjoint validation/test splits to prevent data leakage. We have added Appendix L.1 (Tables 9–10) to clarify the detailed data composition for expert detector SFT, EAC calibration, baseline MLLM SFT, and benchmark evaluation.

B. Model Resources.

All pretrained models are obtained from ModelScope, with detailed model resources and corresponding references provided in Appendix L.2 (Table 11).

C. Computational Cost.

All experiments are conducted using A6000 GPUs. Fine-tuning each expert detector requires 0.5–3.0 GPU-hours, while the central decision-making MLLM requires no additional training. EAC experience acquisition takes approximately 5–10 minutes per sample, requiring around 8 hours for 80 calibration samples. In comparison, baseline MLLM SFT requires 0.5–8.0 GPU-hours per model–dataset combination for one epoch. Detailed computational and annotation costs are provided in Appendices L.3–L.4. Regarding inference efficiency, LocAS-X requires fewer estimated FLOPs than the larger Qwen3.8-27B, while achieving a substantially higher average F1-score (53.94% vs. 36.75%). Additional inference latency analysis is provided in Appendix G. Finally, we provide detailed implementation guidance in the supplementary code, including model adapter weights, training/calibration dataset construction, and benchmark evaluation scripts, to facilitate reproducibility.

AuthorsOpenReview ↗

Our Response to Q3-Q4 and W1-W2

Q3:

Thank you for raising this important question. We would like to clarify that our claim of requiring no dataset-specific training refers to the agentic decision-making framework, rather than the preparation of individual expert detectors. Our contributions consist of two parts:

1. Agentic Localization (LocAS). LocAS itself requires no additional training and dynamically coordinates available restoration and detection tools. In our implementation, four domain-specific detectors are fine-tuned to construct a sufficiently diverse expert pool, as suitable pretrained open-vocabulary experts are not readily available for all domains. These detectors use non-degraded training subsets. Importantly, such fine-tuning is not intrinsic to LocAS and can be avoided when suitable pretrained expert detectors are available.

2. Experience-Aware Calibration (LocAS-X). EAC requires a small number of annotated calibration samples to construct the experience memory, without updating any model parameters. As demonstrated in Appendix D, only 10 annotated images per benchmark (80 in total) are sufficient to achieve substantial performance improvements across the eight benchmarks.

We have further clarified the training data and calibration requirements in Appendix L.1 (Tables 9–10). We emphasize that LocAS-X requires no dataset-specific training of its central decision-making MLLM, while expert detector preparation and experience calibration are separate processes.

Q4:

We have added a Leave-One-Calibration-Out experiment on DAWN to evaluate the cross-dataset transferability of EAC (see Appendix K-ii). After excluding all 10 DAWN-specific entries from the 80-sample experience memory, LocAS-X still achieves an F1-score of 51.69, improving upon LocAS without EAC by 3.98 points.

MethodDAWN Calibration Data (EAC)[email protected] (%)
LocASN/A47.71
LocAS-XNo51.69
LocAS-XYes52.44

Notably, DAWN contains diverse adverse weather conditions, including snow and sandstorms, which are absent from the remaining calibration data. Furthermore, no detector is specifically fine-tuned on DAWN, and no dedicated restoration tools are available for these unseen conditions. These findings suggest that EAC can transfer useful decision-making experience across datasets and provide performance gains even under previously unrepresented degradation conditions.

W1:

Thank you for raising this important concern. As discussed in our response to Q1, we introduce four alternative strategies, including Fixed, Ensembling, Random, and Zero-Shot CLIP, to assess whether the performance gains arise from agentic decision-making rather than merely expanding the tool pool. Following the suggestion from Reviewer ky4g, we further include Zero-Shot CLIP as a stronger baseline that performs scene-aware detector selection and softmax-weighted majority voting. Despite outperforming the other alternatives on average, Zero-Shot CLIP still trails LocAS-X by 16.97 F1 points across three datasets. These results highlight the benefits of agentic reasoning beyond simple scene classification and detector selection.

W2:

We acknowledge that LocAS benefits from integrating multiple expert tools, as our contribution focuses on the overall agentic localization framework rather than an individual detector. Regarding computational efficiency, we provide a detailed inference latency analysis in Appendix G. LocAS employs lightweight restoration tools, while SAIR reduces false-positive predictions and shortens downstream MLLM decoding time, keeping the overall latency comparable to that of a standalone MLLM detector. Moreover, LocAS-X achieves a substantially higher average F1-score than Qwen3.8-27B (53.94% vs. 36.75%) despite requiring fewer estimated FLOPs. These results demonstrate that the performance gains of LocAS-X are not simply attributable to increased computational resources, but reflect the effectiveness of the proposed agentic localization framework.

AuthorsOpenReview ↗

Our Response to W3 and Summary

W3:

We appreciate the reviewer's concern and acknowledge that our original description may have been unclear. As clarified in our response to Q3, the claim of avoiding dataset-specific training refers specifically to the central decision-making MLLM, rather than the entire expert tool pool. While four domain-specific detectors are fine-tuned to construct the expert pool in our implementation, this preparation is not intrinsic to LocAS and can be avoided when suitable pretrained detectors are available. Furthermore, EAC requires a small number of annotated calibration samples to construct the experience memory, but does not involve any model parameter updates. We have clarified these distinctions and provided detailed training and calibration data information in Appendix L.1 (Tables 9–10) of the revised manuscript.

Summary

We sincerely appreciate the reviewer's constructive feedback. We understand that the primary concerns relate to the effectiveness of agentic decision-making, experimental fairness, and potential data leakage. We believe our additional experiments and clarifications provide strong evidence addressing these concerns, demonstrating the critical role of agentic reasoning while maintaining strict separation between training and evaluation data. In particular, DAWN and ExDark provide compelling examples of cross-dataset generalization: none of the expert detectors are fine-tuned on these datasets, yet LocAS maintains strong localization performance under diverse degradation conditions. These findings reinforce our confidence in the effectiveness and generalizability of the proposed agentic localization framework.

We welcome further questions or suggestions for additional experiments and will make every effort to address them during the rebuttal period. Any remaining investigations could also be incorporated into the camera-ready manuscript, where appropriate, or future revisions. We firmly believe that agentic decision-making represents a valuable direction for robust localization under diverse visual degradations.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Our Rebuttal Report and Overall Summary

To Dear Area Chair,

We sincerely thank the reviewers for their constructive feedback. We have addressed their concerns through additional experiments, technical clarifications, and manuscript revisions. All modifications are highlighted in blue. We summarize the four major concerns and our responses below.

1. Novelty and Effectiveness of Agentic Decision-Making.

Our primary contribution is to introduce Agentic Localization as a localization-oriented decision-making paradigm, rather than simply combining existing tools. Unlike previous restoration-oriented agents, LocAS redesigns the workflow for downstream localization through localization-aware restoration (SAIR), adaptive expert coordination and instance-level reasoning (MEL), and experience-aware calibration (EAC).

To demonstrate the necessity of agentic decisions, we have added a controlled ablation study in Fig. 5(b), comparing LocAS-X against Fixed, Ensembling, Random, and Zero-Shot CLIP strategies using the existing tool pools. LocAS-X consistently outperforms these alternatives, exceeding the strongest baseline by 16.97 F1 points on average across three datasets. Together with the existing component ablations in Fig. 5(a) and the 4.65-point improvement from LocAS to LocAS-X, these results provide strong evidence for the effectiveness of agentic decision-making and EAC.

2. Generalization across Unseen Domains and Degradations.

To address concerns regarding domain-specific experts and calibration data, we have added three Leave-One-Out evaluations in Appendix K. First, the DarkFace experiments quantify the complementary contributions of restoration and expert detectors. Second, LocAS-X achieves 51.69 F1 on DAWN without DAWN-specific experience calibration, outperforming LocAS by 3.98 points. Third, when DAWN training data are excluded, YOLOv11-Large drops from 45.77 to 12.43 F1, whereas LocAS-X achieves 51.69 without DAWN-specific calibration or expert fine-tuning.

Together with the existing DAWN and ExDark results, these experiments provide stronger evidence of generalization beyond matched training and calibration domains. We have also clarified the distinction between agent training, expert preparation, and experience calibration.

3. Computational Efficiency and Experimental Fairness.

We have strengthened the comparisons with additional recent baselines, including Qwen3.8-27B in Table 1 and YOLO26-L in Appendix C. LocAS-X achieves 53.94 F1 compared with 36.75 for Qwen3.8-27B, while requiring fewer estimated FLOPs. Furthermore, the existing latency analysis in Appendix G shows approximately 19% additional inference latency compared with a standalone MLLM detector on DarkFace.

We have directed reviewers to this analysis and added computational costs in Appendix L.3, covering expert fine-tuning and offline EAC acquisition. These findings demonstrate a favorable accuracy–efficiency trade-off and indicate that the improvements are not simply attributable to increased computational resources.

4. Implementation Details and Reproducibility.

We have expanded the implementation details by adding Sec. 3.4 and Algorithm 1 to summarize the complete LocAS-X workflow, Appendix J to specify experience retrieval, and Appendix L to document dataset splits, model resources, computational costs, and annotation requirements.

We have clarified the strict separation between training/calibration and evaluation data. We also directed reviewers to existing analyses, including EAC sample-size ablations (Appendix D), tool-selection statistics (Appendix F), latency analysis (Appendix G), and detailed prompts (Appendix I). The supplementary code provides additional implementation and evaluation resources.

Overall, we believe that these additional experiments and revisions have substantially strengthened the paper and addressed the reviewers' principal concerns. The new controlled ablations establish the benefits of agentic decisions, the Leave-One-Out evaluations strengthen the generalization evidence, and the expanded analyses improve experimental transparency and reproducibility.

We sincerely appreciate the Area Chair's time and consideration and respectfully hope these substantial improvements will be considered in the final assessment.

To Dear Reviewers,

We sincerely appreciate your valuable feedback and warmly welcome further comments, questions, and discussions regarding our revised manuscript.

Dear Reviewer,

We sincerely apologize for reaching out once again. As only 24 hours remain in the author rebuttal period, we would be deeply grateful if you could kindly spare a moment from your busy schedule to review our responses. Over the past week, we have sincerely tried our best to address your concerns and improve our manuscript. We fully understand and respect that our responses may not completely resolve your concerns, and we do not wish to place any pressure on you to reconsider your rating. Even if your assessment remains unchanged, we would be truly grateful for a brief reply, as your feedback would help us understand what remains insufficient and how we can further improve our work. We sincerely appreciate your time, patience, and consideration, and apologize again for any inconvenience.

Dear Area Chair and Reviewers,

As the rebuttal period comes to an end, we sincerely thank all three reviewers for their valuable feedback and thoughtful discussions. We are truly grateful that one reviewer has explicitly acknowledged that our additional experiments and clarifications addressed their main concerns. For the other two reviewers, we fully respect their remaining reservations, while sincerely hoping that our subsequent experiments and clarifications have helped resolve some of the outstanding concerns and misunderstandings.

We understand that the current ratings remain unchanged, and we fully respect each reviewer's independent judgment. Nevertheless, if our responses have satisfactorily addressed your main concerns, we would be deeply grateful if you could kindly consider revisiting your rating. For reviewers who still have reservations, we sincerely hope that, if you feel the strengths of our work outweigh its remaining limitations, you might consider supporting a more favorable assessment during the subsequent discussion with the Area Chair.

Regardless of the final outcome, we are truly grateful for your time, patience, and constructive feedback throughout this process. Your comments have greatly helped us improve our work. Thank you all once again!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.