acceptodds
Under review as a conference paper at ICLR 2027

DCAgent: Towards Dense Visual Perception via a Divide-and-Conquer Tool-Use Agent

Abstract

Frontier Vision-Language Models (VLMs) collapse on dense visual perception—tasks that require enumerating or localising large numbers of small, tightly packed objects. The structural cause is a fixed-budget visual encoder that smears small dense objects before they reach the language model. The intuitive remedy—divide-and-conquer crops—has been explored as a hard-coded external pipeline, but never as a learned, agentic capability inside the VLM itself. We present DCAgent, a tool-use agent that natively learns to divide and conquer its own image input through two visual tools, Divide and Merge: Divide splits the current view into sub-pieces the agent itself selects, and Merge aggregates the resulting leaf detections via class-aware non-maximum suppression. The multi-turn tool-use policy is trained with GRPO under an end-to-end final reward combining count MAE and a DETR-style Hungarian-matched detection IoU on the merged output. Three task-specialised agents are trained on SKU-110K, SOD-Aerial, and LVLM-Count from a Qwen3-VL-8B starting point and each surpasses frontier proprietary VLMs on its target dataset. We then consolidate them into a single deployable model with Online Policy Distillation under a four-teacher setup. The merged model has internalised the use of the divide-and-merge tools, not the patterns of any single dataset: it auto-triggers the tools on counting and dense-perception inputs and bypasses them on knowledge-and-reasoning inputs, so the single checkpoint substantially improves on all three training datasets, generalises to CountBench, and maintains the Qwen3-VL-8B base on MMMU, MMMU-Pro, MMStar, and MMBench-V1.1.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Dense Visual Perception, Vision-Language Models, Visual Tool Use, Recursive Visual Inference, Online Distillation Policy

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ez4WOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DCAgent, a tool-use vision-language agent designed to improve dense visual perception, particularly object counting and localization in crowded scenes. The key contribution is an adaptive divide-and-conquer framework that enables a VLM to recursively split images into smaller regions, detect objects locally, and merge predictions while removing duplicates. The authors train this multi-turn tool-use policy using GRPO with an end-to-end reward combining counting accuracy and detection quality, followed by a four-teacher Online Policy Distillation (OPD) approach that consolidates three task-specific agents into a single model while preserving general VLM capabilities. Experiments on SKU-110K, SODA-Aerial, and LVLM-Count demonstrate substantial improvements over the base Qwen3-VL-8B and direct SFT baselines, with additional evaluations on CountBench showing some generalization to unseen counting tasks. Ablation studies further examine the effects of the tool interface, reinforcement learning, overlap ratios, and merging strategies, supporting the effectiveness of the proposed framework.

Critical strengths
  1. Well-motivated Method: The paper addresses a clear limitation of VLMs in dense visual perception through an adaptive divide-and-conquer framework. The integration of multi-turn RL and policy distillation is well motivated and effective.

  2. Illustrative Qualitative Analysis: The paper provides clear qualitative examples across different tasks, effectively demonstrating the learned adaptive image partitioning strategy and improvements in object localization.

Critical weaknesses
  1. Limited Novelty: Similar divide-and-conquer strategies for improving VLM visual perception have been explored in prior work. The proposed approach mainly extends these ideas with reinforcement learning and tool use, raising concerns about its methodological novelty.

  2. Limited Generalizability: The performance improvements are largely restricted to dense object counting and detection tasks, while the model shows performance degradation on general VLM benchmarks. This raises concerns about whether the proposed method provides broadly useful capabilities beyond the specific training domains.

Questions for authors

Can you test the performance of using frontier agents such as Claude Opus for coding to solve this problem?

Reviewer_pX7iOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses the failure of VLM on object counting and dense small-object detection tasks. Instead of relying on hard-coded external pipelines like fixed-grid cropping, the authors propose DCAGENT, a tool-use agent that learns to divide and conquer its own image input through a compact visual interface. The experiments show substantial gains on the dense benchmarks and zero-shot transfer to CountBench while preserving performance on general VLM benchmarks.

Critical strengths

The author reformulates the "crop-and-stitch" framework into an divide and conquer tool used by the agent, which is learnable and flexible.

Critical weaknesses
  1. There are several internal prompts (maybe generated by AI) that reveal flaws in the experiments. Line 845: "AUTHOR-CONFIRM: replace [AUTHOR_PROTOCOL] with the source split for ... if the same public split was used for both calibration and reporting, recompute the score on the untouched remainder." which could means data leakage promblem. Line 832: "AUTHOR-CONFIRM—required before submission. The source bundle does not contain the manifest needed to reconstruct the... Confirm that the aggregate 4,832/241/1,207 counts... match the experiments; otherwise replace them with the actual protocol." which means the exact dataset splits are not verified.
  2. Numerical inconsistencies in the experiments results. In Table 1, Qwen3-VL-8B + direct SFT achieves 15.4 on SKU-110K and 15.6 on LVLM-Count. Tn Table 4, the same setting (Qwen3-VL-8B + direct SFT) achieves 12.5 on SKU-110K and 15.4 on LVLM-Count.
Questions for authors
  1. What is the unresolved notes in Lines 819, 832, and 845?
  2. Why the Qwen3-VL-8B + direct SFT results in Tab 1 conflicts with the results in Tab 4?
  3. The authors state that they ran LVLM-Count and DC2 with Qwen3-VL-8B-Thinking backbone. But the results are not shown in the paper. Can the authors provide the results?
  4. The authors state the four-teahcer OPD is a major contribution over weight-merging, but no weight-merging baselines are evaluated. Can the authors provide the results for this baseline?
Reviewer_S21xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DCAgent studies dense object counting and detection from an image and a task query. It is motivated by the loss of small-object detail under limited visual-token budgets and by redundant processing and boundary duplication in divide-and-conquer pipelines. A pretrained vision-language model first learns valid recursive tool use through rejection-sampled fine-tuning. Task specialists are then optimized with reinforcement learning over complete trajectories: the policy chooses whether to subdivide a crop or emit local detections, while fixed coordinate restoration, core ownership, and nonmaximum suppression assemble the global prediction. Rewards depend on the final merged count and matched detections. On-policy distillation consolidates the specialists with a base-model teacher by matching routed teacher distributions along student-generated histories. At inference, the consolidated student invokes the visual tools recursively and returns deduplicated object locations and a global count.

Critical strengths
  1. The dense-perception problem is relevant, and the introduction clearly identifies resolution loss and boundary duplication as bottlenecks.

  2. The study includes conventional detection tasks, held-out counting, training-stage ablations, and repeated-seed statistics. The explicit coordinate restoration and ownership rules make the proposed workflow understandable.

Critical weaknesses
  1. The paper motivates dense perception but does not establish why learned VLM recursion is preferable to training-free divide-and-conquer or specialized perception systems at comparable cost. LVLM-Count already addresses decomposition and duplicate counting, while DC2 already uses recursive partitioning. The proposed increment is chiefly a learned policy combined with established RL, deterministic merging, and policy distillation. Appendix B.2 describes relevant predecessor reproductions without reporting their numerical results or matched inference budgets. Consequently, the distinctive contribution and accuracy/efficiency advantage remain insufficiently established.

  2. The main experimental limitation is protocol completeness, rather than unfamiliar benchmarks. Appendix B.1 retains author-confirmation notes for aerial-box conversion, counting-composite construction, and calibration/evaluation separation. Table 5 reports more retained SOD trajectories than training images despite a top-one-per-image rule, and direct-SFT results conflict across tables. These issues affect whether the central comparisons are valid. The lack of a controlled base-teacher removal or alternative-consolidation comparison also prevents attributing capability retention specifically to the proposed teacher arrangement.

Questions for authors
  1. Please report LVLM-Count/DC2 and a suitable specialized perception baseline at comparable inference budgets, including average and tail call counts or latency. What benefit specifically requires the learned recursion policy?

  2. Please supply the actual split manifests and calibration IDs, confirm evaluation disjointness, and reconcile the SOD trajectory counts and direct-SFT table entries.

  3. Which action schema was executed: selected quadrants or all quadrants? Why choose the Thinking initialization, and how sensitive is the method to an alternative initialization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.