acceptodds
Withdrawn from ICLR 2027

Generalizing VLMs to Extreme Domains: Benchmarking, Reward-Driven Post-Training, and Scaling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern vision-language models (VLMs) appear general because a single autoregressive interface can solve many visual tasks. Yet task breadth does not imply robustness across visual worlds: current evidence is drawn largely from natural imagery resembling web-scale pretraining data. We investigate whether unified VLM perception survives extreme shifts in image formation and visual statistics. To make this question measurable, we introduce CrossVLM-Bench, a benchmark for both post-training and evaluation across remote sensing, industrial inspection, agriculture, and underwater imagery. It contains 7,362 images and 66,258 image–task pairs, with every image supporting nine tasks spanning recognition, localization, segmentation, grounding, reasoning, and region-level description. Its strictly image-disjoint splits combine human-annotated source geometry with human-verified task and language annotations. Evaluation across four VLM families reveals a systematic breakdown that extends beyond recognition to spatial precision, object-set completeness, and language–region grounding. Target-domain supervised fine-tuning (SFT) partially recovers these capabilities, but optimizes the likelihood of a particular response serialization rather than the correctness of the decoded perceptual outcome. We therefore propose CD-GRPO, a reward-driven post-training framework that recasts extreme-domain adaptation as structured outcome optimization. CD-GRPO verifies complete decoded predictions using task-native semantic and geometric criteria while enforcing valid, complete, and nonredundant output structure. On Qwen3-VL-8B, CD-GRPO raises the average score across four domains and nine tasks from 25.7 to 48.5, exceeding SFT by 6.8 points and the strongest recent perception-oriented RL baseline by 7.7 points under equivalent protocols. Scaling analyses across model capacity, target-domain data, and test-time adaptation further characterize when additional resources translate into domain robustness. Together, our results establish extreme-domain robustness as a distinct axis of VLM generality and structured outcome optimization as an effective route toward it. We will release the benchmark, model, and code.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Vision-Language Models, Cross-Domain Generalization, Multi-task Perception

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KuDmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the robustness of VLMs under extreme visual domains. The authors introduce CrossVLM-Bench covering remote sensing, industrial inspection, agriculture and underwater imagery. The paper also shows several degradation of representative VLMs on these domains. To address the gaps, the authors propose CD-GRPO, a two-stage posting training framework consisting SFT and CD-GRPO for adapting VLMs to extreme visual domains. Experiments across several VLMs and domains show that CD-GRPO consistently outperforms both original models and SFT.

Critical strengths
  1. The authors study the generalization ability of VLMs in real-world or extreme visual domains, which is a worthwhile direction. The paper also addresses some meaningful problems and shows detailed and comprehensive experiments.

  2. Althgouh the novelty of CD-GRPO is limited as it mainly involves task-specific reward design and an additional structural reward, the experiments are comprehensive and the ablation studies clearly demonstrate the effectiveness of CD-GRPO.

Critical weaknesses

The work constructs the CrossVLM-Bench to evaluate the capabilities of VLMs in extreme domains and proposes CD-GRPO to address the resulting performance gap. Although many models show degradation on CrossVLM-Bench, I would expect to see a deeper analysis of why these models fail and how the benchmark can provide guidance for the development of next-generation models, rather than introducing a new dataset and using its training data to do post-training.

Questions for authors
  1. The authors mention that CrossVLM-Bench is supposed to figure out whether domian shift failures share common causes or whether they can be recovered through a unified framework. It's good to see CD-GRPO uses the same reward decomposition to solve the failures across extreme domains, but it does not show that these failures arise from a shared cause. For example, whether the failures in remote sensing and underwater imagery stem from the same type of representation failure, and whether the degradation in detection and segmentation is caused by a shared bottleneck?

  2. Table 2 (b) shows the effect of the structural reward in CD-GRPO, but the authors also use samples whose SFT accuracy lies within intermediate range around 0.5. Can authors provide the effect of the sample-filter strategy?

  3. If the authors can clarify my concerns above, I will raise my rating.

4 comments
AuthorsOpenReview ↗

Response to Reviewer KuDm (Part-1)

Q1. What do the failures teach us?

Response:

We agree that introducing a dataset and improving its scores is not sufficient. Our experiments reveal two limitations that should guide model development: incomplete visual coverage and insufficient supervision of structured predictions.

(1) The errors persist beyond learning a valid response format. Appendix E.4, Table 20 audits the same 900 predictions per model:

Error typeSFT (%)CD-GRPO (Ours) (%)
Malformed output8.63.1
Repetition/over-prediction12.46.8
Missed small/dense objects21.710.2
Spatial localization error18.39.5
Incorrect referring target15.95.4

After SFT, missed small/dense objects and localization errors remain more frequent than malformed outputs. This suggests that reliable formatting is only one part of the problem: models must also discover the required targets and ground them precisely. These remain substantial errors even after CD-GRPO.

(2) The reward ablation tests whether supervising complete predictions helps. Table 2(b) reports:

ObjectiveDIORNEU-DETPlantSegUODD
Format + accuracy37.250.429.955.8
CD-GRPO (Ours)41.656.233.762.3

The gains indicate that accuracy rewards leave useful constraints unexpressed. Correct individual boxes do not guarantee sufficient coverage, nonredundant instances, or consistent counts. This supports explicitly supervising those relationships alongside improving visual perception.

For future models, the implication is to distinguish target discovery, localization, and output consistency rather than treat them as one score. CrossVLM-Bench supports this analysis through complementary annotations on the same images. We will foreground this diagnostic role and connect the existing analyses more clearly to these development priorities.

AuthorsOpenReview ↗

Response to Reviewer KuDm (Part-2)

Q2. Do the domains and tasks share one failure cause?

Response:

No, our results do not establish one representation-level cause across domains. A shared reward decomposition demonstrates a useful adaptation principle, not a shared origin of the errors. We will correct the wording that conflates these claims.

(1) Our additional data analysis shows that the domains present different challenges:

DomainSmall objects (%)Occluded objects (%)
Remote sensing68.414.8
Industrial inspection41.28.6
Agriculture28.331.7
Underwater35.838.9

For example, remote sensing contains a much larger proportion of small objects, while underwater imagery has greater occlusion and different imaging degradations. These observations motivate different visual improvements; they do not support treating the domains as interchangeable.

(2) Detection and segmentation share spatial localization in our pipeline, but segmentation additionally depends on mask conversion. Appendix E.3, Table 19 measures these stages separately:

MethodBox IoU before SAMPoint similarity before SAMMask mIoU after SAM
SFT36.356.327.7
CD-GRPO (Ours)38.788.548.7

Our additional reference-prompt experiment further tests what remains recoverable with the same SAM:

Prompt sourceDIORNEU-DETPlantSegUODD
SFT predictions39.655.631.254.3
CD-GRPO predictions48.260.040.575.2
Reference point54.067.750.778.5
Reference box61.274.257.483.2
Reference box + point65.478.161.686.1

Each entry averages the four segmentation tasks within a domain. Accurate reference prompts substantially improve masks, identifying upstream localization as one remaining limitation. Performance is still imperfect with reference prompts, so mask generation also remains a limitation. This is more specific than attributing detection and segmentation degradation to one common bottleneck.

(3) We will narrow the conclusion accordingly. The evidence supports recurring prediction failures and a reward framework that helps across domains. It does not establish identical internal causes. The benchmark provides a way to investigate these dependencies; we should not present that research opportunity as an analysis already completed.

AuthorsOpenReview ↗

Response to Reviewer KuDm (Part-3)

Q3. How much does sample filtering contribute?

Response:

(1) We select examples with SFT scores around 0.5 to prioritize partially solved cases. Near-perfect examples leave limited room for improvement, while near-zero examples may be difficult to learn from within a small rollout budget. The midpoint is a practical heuristic, not a theoretically optimal threshold; the scores can reflect partial geometric correctness rather than success probability.

(2) Inspired by your comment, we compared this strategy with an equal-size random subset. Both groups used the same SFT checkpoint, annotation-quality checks, rewards, hyperparameters, and update budget.

RL sample selectionDIORNEU-DETPlantSegUODD
Equal-size random subset36.951.429.856.7
Intermediate-difficulty subset41.656.233.762.3
Gain from filtering+4.7+4.8+3.9+5.6

Each value is the nine-task mean within a domain. Filtering contributes materially to the final performance. We will report it as a separate part of the training recipe, rather than attribute the full improvement to reward design.

Dear Reviewer, as the discussion deadline approaches, we would appreciate hearing whether you have any remaining questions or concerns about our paper or rebuttal. We would be happy to provide further clarification or additional information. Thank you again for your time and thoughtful feedback!

Reviewer_6uQUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the robustness of vision-language models under domain shifts involving remote sensing, industrial inspection, agriculture, and underwater imagery. It introduces CrossVLM-Bench, containing 7,362 images and 66,258 image–task pairs across nine tasks, including detection, counting, grounding, segmentation, and region captioning. The paper evaluates four VLM families and reports substantial performance degradation without target-domain adaptation.

To improve performance, the authors propose CD-GRPO, a two-stage post-training approach that first applies supervised fine-tuning and then uses Group Relative Policy Optimization with rewards for output format, task-specific accuracy, and structural integrity. Experiments report improvements over direct inference, SFT, and three perception-oriented reinforcement-learning baselines. The paper also studies model capacity, training-data scale, and test-time adaptation.

Critical strengths
  1. CrossVLM-Bench offers a unified evaluation of nine tasks using image-disjoint training and test splits.

  2. The experiments cover multiple VLM families, and the reward ablation provides evidence that structural constraints improve dense predictions.

Critical weaknesses
  1. The benchmark lacks a clear unifying problem definition. Remote sensing, industrial inspection, agriculture, and underwater imagery differ substantially in sensors, objects, scale, and imaging conditions. The paper labels all four as “extreme domains” but does not define extremeness quantitatively or identify a shared failure mechanism that CD-GRPO specifically addresses. Without such analysis, the benchmark appears to aggregate several unrelated domain-specific datasets.

  2. The necessity and competitiveness of the VLM approach are not established. Most tasks could be handled directly by specialist detectors, segmenters, or visual foundation models. However, the paper compares only against VLMs and VLM-oriented RL methods. It omits specialist detection and segmentation baselines, as well as a tool-augmented VLM that calls external detection or segmentation modules. Autoregressive generation itself introduces malformed outputs, duplication, and cardinality errors that CD-GRPO then corrects. The paper must show what advantage the VLM interface provides over simpler, direct prediction pipelines.

  3. SAM is a major uncontrolled component in the segmentation results. The VLM predicts boxes and points, while SAM produces the evaluated masks. SAM can struggle with small, crowded, low-contrast, deformable, or irregular objects—properties common in the selected domains. The paper provides no oracle experiment using ground-truth prompts, no comparison with alternative mask decoders, and no separation of VLM prompt errors from SAM mask-generation errors. Therefore, the segmentation gains cannot be attributed cleanly to CD-GRPO.

  4. The method demonstrates supervised specialization rather than robust cross-domain generalization. Each model is independently adapted using labeled images from the target domain. The test-time adaptation experiments also start from these target-domain-trained models. No zero-shot, unlabeled, source-free, or cross-domain transfer setting is evaluated. Furthermore, CD-GRPO changes the optimization objective but introduces no network mechanism for the claimed underlying problems, such as small-object perception, unusual viewpoints, low contrast, or irregular boundaries. It is therefore unclear whether it improves visual representations or mainly improves output formatting and completeness.

Questions for authors
  1. What measurable visual challenge is shared by the four domains, and how does CD-GRPO address that challenge rather than merely improving output serialization?

  2. How does CD-GRPO compare with specialist detection and segmentation models, and with a VLM that calls those models as external tools under matched training and inference budgets?

  3. What is the segmentation upper bound when SAM receives ground-truth boxes and points? Can the authors separate errors caused by VLM-generated prompts from errors caused by SAM?

  4. Can the method adapt using no labeled target-domain training images, and can one jointly adapted model generalize across all four domains?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 6uQU (Part-1)

Q1. What unifies the four domains?

Response:

(1) The shared problem is adapting a VLM to multiple perception tasks under substantial changes in visual distribution. We do not assume that remote sensing, industrial inspection, agriculture, and underwater imagery share one physical degradation or representation failure. They share an application requirement: recognizing, locating, counting, segmenting, and describing visual targets through the same instruction-conditioned interface.

CrossVLM-Bench makes this comparison possible by providing nine tasks on every image, with image-disjoint training and test splits. “Extreme” describes these specialized visual settings relative to conventional natural imagery; it is not a calibrated distance from an unknown pretraining distribution. We will clarify this usage.

(2) Our additional measurements describe concrete challenges rather than relying on the domain labels alone:

DomainImagesClassesSmall objects (%)Occluded objects (%)
Remote sensing2,1002068.414.8
Industrial inspection1,800641.28.6
Agriculture2,10011528.331.7
Underwater1,362335.838.9

The challenges are not identical: remote sensing contains many small targets, while agriculture and underwater imagery have substantial occlusion. Our claim is that these different settings benefit from a shared adaptation framework, not that their failures have one common cause.


Q2. Why a VLM, and where are the specialist comparisons?

Response:

(1) No, a VLM is not necessary for every individual task. Our motivation is to study one model that supports geometric, referring, reasoning, and captioning tasks. A detector or segmenter alone does not provide that full interface, although it remains a valid choice when only its native task is needed.

(2) The submission does include specialist comparisons. Appendix D.1, Table 12 adapts DETR and MaskFormer on the same target-domain images and evaluates them on identical test splits:

TaskMethodDIORNEU-DETPlantSegUODD
Detection APDETR12.722.66.022.4
Detection APCD-GRPO (Ours)31.452.115.655.6
Segmentation mIoUMaskFormer17.618.111.121.8
Segmentation mIoUCD-GRPO (Ours)44.848.832.169.2

These results show that supporting nine tasks need not sacrifice performance against the evaluated specialists. We will make this comparison more visible. It controls target-domain images, but not total pretraining resources or computation, and we will not describe it as a compute-matched comparison.

(3) We also compare with externally assisted VLM pipelines. VisionReasoner and OneThinker use SAM-based mask generation, as does our segmentation pipeline. Table 1 reports their reproduced results with the same Qwen3-VL-8B backbone, target-domain training data, and evaluation protocol:

MethodDIORNEU-DETPlantSegUODD
VisionReasoner37.144.828.651.7
OneThinker38.647.625.950.9
CD-GRPO (Ours)41.656.233.762.3

Each entry is the nine-task mean. These comparisons include external mask-generation modules, but do not constitute a budget-matched evaluation of autonomous tool selection. We will distinguish these settings explicitly.

AuthorsOpenReview ↗

Response to Reviewer 6uQU (Part-2)

Q3. Is SAM an uncontrolled source of improvement?

Response:

No, SAM is held fixed across the compared methods. However, the submitted analysis did not fully establish how much performance remained recoverable through better prompts.

(1) Appendix E.3, Table 19 evaluates spatial prompts before SAM and masks afterward:

MethodBox IoU before SAMPoint similarity before SAMMask mIoU after SAM
SFT36.356.327.7
CD-GRPO (Ours)38.788.548.7

Because SAM and its inference configuration are unchanged, these improvements cannot be explained by replacing or retraining the mask decoder. Better localization accompanies better final masks.

(2) Following your comment, we additionally evaluated SAM with reference spatial prompts:

Prompt sourceDIORNEU-DETPlantSegUODD
Base-model predictions28.130.526.036.2
SFT predictions39.655.631.254.3
CD-GRPO predictions48.260.040.575.2
Reference point54.067.750.778.5
Reference box61.274.257.483.2
Reference box + point65.478.161.686.1

Each value averages semantic, instance, referring, and reasoning segmentation within a domain. Reference box–point prompts improve over CD-GRPO predictions by 17.2, 18.1, 21.1, and 10.9 points, respectively. Thus, substantial room remains in upstream localization. Reference prompts also do not produce perfect masks, showing that prompt-to-mask conversion remains imperfect; these results are a reference performance, not a strict mathematical upper bound.

(3) Our aim is to evaluate VLM adaptation under a controlled segmentation pipeline. We therefore use the same frozen SAM checkpoint and inference configuration for every method, holding mask generation fixed. Appendix E.3, Table 19 evaluates localization before SAM alongside the resulting mask quality. CD-GRPO improves both spatial-prompt quality and final masks without changing the decoder. This supports improved upstream localization under the shared SAM pipeline.


Q4. Better perception or merely better formatting?

Response:

The gains are not confined to formatting, although our experiments do not identify a specific internal representation mechanism.

(1) Appendix E.4, Table 20 audits the same 900 predictions per model:

Error typeSFT (%)CD-GRPO (Ours) (%)
Malformed output8.63.1
Repetition/over-prediction12.46.8
Missed small/dense objects21.710.2
Spatial localization error18.39.5
Incorrect referring target15.95.4

The reductions in missed objects, localization errors, and incorrect referring targets concern perceptual predictions, not just valid serialization. Table 19 independently measures improved spatial prompts before mask generation.

(2) Table 2(b) further tests the contribution of structural supervision beyond format-and-accuracy rewards:

ObjectiveDIORNEU-DETPlantSegUODD
Format + accuracy37.250.429.955.8
CD-GRPO (Ours)41.656.233.762.3

The added reward checks category coverage, count–box consistency, valid spatial records, and excessive predictions. These constraints improve complete predictions while retaining task-native geometric and semantic accuracy.

We do not claim that this introduces a dedicated mechanism for low contrast, unusual viewpoints, or irregular boundaries. It changes the supervision used to adapt the existing model. We will distinguish observed improvements in prediction quality from claims about how its internal visual representations change.

AuthorsOpenReview ↗

Response to Reviewer 6uQU (Part-3)

Q5. Target-domain supervision and unseen-domain transfer.

Response:

(1) Yes, the main setting is supervised target-domain adaptation. Each model uses labeled target-domain training images and is evaluated on held-out images. The base-model results provide zero-shot evaluation, but our submitted training experiments do not establish transfer to an unseen domain. We will make this scope explicit.

(2) CD-GRPO does not currently perform unsupervised target-domain RL or source-free adaptation. Its accuracy rewards require reference targets. The additional inference experiments distinguish what can be achieved without updating parameters:

ModelInference strategyDIORNEU-DETPlantSegUODD
BaseDirect inference26.828.219.328.4
BaseFive-shot ICL29.431.721.832.1
BaseTransductive refinement30.132.822.633.0
SFTDirect inference35.052.827.351.7
SFTFive-shot ICL37.154.329.253.6
SFTTransductive refinement37.655.129.854.4
CD-GRPO (Ours)Direct inference41.656.233.762.3
CD-GRPO (Ours)Five-shot ICL43.857.435.163.9
CD-GRPO (Ours)Transductive refinement43.358.335.464.0

ICL uses labeled demonstrations. Transductive refinement uses predictions from the same test image as cross-task feedback, without test labels or parameter updates. Its gains on the base model show that label-free inference refinement is possible.

(3) We do not train one model jointly on all four domains. Our objective is a shared adaptation recipe that can be applied independently to a target domain, starting from a broadly pretrained VLM. This setting reflects applications where practitioners have data from their own domain but need not collect unrelated industrial, agricultural, aerial, and underwater data. The transferable prior comes from the pretrained model; target-domain supervision then adapts that prior to the application. Joint training would instead test cross-domain knowledge sharing and interference under pooled supervision. That is a valid question, but it is not required to establish whether the same adaptation recipe works independently across domains. We will distinguish a shared adaptation framework from a single jointly trained model.

Reviewer_m4NBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles the problems of VLMs not generalizing well to domains underrepresented during training (extreme domains). It curates a benchmark for 4 example domains on various tasks, and demonstrates that a combination of SFT and RL finetuning can greatly improve performance on various tasks. One novelty is the reward used in GRPO, which encourages format validity, performance on the task specific metric, and consistency.

Critical strengths
  • Identifies and tackles a valid problem; whether VLMs generalize to domains beyond the “natural distribution” they were trained on. This does seem to be an important concern and problem. The benchmark released could be useful for this.
  • The method does seem to work, and uniformly and substantially improves performance in all settings.
Critical weaknesses
  • I don’t think anything in this paper demonstrates that these problems are unique to “extreme visual domains”. It has been well established that older VLMs (when naively prompted) are weak on natural images as well, specifically on precise localization (like object detection and segmentation) [1-3]. So while generalization to extreme domains is an important question, nothing in this paper establishes that this is a problem all of these models face because of the domain itself, rather than task difficulty.
  • In Table 1, many baselines score even lower than CD-GRPO after only the SFT stage. Were all of these methods trained after the SFT stage? If so, is the claim that the baselines degrade performance?

[1] Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

[2] DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLM

[3] How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks

Questions for authors

My first two questions are related to addressing the critical weaknesses:

  1. Can you demonstrate that this problem is unique to extreme visual domains, for all of these models (specifically for precise tasks, like segmentation or object detection when there are many small boxes)? Because if not, the contribution reduces to finetuning VLMs on tasks they underperform in.
  2. If the baselines are not trained after the SFT stage, then it isn’t a fair comparison. If they are trained after, why do they degrade performance (are the hyperparameters tuned properly)?

Other questions:

  1. Can you clarify how SFT is done for segmentation? Where do pixel-level ground-truth masks come from in domains without segmentation masks?
  2. Why is the clipping coefficient in GRPO so low (L346). Won’t this effectively amount to only doing a single REINFORCE update per group?
  3. Related to critical weakness 1, what is it about this method that makes it specifically suited for extreme domain generalization?
8 comments
AuthorsOpenReview ↗

Response to Reviewer m4NB (Part-1)

Q1. Domain shift or task difficulty?

Response:

No, we do not claim that localization failures are exclusive to extreme domains. We agree that the cited studies demonstrate difficulties on natural images. Our motivation is to understand whether VLMs can reliably support perception in specialized applications, and how to adapt them when they cannot. We will clarify this distinction in the paper.

(1) We use “extreme” relative to conventional natural imagery. Remote sensing introduces overhead viewpoints and small targets; industrial inspection requires recognizing subtle defects; agriculture involves fine-grained symptoms and occlusion; underwater imagery suffers from color distortion and reduced visibility. Our additional statistics characterize these challenges:

DomainSmall objects (%)Occluded objects (%)
Remote sensing68.414.8
Industrial inspection41.28.6
Agriculture28.331.7
Underwater35.838.9

We do not suggest that every task is harder in these domains. We constructed CrossVLM-Bench to study their shared perception requirements through nine tasks, providing training data for adaptation and image-disjoint test data for evaluation.

(2) Following your comments, we also examined whether stronger inference strategies could recover performance without CD-GRPO:

Model and inference procedureDIORNEU-DETPlantSegUODD
GPT-4V + SoM [1]29.838.520.942.1
GPT-4V + DetToolChain [2]33.443.724.647.8
GPT-4o + task-specific prompt chaining36.948.528.753.4
CD-GRPO (Ours), Qwen3-VL-8B41.656.233.762.3

Each value is the nine-task mean within a domain. Since these systems use different backbones and inference resources, we additionally compared inference strategies using the same Qwen3-VL-8B SFT checkpoint:

MethodDIORNEU-DETPlantSegUODD
SFT35.052.827.351.7
SFT + five-shot ICL37.154.329.253.6
SFT + transductive refinement37.655.129.854.4
CD-GRPO (Ours)41.656.233.762.3

Stronger inference helps, but does not close the gap in these experiments.

(3) We recognize that these comparisons do not fully separate domain shift from task difficulty. We will therefore narrow the claim: our experiments establish the value of structured reward optimization for adaptation across the four evaluated domains, rather than proving that their failures are unique or caused by domain shift alone.

AuthorsOpenReview ↗

Response to Reviewer m4NB (Part-2)

Q2. SFT initialization and baseline degradation.

Response:

(1) Additional training does not guarantee improvement. Table 2(a) shows that extending SFT from one to two epochs lowers performance in every domain. A plausible explanation is overfitting to the limited target-domain supervision: further token-level optimization can reinforce training-specific response patterns without improving object coverage, localization, or prediction consistency on unseen images. In contrast, one SFT epoch followed by one CD-GRPO epoch improves every domain, supporting the value of changing the optimization objective rather than simply extending imitation training.

Training procedureEpochsDIORNEU-DETPlantSegUODD
SFT135.052.827.351.7
SFT-Long233.650.925.849.4
GRPO without SFT235.953.728.652.8
SFT + CD-GRPO (Ours)1 + 141.656.233.762.3

Each score is the nine-task mean within a domain. These results show that continuing optimization can reduce held-out performance, while changing the post-training objective can improve it.

(2) The reward ablation provides further evidence that the objective matters. Table 2(b) reports:

Reward objectiveDIORNEU-DETPlantSegUODD
Format only12.417.810.019.2
Format + structure17.123.613.725.9
Format + accuracy37.250.429.955.8
CD-GRPO (Ours)41.656.233.762.3

Optimizing format or structure without perceptual accuracy is insufficient. Accuracy-based rewards recover substantial performance, and structural supervision adds another 3.8–6.5 points. Thus, applying RL after SFT is not sufficient by itself; the optimization signal must support the evaluated tasks.

(3) For our baselines and ablations, we keep the applicable training hyperparameters fixed to isolate the effect of each component; CD-GRPO receives no additional method-specific hyperparameter tuning. For the competing RL methods, we reproduce their complete post-training pipelines using the same Qwen3-VL-8B base model, target-domain training images, nine-task output space, and test splits, while retaining each method's training procedure and method-specific components.


Q3. Failures beyond extreme domains.

Response:

No, we have not shown that these failures occur only in extreme domains. We agree that this distinction is important and will revise claims that imply exclusivity.

(1) Our motivation is to study whether one VLM can reliably support multiple perception tasks in specialized applications. These settings differ in viewpoint, scale, texture, and imaging conditions, but still require detection, segmentation, counting, and grounding. CrossVLM-Bench provides nine tasks per image, with separate training and test splits, to support both evaluation and adaptation. The existence of similar errors on natural images does not answer how effectively these capabilities transfer to the selected domains.

(2) Our methodological contribution concerns how adaptation is optimized, beyond simply supplying target-domain data. Table 2(b) compares format-and-accuracy GRPO with the addition of structural supervision:

ObjectiveDIORNEU-DETPlantSegUODD
Format + accuracy GRPO37.250.429.955.8
CD-GRPO (Ours)41.656.233.762.3
Gain+4.4+5.8+3.8+6.5

Each value is the nine-task mean within a domain. Under the same data-selection and training setting, structural supervision improves every domain. This supports the proposed objective without requiring the underlying failures to be unique to these domains.

AuthorsOpenReview ↗

Response to Reviewer m4NB (Part-3)

Q4. Segmentation supervision and reference masks.

Response:

(1) SFT teaches the VLM to predict spatial prompts, not pixel masks. The target responses contain boxes, points, and task-dependent instance records, learned through token-level supervision (Sections 3.1–3.2). At inference, the same frozen SAM converts the predicted prompts into masks for all methods.

(2) We retain source masks where available. For datasets without masks, SAM generates candidate masks from the source geometric annotations, which experienced annotators then review. This reduces the considerable cost of drawing masks from scratch while retaining human oversight. The reviewed reference masks are fixed during evaluation and are not generated from the evaluated VLM’s predictions.

(3) Appendix A.3 reports an independent audit of 120 images, with all nine tasks reviewed by two annotators and disagreements resolved by a third. This provides evidence about overall task-annotation quality, but does not separately measure pixel-boundary accuracy. We will explicitly document the mask-review procedure and distinguish SAM’s role in annotation from its role in inference.


Q5. Clipping coefficient.

Response:

We apologize for the reporting error. After checking our training code, we confirm that the clipping coefficient used in the experiments was 0.2; 0.01 is the KL regularization coefficient and was mistakenly reported as the clipping coefficient. We will correct Section 4.1 and thank the reviewer for identifying this error.


Q6. Why is CD-GRPO suitable for extreme-domain adaptation?

Response:

(1) Its suitability comes from the prediction requirements, rather than a domain-specific architecture. Across these settings, the model must identify unfamiliar targets, localize them, and return complete object sets. CD-GRPO directly rewards semantic and geometric accuracy, while checking category coverage, count–box consistency, valid spatial records, and excessive predictions. The task-specific metrics preserve what correctness means for each task; the same reward decomposition is retained across domains.

(2) The reward ablation supports this design. Table 2(b) compares format-and-accuracy GRPO with the complete objective:

Reward objectiveDIORNEU-DETPlantSegUODD
Format + accuracy37.250.429.955.8
CD-GRPO (Ours)41.656.233.762.3
Gain+4.4+5.8+3.8+6.5

Each value is the nine-task mean within a domain. The consistent gains show that structural supervision adds value beyond metric-aligned accuracy optimization across all four settings.

(3) The improvements extend beyond formatting. Appendix E.4, Table 20 reports the following errors on the same 900 predictions per model:

Error typeSFT (%)CD-GRPO (Ours) (%)
Missed small/dense objects21.710.2
Spatial localization error18.39.5
Incorrect referring target15.95.4
Repetition/over-prediction12.46.8

These results support improved coverage and grounding alongside output consistency.

We will clarify the scope of “generalization” and position CD-GRPO as a shared reward framework for domains adaptation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.