acceptodds
Under review as a conference paper at ICLR 2027

DevoSeg: Budget-Aware Self-Evolving Semantic Segmentation under Visual Degradation

Abstract

Semantic segmentation under visual degradation is challenging because class cues and object boundaries deteriorate differently across images. Effective inference therefore calls for decisions that adapt to the current image and improve through experience from earlier cases. We introduce DevoSeg, a self-evolving framework that learns how to deploy a frozen segmentor from completed segmentations. It (1) estimates the complementary, Direct-relative opportunity of policy sets and assigns an image-dependent execution budget; (2) compares masks from selected policies with an independently produced Direct mask using post-execution image–mask evidence; and (3) organizes observed effects and costs as contextual knowledge, with a disjoint-group validation rule for proposed decision-state updates. Separating policy acquisition from mask selection lets past outcomes guide which alternatives to explore while the final decision uses the predictions actually obtained. On 200 ACDC images, DevoSeg’s mask selector improves pooled Cityscapes-19 mIoU by 2.084 percentage points over Direct. Image-adaptive acquisition executes 30.1% fewer candidate policies and reduces their collection time by 23.3% relative to fixed eight-policy acquisition.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
semantic segmentation, self-evolution, experience-guided learning, policy optimization, token-efficient inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Xe6fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the task of semantic segmentation under visual degradation and proposes DevoSeg, a self-evolving framework that learns how to deploy a frozen segmentor from completed segmentations. Its core mechanism comprises three key aspects: (1) estimating the complementarity of the strategy set and allocating an adaptive execution budget for each image; (2) utilizing post-execution image-mask evidence to contrast the masks generated by the selected strategies against independently generated direct-inference masks; and (3) compiling the observed effects and costs into contextual knowledge, and proposing a disjoint-group verification rule for decision state updates. Experimental results demonstrate that the mask selector of DevoSeg improves the mIoU on the Cityscapes dataset by 2.084 percentage points.

Critical strengths

The mathematical formulation in the methodology section is academical.

Critical weaknesses
  1. To ensure a fair comparison in Table 4, SAM 3 should be fine-tuned or adapted using the identical set of 920 annotations to compare its performance against DevoSeg.

  2. The paper is motivated by local, category-dependent failure issues, stating that "an image may preserve the overall layout but lose the boundaries of thin or partially occluded objects." However, the evaluation metric adopted in this paper is Image IoU, which is dominated by large-object categories and is therefore largely insensitive to the fine structures and boundary effects the paper claims to address. Tables 1, 2, and 4 should provide per-category (class-wise) results to evaluate whether the empirical evidence supports the objectives argued in the introduction.

  3. The dataset usage in the experimental evaluation is non-standard. While the ACDC validation set contains 406 images and the test set contains 2,000 images, the experimental section only evaluates on 200 ACDC images, and the paper lacks any explanation for this discrepancy.

  4. The literature review is insufficient. There is currently a substantial body of literature in the field of semantic segmentation under adverse weather conditions, such as [1-5]. Both the related work section and the comparative experiments in Table 4 lack descriptions of and comparisons with these relevant studies.

[1] Brüggemann D, Sakaridis C, Truong P, et al. Refign: Align and refine for adaptation of semantic segmentation to adverse conditions[C]//2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2023: 3173-3183.

[2] Xu Q, Wu Q, Hongtao L, et al. CroPe: Cross-Modal Semantic Compensation Adaptation for All Adverse Scene Understanding[J]. Advances in Neural Information Processing Systems, 2026, 38: 136226-136252.

[3] Lee S, Kim N, Kim S, et al. Frest: Feature restoration for semantic segmentation under multiple adverse conditions[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2024: 1-18.

[4] Li F, Gong Z, Deng Y, et al. Parsing all adverse scenes: Severity-aware semantic segmentation with mask-enhanced cross-domain consistency[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(12): 13483-13491.

[5] Pan Y, Sun R, Li W, et al. Exploring weather-aware aggregation and adaptation for semantic segmentation under adverse conditions[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 13952-13962.

Questions for authors

Q1: What are the experimental results in Table 4 when SAM 3 is fine-tuned or adapted using the same set of 920 annotations?

Q2: What are the per-category (class-wise) results in Tables 1, 2, and 4?

Q3: What is the rationale behind selecting only 200 ACDC images for the evaluation?

Q4: What are the advantages of the proposed method compared to existing literature on semantic segmentation under adverse weather conditions, and how does it compare in terms of performance?

9 comments
AuthorsOpenReview ↗

1. Segmentor adaptation with the same annotations

Thank you for recognizing the mathematical formulation of the method and for your specific suggestions on comparisons with the same annotations, class-wise metrics, and adverse-weather segmentation research. To make the adaptation and class-wise evaluations reproducible, we will release the implementation, same-annotation adaptation configurations, class-wise and boundary evaluation scripts, data lists, and model sources upon acceptance.

1. Results after adapting SAM 3 with the same 920 annotations

We adapt SAM 3 using the same 920 annotations used to learn the decision modules, then fix that checkpoint and run Direct and DevoSeg separately. Adaptation keeps the image and text backbones frozen and updates the remaining parameters. This compares the use of the same annotations for different learning targets and tests whether external operation selection and candidate comparison can still improve an adapted segmentor.

Segmentor settingDecision methodMean per-image IoU (%)Dataset-level mIoU (%)
Frozen SAM 3Direct53.44863.870
Frozen SAM 3DevoSeg54.81266.008
SAM 3 adapted with the same 920 annotationsDirect67.78075.180
SAM 3 adapted with the same 920 annotationsDevoSeg67.86577.413

Adapting SAM 3 with the same annotations substantially improves the segmentor itself. With this stronger segmentor, DevoSeg further increases dataset-level mIoU from 75.180% to 77.413%, a gain of 2.233 percentage points, with a 95% scene-paired confidence interval of [0.188, 4.654]. Mean per-image IoU increases from 67.780% to 67.865%, with a difference interval of [−0.137, 0.295]. These results show how segmentor adaptation and external decision learning can work together: the former improves the underlying predictions, while the latter learns how to make use of candidates produced by the operations.

Using annotations for segmentor adaptation and for decision learning are therefore complementary directions. Reviewer bVbt describes this positioning as “policy learning as a complement to adapting segmentor weights”. The same-annotation comparison above provides a concrete result: the external workflow further improves dataset-level mIoU after the segmentor has been adapted.

AuthorsOpenReview ↗

2. Class-wise metrics and fine-structure evaluation

2. Class-wise results for Tables 1, 2, and 4 and evaluation of fine structures

Supplementary Table 1 below provides the complete 19-class region IoU and Boundary IoU results for the candidate-input ablations. Supplementary Table 2 provides the corresponding decision-component results, and Supplementary Table 3 provides the results for the other segmentation methods in Table 4 and the appendix. We refer to the paper’s Image IoU as “mean per-image IoU” to make its image-level averaging explicit. It first averages classes with nonempty union within each image and then averages across images:

ImageIoUi=1∣Ci∣∑c∈CiTPicTPic+FPic+FNic,\mathrm{ImageIoU}_i=\frac{1}{|C_i|}\sum_{c\in C_i}\frac{TP_{ic}}{TP_{ic}+FP_{ic}+FN_{ic}},

MeanImageIoU=1N∑i=1NImageIoUi.\mathrm{MeanImageIoU}=\frac{1}{N}\sum_{i=1}^{N}\mathrm{ImageIoU}_i.

Here, CiC_{i} contains the classes whose prediction–ground-truth union is nonempty in image ii, and TPicTP_{ic}, FPicFP_{ic}, and FNicFN_{ic} are the true-positive, false-positive, and false-negative pixel counts for class cc in that image. Each valid class within an image receives equal weight, regardless of its pixel area. To address your concern about fine structures and boundaries, we additionally report Boundary IoU and provide the complete 19-class results for Tables 1, 2, and 4, evaluating region recognition and contour quality separately.

Dataset-level mIoU is the metric called Pooled mIoU in the paper. For each class, we sum its intersection and union pixel counts across all images and take their ratio, then average the resulting class IoUs with equal class weights. Thus, mean per-image IoU averages scores calculated separately for each image, whereas dataset-level mIoU calculates class scores from counts pooled across the dataset. The revised name makes this distinction explicit; the calculation and reported values are unchanged.

For the full-input Comparator in Table 1, the two boundary metrics are:

Boundary metricDirectFull-input Comparator
Mean per-image Boundary IoU (%)52.53053.480
Dataset-level Boundary IoU (%)59.37261.807

Mean per-image Boundary IoU improves by 0.950 percentage points, with a 99.5% paired interval of [−0.021, 1.960] adjusted for ten comparisons. Results for thin structures and road users are shown below; Supplementary Tables 1 and 2 provide the complete 19-class results for the input ablations and decision-component comparisons, respectively.

ClassDirect Region IoUFull-input Comparator Region IoUDirect Boundary IoUFull-input Comparator Boundary IoU
Pole59.5260.1561.4063.39
Rider19.2547.3536.5356.91
Car82.2289.4169.6982.92

For example, pole Boundary IoU increases from 61.40% to 63.39%; for riders, region IoU and Boundary IoU increase from 19.25% and 36.53% to 47.35% and 56.91%, respectively. Joint decisions in Table 2 also increase car and bus region IoU from 82.22% and 62.66% to 86.20% and 66.34%. The pole and rider results above correspond to the full-input Comparator in Supplementary Table 1. The car and bus results for joint decisions are provided in Supplementary Table 2.

Boundary IoU uses an inner boundary band with a width of 2% of the image diagonal and excludes neighborhoods of invalid labels. The class-support table accompanying Supplementary Table 3 gives the ground-truth pixel count and number of images containing each class, relating each class score to its actual support.

AuthorsOpenReview ↗

3. Evaluation cohort and complete validation results

3. The 200-image component evaluation and the complete validation results

The component comparisons require multiple candidates and multi-step operations for each image, making them computationally demanding. To complete systematic comparisons within the available computational resources, we use 200 ACDC images covering fog, night, rain, and snow, with 50 images per condition. This evaluation cohort was established using early model observations. We also compare Direct and DevoSeg on all 406 images in the official ACDC validation set, with the following results.

MetricDirectDevoSegGain (pp)97.5% scene-paired interval
Mean per-image IoU (%)53.44854.812+1.363[0.734, 2.370]
Dataset-level mIoU (%)63.87066.008+2.138[0.622, 4.683]

On the complete validation set, DevoSeg outperforms Direct on both metrics, with both confidence intervals for the gains above zero. These results support the overall segmentation benefit of the workflow on ACDC. The evaluation uses ACDC images with the Cityscapes 19-class semantic taxonomy throughout.

Reviewer_ZsudOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DevoSeg addresses semantic segmentation under adverse visual conditions by keeping a SAM 3 segmentation backbone frozen while learning decision modules that choose which optional policies to execute and which resulting mask to retain under an image-dependent budget. A Router selects a compatible policy set before execution,while a Comparator ranks executed candidate masks against an independent Direct prediction.Apart from that,an experience-distillation mechanism stores contextual policy effects and costs, with updates gated by validation on disjoint groups. The evaluation uses 200 ACDC images across four weather conditions and reports improvements in mean per-image IoU over Direct, alongside candidate-collection time savings relative to a fixed policy budget.

Critical strengths
  1. The paper formulates a practical decision problem that is often hidden by methods that execute every available refinement policy. Keeping Direct as an available fallback is sensible and it can reduce the risk that a policy-selection system must always replace the baseline prediction.
  2. The evaluation includes paired uncertainty estimates, component ablations, and a measurement of candidate-collection time. These are useful reporting choices, even though the results leave important uncertainty about statistical robustness and total system cost.
Critical weaknesses
  1. The evaluation is narrow: it uses only 200 ACDC evaluation images and a single frozen SAM 3 backbone. More importantly, the evidence is mixed. The confidence interval for the reported mean per-image IoU gain excludes zero, while the interval for pooled mIoU crosses zero. The selected candidate also has negative per-image gain on 23.5% of images. These results warrant a more cautious interpretation of the method's effectiveness and limit the strength of the paper's generalization claims.
  2. The ablations do not demonstrate that the complete Router–Comparator–budget system is reliably better than the simpler Comparator-only variant: the reported difference is relatively small and its confidence interval crosses zero. Router-only and budget-only variants also underperform Direct. Meanwhile, the reported time saving measures candidate collection only and excludes the Comparator and other serving stages. Thus, neither the claimed benefit of self-evolution nor an end-to-end quality–cost advantage is convincingly established. The external comparisons also involve different training-data protocols, which limits their value as direct evidence of superiority.
Questions for authors
  1. DevoSeg’s Comparator ranks candidate masks by their predicted gains over the Direct baseline. Ground-truth gains are available during training but not at inference, forcing the model to infer mask quality from image–mask cues. This raises an identifiability issue: masks with similar low-level statistics may differ sharply in semantic correctness. What assumptions enable reliable quality estimation from the Comparator’s inputs, and how robust are these assumptions under degradation or distribution shift? Unaddressed, this limitation may cause systematic failure on the hard cases where selection matters most.
  2. Appendix C.1 describes four input arms with identical architecture and initialization, each masking distinct visual blocks, without defining the exact mapping between arms and masked components. Please specify each arm precisely and clarify whether masking corresponds to zeroing, removing, or replacing the input region. These details are required for reproducible ablations and proper interpretation of each evidence stream.
  3. The decision modules are trained on labeled ACDC data and evaluated on 200 held-out images. Their core goal, however, is not segmentation itself but learning a meta-rule to select beneficial policies under specific degradation and budget constraints. It remains unclear whether the reported gains reflect transferable selection logic or dataset- and policy-set-specific correlations, given the limited evaluation and fixed backbone. We ask for evidence that the learned acquisition and selection rules generalize to independent datasets, alternative backbones, and unseen degradation combinations.
  4. The budget head selects K∈{1,3,5,8}K \in \{1,3,5,8\}, with Direct as the zero-gain fallback. If all candidate policies are predicted to underperform Direct, can the Router skip optional policy execution entirely, or must candidates still be generated and passed to the Comparator to retain Direct? Please state the decision rule and its implications for redundant computation.
  5. DevoSeg’s Router and budget head select policies and candidate counts from a predefined action space. The reported performance–cost tradeoff is therefore tied to this fixed policy set, cost assignments, and training degradation distribution. It is unclear whether the learned budget policy approximates a general quality–cost optimum or merely works well for this specific pool. Furthermore, fewer executed policies do not guarantee lower end-to-end latency due to hardware, parallelism, memory, and decision-module overheads. Evidence is needed to verify near-optimality under a realistic end-to-end cost model, rather than only the paper’s candidate accounting scheme.
7 comments
AuthorsOpenReview ↗

1. Quality estimation and Direct selection

Thank you for recognizing the practical problem of choosing operations for each image and the design of retaining Direct as an available output. To enable reproduction of these comparisons, we will release the code, input-ablation configurations, policy settings, and complete timing scripts upon acceptance.

1. What the Comparator estimates and how Direct participates in selection

The Comparator learns the change in segmentation quality relative to Direct. Let xx denote an image, m0m_{0} its Direct mask, mjm_{j} a candidate mask, and yy the training annotation. The supervision target is

Δj=Q(mj,y)−Q(m0,y),\Delta_j=Q(m_j,y)-Q(m_0,y),

where QQ is the per-image IoU metric used in the paper. At inference, yy is unavailable. The model instead predicts the relative gain from inputs zjz_{j} constructed from the image, Direct prediction, actual candidate, and route information:

Δ^j=fθ(zj),Δ^0=0.\widehat{\Delta}_{j}=f_{\theta}(z_{j}),\qquad \widehat{\Delta}_{0}=0.

The output is selected as

\begin{cases} m_{0}, & \max_{j}\widehat{\Delta}_{j}\leq 0, \\\\ m_{j^{\star}},\quad j^{\star}=\arg\max_{j}\widehat{\Delta}_{j}, & \max_{j}\widehat{\Delta}_{j} > 0. \end{cases}$$ The evidence includes image–mask correspondence, regional semantics, Direct context, and route information. Masks with similar areas or shapes can still differ in the visual semantics and boundary correspondence of the regions they cover. Reliable estimation therefore depends on these inputs being informative about relative quality and on the learned relationships remaining sufficiently stable in the target setting. The model learns relative quality from this combined evidence, of which low-level mask statistics are only one part. Section 2 tests the contributions of the input evidence, while Section 3 examines transfer when the dataset or segmentation backbone changes. Direct serves as the zero-gain reference in candidate comparison: the system adopts a candidate only when the Comparator predicts an improvement over Direct; otherwise, it retains Direct. Executing an operation therefore does not require replacing the baseline prediction. The 47 cases you identified reflect a discrepancy between predicted gains and actual improvements; in 31 of them, the candidate pool already contains an output that outperforms Direct. This points to a concrete direction for improvement: strengthening the connection between image semantics, local mask changes, and segmentation gains, while improving gain calibration so that the system can better determine when to adopt a candidate and when to retain Direct. To directly evaluate selection, we hold each image’s Direct mask and $K$ generated candidates fixed and change only the final selection method. Random selection and the Comparator receive identical masks, with 32 seeds used for the random baseline. | Candidate count $K$ | Random selection: mean per-image IoU (%) | Comparator: mean per-image IoU (%) | Gain (pp) | Paired gain interval | | --- | --- | --- | --- | --- | | 1 | 55.074 | 55.786 | +0.712 | [0.334, 1.117] | | 3 | 54.659 | 56.215 | +1.556 | [1.039, 2.112] | | 5 | 54.681 | 56.442 | +1.761 | [1.188, 2.374] | | 8 | 54.503 | 56.652 | +2.148 | [1.462, 2.843] | The four budget comparisons use 98.75% paired intervals, jointly resampling images and random seeds. The Comparator improves mean per-image IoU in all four settings. At $K=8$, dataset-level mIoU also increases from 61.127% to 64.879%, a gain of 3.752 percentage points with an interval of [0.270, 6.845]. Because the candidate sets are identical, this comparison directly attributes the improvement to post-execution output selection. On the complete official ACDC validation set, DevoSeg and Direct achieve the following results: | Metric | Direct | DevoSeg | Gain (pp) | 97.5% scene-paired interval | | --- | --- | --- | --- | --- | | Mean per-image IoU (%) | 53.448 | 54.812 | +1.363 | [0.734, 2.370] | | Dataset-level mIoU (%) | 63.870 | 66.008 | +2.138 | [0.622, 4.683] | Mean per-image IoU (Image IoU in the paper) averages IoU over valid classes within each image, then averages the image scores. Dataset-level mIoU (Pooled mIoU in the paper) first pools each class’s intersection and union counts across images, then computes and averages class IoUs. The names distinguish average image-level performance from class-level performance across the dataset; neither the calculations nor the values have changed. The dataset-level mIoU interval in the 200-image evaluation includes zero; both gain intervals in the full-validation comparison are above zero. Differences are calculated from unrounded values.
AuthorsOpenReview ↗

2. Input ablation settings

2. How the four C.1 input configurations are implemented

All four configurations use the same architecture and retain Direct context and route inputs. They differ only in which candidate visual inputs contribute to the decision. Masking sets the designated input blocks to zero after normalization, with the same network branches and input dimensionality retained in every configuration. The normalization parameters are estimated from the fitting data, so zeroing corresponds to replacing those inputs with their training means. The same masking rules are applied during training and evaluation, with matched initialization, sample order, and update budgets.

ConfigurationRetained candidate visual inputsInputs zeroed after normalizationMean per-image IoU (%)Dataset-level mIoU (%)
Shared context onlyNoneImage–mask and DINO region blocks55.58162.755
DINO region inputsDINO region blockImage–mask block55.83564.450
Image–mask inputsImage–mask blockDINO region block56.40764.835
Full candidate visual inputsBoth blocksNone56.65264.879

The shared-context configuration still retains Direct context, so this ablation measures the incremental contribution of candidate visual evidence. Full inputs improve mean per-image IoU by 1.070 and 0.816 percentage points relative to shared context alone and DINO candidate inputs alone, respectively. The paired intervals adjusted for six comparisons are [0.328, 1.848] and [0.078, 1.565]. Image–mask correspondence provides useful evidence for quality assessment, and the full input achieves the highest values for both metrics among the four configurations.

AuthorsOpenReview ↗

3.1 Transfer across datasets and segmentation backbones

3.1 Transfer across datasets and segmentation backbones

The input ablations and decision-component comparisons use the same 200 images and candidate conditions. Generating multiple candidates and operation combinations for each image makes this a computationally demanding evaluation, and the chosen size allows systematic comparisons within the available resources. To test transfer of the learned selection capability, we apply the decision modules to other datasets and to a different segmentation backbone.

We freeze the existing decision modules and apply them to other datasets without retraining those modules on target-dataset images or annotations. CamVid uses a fixed mapping from 19 to 11 classes. Within each dataset, Direct and DevoSeg use the same class handling and evaluation procedure.

For cross-backbone transfer, we use EVF-SAM2 and compare Direct with DevoSeg on the same BDD100K images, keeping the decision modules frozen. Direct and DevoSeg use the same segmentor in each comparison.

Dataset / segmentorDirect: mean per-image IoU (%)Frozen workflow: mean per-image IoU (%)Direct: dataset-level mIoU (%)Frozen workflow: dataset-level mIoU (%)
CamVid / SAM 365.94565.91772.71272.796
Foggy Zurich / SAM 356.24356.31952.82752.933
BDD100K / SAM 354.33254.64960.70461.431
BDD100K / EVF-SAM223.33623.52234.36735.114

Dataset-level mIoU increases on CamVid, while both metrics increase on Foggy Zurich and BDD100K. The BDD100K gains have 95% image-paired intervals of [0.085, 0.569] and [0.407, 1.148] percentage points, respectively. These comparisons show that the existing decision modules can remain useful without additional target-dataset training supervision.

Both metrics improve with EVF-SAM2, showing that the existing decision modules can also help select outputs from another segmentation backbone. These transfer results apply to the datasets, label mappings, and backbones evaluated here. Transfer to predefined unseen combinations of degradations requires a dedicated evaluation with those combinations held out.

We also adapt SAM 3 using the same annotations, then fix the adapted checkpoint and compare Direct with DevoSeg. Adaptation freezes the image and text backbones and updates the remaining parameters. Each pair uses the same segmentor checkpoint.

Segmentor settingDecision methodMean per-image IoU (%)Dataset-level mIoU (%)
Frozen SAM 3Direct53.44863.870
Frozen SAM 3DevoSeg54.81266.008
Adapted SAM 3Direct67.78075.180
Adapted SAM 3DevoSeg67.86577.413

With adapted SAM 3, the workflow increases dataset-level mIoU from 75.180% to 77.413%. Improving the segmentor and learning how to use it can therefore be complementary.

For external methods such as Refign and FREST, we state the training conditions separately and use their results as context across training regimes. Direct with the same segmentor and matched decision controls provide the comparisons for evaluating DevoSeg’s decisions.

MethodTraining conditionsMean per-image IoU (%)Dataset-level mIoU (%)
Refign-DAFormerCityscapes labels + unlabeled ACDC adaptation51.87765.393
Refign-HRDA*Cityscapes labels + unlabeled ACDC adaptation60.30471.056
FRESTCityscapes source model + source-free ACDC adaptation56.78467.676
DevoSegFrozen SAM 3; external decision modules trained with labels54.81266.008
Reviewer_bVbtOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript introduces DevoSeg, a self-evolving decision framework built around a frozen SAM 3 segmentor for semantic segmentation. The authors argue that direct segmentation and fixed tool pipelines follow predetermined paths that cannot adapt to image-specific degradation. DevoSeg instead separates the problem into two decisions: (i) a Router selects, before execution, a compatible set of policies (e.g., restoration, prompting, refinement) under an image-dependent budget; and (ii) a Comparator selects, after execution, among the resulting candidate masks and an independently executed Direct prediction, using image–mask and region evidence. Completed episodes are distilled into cross-image experience that informs subsequent routing, comparison, and budget allocation, with proposed updates validated on disjoint development groups. Experiments are conducted on 200 ACDC images (fog, night, rain, snow; 50 each) with Cityscapes-19 queries and a frozen SAM 3 backbone.

Critical strengths
  • Clear decision-level decomposition. Separating pre-execution policy acquisition from post-execution mask selection is conceptually clean and matches the different evidence available at each stage.
  • Keeping SAM 3 weights fixed and confining adaptation to external decision state is pragmatic. The framing of “policy learning as a complement to adapting segmentor weights” is a useful deployment perspective.
Critical weaknesses
  • Missing citation of closely related motivation. The central motivation of DevoSeg, adapting a frozen foundation segmentation model through external operations rather than weight updates, is closely related to two prior works that are not cited: Hu et al., “Prompting to Adapt Foundational Segmentation Models,” ACM Multimedia 2024. Yi et al., “Adapt Foundational Segmentation Models with Heterogeneous Searching Space,” ICCV 2025.

  • Evaluation is confined to 200 ACDC images, Cityscapes-19 queries, four weather conditions, and one frozen backbone. The claim that the framework accommodates different policy libraries and semantic queries is not tested beyond this setting. Additional datasets, category systems, degradation types, or online streams would substantially strengthen the generality claim.

  • Although SAM 3 is frozen, the Router, Comparator, and budget head require 920 labeled development groups and 8,640 executed candidate masks. The “self-evolving” framing should be qualified: the decision state evolves, but the initial decision heads are supervised. How these development labels are obtained in deployment, and how the system behaves under distribution shift without fresh labels, is not discussed.

  • The budget is restricted to K∈{1,3,5,8}, and the policy library consists of predefined restoration, prompting, and refinement operations. The paper does not analyze sensitivity to library composition, compatibility constraints, or budget granularity, nor does it compare against simple fixed-policy or random-routing baselines. Without such baselines, it is difficult to attribute gains to the routing mechanism rather than to the availability of multiple policies.

Questions for authors
  1. Qualify the performance claims. Restate the main conclusions as small but directionally consistent mean gains on 200 ACDC images, and explicitly note that pooled-mIoU and joint-vs-Comparator-only intervals cross zero. Avoid language implying broad superiority over adapted or fine-tuned models.
  2. Clarify the external comparison. Separate “same-protocol” comparisons from “context across training regimes” more visibly. Consider reporting a like-for-like comparison against a frozen SAM 3 with the same decision-layer budget, and clearly label ACDC-fine-tuned models as upper-reference context rather than direct competitors.
5 comments
AuthorsOpenReview ↗

1.Prior work and contribution

Thank you for recognizing DevoSeg’s separation of pre-execution policy selection from post-execution mask comparison, and for highlighting the deployment value of adapting a frozen segmentation backbone through external decision mechanisms. To support reproducibility and further research, we will release the implementation, decision modules, policy configurations, and evaluation scripts upon acceptance.

1. Relationship to prior work and DevoSeg’s contribution

Hu et al. investigate how image-processing prompts can adapt a frozen foundation segmentation model. Using a small set of images and annotations, their method learns suitable processing combinations that make the inputs better suited to the existing model. The work also addresses the difficulty of optimizing image-processing operations that are not differentiable, providing an approach to finding effective prompts without modifying the foundation model’s weights. Yi et al. extend this direction by incorporating rule-based image enhancement and learnable transformations into a heterogeneous search space. Their method uses segmentation feedback to search for adaptation operations and applies the resulting policies as input preprocessing. Both studies demonstrate the potential of input processing outside the foundation model and provide important context for our investigation of segmentation improvement with frozen models.

DevoSeg shares the idea of improving outputs through operations outside the foundation model. Our further focus is how to organize these operations for different images under a limited execution budget, and how to determine the final output from their actual results. Features such as low illumination, blur, or unclear boundaries may motivate trying a particular operation, but do not guarantee that it will improve segmentation. For example, restoration may improve visibility in some regions while changing the texture of small objects and disrupting predictions that were originally correct. Consequently, deciding whether an operation is worth executing and whether its output is worth retaining requires evidence from different stages.

With this distinction in mind, the Router uses pre-execution image information and the current state to select operations, whereas the Comparator evaluates the actual candidate masks after execution. Restoration, prompt adjustment, and local refinement can therefore form a workflow under a budget constraint, while Direct remains an available output. Our contribution lies in combining image-dependent operation selection, budgeted execution, and output selection based on the resulting candidates, rather than treating external adaptation of a frozen foundation model itself as a unique contribution. This also motivates the two-stage design: the first stage determines where to allocate computation, and the second determines whether that computation produces a result worth adopting.

AuthorsOpenReview ↗

2.Evaluation and transfer

2. Extended evaluation and applicability across datasets and label taxonomies

We compare DevoSeg with Direct using the same segmentation model weights. In the 200-image ACDC evaluation, the average gains are modest and directionally consistent; both the dataset-level mIoU gain interval and the interval for the joint decision configuration relative to Comparator-only include zero. The component evaluation involves multi-candidate generation and multi-step workflows. To compare the components systematically within the available computational resources, we used a 200-image ACDC cohort covering four weather conditions. The comparison on all 406 images in the official ACDC validation set yields the following results.

The two metrics differ in the order of calculation and aggregation. Mean per-image IoU first averages IoU over the valid classes in each image, then averages these image scores, giving each image equal weight. Dataset-level mIoU first pools each class’s intersection and union pixel counts across the entire dataset, then computes IoU for each class and averages across classes. These are the metrics called Image IoU and Pooled mIoU in the paper; the revised names clarify how they are calculated without changing the calculations or results.

MethodMean per-image IoU (%)Dataset-level mIoU (%)
Direct53.44863.870
DevoSeg with frozen decision modules54.81266.008
Improvement over Direct (percentage points)+1.363+2.138
97.5% scene-paired confidence interval for the improvement[0.734, 2.370][0.622, 4.683]

Both metrics improve, and both gain intervals are above zero. Differences in the table are calculated from unrounded values. The scene-paired intervals are obtained by jointly resampling scenes for the two methods, with a 97.5% confidence level used to account for the two metrics.

For external models, we distinguish two settings and their respective comparison purposes. The first directly applies an existing model to ACDC without further training on ACDC annotations. Cityscapes and ACDC use the same 19-class semantic taxonomy, so these models can make predictions on ACDC directly; the main differences concern image conditions such as weather and illumination. The second further fine-tunes a model using ACDC annotations to adapt it to the target data’s imaging conditions. The ACDC-fine-tuned models in the table belong to this second setting and include the corresponding SegFormer, MaskFormer, and Mask2Former variants.

The tables identify the training data and adaptation procedure for each setting. ACDC-fine-tuned variants serve as upper-reference context with additional target-domain training, helping readers understand performance under different training conditions. DevoSeg’s primary comparisons are against Direct using the same frozen backbone and against other decision strategies under the same action budget, examining the benefit of the overall workflow and the role of its decision mechanisms, respectively.

Without retraining the selector, we further apply DevoSeg to CamVid, Foggy Zurich, and BDD100K. On CamVid, after mapping the 19-class taxonomy to 11 classes, dataset-level mIoU increases from 72.712% to 72.796%. On Foggy Zurich, mean per-image IoU and dataset-level mIoU increase from 56.243% / 52.827% to 56.319% / 52.933%, respectively. On BDD100K, the corresponding metrics increase from 54.332% / 60.704% to 54.649% / 61.431%. These evaluations demonstrate the applicability of the existing decision modules to other datasets, with positive improvements in both metrics on BDD100K.

We also evaluate transfer to EVF-SAM2 on the same BDD100K images, comparing its Direct predictions with DevoSeg while keeping both the segmentor and decision modules frozen. Both metrics improve in this setting as well. Each setting compares Direct with the frozen DevoSeg workflow using the same segmentor.

Dataset / segmentorMethodMean per-image IoU (%)Dataset-level mIoU (%)
CamVid / SAM 3Direct65.94572.712
CamVid / SAM 3DevoSeg with frozen decision modules65.91772.796
Foggy Zurich / SAM 3Direct56.24352.827
Foggy Zurich / SAM 3DevoSeg with frozen decision modules56.31952.933
BDD100K / SAM 3Direct54.33260.704
BDD100K / SAM 3DevoSeg with frozen decision modules54.64961.431
BDD100K / EVF-SAM2Direct23.33634.367
BDD100K / EVF-SAM2DevoSeg with frozen decision modules23.52235.114
AuthorsOpenReview ↗

3.Self-evolution and deployment

3. Self-evolution, supervised initialization, and annotation requirements at deployment

Supervised initialization gives the Router and Comparator the ability to estimate operation benefits and distinguish candidate quality, providing a starting point for learning from subsequent experience. Segmentation operations incur real computational costs, and starting from random decisions could spend substantial resources on ineffective or harmful attempts. Initialization and feedback learning therefore serve different purposes: the former establishes basic decision-making ability, while the latter updates that ability using successes and failures from actual execution.

Self-evolution in DevoSeg forms a loop of execution, feedback, experience accumulation and reuse, and decision updating. The system associates changes between parent and candidate masks with observed segmentation feedback, recording successful corrections and their context as well as operations that cause degradation and the conditions in which they occur. These positive and negative experiences are retrieved and incorporated into subsequent learning to update the Router, Comparator, prompting, and local decision modules. Past execution outcomes can therefore influence future operation selection and candidate comparison. Supervised initialization establishes basic decision-making ability, while subsequent feedback updates the decision parameters and experience state. The foundation segmentation model remains frozen; learning takes place in the decision layer that organizes operations, compares outputs, and uses accumulated experience.

Supervision establishes the initial decision-making ability, after which the learned modules can be deployed directly on a new dataset. The BDD100K evaluation in Section 2 uses the existing frozen modules; BDD images and labels are used only for evaluation, with labels accessed for scoring after predictions have been sealed. The improvement over Direct shows that these learned decisions transfer to BDD100K and can be reused at deployment without supervised retraining on the target dataset.

During deployment inference, the Router and Comparator make decisions from the image, candidate masks, and model state, without requiring ground-truth labels for the current image. Reliable labels are used for initialization and subsequent supervised updates and may come from human annotations or quality-checked corrections obtained during deployment. When new reliable labels are unavailable, the system continues to select operations and compare candidates using fixed decision modules. Further learning updates require reliable feedback.

We also compare four DevoSeg decision settings on the same BDD100K evaluation images under the same action budget: frozen common-start modules, a Router-checkpoint control, full joint training, and training with new experience writeback disabled. Each checkpoint is fixed before evaluation.

MethodMean per-image IoU (%)Dataset-level mIoU (%)
Direct54.33260.704
Common-start frozen configuration54.64961.431
Router-checkpoint control54.69961.576
Full joint-training configuration54.49461.488
Configuration without new experience writeback54.60661.223

The Router-checkpoint control uses the designated Router checkpoint with the remaining modules held at their common settings. Full joint training updates the decision modules and writes execution experience into memory. Disabling new experience writeback retains existing experience but does not add experience generated during this training run. All four settings exceed Direct in both mean per-image IoU and dataset-level mIoU on BDD100K, with the Router-checkpoint control reaching 54.699% and 61.576%, respectively.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

We thank the AC and reviewers for their careful reading and constructive feedback. DevoSeg studies how to choose segmentation workflows for each image within a computational budget and select the final prediction from their outputs. Your comments help us clarify its relation to frozen-segmentor adaptation, the supervision used to learn external decisions, and the roles of acquisition, selection, and feedback updates.

We also appreciate the reviewers' recognition of the following aspects of the paper.

  1. A practical decision problem for deployment. Reviewer Zsud described deciding which refinement operations are worth executing for each image as a “practical decision problem,” and Reviewer bVbt described learning external policies with SAM 3 frozen as “a useful deployment perspective.” DevoSeg brings workflow choice, execution budget, and final-output selection into one decision process, using the image and available predictions to determine the next operation.

  2. A clear separation between planning before execution and judging results afterward. Reviewer bVbt highlighted the “Clear decision-level decomposition” and noted that it “matches the different evidence available at each stage.” The Router chooses a workflow using the image, existing predictions, and remaining budget. The Comparator selects the output using the image-mask evidence produced by execution. We learn these decisions separately because they use different evidence at different stages.

  3. An explicit reference prediction that remains available as an output. Reviewer Zsud noted that “Keeping Direct as an available fallback is sensible.” Direct is generated independently by the segmentor and remains in the candidate set. Running a workflow and accepting its output are separate decisions: the system can stop acquisition early or return Direct after obtaining additional candidates.

  4. Component roles and computational costs that can be examined separately. Reviewer Xe6f identified the mathematical presentation as a strength, writing, “The mathematical formulation in the methodology section is academical.” Reviewer Zsud described the paired uncertainty estimates, component ablations, and acquisition timing as “useful reporting choices.” Holding candidate masks fixed isolates output selection; comparing operation-selection strategies under the same tool library and action cap examines acquisition decisions; timing all execution stages measures deployment cost.

On the complete 406-image ACDC validation set, DevoSeg increases mean per-image IoU from 53.448% to 54.812% and dataset-level mIoU from 63.870% to 66.008%, with both 97.5% scene-paired gain intervals above zero. After adapting SAM 3 with the same 920 annotations, adding DevoSeg increases dataset-level mIoU from 75.180% to 77.413%. These results support external decisions with a fixed segmentor and their combination with segmentor adaptation.

The additional controls connect these results to the decision mechanisms. With identical candidate masks, the Comparator outperforms uniform random selection at all four evaluated budgets. Under the same action cap on MUSES, Router-based execution reduces complete request latency by approximately 24% relative to fixed policies and 23% relative to valid random routing, with quality and latency reported together. Input and policy-library ablations, along with class-wise region and boundary metrics, further examine the sources of improvement.

Without retraining the decision modules, dataset-level mIoU improves on CamVid, and both metrics improve on Foggy Zurich and BDD100K; both also improve when transferring to EVF-SAM2 on BDD100K. For self-evolution, we clarify how execution feedback is accumulated and reused to update subsequent operation selection and candidate comparison while the segmentation backbone remains frozen. The reported BDD100K controls cover frozen decision modules, a Router-checkpoint control, full joint training, and training without new experience writeback. All four configurations outperform Direct on both metrics. These results demonstrate the usefulness of learned decisions on the target dataset and provide comparisons across experience-update settings.

To support reproducibility and further research, we will release the implementation, decision modules, policy configurations, and evaluation scripts upon acceptance. We thank the AC and reviewers again for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.