- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes controlled Reducible Degradation Gap (cRDG), a metric for estimating the training utility of synthetic degradations through short, matched training probes. Based on this metric, the authors introduce CURB, a framework for selecting useful degradation regions under a fixed training budget. Experiments on semantic segmentation and salient object detection demonstrate improvements over several data selection baselines.
Critical strengths
-
Well-motivated problem and sound methodology. The paper addresses an important challenge in synthetic data selection: difficult degradations are not necessarily useful for training. The matched-control formulation offers a principled way to estimate degradation-specific utility while accounting for generic training progress and clean-performance degradation.
-
Improvements across multiple settings. The method improves results on both semantic segmentation and salient object detection, across different architectures, datasets, and data-generation pipelines. The gains are consistent in direction, which suggests practical value beyond a single setup, although their magnitude should be confirmed with repeated runs (see Weaknesses).
Critical weaknesses
-
Novelty relative to proxy-based search. Estimating the value of an augmentation type and magnitude through short proxy training and held-out evaluation is closely related to augmentation policy search (e.g., AutoAugment, Fast AutoAugment) and proxy-based data-mixture selection (e.g., DoReMi, RegMix). These works are not discussed. The added value of the matched control over such approaches should be clarified, ideally with a compute-matched comparison.
-
Insufficient experimental validation. Most results come from single runs without variance estimates, while the key margins are small, so the reliability of the improvements is hard to assess. In addition, over 90% of the synthetic budget is drawn from a single region. The paper should report which region is selected, how it ranks under real retraining utility, and how the result compares with an oracle choice, to show that the gains come from correct ranking rather than a fortunate selection.
Questions for authors
-
Since most of the budget is drawn from a single region, which region is selected in each setting, how does it rank under real retraining utility, and how does CURB compare with an oracle choice? Does spreading the budget across the top few regions change the results?
-
Since probe cost grows linearly with the number of regions, how does the method scale to substantially larger candidate spaces, such as finer severity bins or more degradation families?
6 comments
Response to Reviewer dMGm [2/4]: W1, continued
W1. Protocols of the adapted baselines. Each policy defines a mixture over the same 32 regions, subject to the gated pool's sample capacities. All methods use the same final-training objective and budget (Tab. S8, App. G).
AutoAugment-style search. A policy-gradient controller proposes region mixtures. Each candidate is evaluated by a -step child run from , using mIoU on a fixed held-out twin set as the reward. The five highest-reward candidates passing the clean-loss screen are combined into the final policy, following AutoAugment's policy aggregation. The 66 total runs include one clean-control run.
Fast AutoAugment-style search. Following Fast AutoAugment's fixed-model search, Bayesian optimization minimizes the loss of on held-out twins sampled from candidate mixtures. The top-ranked mixtures are combined without child-model training. Selection costs 0.03 final-run equivalents. At matched total compute, extending final training to steps gives 47.98 versus 47.56 mIoU at , using the same final-training seed.
DoReMi-style reweighting. The 32 regions serve as domains. A reference model is trained from on a uniform region mixture and then frozen. A proxy initialized from uses Group DRO on excess loss relative to the reference. Trajectory-averaged domain weights define the final mixture. Reference and proxy updates, including reference forward passes, are included in the 0.66 selection cost.
RegMix-style regression. Each proxy trains for the probe horizon on a Dirichlet-sampled region mixture, covering sparse and near-uniform allocations. All runs are evaluated on the same fixed held-out twin set and clean held-out scenes. We fit ridge and LightGBM predictors for these losses and select the regression variant by out-of-fold rank correlation. One million candidate mixtures are scored, and the 100 with the lowest predicted twin loss among those passing the clean-loss screen and sample-capacity limits are averaged. The budgets of 66, 128, and 256 total runs each include one clean-control run, and 66 matches the number of CuRB probes.
Candidate clean-loss filtering. For AutoAugment-style search, candidates are screened using their measured clean held-out loss; for RegMix-style regression, using predicted clean loss. Both apply the source-calibrated tolerance relative to the clean-control run. Screening precedes policy aggregation. CuRB instead applies its measured clean-harm constraint to individual regions.
RegMix-style diagnostics. Out-of-fold Spearman correlation on held-out mixtures increases from 0.38 to 0.52 and 0.61 at the three budgets, alongside increasing final mIoU. Additional regression and mixture-selection ablations are reported in Tab. S8.
Response to Reviewer dMGm [3/4]: W2, Q1
W2 and Q1. Repeated runs, the selected region, and the oracle. We repeat final training over three paired seeds, keeping and each selected set fixed. Results are mean ± sample SD and include the submitted run (Tab. 7a).
| Method | In-family mIoU | Dark Zurich | Cityscapes val |
|---|---|---|---|
| CuRB | 51.90±0.23 | 28.60±0.35 | 70.09±0.14 |
| 50.86±0.15 | 27.77±0.17 | 68.64±0.22 | |
| Fixed-Mid | 50.63±0.14 | 27.53±0.21 | 68.99±0.13 |
| All-gen | 49.19±0.18 | 26.62±0.19 | 68.10±0.12 |
CuRB exceeds , Fixed-Mid, and All-gen by 1.04, 1.27, and 2.71 in-family mIoU on average. Its paired gains over are 0.95, 1.24, and 0.92. Across the four hosts in Tab. 2, each retrained with and without CuRB, the mean paired gains over three seeds range from 1.01 to 1.60 mIoU on the four-condition ACDC average and from 1.03 to 1.68 on Dark Zurich, with positive gains in every run. On SOD (NUN, CSOD10K), three seeds give of .878±.004 for CuRB against .866±.002 for uncurated twins (App. E).
Selected region and allocation. In the main setting, CuRB selects fog for 94% of the synthetic budget and low light for the remaining 6%. Tab. S6 reports the regions and shares for every setting. The top region alone, with clean data filling the remaining slots, reaches 51.78±0.18, compared with 51.90±0.23 for the full selection and the single-run source-only reference of 46.46 (Tab. 3). This region therefore accounts for most of the observed gain. The full selection averages 0.12 mIoU higher, while also filling the remaining slots with synthetic rather than clean data. Even splits over the top two and top four regions give 51.76±0.15 and 51.31±0.16 (Tab. 7c), which are 0.14 and 0.59 below the full selection. Neither tested equal split improves on rank-order filling.
Oracle comparison. We retrain each of the 32 regions once under the same final-training protocol, using candidates from that region and clean data to fill the remaining budget. In these single runs, fog achieves 51.84 mIoU and ranks 2/32 overall and 2/25 among feasible regions, with an observed regret of 0.16 mIoU relative to the best feasible single region, fog (52.00). For NUN on CSOD10K, the selected region, low light , fills the whole budget and ranks 3/32 overall in the same single-run retraining, with an observed regret of 0.003 in relative to the best feasible single region. Both oracle comparisons use target-domain evaluation retrospectively and do not affect selection.
Selection stability. In the main setting, across three probe seeds and three cross-fitting partitions, the mean pairwise Spearman correlations between region rankings are 0.95 and 0.94, respectively. The top selected region remains unchanged, and each selected set shares at least 94% of its sample IDs with the default selection. With the final-training seed fixed, the changed selections achieve 51.81 to 52.04 in-family mIoU, compared with 51.97 for the default selection (App. C).
Response to Reviewer dMGm [1/4]: W1
W1. Relation to augmentation search and proxy-based mixture selection. These methods use model-based feedback to select augmentation policies or data mixtures. AutoAugment evaluates policies using child-model validation accuracy, while Fast AutoAugment uses fixed-model density matching. RegMix predicts validation loss from mixture weights using small-model runs. DoReMi optimizes domain weights using a proxy model's excess loss relative to a separately trained reference.
cRDG instead estimates the additional held-out gain from degradation relative to matched clean training. Two K-step probes start from the same checkpoint and use the same scenes, labels, and number of updates. Their augmentation slots contain degraded twins or clean augmentations, and the resulting losses are compared on degraded twins held out from probe updates. The subtraction matters because clean training alone reduces held-out loss by a region-dependent amount, which a score without the control would credit to degradation training. Unlike DoReMi's reference-model loss gap, our control compares two matched training interventions. Clean harm is screened separately. We clarify these distinctions in Related Work.
We test the subtraction directly and adapt all four methods to the same gated 32-region pool, synthetic budget, predictor, and final-training recipe. Selection cost includes method-specific training, calibration, evaluation, and search, excluding shared candidate-pool generation. CuRB costs 0.66 final-run equivalents, including 0.59 for probe updates. Results are mean ± sample SD over three paired final-training seeds, with θ0 and each selected set fixed (Tab. S8).
Control ablation. Dg removes the clean-control subtraction from the ranking score. Both methods use the same 66 probes and clean-harm constraint, giving identical probe-training compute. The subtraction raises Spearman correlation with full-retraining utility from 0.70 to 0.79 on ACDC and from 0.62 to 0.72 on Dark Zurich (Tab. 6). In-family mIoU improves by 1.04 points on average, with paired gains of 0.95, 1.24, and 0.92 (Tab. 7a).
Adapted baselines. At matched selection cost, CuRB exceeds AutoAugment-style, DoReMi-style, and RegMix-style selection by 1.31, 1.43, and 1.18 mIoU. It remains 0.87 ahead of RegMix-style with 256 runs, which uses 3.8 times the selection cost. CuRB exceeds every compared method in all three paired final-training seeds. Implementation protocols and additional diagnostics are reported in Part 2/4.