acceptodds
Under review as a conference paper at ICLR 2027

Concentration, Not Uncertainty: Why Targeted Synthetic Data Doesn't Help Camouflaged Object Detection

Abstract

Camouflaged object detection requires pixel-accurate masks, but obtaining such annotations is slow and costly, making synthetic training images an attractive alternative. Under a fixed generation budget, however, it remains unclear which real-image regions to target for synthetic data generation. We study an uncertainty-guided generation strategy that clusters the unlabelled real images, identifies clusters on which the model is least certain, allocates synthetic generation toward those clusters, and iteratively retrains the model. Across 103 training runs, uncertainty-based targeting does not outperform random allocation. Five independent controls further show that this null result is not an artifact: targeted training sets are measurably different from random sets, but the difference is explained by concentrating the generation budget rather than by where uncertainty is concentrated, as every concentration rule we test reproduces the effect and, on boundary accuracy, so does aiming at the clusters the model was most certain about. Separately, we find substantial data contamination in CHAMELEON, with 50 of its 76 images duplicated from training data despite the standard overlap check reporting zero overlap. Together, these results show that, under a fixed synthetic-data budget, budget concentration, not uncertainty-based targeting, accounts for the observed training-set effects.

Primary area:
datasets and benchmarks
Keywords:
Camouflaged object detection, active learning, synthetic data generation, uncertainty estimation, mean-teacher, benchmark contamination, closed-loop training, dataset duplication

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RL5BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether uncertainty-guided allocation of a fixed synthetic-data budget improves camouflaged object detection (COD). Through controlled comparisons with random, shuffled, reversed, and current-error-guided allocation, the authors find no reliable improvement in the primary metrics. The paper also reports substantial image overlap between CHAMELEON and training data.

Critical strengths
  • The same-shape shuffled and reversed controls provide a meaningful way to isolate the contribution of uncertainty-guided allocation. The extensive experiments make the negative results more informative than a simple comparison against random sampling.
  • The CHAMELEON overlap audit examines re-encoded and geometrically transformed images, offering potentially useful evidence about benchmark integrity.
Critical weaknesses
  • For the primary comparison C10−B, the reported difference is +0.0024+0.0024, with a 9595% confidence interval of [−0.0020,+0.0067][−0.0020, +0.0067]. This interval does not establish equivalence within the proposed margin of (δ=0.005\delta=0.005). The results support a lack of reliable evidence for an advantage, not the stronger conclusion that uncertainty guidance has no meaningful effect.
  • The distributional analyses show that concentrated allocation changes the training-set composition, but this does not establish that concentration explains downstream performance changes. The post-hoc boundary-metric results are suggestive but are not consistent across both architectures.
  • The reported C10 results achieve approximately (Sα=0.719S_\alpha=0.719) on COD10K-test and (0.7690.769) on NC4K, below the reported unsupervised RISE results of 0.7630.763 and 0.8050.805, respectively, and standard supervised SINet-v2 results of approximately 0.8150.815 and 0.8470.847. These results are not directly comparable because the training protocols differ. And the gap raises questions about the effectiveness of the overall pipeline. Stronger protocol-matched baselines and a clearer explanation of the performance gap are needed.
  • The method relies on a fixed generator and a limited foreground pool, with synthetic diversity primarily coming from background changes. Weak clustering structure and limited foreground diversity may constrain the method from the outset. Consequently, the results provide limited evidence about uncertainty-guided synthetic-data generation more generally.
Questions for authors
  • Could the primary confidence intervals and equivalence-test results be presented more prominently, with a clear distinction between failure to detect an effect and evidence of equivalence?
  • What evidence supports the claim that budget concentration explains downstream performance, rather than merely changing the training-set distribution?
  • Can the performance gap be explained through protocol-matched comparisons with supervised and unsupervised COD baselines?
  • Can the conclusions be better bounded to the specific generator, foreground pool, and training setup evaluated?
Reviewer_CausOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether an uncertainty-guided closed-loop strategy can efficiently allocate a fixed synthetic data generation budget (e.g., 1,000 images)

The authors evaluate a standard active-learning-style loop: Cluster unlabeled real target images into k=75 clusters using self-supervised representations (DINOv2)

Measure model uncertainty across clusters using a student–teacher disagreement score

Allocate more synthetic images to clusters where the model is least certain using a softmax temperature distribution.

Append the synthetic data to the training pool and retrain the model

Key Finding — Null Result on Uncertainty: Across 103 training runs, uncertainty-based targeting does not beat allocating synthetic data uniformly at random on downstream segmentation accuracy.

Critical strengths

Authors introduces the "same-shape shuffle" control to decouple budget concentration (shape) from budget direction (aim). This exposes a major drawback present in prior synthetic data allocation literature.

In-Depth Failure Diagnosis: Rather than stopping at a null result, the authors rigorously analyze why the loop failed (clustering silhouette analysis, within vs. between error variance, manifold coverage)

Critical weaknesses

The main contribution of this paper is the validation of null hypothesis i.e. Any non-uniform allocation is separable from a uniform draw with lower effective rank, so the probe result is only as good as a sanity check . From a contribution view point this has limited role in advancing the camouflage object detection methodology.

Bounded Scope: The conclusions are bounded to Camouflaged Object Detection (COD) using the LAKE-RED generator, SINet/SINet-v2 architectures, and background replacement. Findings may not directly transfer to task domains with vastly more expressive generative feedback loops.

Generator Expressiveness Bottleneck: The generator is constrained to reusing fixed foreground cutouts, creating a narrow synthetic distribution (0.13 --0.54 manifold coverage) that may inherently limit the potential benefit of any steering policy

Questions for authors

CORACLE keeps C10's shape. Did you test an oracle with a different shape (e.g., proportional to true error, or concentrated on the truly worst clusters)?

Replicate with one modern backbone and one foreground-varying generator, and scope the claim to the LAKE-RED/S2R-COD pipeline.

Reviewer_dypEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether uncertainty-guided allocation of a fixed synthetic-data budget improves camouflaged object detection (COD). The authors develop a pipeline that clusters unlabeled real images, estimates cluster-level uncertainty from student–teacher disagreement, and allocates synthetic samples toward high-uncertainty clusters. Using SINet and SINet-v2, they compare this strategy against random allocation and several controls designed to separate the effect of uncertainty targeting from the concentration of the allocation budget.

Across 103 training runs, the authors find no consistent improvement from uncertainty-guided targeting. Additional experiments suggest that the distinguishability of targeted and randomly selected synthetic datasets is primarily explained by allocation concentration rather than the direction of the uncertainty signal. The paper further investigates several potential explanations for the negative result, including weak clustering structure, limited alignment between uncertainty and evaluation metrics, and insufficient coverage of the target distribution by synthetic samples.

Separately, the authors identify substantial data contamination in the CHAMELEON benchmark, reporting that at least 50 of its 76 images overlap with COD10K or CAMO training data despite exact file hashing detecting no overlap. They provide a duplicate-detection procedure and discuss the implications for COD evaluation.

Critical strengths
  1. A rigorously controlled study investigating a potentially significant confounding factor in the synthetic data allocation process.

The study’s most notable methodological contribution lies in its distinction between "allocation concentration" and "targeting direction." Comparing uncertainty-based selection solely against random sampling fails to disentangle these two factors; in contrast, control techniques such as "concentration-preserving shuffle" and "reversed-ranking" provide more meaningful experimental comparisons.

The authors’ transparent presentation of these negative—or rather, inconclusive—results further strengthens the study’s empirical contribution.

This is highly significant, as the proposed control methods are not limited to the field of Camouflaged Object Detection (COD) but are also valuable for evaluating other uncertainty-guided data acquisition and augmentation strategies.

  1. A critical finding regarding "benchmark contamination," which holds independent value in its own right.

The analysis of the CHAMELEON dataset represents a substantial contribution. The authors reveal that 50 out of the 76 images in this dataset overlap with the training data—comprising 41 re-encoded copies (with identical dimensions) and 9 matches identified via geometric verification. Crucially, these overlaps could not be detected through simple exact hashing, highlighting the limitations of conventional deduplication methods.

Furthermore, the study demonstrates that different versions of the CHAMELEON mask data lead to significant discrepancies in evaluation results—an observation that underscores the practical importance of this analysis.

Regardless of the ultimate success of the "uncertainty-guided" generation strategy proposed in the paper, these findings have profound implications for assessing the reliability and comparability of existing benchmarks for the COD task.

Critical weaknesses
  1. Statistical Sensitivity and Interpretation of the Negative Result

The difference between targeted and random training is quite small. Synthetic images make up only 18.4% of the training set, and about half are shared across different groups. As noted in Appendix C, only around 8.8% of the full training set actually differs. This may make small improvements hard to detect.

The authors also use twice the pooled standard deviation as their decision rule, which is more conservative than using the standard error. The eight-seed comparison shows a difference of +0.0024, with a 95% confidence interval of [-0.0020, +0.0067]. This rules out the reference improvement of +0.0142, but smaller gains are still possible. Also, the targeting-versus-random comparison does not pass the equivalence test at ±0.005.

Overall, the results show that this method does not bring a large improvement in the tested setting. However, they do not fully rule out smaller gains. I think the paper should make this limitation clearer.

  1. Downstream Validation and Generalizability of the Concentration Hypothesis**

The geometric analysis shows that concentrating samples changes the selected dataset, even when uncertainty scores are shuffled or reversed. However, this does not mean that concentration improves detection performance.

In the post-hoc boundary-metric analysis, both targeted and reversed allocation perform better than random allocation on SINet. However, these gains are much smaller on SINet-v2. This suggests that the effect of concentration may depend on the model architecture.

The experiments also use a fixed pool of LAKE-RED images generated offline. Since no new foreground content is generated, uncertainty guidance can only select from existing samples. This may limit its benefits, especially when the synthetic data does not cover the real data distribution well.

Therefore, the negative results may be partly caused by the generation setup. It is still unclear whether the same conclusion would hold with better synthetic data or an adaptive generation process.

Questions for authors
  1. How well do the results rule out smaller improvements from uncertainty-guided targeting?

The eight-seed results rule out large improvements, but the targeting-versus-random comparison does not pass the equivalence test at ±0.005. Could the authors clarify what level of improvement they consider meaningful for COD? Also, could they provide a paired analysis using the same random seeds? This would help clarify whether the negative result also applies to smaller but useful improvements.

  1. How much do the limited training-set differences and synthetic data diversity affect the conclusion?

As noted in Appendix C, only about 8.8% of the training pool differs between the compared groups. The synthetic data also have limited coverage of the real data distribution. Would the same conclusion hold if the training sets had larger differences or if the generator could produce more diverse foreground content? Additional experiments or discussion on these limitations would help support the generality of the results.

  1. How does CHAMELEON contamination affect existing COD evaluation results?

The paper identifies 50 overlapping images, including 44 that also appear in COD10K-train. Could the authors clarify how these overlaps may affect common COD training and evaluation protocols? In particular, it would be useful to distinguish between supervised training, unlabeled target-domain adaptation, and checkpoint selection. This would help readers better understand the practical impact of the contamination finding.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: revised manuscript and summary of changes

We thank the three reviewers and the AC. Our initial responses are posted; each reviewer now has a two-part follow-up with the results, inline tables and exact locations in the revised PDF. The main text is within the 10-page limit.

Revision colours. In the marked revision, changes addressing RL5B only are blue; changes addressing Caus only are teal; changes addressing dypE only are purple; revisions addressing two or more reviewers are orange. Orange overrides the individual colours. Uncoloured text is unchanged or was only moved. A clean, uncoloured copy has the same content.

The claim, as now stated throughout. Within the tested pipeline (LAKE-RED fixed render pool, S2R-COD protocol, SINet/SINet-v2, N = 1000), uncertainty-guided allocation shows no detectable benefit over random allocation.

  • A pre-registered confirmatory campaign establishes equivalence within ±0.005 SαS_\alpha on the primary cell.
  • Same-shape controls show that the direction of the signal adds nothing detectable.
  • The selector's matching and the budget's concentration explain how the selected training sets differ, but a pre-registered dose-response test found no effect of concentration on accuracy.
  • The CHAMELEON audit is a separate contribution.

Main changes

  • Equivalence, not just non-detection (RL5B, dypE; orange). New Table 3 (§3.1) gives the eight-seed Welch and paired intervals and a pre-registered confirmatory campaign. That campaign establishes equivalence on the primary cell: Δ = +0.0006, 90% CI [−0.0027, +0.0039], TOST pp = 0.015. It is incomplete (16/15/15 of 27 seeds per arm, after power outages), and against the same-shape shuffle on COD10K-test it is neither detected nor equivalent. All 16 eight-seed contrasts are in Table 9 (App. C).
  • Concentration restricted to the training set (RL5B, dypE; orange). The pre-registered dose-response test did not support a downstream effect (Table 7). The three-seed boundary-metric result does not hold at eight seeds and is withdrawn (Table 8). One exploratory CINV−B signal is reported with its correction status.
  • Scope (all three; orange). The abstract, introduction, a new Scope paragraph (§4) and the conclusion bound every claim to the tested generator, pool, clusters, architectures, training and budget. The foreground constraint is quantified, and coverage is reported at every k (App. E).
  • Oracle allocations of other shapes (Caus; teal). Three pre-registered shapes, 12 runs; none resolves a gain (Table 2; App. I). The probe correction: even an unconcentrated quota separates from random under the same selector (§3.2).
  • Protocol-matched baselines (RL5B; blue). RISE, supervised SINet-V2 and our trainer on real masks, under one scorer (Table 4, App. A).
  • Training-set contrast and CHAMELEON protocols (dypE; purple):
    • the decisive comparison differs in 14.1% of the pool, not 8.8%;
    • a larger-budget preflight failed;
    • CHAMELEON shares 32 images with COD10K-test, and 74 of its 76 images are duplicated somewhere in COD10K or CAMO;
    • the overlap's meaning is spelled out per protocol (Table 17, App. F).

New work. 99 trained runs after review:

  • the confirmatory campaign, 46;
  • the dose-response experiment, 35;
  • the oracle shapes, 12;
  • supervised references, 6.

Not run:

  • the SINet-v2 dose-response replication;
  • a higher operating point;
  • a modern transformer detector (no licensed, leak-free training code);
  • a generator that creates new foregrounds (none available).

Each is stated as a limitation, not implied.

Corrections to our posted responses (detailed in each reply):

  • rule sensitivity 0.20, not 0.14;
  • 80%-detectable effect 0.0087, not 0.008;
  • boundary ratios 0.30×/0.64× at eight seeds, not 1.37×/1.34×;
  • decisive contrast 14.1%, not 8.8%;
  • RISE's detector is SINet-V2 trained on pseudo masks for the same images;
  • CHAMELEON copies in CAMO-250 are four, not three.

The manuscript also corrects its pre-registration record: one rule (the oracle campaign's) was committed 62 minutes after its first run started, though before any of its results existed (App. B).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.