- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether uncertainty-guided allocation of a fixed synthetic-data budget improves camouflaged object detection (COD). Through controlled comparisons with random, shuffled, reversed, and current-error-guided allocation, the authors find no reliable improvement in the primary metrics. The paper also reports substantial image overlap between CHAMELEON and training data.
Critical strengths
- The same-shape shuffled and reversed controls provide a meaningful way to isolate the contribution of uncertainty-guided allocation. The extensive experiments make the negative results more informative than a simple comparison against random sampling.
- The CHAMELEON overlap audit examines re-encoded and geometrically transformed images, offering potentially useful evidence about benchmark integrity.
Critical weaknesses
- For the primary comparison C10−B, the reported difference is , with a confidence interval of . This interval does not establish equivalence within the proposed margin of (). The results support a lack of reliable evidence for an advantage, not the stronger conclusion that uncertainty guidance has no meaningful effect.
- The distributional analyses show that concentrated allocation changes the training-set composition, but this does not establish that concentration explains downstream performance changes. The post-hoc boundary-metric results are suggestive but are not consistent across both architectures.
- The reported C10 results achieve approximately () on COD10K-test and () on NC4K, below the reported unsupervised RISE results of and , respectively, and standard supervised SINet-v2 results of approximately and . These results are not directly comparable because the training protocols differ. And the gap raises questions about the effectiveness of the overall pipeline. Stronger protocol-matched baselines and a clearer explanation of the performance gap are needed.
- The method relies on a fixed generator and a limited foreground pool, with synthetic diversity primarily coming from background changes. Weak clustering structure and limited foreground diversity may constrain the method from the outset. Consequently, the results provide limited evidence about uncertainty-guided synthetic-data generation more generally.
Questions for authors
- Could the primary confidence intervals and equivalence-test results be presented more prominently, with a clear distinction between failure to detect an effect and evidence of equivalence?
- What evidence supports the claim that budget concentration explains downstream performance, rather than merely changing the training-set distribution?
- Can the performance gap be explained through protocol-matched comparisons with supervised and unsupervised COD baselines?
- Can the conclusions be better bounded to the specific generator, foreground pool, and training setup evaluated?
General response: revised manuscript and summary of changes
We thank the three reviewers and the AC. Our initial responses are posted; each reviewer now has a two-part follow-up with the results, inline tables and exact locations in the revised PDF. The main text is within the 10-page limit.
Revision colours. In the marked revision, changes addressing RL5B only are blue; changes addressing Caus only are teal; changes addressing dypE only are purple; revisions addressing two or more reviewers are orange. Orange overrides the individual colours. Uncoloured text is unchanged or was only moved. A clean, uncoloured copy has the same content.
The claim, as now stated throughout. Within the tested pipeline (LAKE-RED fixed render pool, S2R-COD protocol, SINet/SINet-v2, N = 1000), uncertainty-guided allocation shows no detectable benefit over random allocation.
Main changes
New work. 99 trained runs after review:
Not run:
Each is stated as a limitation, not implied.
Corrections to our posted responses (detailed in each reply):
The manuscript also corrects its pre-registration record: one rule (the oracle campaign's) was committed 62 minutes after its first run started, though before any of its results existed (App. B).