- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies boldness: whether a model predicts the new label after a meaningful, label-changing edit to an input. It formalizes the requirements for such edits and constructs them by optimizing the latent seed of a fixed diffusion model, combining target-class conditioning with a source-label loss and source regularization. Experiments on ImageNet-100 and COCO report low boldness accuracy across classifiers, robust models, and object detectors, with some failures transferring across models.
Critical strengths
-
The formulation makes source faithfulness an explicit part of evaluating label-changing edits. This helps distinguish a failure to respond to a meaningful edit from an ordinary error on an unrelated image.
-
The experiments cover classification and detection, including robustness-enhanced models and cross-model transfer. This breadth makes the evaluation useful beyond a single architecture, provided the edits are valid.
Critical weaknesses
-
Insufficient validation of the oracle-label changes and source faithfulness. The proposed definition of boldness relies on valid, source-faithful edits whose oracle labels have genuinely changed. However, the experimental validation relies entirely on automated metrics, including TIFA for target semantics, DreamSim for source similarity, and an edit-vs-random similarity gap (Table 2 and Appendix C). These metrics do not necessarily establish that the oracle label has changed or that the edited image remains a meaningful modification of the original sample. For example, an image may satisfy the target description while retaining visual features of the original category, making its true label ambiguous. Similarly, perceptual similarity does not guarantee preservation of source-specific content. Since the reported boldness failures depend directly on these assumptions, the lack of human evaluation or independent semantic-label verification raises concerns about the validity of the constructed examples and the reliability of the main conclusions.
-
Figure 3 illustrates as a large perturbation that falls outside the natural-image manifold . However, Section 2.1 and Section 2.2 define as a valid boldness sample that must remain on . Similarly, is described as being close to , although valid robustness samples are defined as natural images. The authors should clarify this notation or introduce distinct symbols to distinguish candidate or invalid perturbations from valid samples that satisfy the manifold constraint.
-
TIFA (Hu et al., 2023) recommends mPLUG-large as the default VQA model in TIFA v1.0. Could the authors clarify which VQA model was used in this paper? Given the progress in large vision-language models (LVLMs) over the past two years, I suggest adding an ablation using a recent LVLM for semantic validation to examine whether the validity rates and boldness accuracy remain consistent.
-
The experiments rely exclusively on Stable Diffusion 2.1, which is relatively outdated and may introduce generator-specific biases. I strongly suggest evaluating at least one additional, more recent text-to-image model. I encourage the authors to prioritize this experiment. Generating a large number of images would not improve my rating, so please do not put too much strain on yourselves or your servers during the discussion period (just start it first). A carefully selected subset with actual experimental results, including mutiple metrics, would be sufficient and more useful for my assessment.
-
The manuscript does not include active cross-referencing hyperlinks, making navigation difficult. In addition, several figures are not referenced in the main text. Visual figures are intended to assist readers at the exact point where the corresponding concepts are discussed; omitting in-text references disrupts the flow of the presentation and weakens the clarity of the paper.
Questions for authors
See Weakness.