acceptodds
Withdrawn from ICLR 2027

Boldness: Evaluating Models under Label-Changing Edits via Diffusion Latent Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We define boldness as a model's ability to predict the new oracle label under valid nearby label-changing perturbations, complementing robustness under label-preserving perturbations. Evaluating boldness requires constructing natural, source-faithful samples whose oracle labels change. We characterize these requirements and formulate boldness-sample construction as a constrained optimization problem. To make this construction practical, we develop a plug-in diffusion method that optimizes the latent seed of a fixed pretrained conditional generator. The method combines target-class conditioning with source-label loss to search for label-changing edits on which the model retains the source label, while regularizing the latent seed around a source-inversion solution to promote source faithfulness. Target semantics and source faithfulness are validated before evaluation. Experiments on image classification and object detection reveal substantial boldness failures across the evaluated model families, including robustness-enhanced models, highlighting the need to evaluate boldness alongside clean accuracy and robustness.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
model evaluation, adversarial robustness, label-changing perturbations, semantic image editing, diffusion models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_33tAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies boldness: whether a model predicts the new label after a meaningful, label-changing edit to an input. It formalizes the requirements for such edits and constructs them by optimizing the latent seed of a fixed diffusion model, combining target-class conditioning with a source-label loss and source regularization. Experiments on ImageNet-100 and COCO report low boldness accuracy across classifiers, robust models, and object detectors, with some failures transferring across models.

Critical strengths
  • The formulation makes source faithfulness an explicit part of evaluating label-changing edits. This helps distinguish a failure to respond to a meaningful edit from an ordinary error on an unrelated image.

  • The experiments cover classification and detection, including robustness-enhanced models and cross-model transfer. This breadth makes the evaluation useful beyond a single architecture, provided the edits are valid.

Critical weaknesses
  • Insufficient validation of the oracle-label changes and source faithfulness. The proposed definition of boldness relies on valid, source-faithful edits whose oracle labels have genuinely changed. However, the experimental validation relies entirely on automated metrics, including TIFA for target semantics, DreamSim for source similarity, and an edit-vs-random similarity gap (Table 2 and Appendix C). These metrics do not necessarily establish that the oracle label has changed or that the edited image remains a meaningful modification of the original sample. For example, an image may satisfy the target description while retaining visual features of the original category, making its true label ambiguous. Similarly, perceptual similarity does not guarantee preservation of source-specific content. Since the reported boldness failures depend directly on these assumptions, the lack of human evaluation or independent semantic-label verification raises concerns about the validity of the constructed examples and the reliability of the main conclusions.

  • Figure 3 illustrates xBx_B as a large perturbation that falls outside the natural-image manifold M\mathcal{M}. However, Section 2.1 and Section 2.2 define xBx_B as a valid boldness sample that must remain on M\mathcal{M}. Similarly, xRx_R is described as being close to M\mathcal{M}, although valid robustness samples are defined as natural images. The authors should clarify this notation or introduce distinct symbols to distinguish candidate or invalid perturbations from valid samples that satisfy the manifold constraint.

  • TIFA (Hu et al., 2023) recommends mPLUG-large as the default VQA model in TIFA v1.0. Could the authors clarify which VQA model was used in this paper? Given the progress in large vision-language models (LVLMs) over the past two years, I suggest adding an ablation using a recent LVLM for semantic validation to examine whether the validity rates and boldness accuracy remain consistent.

  • The experiments rely exclusively on Stable Diffusion 2.1, which is relatively outdated and may introduce generator-specific biases. I strongly suggest evaluating at least one additional, more recent text-to-image model. I encourage the authors to prioritize this experiment. Generating a large number of images would not improve my rating, so please do not put too much strain on yourselves or your servers during the discussion period (just start it first). A carefully selected subset with actual experimental results, including mutiple metrics, would be sufficient and more useful for my assessment.

  • The manuscript does not include active cross-referencing hyperlinks, making navigation difficult. In addition, several figures are not referenced in the main text. Visual figures are intended to assist readers at the exact point where the corresponding concepts are discussed; omitting in-text references disrupts the flow of the presentation and weakens the clarity of the paper.

Questions for authors

See Weakness.

Reviewer_XwD4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces "boldness", a model's ability to update its prediction when a nearby edit changes the true (oracle) label of an image, as a counterpart to robustness under label-preserving perturbations. The authors argue that prior large-distance constructions can be satisfied by unrelated images, and instead formulate boldness-sample construction as finding a source-faithful, natural edit with a new label on which the model keeps its original prediction. In practice, they optimize the initial noise seed of Stable Diffusion 2.1 under a target-class text prompt, minimizing the evaluated model's cross-entropy on the source label while keeping the seed close to an inversion of the source image and within the Gaussian shell. Generated edits are filtered with TIFA, DreamSim and an edit-vs-random similarity gap. Experiments on ImageNet-100 classifiers (including adversarially robust and AugMix models) and COCO detectors report very low boldness accuracy, and the authors conclude that boldness should be evaluated alongside accuracy and robustness.

Critical strengths

The question the paper asks is a good one. Most evaluation work focuses on whether models stay stable under changes that should not matter, and much less on whether they respond when something that does matter changes. The paper makes this distinction clearly, and the toy example in Figure 2 is a nice illustration that the two failure modes can coexist. I also agree with the critique of pure distance-based constructions: requiring the new image to be far from the source does not make it a meaningful edit of that source, and the paper is right to put source faithfulness at the center. The method itself is simple and builds sensibly on existing plug-in optimization with generative priors, and it is nice to see the evaluation extended beyond classification to object detection. The paper is generally clearly written and honest about the limits of its search (e.g., only one target label per image).

Critical weaknesses

The main claim cannot be separated from ordinary white-box adversarial attacks. The edits are produced by directly minimizing the evaluated model's loss on the source label, with gradients through the model, for many optimization steps. This is essentially an unrestricted adversarial attack carried out in the diffusion latent space, and the generator is free to introduce adversarial texture or keep source-class cues in the image. Under these conditions, near-zero same-model accuracy is what one would expect from any white-box attack, and it does not by itself show that models fail to adapt to semantic changes. The same applies to the robust models: they are only robust within a small ℓp ball, so it is not surprising that an unrestricted generative attack breaks them, and this does not say much about the relationship between robustness and boldness. The key missing experiment is a baseline where the same target-conditioned edit is produced without the classifier loss (e.g., plain inversion-based editing from the same source). If models handle these plain edits correctly and only fail on the optimized ones, the paper is measuring adversarial vulnerability rather than boldness. If they also fail on plain edits, that would be a much more convincing and interesting result.

Whether the oracle label actually changed is never verified by humans. The paper defines the oracle label as human judgment, yet label change is checked only with automated proxies: a VQA-based score, a perceptual similarity threshold, and a comparison against random generations. TIFA can tell whether the target concept is present, but not whether the image's label has changed from the source to the target. Since the optimization explicitly pushes the image to keep whatever makes the model predict the source class, it seems quite likely to produce images that contain both concepts or are genuinely ambiguous. Choosing the target as the model's second most likely class makes this worse, since these are exactly the classes that are easiest to confuse. Some of the qualitative examples look like they could reasonably still be labeled as the source class. A human study on a subset of the retained edits seems necessary before calling these model failures. The edit-vs-random gap criterion is also quite weak, as it only requires the edit to be more similar to the source than a random target-class image.

The metric does not quite match the failure mode being studied. The paper says it focuses on the conservative failure where the model keeps the source label, but boldness accuracy counts any prediction other than the target as a failure. For the cross-model results in particular, it would be important to know how often models keep the source label versus predict some third class; the latter would point more toward edit quality or ambiguity than toward a lack of boldness. The same breakdown is missing for detection, where the drop in boldness mAP could mix several different effects.

Some of the validation statistics are hard to reconcile. In the source-regularization ablation, the setting without regularization has a mean DreamSim close to the acceptance threshold, which suggests a large fraction should fail the source-faithfulness check, yet the reported validity rate is nearly unchanged. Similarly, the mean and spread reported in Table 2 seem to imply a noticeably lower pass rate than the one reported. It is not clear whether these statistics are computed before or after filtering. More generally, the validity rate barely moves across regularization strengths, which makes me wonder how selective the filter really is. The default regularization strength also appears to be chosen to maximize attack success.

Several important details are missing. The detection part is described only at a high level: it is not explained how the edit is restricted to the bounding box, how target categories are chosen, whether the optimization is done against each detector, or how many edits are retained. The inversion uses very few steps, and no reconstruction quality is reported, even though the source-faithfulness penalty relies on it. The number of sampling steps, the computational cost of backpropagating through the sampler, and any variance across runs are not reported. Only a single generator is used.

The novelty of the concept is limited and related work is incomplete. Boldness is closely related to invariance-based adversarial examples and excessive invariance (Jacobsen et al., 2019; Tramèr et al., 2020), and the paper does not clearly explain what is new beyond the name and the construction method. The "nearby" set is never actually defined, so the formalization remains fairly informal, and the edits that pass validation are in fact quite large in pixel space. There is also a body of work on diffusion-based counterfactual generation and generative test sets for evaluating classifiers that would deserve discussion.

Presentation could be improved. The main text does not use the full page limit, yet some of the most important pieces (human validation, the plain-editing baseline, detection details) are missing. Meanwhile, Figures 1 and 2 are conceptual illustrations that take up space without carrying much information, and the qualitative edits in Figures 4 and 5 are too small to judge whether the edits really succeed, which is exactly what readers most need to see. Parts of the appendix (e.g., the closed-form projection onto the Gaussian shell) add little, and the reference list has a duplicated entry.

Questions for authors
  1. What is the boldness accuracy of the evaluated models on target-conditioned edits produced without the classifier loss term, e.g., plain inversion-based editing from the same source seed with the target prompt? This seems essential to separate boldness failures from white-box adversarial vulnerability.

  2. Have you run any human evaluation of the retained edits? In particular, would human annotators assign the target label (and not the source label, or both) to these images? Some examples in Figure 4 look ambiguous to me.

  3. For the failures counted in boldness accuracy, how often does the model keep the source label versus predict some other class? Please report this breakdown for both the same-model and cross-model settings, and for detection.

  4. Were the robust models evaluated in a white-box setting, i.e., with edits optimized against the robust model itself? If so, how do you distinguish their failure from the expected vulnerability of ℓp-robust models to unrestricted attacks?

  5. Are the DreamSim and TIFA statistics in Table 2 and Table 6 computed before or after filtering? Given that the mean DreamSim without regularization is close to the threshold, how is the validity rate nearly unchanged?

  6. How is the edit restricted to the bounding box in the detection experiments, how are target categories chosen, and how many edits are retained per detector?

  7. How good is the source inversion with only five steps? Could you report reconstruction quality, and discuss how inversion errors affect the source-faithfulness penalty?

  8. How does boldness differ conceptually from invariance-based adversarial examples as studied by Jacobsen et al. (2019) and Tramèr et al. (2020), beyond the construction method?

Reviewer_aYqSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a model's ability to predict the correct new label after a valid, source-related semantic edit, termed boldness. It formulates the construction of label-changing examples and implements it through latent optimization of a fixed conditional diffusion generator. The method combines target-class conditioning, an original-label classification loss, source-inversion regularization, and a Gaussian-shell constraint. Generated candidates are filtered using target-consistency and source-similarity measures before evaluation. Experiments on ImageNet-100 classifiers, robustness-enhanced models, and COCO object detectors reveal substantial prediction failures on retained edits, including cross-model transfer.

Critical strengths
  1. The proposed framework explicitly considers target semantics and source faithfulness, providing a structured approach to evaluating label-changing edits beyond conventional perturbation magnitude constraints.

  2. The experiments cover multiple classifier architectures, robustness-enhanced models, and object detectors. The detection extension is particularly valuable because it examines semantic adaptation while separately measuring object localization.

Critical weaknesses
  1. The conceptual contribution is not sufficiently distinguished from existing invariance-based adversarial examples. Previous studies have already investigated true-label changes with unchanged model predictions and large-distance prediction-preserving perturbations (Tramèr et al., 2020; Nie et al., 2025). Although the proposed diffusion-based semantic editing offers a different construction, stronger comparisons are needed to establish its additional contribution.

  2. The reliability of the reported boldness results depends heavily on the validity of the generated label changes. The evaluation relies on TIFA (Hu et al., 2023) and perceptual-similarity thresholds without independent human verification. Moreover, since the optimization explicitly encourages models to retain their original predictions, the extremely low boldness accuracy primarily characterizes adversarially selected failures rather than general adaptation performance.

References

  • Tramèr et al. (2020). Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations. ICML.
  • Nie et al. (2025). A New Type of Adversarial Examples. arXiv:2510.19347.
  • Hu et al. (2023). TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering. ICCV.
Questions for authors
  1. How does the proposed boldness formulation fundamentally differ from existing invariance-based adversarial examples (Tramèr et al., 2020; Nie et al., 2025)? Beyond the use of diffusion-based editing, what is the main conceptual or methodological contribution?

  2. Can the authors provide human verification of a representative subset of generated edits to confirm that the oracle labels have genuinely changed? How sensitive are the reported results to the TIFA and perceptual-similarity thresholds?

  3. Since the optimization explicitly encourages models to retain their original predictions, how should the extremely low boldness accuracy be interpreted? Can the authors compare against semantic edits generated without the original-label classification loss to distinguish general adaptation failures from adversarially induced failures?

Reviewer_RbNVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary
  • This paper introduces Boldness as a model's ability to correctly predict labels under label-changing perturbations, a setting that has received relatively limited attention in conventional adversarial robustness evaluation. It proposes a diffusion-based latent optimization method for evaluating this ability.
  • Whereas conventional robustness evaluates whether a model's predictions remain stable under perturbations that preserve the ground-truth label, Boldness evaluates whether the model correctly predicts the new ground-truth label when a semantic change in the input alters its oracle label.
  • The authors specify four requirements for a valid boldness sample: label-change validity, image validity, source faithfulness, and model failure. They induce semantic changes through target-class conditioning in a pretrained diffusion generator, combining a classification surrogate loss with source regularization to generate images for which the model retains its original prediction. They assess target semantics and source faithfulness using TIFA, DreamSim, and the edit-vs-random similarity gap, while using FID to compare the generated image distribution with the clean-image distribution. This distribution-level comparison is distinct from verifying the validity of individual images. The proposed method is evaluated on image classification and object detection tasks.
Critical strengths

S1: A Meaningful Problem Formulation

  • Whereas conventional adversarial robustness focuses on label-preserving perturbations, this paper evaluates whether a model responds correctly when the ground-truth label changes.
  • This distinction is meaningful: a model should remain stable under irrelevant changes while responding appropriately to changes that alter the semantic content of the input.

S2: A Problem Definition That Accounts for Source-Faithful Semantic Editing

  • Existing large-distance adversarial attacks may admit images from other classes that bear no meaningful relationship to the source image.
  • The paper identifies this limitation and explicitly distinguishes label-change validity, image validity, and source faithfulness.
  • The goal of generating images whose ground-truth labels change while preserving a meaningful connection to the source is well motivated, extending beyond simply finding images that are far from the original input.
Critical weaknesses

W1: Unclear Conceptual Novelty of Boldness and Its Distinction from Conventional Prediction Accuracy

  • Boldness Accuracy, as defined in this paper, has essentially the same form as classification accuracy on images whose oracle labels have changed.
  • If the ground-truth label changes from yy to y′y', predicting y′y' is simply the standard requirement for correct classification. Likewise, retaining the original label yy can be interpreted as misclassification of the modified image.
  • It is therefore unclear whether Boldness constitutes a fundamentally distinct performance metric or measures classification accuracy on a particular set of images produced by label-changing perturbations. In particular, it is difficult to determine whether low Boldness Accuracy reflects an inability to respond appropriately to semantic changes or limitations in generalizing to images that were insufficiently represented during training.
  • The authors should therefore clarify how Boldness differs from prior work and conventional classification accuracy, and why it warrants being defined as an independent model property.

W1-1: Insufficient Analysis of Differences Between the Generator's and Classifier's Training Data Distributions

  • The paper uses Stable Diffusion to generate target-class images and evaluates the classifier's Boldness on these images. However, because the generator and the evaluated classifier were trained on different data distributions, the generated images may contain characteristics that differ from those in the classifier's training distribution.
  • Consequently, it is difficult to distinguish failures to respond to semantic label changes from ordinary classification failures caused by distribution shift in the generated images. Further analysis is needed to clarify whether the proposed method measures the ability to respond to semantic changes or the ability to generalize to generated images.

W2: Limitations of Using FID to Verify Image Validity

  • The paper defines image validity as a necessary condition for a boldness sample. However, in Table 3, the proposed method achieves an FID of 10.3, which is lower than the 12.7 reported for (Nie et al.) but remains substantially higher than the clean–clean reference of 5.6.
  • These results demonstrate a relative improvement over the existing method, but do not establish that the generated images are sufficiently close to the real-image distribution. Thus, although the claim that the proposed method better matches the clean-image distribution is supported as a relative comparison, this evidence alone provides limited support for the validity of the generated images.
  • More importantly, FID measures differences between the feature distributions of image sets; it does not directly verify the naturalness or validity of individual images.
  • It therefore remains unclear whether image validity, which the paper defines as a necessary condition, has been adequately verified. A lower FID than an existing baseline does not resolve this concern on its own.

W3: Effect of Source Regularization

  • The proposed method is designed to preserve source faithfulness through the source-regularization term λz∣z−z∗∣22\lambda_z | z - z^* |_2^2 in Eq. (6), where λz\lambda_z is the regularization coefficient. However, it is unclear whether reducing the latent-space distance translates into improved perceptual similarity in image space.
  • The ablation results in Table 6 also show that DreamSim similarity does not increase monotonically with the source-regularization coefficient λz\lambda_z. For example, DreamSim reaches its highest value of 0.63 at λz=0.01\lambda_z = 0.01, but decreases to 0.62 at λz=0.1\lambda_z = 0.1.
  • Increasing λz\lambda_z places a stronger penalty on deviations from the source inversion seed during optimization, so one might expect a corresponding improvement in similarity to the source image. However, this trend is not consistently observed in the reported results. These findings raise questions about how well latent-space L2 distance captures source faithfulness. The authors should therefore further analyze how effectively source regularization preserves image similarity and examine the relationship between latent-space distance and DreamSim similarity.
Questions for authors

Please refer to the Critical Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.