acceptodds
Withdrawn from ICLR 2027

LU-500: A Logo Benchmark For Concept Unlearning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Concept unlearning is increasingly used to limit protected or unsafe visual concepts in text-to-image models, yet existing evaluations mostly study targets that dominate an image rather than localized company logos. Logos create a distinct failure mode: a small mark can carry the protected concept, remain recognizable from partial visual evidence, and be triggered implicitly by products, storefronts, packaging, or advertisements. We introduce LU-500, a Fortune Global 500 logo-unlearning benchmark containing 9,584 human-verified text-query and logo-image pairs across an explicit track (LUex-500) and an implicit contextual track (LUim-500). Our multi-grained protocol evaluates local removal and non-logo preservation, sup- plements detector outputs with fixed-region and full-output audits, and validates both components against blinded human judgments. Representative inference-time and compatible fine-tuning methods show an erasure–preservation tension under their tested global interventions. This pattern persists with zero-fill-free masked metrics and three seeds, whereas a detect-then-inpaint control attains 78.2% joint success, demonstrating that the benchmark can reward a localized solution rather than mechanically inducing a trade-off. A small SD3/FLUX study yields the same broad operating regimes, providing limited portability evidence. We also analyze ProLU as a prompt-space diagnostic: semantic rewriting can reduce logo triggers, but it is neither weight-level unlearning nor evidence of latent-space entanglement.

Primary area:
datasets and benchmarks
Keywords:
unlearning, logo, dataset

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cXrdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LU-500, comprising 9,584 verified prompt–image pairs across explicit and implicit contextual tracks. It evaluates logo removal and content preservation using local/global metrics, detector audits, human validation, and multi-seed experiments. Comparisons cover inference-time interventions, fine-tuning methods, prompt rewriting, and spatial inpainting.

Critical strengths

S1: This paper provides broad company coverage and 9,584 manually verified examples. Its explicit and contextual tracks offer a useful resource for studying localized logo leakage.

S2: This paper supplements detector outputs with fixed-region checks, full-output audits, blinded judgments, and a spatial control. These checks support the practical value of its localized evaluation.

Critical weaknesses

W1: This paper reports 78.2% joint success in Table 3, but Section 3.3.2 and Algorithm 1 leave the pass/fail criteria and aggregation of human judgments unspecified. The inpainting model and mask settings are also missing.

W2: This paper offers broader brand coverage and localized evaluation, but does not establish its advance over UNBRANDING (2025) or HUB (ICCV 2025). This paper does not report the logo-generation success rate of each backbone before fine-tuning.

Questions for authors

Q1: This paper emphasizes localized logo evaluation. Which failures does it detect that UNBRANDING and HUB miss? On matched logo-removal examples, does its evaluation agree more closely with independent human judgments?

Q2: This paper reports joint success in Table 3. What are the exact removal and preservation criteria and judgment-aggregation rules? Which inpainting model, mask-generation procedure, and dilation settings were used?

Q3: This paper evaluates fine-tuning methods on older backbones in Table 4. Were these evaluations restricted to originals containing visible logos? Please report the original visibility rates and removal results conditioned on those samples.

Reviewer_4BDhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduce LU-500, a benchmark to evaluate unlearning methods for text-to-image for the logo removal problem. The benchmark includes 9584 examples spanning 500 logos with explicit and implicit tracks. The evaluations cover inference-time, fine-tuning, and detect-and-inpaint methods. The overall message: there is a trade-off between logo removal and visual preservation. While detect-and-inpain can balance the trade-off well, it is show output-level effect, not model itself unlearning effect.

Critical strengths
  • The problem setting is interesting (although there is one questionable concern about the "success" in the question section)
  • The benchmark is large-scale with human verification of image quality
  • The evaluation cover a wide range of unlearning methods and text-to-image architectures
Critical weaknesses
  • The benchmark seems model-dependent (i.e., SD3). Although there is a small set for FLUX, most of observations are SD3-conditioned. This raises a concern about an unlearning can optimize using SD3 to achieve high benchmark sores.
  • The metrics include a few limitations: (1) they are strongly relying on logo-region, what if the logo appears somewhere else after unlearning? We still have low logo-detect scores, and potentially high similarity of background since logo could be small; (2) What if you replace logo with unnatural patch? this give you perfect low logo-detect scores and high similarity (as the similarity only cares about background region)
Questions for authors
  • Why background-based method use LPIPS but not CLIP like logo-region metrics?

  • What is the explicit threshold used to translate similarity-based scores to binary decision to compute (joint) success? I unable to to find it in the paper (correct me if I am wrong)

  • Why don't we use VLM agent as judge instead of constructing the model-dependent benchmark? The good side about VLM agent as judge is that it is model-agnostic. The down side is the interpretability, where robust agentic design could help.

  • A broader question: would you consider logo removal and keep product appearance as "success removal"? This removal could result in another copyright issue.

  • Minor comment: I find that figure 1 and figure 4 are very low resolution. They are potentially AI-generated by ChatGPT (verified by https://openai.com/research/verify/). Do you use AI to generate them? If yes, you should mention it explicitly in the Appx. A

Reviewer_mfx6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors of the paper introduce a human-verified benchmark of 9,584 prompts, queries and associated images with logos for surprising company logos in text-to-image generation models. They propose metrics for the local logo removal and non-logo-preservation, evaluation of inference-time interventions, the ProLu prompt rewriting and a detection+inapint tool with a small FLUX portability study. While the general methodology and results are encouraging, the paper lacks the reproducibility standards of a ML venue such as ICLR.

Critical strengths

It is a well established study design (generally spoken) targeting localised brand leakage and separation of explicit and implicit prompt tracks, consisting of local-region, masked-preservation, full-output, fixed-region, detector-audit and a human blinded annotation. The strengths are - at the same time - the biggest weakness of the paper...

Critical weaknesses

... a lot of metrics are heavily under defined and can be only guessed from reading the paper, I will just list a few here:

  • human annotation is blinded - but who annotated with what tool? No characteristics, no code, no statement on what "blindness" means: double-blind? single blinded?
  • you use clip similarity, but "similarity" is not a scientific metric. From your methods section I can only "guess" which model and similarity you mean: probably Vit-B32 / and you mean cosine similarity? You need to be explicit about the checkpoints... Also OpenAI's clip was trained on opaque dataset - so a cleaner solution would be to use open variants of clip, e.g, OpenCLIP or alike
  • You state you use FLUX.1-dev, but cite Flux 1.1 pro 2024...
  • what compute did you use? Have you downloaded full bf16 weights, used quantised version? programming environment?
  • you use pearson requiring data normality and linear assumptions but only very briefly report data and methodology around it...
  • and many more such ambiguities / vague descriptions
Questions for authors

I really want to like the paper, it reads nicely as a "storyline", but it lacks the fundamental description of a reproducible, scientific paper (see weaknesses).

Other minor questions:

  • what are the intended license, redistribution terms, and trademark handling for LU-500 images, prompts, and company metadata?
  • section 4.3 says ProLU can remove textual triggers, but it also removes or alters the user prompt, so the comparison basis should be clearer
  • the paper references released code but gives no anonymous link or evaluation configuration (anonymous GitHub, anonymous hugginface possible without losing double-blindness)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.