acceptodds
Under review as a conference paper at ICLR 2027

Does Memorability Guidance Transfer Beyond the Guiding Critic? Evidence from Latent Diffusion

Abstract

Image memorability is the likelihood that an image will be remembered after viewing. Neural memorability predictors estimate this likelihood and can guide diffusion models toward images with higher predicted memorability. We call the predictor that provides this guidance the guiding critic. Prior work shows that such guidance can also raise average scores from predictors not used for guidance. What remains unclear is (i) whether improvements in predicted memorability transfer consistently beyond the guiding critic at the individual-image level, (ii) whether these improvements arise from the intended guidance direction rather than generic changes introduced by generation or reconstruction, and (iii) whether gradient-based guidance justifies its added computation compared with simple sampling alternatives. We investigate these questions using training-free universal guidance in latent diffusion for both text-to-image generation and real-image editing. The critic supplies gradients during sampling, while the other predictors serve as held-out evaluators that only score the final images. With AMNet as the critic, guidance raises the mean ResMem, ViTMem, and PerceptCLIP scores by 2.87%, 1.69%, and 1.67% on 200 COCO prompts fixed before generation, while reducing CLIP prompt similarity by 1.88%. These gains hold across four seeds on DrawBench and COCO, and on DrawBench they hold whichever of AMNet, ResMem, or ViTMem acts as the critic. At the image level, however, the held-out predictors often disagree, with all three scores rising together on only 69 of 200 COCO image pairs. To separate the effect of guidance from the changes that noising and denoising make on their own, we edit real SUN photographs and compare each edit with a matched unguided reconstruction. Forward guidance raises and reverse guidance lowers the held-out scores, while a random push of the same size has near-zero effect, so the change follows the direction of the critic's gradient. Generating four unguided images and keeping the one the critic scores highest is competitive with guidance on the held-out scores, keeps prompt similarity higher, and runs faster in our benchmark. A five-prompt diagnostic on Stable Diffusion XL shows that predictor scores can rise even when images are visibly malformed. Together, these results show that memorability guidance produces real but modest gains beyond its critic, which hold on average but not reliably per image, cost prompt fidelity, and offer little over selecting the best of a few samples.

Primary area:
generative models
Keywords:
image memorability, diffusion models, guidance, evaluation, transfer, controlled experiments

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iedzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates whether memorability guidance using one predictor improves scores from other predictors. It applies gradient-based guidance to latent diffusion models for image generation and editing. Experiments show improvements across multiple predictors, but also reveal limitations in image quality, text alignment, and consistency.

Critical strengths
  1. The paper investigates whether memorability improvements guided by one predictor can generalize to other predictors. This is an important question, as optimizing a single predictor does not necessarily indicate an actual improvement in image memorability.

  2. The experiments include multiple memorability predictors, datasets, and guidance settings. The analysis of both average improvements and individual-image agreement provides useful insights into the consistency of memorability guidance.

  3. The matched reconstruction, reverse guidance, and random perturbation controls help distinguish the effects of memorability guidance from changes introduced by image reconstruction or generic perturbations.

Critical weaknesses
  1. The paper relies entirely on pretrained memorability predictors without conducting human recognition experiments on the generated or edited images. Improvements in predicted memorability do not necessarily translate into actual improvements in human memory.

  2. All 4 memorability predictors are trained on the same LaMem dataset, which may introduce shared biases. Therefore, improvements across held-out predictors do not necessarily demonstrate generalization beyond their shared training distribution.

  3. The qualitative results show limited perceptible changes between the unguided and guided images, particularly in Figure 3, where neither semantic nor pixel-level differences are clearly noticeable. Despite improvements across multiple memorability predictors, it remains unclear whether these subtle changes have meaningful effects on human memory or offer practical value.

  4. The SDXL experiments show that memorability predictor scores can increase even when the generated images contain obvious visual artifacts and structural distortions. This raises concerns about the reliability of predictor-based optimization, as higher memorability scores do not necessarily correspond to perceptually meaningful or visually plausible improvements.

  5. The proposed method mainly applies existing universal guidance techniques with pretrained memorability predictors, without introducing substantial methodological innovations. The contribution is primarily an empirical evaluation of existing approaches rather than a new memorability manipulation method.

  6. Most experiments are conducted using Stable Diffusion 1.4, while the SDXL evaluation includes only 5 prompts. It remains unclear whether the findings generalize to more recent diffusion or flow-matching models.

Questions for authors
  1. Given that all memorability predictors are trained on LaMem, how can the authors determine whether improvements across held-out predictors reflect meaningful memorability changes rather than shared model biases?

  2. The paper does not include human recognition experiments on the generated or edited images. Is there any behavioral evidence supporting the claim that higher predicted memorability leads to improved human memory performance?

  3. In Figure 3, the guided images show very limited perceptible changes despite significant improvements in predicted memorability. What visual or semantic properties are actually modified by the guidance, and how might these changes affect human memory?

  4. The SDXL experiments show increased memorability scores even when the generated images contain obvious visual artifacts. How do the authors distinguish genuine memorability improvements from the exploitation of predictor sensitivities?

  5. Given that best-of-four sampling achieves comparable results with lower computational cost and better text-image alignment, what practical advantages does gradient-based memorability guidance offer?

Reviewer_oGqYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines whether memorability guidance improves scores from predictors other than the guiding critic. Experiments cover text-to-image generation and real-image editing, with multiple seeds, rotating critics, and matched controls. Held-out scores rise on average, but the predictors often disagree on individual images. Guidance also reduces prompt alignment and adds some source-preservation loss. Best-of-four unguided selection achieves competitive held-out scores with better prompt alignment and lower measured latency. These findings concern predicted memorability; human memory was not measured for the generated or edited images.

Critical strengths
  1. The experiments test transfer beyond the guiding critic. Multiple seeds and rotating critics provide evidence that average held-out score gains are not specific to AMNet.
  2. The editing controls help isolate the effect of guidance. Matched reconstruction, reverse guidance, and norm-matched random controls help distinguish changes caused by reconstruction from those that follow the critic’s gradient direction.
  3. The evaluation considers both individual images and practical costs. The per-image analysis goes beyond average scores, while the best-of-four comparison examines whether guidance justifies its added computation and reduced prompt alignment.
Critical weaknesses
  1. The paper should explain more directly what these experiments establish beyond earlier memorability-guidance studies. Most systematic results also come from one SD1.4 pipeline; the five-prompt SDXL diagnostic provides limited evidence about other models. This limits the scope of the conclusions, though it does not by itself make the audit insufficient.

  2. The size of the per-image disagreement remains unclear. Table 9 counts the signs of predictor changes, but does not show how large the opposing changes are in the Mixed cases. A very small decrease and a substantial decrease are treated alike. Reporting the change distributions and checking whether disagreement persists after treating near-zero changes as negligible would make this finding easier to interpret.

  3. Table 3 supports calling best-of-four competitive, particularly given its better prompt alignment and lower latency. However, paired differences and confidence intervals would help assess the stronger abstract claim that guidance offers little additional benefit.

  4. Several descriptions should be made consistent.

  • Section 4.3 calls CLIP-I cosine distance, while Appendix C and Table 6 treat it as cosine similarity.
  • Section 4.2 describes an orthogonal random direction, but Equation (7) rescales a Gaussian vector without an orthogonal projection.
  • The conclusion describes 69/200 as predictor agreement. Table 9 reports 69 joint increases and 30 joint decreases, so 69/200 specifically measures how often all three scores rise together.
Questions for authors

How large are the opposing predictor changes in the Mixed cases in Table 9? Does the disagreement rate remain high when near-zero changes are treated as negligible?

Could you report paired differences and confidence intervals between guidance and best-of-four for Table 3’s held-out scores? How do these estimates, together with prompt alignment and latency, support the abstract’s conclusion about guidance’s limited additional benefit?

Reviewer_xG8mOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a method to improve the generation of memorable images. To guide the generation the paper uses memorability predictors that are used as critics during the generation process. The observation is that using one critic for guidance also improves the prediction scores of all other critics. At the same time, the image-text alignment, measured using CLIP, is reduced.

Critical strengths
  • The paper is clearly written and easy to follow
Critical weaknesses
  • I am not sure what the novelty with the presented results/methods is. When using a critic to guide the generation it is kind of clear to me that the scores of the other critics are also rising since these were all trained on the same datasets.
  • In all but one experiment only AMNet is used as the critic for steering.
Questions for authors

Q1: Why is SD1.4 used? This is a rather old model. Did you also try it with newer Models?
Q2: I don't fully understand what the intention behind the "best-of-four" experiment is. If we generate four times and take the image with the highest scores, isn't it clear that we increase the scores? At the same time it is kind of expected that the CLIP score is not decreased, because the same text encoder is used in SD1.4 as in the CLIP model. So since guidance is not interfering with the generation, the text-image alignment will be higher. Q3: Why check whether the gradient of the held-out scores follow the direction of the critics gradient? As these models are trained on the same dataset, isn't it pretty clear that the models will most likely have learned the same things? What is the benefit of knowing that the scores follow the direction of the critics gradients?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.