acceptodds
Withdrawn from ICLR 2027

Predicting Generation Failures from Intermediate Representations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generative models based on diffusion or flow matching can produce undesirable, low-quality, or invalid outputs, which we broadly refer to as generation failures. Detecting such failures before the generation is complete can help identify bad samples early and avoid spending additional computation on samples that will eventually be discarded. In this work, we study whether generation failures can be forecast from intermediate representations of the generative model, and how early such signals become useful. We find that simple reconstruction-based models, including vanilla autoencoders and Block Sparse Featurizers (BSF), expose markers of eventual generation failure through their reconstruction residuals and feature usage patterns. On La-Proteina, a flow matching based protein generative model, these signals become predictive very early (first quarter) in generation and outperform the sequence-based baselines at the same stage. For low-entropy failures, we further find that the representation-level signal appears before the same failure becomes visible in the model's predicted sequence. We see a similar trend in image generation, where the reconstruction-based detectors perform comparably to or better than existing detectors on several datasets while 15-45% of the generation process still remains. Together, these results suggest that information about eventual generation failure can become accessible inside the model well before the final sample is produced, and in some cases before the failure is expressed in the generated output. More broadly, the recurrence of this behavior across protein and image generation suggests that reconstructing intermediate representations can serve as a simple, general tool for studying when generation failures become predictable.

Primary area:
interpretability and explainable AI
Keywords:
Hallucinations, early failure detection, diffusion, flow-matching, sparse dictionary learning, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eXmyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this work, the authors show that generation failures, meaning undesirable, low-quality, or invalid outputs, can be detected before the end of generation using predictions derived from intermediate statistics. In particular, the authors show that the reconstruction residuals and feature entropy of intermediate embedding reconstruction using AutoEncoders and BlockSparseFeaturizes. Their method allows for both reduction in the error rate and efficiency savings in compute, by stopping generations destined for failure before they finish.

Critical strengths
  • The method is principled and makes logical sense
  • The experiments are convincing, and the detection rate and compute savings are strong
Critical weaknesses
  • While the detection component seems strong, from what I can see the authors did not evaluate performance gains, not just efficiency savings, that could be obtained with this method.
  • Early failure detection has several prior works in the literature, which I did not see the authors compare with e.g.
    • [IJCV 2026] Betti et al., Hallucination Early Detection in Diffusion Models (arXiv:2604.20354)
    • [CVPR 2026] Cui et al., Diffusion Probe: Generated Image Result Prediction Using CNN Probes (arXiv:2602.23783)
    • [NeurIPS 2025] Römer et al., Failure Prediction at Runtime for Generative Robot Policies (arXiv:2510.09459)
  • The "failure labelling" is entirely model-derived (low entropy, high perplexity), instead of data derived i.e. LDDT, DockQ, etc.
Questions for authors
  • Is the failure detection metric complementary to the confidence head of protein models, or is its detection set a strict subset?
  • Would performing Reinforcement Learning to avoid failed generations using the proposed detector improve performance?
  • Could the authors compare with prior work in failure detection?
  • Does the method work for data-derived badness labels? i.e. LDDT, DockQ, etc.
Reviewer_KtH8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

I this paper, authors introduce a new method for detecting hallucinations in generative models (diffusion/flow models). They propose 2 markers for detecting such a failures - reconstruction error and an entropy-based marker + their fusion. The evaluation show improvement over recent methods in two tasks of protein design and image generation.

Critical strengths
  • The proposed idea shows performance gains against recent works in the field
  • I have a mixed feelings about the quality of evaluation protocol. While authors do use state-of-the-art, novel protein design model, the image-related studies significantly lag behind. For problems such as hallucination detection datasets such as MNIST should be retired and not reported anymore apart from some explanatory experiments. Nevertheless, evaluation with 2 distinct domains is a big plus
Critical weaknesses
  • The main methodological idea, using autoencoder reconstruction error as an outlier score, is a well-established anomaly-detection technique. This work applies it to the activation space of large model for detecting outliers which turn into hallucinations in this case. The contribution is therefore mainly empirical: showing that this combination works for diffusion and flow-matching models. That would be fine if the paper analysed why the signal appears, but for example, the second marker (feature entropy) and the fusion of the two markers have no motivation beyond validation performance.
  • Connected to the previous weakness, the paper lacks motivation for the entropy-based marker. To be honest I’m not sure why we should expect the model to use sparse features more or less broadly when generating failures vs. not. One of the main claims “Detecting such failures before the generation is complete can help identify bad samples early and avoid spending additional computation on samples that will eventually be discarded.“ Is not really supported by numbers. Even assuming a perfect detector, the gain from detecting failures is the trajectory left x failure rate (in terms of proteins here 0.75 * 0.073 ~ 5.4%). This does not include the fact that while discarding generations with suboptimal detector, we also discard many valid generations that need to be regenerated - leading to the reported 1.17 steps boost (out of 400 steps - so around 0.3%) and Fig 3. shows that anything beyond 6.5% discards actually adds compute (including the 20% discard discussed in the same section).
  • While the interpretability of the selected BSF concepts is interesting, it would be interesting to evaluate further what those concepts actually mean in practice

Small details:

  • There is a small issue with the references, they highlight often previous word or its part, not only the author name (e.g. . Alternatively, Tian et al. (2026) in L125)
Questions for authors
  • What is the motivation behind the entropy-based marker?
Reviewer_3ycQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an approach that can predict in advance if a (diffusion or flow-based) generative model produces a bad (unintended) final outcome. The approach involves reconstructing internal activations at earlier generative steps to detect failures at the outcome. If the internal activations are not generated properly and does not cross threshold determined by a validation set then it is considered more likely that the final outcome will be poor. This approach is evaluated on image and protein sequence dataset with success. I do not like the dependence on validation set and the most convincing result for me is early detection of low entropy protein failures.

Critical strengths
  • The presented detectors successfully identify bad outcomes before the generation ends from activations. It is an interesting approach.
  • The paper provides cross-domain evidence of the method.
  • It is a simple approach and a useful analysis with practical relevance.
Critical weaknesses
  • Limited failure definitions: It is difficult to say what a bad outcome in some domains than other. For images it is quite clear and visible but that is not the case in all domains. This is not a criticism of the approach but just a general limitation.
  • The direction and approach is nice but it leads to uneven results. Not all baselines are clearly beaten.
  • There is a clear dependency on the validation set and I am not confident how usefulness of this approach.
Questions for authors
  • How do you know if the reconstruction error indicate generation failure rather than valid but unusual outputs (where do you draw this line)?
  • Why should reconstruction of activation predict the final quality of the output?
  • How well does the method generalize to failure types that are not represented in the validation set?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.