- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
In this work, the authors show that generation failures, meaning undesirable, low-quality, or invalid outputs, can be detected before the end of generation using predictions derived from intermediate statistics. In particular, the authors show that the reconstruction residuals and feature entropy of intermediate embedding reconstruction using AutoEncoders and BlockSparseFeaturizes. Their method allows for both reduction in the error rate and efficiency savings in compute, by stopping generations destined for failure before they finish.
Critical strengths
- The method is principled and makes logical sense
- The experiments are convincing, and the detection rate and compute savings are strong
Critical weaknesses
- While the detection component seems strong, from what I can see the authors did not evaluate performance gains, not just efficiency savings, that could be obtained with this method.
- Early failure detection has several prior works in the literature, which I did not see the authors compare with e.g.
- [IJCV 2026] Betti et al., Hallucination Early Detection in Diffusion Models (arXiv:2604.20354)
- [CVPR 2026] Cui et al., Diffusion Probe: Generated Image Result Prediction Using CNN Probes (arXiv:2602.23783)
- [NeurIPS 2025] Römer et al., Failure Prediction at Runtime for Generative Robot Policies (arXiv:2510.09459)
- The "failure labelling" is entirely model-derived (low entropy, high perplexity), instead of data derived i.e. LDDT, DockQ, etc.
Questions for authors
- Is the failure detection metric complementary to the confidence head of protein models, or is its detection set a strict subset?
- Would performing Reinforcement Learning to avoid failed generations using the proposed detector improve performance?
- Could the authors compare with prior work in failure detection?
- Does the method work for data-derived badness labels? i.e. LDDT, DockQ, etc.