- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies FID evaluation from a new perspective: whether the extreme samples generated by the model have the correct occurrence frequencies. The paper argues that FID cannot guarantee that the tail occurrence rates are consistent with real data, and proposes a tail calibration audit method to recalibrate large generative models. Experimental results demonstrate the effectiveness of the proposed method.
Critical strengths
-
The research motivation of the paper is clear and has strong practical significance.
-
The theoretical justification of the method in the paper is thorough, the method design exhibits good logical coherence, and the delineation of the method’s scope of applicability is also accurate.
-
The experimental design is well-targeted and demonstrates the effectiveness of the proposed method.
Critical weaknesses
The motivation of this paper lies in the fact that when generative models provide samples for downstream systems, the accuracy of rare events is of great importance. One key design of the method design is the definition of severity functionals. However, no close connection has been established between these two; the severity functionals mainly consider statistical anomalies rather than the direct representation of extreme events in downstream systems. Considering that real-world extreme events are difficult to represent directly and accurately, it is suggested that the authors discuss the motivation behind the design of the severity functionals and clarify that this is not a scheme that can be fully and directly oriented toward downstream systems.
Questions for authors
There are no further questions.
I was drawn to this manuscript with strong interest due to its overlap with my own current ongoing research in generative model assessment. I have not examined the technical aspects of this manuscript carefully and so my comments do not regard the technical strengths or weaknesses of this paper. The referees and the author have had very insightful discussions about the pros and cons of the proposed strategy, which I learned a great deal from and won’t comment upon.
I do believe the topic under consideration is an important and timely one—assessing generative models and synthetic data using aggregate scores such as the FID is helpful to some extent, but it lacks the capability of providing more detailed insights to help understand the adequacy of the synthetic data and generative model for a given downstream application.
This particular manuscript considers one such scenario, when the downstream application requires accurate generation of “tail” or “extreme” events. Global metrics such as the FID generally characterize the overall distance between the real and synthetic data. As such, models that are excellent as measured by FID, could misrepresent application-relevant characteristics such as the tail behaviors. This paper proposes one possible method to remedy this issue, which has its own strengths and weaknesses.
More broadly, I believe application-driven assessment of generative models that goes beyond using standard scores such as the FID is a very important area that deserves more attention and effort from our community.