- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies why conditional generators fail on unseen conditions. It builds on prior work that divides out-of-distribution (OOD) error into three parts: target geometry, embedding distortion, and training fit.
The authors revisit the theoretical analysis and clarify the conditions under which some bounds hold. They also introduce two benchmarks based on letter point clouds and single-cell perturbation data. These benchmarks allow more direct measurement of the error terms using Wasserstein distances.
Experiments show that the geometric term strongly correlates with OOD error in both domains. Reweighting and representation regularization have limited effects, while adding real target examples greatly reduces the error. The paper aims to identify when generation errors can be improved by model tuning and when additional data may be needed.
Critical strengths
1. Well-defined and Measurable Benchmarks
The paper introduces two well-designed benchmarks that allow direct measurement of Wasserstein-based error terms, rather than relying on proxies. This provides a clear and controlled setting to study the sources of OOD generation error. I think this is a useful contribution, as it makes the analysis more concrete and easier to validate.
2. Comprehensive Experimental Validation
The experiments are thorough and well organized. The authors evaluate two different domains and include held-out evaluation, fidelity checks, ablation studies, and robustness tests across different settings. They also compare different interventions and show how adding target examples affects the error. These experiments provide good support for the main empirical findings.
Critical weaknesses
1. Limited Generality and Novelty of the Main Finding
The main finding is somewhat expected under the barycentric generation setting. If a model can only combine existing training distributions, changing their weights cannot reach targets outside the reachable set. While the paper provides a careful validation of this result, it is unclear how much new insight this offers beyond the existing theory. The experiments also use a small per-point MLP with barycentric conditioning. It remains unclear whether the same geometric limitations apply to more general conditional generators.
2. Limited Practical Applicability
The paper aims to use geometric information to identify OOD generation limits before training. However, computing the proposed diagnostic requires information about the target distribution. The experiments use Wasserstein distances involving held-out target samples, which may not be available in real OOD scenarios. Although the authors acknowledge this limitation, it remains unclear how the diagnostic can be applied when the target distribution is unknown. This limits the practical impact of the proposed approach.
Questions for authors
1. Generality to Large-scale Generative Models
The experiments mainly use a small per-point MLP and low-dimensional point clouds. How do the authors envision applying this framework to mainstream large-scale generative models, such as diffusion or flow-based models? Would the proposed geometric diagnostic and the conclusions about irreducible OOD error still apply to these models?
2. Practical Use of the Diagnostic
The paper proposes using Term I to identify OOD generation limits before training. However, the experiments use information from held-out target samples to compute Wasserstein distances. How could Term I be estimated when no target samples are available? How practical is the proposed diagnostic in a true zero-shot setting?
Dear Reviewer E5xS,
Because of the discussion periods will close soon. Dont know if my comments have solved all your concerns ?
Thank you again for your time and thoughtful feedback.