- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors present COFAT, a correlation-aware query design for controllability evaluation that removes the penalization of infeasible requests. When testing a specific attribute, COFAT sets the other attributes to their conditional mean given the training data distribution, instead of their marginal mean. The authors test it on tumor images, molecules, and a synthetic generator, showing that measured obedience and validity rise substantially when only the query is changed, claiming that many controllability gaps may be artifacts of query design rather than model failures.
Critical strengths
- I think that treating the query as a source of measurement error is an interesting reframing of controllability evaluation. The mathematics of the paper appears solid, and the authors are honest about the limitations of their approach.
- Experiments are run on 3 real tasks (lesion inpainting, lesion insertion and molecule generation) and compared against seven baseline designs, showing consistent gains over OFAT-µ across domains.
Critical weaknesses
- The gains of COFAT over OFAT-µ are clear. However, compared to other correlation-aware designs, the results are close, and it is difficult to draw a firm conclusion without significance tests, repeated runs with different seeds, or confidence intervals.
- The claim that "many reported controllability gaps may be artifacts of query construction" would require re-evaluating existing published controllability results. As it stands, the evidence comes from a narrow, controlled set of experiments and is too limited to support such a general claim.
Questions for authors
- What is the computational cost of COFAT compared with the other designs? What is the runtime or complexity advantage (if any)? How large is the practical difference, and when should a practitioner prefer COFAT over KNN, whose results are close?
- COFAT evaluates "change s as the data does". In which applications would rare but valid combinations be needed, and when should OFAT or COFAT be used? Is COFAT meant to replace OFAT or complement it?
Summary of our actions (FAR3)
We thank the reviewer for finding the reframing of the query as a source of measurement error interesting, the mathematics solid and our discussion of limitations honest, and for noting the consistent gains over OFAT-μ across three real tasks and seven baseline designs. Following the comments we:
In short: COFAT's gain over the usual OFAT-μ design is significant on all 10 attributes and in every training seed. Among the correlation-aware designs the differences are small and go both ways, so we do not claim that COFAT always obeys best. Its consistent advantages are a lower bias and practical ones: it needs only the correlation matrix, not the training records themselves, so it works when records are few or cannot be shared (e.g. clinical patient data); it stays well defined at and beyond the edges of the data percentiles; it has no parameter to tune; and it is as cheap as OFAT-μ (Question 1, which also says when KNN is preferable).
A detailed response (response 1/2, response 2/2) and full experimental details (experiment 7, experiment 8) are in the follow-up comments below.