COFAT: Correlation-Aware Query Design for Controllable Generation

Conditional generative models should not only produce realistic samples but also reliably follow the requests they are given. Evaluating this requires checking four things: whether the requested attribute combination is *feasible*, whether the generated sample is *obedient* to it, and whether it remains *realistic* and *novel*. Existing controllability evaluations score obedience, realism, or novelty largely in isolation and take feasibility for granted. The dominant query design, One-Factor-at-a-Time with non-swept attributes fixed at their marginal means (OFAT-$\mu$), ignores dependence between attributes. On correlated attribute spaces, it therefore issues requests that rarely or never occur in the data. Generators may then ignore the requested change, undermining the purpose of conditional generation. Our approach offers a new perspective by treating query design as part of controllability evaluation. We introduce a closed-form measure of query infeasibility and COFAT, a family of correlation-aware query designs that set non-swept attributes from their conditional rather than marginal distribution, within a framework that scores feasibility, obedience, realism, and novelty jointly. For a synthetic generator that is perfectly obedient on feasible requests, switching from OFAT-$\mu$ to COFAT raises measured obedience from $0.15$ to $0.99$: the query, not the generator, can be the bottleneck. Across three tasks (lesion inpainting, lesion insertion, and molecule generation) and against seven baseline designs, COFAT raises obedience (e.g., $0.07\to 0.66$ for lesion intensity) and molecule validity ($0$), reduces copying of training samples (improving the novelty score from $33.9$ to $26.9$), and keeps realism comparable. As the target range of the swept attribute is unchanged, these gains come from removing infeasible combinations, not from easing the target. Many reported controllability gaps may therefore be artifacts of query construction, which argues for treating query design as a first-class component of controllability evaluation.