acceptodds
Withdrawn from ICLR 2027

More HASTE, More Speed: Adaptive Sequential Testing for Efficient Text-to-Image Diagnostic Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diagnostic benchmarks test whether text-to-image (T2I) models satisfy conditions such as object presence, attribute binding, and spatial relations across repeated generations. They typically allocate identical image budgets to every prompt, although obvious successes and failures need less evidence than conditions near the validity threshold. We introduce HASTE (Hierarchical Adaptive Sequential Testing for Evaluation), which varies the sample count without omitting any benchmark condition. Conditions are organized by semantic prerequisites: for example, "a car is present" must hold before "the car is red" can hold. HASTE evaluates prerequisite conditions first and uses their results to inform priors for more specific conditions. It then updates each condition's posterior with binary vision-language model judgments and stops when there is sufficient evidence that its success probability lies above or below the benchmark's validity threshold. A minimum sample requirement limits premature stopping, while conditions uncertain at the sample cap use the fixed-budget decision rule. Across six T2I models, HASTE achieves average speedups of 2.60× on DSG and 4.93× on FailureAtlas, with failure-set Jaccard agreement above 0.97 relative to full-budget evaluation. At matched average image budgets, HASTE attains 0.9806 Jaccard agreement on DSG versus 0.9220 for uniform sampling, and 0.9794 on FailureAtlas versus 0.6558 for uniform sampling.

Primary area:
generative models
Keywords:
text-to-image generation, generative model evaluation, structured evaluation, sequential testing, adaptive sampling, failure discovery, vision-language model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LfzPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an evaluation pipeline that accelerates diagnostic evaluation for text-to-image generation. It organizes evaluation conditions into a graph structure and utilizes the estimated success probabilities of parent nodes as priors for child nodes. It then performs early stopping when sufficient evidence is gathered. Compared with full-budget evaluation, the method accelerates the evaluation process while maintaining high agreement with the full-budget evaluation results.

Critical strengths

The idea of structuring evaluation conditions into a graph and utilizing parent information to provide priors for more fine-grained conditions intuitively makes sense. I see the paper as a well-executed study of this evaluation acceleration pipeline, which may have practical benefits in making T2I evaluation more efficient. The paper also presents the idea clearly.

Critical weaknesses
  1. The main novelty of the method is the hierarchical testing, but in the ablation study, Flat Sequential achieves a very similar speedup and reasonably close Jaccard agreement compared to the hierarchical version. It remains unclear how critical the hierarchical design is in practice. Besides, the sample gate seems to be a relatively simple design choice. Overall, I am not fully convinced that the methodological novelty and contributions are strong enough.
  2. The method involves several hyperparameters, such as the parent weight, prior strength, and the sample gate. Although the authors provide some hyperparameter studies in the appendix, the main hyperparameter search is conducted with one model on one dataset. It is still unclear how sensitive the method is to these hyperparameters or whether the same settings generalize across different evaluation scenarios.
Questions for authors

Please refer to the critical weaknesses section.

Reviewer_iA7nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the computational waste caused by allocating identical fixed sample budgets to every condition in diagnostic evaluation of text-to-image models, and proposes HASTE: an adaptive sequential evaluation method based on a condition dependency DAG. The core idea is to evaluate parent conditions first in topological order, use a weighted mixture of the minimum of parent posterior means and the global mean as the Beta prior for child conditions, sequentially update the posterior with VLM binary judgments, stop early when the posterior probability crosses the threshold provided a minimum sample requirement is met, and fall back to the original fixed-budget rule for undecided conditions. Across six T2I models on DSG and FailureAtlas, HASTE achieves image-generation speedups of 2.60× and 4.93× respectively, with failure-set Jaccard agreement above 0.97.

Critical strengths

1、the combination of "graph-guided prior + sequential early stopping" is clean and effective. 2、the matched-budget controlled experiment hits the nail on the head.

Critical weaknesses

1、agreement with the fixed-budget reference is explicitly sacrificed, but the assumption that "the reference itself is correct" is never tested. The entire evaluation chain uses K=50 empirical pass rates and τ=0.5 as ground truth, yet K=50 itself has high variance for boundary conditions near τ. Whether the ~2% disagreement between HASTE and the fixed-budget reference reflects "closer to the true value" or "introduction of new bias" is not addressed by any ground-truth calibration experiment (e.g., using a large K or human annotations as an external reference). This makes the interpretation of Jaccard≈0.98 ambiguous. 2、the interaction between VLM judge error and adaptive stopping is not analyzed. HASTE's stopping boundary depends on the sequential accumulation of VLM binary judgments, and Qwen3-VL-8B as a judge has its own false-positive and false-negative rates (the human-agreement numbers in Appendix J are not reported in the main text). When the VLM has systematic bias, sequential early stopping may "lock in" wrong conclusions faster than fixed-budget evaluation — because early stopping amplifies the influence of a small number of samples. The authors do not discuss the sensitivity of HASTE's decision fidelity to VLM reliability.

Questions for authors

1、when the VLM judge has systematic bias (e.g., systematically over- or under-calling a certain type of attribute), does HASTE's hierarchical prior amplify the parent's bias into the child, potentially reaching wrong conclusions faster than flat sequential methods? Could you provide a "VLM bias injection" robustness experiment, or give theoretical conditions under which prior conduction does not amplify bias?

Reviewer_6MVqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HASTE, an adaptive sequential testing procedure to reduce the number of generated images and VLM queries needed when running diagnostic text-to-image (T2I) benchmarks such as DSG and FailureAtlas. Instead of generating a fixed budget of K images (here K = 50) for every condition and checking whether the empirical pass rate reaches a threshold tau (e.g., tau = 0.5), HASTE evaluates conditions in topological order along the benchmark's prerequisite DAG. For each child condition v, it takes the minimum posterior mean g_v = min_{u in pa(v)} m_u over its parents, blends it with a global mean mu_g = 0.5 via mu_v = rho * g_v + (1 - rho) * mu_g, and forms a Beta(kappa * mu_v, kappa * (1 - mu_v)) prior. As binary VLM judgments arrive sequentially, HASTE updates the Beta posterior and stops early once at least n_min = 5 samples have been observed and the posterior probability Pr(theta_v >= tau | D_t) falls outside (epsilon, 1 - epsilon) with epsilon = 0.05; otherwise it falls back to the empirical decision at t = K. The authors distinguish between independent-stream benchmarks (FailureAtlas, where image and VLM costs both equal sum_v T_v) and shared-stream benchmarks (DSG and T2I-CoReBench, where multiple questions share generated images per prompt so image cost is sum_x max_{v in V_x} T_v while VLM cost is sum_v T_v). Experiments across six open-weight T2I models (plus appendices on T2I-CoReBench, GPT-Image-1.5, and alternative VLMs) report average image speedups of 4.93x on FailureAtlas and 2.60x on DSG with failure-set Jaccard above 0.97.

Critical strengths

1.Practical cost formulation separating shared-stream vs. independent-stream bottlenecks. The distinction in Section 4.1 between independent image streams (C_image = sum_v T_v) and shared prompt-level image streams (C_image = sum_x max_{v in V_x} T_v, C_vlm = sum_v T_v) is simple, clean, and practically relevant. Even when the slowest proposition per prompt caps image-generation savings on DSG at 2.60x, stopping already-decided propositions early still cuts VLM queries by 6.10x (Table 3), which matters for fast/distilled generators like SDXL-Turbo where VLM inference dominates wall-clock cost (Appendix E.4). Thorough experimental hygiene with deterministic trace replay and online verification. Replaying identical pre-generated K = 50 pools across 5 SHA-256 observation-order seeds removes generation noise when comparing sequential stopping rules (Section 5.2, Appendix G.1), and the live online check on FailureAtlas in Appendix H.5 (Table 16) confirms that the offline replay numbers match actual end-to-end execution.

  1. Thorough experimental hygiene with deterministic trace replay and online verification. Replaying identical pre-generated K = 50 pools across 5 SHA-256 observation-order seeds removes generation noise when comparing sequential stopping rules (Section 5.2, Appendix G.1), and the live online check on FailureAtlas in Appendix H.5 (Table 16) confirms that the offline replay numbers match actual end-to-end execution.
Critical weaknesses
  1. Conceptual and mathematical contradiction in how the parent prior works (Eq. 10 vs. FailureAtlas traversal and Figure 5). In Section 4.2 (lines 221-234), the paper motivates hierarchical priors by arguing that a child condition can hold only if its prerequisites hold (theta_v <= theta_u), so a failing parent suggests a low child success probability and helps identify child failures faster. However, prerequisite dependency is a one-sided upper bound (theta_v <= theta_u): a parent failing implies the child fails, but a parent succeeding ("a basketball is present") does not imply that a harder child specialization ("the basketball is square") is likely to succeed. Yet Eq. (10) sets mu_v = rho * g_v + (1 - rho) * mu_g, which symmetrically pushes mu_v above 0.5 whenever g_v > 0.5 This creates two concrete problems: First, in FailureAtlas, a child node is only entered into the evaluation frontier after its parent is verified as VALID (Section 5.1 line 281; Appendix M line 2171). Thus, for every child node actually evaluated in FailureAtlas, g_v is high, meaning the hierarchical prior never lowers mu_v to catch failures faster—it exclusively biases child nodes upward toward VALID. Second, plugging the default hyperparameters from Table 14 (rho = 0.7, mu_g = 0.5, kappa = 5.0, n_min = 5, epsilon = 0.05, tau = 0.5) into Eqs. (8)-(13) directly contradicts Figure 5 (page 15). Suppose a parent passes at t = 5 with 5/5 successes, giving m_u = (1 + 5)/(2 + 5) = 6/7 (approx. 0.857). Its child gets mu_v = 0.7 * (6/7) + 0.3 * (0.5) = 0.75, so (alpha_v, beta_v) = (3.75, 1.25). If the child is a complete failure ("square basketball", 0% pass rate) and observes s_5 = 0, f_5 = 5, its posterior at t = 5 is Beta(3.75, 6.25). The tail probability is p_{v,5} = 1 - I_{0.5}(3.75, 6.25) = 0.213, which is nowhere near the stopping threshold epsilon = 0.05 (in fact, starting from Beta(3.75, 1.25), even consecutive failures cannot bring 1 - I_{0.5}(3.75, 1.25 + t) below 0.05 until t = 9, whereas a flat Beta(1,1) prior stops at t = 5 since 1 - I_{0.5}(1, 6) = 0.0156 <= 0.05). Yet Figure 5 explicitly labels "square basketball" (0% pass rate) and "textile lettuce" (12% pass rate) as stopping as HASTE: Failure (n = 5). How is p_{v,5} <= 0.05 achieved at n = 5 for a child of a valid parent under Table 14's parameters?

  2. Unexplained discrepancy between Table 2 and Table 15 on DSG, where the flat prior outperforms the default hierarchical prior. In Appendix G.7 (Table 15, page 30), the authors report a 372-configuration grid search on DSG (SD 1.5, tau = 0.5, averaged over the 5 observation orderings). In Table 15, the paper's chosen "Conservative Default (Min, rho = 0.70, mu_g = 0.50, kappa = 5.00)" achieves a speedup of 1.3633x and Failure Jaccard of 0.9831, which is outperformed on both speedup and Jaccard by the "Flat Prior Anchor (Beta(1,1))" at 1.3777x speedup and 0.9915 Jaccard (and the fastest setting in the entire 372-configuration sweep is 1.4282x). However, in the main paper (Table 2, page 7, and Table 11, page 24), HASTE on DSG (SD 1.5, tau = 0.5) with the same Table 14 default configuration is reported as having a 1.70x speedup and 0.9736 Jaccard. Similarly, in the FailureAtlas ablation (Table 4, page 9), adding the hierarchical prior on top of the sample-gated flat Beta(1,1) model changes image speedup from 4.89x to 4.92x (a 0.6% difference). Together, these numbers suggest that the n_min = 5 gated Beta test is doing almost all of the work, while the hierarchical prior propagation adds minimal speedup (or slightly hurts on DSG in Table 15) and introduces mismatch between Table 2 and Table 15.

  3. Multiple corrupted table entries and a hallucinated author list in the bibliography. While the presentation is generally clear, there are several data/editing errors that need cleanup: In Table 7 (page 17), the Avg. Img. and Avg. Query columns are swapped for KL-CS, SPRT, GrAPL, and HASTE (e.g., HASTE's average image speedup of 2.42x and query speedup of 6.80x from Table 6 are placed in reversed columns). In Table 17 (page 35), the K = 50 and K = 75 entries are swapped for SPRT on FailureAtlas (6.04x at K = 50 vs. 4.19x at K = 75, whereas Table 1 reports 4.19x at K = 50), and for Hoeffding-CS, SPRT, and GrAPL on DSG (1.33x/1.21x, 2.87x/2.17x, and 2.43x/2.08x vs. Table 2's K = 50 values of 1.21x, 2.17x, and 2.08x). On page 9 (line 462), FailureAtlas speedup across tau is written as "1.99x to 1.82x" instead of 4.82x. In the References (lines 644-647), the CVPR 2024 paper "FlashEval: Towards fast and accurate evaluation of text-to-image diffusion generative models" is cited with a fabricated author list ("Linqi Zhao, Shengyu Zhang, Jiawei Dong, Haozhe Yang, Lele Chen, Zhaowei Zheng, Lingxuan Zhu, Meng Shen, Yujia Wang, Zhichao Wang, et al."), whereas the actual authors of that paper are Lin Zhao, Tianchen Zhao, Zinan Lin, Xuefei Ning, Guohao Dai, Huazhong Yang, and Yu Wang.

Questions for authors

1.Reconciling Eqs. (8)-(13) and Table 14 with Figure 5 and FailureAtlas: Can you walk through the exact arithmetic for how a child node in FailureAtlas (whose parent passed with m_u >= 0.5) can terminate as FAILURE at n_min = 5 (as shown for "square basketball" and "textile lettuce" in Figure 5) when rho = 0.7, mu_g = 0.5, kappa = 5.0, and epsilon = 0.05? With those hyperparameters, alpha_v = kappa * (rho * g_v + (1 - rho) * mu_g) >= 2.5 (and approx. 3.75 when g_v approx. 0.86), so even s_5 = 0, f_5 = 5 yields p_{v,5} = 1 - I_{0.5}(alpha_v, beta_v + 5) > 0.05 (and if s_5 = 1, p_{v,5} is even larger). Did the code used to produce Figure 5 and the FailureAtlas results use a different prior formula, a smaller kappa, or treat root/child nodes differently than written in Section 4.2? 2. Discrepancy between Table 2 and Table 15 on DSG (SD 1.5): Why does the default configuration (Min, rho = 0.70, mu_g = 0.50, kappa = 5.00) achieve 1.3633x speedup and 0.9831 Jaccard on DSG (SD 1.5, tau = 0.5, 5 seeds) in Table 15 (where Flat Beta(1,1) gets 1.3777x and 0.9915 Jaccard), whereas Table 2 and Table 11 report 1.70x speedup and 0.9736 Jaccard for the same model and benchmark? Also, what does the ablation of Flat Sequential (Beta(1,1), n_min = 5) vs. HASTE look like across all 6 models on DSG (analogous to rows c and d of Table 4)? 3. One-sided prerequisite upper bound: Since semantic prerequisite means theta_v <= theta_u, why not clamp the prior contribution upward via g_v = min(mu_g, min_{u in pa(v)} m_u) so that failing parents pull the child prior down while succeeding parents leave the child at the neutral prior mu_g = 0.5 rather than inflating alpha_v?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.