acceptodds
Under review as a conference paper at ICLR 2027

Criterion-Aware Factorial Optimization with Applications in Black-Box Text-to-Image Systems

Abstract

Prompt optimization is commonly formulated as free-form search over instructions using a scalar reward, which obscures both criterion-level failures and the effects of individual prompt operations. We instead formulate prompt optimization as a finite factorial design problem. A transformation is composed of interpretable two-level factors, and each generated output is evaluated using multiple binary criteria. We introduce Factorial Upper Confidence Bound Algorithm (FACT-UCB), which fits criterion-specific factorial logistic models to share information across configurations, estimate main and interaction effects, and enforce explicit pass-probability constraints. The method combines uncertainty-aware adaptive sampling with forced exploration to maintain identifiability. We further establish fixed-confidence recovery of the best feasible transformation and an anytime-valid stopping rule. We conduct a text-to-image case study with seven factors, 128 configurations, nine criteria, and 999 adaptive queries, the fitted model suggests concise, Markdown-structured, positively phrased instructions on an industrial 3D pre-generation dataset. Together, these results provide an explicit framework for interpretable, criterion-aware optimization of black-box generative systems.

Primary area:
optimization
Keywords:
Prompt Optimization, Factorial Design, Text-to-image Model, Generative Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nFEAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FACT-UCB, a criterion-aware prompt optimization framework that formulates prompt design as a factorial optimization problem. It decomposes prompt transformations into interpretable factors and models their effects on multiple evaluation criteria, rather than relying on a single scalar reward. The method is evaluated in a pre-3D generation case study, where prompts are optimized to generate 2D reference images that better satisfy criteria relevant to downstream 3D asset generation, such as object completeness, background cleanliness, count, orientation, and style.

Critical strengths
  1. The factorial formulation is interpretable and operationally meaningful. By modeling reusable prompt operations as factors rather than treating complete prompts as unrelated arms, the framework can estimate factor-level and interaction effects while sharing information across configurations. Keeping evaluation criteria separate also exposes trade-offs and supports criterion-wise feasibility constraints that would be obscured by a single scalar score.
  2. The paper is appropriately cautious about its empirical evidence. The authors do not treat the adaptively fitted optimum as definitively validated: they explicitly discuss model misspecification, winner’s curse, evaluator uncertainty, the lack of downstream 3D validation, and the inconclusive fresh-sample comparison. The paired blocked confirmatory experiment further reports confidence intervals, an equivalence test, and a power calculation rather than relying only on the fitted ranking.
Critical weaknesses
  1. The theoretical guarantees do not yet cover the claimed recovery of the best feasible configuration. Section 4 explicitly restricts the analysis to the unconstrained setting C=∅C=\varnothing, and the appendix establishes recovery and stopping guarantees only for this setting. When hard constraints such as minimum pass-probability thresholds are imposed, can Algorithm 1 be shown to return a feasible and optimal or near-optimal configuration with a finite-sample guarantee?
  2. The experiments do not yet sufficiently establish that FACT-UCB selects higher-quality prompt configurations than existing methods. Although the paper compares several algorithms, the runs use different task sequences and service deployments, and the selected configurations are evaluated primarily using FACT-UCB’s own predicted scores.
Questions for authors
  1. Can the authors extend the analysis in Appendix A to the constrained procedure in Algorithm 1, specifying the confidence bounds and any feasibility-margin assumptions needed for valid certification, stopping, and best-feasible recovery when C≠∅C\neq\varnothing?
  2. Building on Appendix B.2, can the authors repeat the comparisons under matched prompt streams and deployments, and independently evaluate every method’s final recommendation on shared held-out prompts? Please also include the main-effect-only and flexible-surrogate ablations.
  3. Empirical validity and internal consistency: Can the authors provide the complete nine-item rubric and a human-audited calibration of the VLM evaluator? There is a confusing inconsistency between the configuration (1,1,1,0,0,1,1) reported in Section 5.2/Table 2 and 1111011 reported in Table 3, although the appendix clarifies that they come from different judge-specific runs. Please make this distinction explicit in Section 5.2 and Table 3, and clarify which discovery run the confirmatory experiment is intended to validate.
  4. Claimed constrained and pre-3D use cases: Can the authors demonstrate at least one experiment with C≠∅C\neq\varnothing and verify that the returned configuration actually satisfies the constraints? For the pre-3D application, can they evaluate downstream reconstruction quality or expert assessment rather than only VLM-based image criteria?
Reviewer_pZDoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces FACT-UCB for optimizing interpretable combinations of prompt transformations. Criterion-specific factorial logistic models share information across configurations, while UCB sampling and forced exploration support selection. An unconstrained fixed-confidence result accompanies a text-to-image study involving seven factors, 128 configurations, nine criteria, and 999 queries.

Critical strengths

The factorial formulation provides an interpretable alternative to treating complete prompts as unrelated arms. Modeling criteria separately exposes operational tradeoffs, while shared main and interaction effects could improve sampling efficiency.

Critical weaknesses
  • The theoretical result does not establish the advertised constrained guarantee in my opinion. Consequently, the supplied theorem does not establish best-feasible recovery under criterion thresholds. The experiment also does not document the persistent forced-exploration schedule or stopping certificate required by the analysis.

  • There is a mismatch between table 2 and 3: able 2 identifies 1110011, fitted score 0.9345, as the winner, whereas Table 3 tests 1111011, predicted score 0.8793, as “optimal.”

Questions for authors
  • Where is the constrained recovery result, and which forced-exploration schedule and certification procedure were implemented?

  • Which configuration and fitted model were independently tested? Please reconcile Tables 2–3 and explain the change of evaluator between discovery and confirmation.

  • Under the same query budget, how does FACT-UCB compare with balanced/random allocation and a main-effects-only logistic model?

Reviewer_n9ktOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a FACT-UCB algorithm to efficiently search for the prompt rewriting configuration with the best average criterion score. The prompt rewriting configuration contains seven factors, each with two options, and the criterion score consists of nine binary evaluation criteria. The method uses a generalized linear factorial model to estimate the probability of each criterion based on the individual factors and their pairwise interactions. It then uses UCB to adaptively select configurations for evaluation, considering both the predicted score and its uncertainty.

The experiments are conducted on text-to-image generation for 3D game assets, using GPT-4o to rewrite prompts according to the factor configurations, and using Gemini-2.5-Flash-Image as the image generation model. The authors conduct 999 adaptive queries and report the configuration with the highest predicted score.

Critical strengths

Factorizing the prompt rewriting configuration and using a UCB algorithm for configuration selection intuitively make sense. The factorial structure allows information sharing across different configurations, potentially improving search efficiency. It could be an interesting strategy for prompt optimization.

Critical weaknesses

My main concern is with the experimental part. The results lack comparison with other prompt optimization strategies, including simple baselines such as randomly sampling prompt configurations and selecting the one with the highest average criterion score. It is unknown how the method performs compared to these baselines, which raises concerns about the effectiveness of the optimization method. Besides, the method is evaluated on only one text-to-image generation task and dataset setting, using one image generation model (Gemini) and one prompt rewriting model (GPT-4o). The paper also provides only one qualitative example.

Questions for authors

How does the proposed method compare with other baseline methods, such as random search, under the same query budget? (See the weakness section.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.