- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FACT-UCB, a criterion-aware prompt optimization framework that formulates prompt design as a factorial optimization problem. It decomposes prompt transformations into interpretable factors and models their effects on multiple evaluation criteria, rather than relying on a single scalar reward. The method is evaluated in a pre-3D generation case study, where prompts are optimized to generate 2D reference images that better satisfy criteria relevant to downstream 3D asset generation, such as object completeness, background cleanliness, count, orientation, and style.
Critical strengths
- The factorial formulation is interpretable and operationally meaningful. By modeling reusable prompt operations as factors rather than treating complete prompts as unrelated arms, the framework can estimate factor-level and interaction effects while sharing information across configurations. Keeping evaluation criteria separate also exposes trade-offs and supports criterion-wise feasibility constraints that would be obscured by a single scalar score.
- The paper is appropriately cautious about its empirical evidence. The authors do not treat the adaptively fitted optimum as definitively validated: they explicitly discuss model misspecification, winner’s curse, evaluator uncertainty, the lack of downstream 3D validation, and the inconclusive fresh-sample comparison. The paired blocked confirmatory experiment further reports confidence intervals, an equivalence test, and a power calculation rather than relying only on the fitted ranking.
Critical weaknesses
- The theoretical guarantees do not yet cover the claimed recovery of the best feasible configuration. Section 4 explicitly restricts the analysis to the unconstrained setting , and the appendix establishes recovery and stopping guarantees only for this setting. When hard constraints such as minimum pass-probability thresholds are imposed, can Algorithm 1 be shown to return a feasible and optimal or near-optimal configuration with a finite-sample guarantee?
- The experiments do not yet sufficiently establish that FACT-UCB selects higher-quality prompt configurations than existing methods. Although the paper compares several algorithms, the runs use different task sequences and service deployments, and the selected configurations are evaluated primarily using FACT-UCB’s own predicted scores.
Questions for authors
- Can the authors extend the analysis in Appendix A to the constrained procedure in Algorithm 1, specifying the confidence bounds and any feasibility-margin assumptions needed for valid certification, stopping, and best-feasible recovery when ?
- Building on Appendix B.2, can the authors repeat the comparisons under matched prompt streams and deployments, and independently evaluate every method’s final recommendation on shared held-out prompts? Please also include the main-effect-only and flexible-surrogate ablations.
- Empirical validity and internal consistency: Can the authors provide the complete nine-item rubric and a human-audited calibration of the VLM evaluator? There is a confusing inconsistency between the configuration (1,1,1,0,0,1,1) reported in Section 5.2/Table 2 and 1111011 reported in Table 3, although the appendix clarifies that they come from different judge-specific runs. Please make this distinction explicit in Section 5.2 and Table 3, and clarify which discovery run the confirmatory experiment is intended to validate.
- Claimed constrained and pre-3D use cases: Can the authors demonstrate at least one experiment with and verify that the returned configuration actually satisfies the constraints? For the pre-3D application, can they evaluate downstream reconstruction quality or expert assessment rather than only VLM-based image criteria?