- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses a useful question: whether question-level dependency graphs can support more informative text-to-image evaluation and model selection. However, several concerns limit the conclusions.
Critical strengths
The paper includes useful controls, particularly pre-masking outcomes for dependency analysis, prompt-cluster bootstrapping, and appropriately qualified routing claims.
Critical weaknesses
- Model-dependent selection in scoring. Dependency masking evaluates each model’s downstream capabilities on a different surviving subset. This avoids repeated penalties but can favor models that satisfy prerequisites only on easier examples. Report prerequisite coverage and unconditional success alongside conditional scores. High rank correlation with an alternative score does not resolve this issue.
- Insufficient validation in the supplied manuscript. Construction rules and automated auditing do not establish question atomicity, coverage, graph correctness, or evaluator accuracy. Independent validation is needed because errors in prerequisite judgments propagate into downstream eligibility and capability scores. Relevant evidence may be in the referenced supplement, which is not included in this PDF.
- Cross-prompt matching does not isolate composition effects. Matching an entity across otherwise different prompts does not control scene difficulty or attached constraints. The reported root-success differences should be interpreted as associations unless supported by matched prompt variants and clearly specified aggregation.
- The aggregation result has limited scope. Lower noise variance under independent, equal-variance errors does not establish that question weighting is preferable to prompt weighting. These rules weight the evaluation population differently, and correlated evaluator errors require additional analysis.
- Routing comparisons are incomplete. Fixed generators and uniform random selection do not isolate the benefit of question-level profiles. Compare against a cost-matched prompt-independent policy and a simple prompt-level router.
Questions for authors
See weaknesses
3 comments
Thanks to the authors for the new results. The rebuttal is encouraging at the same time I think it is important to have rigor in the newly reported metrics for example inter annotator agreement for the human evaluation. Some case analysis into the first table would be useful. I am willing to reconsider my score if the rigor is improved
Thank you for the encouraging follow-up. We add Cohen's for the human reviews, followed by a case addressing your concern about different surviving subsets.
1. Cohen's kappa between human reviewers
"inter annotator agreement for the human evaluation."
Response. The paper's VQA answer agreement, 152/178 (85.4%), compares the automatic judge with the final adjudicated human answers. Your request concerns consistency between the two human reviewers. We therefore add Cohen's for their judgments before adjudication:
| Human review | Cohen's |
|---|---|
| Question quality, tags and image answers | 0.825 |
| Recorded DSG edges | 0.664 |
The question/tag/VQA audit covers 200 bilingual records, checking question quality, capability tags, VQA answers, and question validity. Across these checks, the two reviewers gave identical labels on 1,081 of 1,164 applicable judgments. This is the separate two-reviewer result in the original audit; comes from the same audit's reviewer statistics.
For DSG, we calculated from the existing two annotators' judgments on 300 edges from 30 concepts with paired English/Chinese graphs. Necessity and direction were judged separately and had identical counts: both annotators accepted 296 edges, rejected 2, and disagreed on 2. These labels give .
The values use the two human reviewers' labels before adjudication. Specifically, , where is the proportion of matching human labels and is chance agreement calculated from the two reviewers' label proportions. A third reviewer resolved disagreements; in the question/tag/VQA audit, this reviewer also checked 20 fully agreed records, overturning none.
Finding missing edges is an open-ended search and is harder than judging a specified edge. We therefore report confirmed omissions: edges independently identified as missing by both annotators and retained after human adjudication. This gives 14 confirmed missing edges relative to 431 estimated valid recorded edges in the sampled graphs (3.25%).
2. Case analysis: different surviving subsets
"Some case analysis into the first table would be useful." Original concern: "can favor models that satisfy prerequisites only on easier examples."
Response. We agree that a high conditional score can reflect a small surviving subset. Consider the same Chinese prompt requesting an elliptical bedroom with a pyramid-shaped photo frame at the top, contrasting curved and angular forms:
| Question (English translation) | HiDream-I1 | ERNIE-Image |
|---|---|---|
| Is there a room? | Yes | Yes |
| Is the room elliptical? | Yes | No |
| Is there a photo frame? | No (excluded) | Yes |
| Is the photo frame pyramid-shaped? | No (excluded) | Yes |
| Is the photo frame at the top of the room? | No (excluded) | Yes |
| Is there a contrast between curved and angular forms? | Yes | Yes |
These are the saved VQA decisions; "excluded" means marked unevaluable. On the two shape questions, HiDream-I1's conditional success is 100% (1/1), versus ERNIE-Image's 50% (1/2), because HiDream-I1's frame-shape question is excluded when the frame is absent. Both models complete one of two shape requirements: unconditional success is 50% for both, while coverage is 50% versus 100%.
For all six non-text questions, the three metrics in the first table give:
| Model | Coverage | Conditional success | Unconditional success |
|---|---|---|---|
| HiDream-I1 | 3/6 (50%) | 3/3 (100%) | 3/6 (50%) |
| ERNIE-Image | 6/6 (100%) | 5/6 (83.33%) | 5/6 (83.33%) |
This single-prompt contrast is large. Over the full benchmark, replacing conditional non-text scores with unconditional scores in HCQ (OCR unchanged) preserves the English ordering; only two adjacent Chinese model pairs swap. Overall model differences are broadly consistent, while score changes remain larger for low-coverage models.
Conditional scoring measures capability when a requirement can be evaluated, avoiding repeated penalties for an absent prerequisite. Coverage and unconditional success show how often the model reaches the task and succeeds over the full question set. The supplement and its accompanying data already report all three for every non-text capability, model, and language, alongside HCQ. They make the different surviving subsets visible.
We will add these human-reviewer values and this worked case to the supplementary material.
Thank you again for your constructive feedback and willingness to reconsider your assessment. Your suggestions have helped us present the evidence more clearly and rigorously.
Thank you for the detailed critique and requested controls.
W1: Model-dependent selection
Response. We now report coverage (eligible/all questions), conditional success (eligible correct/eligible), and unconditional success (eligible correct/all). Existing non-text records give:
These non-text averages differ from HCQ, which balances capabilities and includes OCR. Missing outputs stay in the full-set denominator, separately from prerequisite failures. We also report all three quantities per non-text capability.
The masking ablation counts dependency-invalid questions as failures; OCR is unchanged and missing/unparsable answers stay excluded. HCQ drops 0.64-1.82 pp in EN and 0.49-1.21 pp for 11 ZH models; ZH exceptions are HiDream-I1 (65.02% to 59.57%) and FLUX.1-dev (32.75% to 14.77%). EN ranks stay unchanged; two adjacent ZH pairs swap. Coverage makes the scoring subsets explicit.
W2: Independent validation
Response. The original bilingual 200-record challenge audit used two human reviewers and adjudication. We added an audit of 300 DSG edges from 30 concepts with paired EN/ZH graphs. A/B independently checked necessity/direction; human C adjudicated.
W3: Cross-prompt interpretation
Response. For each model/language, the reference averages the same entity's success in at least three other evaluable prompts, excluding the target. References are not selected by scene, outcome, or constraint count and need not be simpler. Local includes all evaluable roots in eligible components; cross-prompt includes roots meeting the reference threshold. Each prompt has equal weight, split among eligible components and included roots.
Unlike the overall entity-view score, this analysis links the same entity across combinations of attributes, relations, and other requirements. Higher rates reflect more reliable entity presence in these contexts, one aspect of compositional ability. Their gap describes the observed contexts and root populations. We clarify these rules beside Figure 1.
W4: Aggregation and correlated errors
Response. HCQ weights valid atomic requirements equally within capabilities and balances five reporting views. Prompt-first gives a one-question prompt the same within-capability weight as one with several questions. In our 1,000-prompt analysis, this gives a short-prompt judgment 48.4% more weight in EN and 41.8% more in ZH. We choose question weighting because the atomic requirement is our evaluation unit. The variance result assumes independent, equal-variance judging errors; HCQ itself does not. Correlated errors limit the variance result, without making prompt-first weighting preferable. We clarify this distinction.
W5: Routing baselines
Response. Our goal is to demonstrate useful routing signals with a simple rule. Q-Profile-C maps 21 question tags to five groups and uses active groups and training-fold performance to select generators. We added both controls to the same five-fold evaluation of 13,132 held-out EN/ZH requests:
Costs count generation only. Intervals are Q-Profile-C minus each baseline, using paired conceptual-prompt bootstrap. The annotations support effective routing on held-out prompts.