- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SCOPE addresses the loss of user requirements across stages of image generation. It decomposes a prompt into entities, constraints, and unknowns with stable identifiers, maintaining evidence, verification results, and repair records. Unresolved knowledge or reasoning gaps trigger retrieval or reasoning, whereas specified requirements that remain visually unsatisfied trigger prompt rewriting, local editing, or regeneration. The paper introduces the 300-instance Gen-Arena benchmark and strict EGIP metric, requiring every entity and constraint to pass. Experiments compare SCOPE with Mind-Brush and GEMS using two generation backends, a shared controller, and a maximum of three image calls, alongside evaluation on WISE-V and MindBench. The authors report higher overall pass rates, lower orchestration overhead, three complete runs, and blinded evaluation on 120 shared prompts to support persistent requirement tracking.
Critical strengths
-
Persistently linking requirements, evidence, and repair targets is a practical design. The results go beyond a single run on the new benchmark: improvements are reasonably consistent across two generation backends, three complete runs, and external benchmarks.
-
Blinded human evaluation on 120 shared prompts preserves the automated ranking across all three agentic methods (Table 8). This reduces concern that the automated evaluator simply favors one output style and is important positive evidence.
Critical weaknesses
-
The submitted implementation does not clearly map to the main tables. The paper uses Kimi-K3 and Qwen/Z-Image, whereas the code invokes Codex, implements a Gemini JDCloud image adapter, and reports different README scores. This may be a versioning issue, but the available materials do not reproduce the paper's configuration. It is my main reason for a tentative weak reject.
-
Candidate selection mixes entity and constraint failure counts rather than prioritizing entities. Final evaluation also requests all constraints instead of skipping them after an entity failure (
stages.py:417–440;gen_arena_eval.py:330–364). With judgments fixed, the latter gives the same final AND result, but the evaluation process and item-level statistics differ. The experimental protocol needs to be identified. -
The drop from 0.56 to 0.40 without the specification supports structured records (Table 3). However, structured formatting, more explicit checklists, and cross-stage tracking remain bundled. The ablation does not isolate continuity of commitments as the mechanism behind the gain.
Questions for authors
-
Which configuration and source version produced the main tables, and can the corresponding logs be provided?
-
Which selection and gating rules were used, and do the results hold after making them consistent?
-
What happens if the structure is retained but ID links or state updates are removed?