- Rating
- 3
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper uses instance-specific checklists to represent fine-grained constraints in image editing tasks. It then proposes CR-OPSD to distill checklist- and reasoning-based teacher supervision into a model that directly predicts four-dimensional reward scores. The main goal is to preserve detailed evaluation capabilities while reducing the inference cost of long reasoning traces during online RL.
Critical strengths
The paper has a clear problem formulation and presents a coherent pipeline covering checklist construction, privileged distillation, and downstream RL. The method achieves strong results across several reward benchmarks while substantially reducing the number of output tokens required during inference. This makes the approach practically relevant for large-scale reward evaluation.
Critical weaknesses
-
The central claim is that reasoning can be effectively distilled away, but the contribution of the original reasoning is not sufficiently analyzed. The current experiments mainly compare final accuracy and output length. They do not fully address: how much reasoning contributes to different types of examples; first, whether the distilled direct scorer degrades on long-tail cases; then, whether the reasoning and distilled models have comparable calibration; and whether the distilled model is more vulnerable to distribution shift. The authors should construct additional OOD samples, including small localized edits, multi-object edits, negation, complex spatial relations, unseen attribute combinations, outputs from unseen editing models, and different image-quality distributions. The reasoning model, direct SFT model, and CheckDistill should then be compared.
-
The training labels and checklists rely heavily on Gemini-generated supervision. Although human preferences are used for filtering, the final dimension scores and rationales are still mainly generated by an automatic teacher. The paper should report agreement between automatic annotations and human judgments, or evaluate robustness across multiple independent teachers.
-
The efficiency analysis focuses mainly on token count and does not fully measure deployment cost.
Questions for authors
See Critical Weaknesses
4 comments
W2: Agreement between automatic annotations and human judgments
We add a human evaluation of the automatic annotations in CheckDistill-40K and include the analysis in Appendix K. We randomly sample 100 examples and assess five dimensions of annotation quality: score agreement, checklist correctness, checklist completeness, reasoning correctness, and score-reason consistency.
For score agreement, we compute the mean absolute score deviation () to measure how closely the automatically assigned scores () match the human ratings (). We define () as the average of () across examples.
For the remaining four dimensions, human annotators rate the automatic annotations on a scale from 1 to 5. We extend this assessment to 2K examples using GPT-6.1 Sol with the consistency assessment template in Appendix K.
The score annotations in CheckDistill-40K agree well with human judgments (). Both human annotators and GPT-6.1 Sol give favorable ratings for checklist correctness, checklist completeness, reasoning correctness, and score-reason consistency. In addition, for GPT-6.1 Sol, the difference between the data construction template () and the scoring template () indicates that checklist-based data construction yields score annotations that agree more closely with human ratings.
| Evaluator | IF (, ↓) | AP (, ↓) | CP (, ↓) | PQ (, ↓) | Overall (, ↓) |
|---|---|---|---|---|---|
| Gemini-3.0-Flash | 1.55 | 0.70 | 0.55 | 2.70 | 0.99 |
| GPT-6.1 Sol (data construction template) | 0.90 | 0.45 | 0.10 | 0.80 | 0.45 |
| GPT-6.1 Sol (scoring template) | 1.35 | 0.60 | 0.35 | 0.80 | 0.65 |
Table: Evaluator score agreement with human ratings. is computed as the average of across examples. Lower indicates closer agreement.
| Evaluator | Checklist Correctness ↑ | Checklist Completeness ↑ | Reasoning Correctness ↑ | Score-Reason Consistency ↑ |
|---|---|---|---|---|
| GPT-6.1 Sol | 4.78 | 4.58 | 3.49 | 4.67 |
| Human | 4.86 | 4.66 | 4.03 | 4.82 |
Table: Checklist and Rationale Consistency of Automatic Annotations as Assessed by GPT-6.1 Sol and Human Annotators.
W3: Efficiency Analysis
We extend our inference efficiency evaluation to include mean generation time per request, throughput, and total runtime alongside token counts and accuracy. We conduct these experiments on MMRB2 Single and include this analysis in Appendix L as well.
CheckDistill processes 1.62× as many pairs per second as SpatialReward and 2.57× as many as EditScore, while achieving the highest accuracy among the three models.
| Model | Input tokens | Output tokens | Generation time (s/request) ↓ | Throughput (pairs/s) ↑ | Total runtime ↓ | Accuracy ↑ |
|---|---|---|---|---|---|---|
| EditScore | 8,639 | 346 | 4.91 | 0.195 | 70m 26s | 57.90% |
| SpatialReward | 7,102 | 496 | 2.98 | 0.310 | 44m 09s | 67.10% |
| CheckDistill | 4,617 | 26 | 0.76 | 0.501 | 27m 20s | 73.36% |
Table: Efficiency and accuracy on MMRB2 Single.
Thank you again for your thoughtful feedback. We hope our responses address your concerns and would be happy to discuss any remaining questions.
Review
Thank you for the response. Follow-up on Weakness 1: (1) Beyond a single qualitative example (Figs. 13–14), could you report quantitative calibration or score agreement (e.g., MAE, correlation) between the reasoning and distilled variants across the benchmark? Besdes, since 51 OOD tasks is relatively small, could you provide a breakdown by edit type (e.g., negation, spatial relations, local edits) to check whether distillation degrades on specific long-tail cases?
We are glad to receive your follow-up comments! We report score agreement between the reasoning and distilled variants across the OOD benchmark in Table 12 of Appendix J, with an overall mean absolute score deviation of 0.85, indicating close agreement.
We further provide accuracy by edit type on the OOD benchmark in the table below. For CheckDistill-40K, negated instructions (1.5% of the dataset) and attribute combinations (including unseen attributes in the OOD benchmark) represent long-tail cases, while each of the other types accounts for over 10% of CheckDistill-40K. As you anticipated, the distilled model shows some decline in accuracy compared with the reasoning model on these long-tail cases. Nevertheless, compared with the direct SFT model, CheckDistill achieves a better balance between accuracy and efficiency.
We believe the decline may stem from limited training coverage of these long-tail cases, making it harder for CheckDistill to generalize its implicit evaluation capabilities to them. This gap could be reduced by expanding the reward training data to include more long-tail data. Alternatively, it could be mitigated during RL by using CheckDistill as an efficient reward model for most editing tasks and switching to a reasoning model for challenging OOD cases.
| Model | Small Local Edit | Negated Instruction | Complex Spatial Relation | Multi-Object Edit | Attribute Combination | Overall Accuracy ↑ |
|---|---|---|---|---|---|---|
| SFT (None → S) | 14.29% | 16.67% | 30.00% | 25.00% | 7.14% | 17.65% |
| CheckDistill (Reasoning) | 14.29% | 33.33% | 30.00% | 25.00% | 21.43% | 25.49% |
| CheckDistill (Distilled) | 28.57% | 25.00% | 40.00% | 37.50% | 7.14% | 25.49% |
Table: Ranking accuracy on the OOD benchmark (including accuracy by edit type).
We sincerely appreciate your time. We hope this response addresses your concerns and wish you all the best.
Thank you for your careful reading of our submission and for the clear suggestions on how to strengthen the evaluation. We respond to each of the three points below.
W1: OOD Evaluation and Analysis
Following your suggestion, we construct an OOD benchmark for evaluation in Appendix J. We randomly sample 51 tasks from CPI-Intelligent-Bench [1], covering fine-grained local edits, multi-object edits, edits with multiple reference images, negated instructions, complex spatial relations, and unseen attribute combinations. For each task, human annotators rank three candidate edits generated by Nano Banana Pro, Qwen-Image-2.0-Pro, and Seedream 5.0 Pro without knowing which model produced each image. Figures 11 and 12 provide visualizations of the benchmark.
On the OOD benchmark, we use 3-pair ranking accuracy to compare eight model variants, including Gemini-3.0-Flash with the data construction template (Appendix D) and with the scoring template (Appendix E), Qwen3.5-9B (None → S) and three SFT variants (None → S, None → R+S, and None → C+J+S) from Section 4.3, and the reasoning and distilled variants of CheckDistill.
As shown in Table, distilled CheckDistill achieves strong performance on the OOD benchmark, attaining 25.49% ranking accuracy and outperforming direct-scoring SFT (17.65%) and SFT with explicit rationales (21.57%). Its performance is comparable to both SFT (None → C+J+S) and the reasoning variant of CheckDistill, highlighting the benefits of checklist-guided reasoning for tasks requiring fine-grained visual understanding. We further present a scoring example for the reasoning and distilled variants of CheckDistill (Figures 13 and 14). The two variants show similar scoring behavior and have comparable calibration. These results demonstrate that CheckDistill effectively internalizes the evaluation capabilities of the reasoning model without requiring explicit reasoning at inference time.
Table: Ranking accuracy on the OOD benchmark.
References: