- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose EditStress-Bench, a benchmark for evaluating the reliability of image editing reward models under increasing search budgets. Using HICO-DET-derived human–object editing tasks, it compares static reward–human agreement with the human-rated quality of outputs selected through Best-of-N and iterative refinement, revealing that the agreement does not necessarily translate into effective selection. The authors also introduce factor-guided reward patching, which uses controlled perturbations to identify weaknesses in detecting specific editing defects, predict failures amplified by search. The approach is evaluated through search with Overall Human Quality (OHQ), a holistic score assessed by a separate three scorers.
Critical strengths
- Separating holistic quality assessment from defect annotation also reduces overlap between the evidence used to evaluate quality and diagnose failures.
Critical weaknesses
-
It seems that the discussed problem can be mitigated simply by adopting stronger reward models. Beyond GPT-4o judge (Table 1), evaluating recent strong VLMs such as Claude Opus 5.5 or GPT-6 Astra as reward model would help establish whether the mismatch between static agreement. The authors should demonstrate that their approach addresses a persistent practical limitation and provides gains beyond those obtainable simply by enhancing the underlying reward model.
-
EditStress-Bench’s reliance on HICO-DET limits both its task coverage and its evaluation framework to human-centric editing scenarios. The five factors is tailored to human–object interactions, most explicitly through human fidelity and the emphasis on action completion and contact consistency. This specialization limits the framework’s applicability to broader image editing tasks, weakening its representativeness as a general well-known image editing benchmark such as PIE-Bench [1]. More diverse tasks and evidence of transfer beyond human-centric scenarios are needed to support broader claims.
[1] Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Code (Ju et al., ICLR 2024)
Questions for authors
-
The practical relevance of the problem setting is insufficiently established. In particular, the paper provides limited justification for best-of-N scaling with large N in image editing tasks. Without supporting references or concrete use cases, the motivation for making this setting a central focus of the benchmark remains weak.
-
The distinction between the default (D) and alternative (A) settings in Table 1 is unclear and hard to understand. Could the authors explain (or define) precisely on Section 4 how these settings differ and why both are considered?
4 comments
Responses to Reviewer 298B (Part II)
- Q1: We have added references and use cases to the introduction (Sec. 1). Reward scores are used for Best-of-N reranking of image editing candidates (Luo et al., 2026) and to guide inference-time search in image generation (Ma et al., 2025). They also provide group-relative rewards for reinforcement learning (Liu et al., 2025; Xue et al., 2025). In each case, the reward repeatedly selects among many candidates, which is the pressure that EditStress-Bench measures. We are also encouraged that Reviewers g2x7 and EJ17 found this search-time setting practically relevant.
Importantly, the finding does not depend on a large N. In the initial version, Appendix B.5 (Table 10) already showed that source-aware Best-of-N rank transfer falls from 0.479 at four calls to 0.006 at eight calls. The revised Sec. 4.2 now states this in the main text. Larger budgets show how far selection can drift, for example the decline of three default settings between N = 16 and N = 64 (Table R3).
- Q2: We have defined the D/A settings in Sec. 4.1 and added a Composition column to Table 1. D and A share the backbone of each family and differ in scoring head, rubric weighting or aggregation. In five families, D uses the native or single scalar output and A adds factor information. For VIEScore, D weights the rubric items unequally and A weights them equally. Full specifications are given in Appendix A.2. Both settings are considered because they hold the backbone fixed and change only the scoring composition, which isolates its effect on search-time quality. At N = 64, D selects outputs 2.1 to 6.7 points below A under Best-of-N in all five source-aware families, with four intervals excluding zero (Table R2).
We hope these responses address the reviewer's concerns, and we would be glad to discuss any remaining questions.
Thank you for the detailed response and additional experiments. The evaluations with stronger judges substantially address my concern that the observed failures might simply reflect limitations of the original reward models.
The PIE-Bench results also provide useful evidence that the search-time failure phenomenon extends beyond human–object interaction editing, which showing robustness. My small remaining concern is whether the benefits of factor-guided patching generalize as well, although the new experiments appear to establish transfer of the failure phenomenon.
Overall, the response has resolved most of my initial concerns. I now lean slightly toward acceptance and have revised my rating to Weak Accept.
Response to Reviewer 298B
To reviewer 298B: We sincerely thank the reviewer for the positive feedback and the raised rating. We are glad that the frontier-judge and PIE-Bench results resolve the main concerns. Regarding the generalization of factor-guided patching, the current evidence is summarized below.
-
Across scorers and factors: Patching is applied to three scorers with different diagnosed factors, namely SP for InternVL-Reg (D), NP for EditReward (A) and HF for EditScore (D). In all three cases, targeted updates raise OHQ by 1.6 to 2.1 points under rerun search on the held-out patch tasks, compared with 0.3 to 0.4 for equal-budget random updates (Table 4).
-
Beyond the original factor set: Patching relies on the link between low sensitivity to a factor and its enrichment after search. This link also holds for the five additional categories identified in the residual audit (Table 25, Appendix F.5) and when whole reward families are held out from fitting (Table R4). Since the procedure only requires a factor set with calibration pairs, the same targeting can be applied with factors suited to other editing domains.
Thanks again for the constructive discussion, which has helped improve the paper.
Responses to Reviewer 298B (Part I)
To reviewer 298B: Thanks for the constructive comments. Responses to specific comments are given below, and the corresponding revisions are highlighted in blue in the revised paper.
The two frontier judges lead static agreement and small-budget search, yet fall to 11th and 12th among sixteen configurations at N = 64. With joint 95% intervals, rank transfer at N = 64 is −0.389 [−0.653, −0.051] among the fourteen source-aware configurations. For all sixteen configurations it is −0.176 [−0.403, 0.138], following the same pattern as Table 6. On the same backbones, the alternate settings score five rubric items and select outputs 3.8 and 4.0 points higher at N = 64. A stronger backbone therefore does not prevent search-time failures, whereas scoring composition changes selected quality. This is the gap that factor-guided patching addresses, raising OHQ by 1.6 to 2.1 points for a fixed backbone (Table 4).
(1) Transfer to PIE-Bench. We sample 100 non-human tasks from PIE-Bench (Ju et al., 2024) with a fixed seed, covering 29 object, 27 attribute, 23 background and 21 style edits. All tasks use the official editing instructions. Qwen-Image-Edit generates 64 candidates per task, the twelve configurations select from shared nested pools, and OHQ follows the same three-rater blinded protocol (Sec. 4.3, Appendix B.8).
On PIE-Bench, D selects outputs 1.9 to 2.8 points below A in four families, and these D settings lose 1.0 to 1.9 points from N = 16 to N = 64. All eight intervals exclude zero (Table R8). EditReward behaves the same way on both datasets. Rank transfer over the twelve configurations falls from −0.007 at N = 16 to −0.182 at N = 64 (Table R7).
(2) Factor set. The five factors are specific to human-object interaction by design. Profiling, enrichment and targeted patching only require a factor set with calibration pairs. In the initial version, Appendix F.5 already applied the same regression to five categories identified in the residual audit. The sensitivity-enrichment slope remains negative for these categories alone (−3.799 [−4.254, −3.294] under Best-of-N) and for the union of ten factors (−4.109 [−4.433, −3.774]) (Table 25). The framework can therefore be instantiated with factors suited to other editing domains.