acceptodds
Under review as a conference paper at ICLR 2027

SVGFaith: A Two-Layer Meta-Evaluation of Faithfulness Metrics for Text-to-SVG Generation

Abstract

Text-to-SVG systems are commonly evaluated by applying raster-image metrics to rasterised outputs, yet whether these scores capture fine-grained prompt faithfulness remains unclear. We introduce SVGFaith, a two-layer meta-evaluation framework in which the more faithful candidate of each pair is determined by construction or by human judgment. SVGFaith-Diagnostic contains controlled contrastive pairs across eight capability axes, split evenly between two directions: two renderings compared against one prompt, and two descriptions compared against one rendering. SVGFaith-Arena contains human-annotated comparisons of real generator outputs. On SVGFaith-Diagnostic, CLIP and three other image–text embedding metrics reach at least pairwise accuracy on object identity but at most on spatial relations and attribute binding, where chance is . We further introduce SVGFaith-Score, an interpretable pointwise metric. A domain-tuned decomposer turns each prompt into a dependency graph of atomic visual requirements, a visual question answering model scores whether each requirement is satisfied, and a requirement's score is discounted when its prerequisites score low. In addition to the overall score, the per-requirement scores show which requirements are judged unsatisfied and which inherit discounts from their prerequisites. SVGFaith-Score significantly outperforms all nine baselines, including a Qwen3.5-122B-A10B judge, on Arena and in both Diagnostic directions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
text-to-SVG generation, vector graphics, faithfulness evaluation, multimodal evaluation, metric meta-evaluation, visual question answering, interpretable evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1HmnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SVGFaith, a two-layer meta-evaluation framework for text-to-SVG faithfulness. SVGFaith-Diagnostic contains 1,842 controlled contrastive pairs across eight capability axes and two directions, while SVGFaith-Arena contains 1,327 human-annotated comparisons of real generator outputs. The authors propose SVGFaith-Score, an interpretable pointwise metric that decomposes a prompt into an atomic requirement graph, scores each requirement with a VQA backend, and propagates discounts from prerequisite requirements. Experiments show that SVGFaith-Score achieves 0.880 pairwise accuracy on SVGFaith-Arena, outperforming all nine baselines.

Critical strengths
  1. The two-layer design separates controlled diagnosis from real-output reliability: Diagnostic pairs localize specific metric blind spots, while human-annotated Arena pairs test real generator outputs without using any evaluated metric to define the reference ordering.

  2. SVGFaith-Score is interpretable and deployable. It leads nine baselines on Arena and both Diagnostic directions, and the ablation dropping Arena accuracy to 0.073 with base Qwen3.5-9B as decomposer shows the gain comes from domain-tuned decomposition.

Critical weaknesses
  1. Table 2 compares against mostly natural-image metrics such as CLIP, SigLIP, PickScore, HPS, and ImageReward. These metrics were not designed for SVG and transfer poorly to vector graphics, while SVG-specific evaluators such as SVG-Score, SVGauge (described in the related work) are not included in the comparison.

  2. All evaluations are conducted only on the authors’ own SVGFaith benchmarks. SVGFaith-Decomposer is domain-fine-tuned, while the baselines are not tuned for SVG, so its advantage may partly reflect benchmark-specific tuning rather than general evaluator quality. The metric should also be tested on independent SVG benchmarks to show that the gain transfers beyond the proposed evaluation sets.

Questions for authors

Could the authors validate SVGFaith-Score on multiple independent SVG benchmarks beyond the proposed SVGFaith, to demonstrate that its effectiveness generalizes across datasets?

2 comments

Thank you for the questions on baselines and external validation.

Table 2 evaluates the metrics that text-to-SVG papers actually use in practice: image–text embeddings, preference reward models, VQAScore, and a VLM judge. Their poor transfer to vector graphics is one of our main empirical findings rather than a flaw in baseline selection. The table also goes beyond natural-image embeddings: FlanT5-XXL VQAScore and the Qwen3.5-122B-A10B judge are included, and the 122B judge is the strongest baseline in both Diagnostic directions (0.902 and 0.857 at τ = 0.01).

SVG-Score and SVGauge are indeed the closest SVG-specific evaluators, and we will make their status explicit in Section 4.1. Neither had a runnable public release at submission or at the time of this response. SVG-Score provides a project page, while the paper notes that its dataset, weights, and code will be released later. SVGauge states that its SHE dataset will be released upon acceptance, and no public implementation is available. Moreover, SVGauge is reference-based: it compares a generated SVG against a ground-truth reference SVG, whereas SVGFaith-Score is reference-free and scores a single rendering against the prompt alone. Once their official releases become available, we will add both to our comparison.

We do not train the 9B decomposer to beat the 397B teacher. Prompting the untuned 397B teacher as the decomposer already reaches 0.876 on Arena, statistically indistinguishable from the released 9B at 0.880 (95% CI [−0.012, +0.005]). Keeping a 397B model in the scoring loop is too expensive for routine evaluation and for community use after release. We therefore distill that decomposition ability into a 9B model so the metric can run at a much lower GPU cost. The VQA backend remains the off-the-shelf Qwen3.5-9B, and training does not use SVGFaith’s human preference labels. With that same untrained backend, a single holistic query already reaches 0.850 on Arena. The gain is not from fitting the benchmark labels.

Existing public SVG benchmarks (such as SVGenius, SVGEditBench, and VectorGym) evaluate generation or editing models rather than evaluators: they do not provide paired candidates with ground-truth faithfulness orderings, which meta-evaluation requires. The human-rated sets in SVG-Score and SVGauge would be the right external testbeds, and both are not yet publicly available. We will evaluate SVGFaith-Score on them as soon as they are released.

AuthorsOpenReview ↗

Follow-up: Revised Manuscript Updated (Section 4.1 & Additional Controls)

Dear Reviewer 1Hmn,

Thank you again for your time and for noting that your soundness concerns are minor/fixable. We wanted to let you know that we have uploaded the revised manuscript incorporating our clarifications:

  1. Section 4.1 now explicitly documents the unavailability of public code, model weights, and datasets for both concurrent SVG-Score and reference-based SVGauge at evaluation time, as well as the protocol distinction between reference-based comparison and reference-free prompt-faithfulness scoring.
  2. Section 4.4 adds further matched controls alongside Table 3, reinforcing that untuned Qwen3.5-397B-A17B + untuned Qwen3.5-9B already reaches 0.876 on Arena (statistically indistinguishable from our distilled 9B decomposer at 0.880, with zero prompt overlap between decomposer training data and SVGFaith), confirming that performance does not come from fitting benchmark labels.

Please let us know if our response and these manuscript updates address your remaining concerns, or if there is anything further we can clarify during the discussion period.

Reviewer_27wkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether commonly used evaluation metrics reliably capture prompt faithfulness in text-to-SVG generation. It introduces SVGFaith-Diagnostic, containing 1,842 controlled comparisons across eight capability axes and two evaluation directions, and SVGFaith-Arena, containing 1,327 human-labelled comparisons of real generator outputs. The paper also proposes SVGFaith-Score, which decomposes prompts into atomic requirements, scores them using VQA, and aggregates their scores through soft prerequisite gating. Experiments compare nine baseline metrics, analyze constraint-specific and generator-conditioned failures, and report component controls and sensitivity to tie handling.

Critical strengths
  1. The two-layer evaluation design addresses a meaningful methodological problem. Controlled violations help identify specific metric weaknesses, while human-labelled generator outputs test whether these findings extend to realistic cases with multiple simultaneous errors. Reporting these layers separately is useful for understanding what an evaluation metric actually measures.

  2. The empirical analysis is informative and reasonably transparent. The paper reports per-axis results, paired bootstrap comparisons, threshold sweeps, and alternative treatments of ties. In particular, Section 4.2 distinguishes ranking correctness from decision coverage and acknowledges that the large judge has higher conditional accuracy on non-tied Arena pairs. SVGFaith-Score also leads the evaluated baselines at zero threshold, so its reported advantage is not solely an artefact of the nonzero tie band.

Critical weaknesses
  1. Methodological novelty is not well isolated from prior work. As noted in Appendix A.3, SVGFaith-Score is conceptually close to DSG and Soft-TIFA. However, the experiments do not clearly isolate the actual contribution of the proposed soft prerequisite gating. The authors fix α=0.5 throughout, without providing crucial baselines such as the un-gated case (α=0) or DSG-style hard-gating on the exact same decomposed requirements and VQA outputs. While Table 3 shows that decomposition helps over holistic scoring, it does not justify why the specific dependency aggregation is necessary. Furthermore, the failure of the off-the-shelf 9B decomposer is primarily due to hitting the reasoning token limit rather than semantic errors. This confirms that fine-tuning helps with instruction following/output formatting, but leaves it unclear how much the individual training stages (SFT vs. GRPO) contribute to actual semantic decomposition quality.

  2. Potential bias introduced by metric-triggered label review in Arena. In Appendix C.3, the authors state that after evaluating all metrics, they manually inspected cases where the majority human label disagreed with several evaluated metrics, resulting in 12 label flips and 25 removals. Crucially, pairs where metrics agreed with the human majority were not re-examined. While cleaning bad annotations is understandable, selecting candidates for manual review based on the evaluated metrics' predictions introduces an circular dependency between the benchmark and the evaluated models. Appendix C.7 only ablates the expert-adjudicated pairs, not these post-hoc corrected/removed samples. To ensure the Arena results are unbiased, the paper should report metric performance on the original 1,352 decided pairs prior to this metric-dependent cleanup.

  3. Presentation (secondary concern). The main narrative is understandable, but important methodological comparisons and evaluation details are dispersed across Appendices A.3, C.3-C.7, and E.4, requiring repeated cross-referencing. Several paragraphs also end with only one or two words, such as "conventions." on page 3, "aggregation." on page 5, and "present." on page 6, creating avoidable whitespace. More compact editing and a concise main-text summary of the closest methods and annotation-review stages would improve readability and use of space.

My weak-reject recommendation is primarily based on the first and second substantive concerns above. Matched aggregation controls and robustness results using the pre-review Arena labels could materially improve my assessment.

Questions for authors
  1. Could you provide metric rankings and accuracy on the original 1,352 Arena pairs before the metric-triggered review (reverting the 12 flips and including the 25 removed pairs)? Additionally, during this review, were the annotators blind to metric predictions?
  2. Keeping the decomposer and VQA model fixed, what is the performance when α=0 (no gating) and α=1 with hard gating? Does the soft gating mechanism provide consistent gains on both Diagnostic and Arena?
  3. Could you report the usable-graph rate and downstream accuracy across the different training stages (Teacher SFT, Rejection-sampling SFT, and GRPO)? This would clarify whether RL improves semantic decomposition or simply fixes formatting/timeout failures.
3 comments
AuthorsOpenReview ↗

Response Q1: Blind Arena Review & Pre-review 1,352 Rankings

Thank you for the constructive and detailed review. We have run all three requested experimental controls without changing any model weights or hyperparameters. This comment answers Question 1 / Weakness 2; matched gating (Q2) and training-stage results (Q3) follow in two short comments.

1. Pre-Review 1,352 Arena Pairs and Blindness of Manual Review

Were inspectors blind to metric predictions? Yes—per-pair inspection was blind. An automated script first pooled pairs whose majority vote conflicted with multiple evaluated metrics (including baselines). In the review interface, the inspector saw only the prompt and the two renderings in randomized left–right order, with all metric scores, predicted winners, prior majority labels, and generator identities hidden. Under the same faithfulness rubric, inspected pairs were kept unchanged, flipped when the majority was clearly wrong (12), or removed when invalid/indeterminable (25); most pooled candidates were left unchanged.

Rankings on the untouched 1,352 pairs (τ=0.01\tau=0.01; revert 12 flips, restore 25 removals):

RankMethodPre-review 1,352W / T / LFinal 1,327
1SVGFaith-Score0.8651170 / 38 / 1440.880 (#1)
2Qwen3-VL-Embedding-8B0.8541155 / 46 / 1510.843 (#2)
3ImageReward v1.00.8171105 / 47 / 2000.819 (#3)
4FlanT5-XXL VQAScore0.7801054 / 72 / 2260.784 (#4)
5SigLIP-L/16-3840.7711043 / 164 / 1450.775 (#5)
6HPS v2.10.7621030 / 196 / 1260.769 (#6)
7LLM-as-a-Judge (122B)0.7511015 / 247 / 900.754 (#7)
8Long-CLIP-L/140.738998 / 168 / 1860.739 (#8)
9CLIP-L/140.700946 / 185 / 2210.699 (#9)
10PickScore v10.598808 / 499 / 450.605 (#10)
  • Ranks #1–#10 are identical on the untouched 1,352 and final 1,327 sets; SVGFaith-Score is #1 on both.
  • SVGFaith-Diagnostic (1,842 pairs; no post-hoc review) further supports the conclusion: SVGFaith-Score reaches 0.941 / 0.953 vs. 0.717 / 0.668 (Qwen3-VL-Embedding-8B) and 0.902 / 0.857 (122B judge).
  • We have reported this table in revised Sections 4.2 and Appendix C.7 with an explicit description of the blind inspection interface.
AuthorsOpenReview ↗

Response Q2: DSG/Soft-TIFA Differences & Matched Gating Controls

2. Distinctions from DSG / Soft-TIFA and Matched Gating Controls (Question 2 / Weakness 1)

Continuation of our response (Q1 in the previous comment; Q3 next).

Structural differences. Original DSG uses a three-stage few-shot LLM pipeline and binary VQA; Soft-TIFA uses flat, benchmark-supplied atoms and unnormalized open/closed probabilities. SVGFaith-Score uses single-stage 9B decomposition, satisfaction-aligned Yes/No queries (so Yes always means satisfied), unique-ancestor soft gating, and a fixed denominator over all NN requirements (Eq. 3). As additional matched controls with the same untrained Qwen3.5-9B VQA backend: satisfaction-aligned phrasing alone lifts Diagnostic negation from 0.618 to 0.870; three-stage DSG / flat Soft-TIFA (AM) reach Arena 0.658 / 0.797, below SVGFaith-Score 0.880.

Matched gating on identical graphs and 9B VQA outputs (τ=0.01\tau=0.01; Diagnostic = 8-axis equal-weight means):

Aggregation / gatingArena 1,327 (W/T/L)Diag. ImageDiag. Text
No gating (α=0\alpha=0)0.875 (1161/50/116)0.959 (884/27/10)0.962 (889/18/14)
Soft gating (α=0.5\alpha=0.5)0.880 (1168/37/122)0.941 (867/33/21)0.953 (881/14/26)
Continuous hard (α=1.0\alpha=1.0)0.872 (1157/39/131)0.923 (849/35/37)0.938 (868/24/29)
DSG binary hard (zero)0.749 (994/219/114)0.829 (766/118/37)0.881 (818/82/21)
DSG binary hard (drop)0.756 (1003/211/113)0.839 (776/118/27)0.878 (815/93/13)

Direct answer: Soft gating beats all hard-gating variants on every set, with an expected trade-off vs. α=0\alpha=0:

  1. Binary hard gating inflates Arena ties (37→219) and loses 7–13 points; α=1.0\alpha=1.0 also trails α=0.5\alpha=0.5 everywhere.
  2. On Arena, soft gating resolves 13 ties (50→37) and lifts 0.875→0.880 (comparable to mean individual-annotator agreement 0.875). On Diagnostic, where parents are preserved by construction, soft gating mildly compresses margins near τ=0.01\tau=0.01 (0.959/0.962→0.941/0.953) while remaining well above the strongest baseline (0.902/0.857).
  3. Thus satisfaction-aligned decomposition and continuous VQA drive the main gain; soft unique-ancestor gating mainly enforces prerequisite consistency on multi-error real outputs and enables node-level attribution (qiq_i vs. qi−giq_i-g_i). We have added this table to Section 4.4.
AuthorsOpenReview ↗

Response Q3: Training-Stage Ablation & Presentation Fixes

3. Training Stages + Presentation (Question 3 / Weakness 1)

Continuation of our response (Q1–Q2 in prior comments).

As in Table 3, the untuned 397B teacher already reaches Arena 0.876; we distill into a 9B decomposer for latency/cost. Holding the untrained 9B VQA backend and α=0.5,τ=0.01\alpha=0.5, \tau=0.01 fixed under the single-pass thinking protocol (no fallback; truncated outputs scored as 0), plus 397B-judged fact-extraction F1 on 164 held-out prompts:

StageUsable (Arena / Diag.)Strict / Source F1Single-pass ArenaSingle-pass Diag. Image
Base Qwen3.5-9B8.3% / 6.8%0.095 / 0.1000.073 (97/1221/9)0.063 (61/859/1)
Stage 1: Teacher SFT90.6% / 91.4%0.783 / 0.8520.796 (1056/156/115)0.868 (801/101/19)
Stage 2: Rejection-SFT94.9% / 94.9%0.836 / 0.8840.842 (1117/99/111)0.896 (826/70/25)
Stage 3: GRPO (final)96.3% / 96.9%0.857 / 0.9220.850 (1128/83/116)0.909 (838/62/21)

(Deployed SVGFaith-Score adds one no-thinking fallback on the remaining 3.7% [49/1,327] truncated prompts, reaching the reported 0.880 / 0.941.)

Division of labor:

  1. Core semantic decomposition comes mainly from Teacher SFT. With the same fallback retry, all three fine-tuned stages converge near the teacher ceiling (about 0.88 on Arena and 0.94–0.95 on Diagnostic; teacher 0.876).
  2. Rejection-SFT and GRPO mainly cut residual truncations and preserve coverage: single-pass timeouts fall nearly two-thirds (9.4%→3.7% on Arena), accuracy rises monotonically (0.796→0.842→0.850; 0.868→0.896→0.909), and source-aware F1 rises 0.884→0.922 (non-truncated valid: 0.960→0.975). We have added this table to Section 4.4 of the revised manuscript.

4. Presentation

In the uploaded revision we have: (1) added a concise main-text DSG/Soft-TIFA comparison (Sec. 3.3) and the Arena review funnel + pre-review 1,352 results (Secs. 2.2 / 4.2); (2) placed both matched-gating and training-stage tables directly in Sec. 4.4; and (3) tightened paragraph endings on pp. 3/5/6 (conventions., aggregation., present.).

Reviewer_oi8fOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SVGFaith evaluates text-to-SVG faithfulness metrics using controlled diagnostic pairs and human-labelled generator outputs. Existing metrics often miss spatial relations, attribute binding, and negation. The proposed SVGFaith-Score checks a DAG of yes/no requirements and outperforms all baselines, reaching 0.880 on Arena. It still struggles when a strong generator produces the worse output.

Critical strengths
  1. The two layers complement each other: Diagnostic identifies specific failures, while Arena tests real outputs. The per-axis results are particularly useful.

  2. The paper reports limitations clearly, including tie sensitivity, a smaller advantage under half-credit scoring, and its own metric’s drop to 0.516 when Claude loses.

  3. Statistical checks are thorough, with paired bootstraps, confidence intervals for agreement, and sensitivity analysis on adjudicated pairs.

Critical weaknesses
  1. The component gains are limited. Decomposition improves over holistic VQA by only 0.03, from 0.850 to 0.880. The untuned decomposer’s collapse reflects token-budget exhaustion, while the untuned 397B teacher already reaches 0.876. Without an α ablation, the benefit of ancestor gating over averaging remains unclear. Much of the headline margin comes from weaker or older baselines.

  2. Label circularity is a concern. After inspecting metric outputs, the authors flipped 12 Arena labels and removed 25 pairs, reviewing only disagreements with the human majority. The Qwen-based Diagnostic filter may also favor errors detectable by the Qwen scoring backend. Raw agreement is modest (α = 0.568), and 620 majority ties were excluded, weakening the claim of matching human agreement.

Questions for authors

Please address the concerns above about component ablations and possible bias in benchmark filtering and label revision.

1 comment
AuthorsOpenReview ↗

Response to Reviewer oi8f: Component Gains, Gating () Ablation & Pre-Review 1,352 Robustness

Thank you for the thorough review and for recognizing our two-layer design, limitation analyses, and statistical checks. Below we address both concerns with matched controls (no weight or hyperparameter changes), all added to the revised paper (Secs. 2.2, 4.2, 4.4; App. C.3, C.7).

1. Component Gains, Strong Baselines, Gating (α\alpha), and 9B Distillation

(a) +0.030 (0.850 → 0.880) over holistic VQA is substantial. With the same untrained Qwen3.5-9B VQA backend, decomposition resolves 40 more Arena pairs (1128/69/130 → 1168/37/122, CI [+0.012, +0.048]), cuts ties nearly in half (69 → 37), and reduces non-wins from 15.0% to 12.0% (20% relative reduction)—closing the entire remaining gap to single-annotator agreement (0.875), while adding node-level localisation (qiq_i) and prerequisite attribution (qi−giq_i - g_i).

(b) Margins hold against strong 2026 baselines. At τ=0.01\tau=0.01, no baseline holds up on both layers: Qwen3-VL-Embedding-8B leads baselines on Arena (0.843) yet drops to 0.717 / 0.668 on Diagnostic image/text, while Qwen3.5-122B-A10B Judge leads on Diagnostic (0.902 / 0.857) yet drops to 0.754 on Arena (17.9% ties). Using 9B models, SVGFaith-Score beats every baseline in every column (0.880, 0.941, 0.953; paired gaps vs. 122B: +0.127, +0.039, +0.095, all CIs > 0).

(c) Matched α\alpha and hard-gating ablation (fixed graphs + fixed 9B VQA, τ=0.01\tau=0.01).

Aggregation / gatingArena 1,327 (W/T/L)Diag. ImageDiag. Text
No gating, plain mean (α=0\alpha=0)0.875 (1161/50/116)0.959 (884/27/10)0.962 (889/18/14)
Soft gating (α=0.5\alpha=0.5, default)0.880 (1168/37/122)0.941 (867/33/21)0.953 (881/14/26)
Continuous hard (α=1.0\alpha=1.0)0.872 (1157/39/131)0.923 (849/35/37)0.938 (868/24/29)
DSG binary hard (zero / drop)0.749 / 0.756 (ties 219/211)0.829 / 0.8390.881 / 0.878

α=0.5\alpha=0.5 beats α=1.0\alpha=1.0 and DSG binary hard gating across all sets (binary hard inflates Arena ties 37→219, losing 12–13 pts). Vs. α=0\alpha=0, α=0.5\alpha=0.5 resolves 13 ties and lifts Arena 0.875→0.880 where primary entities can be missing; on Diagnostic—where parents are preserved by construction—multiplying by qj<1q_j<1 mildly narrows margins near τ=0.01\tau=0.01 while staying well above 122B (0.902/0.857). Thus decomposition + continuous VQA drive primary discrimination, while soft gating enforces prerequisite consistency and attribution.

(d) 397B teacher (0.876) and training stages. That the untuned 397B teacher reaches 0.876 confirms the paradigm works without fitting benchmark labels (training prompts have zero overlap with SVGFaith). Since base 9B exhausts its token budget on 91.7% of Arena pairs (8.3% usable), we distill 397B into a low-cost 9B model. In single-pass evaluation (no fallback), Teacher SFT imparts core decomposition (90.6% usable, 0.796 Arena / 0.868 Diag.), while Rejection-SFT and GRPO cut truncations by nearly two-thirds (9.4%→3.7%), lifting single-pass accuracy to 0.842→0.850 (0.896→0.909 Diag.) and source-aware F1F_1 from 0.852 to 0.922 (0.880/0.941 with one no-thinking fallback on the remaining 3.7%).

2. Label Revision, Diagnostic Filtering, and Human Agreement

(a) Blind review + untouched 1,352 Arena pairs. Metric disagreements (including baselines) only pooled candidates; per-pair inspection was strictly blind (prompt + two randomised renderings only; scores, predicted winners, prior votes, and generator IDs hidden). Reverting all 12 flips and restoring all 25 removals on the untouched 1,352 pairs (τ=0.01\tau=0.01) leaves all 10 method ranks (#1–#10) identical to the final 1,327 set: #1 SVGFaith-Score 0.865 (1170/38/144), #2 Qwen3-VL-Embedding-8B 0.854 (1155/46/151), #3 ImageReward 0.817, #4 VQAScore 0.780, #5 SigLIP 0.771, #6 HPS 0.762, #7 122B Judge 0.751, #8 Long-CLIP 0.738, #9 CLIP 0.700, #10 PickScore 0.598.

(b) Diagnostic Qwen filter. The gate uses external Qwen3.6-27B (not the 9B scorer), only discards pairs without editing candidates or choosing AA vs. BB, and precedes 3-annotator review (1,799 unanimous + 43 expert). Arena has no semantic gate, and SVGFaith-Score leads there too (1,352 and 1,327).

(c) α=0.568\alpha=0.568 and 620 majority ties. Pairwise evaluation tests strict A≻BA \succ B, so majority-tie pairs have no strict winner by definition. α=0.568\alpha=0.568 (71.2% pairwise agreement) is over 3-way (A/B/tie) votes on a challenge sample oversampling close pairs (62.8% high-value/uncertain), where splits are mostly narrow win vs. tie. The 0.875 reference scores individual votes on the exact 1,327 pairs under the same strict rule (annotator ties also score 0; leave-one-out is 0.873).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.