- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SVGFaith, a two-layer meta-evaluation framework for text-to-SVG faithfulness. SVGFaith-Diagnostic contains 1,842 controlled contrastive pairs across eight capability axes and two directions, while SVGFaith-Arena contains 1,327 human-annotated comparisons of real generator outputs. The authors propose SVGFaith-Score, an interpretable pointwise metric that decomposes a prompt into an atomic requirement graph, scores each requirement with a VQA backend, and propagates discounts from prerequisite requirements. Experiments show that SVGFaith-Score achieves 0.880 pairwise accuracy on SVGFaith-Arena, outperforming all nine baselines.
Critical strengths
-
The two-layer design separates controlled diagnosis from real-output reliability: Diagnostic pairs localize specific metric blind spots, while human-annotated Arena pairs test real generator outputs without using any evaluated metric to define the reference ordering.
-
SVGFaith-Score is interpretable and deployable. It leads nine baselines on Arena and both Diagnostic directions, and the ablation dropping Arena accuracy to 0.073 with base Qwen3.5-9B as decomposer shows the gain comes from domain-tuned decomposition.
Critical weaknesses
-
Table 2 compares against mostly natural-image metrics such as CLIP, SigLIP, PickScore, HPS, and ImageReward. These metrics were not designed for SVG and transfer poorly to vector graphics, while SVG-specific evaluators such as SVG-Score, SVGauge (described in the related work) are not included in the comparison.
-
All evaluations are conducted only on the authors’ own SVGFaith benchmarks. SVGFaith-Decomposer is domain-fine-tuned, while the baselines are not tuned for SVG, so its advantage may partly reflect benchmark-specific tuning rather than general evaluator quality. The metric should also be tested on independent SVG benchmarks to show that the gain transfers beyond the proposed evaluation sets.
Questions for authors
Could the authors validate SVGFaith-Score on multiple independent SVG benchmarks beyond the proposed SVGFaith, to demonstrate that its effectiveness generalizes across datasets?
2 comments
Follow-up: Revised Manuscript Updated (Section 4.1 & Additional Controls)
Dear Reviewer 1Hmn,
Thank you again for your time and for noting that your soundness concerns are minor/fixable. We wanted to let you know that we have uploaded the revised manuscript incorporating our clarifications:
- Section 4.1 now explicitly documents the unavailability of public code, model weights, and datasets for both concurrent SVG-Score and reference-based SVGauge at evaluation time, as well as the protocol distinction between reference-based comparison and reference-free prompt-faithfulness scoring.
- Section 4.4 adds further matched controls alongside Table 3, reinforcing that untuned
Qwen3.5-397B-A17B+ untunedQwen3.5-9Balready reaches0.876on Arena (statistically indistinguishable from our distilled 9B decomposer at0.880, with zero prompt overlap between decomposer training data and SVGFaith), confirming that performance does not come from fitting benchmark labels.
Please let us know if our response and these manuscript updates address your remaining concerns, or if there is anything further we can clarify during the discussion period.
Thank you for the questions on baselines and external validation.
Table 2 evaluates the metrics that text-to-SVG papers actually use in practice: image–text embeddings, preference reward models, VQAScore, and a VLM judge. Their poor transfer to vector graphics is one of our main empirical findings rather than a flaw in baseline selection. The table also goes beyond natural-image embeddings: FlanT5-XXL VQAScore and the Qwen3.5-122B-A10B judge are included, and the 122B judge is the strongest baseline in both Diagnostic directions (0.902 and 0.857 at τ = 0.01).
SVG-Score and SVGauge are indeed the closest SVG-specific evaluators, and we will make their status explicit in Section 4.1. Neither had a runnable public release at submission or at the time of this response. SVG-Score provides a project page, while the paper notes that its dataset, weights, and code will be released later. SVGauge states that its SHE dataset will be released upon acceptance, and no public implementation is available. Moreover, SVGauge is reference-based: it compares a generated SVG against a ground-truth reference SVG, whereas SVGFaith-Score is reference-free and scores a single rendering against the prompt alone. Once their official releases become available, we will add both to our comparison.
We do not train the 9B decomposer to beat the 397B teacher. Prompting the untuned 397B teacher as the decomposer already reaches 0.876 on Arena, statistically indistinguishable from the released 9B at 0.880 (95% CI [−0.012, +0.005]). Keeping a 397B model in the scoring loop is too expensive for routine evaluation and for community use after release. We therefore distill that decomposition ability into a 9B model so the metric can run at a much lower GPU cost. The VQA backend remains the off-the-shelf Qwen3.5-9B, and training does not use SVGFaith’s human preference labels. With that same untrained backend, a single holistic query already reaches 0.850 on Arena. The gain is not from fitting the benchmark labels.
Existing public SVG benchmarks (such as SVGenius, SVGEditBench, and VectorGym) evaluate generation or editing models rather than evaluators: they do not provide paired candidates with ground-truth faithfulness orderings, which meta-evaluation requires. The human-rated sets in SVG-Score and SVGauge would be the right external testbeds, and both are not yet publicly available. We will evaluate SVGFaith-Score on them as soon as they are released.