- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EviVQA, a Chinese multimodal e-commerce customer-service dataset with about 104K instances built from real production records. Given a user question and the retrieved evidence (product images, OCR, product info, dialogue history and historical QA), the model must decide whether to Answer, Ask for clarification, Abstain, or Escalate. The authors annotate each sample with a four-layer scheme and compile it into "certificates" for evaluating action selection and factual compliance, with response quality scored separately by an LLM judge. They also train Qwen3-VL-4B with SFT followed by GRPO, adding certificate-based action and factual rewards. Their main finding is that higher response-quality scores do not necessarily mean the model chooses the right action. The trained model improves action selection over both the base model and the strongest general-purpose model, but this does not carry over to factual compliance or response quality.
Critical strengths
- The paper tackles a real, customer-centric problem. It goes beyond answer correctness and asks whether a model should answer, ask, abstain, or escalate.
- The dataset comes from real production records and preserves the retrieved evidence with its missing, conflicting and variant-dependent information, rather than keeping only answerable questions.
- The annotation framework is well documented. The four-layer schema has explicit decision rules and clearly defined certificates, and the annotation prompts are included in the appendix.
- The experiments give several insightful findings. For example, fluent, high-quality responses often still take the wrong action, and even responses with the top judge score show many action errors (Fig. 5).
Critical weaknesses
- Annotation quality is the main concern. About 87% of the data (Bronze) has only one LLM annotation, and agreement between models on the target action is low (κ ≈ 0.55). The paper describes an expert validation but does not report inter-annotator agreement or say clearly which samples were reviewed
- Most of the data goes unused. The authors use only a small fraction of the dataset for training and evaluation, so most of it is never used in the experiments. The value of a 100K-scale dataset is therefore not justified.
- Marginal improvement. The trained model gives only small gains in action accuracy, and the authors themselves treat the gains in their ablation as descriptive. The base model still has a better factual score and fewer violations.
- Shallow verifier. The factual verifier relies largely on substring and keyword matching, which may not be accurate.
- Limited generality and availability. The dataset is Chinese-only and comes from e-commerce production records, which may limit its broader applicability. It is also unclear how much of the benchmark will actually be released publicly.
- Minor presentation issues, such as inconsistent model names and typos.
Questions for authors
Please refer to Weaknesses Section
We combine cross-model checks with human review: 13,152 examples have multiple model annotations. Two domain experts reviewed evaluation L4 target actions; approximately 95% required no revision, and the remainder were corrected (Appendix A.5). We will expand dual-model coverage of Bronze and prioritize disagreements for review.
The dataset covers diverse service scenarios, evidence conditions, and actions. Our controlled study compares data selection and rewards under a matched budget. We plan to extend training to 2,000 distinct examples with the same initialization and reward configuration, then evaluate action selection and factual compliance on the same test set.
The combined configuration exceeds Base by 1.75 percentage points in accuracy and 2.61 points in macro-F1. Tables 3–4 also show the factual trade-off:
Gate violations cover specific certificate constraints—over-refusal, explicit wrong answers, and prohibited claims—not general safety incidents. The gate reward weight is zero (Table 12), so this diagnostic is not directly optimized; action and factual rewards are active. Reporting these dimensions separately exposes the trade-off. GRPO results average three seeds; Base is evaluated once. Factorial contrasts are not significant after correction.
The verifier provides transparent, reproducible, constraint-specific checks beyond raw keyword matching (Appendix C.2):
These checks identify specific constraint violations, while an LLM judge separately assesses response quality. We plan to add semantic checks for paraphrases, negation, and complex conditions, evaluated through human spot checks, while retaining deterministic checks for explicit facts.
The dataset focuses on Chinese e-commerce customer service. We plan to de-identify release data under company policies and applicable data-protection requirements, followed by stricter privacy and security screening. We will release approved data and specify its scope and access method after review.
We will standardize model/checkpoint names against evaluation records and correct typos and configuration labels.