acceptodds
Under review as a conference paper at ICLR 2027

EviVQA: A Dataset and Benchmark for Evidence-Grounded Evaluation of E-Commerce Visual Question Answering

Abstract

E-commerce customer service requires multimodal models to decide not only what to answer, but whether to answer, ask for clarification, abstain, or escalate. Existing benchmarks centered on answer correctness or overall response quality do not fully capture these decisions when evidence is incomplete, conflicting, or beyond available service capabilities. We introduce EviVQA, a Chinese multimodal dataset and benchmark comprising 103,949 instances derived from production e-commerce records. Its retrieval-native inputs combine product images, OCR text, structured attributes, dialogue context, and historical question–answer pairs, while preserving missing information, variant ambiguity, and conflicting evidence. A four-layer annotation scheme connects service scope, evidence sufficiency and information deficits, grounded facts, response constraints, and target actions. These annotations are compiled into decision and evidence certificates, enabling separate evaluation of action selection, factual compliance, and model-judged response quality. On human-annotated, action-balanced examples, we find that higher response-quality scores do not consistently correspond to better action selection, revealing a gap between fluent responses and appropriate customer-service behavior. Combining data curation with certificate-based rewards improves action selection over strong general-purpose and same-backbone baselines, including a 7.6% relative gain in macro-F1 over the strongest fully evaluated general-purpose model. The results further show that improvements in action selection do not necessarily translate into uniform gains in factual compliance or response quality. EviVQA provides a benchmark for studying evidence-grounded customer-service decision making under incomplete information and limited service capabilities.

Primary area:
datasets and benchmarks
Keywords:
E-commerce, Visual question answering, Multimodal benchmarks, Evidence grounding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PuDCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EviVQA, a Chinese multimodal e-commerce customer-service dataset with about 104K instances built from real production records. Given a user question and the retrieved evidence (product images, OCR, product info, dialogue history and historical QA), the model must decide whether to Answer, Ask for clarification, Abstain, or Escalate. The authors annotate each sample with a four-layer scheme and compile it into "certificates" for evaluating action selection and factual compliance, with response quality scored separately by an LLM judge. They also train Qwen3-VL-4B with SFT followed by GRPO, adding certificate-based action and factual rewards. Their main finding is that higher response-quality scores do not necessarily mean the model chooses the right action. The trained model improves action selection over both the base model and the strongest general-purpose model, but this does not carry over to factual compliance or response quality.

Critical strengths
  • The paper tackles a real, customer-centric problem. It goes beyond answer correctness and asks whether a model should answer, ask, abstain, or escalate.
  • The dataset comes from real production records and preserves the retrieved evidence with its missing, conflicting and variant-dependent information, rather than keeping only answerable questions.
  • The annotation framework is well documented. The four-layer schema has explicit decision rules and clearly defined certificates, and the annotation prompts are included in the appendix.
  • The experiments give several insightful findings. For example, fluent, high-quality responses often still take the wrong action, and even responses with the top judge score show many action errors (Fig. 5).
Critical weaknesses
  • Annotation quality is the main concern. About 87% of the data (Bronze) has only one LLM annotation, and agreement between models on the target action is low (κ ≈ 0.55). The paper describes an expert validation but does not report inter-annotator agreement or say clearly which samples were reviewed
  • Most of the data goes unused. The authors use only a small fraction of the dataset for training and evaluation, so most of it is never used in the experiments. The value of a 100K-scale dataset is therefore not justified.
  • Marginal improvement. The trained model gives only small gains in action accuracy, and the authors themselves treat the gains in their ablation as descriptive. The base model still has a better factual score and fewer violations.
  • Shallow verifier. The factual verifier relies largely on substring and keyword matching, which may not be accurate.
  • Limited generality and availability. The dataset is Chinese-only and comes from e-commerce production records, which may limit its broader applicability. It is also unclear how much of the benchmark will actually be released publicly.
  • Minor presentation issues, such as inconsistent model names and typos.
Questions for authors

Please refer to Weaknesses Section

1 comment
  1. We combine cross-model checks with human review: 13,152 examples have multiple model annotations. Two domain experts reviewed evaluation L4 target actions; approximately 95% required no revision, and the remainder were corrected (Appendix A.5). We will expand dual-model coverage of Bronze and prioritize disagreements for review.

  2. The dataset covers diverse service scenarios, evidence conditions, and actions. Our controlled study compares data selection and rewards under a matched budget. We plan to extend training to 2,000 distinct examples with the same initialization and reward configuration, then evaluate action selection and factual compliance on the same test set.

  3. The combined configuration exceeds Base by 1.75 percentage points in accuracy and 2.61 points in macro-F1. Tables 3–4 also show the factual trade-off:

    ModelAccuracy (%)Macro-F1 (%)Factual scoreGate violations (%)
    Base45.7539.940.9114.00
    D₀R₀46.2541.040.8827.00
    D₁R₁47.5042.550.8807.58

    Gate violations cover specific certificate constraints—over-refusal, explicit wrong answers, and prohibited claims—not general safety incidents. The gate reward weight is zero (Table 12), so this diagnostic is not directly optimized; action and factual rewards are active. Reporting these dimensions separately exposes the trade-off. GRPO results average three seeds; Base is evaluated once. Factorial contrasts are not significant after correction.

  4. The verifier provides transparent, reproducible, constraint-specific checks beyond raw keyword matching (Appendix C.2):

    TargetCheck
    Numeric values/rangesUnit normalization and numeric/range comparisons
    Textual valuesSubstring comparisons
    Conditional factsFact checks plus condition keywords
    Wrong answers/prohibited claimsCertificate-based substring checks

    These checks identify specific constraint violations, while an LLM judge separately assesses response quality. We plan to add semantic checks for paraphrases, negation, and complex conditions, evaluated through human spot checks, while retaining deterministic checks for explicit facts.

  5. The dataset focuses on Chinese e-commerce customer service. We plan to de-identify release data under company policies and applicable data-protection requirements, followed by stricter privacy and security screening. We will release approved data and specify its scope and access method after review.

  6. We will standardize model/checkpoint names against evaluation records and correct typos and configuration labels.

Reviewer_MEWfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EviVQA, a dataset and benchmark for evidence-grounded visual question answering in Chinese e-commerce customer service. It contains 103,949 instances combining user queries, dialogue context, product images, OCR, structured product information, and historical QA. Its four-layer annotation framework connects service scope, evidence sufficiency and deficit attribution, evidence-grounded factual constraints, and four response actions: Answer, Ask, Abstain, and Escalate. These annotations are compiled into certificates for evaluating action selection and certificate-scoped factual compliance separately from response quality.

The paper evaluates models on a 400-instance, action-balanced subset with expert-validated target-action labels. It also conducts a three-seed factorial experiment examining certificate-rich training data and certificate-based rewards. The results illustrate discrepancies between response-quality scores and action appropriateness. The combined training configuration produces modest descriptive improvements in action selection, without consistent improvements in factual compliance; the authors explicitly report that the factorial contrasts are not statistically significant after correction.

Critical strengths
  1. A useful formulation of evidence-grounded response decisions in a realistic setting. EviVQA goes beyond answer correctness by distinguishing whether a system should answer, request indispensable information, abstain, or escalate. Connecting these actions to evidence sufficiency and the source of missing information makes the benchmark more diagnostic than a single response-quality score. Its production-derived examples preserve incomplete evidence, product-variant ambiguity, and service constraints, making this a consequential contribution for evaluating customer-service systems.

  2. Controlled experiments with transparent reporting of limitations and trade-offs. The three-seed factorial experiment separates the effects of the data-selection policy and reward configuration. The paper reports action metrics, factual compliance, and response quality, and acknowledges both the absence of corrected statistical significance and the lack of uniform improvement across metrics. This supports a measured interpretation of the results and provides a useful basis for subsequent research.

Critical weaknesses
  1. The response-action classifier lacks reported validation against independent human judgments. Appendix C.2 identifies the realized action primarily from the response’s primary clause, extending cue search to the full response under specified hedging conditions. This raises a plausible concern for mixed responses that provide supported information before asking a necessary clarification question. Figure 8, Case 1 illustrates such a response structure, although it does not establish that the implemented classifier actually misclassifies it. The expert validation described in Appendix A.5 concerns the target-action labels, rather than the classifier’s interpretation of generated responses. The manuscript does not report per-action agreement between this classifier and independent human judgments. Because action recognition is used in both reward computation and evaluation, this missing validation limits confidence in the reported action-specific failure rates and small performance differences. 29905_EviVQA_A_Dataset_and_Ben

  2. The validation of evidence certificates and the boundary of permitted inference are insufficiently documented. Appendix A.5 establishes expert review of target actions, but does not describe a comparable audit of L3 evidence values, provenance, required qualifiers, and forbidden claims. These fields determine what the verifier rewards or penalizes, making their reliability central to the benchmark’s factual-compliance measurements. There is also an ambiguity in the annotation policy. Figure 7 prohibits using common sense to fill in answers, whereas Figure 8, Case 2 explicitly invokes common sense to recommend against continuous charging despite missing product-specific information about overcharge protection. This could represent permissible general precautionary advice, rather than an incorrect annotation. Nevertheless, the paper should clarify that distinction and explain how it is applied consistently when constructing certificates and evaluating responses.

Questions for authors
  1. Evaluation validation: Have independent human annotators assessed the realized actions of generated responses, particularly responses combining a partial answer with a clarification request or escalation? Please report the classifier’s per-action agreement or confusion matrix on an audited sample, and whether the very low Ask recall and main model comparisons persist under human action judgments.

  2. Annotation scope and policy clarification: Did expert validation cover only target-action labels, or also L3 evidence values, provenance, qualifiers, and forbidden claims? Please specify the audited sample sizes and any available correction rates. Additionally, how does the policy distinguish permissible general precautionary advice, as in Figure 8, Case 2, from answers that impermissibly supplement the supplied evidence with common sense?

1 comment
  1. Thank you for highlighting the distinction between target-action annotation and recognition of actions realized in generated responses. To address this issue, we propose randomly sampling 200 existing model-generated responses for independent human assessment and comparison with the classifier’s predictions. The following results are simulated illustrations, not findings from a completed human audit.

    ActionPrecision (%)Recall (%)F1 (%)
    Answer88.8996.5592.56
    Ask83.3350.0062.50
    Abstain89.2989.2989.29
    Escalate94.1288.8991.43
    Macro average88.9181.1883.94

    In this simulated example, overall agreement is 89.50%, but Ask recall is only 50.00%. Of the 20 responses labeled Ask by the hypothetical human judgments, eight are classified as Answer. This illustrates how high overall agreement can conceal missed clarification actions. Responses that provide partial information before requesting necessary clarification are therefore an important focus for the audit. The aggregate Escalate recall of 88.89% does not establish accuracy specifically for responses combining partial answers with escalation.

    These metrics measure the classifier’s recognition accuracy. The original Ask recall of 2.7%–3.3% instead measures whether models ask for clarification when the target action is Ask. Determining whether low Ask recall and the main model comparisons persist under human judgments requires actual annotations linked to target actions and model identities; the simulated results cannot establish those conclusions.

  2. We combine model-based evaluation with human review. The LLM judge has access to the complete sample information, including L3 evidence and associated constraints. Human experts focus on reviewing L4 target actions and assessing whether model responses are acceptable. This division of responsibilities concentrates limited human-review resources on final decisions and response acceptability. We will clarify the responsibilities, review scope, and sample sizes for both forms of evaluation in the revised manuscript.

    Regarding Figure 8, Case 2, we will clarify the scope of the annotation rule prohibiting the use of common sense to fill in answers. This rule prohibits supplementing unsupported product-specific facts with common sense; it does not exclude explicitly framed general precautionary advice. The advice in this case should not be interpreted as indicating that the product lacks overcharge protection, nor should it serve as evidence of that product property. We will explicitly distinguish product facts from general advice in the annotation guidelines so that certificate construction and response evaluation follow the same boundary.

Reviewer_2ESSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EviVQA, a multimodal dataset and benchmark for Chinese e-commerce customer service. Unlike conventional visual question answering benchmarks that primarily focus on answer correctness, EviVQA aims to evaluate whether models can select appropriate response actions when faced with incomplete, conflicting, or insufficient evidence. The benchmark defines four response actions: Answer, Ask, Abstain, and Escalate. To support these decisions, the authors propose a four-layer annotation framework covering user intent and service scope, evidence sufficiency and information-deficit attribution, evidence grounding and verifiability, and target response actions. The authors further convert these annotations into decision certificates and evidence certificates, which specify evidence-supported facts, prohibited claims, required qualifications, and evidence provenance. These certificates are used to separately evaluate action selection, factual compliance, and response quality, while also providing additional supervision signals for reinforcement learning. Overall, the main contribution of this work is a dataset and evaluation framework for studying evidence-grounded decision-making in multimodal customer-service scenarios.

Critical strengths
  1. The authors argue that the decision space for customer-service agents in multimodal e-commerce question answering should extend beyond simply answering or refusing to answer, highlighting the necessity of Ask and escalation Escalate actions. This work also provides a valuable perspective on trustworthy decision-making in customer-service scenarios, where agents must operate under evidence constraints and within the boundaries of their service capabilities.

  2. EviVQA is constructed from real-world e-commerce customer-service records and preserves challenging scenarios involving missing information, conflicting evidence, product-variant ambiguity, and limitations on external-system access. Its scale and task coverage provide a useful foundation for studying evidence-grounded multimodal decision-making.

  3. The paper incorporates supported content, prohibited claims, required qualifiers, and evidence provenance into a structured representation, and attempts to translate this information into executable evaluation and training signals. This design enables certain factual errors to be verified through programmatic rules, rather than relying entirely on holistic evaluations by LLM-as-a-Judge.

Critical weaknesses
  1. The improvements over the base and same-backbone models are modest, raising questions about the effectiveness of the proposed dataset for training. Ablation results suggest that most gains come from reward design rather than data curation.

  2. Most samples are annotated by a single model, and inter-model agreement on target actions is only moderate, suggesting potential ambiguity and noise in the annotations.

  3. The benefits of the four-layer annotations and evidence certificates over simpler approaches remain unclear. Nearly all External system (22,722/22,724) and Human decision (4,365/4,368) samples are labeled Escalate, suggesting that service-scope classification alone may explain some decisions. Comparisons with simpler baselines are needed.

Questions for authors
  1. Given the limited performance gains and the larger contribution of reward design, how can the authors further demonstrate the effectiveness and value of the proposed dataset?

  2. How do the authors ensure the quality and reliability of the annotations, considering that most samples are annotated by a single model and inter-model agreement on target actions is only moderate?

  3. Can the authors compare their framework with simpler rule-based or service-scope classification baselines to demonstrate the necessity and effectiveness of the four-layer annotations and evidence certificates?

1 comment
  1. Thank you for this suggestion. EviVQA serves two purposes: identifying action errors that response-quality scores miss, and supplying action and evidence signals for training.

    Figure 2 illustrates the first purpose. Judge pass rates increase from 54.6% to 82.3%, while the non-refusal proxy on teacher-answerable cases falls from 68.2% to 40.4%. A higher response score therefore does not necessarily mean a more appropriate service action.

    For training, the combined configuration achieves 42.55% macro-F1 in Table 3, exceeding the strongest fully evaluated general-purpose model by 3.01 points and the same-backbone Base and SFT models by 2.61 and 1.67 points. Table 4 separates data selection from reward augmentation:

    ConfigurationAction accuracy (%)Macro-F1 (%)
    Baseline D₀R₀46.2541.04
    Reward augmentation only D₀R₁47.0841.95
    Data selection only D₁R₀46.3341.31
    Combined D₁R₁47.5042.55

    The reward effect is larger. However, the data factor compares two selection policies within EviVQA, not training with versus without EviVQA. These three-seed results demonstrate a preliminary training use; the factorial contrasts are not significant after correction, so we do not claim an established training advantage.

  2. We improve annotation reliability through cross-model checks and expert review. Already, 13,152 examples have multiple model annotations. Two domain experts also reviewed evaluation target actions: approximately 95% required no revision, and the remaining cases were reviewed and corrected.

    Appendices A.3 and A.5 report:

    CheckResult
    Multi-model coverage13,152 examples, or 12.65%
    Service-scope agreement85.08%, κ=0.675
    Target-action agreement66.16%, κ=0.549
    Expert review of evaluation targetsApproximately 95% accepted without revision; remaining cases corrected

    Action agreement shows that some boundaries remain difficult. We plan to expand dual-model annotation of Bronze examples and use disagreements to identify cases for review.

  3. Service scope explains many escalation decisions, but not all actions. In Table 8, examples within the same Public evidence scope have three major target actions:

    Service scopeAnswerAskAbstainEscalate
    Public evidence54,0526,46514,2652
    External system02022,722
    Human decision0304,365

    After identifying scope, a system still needs to decide whether evidence is sufficient and who can supply missing information. This is the role of L2. L3 specifies allowable facts and conditions: for example, evidence stating “basic version: 3L; upgraded version: 5L” does not support “all versions are 5L.” The hierarchy supports both action decisions and checks on response content.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.