acceptodds
Withdrawn from ICLR 2027

ComplaintBench: A Dataset and Benchmark for Evidence-Grounded Payment Complaint Assessment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce ComplaintBench, a dataset and diagnostic benchmark for evidence-grounded risk assessment in Chinese-language payment complaints: 87,359 records with 286,582 screenshot references, each carrying a narrative, chat and payment screenshots, a binary positive / non-positive decision, and one of 32 operational categories. The decision policy places ordinary disputes and insufficient-evidence cases in the non-positive class, so “unverifiable” is a legitimate outcome rather than an abstention. A two-stage reference pipeline extracts source-linked evidence from text and screenshots with a frozen vision-language model and then classifies from the resulting text view; we compare it with six hosted services under matched evidence and under direct raw input. At 16.93% positive prevalence, accuracy is not a usable criterion—a constant non-positive control scores 83.1% accuracy, above every evaluated system, with zero recall—so we report BalancedSuccess alongside recall and false-positive rate. Fine-tuning determines the error direction: the complaint-fine-tuned model reaches 75.7% recall at a 17.0% false-positive rate, against 74.8% recall at 43.5% for the same model without complaint-specific fine-tuning (BalancedSuccess 79.4 versus 65.7), whereas every hosted service buys 71–100% recall only at a 34–65% false-positive rate that concentrates on insufficient-evidence records. A de-identified 20,000-record subset with its extracted evidence is released as supplementary material.

Primary area:
datasets and benchmarks
Keywords:
datasets and benchmarks, evidence-grounded reasoning, payment complaint assessment, multimodal evidence, evidence sufficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5RTSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ComplaintBench introduces 87,359 Chinese-language payment complaints with 286,582 screenshot references and 32 operational categories, including insufficient evidence as a non-positive outcome. Its two-stage pipeline combines frozen multimodal evidence extraction with supervised classification, achieving 75.7% recall at 17.0% FPR with a fine-tuned Qwen3.5-9B. However, inspection of the released 20,000-record dataset reveals inconsistencies with the documented evidence schema and limitations in reproducibility.

Critical strengths
  1. Large, practically relevant dataset with meaningful evidence-sensitive evaluation: The dataset addresses payment-risk assessment involving complaint narratives, transaction screenshots, and incomplete supporting evidence. Its explicit insufficient-evidence category and use of BalancedSuccess, recall, FPR, and subtype-level diagnostics provide a more meaningful evaluation than accuracy alone in a highly imbalanced setting.
  2. Detailed experimental analysis with appropriately qualified findings: The paper distinguishes cached-evidence classification from direct multimodal assessment and investigates sensitivity to different evidence components and compression strategies. Its full-population closure-rule experiment reports a null result rather than overstating improvements observed in smaller subgroup experiments, strengthening the credibility of that analysis.
Critical weaknesses
  1. The released dataset does not consistently match the claimed evidence representation and cannot reproduce the benchmark evaluation: Independent inspection of the 20,000 public records finds that only 1,950 (9.75%) contain all seven evidence fields specified in the paper, despite Appendix C describing structural validation requiring these fields. The archive also lacks an identifiable manifest for the frozen 7,000-record test partition, while only 1,000 actual screenshots are supplied for 57,771 image references. These discrepancies make it unclear whether the public representation matches the evidence used in the experiments and prevent independent reproduction of the reported evaluation.
  2. The experiments do not sufficiently establish evidence-grounded reasoning or isolate the benefits of supervised adaptation: The frozen evidence extractor is not quantitatively evaluated for factual correctness, source support, or omission of decision-relevant information, while the inherited operational labels are not compared explicitly against independently adjudicated labels. Furthermore, only Q-SFT receives extensive task-specific training, and the lack of comparably trained baselines or calibrated operating-point comparisons makes it difficult to determine whether its advantage reflects better evidence assessment or adaptation to the labeling policy.
Questions for authors
  1. Why do only 1,950 of the 20,000 released records contain all seven required evidence fields, despite the manuscript's claim that retained evidence records pass structural validation? Is the public dataset an intentionally reduced representation, and if so, how can it be reconstructed into the exact evidence format used for evaluation?
  2. Which public records correspond to the frozen 7,000-record test partition, and how can researchers reproduce the reported experiments or evaluate new models against the same reference set? Can the authors provide the evaluation manifest, scoring code, and an accessible evaluation procedure given the restricted screenshots and unavailable fine-tuned checkpoint?
  3. Can the authors quantify the frozen extractor's factual accuracy, source-reference correctness, and omission rate on an independently annotated subset? How frequently do independently adjudicated labels disagree with the inherited operational labels, particularly for insufficient-evidence cases?
  4. Does Q-SFT retain its reported advantage against other supervised classifiers trained on the same structured evidence and against hosted models evaluated at calibrated, matched-recall operating points?
Reviewer_FShGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ComplaintBench, a dataset and benchmark for evidence-grounded risk assessment in Chinese-language payment complaints. It contains 87,359 records with 286,582 screenshot references, each associated with a complaint narrative, a binary positive/non-positive label, and one of 32 operational categories. A central design choice is to treat ordinary disputes, explicitly benign cases, and insufficient-evidence cases as non-positive, making “the submitted evidence does not justify a positive decision” a legitimate outcome.

The authors propose a two-stage reference pipeline: a frozen vision-language model extracts structured, source-linked evidence from narratives and screenshots; a fine-tuned text classifier then predicts the binary decision and auxiliary category from the extracted representation. Their evaluation compares this pipeline with the same model family without complaint-specific fine-tuning and several hosted models, under both shared evidence-conditioned (EC) and direct raw-input (RI) conditions. The evaluation emphasizes recall, false-positive rate (FPR), and BalancedSuccess rather than accuracy alone, given the dataset’s 16.93% positive prevalence.

Critical strengths
  1. A meaningful and clearly specified task. The paper focuses on whether the submitted evidence supports a risk decision, rather than simply detecting negative sentiment or classifying complaint text. Treating insufficient evidence as a substantive non-positive outcome is particularly relevant to the intended application.

  2. Evaluation metrics that expose operational trade-offs. The authors correctly show why accuracy is misleading under class imbalance: always predicting non-positive achieves 83.1% accuracy but has zero recall. Reporting recall and FPR alongside BalancedSuccess makes the trade-off between detecting positive cases and avoiding unsupported positive decisions visible.

Critical weaknesses
  1. Limited reproducibility of the main experimental results. All reported experiments appear to rely on the full 87,359-record corpus, which is not publicly available, while the public release contains only a 20,000-record subset. The paper does not report a separate evaluation using only the publicly accessible subset. Consequently, researchers without restricted access cannot independently verify the main empirical findings using the released data. Given that the paper's primary contributions are a dataset and benchmark, I consider that the authors should report results using a reproducible train/validation/test split drawn exclusively from the public subset, including the corresponding baseline performance and evaluation protocol. This would allow the community to assess the reported findings independently, even if the full corpus must remain restricted for privacy reasons.
  2. Related-work coverage. The paper discusses MulComp and general evidence-verification benchmarks, but does not systematically compare ComplaintBench with existing complaint datasets. A comprehensive study contrasting task, modality, labels, evidence assumptions, and evaluation would better substantiate the novelty claim of this framework.
  3. Limited scope of the reference-label audit. The independent audit covers only 300 records from the 7,000-record test set, while the dataset contains 32 operational categories. Although the reported first-pass agreement is high, the paper does not establish whether this sample is sufficient to characterize label reliability across the full test set and individual categories. The authors should justify the audit sample size and selection procedure and report category-level agreement where feasible.
Questions for authors
  1. Can you report a fully reproducible public-subset experiment? This would allow independent verification of results without access to the restricted corpus.
  2. How does ComplaintBench compare systematically with existing complaint datasets? This would make the benchmark’s novelty easier to assess.
  3. How sensitive are the findings to rare categories and alternative taxonomies?
Reviewer_yXvuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ComplaintBench is a Chinese-language dataset and benchmark for evidence-grounded payment-complaint assessment. It contains 87,359 records and 286,582 screenshot references, with narratives, transaction/conversation screenshots, a binary decision, and one of 32 operational categories. Ordinary disputes, explicitly benign cases, and insufficient-evidence cases are non-positive under the benchmark policy. A frozen vision-language model extracts seven structured evidence fields, after which a Qwen3.5 9B text classifier is fine-tuned on the compressed representation. The paper distinguishes shared cached-evidence evaluation from direct raw-input multimodal assessment and emphasizes recall, false-positive rate, and BalancedSuccess because positive prevalence is only 16.93%. On 7,000 test records, the fine-tuned reference reaches 75.7% recall at 17.0% FPR, whereas hosted services generally obtain higher recall at much larger FPR. A nested 300-record audit checks reference labels, and a de-identified 20,000-record public tier is released.

Critical strengths
  1. The task makes a useful distinction between an allegation and evidence supporting it. Treating insufficient evidence as a valid outcome makes the benchmark more realistic.
  2. The main comparison exposes errors that accuracy alone hides. Under the same cached evidence, fine-tuning reduces FPR from 43.5% to 17.0% with similar recall; constant-label controls help interpret the class imbalance.
  3. The authors document the taxonomy, input limits, label audit, and access restrictions. They also acknowledge that cached-evidence and raw-input results cannot isolate the benefit of the two-stage pipeline.
Critical weaknesses
  1. The shared evidence cache has only been inspected qualitatively. There are no measured rates of incorrect facts, missing facts, or wrong source references, making it difficult to tell how much error comes from extraction versus classification
  2. The 300-record audit reports strong overall agreement, but it includes only 51 positive records across a 32-category task. This leaves limited evidence about rare-category label quality, even though the audit is useful for checking the sampled records
  3. The split checks rule out identical inputs and shared image files, but do not establish that the same people, accounts, or very similar complaints are absent from both training and test sets. The paper also does not test on a later time period. This leaves uncertainty about performance on genuinely new cases
Questions for authors

See Weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.