- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ComplaintBench introduces 87,359 Chinese-language payment complaints with 286,582 screenshot references and 32 operational categories, including insufficient evidence as a non-positive outcome. Its two-stage pipeline combines frozen multimodal evidence extraction with supervised classification, achieving 75.7% recall at 17.0% FPR with a fine-tuned Qwen3.5-9B. However, inspection of the released 20,000-record dataset reveals inconsistencies with the documented evidence schema and limitations in reproducibility.
Critical strengths
- Large, practically relevant dataset with meaningful evidence-sensitive evaluation: The dataset addresses payment-risk assessment involving complaint narratives, transaction screenshots, and incomplete supporting evidence. Its explicit insufficient-evidence category and use of BalancedSuccess, recall, FPR, and subtype-level diagnostics provide a more meaningful evaluation than accuracy alone in a highly imbalanced setting.
- Detailed experimental analysis with appropriately qualified findings: The paper distinguishes cached-evidence classification from direct multimodal assessment and investigates sensitivity to different evidence components and compression strategies. Its full-population closure-rule experiment reports a null result rather than overstating improvements observed in smaller subgroup experiments, strengthening the credibility of that analysis.
Critical weaknesses
- The released dataset does not consistently match the claimed evidence representation and cannot reproduce the benchmark evaluation: Independent inspection of the 20,000 public records finds that only 1,950 (9.75%) contain all seven evidence fields specified in the paper, despite Appendix C describing structural validation requiring these fields. The archive also lacks an identifiable manifest for the frozen 7,000-record test partition, while only 1,000 actual screenshots are supplied for 57,771 image references. These discrepancies make it unclear whether the public representation matches the evidence used in the experiments and prevent independent reproduction of the reported evaluation.
- The experiments do not sufficiently establish evidence-grounded reasoning or isolate the benefits of supervised adaptation: The frozen evidence extractor is not quantitatively evaluated for factual correctness, source support, or omission of decision-relevant information, while the inherited operational labels are not compared explicitly against independently adjudicated labels. Furthermore, only Q-SFT receives extensive task-specific training, and the lack of comparably trained baselines or calibrated operating-point comparisons makes it difficult to determine whether its advantage reflects better evidence assessment or adaptation to the labeling policy.
Questions for authors
- Why do only 1,950 of the 20,000 released records contain all seven required evidence fields, despite the manuscript's claim that retained evidence records pass structural validation? Is the public dataset an intentionally reduced representation, and if so, how can it be reconstructed into the exact evidence format used for evaluation?
- Which public records correspond to the frozen 7,000-record test partition, and how can researchers reproduce the reported experiments or evaluate new models against the same reference set? Can the authors provide the evaluation manifest, scoring code, and an accessible evaluation procedure given the restricted screenshots and unavailable fine-tuned checkpoint?
- Can the authors quantify the frozen extractor's factual accuracy, source-reference correctness, and omission rate on an independently annotated subset? How frequently do independently adjudicated labels disagree with the inherited operational labels, particularly for insufficient-evidence cases?
- Does Q-SFT retain its reported advantage against other supervised classifiers trained on the same structured evidence and against hosted models evaluated at calibrated, matched-recall operating points?