- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CaseTCM, a benchmark for case-preserving cross-source retrieval in traditional Chinese medicine, where answers must be traced to the correct source case rather than only matching a final string. Built from 17 disease monographs and the Chinese Pharmacopoeia, it contains 3,646 pathology questions graded against verbatim case fields and 1,237 herb-name-masked reverse-bridge probes, along with an evaluation protocol that scores both answers and evidence provenance. The authors also provide CaseRouter, a deterministic reference system that routes each question to a single case and copies the requested field. CaseRouter reaches 0.866 target hit compared with 0.545 for the strongest RAG baseline, and most of its remaining errors come from selecting the wrong case. Based on these results, the authors argue that the main bottleneck is preserving case structure rather than retrieval scale.
Critical strengths
- Well-motivated problem in TCM evidence routing. The paper identifies a clear gap in existing Chinese medical benchmarks, which grade only final answers. Requiring answers to be traced to the correct source case reflects how TCM clinicians actually reason across disease monographs and the pharmacopoeia.
- Carefully constructed and reproducible benchmark. Gold answers are taken verbatim from source fields, with automatic coverage checks (99.3%) and a human audit showing high agreement (κ = 0.88). All benchmark files, the evaluation parser, and the reference implementation are frozen and released, supporting reproducibility.
- Diagnostic evaluation protocol with transparent reporting. Beyond aggregate scores, the protocol includes routing controls from random to oracle cases, a paraphrase robustness probe, and an error taxonomy that locate where difficulty lies. The authors also honestly report negative results, such as typed field scoring providing no benefit over raw-text matching.
Critical weaknesses
- Insufficient evidence that the benchmark measures case preservation. The paper attributes the gap between CaseRouter and baselines (0.87 vs. 0.55) to case preservation. However, questions are template-generated from the case fields, and CaseRouter retrieves those same fields by key lookup, so the gap may reflect direct access to the question-generation structure rather than case preservation itself. Case-level BM25 alone reaches 0.815, and paraphrasing drops exact-key routing from 0.84 to 0.38, suggesting heavy template dependence. Baselines should be given the same case-level data, and evaluation on naturally phrased questions is needed.
- Unclear added value of provenance scoring. Since gold answers are verbatim case fields, selecting the wrong case usually yields a wrong answer, so answer-level metrics already capture most of the retrieval quality. Indeed, system rankings are identical under target hit and supported-answer rate, and the routing-error analysis (94.3%) applies only to CaseRouter, where it follows by design. For baselines, provenance scoring only checks whether the gold string appears in the retrieved context, which differs little from passage-level provenance. The authors should report how often baselines answer correctly from the wrong case and whether provenance metrics ever lead to different conclusions.
Questions for authors
Q1: Could the authors report baseline results (e.g., BM25 and Cog-RAG) when given the same case-level data that CaseRouter uses, and on paraphrased or naturally phrased questions? This would help separate the effect of case preservation from access to the question-generation structure.
Q2: Could the authors report, for each baseline, how often answers are correct but come from the wrong case, and whether provenance metrics ever change the conclusions drawn from answer-only metrics?