acceptodds
Under review as a conference paper at ICLR 2027

CaseTCM: A Benchmark for Case-Preserving Cross-Source Retrieval in Traditional Chinese Medicine

Abstract

Question answering in traditional Chinese medicine (TCM) depends on links between disease-monograph pathology reasoning and pharmacopoeia herb grounding, and a usable answer has to remain traceable to the evidence route a clinician would recognize. Existing medical and TCM benchmarks, however, grade closed-book answers against a key: they ask whether a final string is right, not whether a system can recover the source case that underwrites it and cross between two structurally different sources while keeping that case intact. We present CaseTCM, a benchmark for case-preserving cross-source retrieval in TCM. It comprises pathology questions drawn from 17 disease monographs, graded against slot-structured gold answers taken verbatim from source fields, together with herb-name-masked, candidate-constrained reverse-bridge probes over five books; the frozen benchmark files ship with a matching protocol of shared prompts, one parser, routing controls, a paraphrase robustness probe and an error taxonomy. Evaluating dense, sparse and graph retrieval, a structured RAG comparator and a deterministic reference system shows that the ceiling here is set by preserving the source case template rather than by retrieval scale: flat retrieval plateaus far below the reference system, typed field scoring adds nothing once candidates are bounded by the routed case, exact-key routing is paraphrase-brittle until a lexical fallback is added, and almost all remaining errors are wrong-case routing rather than answer realization. We anticipate that CaseTCM will support retrieval systems whose evidence routes can be inspected field by field. The benchmark files, the evaluation protocol and the reference implementation are released at https://anonymous.4open.science/r/Case-tcm-code.

Primary area:
datasets and benchmarks
Keywords:
retrieval-augmented generation, benchmark, provenance, cross-source retrieval, traditional Chinese medicine

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3NuGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CaseTCM, a benchmark for case-preserving cross-source retrieval in traditional Chinese medicine, where answers must be traced to the correct source case rather than only matching a final string. Built from 17 disease monographs and the Chinese Pharmacopoeia, it contains 3,646 pathology questions graded against verbatim case fields and 1,237 herb-name-masked reverse-bridge probes, along with an evaluation protocol that scores both answers and evidence provenance. The authors also provide CaseRouter, a deterministic reference system that routes each question to a single case and copies the requested field. CaseRouter reaches 0.866 target hit compared with 0.545 for the strongest RAG baseline, and most of its remaining errors come from selecting the wrong case. Based on these results, the authors argue that the main bottleneck is preserving case structure rather than retrieval scale.

Critical strengths
  1. Well-motivated problem in TCM evidence routing. The paper identifies a clear gap in existing Chinese medical benchmarks, which grade only final answers. Requiring answers to be traced to the correct source case reflects how TCM clinicians actually reason across disease monographs and the pharmacopoeia.
  2. Carefully constructed and reproducible benchmark. Gold answers are taken verbatim from source fields, with automatic coverage checks (99.3%) and a human audit showing high agreement (κ = 0.88). All benchmark files, the evaluation parser, and the reference implementation are frozen and released, supporting reproducibility.
  3. Diagnostic evaluation protocol with transparent reporting. Beyond aggregate scores, the protocol includes routing controls from random to oracle cases, a paraphrase robustness probe, and an error taxonomy that locate where difficulty lies. The authors also honestly report negative results, such as typed field scoring providing no benefit over raw-text matching.
Critical weaknesses
  1. Insufficient evidence that the benchmark measures case preservation. The paper attributes the gap between CaseRouter and baselines (0.87 vs. 0.55) to case preservation. However, questions are template-generated from the case fields, and CaseRouter retrieves those same fields by key lookup, so the gap may reflect direct access to the question-generation structure rather than case preservation itself. Case-level BM25 alone reaches 0.815, and paraphrasing drops exact-key routing from 0.84 to 0.38, suggesting heavy template dependence. Baselines should be given the same case-level data, and evaluation on naturally phrased questions is needed.
  2. Unclear added value of provenance scoring. Since gold answers are verbatim case fields, selecting the wrong case usually yields a wrong answer, so answer-level metrics already capture most of the retrieval quality. Indeed, system rankings are identical under target hit and supported-answer rate, and the routing-error analysis (94.3%) applies only to CaseRouter, where it follows by design. For baselines, provenance scoring only checks whether the gold string appears in the retrieved context, which differs little from passage-level provenance. The authors should report how often baselines answer correctly from the wrong case and whether provenance metrics ever lead to different conclusions.
Questions for authors

Q1: Could the authors report baseline results (e.g., BM25 and Cog-RAG) when given the same case-level data that CaseRouter uses, and on paraphrased or naturally phrased questions? This would help separate the effect of case preservation from access to the question-generation structure.

Q2: Could the authors report, for each baseline, how often answers are correct but come from the wrong case, and whether provenance metrics ever change the conclusions drawn from answer-only metrics?

Reviewer_GQ1FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents CASETCM, a benchmark for testing whether retrieval systems can preserve the case structure used in traditional Chinese medicine. It includes 3,646 pathology questions from 17 disease monographs and 1,237 herb-name-masked reverse-bridge questions from five books. The benchmark scores both the answer and its evidence path, requiring a system to identify the correct case, retain the syndrome, pathogenesis, therapy, and formula relationships, and connect the case to the relevant pharmacopoeia entry.

The main contributions are the structured benchmark, the frozen evaluation protocol, provenance-aware metrics, routing and paraphrase controls, and the deterministic CaseRouter reference system. The authors compare it with dense, sparse, and graph-based retrieval methods. CaseRouter obtains a target-hit score of 0.8661, compared with 0.5448 for the strongest baseline, and reaches 0.9863 on the reverse-bridge task. Ablation studies and error analysis indicate that correct case routing and construction of the local herb candidate set account for most of the performance gain, while the remaining errors are mainly caused by routing to the wrong case.

Critical strengths
  1. It evaluates a concrete problem that standard QA benchmarks miss. CASETCM requires the system to recover the correct TCM case and its supporting evidence, so answers assembled from neighboring cases can be identified as errors.

  2. The experiments help explain the source of the performance gains. The routing controls, paraphrase test, ablations, and error analysis separate case selection from answer generation and show that most errors come from routing to the wrong case.

Critical weaknesses
  1. The comparison is not fully fair. CaseRouter uses typed case records and copies answers directly, while the baselines retrieve generic chunks and generate responses. The reported gap therefore may reflect this privileged setup rather than better retrieval.

  2. The benchmark may favor exact matching. Questions are generated from the same case slots used for routing and often retain explicit anchors. Results on held-out cases, books, or naturally written queries are not shown, so generalization remains uncertain.

Questions for authors
  1. How do you ensure that the comparison with the generative baselines is fair, given that CaseRouter directly accesses typed case fields and copies answers from them?

  2. Were any cases, books, or question templates held out during benchmark construction and routing development? If not, how do you assess generalization beyond the benchmark’s exact anchors?

  3. Which CaseRouter configuration corresponds to each reported score, especially the differences between Tables 3, 5, and 6?

Reviewer_DDFSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CaseTCM introduces a benchmark for case-preserving retrieval in traditional Chinese medicine (TCM), covering 3,646 pathology questions and 1,237 cross-source herb retrieval probes. It evaluates several RAG methods against a deterministic reference system, CaseRouter, which retrieves structured cases and directly copies target fields. The authors argue that preserving case structure, rather than retrieval scale, is the main bottleneck.

Critical strengths
  1. Meaningful benchmark design. Evaluating evidence provenance and case consistency rather than answer correctness alone is valuable. The routing controls, paraphrase tests, and error taxonomy provide useful diagnostics.

  2. Substantial engineering effort. The benchmark covers multiple books and question types, with released code, evaluation protocols, and detailed documentation.

Critical weaknesses
  1. Evaluation design favors the reference system and does not isolate retrieval quality. Questions are generated from structured case fields using fixed templates, allowing CaseRouter to reverse-match the question and copy the gold answer directly, while baselines rely on a 4B generator. This advantage is highly sensitive to wording: exact routing drops from 0.84 to 0.38 under paraphrasing (Section 6.3). More importantly, even on the anchor book with only 39 cases, RAG baselines perform poorly (Table 4), suggesting that generation or answer matching, rather than retrieval failure, may explain much of the gap. Without reporting gold-case retrieval recall for baselines, the central claim about evidence drift remains unsubstantiated.

  2. Questionable data provenance and insufficient validation. The 17 source books are not explicitly identified, and the released data appear to represent guideline-style syndrome entries rather than clinical cases. The pharmacopoeia-derived entries also raise concerns about source fidelity, including unexpected fields, missing cautions, and patent-medicine entries. Furthermore, redistribution rights are unclear despite the CC BY claim. Human validation covers only 50 of 4,883 items, focuses on agreement with extracted fields, and excludes reverse-bridge probes. These issues undermine confidence in the benchmark's validity and release readiness.

Questions for authors
  1. Can the authors report gold-case retrieval recall for each baseline, separately from answer accuracy? How would results change under paraphrased questions and gold-case-based evaluation, rather than exact-template matching?

  2. What are the exact sources and redistribution rights of the released data? Can the authors explain the pharmacopoeia inconsistencies and expand human validation to include extraction fidelity and reverse-bridge questions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.