- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MedShard, a task-structured inference harness for clinical-document quality control. It combines field normalization, evidence-preserving field views, deterministic prechecks, and focused per-rule judgments. For each rule, it uses development data to choose the LLM tier, regex tier, or their union. On 250 Chinese admission notes evaluated against 19 QC rules, MedShard-27B raises micro-F1 from 0.355 with direct prompting to 0.751. The routed 4B system beats the direct-prompt 122B-A10B baseline.
Critical strengths
- The method fits the task well. Each rule can be checked on its own, and the findings can be combined by union. MedShard uses this structure to link each rule to the fields, checks, and judgment it needs. The source of each finding is also recorded, which helps human review.
- All four models improve over their own direct-prompt baselines. Paired record-level bootstrap comparisons and per-rule results help show where the two tiers help each other and where routing fails.
- The paper compares the complete LLM tier, blanket union, and calibrated routing. It also reports poor or mixed results on the scope probes. These tests help explain when the method may be useful.
Critical weaknesses
- The per-rule routes and three targeted template corrections are chosen using only 50 records. These choices may depend too much on this small set. The paper does not show how stable they are across different development sets.
- The main experiment uses one Chinese corpus of 300 records. Format normality and information completeness account for 701 of the 765 test findings (about 92%). The reported micro-F1 therefore may mainly reflects these common rule families. Testing on more benchmarks, such as MedErrBench [1] and MedRECT [2] will help make the paper more convincing.
- Eight rules use precheck-only paths. The LLM tier also includes field normalization and deterministic prechecks. It's unclear how much these steps help compared with rule-level LLM decomposition. Table 3 reports different development branches rather than a controlled test of these parts.
[1] Ma C, Zhang Y, Al-Jazzazi Y, et al. MedErrBench: A Fine-Grained Multilingual Benchmark for Medical Error Detection and Correction with Clinical Expert Annotations. Findings of the Association for Computational Linguistics: ACL 2026.
[2] Iwase N, Okuyama H, Iwasawa J. MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts. https://arxiv.org/abs/2511.00421v2
Questions for authors
Section 4.5 requires labeled data to calibrate the system at a new site. What should users do when local labels are unavailable in new hospital?
The paper can disable the prechecks while keeping the same field views and focused judge prompts. In this test, the focused judge should also check the eight rules that currently use precheck-only paths.