acceptodds
Under review as a conference paper at ICLR 2027

MedShard: Task-Structured Inference Improves Clinical Quality Control across Model Scales

Abstract

Structured clinical-document audits require a language model to track many rules, locate their evidence, and emit a variable number of findings. We introduce MedShard, a task-structured inference harness that compiles this workload into rule-specific execution paths. MedShard preserves rule-relevant fields, resolves eligible cases with deterministic prechecks, and combines focused model judgments with a deployed regex system through development-calibrated per-rule routing. On 250 Chinese admission notes evaluated against 19 audit rules, MedShard with a fixed 27B model achieves micro-F1 of 0.751 (95% CI [0.726, 0.776]), compared with 0.355 for direct prompting and 0.621 for the regex system. The complete LLM tier reaches 0.702, and calibrated routing improves on blanket union by 0.016 F1. Across 4B, 9B, 27B, and 122B-A10B configurations, MedShard improves F1 by 0.379–0.532 over each checkpoint's direct-prompt baseline. The routed 4B system exceeds direct 122B-A10B, while dense 27B achieves the highest routed score. On two mutually exclusive error-type benchmarks, decomposition increases detection precision while reducing type accuracy. These results connect rule-level inference design, evidence-source complementarity, and output structure to the performance of locally deployed clinical-document quality control.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
clinical document quality control, large language models, inference harness, medical record auditing

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_ndSJOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MedShard, a task-structured inference harness for clinical-document quality control. It combines field normalization, evidence-preserving field views, deterministic prechecks, and focused per-rule judgments. For each rule, it uses development data to choose the LLM tier, regex tier, or their union. On 250 Chinese admission notes evaluated against 19 QC rules, MedShard-27B raises micro-F1 from 0.355 with direct prompting to 0.751. The routed 4B system beats the direct-prompt 122B-A10B baseline.

Critical strengths
  1. The method fits the task well. Each rule can be checked on its own, and the findings can be combined by union. MedShard uses this structure to link each rule to the fields, checks, and judgment it needs. The source of each finding is also recorded, which helps human review.
  2. All four models improve over their own direct-prompt baselines. Paired record-level bootstrap comparisons and per-rule results help show where the two tiers help each other and where routing fails.
  3. The paper compares the complete LLM tier, blanket union, and calibrated routing. It also reports poor or mixed results on the scope probes. These tests help explain when the method may be useful.
Critical weaknesses
  1. The per-rule routes and three targeted template corrections are chosen using only 50 records. These choices may depend too much on this small set. The paper does not show how stable they are across different development sets.
  2. The main experiment uses one Chinese corpus of 300 records. Format normality and information completeness account for 701 of the 765 test findings (about 92%). The reported micro-F1 therefore may mainly reflects these common rule families. Testing on more benchmarks, such as MedErrBench [1] and MedRECT [2] will help make the paper more convincing.
  3. Eight rules use precheck-only paths. The LLM tier also includes field normalization and deterministic prechecks. It's unclear how much these steps help compared with rule-level LLM decomposition. Table 3 reports different development branches rather than a controlled test of these parts.

[1] Ma C, Zhang Y, Al-Jazzazi Y, et al. MedErrBench: A Fine-Grained Multilingual Benchmark for Medical Error Detection and Correction with Clinical Expert Annotations. Findings of the Association for Computational Linguistics: ACL 2026.

[2] Iwase N, Okuyama H, Iwasawa J. MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts. https://arxiv.org/abs/2511.00421v2

Questions for authors

Section 4.5 requires labeled data to calibrate the system at a new site. What should users do when local labels are unavailable in new hospital?

The paper can disable the prechecks while keeping the same field views and focused judge prompts. In this test, the focused judge should also check the eight rules that currently use precheck-only paths.

Reviewer_JiNyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce MedShard a regex and deterministic tool harness for using locally deployed large language models for structured clinical document audits. While for many everyday AI tasks a common solution is to switch to more powerful models provided via an API, this is not always possible in the healthcare setting creating a need for effective harnesses (or fine-tuning) of models to achieve high performance on specific tasks on local hardware. The evaluate their harness on admission notes and error-type benchmarks demonstrating that in some cases the harness improves upon the base models of various sizes.

Critical strengths

Developing harnesses for local model deployment of models in healthcare is an important and timely topic.

Critical weaknesses

Components of the system are not novel and have been used in similar form in many other systems across domains.

The authors only evaluate their main system on small benchmarks and models from a single model family. Please run Llama, Mistral etc.

No expert analysis/verification what the model is actually retrieving.

Very weak baselines, would expect at least a RAG system and other related works for LLMs to access information such as GraphRAG, Cache-Augmented Generation (CAG), ReAct etc.

Questions for authors

Please discuss fine-tuning in the introduction currently only focused on model size

I find it a bit confusing that the 122B model barely does better than direct prompting. These models would be able to write suitable regex code themselves for tool usage probably. What tools did they have access to?

"Eight rules are decided this way in the final 27B policy and issue no model call." is this not just over fitting on the specific dataset by avoiding the LLM and just hardcoding the solution?

"Three rules—a present-illness severity requirement, the scope of a laboratory-format rule, and a diagnosis-completeness condition—kept failing under the shared template, so each received a rewritten rule-specific template. " this also feels like a post-hoc engineering to overfit the specific benchmark rather than a reusable framework

Can the authors explain why F1 score is the best metric to use for the evaluation. Also please add overall accuracy and by subtasks to the analysis.

Can the authors please elaborate what the ML novelty is as to me it seems like their system is a straightforward application of a regex and a manually engineered system for a specific task.

Can the authors please explain how 4.4 supports the effectiveness of their method, seems like a mixed bags of results against a naive baseline to me.

If the main contribution is an effective harness I would like to see a Pareto efficiency plot of model size against F1/accuracy etc. With the Qwen model with harness dominating all other models.

Can the authors please include an English admission note benchmark as well since reproducibility is much harder this way for non Chinese speakers (I do like the inclusion of a non-English benchmark since this is generally underexplored in the field.

Reviewer_QtzCOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper is a methodology paper, where the main contribution is an inference harness for clinical document routing. The main problem it is solving is that clinical documents require quality checks, and existing systems (regex-based) occasionally are inflexible. Naturally, LLM could be used to enhance these quality assurance steps. As such, the authors introduce a harness / workflow which normalizes clinical notes, breaks it down into smaller chunks, decomposing it into rule-specific sections, applies prechecks, and decide which one to route to a LLM, regex, or union of the two systems.

Experiments-wise, the authors reported a significantly higher finding-level F1 over direct prompting a model to sanitize the document. The paper also includes several ablations and routing comparisons etc.

Critical strengths
  1. The paper addresses a practical problem. Clinical document audit is an annoying task with a multitude of possible failure modes e.g. formatting, completeness, and cross-field inconsistencies etc., which may be difficult to handle reliably in one model call. As such, the approach of breaking it down into smaller, rules-based chunks is sound. It also considers factors such as local inference and integration into existing pipeline (base regex system)
  2. The experiments include several model scales and candidly report negative results
Critical weaknesses
  1. Limited novelty and unclear contribution. I could be overlooking this paper, but it seems that the main approach that brought most of the gains were in 1. breaking down the note into chunks, 2. per-rule prompting, 3. rule-specific routing between LLM and reged-based system. These are mostly standard approaches in harness engineering, and the paper shows that this combination works on a specific task, but it is less clear what new general principle or method is being established (or even that this method is generalised across other datasets, see next weakness).

  2. The main evidence is not super convincing. In particular, the test set has 250 notes, and most findings concern format and completeness rules. As such, the headline micro-F1 may mostly reflect gains on the structured checks rather than broader clinical-document auditing. While the comparison over direct prompting seems to give strong numbers, the claim it supports might just be that "a single, large LLM call is less accurate than smaller, structured chunks for this specific context of medical document audit"

  3. The claims about routing and task scope are also possibly overstated. Routing improves only moderately improves over taking the union of both systems. Also, it is calibrated from 50 records, so its stability is unclear. The benchmarks with other dataset further show that decomposition can hurt mutually exclusive classification , and the authors did not isolate output composability from other differences between the tasks.

Questions for authors
  1. How strong is the direct-prompt baseline? It seems that the direct prompt version receives rule summaries, while the per-rule calls receive detailed criteria and examples. Can you include examples? Can you investigate whetehr the gain remains if the direct baseline gets comparable instructions and its outputs are also deduplicated?

  2. How stable is the routing with only 50 development notes? Does the improvement over blanket union hold when routes are recalibrated on resampled development sets, especially for rules with few or no positive examples?

  3. Why are the four zero-F1 rules kept on precheck-only paths? What justified ruling out an LLM call for these rules, particularly when some had no positive development examples?

Reviewer_PNThOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MedShard, an inference harness for quality control of clinical notes against a set of explicit rules. Instead of checking all rules in one prompt, it handles each rule separately. Specifically, a deterministic precheck first resolves the cases it can, and the remaining cases go to an LLM judge that sees only the fields relevant to that rule. A per-rule route, selected on a small development set, then decides whether each rule uses the output of this LLM tier, the hospital's existing regex system, or their union. The authors evaluate the method on a Chinese admission-note dataset with 19 rules and report substantial gains over direct prompting across four model sizes. They also use two error-type benchmarks with mutually exclusive labels to argue that the approach works best when rule outputs can be combined by union.

Critical strengths
  1. The paper addresses a realistic setting: clinical notes often cannot leave the hospital, and many sites already run rule-based checkers. MedShard works within these constraints and improves consistently over direct prompting across four model sizes. This makes the work relevant to hospitals facing the same constraints.
  2. The reporting is transparent. The authors report per-rule results, failure cases, a negative scope result, and the extra inference cost. This makes the claims easy to check, and most of my concerns below can likely be addressed with outputs the authors already have.
Critical weaknesses
  1. It is unclear how much of the gain comes from the language model. Several rules are decided entirely by hand-written prechecks, and most findings in the dataset are format or completeness checks that deterministic code may handle well. The paper does not separate the contribution of prechecks from that of model calls. This matters because the main claim is about inference structure around a model, and without this breakdown the claim is hard to evaluate.
  2. The evidence base is narrow. All main results come from one dataset, and the per-rule routes are selected on a very small development set where some rules have no positive examples. The paper does not show how stable the routes and final results are across different splits. This matters because it limits how much the results can be expected to hold in other settings.
Questions for authors
  1. Of the final system's correct findings, how many come from prechecks, model calls, and the regex tier? A breakdown by rule family would be especially helpful.
  2. Direct 9B scores lower than direct 4B, which may point to output parsing problems. What are the parse-failure rates of the direct baselines, and how do the scores of the direct baselines and other comparators change after removing duplicate predictions, as MedShard does for its own output?
  3. How stable are the route choices and final results across different development/evaluation splits?
  4. Table 3 appears to report evaluation-set scores for intermediate configurations. Were evaluation results consulted at any point during development, including for the three targeted template corrections?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.