- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3(+1)
Read review
Paper summary
The authors present a novel approach called ReCIPE that learns "rules" (or recipes, as they call them) needed to prepare the data (from notes/ EHR to a structured database for downstream tasks), and makes use of these rules when relevant, when similar cases are encountered in the data in a subsequent step. The authors claim that this approach is more cost-effective than baselines while being generalizable to new patients. The authors tested this approach on MIMIC-IV-Graph (new benchmark) and an expert-validated unnamed oncology dataset.
Critical strengths
-
The approach is interesting, and learning rules to automate and streamline data processing.
-
It could deliver significant cost savings, and its ability to trace evidence makes it well-suited for healthcare.
-
I also acknowledge the authors' effort to include expert validation, which I know can be time-consuming and yet critical.
Critical weaknesses
- Generalizability and note variability: Different notes can have different styles, but the author does not address this potential generalizability limitation. Also, the proposed benchmark (MIMIC-IV-Graph) doesn't appear to be expert-validated, raising questions about its reliability.
- Assumptions and limitations are not clear: What are the assumptions and limitations of the work?
- The verifier is the same LLM used for extraction, which can create a shared blind spot.
- Baselines: The authors mention multiple related works in their "Clinical Information Extraction" section, such as CLEAR and HARMON-E, but do not compare their results against these baselines (which are also more recent than EHRAgent).
- No ablation study: For example, the authors claim that "[...] identifying suitable sources for each target can involve many unproductive extraction attempts", but they did not demonstrate it.
- There is not enough information/ detail about the Direct LLM and Hybrid-RAG, which makes it impossible to judge the credibility of these baselines.
- I am concerned about how the approach might create an unreliable preprocessing pipeline. The authors do not explain how an expert could be integrated into this approach to validate/verify the data (for example, for the "CONFLICTs" or if the approach misses data, or wrongly extracts without proper context (e.g., in the case of epistemic erosion)
- Some design choices are not justified, such as the equally weighted harmonic mean.
- The authors didn't share any code for reproducibility or to investigate the MIMIC-IV-Graph construction.
Questions for authors
- What are the assumptions and limitations of the work? Are there any assumptions related to the data (notes and EHR)?
- For the "CONFLICT" data, and for the whole approach in general, how does the author suggest integrating an expert into the loop?
- What prompts were used? For Hybrid-RAG, how was the text corpus curated and chosen exactly?
- How does the approach fare for notes that are not ICU admissions?
- Is there a metric or ratio between tokens used and performance that could help characterize the trade-off (or gain) between the different approaches? Notably
- Why use an equally weighted harmonic mean for the score?
- Can you evaluate ReCIPE and the strongest baseline with MIMIC-IV but with expert annotations?
- Can you break down the results per field (instead of summarizing by dates or numbers, among others)?
- Can you also show the details of the mapping/recipes?
3 comments
Response to Reviewer wx6n (Part 2/3)
Q3. What prompts were used? For Hybrid-RAG, how was the text corpus curated and chosen exactly?
Response. ReCIPE uses separate prompts for route generation, extraction, and verification; the baselines share extraction instructions and add prompts for their respective stages. Hybrid-RAG automatically builds its corpus from all nonempty records in the current admission's 18 permitted source fields, including notes and structured values. No external documents, cross-patient records, or target reference answers are included. The prompt texts and complete corpus construction and retrieval settings are provided in Appendix H (Prompts and Settings) of the revised manuscript.
Q4. How does the approach fare for notes that are not ICU admissions?
Response. We designed MIMIC-IV-Graph to cover diverse clinical fields and extraction tasks within the same admission. Requiring linked ED and ICU stays, together with nonempty discharge summaries and radiology reports, brings complementary structured and textual sources into a common evaluation setting. Including ICU stays expands the benchmark’s temporal and categorical targets to cover ICU admission and discharge times and first and last care units. This selection supports broader field coverage.
An ICU stay is a benchmark inclusion criterion, not a requirement of ReCIPE's execution model. The executor takes declared fields and available records; ICU-specific routes require their own inputs, while unrelated routes can execute without them. For a non-ICU setting, the task should specify which fields apply and retain missing or inapplicable statuses for unsupported fields.
Our hospital experiment provides evidence in a setting without ICU-based selection. It uses oncology records without restrictions on ICU admission. Five physicians conducted a blinded review of 400 sampled disagreements between ReCIPE and the hospital’s manual/regex Reference. Their aggregated judgments identified 396 cases (99%) as correct only for ReCIPE and four as correct for both methods.
Q5. Is there a metric or ratio between tokens used and performance that could help characterize the trade-off (or gain) between the different approaches? Notably
Response. We add a two-dimensional comparison of mean tokens per test admission and Overall extraction score for all nine methods. The horizontal axis is logarithmic. The upper-left corner indicates higher scores and lower token consumption.
Overall score versus mean tokens per test admission.
As a supplementary descriptive ratio, let be Overall Score and the mean token count per admission. Define
which reports score per 1,000 tokens. The results are shown below:
| Method | Tokens/admission (k) | Overall | : score per 1,000 tokens |
|---|---|---|---|
| Direct LLM | 917.26 | 0.213 | 0.00023 |
| Hybrid-RAG | 349.34 | 0.393 | 0.00112 |
| HippoRAG 2 | 420.01 | 0.395 | 0.00094 |
| CAESURA | 99.37 | 0.845 | 0.00850 |
| EVAPORATE-CODE+ | 26.82 | 0.526 | 0.01961 |
| OmniTQA | 58.05 | 0.821 | 0.01414 |
| DocETL | 68.11 | 0.832 | 0.01222 |
| EHRAgent | 131.77 | 0.836 | 0.00634 |
| ReCIPE | 2.84 | 0.967 | 0.34049 |
ReCIPE improves Overall from 0.845 to 0.967 relative to CAESURA while reducing mean tokens from 99.37k to 2.84k, a 97.1% reduction and approximately 40.0× higher . Relative to EVAPORATE-CODE+, the baseline with the lowest token consumption, ReCIPE uses 89.4% fewer tokens and improves Overall from 0.526 to 0.967. ReCIPE occupies the best score/token position among these operating points.
Q6. Why use an equally weighted harmonic mean for the score?
Response. The harmonic mean is more sensitive to low type scores than the arithmetic mean, limiting how much strong performance in one type can offset weaknesses in another. Equal weights balance Enum, Numeric, Datetime, and Text; weighting by field counts would instead give Text 60.5% (26/43) of the weight and Datetime only 4.7% (2/43).
We also checked whether this choice affects the main comparison. ReCIPE ranks first under all four aggregations, with CAESURA the strongest baseline in each case:
| Aggregation | ReCIPE | CAESURA |
|---|---|---|
| Equal type weights, harmonic mean | 0.967 | 0.845 |
| Equal type weights, arithmetic mean | 0.969 | 0.855 |
| Type harmonic mean weighted by field counts | 0.967 | 0.909 |
| Equal weight per field, arithmetic mean | 0.969 | 0.917 |
Response to Reviewer wx6n (Part 3/3)
Q7. Can you evaluate ReCIPE and the strongest baseline with MIMIC-IV but with expert annotations?
Response. We agree that expert annotation would provide an independent check of the MIMIC-IV-Graph reference values. We added a blinded physician comparison of ReCIPE and CAESURA, the strongest baseline in Table 1. Five physicians each reviewed 400 admission–field output disagreements from the MIMIC-IV-Graph test set, with 100 per field type, producing 2,000 judgments. They assessed the field definitions, randomly assigned A/B outputs, and original source records. Method identities and benchmark reference answers were hidden.
The sample covers 32 target fields across 300 admissions and 262 patients. Both methods return nonempty outputs on 340 of the 400 pairs. This is a diagnostic sample: stratified sampling seeds were selected to achieve 99.000% exact agreement between ReCIPE and the automatic references (396/400).
The table reports the final submitted physician judgments, with 500 judgments per type. A judgment of both correct counts toward both methods' correctness percentages.
| Field type | Reviewed pairs | ReCIPE judged correct (%) | CAESURA judged correct (%) | Inter-rater agreement (%) |
|---|---|---|---|---|
| Enum | 100 | 99.200 | 0.800 | 98.400 |
| Numeric | 100 | 90.200 | 11.800 | 76.900 |
| Datetime | 100 | 99.600 | 0.800 | 98.400 |
| Text | 100 | 93.400 | 18.400 | 63.500 |
| Overall | 400 | 95.600 | 7.950 | 84.300 |
Across the 2,000 judgments, 1,841 favor only ReCIPE, 88 favor only CAESURA, 71 judge both correct, and none judge neither correct. Following the paper's plurality rule, all 400 pairs are judged correct only for ReCIPE: 246 receive unanimous votes, 149 receive 4–1 votes, three receive 3–2 votes, and two receive 3–1–1 votes. Inter-rater agreement follows the paper's definition: the mean fraction of the ten physician pairs giving the same four-way judgment for each reviewed entry.
Q8. Can you break down the results per field (instead of summarizing by dates or numbers, among others)?
Response. Yes. We provide all 43 field scores for ReCIPE and all eight baselines, using the same predictions and scoring rules as the type-level results. The complete tables are in Appendix I (Full Per-field Evaluation Results) of the revised manuscript.
Q9. Can you also show the details of the mapping/recipes?
Response. A mapping identifies which input fields are used to recover a target field. The recipe describes how to turn those inputs into the target value. It specifies, for example, which dictionary to consult, which text pattern or prompt to use, and how to match related records. The table below shows examples from the final graph.
| Target | Inputs | Recipe |
|---|---|---|
| Diagnosis description | ICD code and version | Align the code and version from the same source record, then query the ICD dictionary using both values. |
| Medication dose | Medication instructions extracted from a note section | Extract the numeric dose and retain the instruction identifier, keeping it aligned with the drug name and unit from that instruction. |
| Discharge location | Discharge-disposition section | Match explicit category names; otherwise apply an LLM prompt restricted to documented destinations. For example, “home” alone does not establish home health care. |
| Documented dose difference | Admission/discharge drug names, doses, and units | Keep doses in mg, normalize drug names for case and whitespace, and require a unique match within the same note. Compute discharge minus admission dose; skip ambiguous matches. |
A recipe can take another recipe's output as its input. For medication-dose extraction, this can be written as
\to \text{Medication section} \to \text{Medication instructions} \to \text{Doses}.$$ Each arrow represents a recipe: one locates the medication section, the next separates individual instructions, and the last extracts their doses. Each dose remains linked to its original instruction. Several operations can also be grouped within one recipe: the dose-difference recipe combines unit filtering, medication matching, and subtraction. We also retain the source evidence and the recipe used for each result. For a medication dose, this lets a reviewer return to the original instruction and check the drug name, dose, and unit together.
Response to Reviewer wx6n (Part 1/3)
We thank the reviewer for their interest in learning reusable rules to streamline clinical data processing, and for highlighting the potential cost savings and value of evidence traceability in healthcare. We also appreciate their acknowledgment of our efforts to include expert validation.
We respond to each concern and question below.
Response. We make three assumptions about the input data and extraction task:
Limitations. ReCIPE relies on the available records and cannot reliably recover information for which they contain no supporting evidence. Our MIMIC evaluation uses a cohort with linked ED and ICU stays, with reference values derived from structured tables and extraction rules. The hospital review is limited to sampled disagreements. Further evaluation is needed to assess generalizability across other hospitals, note types, and patient populations.
Response.
For
CONFLICTcases, experts could review competing outputs against the original evidence. This status occurs when candidate outputs for the same target remain inconsistent after normalization, either in their values or in the occurrences they cover. It may reflect contradictory source records, extraction errors, or differences in temporal scope or record alignment. ReCIPE leaves the target unresolved and retains the candidate evidence. Experts would see the target definition and competing values alongside the original source context, including relevant dates and units. They could confirm a supported candidate, correct an extraction, add missing occurrences, or leave the result unresolved if the evidence remains contradictory.For the overall workflow, expert involvement could also address errors that do not produce a
CONFLICT. We suggest three points of integration: