acceptodds
Under review as a conference paper at ICLR 2027

ReCIPE: Collaborative Agents Learning Reusable Procedures for Clinical Information Processing and Extraction

Abstract

Preparing structured clinical data for healthcare agents requires extracting predefined fields from heterogeneous notes and tables. This process is hindered by conflicting evidence, costly source discovery, and limited reuse of extraction experience across patients. We introduce ReCIPE (usable linical nformation rocessing and xtraction), a multi-agent system that learns reusable extraction procedures represented in an evolving typed dependency graph, RouteGraph. Field-semantic retrieval guides a compiler agent in proposing executable routes, while a verifier agent checks their trial outputs against patient evidence. GraphIE, our extraction algorithm, executes these routes with evidence propagation and consistency checks, supporting jointly required inputs and alternative routes. Accepted procedures accumulate across training patients without accessing target reference values or retaining patient-specific answers. We also introduce MIMIC-IV-Graph, a new benchmark pairing patient-specific source values with explicit source and target field definitions, covering 43 targets across four types. Starting from learned graph snapshots with admission-local route discovery, ReCIPE outperforms eight baselines in overall extraction quality on unseen patients under a shared language model backbone. Compared with the direct usage of LLM, it improves the overall score from 0.213 to 0.967. Per admission, it reduces test-time token consumption by 99.7% and LLM calls by 96.9%. In a blinded review by five physicians, 396 of 400 reviewed output differences with an oncology workflow combining manual abstraction and simple regular expressions are judged correct only for ReCIPE. These results support evolving, evidence-grounded procedures for accurate and efficient clinical data preparation.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Clinical Information Extraction, Multi-Agent Systems, Typed Dependency Graphs, Procedural Memory, Workflow Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wx6nOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
3(+1)
Read review
Paper summary

The authors present a novel approach called ReCIPE that learns "rules" (or recipes, as they call them) needed to prepare the data (from notes/ EHR to a structured database for downstream tasks), and makes use of these rules when relevant, when similar cases are encountered in the data in a subsequent step. The authors claim that this approach is more cost-effective than baselines while being generalizable to new patients. The authors tested this approach on MIMIC-IV-Graph (new benchmark) and an expert-validated unnamed oncology dataset.

Critical strengths
  1. The approach is interesting, and learning rules to automate and streamline data processing.

  2. It could deliver significant cost savings, and its ability to trace evidence makes it well-suited for healthcare.

  3. I also acknowledge the authors' effort to include expert validation, which I know can be time-consuming and yet critical.

Critical weaknesses
  1. Generalizability and note variability: Different notes can have different styles, but the author does not address this potential generalizability limitation. Also, the proposed benchmark (MIMIC-IV-Graph) doesn't appear to be expert-validated, raising questions about its reliability.
  2. Assumptions and limitations are not clear: What are the assumptions and limitations of the work?
  3. The verifier is the same LLM used for extraction, which can create a shared blind spot.
  4. Baselines: The authors mention multiple related works in their "Clinical Information Extraction" section, such as CLEAR and HARMON-E, but do not compare their results against these baselines (which are also more recent than EHRAgent).
  5. No ablation study: For example, the authors claim that "[...] identifying suitable sources for each target can involve many unproductive extraction attempts", but they did not demonstrate it.
  6. There is not enough information/ detail about the Direct LLM and Hybrid-RAG, which makes it impossible to judge the credibility of these baselines.
  7. I am concerned about how the approach might create an unreliable preprocessing pipeline. The authors do not explain how an expert could be integrated into this approach to validate/verify the data (for example, for the "CONFLICTs" or if the approach misses data, or wrongly extracts without proper context (e.g., in the case of epistemic erosion)
  8. Some design choices are not justified, such as the equally weighted harmonic mean.
  9. The authors didn't share any code for reproducibility or to investigate the MIMIC-IV-Graph construction.
Questions for authors
  1. What are the assumptions and limitations of the work? Are there any assumptions related to the data (notes and EHR)?
  2. For the "CONFLICT" data, and for the whole approach in general, how does the author suggest integrating an expert into the loop?
  3. What prompts were used? For Hybrid-RAG, how was the text corpus curated and chosen exactly?
  4. How does the approach fare for notes that are not ICU admissions?
  5. Is there a metric or ratio between tokens used and performance that could help characterize the trade-off (or gain) between the different approaches? Notably
  6. Why use an equally weighted harmonic mean for the score?
  7. Can you evaluate ReCIPE and the strongest baseline with MIMIC-IV but with expert annotations?
  8. Can you break down the results per field (instead of summarizing by dates or numbers, among others)?
  9. Can you also show the details of the mapping/recipes?
3 comments
AuthorsOpenReview ↗

Response to Reviewer wx6n (Part 1/3)

We thank the reviewer for their interest in learning reusable rules to streamline clinical data processing, and for highlighting the potential cost savings and value of evidence traceability in healthcare. We also appreciate their acknowledgment of our efforts to include expert validation.

We respond to each concern and question below.

Q1. What are the assumptions and limitations of the work? Are there any assumptions related to the data (notes and EHR)?

Response. We make three assumptions about the input data and extraction task:

  1. Target fields are clearly defined. Each field has a definition specifying its meaning and data type, as well as units, allowed categories, and normalization rules where applicable. The definition also specifies how to distinguish repeated occurrences and how to distinguish missing information from an explicit negative finding.
  2. Records are correctly grouped by patient and admission. We assume that the provided notes and structured EHR records are organized by patient and, within each patient, by admission, with no mixing of records across patients or admissions. The original text and record identifiers are preserved so that extracted values can be traced back to their sources.
  3. Recipe reuse depends on compatible field definitions and documentation. Reusing a recipe assumes that the field definitions and documentation patterns it was built for remain applicable. Changes in note templates, abbreviations, language, or coding systems may require adapting the recipe and revalidating its outputs.

Limitations. ReCIPE relies on the available records and cannot reliably recover information for which they contain no supporting evidence. Our MIMIC evaluation uses a cohort with linked ED and ICU stays, with reference values derived from structured tables and extraction rules. The hospital review is limited to sampled disagreements. Further evaluation is needed to assess generalizability across other hospitals, note types, and patient populations.

Q2. For the "CONFLICT" data, and for the whole approach in general, how does the author suggest integrating an expert into the loop?

Response.

For CONFLICT cases, experts could review competing outputs against the original evidence. This status occurs when candidate outputs for the same target remain inconsistent after normalization, either in their values or in the occurrences they cover. It may reflect contradictory source records, extraction errors, or differences in temporal scope or record alignment. ReCIPE leaves the target unresolved and retains the candidate evidence. Experts would see the target definition and competing values alongside the original source context, including relevant dates and units. They could confirm a supported candidate, correct an extraction, add missing occurrences, or leave the result unresolved if the evidence remains contradictory.

For the overall workflow, expert involvement could also address errors that do not produce a CONFLICT. We suggest three points of integration:

  1. Before deployment: A clinical expert could review field definitions, source mappings, and representative recipe outputs. This review would establish what evidence supports each target and how temporal scope, units, negation, and uncertainty should be handled. For example, a tentative finding should retain its uncertainty in the extracted output. Experts could revise ambiguous definitions or recipes before they are applied to further records.
  2. During routine review: Experts could review missing or insufficient-evidence outputs and a sample of apparently successful results against the source records. This would help identify omitted information, loss of context, or errors shared by multiple routes, even when their outputs agree. Corrections would retain the supporting evidence so that reviewers can check why a value was changed.
  3. When updating recipes: Expert corrections could be retained with their supporting evidence to identify recurring problems, such as selecting the wrong note section or losing a contextual qualifier. These findings could guide recipe revisions. Before reuse, the revised recipes would be checked on separate development cases, including previously correct examples, to assess whether they address the problem without introducing new errors.
AuthorsOpenReview ↗

Response to Reviewer wx6n (Part 2/3)

Q3. What prompts were used? For Hybrid-RAG, how was the text corpus curated and chosen exactly?

Response. ReCIPE uses separate prompts for route generation, extraction, and verification; the baselines share extraction instructions and add prompts for their respective stages. Hybrid-RAG automatically builds its corpus from all nonempty records in the current admission's 18 permitted source fields, including notes and structured values. No external documents, cross-patient records, or target reference answers are included. The prompt texts and complete corpus construction and retrieval settings are provided in Appendix H (Prompts and Settings) of the revised manuscript.

Q4. How does the approach fare for notes that are not ICU admissions?

Response. We designed MIMIC-IV-Graph to cover diverse clinical fields and extraction tasks within the same admission. Requiring linked ED and ICU stays, together with nonempty discharge summaries and radiology reports, brings complementary structured and textual sources into a common evaluation setting. Including ICU stays expands the benchmark’s temporal and categorical targets to cover ICU admission and discharge times and first and last care units. This selection supports broader field coverage.

An ICU stay is a benchmark inclusion criterion, not a requirement of ReCIPE's execution model. The executor takes declared fields and available records; ICU-specific routes require their own inputs, while unrelated routes can execute without them. For a non-ICU setting, the task should specify which fields apply and retain missing or inapplicable statuses for unsupported fields.

Our hospital experiment provides evidence in a setting without ICU-based selection. It uses oncology records without restrictions on ICU admission. Five physicians conducted a blinded review of 400 sampled disagreements between ReCIPE and the hospital’s manual/regex Reference. Their aggregated judgments identified 396 cases (99%) as correct only for ReCIPE and four as correct for both methods.

Q5. Is there a metric or ratio between tokens used and performance that could help characterize the trade-off (or gain) between the different approaches? Notably

Response. We add a two-dimensional comparison of mean tokens per test admission and Overall extraction score for all nine methods. The horizontal axis is logarithmic. The upper-left corner indicates higher scores and lower token consumption.

Overall score versus mean tokens per test admission.

As a supplementary descriptive ratio, let SmS_m be Overall Score and TmT_m the mean token count per admission. Define

Em=1000SmTm,E_m=\frac{1000S_m}{T_m},

which reports score per 1,000 tokens. The results are shown below:

MethodTokens/admission (k)OverallEmE_m: score per 1,000 tokens
Direct LLM917.260.2130.00023
Hybrid-RAG349.340.3930.00112
HippoRAG 2420.010.3950.00094
CAESURA99.370.8450.00850
EVAPORATE-CODE+26.820.5260.01961
OmniTQA58.050.8210.01414
DocETL68.110.8320.01222
EHRAgent131.770.8360.00634
ReCIPE2.840.9670.34049

ReCIPE improves Overall from 0.845 to 0.967 relative to CAESURA while reducing mean tokens from 99.37k to 2.84k, a 97.1% reduction and approximately 40.0× higher EmE_m. Relative to EVAPORATE-CODE+, the baseline with the lowest token consumption, ReCIPE uses 89.4% fewer tokens and improves Overall from 0.526 to 0.967. ReCIPE occupies the best score/token position among these operating points.

Q6. Why use an equally weighted harmonic mean for the score?

Response. The harmonic mean is more sensitive to low type scores than the arithmetic mean, limiting how much strong performance in one type can offset weaknesses in another. Equal weights balance Enum, Numeric, Datetime, and Text; weighting by field counts would instead give Text 60.5% (26/43) of the weight and Datetime only 4.7% (2/43).

We also checked whether this choice affects the main comparison. ReCIPE ranks first under all four aggregations, with CAESURA the strongest baseline in each case:

AggregationReCIPECAESURA
Equal type weights, harmonic mean0.9670.845
Equal type weights, arithmetic mean0.9690.855
Type harmonic mean weighted by field counts0.9670.909
Equal weight per field, arithmetic mean0.9690.917
AuthorsOpenReview ↗

Response to Reviewer wx6n (Part 3/3)

Q7. Can you evaluate ReCIPE and the strongest baseline with MIMIC-IV but with expert annotations?

Response. We agree that expert annotation would provide an independent check of the MIMIC-IV-Graph reference values. We added a blinded physician comparison of ReCIPE and CAESURA, the strongest baseline in Table 1. Five physicians each reviewed 400 admission–field output disagreements from the MIMIC-IV-Graph test set, with 100 per field type, producing 2,000 judgments. They assessed the field definitions, randomly assigned A/B outputs, and original source records. Method identities and benchmark reference answers were hidden.

The sample covers 32 target fields across 300 admissions and 262 patients. Both methods return nonempty outputs on 340 of the 400 pairs. This is a diagnostic sample: stratified sampling seeds were selected to achieve 99.000% exact agreement between ReCIPE and the automatic references (396/400).

The table reports the final submitted physician judgments, with 500 judgments per type. A judgment of both correct counts toward both methods' correctness percentages.

Field typeReviewed pairsReCIPE judged correct (%)CAESURA judged correct (%)Inter-rater agreement (%)
Enum10099.2000.80098.400
Numeric10090.20011.80076.900
Datetime10099.6000.80098.400
Text10093.40018.40063.500
Overall40095.6007.95084.300

Across the 2,000 judgments, 1,841 favor only ReCIPE, 88 favor only CAESURA, 71 judge both correct, and none judge neither correct. Following the paper's plurality rule, all 400 pairs are judged correct only for ReCIPE: 246 receive unanimous votes, 149 receive 4–1 votes, three receive 3–2 votes, and two receive 3–1–1 votes. Inter-rater agreement follows the paper's definition: the mean fraction of the ten physician pairs giving the same four-way judgment for each reviewed entry.

Q8. Can you break down the results per field (instead of summarizing by dates or numbers, among others)?

Response. Yes. We provide all 43 field scores for ReCIPE and all eight baselines, using the same predictions and scoring rules as the type-level results. The complete tables are in Appendix I (Full Per-field Evaluation Results) of the revised manuscript.

Q9. Can you also show the details of the mapping/recipes?

Response. A mapping identifies which input fields are used to recover a target field. The recipe describes how to turn those inputs into the target value. It specifies, for example, which dictionary to consult, which text pattern or prompt to use, and how to match related records. The table below shows examples from the final graph.

TargetInputsRecipe
Diagnosis descriptionICD code and versionAlign the code and version from the same source record, then query the ICD dictionary using both values.
Medication doseMedication instructions extracted from a note sectionExtract the numeric dose and retain the instruction identifier, keeping it aligned with the drug name and unit from that instruction.
Discharge locationDischarge-disposition sectionMatch explicit category names; otherwise apply an LLM prompt restricted to documented destinations. For example, “home” alone does not establish home health care.
Documented dose differenceAdmission/discharge drug names, doses, and unitsKeep doses in mg, normalize drug names for case and whitespace, and require a unique match within the same note. Compute discharge minus admission dose; skip ambiguous matches.

A recipe can take another recipe's output as its input. For medication-dose extraction, this can be written as

\to \text{Medication section} \to \text{Medication instructions} \to \text{Doses}.$$ Each arrow represents a recipe: one locates the medication section, the next separates individual instructions, and the last extracts their doses. Each dose remains linked to its original instruction. Several operations can also be grouped within one recipe: the dose-difference recipe combines unit filtering, medication matching, and subtraction. We also retain the source evidence and the recipe used for each result. For a medication dose, this lets a reviewer return to the original instruction and check the drug name, dose, and unit together.
Reviewer_V2LUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies extracting predefined fields from clinical notes and tables to build structured clinical data. The authors propose ReCIPE, a multi-agent system that stores reusable extraction procedures in a typed dependency graph (RouteGraph). A compiler agent proposes routes, a verifier agent checks them against patient evidence, and GraphIE executes and extends the graph. The authors also build a benchmark called MIMIC-IV-Graph from MIMIC-IV, and show better extraction quality and lower LLM cost than baselines.

Critical strengths

S1. The method lowers dependence on the LLM. Procedures are accumulated in a typed dependency graph over patients, and at test time the system runs the stored graph and only searches for new routes when a field is left unresolved. LLM calls per admission are few (2.6), and the cost gap to the baselines is large. Given the cost, this seems better suited to real deployment than the baselines.

S2. The physician review of the outputs and the comparison with a physician-designed extraction programs help show that the method is useful beyond the benchmark numbers.

Critical weaknesses

W1. The core results are only on MIMIC-IV-Graph, which the authors build and list as their first contribution. Why a new benchmark is needed, and how it differs from existing ones, gets one sentence in the Introduction and nothing in Related Work. Since the benchmark is both a main contribution and the only evaluation setting, this needs a proper analysis.

W2. Full per-field results are missing. Table 1 gives type averages, and per-field numbers appear only for Gref vs. G80 (Table 11). I cannot tell whether the gain from graph-based extraction comes from fields that reduce to lookups and joins, or whether it also holds against the baselines on fields like in-hospital mortality and discharge location, where no explicit phrase gives the value. Per-field results for all eight baselines would help.

Questions for authors

Q1. The Abstract and Introduction report cost reductions (99.7% tokens, 96.9% calls) only against Direct LLM, the weakest baseline. The chosen comparison makes the efficiency gain look larger than it is.

2 comments
AuthorsOpenReview ↗

Response to Reviewer V2LU (Part 1/2)

We appreciate the reviewer's emphasis on how reusing procedures reduces LLM calls and cost, making ReCIPE promising for practical deployment. We also thank them for noting that physician review and comparison with physician-designed extraction programs help show its usefulness beyond benchmark scores.

We respond to each concern and question below.

W1. The core results are only on MIMIC-IV-Graph, which the authors build and list as their first contribution. Why a new benchmark is needed, and how it differs from existing ones, gets one sentence in the Introduction and nothing in Related Work. Since the benchmark is both a main contribution and the only evaluation setting, this needs a proper analysis.

Response. We agree that the benchmark's purpose and its relation to prior work should be clearer. Existing benchmarks address note-based QA [1], text-to-SQL [2, 3], medication QA [4], and consistency between notes and tables [5]. MIMIC-IV-Graph focuses on admission-level data preparation: recovering the same 43 target fields from 18 permitted source fields, with specified data types, normalization rules, and alignment of occurrences across related fields.

These related benchmarks do not directly provide the complete, aligned reference records needed for this task. Adapting them would require additional field definitions and reference annotations. MIMIC-IV-Graph provides these specifications together with a patient-disjoint protocol for measuring extraction quality and test-time cost when procedures learned from training patients are reused on unseen patients. The differences and our contributions are shown in the following table.

BenchmarkClinical notesStructured tablesMulti-step processingFixed record extraction¹Procedure reuse evaluation²
emrQA [1]✓✓
MIMICSQL [2]✓✓
EHRSQL [3]✓✓
DrugEHRQA [4]✓✓✓
EHRCon [5]✓✓✓
MIMIC-IV-Graph (ours)✓✓✓✓✓

¹ Fixed record extraction: Extracting the same predefined fields for every admission, with specified data types, normalization rules, and aligned occurrences across related fields. Returning multiple answers to a question does not by itself meet this definition.

² Procedure reuse evaluation: Measuring extraction quality and test-time cost when executable extraction procedures learned from training patients are applied to unseen patients. A standard train/test split alone does not meet this definition.

References

[1] Anusri Pampari et al. emrQA: A Large Corpus for Question Answering on Electronic Medical Records. EMNLP, 2018.

[2] Ping Wang, Tian Shi, and Chandan K. Reddy. Text-to-SQL Generation for Question Answering on Electronic Medical Records. The Web Conference (WWW), 2020.

[3] Gyubok Lee et al. EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records. NeurIPS, 2022.

[4] Jayetri Bardhan et al. DrugEHRQA: A Question Answering Dataset on Structured and Unstructured Electronic Health Records For Medicine Related Queries. LREC, 2022.

[5] Yeonsu Kwon et al. EHRCon: Dataset for Checking Consistency between Unstructured Notes and Structured Tables in Electronic Health Records. NeurIPS, 2024.

AuthorsOpenReview ↗

Response to Reviewer V2LU (Part 2/2)

W2. Full per-field results are missing. Table 1 gives type averages, and per-field numbers appear only for Gref vs. G80 (Table 11). I cannot tell whether the gain from graph-based extraction comes from fields that reduce to lookups and joins, or whether it also holds against the baselines on fields like in-hospital mortality and discharge location, where no explicit phrase gives the value. Per-field results for all eight baselines would help.

Response. To examine whether the gains are confined to tabular recovery, we progressively exclude targets involving table lookup, alignment, or normalization, whole-section extraction, and documented medication-dose differences. ReCIPE remains ahead of all eight baselines on each remaining subset:

Targets retained after cumulative exclusionsFieldsReCIPEStrongest baselineBaseline scoreDifference
Exclude 11 tabular recovery targets320.9683CAESURA0.9113+0.0571
Additionally exclude 9 whole-section targets230.9560CAESURA0.8765+0.0794
Additionally exclude the documented dose-difference target220.9540CAESURA0.9155+0.0385

The 11 tabular targets comprise five dictionary mappings, four ICU fields, creatinine normalization, and previous service. The nine section targets are all fields with the prefix t.section.. Exclusions are cumulative. Each subset score is the arithmetic mean of its original field scores.

ReCIPE also outperforms all baselines on in-hospital mortality:

Field (Macro-F1)nnReCIPEStrongest baselineBaseline scoreDifference
In-hospital mortality6200.8306DocETL0.5665+0.2641

The complete results for all 43 fields and all eight baselines are provided in Appendix I (Full Per-field Evaluation Results) of the revised manuscript.

Q1. The Abstract and Introduction report cost reductions (99.7% tokens, 96.9% calls) only against Direct LLM, the weakest baseline. The chosen comparison makes the efficiency gain look larger than it is.

We agree that highlighting efficiency gains only against Direct LLM gives an incomplete picture of the comparison with stronger baselines. We have revised the Abstract and Introduction to report gains relative to CAESURA, the strongest baseline in overall extraction quality. Under this comparison, ReCIPE still reduces average test-time token consumption per admission by 97.1% (99.37k to 2.84k) and LLM calls by 60.7% (6.626 to 2.603), while improving the overall extraction score from 0.845 to 0.967 (Table 1).

The corresponding changes to the Abstract and Introduction are highlighted in blue in the revised manuscript.

Reviewer_U9VSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReCIPE, a multi-agent system for extracting predefined fields from clinical notes and tables. Extraction procedures are stored as routes in a typed dependency graph, where each route applies an operator from a fixed library (e.g., regex or LLM extraction, dictionary lookup, arithmetic). A compiler agent proposes routes from field definitions, a verifier agent checks their outputs against the patient's records, and accepted routes are reused for later patients. The graph is executed by GraphIE, which tracks evidence and flags conflicts. The authors also build a benchmark from MIMIC-IV, with references derived from table values and fixed rules. They report that ReCIPE outperforms eight baselines with much lower test-time token cost, outperforms a physician-designed graph, and is favored by physicians over an existing oncology workflow on differing outputs.

Critical strengths
  1. Storing extraction procedures, rather than answers, as an executable graph is a practical idea. Outputs come with evidence, and test-time cost is much lower than running a language model on full records. Both matter for large-scale clinical data preparation.

  2. The evaluation includes more than a benchmark table. Learned routes are tested on unseen patients across training snapshots and compared with a physician-designed graph, and ReCIPE is also evaluated on real oncology records with a blinded physician review. This makes it possible to look at what the method learns, not only its final score.

Critical weaknesses
  1. Storing extraction procedures, rather than answers, as an executable graph is a practical idea. Outputs come with evidence, and test-time cost is much lower than running a language model on full records. Both matter for large-scale clinical data preparation.

  2. The evaluation includes more than a benchmark table. Learned routes are tested on unseen patients across training snapshots and compared with a physician-designed graph, and ReCIPE is also evaluated on real oncology records with a blinded physician review. This makes it possible to look at what the method learns, not only its final score.

Questions for authors
  1. In the final graph, how many routes use LLM or hybrid extraction (including inside composites), and how many are purely deterministic?
  2. The Overall score weights each field type equally, although Datetime has far fewer fields than Text. How do the main comparisons change under a field-weighted average?
  3. For the oncology review, how many outputs differed in total, how were the excerpts shown to physicians selected, and how was ReCIPE set up for these fields?
  4. What is the training cost of building the final graph, and how does it compare with the baselines' one-time compilation cost?
  5. Could the authors provide the prompts and settings used for the baselines?
2 comments
AuthorsOpenReview ↗

Response to Reviewer U9VS (Part 1/2)

We thank the reviewer for highlighting the practical value of reusable extraction procedures, traceable evidence, and lower test-time cost. We also appreciate their positive assessment of our evaluation across training snapshots and unseen patients, the comparison with a physician-designed graph, and the blinded review of real oncology records.

We respond to each concern and question below.

Q1. In the final graph, how many routes use LLM or hybrid extraction (including inside composites), and how many are purely deterministic?

Response. The stored final graph G80G_{80} contains 47 routes: 4 hybrid routes (including one composite) and 43 deterministic routes. The large deterministic share is mainly due to 32 EXTRACT (mode: regex) routes; excluding these leaves 11 deterministic and 4 hybrid routes. The LLM also contributes to learning the deterministic routes: the compiler proposes their input bindings and operator parameters, and the verifier checks trial outputs against patient evidence. Reusing these learned procedures reduces the need for repeated model inference across patients.

Route implementationRoutesUses model-based extraction?
LOOKUP5No
AGGREGATE1No
PARSE_NORMALIZE1No
FILTER_JOIN4No
EXTRACT, mode: regex32No
COMPOSITE1Yes
EXTRACT, mode: hybrid3Yes
Total474 hybrid; 43 deterministic

Q2. The Overall score weights each field type equally, although Datetime has far fewer fields than Text. How do the main comparisons change under a field-weighted average?

Response. ReCIPE remains first under a field-weighted arithmetic average: 0.969 vs 0.917 for CAESURA, the strongest baseline under this aggregation. It also ranks first when the harmonic mean is weighted by field counts. The full comparison is given below.

The benchmark contains 43 targets: 11 Enum, 4 Numeric, 2 Datetime, and 26 Text. Let fjf_j be the score for field jj, using its original metric, and let sts_t be the arithmetic mean of field scores within type tt. The field counts are

s_t=\frac{1}{n_t}\sum_{j\in\mathcal F_t}f_j.$$ We compare four explicitly defined aggregations, keeping predictions, field metrics, and eligibility unchanged: | Aggregation | Definition | Interpretation | | -------------------- | ----------------------------------------------------- | ------------------------------------------------------ | | $H_{\mathrm{type}}$ | $4/\sum_t s_t^{-1}$ | Paper's Overall: equal type weights, harmonic mean | | $A_{\mathrm{type}}$ | $\frac{1}{4}\sum_t s_t$ | Equal type weights, arithmetic mean | | $H_{\mathrm{field}}$ | $43/\sum_t n_t s_t^{-1}$ | Harmonic mean of type scores, weighted by field counts | | $A_{\mathrm{field}}$ | $\dfrac{1}{43}\sum_j f_j=\dfrac{1}{43}\sum_t n_t s_t$ | Equal weight for every field, arithmetic mean | | Aggregation | Direct LLM | Hybrid-RAG | HippoRAG 2 | CAESURA | EVAPORATE-CODE+ | OmniTQA | DocETL | EHRAgent | **ReCIPE** | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | $H_{\mathrm{type}}$ (paper) | 0.213 | 0.393 | 0.395 | 0.845 | 0.526 | 0.821 | 0.832 | 0.836 | **0.967** | | $A_{\mathrm{type}}$ | 0.297 | 0.455 | 0.456 | 0.855 | 0.661 | 0.828 | 0.841 | 0.846 | **0.969** | | $H_{\mathrm{field}}$ | 0.285 | 0.421 | 0.427 | 0.909 | 0.671 | 0.866 | 0.892 | 0.899 | **0.967** | | $A_{\mathrm{field}}$ | 0.351 | 0.451 | 0.458 | 0.917 | 0.769 | 0.872 | 0.898 | 0.909 | **0.969** | **ReCIPE ranks first under all four aggregations**, with CAESURA the strongest baseline in each case. ReCIPE's lead is consistent across these aggregation choices.
AuthorsOpenReview ↗

Response to Reviewer U9VS (Part 2/2)

Q3. For the oncology review, how many outputs differed in total, how were the excerpts shown to physicians selected, and how was ReCIPE set up for these fields?

Response.

(1) Total number of differing outputs.

Among the aligned field outputs processed to date, ReCIPE and the hospital’s manual Reference differed in 22.05% (5,077,748/23,024,962). We reviewed 400 sampled disagreements. The remaining outputs have not undergone independent physician verification because of the time required.

(2) Selection of review items and source excerpts.

We randomly sampled 100 disagreements per field type (Enum, Numeric, Datetime, and Text) before physicians assessed correctness. Since ReCIPE returns source evidence with its predictions, we highlighted the passages it cited in the original text. The full text remained available to physicians, allowing them to examine surrounding context and evidence relevant to either output. Five physicians independently reviewed each field's definition and both outputs, with method identities hidden and candidate values randomly assigned to A/B. They judged whether only A, only B, both, or neither was correct.

(3) ReCIPE setup on the hospital data.

We used Qwen3.5-27B, as in MIMIC-IV-Graph, and started from an empty RouteGraph. Physicians defined the hospital fields, specifying their clinical meaning and output requirements. ReCIPE used these definitions to propose and execute recipes on the hospital records, check their outputs against source evidence, and retain supported recipes for reuse. Because the hospital dataset is private, we cannot disclose the detailed field definitions or clinical record examples.

Q4. What is the training cost of building the final graph, and how does it compare with the baselines' one-time compilation cost?

Response. Building G80G_{80} on 2,000 training patients (2,487 admissions) requires 5.009M tokens, including route discovery, execution, verification, and retries. Among baselines, only EVAPORATE-CODE+ has a separate one-time compilation stage: synthesizing the extraction-function bank reused across admissions. This stage uses 2.285M tokens. Additional one-time compilation overhead is negligible for the other evaluated adaptations; their query-specific planning and code generation are counted as online execution costs.

Despite ReCIPE's higher upfront cost, combining these data with Table 1 yields estimated totals of 6.770 M tokens for ReCIPE vs 18.913 M for EVAPORATE-CODE+, corresponding to 64.2% fewer tokens overall.

Q5. Could the authors provide the prompts and settings used for the baselines?

Response. All eight baselines use Qwen3.5-27B with temperature 0.0 and the same test admissions and public field contracts. We use a shared extraction prompt, supplemented with instructions for each method and stage. We acknowledge that these details are essential to the paper's reproducibility, so we have formally included the prompt templates and shared model settings in Appendix H (Prompts and Settings) of the revised manuscript, rather than presenting them only in this response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.