acceptodds
Under review as a conference paper at ICLR 2027

TEMPLAR: Agentic Induction and Evolution of Standardized Radiology Reporting Templates from Large-Scale Clinical Corpora

59 views

Abstract

Structured radiology reporting mitigates the heterogeneity of free-text reports, yet its benefits depend on high-quality reporting templates. In practice, such templates are conventionally built through labor-intensive expert consensus and therefore vary across institutions and lag behind evolving clinical practice. Large language models (LLMs) enable automated template induction, but existing approaches remain limited: single-LLM induction is constrained by context length, and the corpus-scale method ASTAR produces a static, closed-corpus template without external grounding or downstream adaptation. To address these limitations, we propose TEMPLAR, a TEMPLate-centric Agentic framework for inducing and evolving standardized Radiology reporting templates from large-scale clinical corpora. TEMPLAR treats the template as a persistent central state maintained alongside two provenance-aware knowledge graphs, namely an anatomical graph that constrains template construction and a diagnostic graph that supports finding-to-diagnosis reasoning. Three agents operate on this state. The Induction Agent derives canonical clinical slots from anatomy-constrained Span-Triple atoms via dual-view similarity clustering; the Evolution Agent then assembles these slots into a hierarchical template and revises it under consistency constraints, external clinical evidence, and downstream structuring feedback; and the Clinical Agent applies the evolved template to report structuring, reconstruction, and diagnostic reasoning. Across four datasets (PETCT, ViMed, MERLIN, and BDMAP), TEMPLAR outperforms ASTAR, three medical LLMs, and six general-purpose LLMs in coverage, information fidelity, and diagnostic fidelity, while achieving the highest or tied-highest LLM-rated template quality. Its fidelity advantages over ASTAR persist under cross-dataset transfer, and cumulative ablations support complementary contributions of its key components.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Anatomical knowledge graphs, Dual-view clustering, Evidence-based refinement, Feedback-driven calibration, Diagnostic reasoning.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tuVoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes TEMPLAR, which automatically induces and refines structured radiology reporting templates through anatomy-guided concept clustering, knowledge graphs, external evidence retrieval, and feedback-based refinement. The framework is evaluated on four report datasets for template coverage, reconstruction fidelity, and diagnostic fidelity.

Critical strengths

Automated template construction addresses a practical need by reducing manual design effort and improving report standardization.

Critical weaknesses
  1. Adaptive report structuring is clinically meaningful, but its value is limited. For corpora of current size, iterative template induction using a strong GPT or Claude model, followed by minor manual refinement, is a straightforward alternative worth comparing against.
  2. State-of-the-art models, such as GPT and Gemini, perform unexpectedly poorly, trailing the proposed method by more than ten points. This raises concerns about comparison fairness. Please clarify how these baselines were implemented.
  3. The datasets are too small to support the “large-scale” description, with even the largest containing fewer than 5,000 reports.
  4. The experiments use only subsets of the original datasets. For example, MERLIN originally contains over 20,000 reports, but only 4,080 are used here.
Questions for authors
  1. What effect would be achieved if GPT or Claude were to be optimized carefully?
  2. Why were only subsets of the original datasets used, and how were these subsets selected?
Reviewer_Gn6GOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles automatic induction of structured radiology reporting templates from free-text corpora. It builds on ASTAR and adds a few things. Slot induction is done per anatomical region with a BM25 plus embedding similarity graph and Leiden clustering instead of global k-means. The induced template is then revised, grounded with evidence retrieved from PubMed, Europe PMC and Google which gets attached as per-slot notes, and calibrated once using feedback from actual structuring runs. Two knowledge graphs, one anatomical and one diagnostic, are built alongside and used downstream for structuring, report reconstruction and diagnosis inference. Experiments on PETCT, ViMed, MERLIN and BDMAP compare against ASTAR and nine LLMs on coverage, LLM and expert rated template quality, reconstruction fidelity and LLM judged diagnostic fidelity, plus cross-dataset transfer and cumulative ablations. TEMPLAR reports the best numbers across the board.

Critical strengths
  1. The problem is real and under-explored. Most structured reporting work assumes the template exists and only automates filling it in. Going after template construction itself is a reasonable gap and the critique of ASTAR being static and closed-corpus is fair.
  2. The evaluation is broad. Four datasets, ten baselines, two transfer settings, a six-stage cumulative ablation and a separate pairwise ablation for revision and evidence notes. Not many papers in this space go this far.
  3. The failure analysis in C.6 is honest and useful. Documenting which baselines could not complete the protocol and why, including truncation and repetition degeneration, is the kind of thing most people quietly drop.
  4. The paper reports where it loses, for example the expert score on BDMAP where Gemini-3.5-Flash-Lite comes out ahead.
Critical weaknesses
  1. The main comparison is confounded. The reconstruction and diagnosis gains over ASTAR and the LLM baselines come partly from pipeline components that have nothing to do with the template, namely the retrieved training sentences used during reconstruction and the diagnostic graph used during reasoning. The ablation shows this directly. M4 adds RAG and ROUGE-1 on PETCT goes from 0.6473 to 0.6700 with the same template, and M6 adds the diagnostic graph and PDA jumps. As far as I can tell the baselines in Tables 1 and 2 do not get these. So the headline numbers compare ASTAR template with a bare pipeline against TEMPLAR template with a full pipeline. The fair number is the M5 vs M4 gap, which is still positive but much smaller than what is claimed. All baseline templates need to be run through the same Clinical Agent.
  2. Almost everything is LLM as judge with no statistics. Coverage is LLM matching, template quality is GPT-5.5, diagnostic fidelity is DeepSeek. There are no confidence intervals, no significance tests, no variance over runs. PETCT has 49 test cases. The expert score of 4.250 appearing identically on three datasets suggests one or two raters, and the number of experts, their background and agreement are never reported. On PETCT the LLM template quality ties with MedGemma and ClinFusion, which are the worst models on every other metric, so the judge does not seem to discriminate much.
  3. The agentic and evolution framing is overstated. It is a fixed prompt chain with no planning or loops, the calibration runs once, and the abstract says three agents while the text has four. The motivation is templates lagging behind evolving practice but there is no experiment where the template is actually updated over time.
Questions for authors
  1. In Tables 1 and 2, are the reconstruction and diagnostic numbers for ASTAR and the LLM baselines produced with the same Clinical Agent as TEMPLAR, including the top 80 retrieved training sentences and the diagnostic graph? If not, please report all baseline templates under the identical downstream pipeline. This is the single thing that would most change my view of the paper.
  2. How many expert raters were used, what was their background, were they blinded to method identity, and what was the inter-rater agreement?
  3. Can you provide confidence intervals or a significance test for the main comparisons, especially on PETCT with 49 test cases, and variance over repeated LLM judge runs?
  4. Does the overall conclusion in Table 5 hold if BDMAP is excluded or reported separately, given its reports are machine generated?
Reviewer_EERHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a framework to automatically create standardized radiology reporting templates from existing clinical reports. It proposes to use multiple agents to extract and organize clinical information into templates. The paper also uses external medical knowledge and feedback to further improve these templates. Through experiments on four datasets, the paper shows improvements in template coverage, information preservation, and diagnostic accuracy over existing methods.

Critical strengths
  1. The idea of automatically generating and refining radiology reporting templates is interesting. The use of external medical knowledge and feedback to improve the templates is a useful addition.

  2. The paper incorporates anatomical and diagnostic knowledge graphs to organize clinical information and support diagnostic reasoning.

  3. The paper presents extensive experiments on four datasets using multiple evaluation metrics. It also includes ablation studies and cross-dataset evaluations to validate the effectiveness and generalizability of the proposed method.

Critical weaknesses
  1. Clinical significance: My main concern is the clinical need for automatically generating new reporting templates, as many hospitals already use standardized templates developed by clinical experts. It is unclear when the proposed method would be preferred over existing templates. I believe a comparison with expert-designed clinical templates would be useful to better understand the practical significance of this work.

  2. Clinical evaluation: Although the paper includes expert evaluation of the generated templates, the reconstructed reports and diagnostic outputs are primarily evaluated using automated metrics and LLM judgments. I think an independent clinician evaluation would be useful to assess whether clinically important information is preserved. It would also be helpful to clarify the number of experts involved in the template evaluation and report inter-rater agreement.

  3. Evaluation metrics: The paper primarily uses ROUGE, chrF, and BERTScore to evaluate information preservation. However, these metrics may not adequately capture clinically important errors, including hallucinations and omissions. I believe an additional evaluation using recent clinically grounded metrics such as GREEN, CRIMSON, and ICARE, where applicable to the evaluated modalities, would strengthen the results. These approaches have shown agreement with clinician assessments and could provide a more reliable evaluation of clinical consistency.

Questions for authors

I would be open to raising my rating if the authors can adequately address the concerns mentioned above.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Reply and Summary of Revisions

We thank EERH for the questions on clinical value, Gn6G for the guidance on fair comparison and evaluation reliability, and tuVo for examining the methodological contribution and baseline design. These comments guided the following experiments and clarifications. Manuscript revisions are highlighted in yellow.

  1. Clinical motivation and public-template baselines (EERH). We clarified how automated induction supports template development and adaptation across clinical settings, and added public PET/CT and CT reporting templates as baselines (Secs. 1 and 4.1; Tables 1–2).

  2. Controlled comparisons (Gn6G, tuVo). We reevaluated every method's template with the same Clinical Agent for structuring, reconstruction, and diagnosis. We also clarified that the LLM baselines process the complete training split using shared report batches and instructions (Sec. 4.1; Tables 1–2; Appendix C.2).

  3. Clinically grounded evaluation (EERH, Gn6G). We added GREEN to assess clinically relevant reconstruction errors (Tables 1–2; Appendix B.3.3) and expanded physician assessment of template design and clinical utility, with raters blinded to method identity (Sec. 4.2; Figs. 3 and 6; Appendices B.3.2 and D.3).

  4. Evaluation reliability (Gn6G). We documented the revised evaluator configuration and added repeated evaluations and case-paired statistical analyses to assess run-to-run variation and statistical uncertainty under the common protocol (Tables 1–2 and 8–11; Appendices A and D.2).

  5. Agent roles and template evolution (Gn6G). We clarified the task-agent roles, their fixed execution order, and the separate Evaluation Agent. Template evolution focuses on coarse-to-fine correction of template structure and slot semantics, including one round of downstream feedback refinement. Longitudinal adaptation to changing clinical practice is a future direction (Abstract; Secs. 1, 3.2, and 5; Appendix B.2).

  6. Strong models and task-specific design (tuVo). We discuss why general model capability does not guarantee the best clinical template quality, and how TEMPLAR addresses the organization and refinement of clinical concepts explicitly. GPT/Claude backbones can also be incorporated into this procedure (discussion in our response to tuVo, W1/Q1; method in Secs. 3.1–3.2).

  7. Clinical privacy and local deployment (tuVo). We discuss institutional restrictions on external processing of clinical reports and the local deployment option offered by the open-weight counterpart of our Qwen template-construction backbone (discussion in our response to tuVo, W1/Q1; experimental backbone configuration in Appendix A).

  8. Cohort selection and sensitivity analysis (Gn6G, tuVo). We clarified cohort selection and train/test separation, and added a separate aggregate comparison excluding the machine-generated BDMAP reports (Secs. 4.1–4.2; Table 7; Appendices C.1 and D.1).

The reviewer-specific responses provide the supporting results and manuscript locations for each update.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.