acceptodds
Under review as a conference paper at ICLR 2027

CATC: Coordinating Clinical Agent Team via Runtime Local–Global Trace Audit

Abstract

Clinical large language model agent teams coordinate specialized roles to solve complex clinical tasks. Existing approaches primarily verify individual responses or final outputs, leaving failures that emerge and propagate through inter-agent communication and context management unobserved during execution. We introduce Coordinating Clinical Agent Team (CATC), a framework for coordinating a clinical agent team via runtime local–global trace audit. CATC follows a two-stage design. An independent auditor combines dynamically generated code checks with semantic reasoning over an incremental interaction graph to identify local agent failures and global inconsistencies in communication and context propagation. CATC then converts structured audit findings into targeted corrections by the agent team, while an independent verifier controls whether each correction is accepted. We evaluate CATC across multiple clinical datasets spanning perception-oriented and cognition-oriented tasks, examining audit accuracy, failure localization, final task performance, intervention safety, and computational cost. These results establish runtime trace audit as a mechanism for converting additional agent-team computation into more reliable clinical reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Clinical Large Language Models, Agent-Team Coordination, Runtime Auditing, Trace Audit, Failure Localization, Context Propagation, Clinical Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TGJyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CATC, a runtime framework for clinical multi-agent LLM systems. A read-only auditor builds an incremental provenance graph over the execution prefix and checks it for local and cross-agent inconsistencies. The method is evaluated on six clinical datasets under five orchestration patterns against a vanilla multi-agent system and five verification baselines.

Critical strengths

The strict online information boundary, which excludes future events and gold labels, is stated clearly. Separating detection from intervention is a careful choice.

Critical weaknesses
  1. The headline gain rests largely on one dataset with an unstable baseline. The vanilla baseline on BioMed-R1 ranges from 23.5 to 70.6 with the same backbone at temperature 0. A spread this large points to answer-format, parsing, or aggregation failures rather than differences in clinical reasoning. CATC's largest improvements occur exactly where B0 is lowest. Its format/contract checks could plausibly recover these cases without any improvement in clinical evidence handling. The authors should break down the failures that CATC rescues on this dataset by anomaly family.
  2. Audit quality is claimed but never measured directly. The abstract says the evaluation covers audit quality and intervention safety, yet the paper defines no ground-truth anomaly annotations. Figure 4 reports a "Stage 1 F1" without stating what reference it is computed against. The ablation gives only values normalized to 100, with no absolute numbers, no dataset or orchestration specification, and no definitions of "protocol quality," "rescue," or "damage rate." As a result, readers cannot tell whether the auditor detects real handoff failures or whether downstream gains come from extra review calls.
  3. The compute-matching claim is not credible as stated. The paper says B0 to B5 and CATC share the same call budget K. However, CATC adds an external auditor that generates and executes code and performs semantic reasoning after every event. The paper does not say which model serves as the auditor, whether its tokens and calls count toward K, or what CATC adds in latency and cost per task. If auditor computation sits outside the budget, CATC is being compared against baselines that were given less computation.
  4. The baselines are under-specified, and some behave implausibly. The single-critic baseline (B2) reproduces B0 exactly in most cells, which suggests the critic seldom affects the output. The online auditor baseline (B5) is the comparison closest to CATC, but it is described only by name. The paper does not compare against the trace-auditing methods it cites, such as AgentForesight, AgentTether, or AgentSpec. It also lacks a strong single-agent baseline in the main table. Such a baseline would show whether the multi-agent setup is needed at all, which matters given the small gains on the perception datasets.
  5. The method is hard to reproduce, and part of the analysis adds little. Section 3 is mostly interface and schema definitions. Key mechanisms, including how the codebook is compiled, how the DSL constrains generated programs, and how semantic checks such as NLI are carried out and calibrated, are not specified enough to reimplement. Figure 5a conditions on tasks that already show monotonic improvement, which makes the selection circular.
Questions for authors

See weaknesses above.

1 comment
AuthorsOpenReview ↗

Clinical Corrections, Evaluation Evidence, and Reproducibility

Thank you for the detailed questions. We address the clinical corrections, evaluation, compute, baselines, and reproducibility below.

C1. Clinical corrections. Among 81 inspected BioMed-R1 task-topology instances scored incorrect for B0 and correct for CATC, nine involved medical-knowledge or reasoning corrections, five aggregation/consistency corrections, and 61 protocol recovery. The remaining six concerned problems the agents could not resolve reliably. Under M1 and M3, CATC corrected B0's interpretation of a negative modified Allen test and the resulting radial-artery puncture decision, consistent with WHO guidance: https://www.ncbi.nlm.nih.gov/books/NBK138652/. Section 4.2 reports these corrections alongside the predominance of protocol recovery (61/81, 75.3%) in the inspected set.

C2. Evidence for auditing and coordination. The paired task analysis gives a macro gain of +6.6775 pp (95% CI [5.5114, 7.8552]). Component ablations examine the contribution of graph views, routing, provenance, and lifetime control. Trace inspection identifies substantive medical-reasoning corrections, while Appendix E records a numerical inconsistency, its witness-grounded GraphPatch, targeted CCI delivery, and a separately executed downstream correction. Together, these results provide complementary evidence of task-level effectiveness and concrete error-correction behavior. We appreciate the suggestion to extend this evidence with independent human evaluation of audit quality and clinical safety. We plan manual review of anomaly annotations and intervention outcomes. Organizing and completing such an assessment rigorously is difficult within the limited rebuttal period.

C3. Compute accounting. Section 4.1 and Appendices B-C now clarify that conditions share the base execution flow, with additional audit/repair computation counted separately and no shared cumulative call/token budget. K describes the base topology. CATC adds 958 calls to 12,192 base calls (+7.86%). The estimated token increase is 4.27%, including extrapolation for missing telemetry. The runtime auditor uses deterministic Python checks, with extra model calls coming from selected debate executions.

C4. Baselines and single-agent comparisons. B0-B4 use official AgentWorld, Self-Refine, MultiAgent Research Ideator single-critic, ReConcile, and DoVer implementations. B5 combines AgentForesight, AgentSpec, and ProbGuard. Table 1, Section 4.1, and Appendix C now document their sources and configurations. The B2 critic ran for all 2,545 task-orchestration instances and revised 23 final outputs: four improved, four degraded, and 15 were unchanged relative to B0. This is consistent with the limited aggregate gain. Appendix C.1 specifies B5's prefix replay, one-call joint diagnostic views, first-alarm stopping, one local repair, and candidate acceptance.

S1 uses iterative reasoning by one agent. S2 generates candidates and selects or aggregates with the same agent. Both use the same model and persona, with call allowances aligned to the MAS topology. We randomly sampled six available paired comparisons for supplementary analysis. Tokens are mean realized usage across two paired replicates.

ComparisonDatasetMAS-SA (pp)W/L/TTokens MAS/SA
M5/S1-M5MedCalc+7.6516/5/6417.8k/13.6k
M2/S1-M2MedCalc+4.1212/8/656.35k/6.75k
M2/S2-M2BioMed-R1+4.128/3/744.50k/4.35k
M2/S2-M2PMOA-TTS+3.5760/22/518.2k/20.8k
M4/S1-M4MedCalc+3.5315/9/616.09k/6.52k
M2/S2-M2EHRCon+2.2611/6/1243.9k/53.2k

C5. Figure 5 and reproducibility. We replaced the outcome-selected subset with paired score changes for all 509 tasks. Both steps, S0 to mean(S1) and mean(S1) to mean(MAS), are positive for 199 tasks (39.1%). At least one step is unchanged or lower for 310 tasks (60.9%).

Appendix E now links a deterministic inconsistency check, witness-grounded GraphPatch, routing, and one-call CCI injection. A separately recorded downstream call received source evidence and a recheck instruction, without the precomputed result or gold, and returned 38.0 mL/min, matching gold offline. The appendix identifies the additional source evidence, case-adapter inputs, and two recorded executions.

Reviewer_iKDmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a clinical agent system that is designed to have local and global trace auditing. The system is evaluated on six datasets with the authors arguing that the results support the conclusion that the trace auditing improves clinical reasoning. The paper builds on a large existing literature on clinical reasoning frameworks.

Critical strengths

The paper is on a interesting research topic. The use of LLMs and agent systems is an important and interesting research topic. There are many considerations and the ability to audit and reason about parts of these agentic systems will be valuable. The authors have designed a system and evaluated it on a number of datasets. The authors have presented a reasonable amount of experim

Critical weaknesses

I have a number of concerns about the positioning and experiments in the paper.

The first is that the author cite many papers a vast majority of which are unreviewed ArXiv papers. The is nothing wrong with citing ArXiv papers, however the volume of unpeer-reviewed work alerted me to the related work. Specifically there are many peer reviewed foundational papers in this space (clinical reasoning with LLMs and agentic clinicial reasoning with LLMs) which are not referenced. This makes me concerned that the authors are not aware of this work and have not compared their system to these prior baselines. I recognize that the field is moving fast but think it significantly weakens the paper if the work is not positioned well.

I find the way the results are presented confusing. The authors present ablations and a baseline but it isn't clear to me if these baseline represent prior published work or are essentially the author's own ablations/baselines. It isn't clearly articulated.

Some of the figures (e.g. Fig. 5) are essentially unreadable and need a lot of work to make them useful and informative to the reader.

The results are dense and not very readable, they are essentially just a list on numbers and the discussion of the results is very limited.

Questions for authors

Can you explain why you did not reference, and use as baselines, more established agentic frameworks in the space?

Can you clarify which of the baseline(s) are from prior work? You say CATC improves over the "vanilla multi-agent baseline," but there are many multi-agent baselines in prior work.

Why did you not choose to include other much more established baselines?

Can you improve the presentation of the figures and have been explanation of the results?

2 comments
Reviewer_iKDmOpenReview ↗

No response from the authors

I don't see a response from that authors.

AuthorsOpenReview ↗

Clinical Literature, Baseline Design, and Result Interpretation

Thank you for identifying gaps in the literature positioning, baseline description, and presentation. We have revised these parts of the manuscript and address each concern below.

C1. Peer-reviewed clinical literature and publication metadata. We have expanded Section 2 to include MedAgents (Findings of ACL 2024), MDAgents (NeurIPS 2024), AMIE (Nature 2025), and MAC (npj Digital Medicine 2025). The discussion now covers role-based medical reasoning, complexity-adaptive collaboration, diagnostic-dialogue evaluation, and multidisciplinary diagnostic consultation. These works provide the clinical context for CATC's focus on runtime trace auditing and targeted coordination. We updated the publication metadata of 24 references, including replacing arXiv-only descriptions with the corresponding conference and journal records. Of the updated entries cited in the current manuscript, 19 now identify formal conference or journal publications, including ReAct, Reflexion, Self-Refine, and ReConcile.

C2. Which baselines use prior implementations? B0-B4 use the official implementations of AgentWorld, Self-Refine, the single-critic baseline from MultiAgent Research Ideator, ReConcile, and DoVer, respectively. B5 is our composite reproduction of prefix auditing from AgentForesight, constraint enforcement from AgentSpec, and proactive monitoring from ProbGuard. Table 1 now cites each B0-B5 row. Section 4.1 and Appendix C document definitions, implementation sources, repository links, and the mechanisms evaluated. These conditions are distinct from CATC's component ablations. "Vanilla MAS" refers specifically to B0 in this evaluation. Our experiments address the full agent-collaboration protocol across multiple design patterns. Many of the compared approaches optimize a particular pattern or component, making direct system-level comparison difficult without aligning the collaboration protocol. Appendices B and C now describe and illustrate all five orchestration patterns and the B0-B5 mechanism conditions.

C3. Why not compare complete established agentic frameworks? Our comparison asks how refinement, critique, consensus, attribution/repair, and auditing affect a given collaboration topology under a common clinical protocol. The implementations share the backbone, task IDs, splits, context limits, output contracts, and evaluators. Additional computation is accounted for separately. Comparing complete frameworks would also change role prompts, orchestration, tools, memory, and interfaces, making the resulting score difference difficult to attribute to auditing and coordination. The current design therefore evaluates mechanisms across five topologies, using established implementations rather than only internal ablations. This answers a narrower question than a complete clinical-framework benchmark. The added clinical citations improve positioning. They do not imply new head-to-head experiments with those systems.

C4. Figure readability. We redrew Figure 5 as a single plot with larger labels, a simpler legend, and a caption explaining the paired score changes and markers. It now includes all 509 tasks, without filtering by the direction of either change. The backbone-model comparison is presented separately as Figure 6 in Section 4.4. We corrected its model-series labels to align the GLM-5.2 series with Table 1 and updated the difference signs accordingly. Appendices B and C also include orchestration and baseline schematics to clarify the experimental conditions.

C5. Interpretation of the results. We reorganized Section 4.2 to explain task dependence, the nature of the observed improvements, and baseline behavior, reducing repeated numerical comparisons. The discussion distinguishes the large BioMed-R1 gain from the smaller EHRCon and CaseReport changes. Among 81 inspected BioMed-R1 task-topology improvements, 61 involved protocol recovery, nine medical knowledge or reasoning corrections, five aggregation or consistency corrections, and six problems the agents could not resolve reliably. These counts characterize the inspected instances rather than decomposing the aggregate net gain. We also discuss B3's 0.4 pp higher mean under M1 and CATC's highest mean under the other four topologies. Appendix E provides a numerical-correction example with its additional source evidence, case-adapter inputs, and two separately recorded executions stated explicitly. The revised discussion connects the numerical results to their interpretation and evidential scope.

Reviewer_JKLEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Clinical Agent Team Coordination (CATC), a framework for auditing and guiding clinical multi-agent systems during execution. A read-only auditor combines generated code checks with semantic reasoning over an incremental interaction graph to identify errors within individual agents and inconsistencies across agents. Each finding retains its supporting evidence and is routed to a subsequent agent invocation as a structured review constraint. The resulting events are then audited again. The evaluation covers 509 tasks from six datasets under five collaboration patterns and reports improvements over the corresponding vanilla systems, alongside comparisons with verification baselines and component ablations.

Critical strengths

1 The paper addresses a useful problem Important evidence can be lost during handoffs, and unsupported claims can be repeated by later agents. These errors may persist even when the agents eventually agree. CATC checks both individual agents against the evidence available to them and the transmission of information across the team. This is a useful way to identify failures that may be difficult to locate by inspecting the final answer alone.

2 The method sets clear limits on intervention CATC first examines completed interactions and then directs its findings to a later agent that can act on them. Anomaly records retain their evidence sources, and injected constraints have an explicit scope and lifetime. The paper also specifies how to handle insufficient information and the absence of a suitable intervention target. These details make the overall procedure easier to understand and help define what the auditor is allowed to do.

3 The experiments cover several tasks and collaboration patterns The evaluation spans six datasets and five collaboration patterns under shared call and token budgets. CATC improves on the vanilla system in all 30 dataset-topology combinations, with an average gain of about 6.76 percentage points, and achieves the highest mean performance in four of the five patterns. The ablations also show that components such as local and global checks and target routing affect the results.

Critical weaknesses

1 The gains are uneven across tasks This is a major concern for me. The average gain on BioMed-R1 is about 25.65 percentage points, compared with only 0.31 on EHRCon and 0.90 on CaseReport. The overall improvement is therefore strongly influenced by BioMed-R1. Although the other tasks also improve, some gains are small, and the paper does not provide confidence intervals or paired significance tests to show whether they are reliable. I would like to understand which errors CATC corrects and why it helps substantially on some tasks but has little effect on others. An analysis by error type would help establish whether CATC broadly improves collaboration or is especially effective for a narrower set of problems.

2 Actual token usage and response latency are not reported The authors use shared budgets, deterministic checks, and asynchronous execution to control overhead. However, the paper does not report how many tokens the full system actually consumes or how long it takes to complete a task. It also does not break down the costs of audit-program generation, semantic checks, and constraint injection. In particular, an audit finding must be ready before the target invocation for the constraint to take effect. It is unclear how often this happens in practice. End-to-end latency, token usage by component, and the proportion of interventions completed in time would help assess the practical value of the method.

3 The absolute reliability of the auditor remains unclear Figure 4 normalizes the full CATC system to 100 and shows the relative changes after removing components. This helps assess whether a component matters, but it does not tell us the actual audit F1 or how often an intervention causes harm. The source of the anomaly labels and the procedure used to establish them also need more explanation. A higher final task score can hide cases where CATC corrects some errors while changing previously correct answers into incorrect ones. I would like to see the rates of successful corrections and harmful interventions, in addition to the net improvement in task performance.

4 The comparison with B5 needs more detail B5 is itself an online auditor with repair, so its basic purpose is close to that of CATC. The paper does not sufficiently explain which interaction records B5 can access, which checking tools it can use, or how it carries out a repair after finding a problem. These details matter when interpreting the comparison. Even with the same budget, differences in available information or tools could contribute to the performance gap. A clearer account of the two configurations would help establish how much of the gain comes from CATC's graph structure and target routing.

Questions for authors

See above

1 comment
AuthorsOpenReview ↗

Statistical Evidence, Computational Cost, and Audit Reliability

Thank you for the detailed questions. We have added paired statistical evidence, clarified the scope of the compute comparison, and documented the B5 configuration.

C1. Uneven gains and statistical support. The gains are task-dependent. The equal-weight mean over six datasets and five topologies is +6.6775 percentage points (pp), with a 95% CI of [5.5114, 7.8552]. The CI uses 12,000 dataset-stratified task-block bootstrap samples, preserving each task's five topology results. The overall task-block sign-flip test uses 200,000 Monte Carlo draws (p approximately 5.0e-6). There are 509 task blocks and 2,545 paired task-topology records. The latter are not treated as independent tasks. The 30 dataset-topology comparisons use two-sided exact Wilcoxon signed-rank tests, with Benjamini-Hochberg correction over these 30 tests only. Mean CATC-B0 differences are positive in all 30 comparisons. MedR-Bench uses a continuous composite diagnosis-and-treatment score. All five topologies show statistically significant gains on BioMed-R1, MedR-Bench, and PMOA-TTS. MedCalc-Bench, EHRCon, and CaseReportBench also show positive mean gains, although these smaller effects do not reach the corrected significance threshold. Sections 4.1-4.2 now report this task-dependent pattern.

C2. Model usage, latency, and intervention timing. CATC adds 958 model calls to 12,192 base calls (+7.86%). Estimated total tokens are 48.831M for B0, 50.918M for CATC (+4.27%), and 60.270M for B5 (+23.43%). These token totals include extrapolation from sparse telemetry: coverage is 56/12,192 base calls, 2/958 CATC extra calls, 4/5,002 B5 auditor calls, and 1/263 B5 repair calls.

The evaluated CATC auditor runs deterministic Python checks. The extra model requests come from the selected B3 debate executions for 150 high-risk cases. Relative to B0, mean end-to-end overhead under M1-M5 is 50.861, 5.853, 12.733, 10.579, and 3.091 seconds (3.75%, 0.45%, 0.93%, 0.72%, and 0.21%). Python auditing averages 0.000706 seconds per task. All 110 eligible interventions were prepared and injected before their target call. Clinical corrections are also documented: among 81 inspected BioMed-R1 task-topology improvements, nine involved medical-knowledge or reasoning corrections and five aggregation or cross-agent consistency corrections. Appendix E further records an evidence-guided numerical correction from 53.2 to 38.0 mL/min, with a separately recorded downstream execution.

C3. Absolute audit reliability and intervention safety. Thank you for this valuable suggestion. Independent human evaluation of audit quality and clinical safety would be a meaningful extension of this work. Our current experiments provide complementary evidence through paired task outcomes, component ablations, trace inspection of medical-reasoning corrections, and the worked audit-to-correction example in Appendix E. We plan to extend this evaluation with independent manual review of anomaly annotations and intervention outcomes, including successful corrections and potentially harmful changes. A rigorous assessment requires a consistent annotation protocol and coordinated human review, which is difficult to organize and complete within the limited rebuttal period.

C4. B5 implementation and repair configuration. Section 4.1 and Appendix C.1 now specify B5 as a composite of AgentForesight, AgentSpec, and ProbGuard. It replays prefixes of completed traces, rather than interrupting a live MAS. One auditor call jointly produces three diagnostic views from the visible prefix and task context. Gold answers and future events are hidden. The views are not independent models or independent statistical signals. An alarm occurs on a Spec violation, agreement of at least two views, or a maximum risk score of 0.85. The score is uncalibrated. Auditing stops at the first alarm and permits one local repair. Missing, blank, abstaining, or contract-invalid candidates retain B0's answer. Appendix C.1 also specifies prefix selection, repair targeting and feedback, and resource accounting. B5 evaluates this composite condition, not three separate full-system replications. Its replay timing is not a test of live pre-invocation intervention capability.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Core Contribution and Summary of Revisions

Thank you for considering our submission. CATC addresses errors that arise during collaboration in clinical LLM teams. An agent can omit a relevant finding during a handoff, or several agents can carry the same unsupported assumption into the final answer. CATC audits the execution trace as the team runs, links each detected inconsistency to its supporting evidence, and sends a targeted constraint to a subsequent agent call. The same audit and coordination interface works across five orchestration patterns.

The revised manuscript makes three changes in response to the reviews:

  1. Quantify where CATC helps. We added paired statistical analyses across 509 tasks, six datasets, and five orchestration patterns. CATC improves on the vanilla multi-agent baseline by 6.68 percentage points on average (95% CI: 5.51–7.86). Mean gains are positive in all 30 dataset–topology comparisons. Gains on BioMed-R1, MedR-Bench, and PMOA-TTS are significant across all five patterns after multiple-testing correction. Gains on the other three datasets are smaller and do not reach that threshold.
  2. Show what CATC corrects. Section 4.2 now distinguishes protocol recovery from medical reasoning and cross-agent consistency corrections. Among 81 inspected BioMed-R1 improvements, 61 involved protocol recovery, nine involved medical knowledge or reasoning, and five involved aggregation or consistency. Appendix E traces how a detected inconsistency led to a targeted recheck that corrected a clinical calculation.
  3. Clarify how the comparisons were made. We expanded the clinical related work, documented the baseline implementations and B5 configuration, and clarified how additional computation is counted. Figure 5 now shows the paired changes for all 509 tasks, replacing the subset selected for improvement at both steps. The rebuttal replies separately report resource estimates and six supplementary single-agent comparisons.

These revisions clarify where runtime trace auditing improves benchmark outcomes and show how individual corrections occur.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.