- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes CATC, a runtime framework for clinical multi-agent LLM systems. A read-only auditor builds an incremental provenance graph over the execution prefix and checks it for local and cross-agent inconsistencies. The method is evaluated on six clinical datasets under five orchestration patterns against a vanilla multi-agent system and five verification baselines.
Critical strengths
The strict online information boundary, which excludes future events and gold labels, is stated clearly. Separating detection from intervention is a careful choice.
Critical weaknesses
- The headline gain rests largely on one dataset with an unstable baseline. The vanilla baseline on BioMed-R1 ranges from 23.5 to 70.6 with the same backbone at temperature 0. A spread this large points to answer-format, parsing, or aggregation failures rather than differences in clinical reasoning. CATC's largest improvements occur exactly where B0 is lowest. Its format/contract checks could plausibly recover these cases without any improvement in clinical evidence handling. The authors should break down the failures that CATC rescues on this dataset by anomaly family.
- Audit quality is claimed but never measured directly. The abstract says the evaluation covers audit quality and intervention safety, yet the paper defines no ground-truth anomaly annotations. Figure 4 reports a "Stage 1 F1" without stating what reference it is computed against. The ablation gives only values normalized to 100, with no absolute numbers, no dataset or orchestration specification, and no definitions of "protocol quality," "rescue," or "damage rate." As a result, readers cannot tell whether the auditor detects real handoff failures or whether downstream gains come from extra review calls.
- The compute-matching claim is not credible as stated. The paper says B0 to B5 and CATC share the same call budget K. However, CATC adds an external auditor that generates and executes code and performs semantic reasoning after every event. The paper does not say which model serves as the auditor, whether its tokens and calls count toward K, or what CATC adds in latency and cost per task. If auditor computation sits outside the budget, CATC is being compared against baselines that were given less computation.
- The baselines are under-specified, and some behave implausibly. The single-critic baseline (B2) reproduces B0 exactly in most cells, which suggests the critic seldom affects the output. The online auditor baseline (B5) is the comparison closest to CATC, but it is described only by name. The paper does not compare against the trace-auditing methods it cites, such as AgentForesight, AgentTether, or AgentSpec. It also lacks a strong single-agent baseline in the main table. Such a baseline would show whether the multi-agent setup is needed at all, which matters given the small gains on the perception datasets.
- The method is hard to reproduce, and part of the analysis adds little. Section 3 is mostly interface and schema definitions. Key mechanisms, including how the codebook is compiled, how the DSL constrains generated programs, and how semantic checks such as NLI are carried out and calibrated, are not specified enough to reimplement. Figure 5a conditions on tasks that already show monotonic improvement, which makes the selection circular.
Questions for authors
See weaknesses above.
Clinical Corrections, Evaluation Evidence, and Reproducibility
Thank you for the detailed questions. We address the clinical corrections, evaluation, compute, baselines, and reproducibility below.
C1. Clinical corrections. Among 81 inspected BioMed-R1 task-topology instances scored incorrect for B0 and correct for CATC, nine involved medical-knowledge or reasoning corrections, five aggregation/consistency corrections, and 61 protocol recovery. The remaining six concerned problems the agents could not resolve reliably. Under M1 and M3, CATC corrected B0's interpretation of a negative modified Allen test and the resulting radial-artery puncture decision, consistent with WHO guidance: https://www.ncbi.nlm.nih.gov/books/NBK138652/. Section 4.2 reports these corrections alongside the predominance of protocol recovery (61/81, 75.3%) in the inspected set.
C2. Evidence for auditing and coordination. The paired task analysis gives a macro gain of +6.6775 pp (95% CI [5.5114, 7.8552]). Component ablations examine the contribution of graph views, routing, provenance, and lifetime control. Trace inspection identifies substantive medical-reasoning corrections, while Appendix E records a numerical inconsistency, its witness-grounded GraphPatch, targeted CCI delivery, and a separately executed downstream correction. Together, these results provide complementary evidence of task-level effectiveness and concrete error-correction behavior. We appreciate the suggestion to extend this evidence with independent human evaluation of audit quality and clinical safety. We plan manual review of anomaly annotations and intervention outcomes. Organizing and completing such an assessment rigorously is difficult within the limited rebuttal period.
C3. Compute accounting. Section 4.1 and Appendices B-C now clarify that conditions share the base execution flow, with additional audit/repair computation counted separately and no shared cumulative call/token budget. K describes the base topology. CATC adds 958 calls to 12,192 base calls (+7.86%). The estimated token increase is 4.27%, including extrapolation for missing telemetry. The runtime auditor uses deterministic Python checks, with extra model calls coming from selected debate executions.
C4. Baselines and single-agent comparisons. B0-B4 use official AgentWorld, Self-Refine, MultiAgent Research Ideator single-critic, ReConcile, and DoVer implementations. B5 combines AgentForesight, AgentSpec, and ProbGuard. Table 1, Section 4.1, and Appendix C now document their sources and configurations. The B2 critic ran for all 2,545 task-orchestration instances and revised 23 final outputs: four improved, four degraded, and 15 were unchanged relative to B0. This is consistent with the limited aggregate gain. Appendix C.1 specifies B5's prefix replay, one-call joint diagnostic views, first-alarm stopping, one local repair, and candidate acceptance.
S1 uses iterative reasoning by one agent. S2 generates candidates and selects or aggregates with the same agent. Both use the same model and persona, with call allowances aligned to the MAS topology. We randomly sampled six available paired comparisons for supplementary analysis. Tokens are mean realized usage across two paired replicates.
C5. Figure 5 and reproducibility. We replaced the outcome-selected subset with paired score changes for all 509 tasks. Both steps, S0 to mean(S1) and mean(S1) to mean(MAS), are positive for 199 tasks (39.1%). At least one step is unchanged or lower for 310 tasks (60.9%).
Appendix E now links a deterministic inconsistency check, witness-grounded GraphPatch, routing, and one-call CCI injection. A separately recorded downstream call received source evidence and a recheck instruction, without the precomputed result or gold, and returned 38.0 mL/min, matching gold offline. The appendix identifies the additional source evidence, case-adapter inputs, and two recorded executions.