- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes TRIM, a 7B LoRA controller that intervenes in a frozen medical agent’s workflow by choosing among CONTINUE, FINISH, JUMP and REQUEST at host-defined decision points. It reads statistics from a “Diagnostic Process Forest”, which indexes the host’s labeled past executions by workflow position and stores each action’s error rate, cost saving and support. Training uses SFT on the cheapest correct historical paths, then GRPO with a correctness-first, cost-second reward. On DiagGym, accuracy rises from 51.39% to 56.71% with 52.0% fewer model calls. Across 14 host–dataset pairs on MedQA, SLAKE-En and MIMIC-CXR, calls fall by 32.6% on average with no accuracy loss. The only baselines are the native workflow and an untrained “forest-only” controller.
Critical strengths
- A meaningful decision problem with a practical, host-agnostic formulation. Deciding when the evidence is sufficient and when to keep gathering it is a real problem for diagnostic agents, and current multi-agent systems clearly over-deliberate.
- Transparent accounting and failure analysis. The appendix separates host and controller decisions and calls, uses group-disjoint retrieval to avoid leakage, reports where the method does not help (MedRAX, MMedAgent) and why, and discloses confounds such as the MedAgents/SLAKE readout bug.
Critical weaknesses
W1. Incremental novelty, obscured by inconsistent terminology
- Prior work: Learned controllers over frozen agents with cost-aware RL already exist. 1) Puppeteer (Dang et al., “Multi-Agent Collaboration via Evolving Orchestration,” NeurIPS 2025) trains an orchestrator to select the next agent, rewarding correctness minus λ·cost. 2) ToolOrchestra (Su et al., ICML 2026) trains an 8B orchestrator with outcome and efficiency rewards to choose models and tools at each turn. Neither is cited, and Yi & Song (2026) is cited but not differentiated.
- DPForest contribution: The forest statistics alone barely help (forest-only on DiagGym: +0.85 pts, −1.3% calls), and no trained controller without them is reported.
- “Tree”: Host workflows are sequences (Sec. 3.1), and nearly all cases share one path (support 10,176/10,178, Table 8). The structure is a prefix-keyed lookup table (App. B.1).
- “Forest”: There is one tree per task root, but the root is defined only for DiagGym.
- “Host”: The term refers variously to an agent, a model, a runtime and an environment.
W2. The experiments do not test the claimed problem, and the cost claims are incomplete
- Task mismatch: MedQA and SLAKE are knowledge-based QA with all information given upfront, so the “decisions” reduce to early stopping of multi-agent discussion. DiagGym, the only sequential-diagnosis setting, offers just two exams (CBC, lipase).
- Missing baselines: There is no agreement-based stopping, cascade/router or efficient-MAS baseline.
- Confounds: Native multi-agent runs are not matched reruns (App. D.2), and 17.4 of the 22.5-point MedAgents/SLAKE gain comes from bypassing a host bug.
- Hidden cost: All cost figures exclude labeled trajectory collection (≈376k MedAgents calls on MedQA; ≈0.5B tokens for MedAgent-Pro) and up to ~16k GRPO rollouts per pair. Latency is estimated, not measured.
Questions for authors
- How does TRIM differ, conceptually and empirically, from Puppeteer, ToolOrchestra and Yi & Song (2026)? Please report the same SFT+GRPO controller trained without forest statistics, to isolate the DPForest’s contribution.
- The single-pass scorer “follows” the System One model of Almeida (2026), a blog post released on September 15, 2026, three days before the ICLR abstract deadline. Was this design adopted after that date, and were all reported runs trained with it? If it was developed independently, what does the citation add beyond a standard candidate-scoring head (as in reward models or multiple-choice heads)?
- What is the total offline cost per pair (calls, tokens, GPU-hours) of trajectory collection, labeling, synonym merging and GRPO rollouts, and after how many test queries does TRIM break even against the native host? Most agent frameworks do not require labeled historical trajectories; how would TRIM work when ground-truth labels are delayed or unavailable?
- If an off-the-shelf decision model motivates the controller, why is SFT+GRPO needed? GRPO adds only +0.94 points and 13.6% fewer calls over SFT on average, and JUMP/REQUEST are rare (18 and 62 across 827 patients). Would a zero-shot decision model, or a lightweight classifier on forest statistics, match these results?
- What is the task root for the 14 non-DiagGym pairs, how many trees does each forest contain, and how many nodes have more than one child? Please define “host” once and use it consistently.
- Can you rerun Native under the same generation settings as the controlled conditions and report confidence intervals?
7 comments
Rebuttal for EzYA on Critical Weaknesses
W2.3. Confounds.
- Within each host–dataset pair, Native and all controlled conditions use the same official host configuration and evaluator. We have clarified this in Appendix D.2.
- We acknowledge that much of the MedAgents/SLAKE gain comes from bypassing a native readout defect: it incorrectly applies a yes/no instruction to 290 of 645 open-ended questions, achieving only 2.4% accuracy on this subset. Returning the existing synthesized report with forest-only stopping achieves 66.2%, accounting for 17.4 percentage points of the overall gain. This illustrates a practical benefit of workflow control, although the gain is specific to this host defect. We retain the native prompts unchanged and report that, excluding this pair, the average accuracy gain across the remaining 13 pairs is still 3.06 percentage points.
W2.4. Offline cost and latency. We now report the full offline cost per host–dataset pair, including native trajectory collection, answer labeling, synonym merging, and GRPO rollouts (Table 11; see Q3). Trajectory collection executes the native host on the fitting split; suitable existing logs can be reused when available. These one-time preparation and training costs are reported separately from online inference costs. We have also verified inference times against recorded execution logs and revised the corresponding tables and text (Tables 1 and 9, Appendix F.1).
Rebuttal for EzYA on Questions
Q1. Prior work and the DPForest ablation. See W1.1 and W1.2.
Q2. Relation to Almeida (2026). TRIM’s single-pass scorer was developed independently before Almeida (2026) appeared, and all results in Tables 1–3 use this design. At each decision point, the host exposes a finite set of legal actions that the controller scores in one forward pass. The citation denotes conceptual similarity, not design provenance; we have revised the corresponding sentences in the Introduction and Related Work. The scoring head is standard, as in reward models and multiple-choice classifiers. Our contribution is the history-grounded framework for optimizing existing agent workflows, rather than the scoring head itself.
Q3. Offline cost, break-even, and labels.
- TRIM optimizes an existing agent rather than solving tasks on its own, so its offline setup depends on the host. Historical trajectories are collected by running the original host on the fitting cases with its configuration unchanged. Together with reference-based evaluation of the final answers, these executions provide TRIM’s evidence and supervision; no step-level intervention labels are required, only the correctness of final answers. The collection cost per case therefore equals the host’s native cost, whose mean tokens and time are reported in Table 9. We measured the remaining offline cost for each pair in Table 11 (calls, tokens, and GPU-s for labeling, synonym merging, and GRPO). All of these costs are paid once per host–dataset pair, whereas the savings recur on every query. The break-even point therefore depends on how often the optimized host is used: the more queries it serves, the more the one-time cost is amortized.
- When labels are delayed, TRIM can be refitted as they arrive, since it uses them only offline; in clinical settings, labels such as discharge diagnoses arrive naturally after each admission. Labels are also a prerequisite for the host itself: medical agents are developed and validated on labeled data, and TRIM uses the same reference labels.
- Break-Even: We computed the break-even point in host and tool calls according to the Table 11 offline cost. Dividing each pair’s total offline calls by the per-query reduction in M gives the following. Full TRIM (SFT+GRPO) breaks even after about 1.1×10⁵ queries on MedAgent-Pro/CXR and 2.4×10⁵ on MedAgents/SLAKE, up to 1.8×10⁶ on MDAgents/MedQA. The median over the 13 pairs with savings is 7.2×10⁵, or 42–264 times the size of the fitting set. These figures exclude trajectory collection when existing logs can be reused; including it adds at most 4%. Most of this cost comes from GRPO, which accounts for 92–97% of offline compute but mainly buys accuracy, not further savings. On DiagGym, GRPO reduces M by only 0.086 relative to SFT, yet raises accuracy by 3.14 points. SFT alone breaks even far sooner, at 1.9–27 times the fitting-set size including trajectory collection. That is about 1.9×10⁴ queries on MedAgents/MedQA and 1.2×10⁴ on MedAgents/SLAKE. It keeps most of the call reduction: on DiagGym it reaches 50.4% of the 52.0% achieved by SFT+GRPO. Practitioners can therefore choose SFT for fast amortization, or SFT+GRPO when the additional accuracy justifies a longer deployment. Two caveats apply. Controller forwards are excluded on both sides: each is one scoring pass without generation. On pairs where online tokens increase, most notably DiagGym (4,028 to 8,382), there is no token-based break-even, and the savings show up in host calls and measured latency instead. MedRAX and MMedAgent never break even because TRIM barely changes their workflows. We have added the per-pair values to Appendix F.6.
Q4. Why SFT+GRPO.
- Neither a zero-shot decision model nor a rule on forest statistics matches the trained controller on DiagGym (Table 1). A threshold rule on forest statistics (Forest-T) barely changes the native run (52.24%, M = 5.336). Prompting the untrained backbone with the same evidence (Forest-P) reaches 52.84% at M = 3.944, and general-purpose backbones do worse: Qwen2.5-7B falls below native accuracy (51.15%), and Qwen2.5-1.5B executes both examinations for every patient (M = 5). After SFT+GRPO, all three backbones reach 56.59–56.95% at about half the native calls (Table 10).
- GRPO’s gain is largest where intervention actions matter: on DiagGym it raises accuracy from 53.57% to 56.71%. The 18 jumps and 62 requests counted earlier understate this. The controller issues 398 requests in total: 62 add an examination absent from the native run, and the other 336 change the order of examinations the native run also performs. Order matters because the frozen host reads results sequentially: ordering lipase before CBC alone raises accuracy from 54.05% to 56.59% (Appendix F.5). Accordingly, masking REQUEST lowers accuracy by 2.54 points, and masking JUMP by 0.60 points (Table 2).
Rebuttal for EzYA on Critical Weaknesses
We thank the reviewer for the detailed feedback and for recognizing TRIM’s practical, host-agnostic formulation and failure analysis. The suggested references and comments on terminology and cost have helped us clarify TRIM’s positioning and evaluation. We address each weakness (W) and question (Q) below.
W1.1. Relation to prior work. TRIM addresses a different problem: improving an existing functioning agent’s workflow instead of building a new one. Puppeteer learns which agent to activate next, while ToolOrchestra learns to select and invoke models and tools. Yi & Song (2026) learn a general harness control layer from offline rollouts, without historical evidence tied to specific workflow positions. In contrast, our TRIM operates within an existing host’s workflow and legal transitions, learning when to continue, stop, skip a stage, or request an additional operation while keeping the host’s models frozen. Its distinguishing feature is the combination of workflow-indexed historical evidence and a learned intervention policy for existing medical agents. To our knowledge, TRIM is the first framework to combine workflow-indexed historical evidence with learned runtime interventions for existing medical-agent workflows. We have expanded Related Work to clarify these distinctions.
W1.2. DPForest contribution.
W1.3-W1.5. “Tree”, “Forest”, and “Host”.
W2.1. Task mismatch. We agree that on MedQA and SLAKE, where all information is given upfront, control reduces mainly to stopping multi-agent deliberation. Such QA stopping is also a core target of TRIM: repeated discussion accounts for most of these hosts’ cost (e.g., 26.9 of 36.91 calls per question on MedAgents/MedQA go to voting, revision, and readout; Appendix F.3). Sequential evidence gathering is evaluated in two settings. DiagGym provides verified outcomes for every examination path; its two examinations are a limitation, but they allow every path to be enumerated and analyzed (Figure 3, Table 2). MedAgent-Pro on CXR provides a longer tool-based workflow, checking guideline-derived indicators one at a time across 12 findings with 40 calls per native study, where TRIM reduces M to 28.14 while raising balanced accuracy from 85.38% to 88.50%. We acknowledge that richer sequential-diagnosis benchmarks would further strengthen the evaluation.
W2.2. Missing baselines. Since TRIM optimizes an existing agent, we compare it with that host’s native workflow, including its built-in stopping and routing mechanisms. These include unanimous approval in MedAgents, weighted voting in ReConcile, and consensus checks in DyLAN; TRIM reduces calls across all three (Table 3). DiagGym and MedAgent-Pro additionally evaluate control of sequential evidence gathering. These comparisons assess TRIM’s added value over existing host mechanisms. TRIM is complementary to routing and cascading rather than a replacement for them. We have not compared against general-purpose model cascades or routers such as FrugalGPT or RouteLLM, which choose among models rather than control steps within a single agent’s workflow.